Tuesday, 2014-03-11

clarkbsdague: jogo 2014-03-10 23:56:33,409 Pushing 732795 log lines. is the problem00:00
clarkbthat is from the libvirtd logs00:00
*** esker has joined #openstack-infra00:01
clarkbI think we must have a lot of those http://logs.openstack.org/89/74889/6/check/check-tempest-dsvm-savanna-postgres-nv/1b275b7/logs/libvirtd.txt.gz00:01
jeblairclarkb: that is several lines.00:01
clarkbjeblair: yeah its several orders of magnitude more than what I expect00:01
* jogo hopes he doesn't crash his browser00:02
clarkbhttp://logs.openstack.org/14/77814/1/check/check-tempest-dsvm-full/b1beec5/logs/libvirtd.txt.gz is a nicer file00:02
clarkbmuch smaller00:02
jeblairclarkb: maybe we should have a "wc -l" check on log lines in the gate.00:02
SergeyLukjanovOh, my phone is warming up trying to open this00:02
clarkbSergeyLukjanov: hahahaha00:02
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: dequeue abandoned changes  https://review.openstack.org/6546700:02
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Cancel a build even if not found  https://review.openstack.org/7939100:02
clarkbSergeyLukjanov: careful00:02
* StevenK waits for "Now my phone is glowing red"00:03
clarkbthings that are silly, running watch on a pidfile, but it has freed me up to do other things00:03
clarkbjogo: though that looks like dims change so maybe not the root of all evil00:04
clarkbI just noticed that it has a worker way bogged down00:05
jogololz 60MB uncompressed00:06
*** rfolco has joined #openstack-infra00:07
fungijeblair: so you're saying <1300 seems like a better option for minimizing impact? i can do 1200 on wednesday instead in that case00:08
fungiapparently the project "wakes up" at ~9am usa east coast time00:09
fungiinteresting metric00:09
openstackgerritJames E. Blair proposed a change to openstack-infra/gear: Be more careful with locks  https://review.openstack.org/7946900:10
fungii'm very curious why the graphite graphs seem to be in edt even though i have *all* my systems set with utc as local time. is firefox trying to be way, way, way too smart?00:11
clarkbfungi: yes  Ithink so00:11
clarkbfungi: it was showing me MST at one point00:11
fungilike, callout to some external service which geocodes your source ip address based on estimated timezone of your service provider too smart?00:12
clarkbno I don't think that smart00:12
clarkbbut maybe?00:12
clarkbalso DST00:12
clarkbI have been awake since what feels like 5am00:12
jeblairfungi: i would prefer earlier; i feel like people are already plenty busy by the time i wake up.  but i don't want to push you into a time you aren't comfortable with.  if that's the case, then i fully support just doing this on friday00:12
fungii'm perfectly comfortable whenever. SergeyLukjanov: let's do 12:00 utc on wednesday. i'll send an announcement00:13
* fungi spent decades doing 3am maintenance on sundays and wednesdays. this is a cake walk00:13
jeblairDefault: America/Chicago00:16
jeblairgraphite ^00:16
*** weshay has joined #openstack-infra00:16
jeblairso it's on CST because it hasn't restarted since the switch00:16
*** blamar has quit IRC00:17
fungiwhere does it get "TIME_ZONE" from, and can we correct that to utc somewhere?00:17
jeblairi think they made that a lot better in the latest version (using the browser more), but that requires django 1.400:17
*** atiwari has quit IRC00:17
jeblairfungi: yeah, a local settings file, i'm working up a patch00:17
fungijeblair: oh--cool, thanks@00:17
jeblairhttp://graphite.readthedocs.org/en/latest/config-local-settings.html and http://en.wikipedia.org/wiki/Zoneinfo#Names_of_time_zones00:18
jeblairsuggest that it should probably be "Etc/UTC" ?00:18
fungianswering the question of "why aren't these graphs correct for my local time zone?" is a lot easier if the answer is "utc, suck it up" rather than "chicago, _of course_"00:18
* jeblair cancels his move to chicago00:19
fungiEtc/UTC yeah00:19
fungithough i think posix may also just include a bare "UTC" on top of that00:20
jeblairyeah, this is the var they got rid of on master, so this will hopefully be a short lived fix (once we move to 14.04)00:20
fungii know on some fairly ancient posix-compliant systems i used to just feed in "UTC" (with no prefix) and they would dtrt00:20
jeblairokay, i just tested 'Etc/UTC' in production00:21
jeblairlooks like that worked00:21
*** weshay has quit IRC00:21
jeblairnow i just have to figure out how to get it to stop using the nonsensical "12AM" and "12PM"00:21
fungibest place to test. also, shoot from the hip00:21
*** matsuhashi has joined #openstack-infra00:21
fungi12am and 12pm are ancient earth terms for 00:00 and 12:0000:22
fungii learnt that in my stasis pod instructional programs00:22
clarkbjogo: fwiw looking at indexing rates and so on, I think something changed recently in the logs we are indexing00:23
jeblairit's totally 12 hours before noon!  no it's definitely 12 hours after noon!  is it ever actually noon?00:23
clarkbjogo: we are actually doing a good job from a rate perspective which makes me think we bumped the amount of data to index somewhere00:24
jogothe libvirt file?00:24
clarkbjogo: it might be libvirt or something else00:24
*** yamahata__ has joined #openstack-infra00:24
fungijeblair: noon is irrelevant when we all move subsurface to avoid being bathed in deadly solar radiation00:24
clarkblibvirt stood out to me because we spent a bunch of time indexing that one file but that is dims test change00:24
clarkbjogo: so its impact may be small overall00:24
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Have graphite use the One True Timezone.  https://review.openstack.org/7949500:24
jeblairfungi: The sooner the better says this reporter!00:25
jogoclarkb: ahh Hmm00:25
devanandafor the python ironic agent project, what's the deciding factor whether to use a separate launchpad group/bugs/etc vs. share with ironic?00:26
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add a script to manage IRC perms  https://review.openstack.org/7848300:26
fungidevananda: laziness00:27
fungidevananda: or, to use a synonym, convenience00:27
fungi(whichever works better for your community)00:27
devanandaif there's no policy on it, we'll be convenient00:28
*** sarob_ has quit IRC00:28
fungiplease be convenient. it leaves you more time to help us fix things! ;)00:28
*** bhuvan_ has quit IRC00:28
*** bhuvan has quit IRC00:28
clarkbjogo: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all logs.o.o seems like it may be pushing more bits recently00:28
*** sarob_ has joined #openstack-infra00:28
fungiclarkb: ~1kjph was the tipping point we saw before between "logstash keeps up" and "logstash falls behind"00:30
clarkbfungi: that sounds about right00:30
fungiand we did hit that around when the steady climb began, but i figured it would have recovered by now00:30
*** ianw has quit IRC00:31
*** ianw has joined #openstack-infra00:32
jeblairyeah, we've been << 1kjph for >2h now00:33
*** sarob_ has quit IRC00:33
*** blamar has joined #openstack-infra00:34
*** nati_uen_ has quit IRC00:35
jeblairbtw, i find it very interesting that there are now _several_ openstack manual jobs stuck on queued jobs, and they all seem to be either openstack-config-reference or openstack-user-guide-admin.00:37
*** sarob_ has joined #openstack-infra00:37
*** thomasem has joined #openstack-infra00:38
*** wchrisj has quit IRC00:39
jeblairclarkb: what's your latest thinking on logstash?  anything i can help with?00:39
*** esker has quit IRC00:39
clarkbjeblair: if folks want to grep through log-worker-* logs to see which files are taking a long time or having lots of lines we might be able to pinpoint why things have changed00:39
*** CaptTofu has quit IRC00:39
clarkbthe backend throughput seems to be fine so I think it is something further upstream like files grew00:39
*** esker has joined #openstack-infra00:39
StevenKHeh. Zuul has two gate jobs marked as waiting for the window that are running jobs00:40
jeblairStevenK, clarkb: ^ that seems bug-like.00:40
fungiStevenK: that can happen if they were already running before it shrank the window?00:40
clarkbfungi: yes that00:41
*** jungleboyj has joined #openstack-infra00:41
clarkbfungi: we won't kill jobs that already started because we may as well try to use those resources00:41
jeblairso more of a ui bug.00:41
*** wchrisj has joined #openstack-infra00:41
StevenKI'd agree with that. The windowing stuff doesn't seem to "fit" well in the UI00:42
jeblairclarkb: logstash worker 03 is dead00:42
jeblairaccording to graphite00:42
*** sarob_ has quit IRC00:42
*** yamahata has joined #openstack-infra00:42
clarkbjeblair: huh it is connected, but I believe it (watchdog won't catch it if it is still connected)00:43
*** wchrisj has quit IRC00:43
jeblairclarkb: i just mean the host is doing no work00:43
clarkbjeblair: ya00:43
*** thomasem has quit IRC00:43
clarkbjeblair: 11 and 12 too00:43
*** esker has quit IRC00:44
clarkbjeblair: 14 and 16 recently flapped, but I don't see anything in the logs about 3,11,1200:45
jeblairclarkb: yeah.  also, some hosts are moving a lot more data than normal, but maybe those hosts are just dealing with that very large job you found?00:45
clarkbjeblair: want to dump the jvm for the logstash process there?00:45
clarkbjeblair: or picking up the slack00:46
jeblairclarkb: will do00:46
openstackgerritA change was merged to openstack-infra/config: Have graphite use the One True Timezone.  https://review.openstack.org/7949500:46
clarkbjeblair: then I think we should restart the logstash-indexer service on 3,11,1200:46
portantejogo: you asked about devstack/swift changes?00:46
clarkbjeblair: also possible that the spike in traffic there is related to a spike in the size of the logs00:46
jeblairclarkb: so i have to install the jdk on 0300:47
jeblairit's going to install lots of packages00:47
clarkb:/ I think that should be ok00:47
clarkbits not great but not terrible either00:47
jeblairyeah, those hosts are easy to rebuild00:47
jeblairoh yeah it's jruby00:49
jeblairclarkb: ~root/stack on 0300:49
portantejogo: see commit f19ccb6 in devstack00:49
fungijruby? is that some mutant hybrid offspring of the two languages i'm secretly thinking of?00:50
clarkbjeblair: thanks, I will dig through that though probably not today and submit a big upstream if necessary00:50
jeblairfungi: heh, it's a pretty common way to run ruby, actually, esp. in 'enterprise' environments00:50
clarkbjeblair: I do see that no deadlocks were detected00:50
jeblairfungi: it's ruby running in the jvm00:50
fungiso, like, i already have this java application running in a jvm, and now i think it might be hipper if it grew a little ruby too?00:51
jeblairfungi: it's much easier to drop a ruby app in a tomcat container as a war than to figure out how to actually run a ruby server.  :)00:51
jheskethclarkb: okay, so just (re)reviewing your patch... I forget now, how would the override location get its refs? can gerrit push them to other locations?00:51
fungijeblair: without being run out of town on rails. got it00:51
clarkbjhesketh: yup, you need to setup gerrit replication in your gerrit serve rto replicate to the other location00:52
*** Mithrandir has quit IRC00:52
jeblairclarkb: yeah, the stacktrace isn't looking terribly helpful since it's mostly just ruby interpreter stuff00:52
jeblairclarkb: there's a little bit like we can see one thread is waiting on an epoll, but there's no context for what that thread might be doing00:53
fungijhesketh: bonus points for spotting the race condition in that design00:53
*** Mithrandir has joined #openstack-infra00:53
jogoportante: thanks, not sure how that would break things though00:53
jeblairrather, several threads waiting on epoll00:53
jheskethfungi: which now?00:53
fungijhesketh: sometimes gerrit falls behind on replication00:54
jogoportante: its a red herring ahh00:54
jogoSTDOUT: 2014-03-10 22:40:50.423 26911 DEBUG root [-] HTTP PERF: 0.00207 seconds to GET /sdb1/503/AUTH_64dd76f9ce8f4c15b2c81d3cc8ec3da3?format=json) getresponse /opt/stack/new/swift/swift/common/bufferedhttp.py:128 (txn: tx1219402b41d040dbbebc2-00531e3f72)00:54
jeblairclarkb: oh, wait some of those actually have elasticsearch methods....00:54
jheskethoh yeah, that's what the poll would be for00:54
fungiright, polling would help. i should look at the patch00:54
jeblairfungi: it's not as funny if you do that00:55
jheskethit's already there :-)00:55
fungi'tis what i figured after you said it ;)00:55
*** wchrisj has joined #openstack-infra00:55
fungiclarkb gets the bonus points in that case00:56
jheskethindeed :-)00:56
jeblairclarkb: the two threads that are doing this look mildly interesting:00:57
jeblair - org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame)00:57
clarkbjeblair: and there are a bunch of threads polling00:58
clarkbjeblair: this UTC is messing with me. I am going to head home now. I think you can safely `restart logstash-indexer` on those nodes whenever you are ready00:59
jeblairclarkb: ok01:00
clarkbjeblair: I have a hunch that the es client lib is buggy01:00
clarkbresulting in this bevhaior and hte behavior that needs a watchdog01:00
jogoclarkb: silly elasticSearch question01:01
*** wenlock has quit IRC01:01
clarkbjogo: sure01:01
jogohow do I search for the strong "503 "01:01
clarkbstrong "503 "?01:01
jogowhen I try the spaces seem to be stripped01:01
jogosearch for the string*01:01
clarkboh, you mean verbatim01:01
clarkbquoting it should work01:01
clarkbmessage:"503 "01:01
*** SumitNaiksatam has joined #openstack-infra01:02
clarkbsince you have to unquote if you just want terms01:02
jogomessage:"503 " AND filename:"logs/syslog.txt" AND syslog_program:"proxy-server"01:02
jheskethFYI, turbo-hipsters mirror problems are solved by pointing to openstacks pypi01:02
jogodoesn'tw ork01:02
jeblairclarkb: maybe there was nothing wrong with the indexer01:03
jeblairclarkb: the workers are not running on 03 because the oom killer hit them01:03
jeblairclarkb: what's your current target workers/node?01:03
fungijhesketh: awesome! now is the time when i make the obligatory statement about pypi.openstack.org being on a system shared with hosting a lot of other stuff, a single point of failure, an incomplete mirror, not behind a cdn like pypi.python.org not providing https with a server certificate pip can validate, and so on01:04
jheskethyes, but it's significantly more useful than a mirror behind a broken CDN01:04
fungijhesketh: no comment ;)01:05
dstufftBroken CDN?01:05
clarkbjeblair: oh gah. 2 is safe, 3 helps keep queue down01:05
jeblairclarkb: there's a flaw in that.  :)01:05
clarkbjeblair we should probably just go with 2 for now01:05
clarkbjeblair yes I know01:06
fungijhesketh: definitely provide dstufft with your symptoms so it does not stay broken01:06
*** fcarpenter has joined #openstack-infra01:06
dstufftis jhesketh mikal on twitter01:06
dstufftthere's a person in here named that too01:06
fungidstufft: they're able to throw crumpled wads of paper at one another, i believbe01:07
jheskethdstufft: we work together but are not the same people01:07
jeblairwhich is actually how twitter works too01:07
dstufftI was talking to mikal on twitter but then got distracted by RL stuff :/01:07
fungijeblair: i thought twitter was limited to those little slips of paper they used in fortune cookies?01:07
jheskethfungi: actually we'd have throw the paper a very long way as we live in different states01:07
dstufftjhesketh: so I'm like a PyPI administrator and python-infra person01:08
jheskethah cool01:08
jheskethyeah mikal was poking at our pypi mirror01:08
fungijhesketh: oh, you and mikal share edges of a large desert then01:08
clarkbjeblair can you check 11 12 and 16 too?01:08
dstufftjhesketh: can you give me full output of the blowy uppy01:08
clarkbI am leaving office now01:08
jeblairclarkb: i will check all of them01:08
jheskethdstufft: I'll put something useful together01:09
jeblairclarkb: i will cause all of the hosts to be running 2 workers01:09
dstufftif this is using openstack's mirror script then it's even more useful because i'm also a pip developer and IIRC the mirror script uses pip :V01:09
jogoclarkb: any ideas?01:09
jeblairjogo: about what?01:09
fungidstufft: the openstack pypi-mirror script totally uses pip01:09
fungia ton01:09
*** rfolco has quit IRC01:09
fungino idea if that's what jhesketh and mikal are using though01:10
*** Daisy_ has joined #openstack-infra01:11
jogojeblair: how to get message:"503 " AND filename:"logs/syslog.txt" AND  syslog_program:"proxy-server01:12
jogoto use those spaces around the 50301:13
dstufftperiodic reminder that openstack should get rid of it's external urls and then just use bandersnatch :D01:13
fungidstufft: we have fewer and fewer every day01:13
fungisoon we'll be at the point of forking/reimplementing the remainder, i suspect01:14
clarkbbandersnatch doesnt solve wheels thoygh01:14
fungidoes bandersnatch not mirror wheels?01:14
fungior does it just not generate platform-specific wheels for you on the fly while it mirrors?01:15
dstufftit doesn't generate01:15
*** bada_ has quit IRC01:15
fungimaybe that's a feature we want to contribute to it when the time comes01:15
*** emagana has joined #openstack-infra01:15
*** CaptTofu has joined #openstack-infra01:15
fungirather than reinventint the...01:16
fungiround things01:16
*** dkliban has joined #openstack-infra01:19
jeblairfungi, clarkb: i restarted all the workers but i don't think they're getting jobs01:19
jeblairmaybe they aren't restarted...01:20
jeblairoh, i think they don't like to be restarted right after the indexer01:20
jeblairwill do another pass01:21
fungii don't think i've restarted the logstash workers before, bit i'll pull up these useful javascript-based utilities clarkb recommended and see whether they say anything helpful01:21
clarkbjeblair ya if tcp 9999 isnt open they get cranky01:21
clarkband jruby does that slowly01:22
*** wchrisj has quit IRC01:22
jeblairokay gearman rerports 32 online01:22
*** rpodolyaka has quit IRC01:23
*** blamar has quit IRC01:23
fungiboth bigdesk and elasticsearch-head show pictures which incorporate workers 1-1601:23
jeblairyeah, they are all running01:24
jeblairi'm seeing a lot of really big libvirt files01:24
jeblair938219 lines01:24
jeblairi think that's a big problem01:24
jeblairbut i have to go to dinner now01:24
fungigo eat01:24
fungii wonder whether something changed in our libvirt logging01:25
clarkbits possible01:25
fungimillion-line logfiles?01:25
clarkbor nova made it cranky01:25
*** rpodolyaka has joined #openstack-infra01:25
*** wenlock has joined #openstack-infra01:28
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Remove false positives from query 1240256  https://review.openstack.org/7950301:28
*** thuc has joined #openstack-infra01:30
*** thuc_ has joined #openstack-infra01:30
*** thuc has quit IRC01:34
jogojeblair: there has bee na libvirt failure spike recently http://status.openstack.org/elastic-recheck/01:37
*** rfolco has joined #openstack-infra01:37
fungiwhich could explain an uptick in libvirt log sizes, if they're full of failspam01:38
jogofungi: so the version of libvirt  appears to not have changed01:39
jogoI wonder if the kernel has01:39
fungijogo: oh, maybe... rackspace supposedly redid their base images last week?01:39
jogofungi: how do we confirm01:39
*** nosnos has joined #openstack-infra01:40
*** alexpilotti has quit IRC01:40
fungiLinux py3k-precise-hpcloud-az1-2518397 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux01:40
fungiLinux py3k-precise-rax-dfw-2518514.slave.openstack.org 3.2.0-60-generic #91-Ubuntu SMP Wed Feb 19 03:54:44 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux01:40
fungiwell, wrong image variants, but the precise kernels we use on them should be consistent at each provider anyway01:41
fungibuilt remarkably close together01:41
jogothis may be nothing, the spike may just be a spike in development01:42
fungior spacetime compression due to a passing black hole01:42
jogooh wait01:43
clarkbno libvirt thing is anamolous01:43
jogounless kibana derped out on me01:43
jogoeither my kibana died or that started on 3/701:44
jogoright around the spike01:44
jogomy kibana did die01:44
jogobut still01:44
jogocan someone else confirm01:46
*** emagana has quit IRC01:49
znsHi folks - may I get input on pep257 testing tools in requirements? https://review.openstack.org/#/c/79054/01:49
jogofungi: yup new kernel on 3/701:51
jogoor at least no no hits for message:"kernel_version=3.2.0-60-virtual" AND filename:"logs/devstacklog.txt" before then01:51
*** rpodolyaka has quit IRC01:52
*** rpodolyaka has joined #openstack-infra01:52
*** dkliban has quit IRC01:52
jogoahh 3.2.0-59-virtual worked better01:52
jogoclarkb: ^ I think we found the issue01:53
*** wchrisj has joined #openstack-infra01:53
*** mgagne has quit IRC01:53
fungizul will want to hear about it when he's awake (or maybe he'd rather not)01:53
jogofungi: about to head home and AFK for a bit, I'll file a bug and can you follow up01:55
*** thuc has joined #openstack-infra01:55
jogonot even sure where to file the bug01:55
*** thuc_ has quit IRC01:55
jogofungi: ^ I didn't know where to file it01:57
fungiand i, similarly, have no idea what to do with it now that you have ;)01:58
jogowendar: ^ any ideas01:58
fungiwe'll find a home for it, after some shopping around, have no doubt01:58
jogofungi: so this is really bad for nova01:58
jogonova essentially has a massive bug we have to release with01:58
jogounless dims can pull some magic01:59
* jogo hopes dims is magical01:59
fungigo go gadget 3.2.0-61?01:59
jogofungi: in other news W screen-ceilometer-collector.txt.gz10-Mar-2014 W I'M02:02
jogo9.7M WAT02:02
fungi9.6m is complaining about too-old mongodb probably02:02
jogo/opt/stack/new/ceilometer/ceilometer/openstack/common/db/sqlalchemy/session.py:526: DeprecationWarning: BaseException.message has been deprecated as of Python 2.602:02
jogoyeah not cool02:02
fungi(unless you're running python 2.5)02:02
*** antonym has quit IRC02:03
*** antonym has joined #openstack-infra02:03
*** khyati has quit IRC02:06
jogolibvirtd.txt.gz11-Mar-2014 W  I'M02:07
jogobesides the paste fail02:07
* clarkb eats dinner and cries02:08
StevenKSaving to: 'screen-ceilometer-collector.txt.gz’02:08
StevenK    [                  <=>                  ] 118,647,609  491KB/s02:08
jogoStevenK: try libvirtd!02:08
StevenKI'd prefer Apache to send me the gz02:09
StevenKRather than now 140MiB of text02:09
*** Ryan_Lane has quit IRC02:10
*** Ryan_Lane has joined #openstack-infra02:10
*** rpodolyaka has quit IRC02:10
jogoStevenK: does firefox work with those files?02:10
clarkbStevenK it will if you ask for it02:11
clarkbwget and curl dont by default02:11
StevenKSilly things02:11
*** yamahata has quit IRC02:11
*** yamahata__ has quit IRC02:12
*** Ryan_Lane has quit IRC02:14
*** reed has quit IRC02:16
jogofungi: during this rename can we sort out the future of gantt?02:18
jogoif not what is blocking that02:18
fungijogo: have we determined that there's something preventing nova cores who don't care about gantt to ignore/not subscribe to reviews for it?02:18
fungiput it on the agenda for tomorrow's infra meeting https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting and we can hash it out if it's urgent to get added02:19
*** fcarpenter has quit IRC02:21
fungijogo: though the nova program probably should unfriend it officially first regardless... https://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n1402:21
fungithat's a lot more official than some prefix on a url02:21
lifelessjogo: why you no like gantt :(02:22
lifelessjogo: (teasing, mostly)02:22
*** rfolco has quit IRC02:24
*** xchu_ has joined #openstack-infra02:27
* fungi is sort of caught up on some of the things he promised people he would do today, so sort of disappearing for a while to go nap02:27
*** zns has quit IRC02:28
*** mgagne has joined #openstack-infra02:28
*** dstufft|laptop has joined #openstack-infra02:29
lifelessfungi: https://review.openstack.org/#/c/76588/ - any chance of a review of this, and a release perhaps ?02:29
lifelesswe're hit by it every week or so and have to restart geard...02:29
*** rpodolyaka has joined #openstack-infra02:30
*** zhiwei has joined #openstack-infra02:31
*** derekh has quit IRC02:33
*** thuc has quit IRC02:33
*** thuc has joined #openstack-infra02:33
*** zhiwei has quit IRC02:36
*** thuc has quit IRC02:38
*** mgagne has quit IRC02:38
*** zhiwei has joined #openstack-infra02:38
*** zhiyan_ is now known as zhiyan02:51
*** malini is now known as malini_afk02:52
*** mgagne has joined #openstack-infra02:52
*** harlowja is now known as harlowja_away02:54
*** Sukhdev has joined #openstack-infra02:57
*** mgagne has quit IRC02:58
*** rpodolyaka has quit IRC02:58
*** dkliban has joined #openstack-infra03:04
*** harlowja_away is now known as harlowja03:06
*** vkozhukalov has quit IRC03:09
*** fifieldt has joined #openstack-infra03:09
*** dkliban has quit IRC03:13
*** CaptTofu has quit IRC03:14
*** khyati has joined #openstack-infra03:16
*** dkliban has joined #openstack-infra03:16
*** krotscheck has quit IRC03:19
*** luisg has quit IRC03:22
*** dcramer__ has joined #openstack-infra03:22
*** luis_ has joined #openstack-infra03:22
*** zhiwei has quit IRC03:22
*** gokrokve has quit IRC03:22
*** zhiwei has joined #openstack-infra03:22
*** dripton has quit IRC03:23
*** dripton has joined #openstack-infra03:24
harlowjainteresting, has anyone seen jenkins do 'sudo: 3 incorrect password attempts ERROR: gate_hook failed' , this then causes the job to fail03:25
* harlowja not trying to guess the password (i swear)03:26
StevenKharlowja: Haha03:26
StevenKharlowja: That seems a little wrong for a jenkins job03:26
harlowjadef seems odd, maybe already open bug03:26
*** caleb_ has joined #openstack-infra03:27
StevenKI would have expected passwordless sudo, TBH03:27
harlowjaya, pretty weird03:27
*** gokrokve has joined #openstack-infra03:27
harlowjamaybe someone was changing the password or something recently03:28
*** lttrl has joined #openstack-infra03:29
harlowjaanyways, https://bugs.launchpad.net/openstack-ci/+bug/1290669 i guess03:30
harlowjamy guess, mr.jenkins forgot the password03:31
harlowja*mr jenkins is pretty old afterall03:32
caleb_Does our instance of Gerrit expose the REST API? I get a 404 at https://review.openstack.org/projects.03:35
*** Daisy_ has quit IRC03:35
harlowjacaleb_ not afaik03:35
harlowjacaleb_ although u can do it via ssh, https://github.com/gd-imelnikov/gerda/blob/master/bin/gerda#L193 (is an example that lists projects)03:35
pleia2caleb_: we're still running 2.4, which doesn't have it (upgrade to 2.8 is in the works)03:35
*** enikanorov_ has joined #openstack-infra03:36
caleb_harlowja, pleia2: Thanks.03:37
caleb_pleia2: won't quote you later but any idea on when the 2.8 upgrade will land?03:37
clarkbharlowja: thats an interesting fail. there are a bunch of sudos prior to that one that succeed03:37
clarkbharlowja: not sure how that happened03:37
*** enikanorov has quit IRC03:38
harlowjaclarkb mr.jenkins fogot i think03:38
clarkbharlowja: http://logs.openstack.org/14/70914/6/check/check-tempest-dsvm-full/56f2985/logs/devstack-gate-setup-host.txt.gz no I think it may be related to the users being su'd to03:39
clarkbtempest and stack both want a tty for some reason but root is happy without it03:39
clarkbwhich must be another image thing03:39
pleia2caleb_: we're pretty close, zaro would probably know more estimate-wise03:40
harlowjaclarkb there is a previous command for 'sudo useradd -U -s /bin/bash -m tempest'03:40
*** matsuhashi has quit IRC03:41
harlowjaand corresponding tempest ALL=(root) stuffs03:42
clarkbharlowja: those are rules for tempest to sudo03:42
clarkbnot to sudo to the tempest user03:42
*** thuc has joined #openstack-infra03:44
*** thuc has quit IRC03:48
harlowjaclarkb hmmm03:49
harlowjawonder why its doing that then03:49
*** amcrn has quit IRC03:50
clarkbharlowja: best guess is we have require tty in our sudoers file now for not root03:51
* clarkb logs into some machines03:52
jogoso the gate seems to moving pretty slow these days03:53
harlowjamr.jenkins take a nap jogo03:53
jogoand 9 patches in post-gate03:53
jogoahh good ol' resource starvation03:54
zarocaleb_: patches for gerrit 2.8 upgarde are up on gerrit.  just need approvals.03:54
jogoI wonder what happens if we drop the recheck rate03:54
jogosdague: ^03:54
jogowe just don't have enough capacity it seems like03:55
clarkbjogo: tests are failing03:55
clarkbbecause sudo03:56
jogoclarkb: ?03:56
harlowjajogo https://bugs.launchpad.net/openstack-ci/+bug/129066903:56
clarkbwe have an hour old dfw image I bet its related to that03:56
jogoclarkb: ahh well we have been this way fora few days I think03:57
jogoerr thats a lie03:57
jogothere was the weekend03:57
clarkbwe merged the second most commits ever last week iirc03:57
jogotrue but things were slowish03:57
jogoso things aren't so bad we don't clear out the queue03:57
jogoso delay shouldn't hurt the flow03:58
jogoclarkb: long as gate empties out every night we don't get a buildup03:58
clarkbright but if every test fails that doesn't happen03:59
jogoclarkb: also http://jogo.github.io/gate/'03:59
jogojobs be less stable03:59
jogowhich I think is the damn kernel03:59
clarkbno its the sudo thing...04:00
*** markmcclain has quit IRC04:00
*** pcrews has quit IRC04:00
jogoclarkb: that is the most recent part of the spike04:00
jogosee http://status.openstack.org/elastic-recheck/04:00
clarkbI mean libvirt mgiht be related04:00
jogolibvirt march 7yh04:00
jogoclarkb: oh so anyway logstash04:00
clarkbjogo: what do you want me to do?04:01
jogothat line keeps rising04:01
jogoclarkb: a miracle?04:01
clarkbmake nova stop hurting libvirt04:01
clarkbor stop indexing libvirt. those are your options04:01
jogoclarkb: so that rise is newer then the kernel bump04:01
*** jepoy has joined #openstack-infra04:01
* jogo goes through git logs04:01
clarkbI don't think kernel bump is to blame honestly04:01
clarkbits more likely a nova change04:01
clarkbor some combination of the two04:01
clarkbubuntu does a pretty good job with their lts kernels04:02
clarkband that is an lts kernel too so upstream halp04:02
jogoadded logs maybe a combination04:02
jogorise in bug 1254872  lines up with kernel change exactly04:02
jogoclarkb: are we 100 sure the issue is the libvirt logs?04:05
clarkbjogo: yes its several orders of mangitude more data we must filter and index04:05
clarkbI mean 99.99% sure04:05
clarkbjogo: when I looked at the es systems they were indexing at record rates04:05
clarkbbecuase we were shoving a ton more data through it04:05
clarkbbasically the index pipe is only so big04:05
clarkbwhen you suddenly go from tens of thousands of log lines per job to million range stuff grinds to a halt04:05
jogoso timeline loks like this started last night?04:05
jogooh wait04:05
jogoit ahppend this afternoon04:05
jogoI wonder ...04:08
*** david-lyle has joined #openstack-infra04:08
clarkbjogo: that wouldn't surprise me if it is the cause04:10
*** CaptTofu has joined #openstack-infra04:10
*** yamahata__ has joined #openstack-infra04:11
jogoclarkb: in that case my vote is pull from ES04:11
clarkbjust delete the lines in the client yaml config04:11
jogoI can do the patch so you can review04:11
clarkbbut first I need to finish debugging this sudo problem04:11
clarkbbecause nothing is happening until that gets fixed04:11
*** yamahata has joined #openstack-infra04:12
clarkbI think jeblair's sudo change broke it04:12
clarkbhttps://review.openstack.org/#/c/72777/ is the change04:12
*** pcrews has joined #openstack-infra04:13
*** markmcclain has joined #openstack-infra04:14
clarkbhttps://review.openstack.org/#/c/72777/2/modules/jenkins/files/jenkins-sudo.sudo I don't see anything wrong with that, jenkins user can run ALL commands as root with no password04:14
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Revert "Add logstash rules to parse libvirtd.txt"  https://review.openstack.org/7953504:14
clarkbwhere does requiretty come into affect04:14
*** krotscheck has joined #openstack-infra04:15
jogoclarkb: ^04:15
*** nati_ueno has joined #openstack-infra04:15
*** CaptTofu has quit IRC04:15
clarkbjogo: honestly a 30MB libvirt log file compressed doesn't really help anyone04:15
clarkbjogo: I am good with that change,but maybe we should consider not logging so verbosely04:15
jogoclarkb: we are debugging libvirt failing04:16
jogoit helps libvirt developers and we have one04:16
jogoso its for him04:16
clarkbbut he doesn't need it 15k times a day04:16
clarkbflip that switch when you need it04:16
jogo   disable client side libvirt debug logging04:16
jogo    and tune server side libvirt logging to the values that danpb04:16
jogo    suggested would be useful on the openstack-dev mailing list.04:16
jogoclarkb: I want to give danpb all the data he needs, at least for a day or two04:17
jogothis really should be an experimental job04:17
clarkbbut its not useable data04:18
clarkbyou need hadepp04:18
*** pcrews has quit IRC04:18
jogoclarkb:  perhaps but I am not comfortable making that judgement call without danpb ... I think stop indexing is the quickest solution, unless we want to add an experimental job with this instead04:20
jogoclarkb: also unless dtroyer is up this is easier to merge at this hour04:20
clarkbI don't plan on merging ^ tonight04:21
clarkbits late and the sudo thing is more important04:21
*** sweston has quit IRC04:21
jogoahh in that case04:23
*** Daisy_ has joined #openstack-infra04:23
*** talluri has joined #openstack-infra04:24
*** wchrisj has quit IRC04:24
jogoso the sudo issue only pops up when you try becoem a different user bsides root04:25
jogo2014-03-11 03:18:08.207 | + sudo -u stack mkdir -p /opt/stack/new/.pip04:25
clarkbI have held 255148904:26
clarkbjogo: yes, jenkins sudo fails04:26
clarkboh duh I see it now04:26
jogofrom http://logs.openstack.org/14/70914/6/check/check-tempest-dsvm-full/56f2985/logs/devstack-gate-setup-host.txt.gz04:27
*** sdake has quit IRC04:27
clarkbya I should have a fix proposed shortly04:27
clarkbgah the node I held doesn't have the derpage yet04:28
SlickNikI came here to report sudo issues with tempest tests, but you guys are already on it. :)04:30
SlickNikMuch thanks!04:30
morganfainbergclarkb, ++ woot and infra is on it already!04:30
jogoclarkb: so I can't write a ER query for it yet, becuase the long aren't in ;)04:31
clarkbjogo: you don't need one04:32
jogofor http://status.openstack.org/elastic-recheck/data/uncategorized.html04:33
jogoanyway I don't want to distract you from fixing it04:33
openstackgerritClark Boylan proposed a change to openstack-infra/config: Allow jenkins to sudo as any user.  https://review.openstack.org/7953604:34
clarkbmordred: jeblair fungi SergeyLukjanov ^ so that is a bit important04:34
clarkbI don't see any immediate issues with allowing it to sudo as any user04:34
mordredclarkb: looking04:34
clarkbsince the one user we intended on allowing before was root04:34
mordredclarkb: ok04:35
clarkbmordred: do you think we should just push that through and trigger some image rebuilds?04:37
clarkbI have a hunch fungi and jeblair aren't going to be around soon04:37
clarkbSergeyLukjanov might but he doesn't sleep04:37
clarkboh darn I didn't put the bug in the commit message /me fixes04:38
mordredclarkb: yah04:38
openstackgerritClark Boylan proposed a change to openstack-infra/config: Allow jenkins to sudo as any user.  https://review.openstack.org/7953604:39
clarkbI will give it long enough to check test then approve04:39
mordredclarkb: I support you +Aing it in04:39
*** matsuhashi has joined #openstack-infra04:40
*** sarob_ has joined #openstack-infra04:42
*** esker has joined #openstack-infra04:43
*** zehicle_at_dell has joined #openstack-infra04:45
*** zns has joined #openstack-infra04:45
*** sweston has joined #openstack-infra04:46
*** sarob_ has quit IRC04:46
*** lttrl has quit IRC04:47
*** esker has quit IRC04:47
clarkbgah the gate is eating all the nodes04:52
clarkbmordred: should I god mode submit it?04:52
*** krotscheck has quit IRC04:52
clarkbI really hate doing that but given the situation...04:52
mordredclarkb: yes. you want me to?04:52
clarkbmordred: sure04:53
clarkbthen I can trigger a bunch of nodepool image udpates04:53
*** wchrisj has joined #openstack-infra04:54
*** Sukhdev has quit IRC04:57
*** esker has joined #openstack-infra04:57
*** krotscheck has joined #openstack-infra04:57
clarkbmordred: were you going to do it?04:57
mordredclarkb: yah04:58
clarkbok it hasn't merged yet so was wondering04:58
openstackgerritA change was merged to openstack-infra/config: Allow jenkins to sudo as any user.  https://review.openstack.org/7953604:59
mordredclarkb: done04:59
*** arosen-home has joined #openstack-infra05:01
arosen-homeLooks like there was some interesting CI event within the hour :/05:01
arosen-homeGuess you guys already know all about it though.05:02
StevenKarosen-home: The fix was merged 4 minutes ago05:03
StevenKNow we have to wait for image builds05:03
*** wchrisj has quit IRC05:04
*** oda-g_ has joined #openstack-infra05:05
*** jepoy_ has joined #openstack-infra05:05
StevenKclarkb: What's the rough ETA for less explosions?05:06
clarkbStevenK: huh half hour ish for new images then nodes will need to cycle out05:06
clarkbif images come up qucik enough I can manually delete things05:06
clarkbbut its getting late here and i had a really early morning05:06
mordredclarkb had to do an HP thing for me05:07
* mordred hangs head05:07
clarkbmordred: it wouldn't have been so bad if DST didn't happen05:07
*** jepoy has quit IRC05:08
*** caleb_ has quit IRC05:09
oda-g_It seems the gate become very bad. All check and gate jobs are going to fail.05:12
clarkboda-g_: yes see scrollback, basically a bad sudo rule for jenkins merged. fix is inflight05:13
jeblairclarkb: need help?05:13
*** gokrokve has quit IRC05:13
oda-g_clarkb: Thanks for your infomation.05:14
*** caleb_ has joined #openstack-infra05:14
*** oda-g_ has quit IRC05:14
clarkbjeblair: I think I have it udner control. mordred pushed the fix through and I have image updates going05:15
clarkbjeblair: only devstack-* nodes seem to be affected so focusing on those05:15
jeblairclarkb: what's the deal with logstash?05:15
clarkbjeblair: sdague made a change to devstack that made libvirt logs blow up05:15
clarkbjeblair: I told jogo either we stop indexing libvirt or fix the logs being massive05:16
jeblairclarkb: we need to make them not be massive because they also take up space on logs.o.o05:16
mordredjeblair: ++05:16
clarkbjeblair: image updates are running in screen on nodepool.o.o now05:16
clarkbjeblair: yup05:16
jeblairclarkb: or we need to not copy them at all05:16
clarkbthis vmware image makes me cranky05:17
jeblairclarkb: have a link to the devstack change?05:17
clarkbmordred: we should kill that with fire imo05:17
clarkbjeblair: ya one sec05:17
jeblairclarkb: no kidding05:17
clarkbjeblair: https://review.openstack.org/#/c/79346/05:17
mordredyeah. I want to kill the vmware image05:17
mordredwhat about a filter for *.vmdk? we're never going to consume those in the gate itself05:18
jeblairmordred: that would work if you don't want to make the argument it shouldn't be in devstack anyway05:18
clarkbI can make that argument :)05:18
mordredjeblair: well, I do - but I have +2 on d-g05:18
clarkbrax-iad was cranky I finally have it building the second image now05:19
clarkball devstack-precise* images are building across all azs now05:19
jeblairthat logic makes me sad05:19
clarkbjeblair: did you see the sudo fix?05:20
jeblairclarkb: do you recall the procedure for "we have a broken image in nodepool?"05:20
clarkbjeblair: ? delete the old images?05:20
jeblairclarkb: delete the broken image05:21
clarkboh right because it will fall back gah05:21
jeblairyeah.  pretty much an immediate fix to such a problem05:21
jeblairclarkb: i think we should still do that, and soon.05:21
clarkbjeblair: ok, I will do that now05:21
*** caleb_ has quit IRC05:21
*** nicedice has quit IRC05:21
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Skip downloading giant VMware images  https://review.openstack.org/7953705:22
jeblairclarkb: because as soon as your new images build, nodepool will think it's okay to delete the good ones (because they would be N-2)05:22
SergeyLukjanovclarkb, /me agreed with https://review.openstack.org/#/c/79536/05:22
mordredSergeyLukjanov: are you STILL awake? or waking up?05:22
clarkbjeblair: yup05:22
SergeyLukjanovmordred, already wake up :)05:22
* mordred hands SergeyLukjanov a sandwich05:23
SergeyLukjanovmordred, and coffee pls ;)05:23
SergeyLukjanovmordred, /me need to order kitchen furniture today heh05:24
*** esker has quit IRC05:24
jeblairmordred: i think we should see if the rest of the devstack core would be okay with removing that from devstack first, then go with your approach if it fails05:24
clarkbjeblair: done, all images are either more than a day old or less than an hour old05:24
jeblairclarkb: cool05:25
clarkbfor devstack-precise*05:25
SergeyLukjanovclarkb, yay!05:25
clarkbthe other images don't need sudo so should be fine and i am not rebuilding there either05:25
SergeyLukjanovjeblair, IMO the more safe option is just have a list of needed images in d-g05:25
jeblairSergeyLukjanov: I don't want to reimplement devstack in devstack-gate; it should be minimal, and devstack-gate should run it as close to the default as possible05:26
jeblairSergeyLukjanov: part of the reason for that is the intention is to test what developers and users actually get when they run devstack05:26
jeblairSergeyLukjanov: basically, aside from saying things like "i want to run (or don't want to run) sahara" devstack-gate shouldn't do much configuration of devstack05:28
mordredjeblair: works for me05:28
mordredjeblair: should I submit a patch to devstack to kickstart the conversation?05:29
SergeyLukjanovMy point is that devstack could install more images than we need for d-g05:29
jeblairmordred: sure; i added dtroyer to the config review too so he'll see that background.  and maybe this ping.  :)05:29
*** sweston has quit IRC05:30
jeblairSergeyLukjanov: perhaps, and that's okay -- but the question is whether every possible image needs to be in devstack05:30
SergeyLukjanovFor ex. In savanna we already have more than 10 useful images05:30
SergeyLukjanovYup, agreed05:30
SergeyLukjanovBah, sahara05:31
jeblairSergeyLukjanov: i want _at least_ every image we might possibly use in the gate to be in devstack, because that's how they get cached on our test nodes05:31
*** dkliban has quit IRC05:31
mordredjeblair: main objection I could see is that vmware has stepped up to the plate with third party testing - and it's possible they're also using devstack to do that05:32
jeblairSergeyLukjanov: and of course, since they are in the gate, that means they are basic configurations that developers should be able to test easily05:32
SergeyLukjanovAnd dib built images too?05:32
jeblairmordred: yeah, but maybe there's an easy way for them to get that information into devstack without it actually being in the tree05:32
jeblairmordred: it's just that i don't see most developers using vmware images05:32
jeblairmordred: because it's proprietary software, and we're writing free software05:33
jeblairamong other reasons05:33
mordredjeblair: I say this mainly because of following the fact that there are vsphere logic branches05:33
mordredjeblair: I mean, I agree with you ...05:33
mordredjust bringing up the possible pushback05:33
jeblairmordred: i'm aware of possible pushback; that's why i'm not saying no to your change, just that we ought to take a look at the idea.05:34
jeblairmordred: if the current state of the art is the best way for developers to test the vsphere nova driver, that's fine05:35
*** dkliban has joined #openstack-infra05:35
jeblairwe can do the cat-and-mouse thing05:35
mordredit's also REALLY easy for the vsphere folks to drop an IMAGE_URL into their localrc05:36
*** zehicle_at_dell has quit IRC05:36
*** kashyap has left #openstack-infra05:38
*** kashyap has joined #openstack-infra05:38
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Temporarily stop copying libvirt log  https://review.openstack.org/7954005:42
jeblairclarkb: ^05:42
clarkbjeblair: lgtm05:42
*** sarob_ has joined #openstack-infra05:43
jeblairmordred, SergeyLukjanov: ^ want to aprv that?05:43
jeblairi approved it05:48
*** sarob_ has quit IRC05:49
*** yjiang51 has joined #openstack-infra05:49
*** yjiang51 has left #openstack-infra05:49
jeblairthere are only 3 jenkins log workers05:50
jeblairone on 13, 2 on 1605:50
clarkbjeblair: I wonder if they are ooming due to the size of that file05:51
clarkbit does load them into memory to unzip05:51
StevenKclarkb: Are the image builds all done?05:51
clarkbStevenK: no, but I killed broken images so things should be happier nowish05:51
clarkbjeblair: it could be better about that, unzipping incrementally and writing to disk, but this has never been a problem before05:52
*** amcrn has joined #openstack-infra05:52
*** sweston has joined #openstack-infra05:53
*** yaguang has joined #openstack-infra05:53
StevenKclarkb: It was a fairly handy way to clear out gate and check :-P05:53
jeblairclarkb: it's streaming; it should just do it 1 line at a time..05:53
*** nati_ueno has quit IRC05:54
jeblairclarkb: (i don't mean to say i think that's what it is doing, but rather i think what it probably should do if it isn't)05:54
StevenKHaha, so much stuff has failed that Jenkins thinks that check-tempest-dsvm-postgres-full now takes 15 minutes.05:54
*** rpodolyaka has joined #openstack-infra05:54
jeblairclarkb: just spot checking 12:05:54
jeblair2014-03-11 01:25:02,021 Retrieving: http://logs.openstack.org/55/74955/2/check/check-tempest-dsvm-full/48e7357/logs/libvirtd.txt.gz?level=INFO05:54
jeblair2014-03-11 01:40:10,452 Decompressing gzipped source file.05:54
clarkbjeblair: yeah it could do that too, maybe not one line at a time since gzipped but some chink at a time05:55
jeblairthat's the tail of one of the worker logs05:55
*** nati_ueno has joined #openstack-infra05:55
jeblairi don't see the oom killer being activated though05:55
jeblairclarkb: so i think we lost information on why that might have died05:55
clarkbjeblair: I would've expected tracebacks if not oomkiller, but I would agree with your statement given the log05:56
jeblairyeah me too05:56
StevenKRargh, nodes haven't cycled out05:57
jeblairclarkb: oh wait, it did oom, i just missed the log entries05:57
jeblairclarkb: yeah, both workers on 12 hit the oom killer05:58
*** CaptTofu has joined #openstack-infra05:58
*** ildikov_ has quit IRC05:59
jeblairStevenK: jobs started after 05:30 should be ok05:59
jeblairand yet that's clearly not the case since https://jenkins04.openstack.org/job/check-tempest-dsvm-full/6131/ failed06:00
*** krotscheck has quit IRC06:00
clarkbjeblair: I think there were a few nodes out there on the old image06:00
SergeyLukjanovjeblair,  https://review.openstack.org/79540 lgtm06:00
StevenKjeblair: I retried my nova patch about 6 minutes ago06:01
jeblairclarkb: there must have been a few building; the one i just looked at was building for 43 minutes06:01
jeblairclarkb: i forgot rax can take so long06:01
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Temporarily stop copying libvirt log  https://review.openstack.org/7954006:02
*** CaptTofu has quit IRC06:02
StevenKhttps://jenkins05.openstack.org/job/check-tempest-dsvm-neutron-heat-slow/1288/ for example06:03
jeblair(restarting tests ^)06:03
clarkbI should probably rewrite the log workers to use co routines and deal with a line at a time (ish)06:03
jeblairStevenK: yeah, it might take a bit longer than i thought due to some nodes taking a long time to build06:04
clarkbI need to crash shortly06:04
clarkbwas a long day and I find myself unable to think properly06:04
*** matsuhashi has quit IRC06:05
jeblairclarkb: me too06:06
jeblairi think we need to wait until after https://review.openstack.org/79540 merges, then dump the logstash queue and restart the workers06:06
jeblairclarkb: i mean, someone does, not necessarily us.  :)06:07
clarkbya I am going to go read red mars now so that the brain can shutdown proeprly06:07
clarkbI am running windows 9506:07
jeblairclarkb: i don't understand what you are saying except i'm pretty sure it all just means it's your bedtime.  good night.  :)06:08
clarkbjeblair: you don't remember windows 95 needing to be shut down properly?06:08
SergeyLukjanovclarkb, good night :)06:08
*** matsuhas_ has joined #openstack-infra06:08
clarkbit would be quite cranky if you dropped power out from under it06:08
SergeyLukjanovclarkb, I remember06:08
*** harlowja is now known as harlowja_away06:08
clarkb"It is now safe to shutdown the computer" or some such06:09
jeblairclarkb: no, actually, but i remember that ext2 behaved similarly.  :)06:09
SergeyLukjanovclarkb, I was about 8 years old and it was painful to not forget to do it06:09
jeblairfungi, mordred: (probably fungi unless mordred stays up late).  Can you please do the following after https://review.openstack.org/#/c/79540/ merges?  http://paste.openstack.org/show/73108/06:14
jeblair(that's about 1.5 from now probably)06:15
mordredjeblair: I'll do it if I'm up06:17
SergeyLukjanovjeblair, if you have instructions than I can do such stuff when everyone else sleeping06:24
jeblairSergeyLukjanov: unfortunately it's a root-access task.  we should work toward getting you root access.  :)06:24
SergeyLukjanovjeblair, :)06:25
SergeyLukjanovheh, my phone is updated to iOS 7.106:25
SergeyLukjanovand now call screen contains only circles... wat?06:26
openstackgerritZhiwei Chen proposed a change to openstack-infra/config: Rename a StackForge project  https://review.openstack.org/6478806:26
*** vkozhukalov has joined #openstack-infra06:27
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Rename savanna group to sahara  https://review.openstack.org/7954306:31
*** dstufft|laptop has quit IRC06:32
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Remove logging of sahara projects to #savanna  https://review.openstack.org/7954506:32
jeblairsome of the new images that clarkb kicked off seem to be in use now and also seem to be working06:33
*** nati_ueno has quit IRC06:34
SergeyLukjanovjeblair, that's awesome06:34
*** skraynev has quit IRC06:34
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Document bad-image procedure for nodepool  https://review.openstack.org/7954606:34
SergeyLukjanovjeblair, oh, that's much easier than I thought06:37
SergeyLukjanovjeblair, re 7954606:37
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Remove savanna from logstash/jenkins-log-client.yaml  https://review.openstack.org/7954806:38
jeblairSergeyLukjanov: yeah, we keep forgetting that it's that easy.  but nodepool is designed for this; hopefully if we all review those docs we'll remember.  :)06:38
*** afazekas has quit IRC06:39
*** denis_makogon has joined #openstack-infra06:39
*** fifieldt has quit IRC06:41
*** wenlock has quit IRC06:43
*** gyee has quit IRC06:43
*** arosen-home has quit IRC06:44
*** skraynev has joined #openstack-infra06:48
*** khyati has quit IRC06:51
mordredjeblair: bah06:58
*** rpodolyaka has quit IRC07:01
*** rpodolyaka has joined #openstack-infra07:01
*** rpodolyaka has quit IRC07:01
*** rpodolyaka has joined #openstack-infra07:01
*** rpodolyaka has quit IRC07:02
AJaegerGood morning, we still have many manual changes queued at http://status.openstack.org/zuul/ - since 40h07:03
AJaegerThose could be easily killed, every job does the same...07:03
StevenKDo we think the gate has recovered now?07:07
*** yolanda has joined #openstack-infra07:07
*** mkoderer has joined #openstack-infra07:17
*** arosen has joined #openstack-infra07:20
*** afazekas has joined #openstack-infra07:28
*** zns has quit IRC07:37
*** davidhadas has quit IRC07:39
*** Daisy_ has quit IRC07:42
*** jamielennox is now known as jamielennox|away07:44
*** CaptTofu has joined #openstack-infra07:46
*** flaper87|afk is now known as flaper8707:50
*** arosen has quit IRC07:50
*** vkozhukalov has quit IRC07:50
*** CaptTofu has quit IRC07:51
*** markmcclain has quit IRC07:53
*** rcarrillocruz1 has joined #openstack-infra07:57
*** mrodden1 has joined #openstack-infra07:58
*** rcarrillocruz has quit IRC07:59
*** mrodden has quit IRC08:00
*** mrda is now known as mrda_away08:01
*** jlibosva has joined #openstack-infra08:03
*** pblaho has joined #openstack-infra08:05
*** eharney has joined #openstack-infra08:06
*** rcarrillocruz has joined #openstack-infra08:07
*** rcarrillocruz1 has quit IRC08:08
*** e0ne has joined #openstack-infra08:10
*** dizquierdo has joined #openstack-infra08:13
*** yamahata has quit IRC08:14
*** yamahata__ has quit IRC08:15
*** jgallard has joined #openstack-infra08:16
*** Daisy_ has joined #openstack-infra08:20
GheRiverodid git.openstack.org go down last night?08:21
*** jepoy_ has quit IRC08:23
*** jepoy has joined #openstack-infra08:23
*** jcoufal has joined #openstack-infra08:24
*** jepoy_ has joined #openstack-infra08:26
*** morganfainberg is now known as morganfainberg_Z08:27
*** jepoy has quit IRC08:29
*** yamahata__ has joined #openstack-infra08:30
*** yamahata has joined #openstack-infra08:30
*** jepoy has joined #openstack-infra08:33
*** jepoy_ has quit IRC08:37
*** rlandy has joined #openstack-infra08:41
*** bogdando has quit IRC08:42
*** jepoy has quit IRC08:43
*** rcarrillocruz1 has joined #openstack-infra08:46
*** rcarrillocruz has quit IRC08:46
*** andreaf has joined #openstack-infra08:48
*** rossella_s has joined #openstack-infra08:50
*** spredzy has quit IRC08:51
*** spredzy has joined #openstack-infra08:51
*** denis_makogon_ has joined #openstack-infra08:55
*** vponomaryov has joined #openstack-infra08:58
*** Pradnya has joined #openstack-infra08:59
*** Pradnya has left #openstack-infra08:59
*** denis_makogon has quit IRC08:59
*** Pradnya has joined #openstack-infra08:59
PradnyaHi, I am contributing to Openstack first time. I am trying to submit the test cases in openstack Tempest. I am facing issue with SSH key authentication. I am getting "Permission denied" error when I am issuing the command "git review -s". I am following this link: "https://wiki.openstack.org/wiki/GerritWorkflow". Please help in submission in tempest.09:00
*** jcoufal has quit IRC09:00
*** jcoufal_ has joined #openstack-infra09:00
*** derekh has joined #openstack-infra09:01
*** jcoufal_ has quit IRC09:02
*** yamahata has quit IRC09:04
*** yamahata__ has quit IRC09:04
ruhePradnya: hi. did you register your email and public key in gerrit?09:05
*** pbelanyi has joined #openstack-infra09:05
*** jpich has joined #openstack-infra09:06
PradnyaI registered my email id and SSh key09:07
ruhecan you try this command:09:07
ruhessh -p 29418 review.openstack.org09:07
Pradnyai am getting the error "Permission denied" when i am giving "git review -s"09:07
*** denis_makogon_ has quit IRC09:08
*** matsuhas_ has quit IRC09:08
Pradnyacloud@idmlocal:~/tempest$ ssh -p 29418 review.openstack.org Permission denied (publickey).09:08
PradnyaError is pasted in paste.openstack.org09:08
*** hashar has joined #openstack-infra09:09
*** matsuhashi has joined #openstack-infra09:09
ruhePradnya: is your username "cloud" registered in gerrit settings at https://review.openstack.org/#/settings/ ?09:09
Pradnyamy username is Hemanth on gerrit09:10
*** jooools has joined #openstack-infra09:10
ruheso, either need to update your username in gerrit settings09:10
ruheupdate your gitreview settings09:11
ruheusername should be the same on your local devbox and in gerrit09:11
Pradnyalet me try09:11
*** zhiwei has quit IRC09:13
*** Ryan_Lane1 has quit IRC09:13
*** jcoufal has joined #openstack-infra09:15
PradnyaI have craeted the user with my user name. "Hemanth@idmlocal" will this do?09:18
*** johnthetubaguy has joined #openstack-infra09:19
*** Daisy_ has quit IRC09:19
*** lahoucine has left #openstack-infra09:21
*** unicell has quit IRC09:22
*** ildikov_ has joined #openstack-infra09:24
*** julienvey has joined #openstack-infra09:24
*** alff has joined #openstack-infra09:25
ruhePradnya: it shouldn't contain hostname part09:26
PradnyaWill try that09:26
*** unicell has joined #openstack-infra09:28
*** CaptTofu has joined #openstack-infra09:35
*** alff has quit IRC09:35
*** zhiwei has joined #openstack-infra09:37
*** CaptTofu has quit IRC09:40
*** johnthetubaguy has quit IRC09:44
*** johnthetubaguy has joined #openstack-infra09:44
*** sarob_ has joined #openstack-infra09:48
*** jp_at_hp has joined #openstack-infra09:49
*** Pradnya has quit IRC09:49
*** sarob_ has quit IRC09:54
*** johnthetubaguy1 has joined #openstack-infra09:57
*** johnthetubaguy has quit IRC10:01
*** saju_m has joined #openstack-infra10:05
*** yamahata__ has joined #openstack-infra10:08
*** yassine has joined #openstack-infra10:09
*** yamahata has joined #openstack-infra10:09
*** xchu_ has quit IRC10:15
*** saju_m has quit IRC10:18
*** rcarrillocruz has joined #openstack-infra10:18
rcarrillocruzhmm, just cloned zuul git repo10:20
rcarrillocruzran tox10:20
rcarrillocruzand got 'tox.ConfigError: ConfigError: substitution key 'posargs' not found'10:20
rcarrillocruzis this an upstream bug?10:20
*** rcarrillocruz1 has quit IRC10:20
*** eharney has quit IRC10:23
ihrachysrcarrillocruz: downgrade tox to 1.610:26
ihrachysit's an issue with tox 1.710:26
*** matsuhashi has quit IRC10:29
*** rcarrillocruz1 has joined #openstack-infra10:30
*** saju_m has joined #openstack-infra10:30
*** rcarrillocruz has quit IRC10:31
*** andreaf has quit IRC10:35
rcarrillocruz1k, thx ihrachys10:37
sdaguejeblair / clarkb: we made the libvirt change so that we could finally debug the top bug in the gate10:44
*** yaguang has quit IRC10:47
*** markmc has joined #openstack-infra10:48
*** dcramer__ has quit IRC10:48
*** sarob_ has joined #openstack-infra10:51
*** saju_m has quit IRC10:54
*** sarob_ has quit IRC10:55
*** bobba has joined #openstack-infra11:00
*** bobba is now known as GnomeKing11:00
*** vogxn has joined #openstack-infra11:01
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: ZMQ Event Publisher plugin support added  https://review.openstack.org/7756111:01
*** vogxn has quit IRC11:02
*** CaptTofu has joined #openstack-infra11:05
*** johnthetubaguy1 has quit IRC11:09
*** dkliban has quit IRC11:13
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin  https://review.openstack.org/7360111:14
*** mbacchi has joined #openstack-infra11:14
*** dizquierdo is now known as dizquierdo_afk11:16
*** dkliban has joined #openstack-infra11:16
*** hashar has quit IRC11:17
*** rcarrillocruz has joined #openstack-infra11:17
*** hashar has joined #openstack-infra11:19
*** rcarrillocruz1 has quit IRC11:20
*** talluri has quit IRC11:26
*** jgallard has quit IRC11:26
*** talluri has joined #openstack-infra11:26
*** eharney has joined #openstack-infra11:27
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Ruby metrics plugin support added  https://review.openstack.org/7352811:28
*** rcarrillocruz1 has joined #openstack-infra11:28
*** rcarrillocruz has quit IRC11:29
*** talluri has quit IRC11:30
*** rcarrillocruz has joined #openstack-infra11:32
*** andreaf has joined #openstack-infra11:35
*** rcarrillocruz1 has quit IRC11:35
*** malini_afk is now known as malini11:37
*** ociuhandu has quit IRC11:41
openstackgerritSean Dague proposed a change to openstack-infra/config: stop indexing libvirt logs  https://review.openstack.org/7958611:42
sdagueSergeyLukjanov / fungi ^^^^^^^^^11:42
*** dkliban has quit IRC11:47
*** pradnya has joined #openstack-infra11:49
SergeyLukjanovsdague, hey11:50
*** sarob_ has joined #openstack-infra11:52
kashyapsdague, Hm, that bug 1254872 has been around for quite a while, I noticed some discussion flying by b/n you and danpb on #openstack-nova, any root-cause analysis on it yet?11:53
sdaguekashyap: nope11:53
sdaguedanpb said he needed more logs11:54
SergeyLukjanovsdague, https://review.openstack.org/#/c/79586/ approved11:54
pradnyaHi, I am contributing to Openstack first time. I am trying to submit the test cases in openstack Tempest. I am facing issue with SSH key authentication. I am getting "Permission denied" error when I am issuing the command "git review -s". I am following this link: "https://wiki.openstack.org/wiki/GerritWorkflow". Please help in submission in tempest.11:55
pradnyaplease help11:55
kashyapsdague, Okay, I'll check with him if I can help debug it or setup a minimal env to reproduce it, looks like a pesky case11:55
sdaguekashyap: thanks!11:56
pradnyaPlease help in tempest submission11:56
sdaguekashyap: the logs should be on in the gate now11:56
kashyapsdague, I attempted in the past couldn't just narrow it down11:57
kashyapsdague, If you have a pointer that'd be appreciated, my look ups have been darn slow :(11:57
sdaguepradnya: did you upload your public key to gerrit11:57
*** hashar has quit IRC11:57
*** lcostantino has joined #openstack-infra11:57
pradnyasdague: yes11:57
*** sarob_ has quit IRC11:58
*** ArxCruz has joined #openstack-infra12:00
pradnyasdague: http://paste.openstack.org/show/73132/12:00
sdagueare you using the right username?12:01
pradnyauser name in gerrit is : Hemanth12:02
pradnyaon my local system12:02
*** changbl has quit IRC12:02
*** pradnya has left #openstack-infra12:03
sdaguewhat is it in gerrit?12:03
sdagueoh, sorry, I see you answered that12:03
*** pradnya has joined #openstack-infra12:03
*** weshay has joined #openstack-infra12:03
pradnyagot discoonnected12:03
sdaguehttps://review.openstack.org/#/settings/ssh-keys - that lists your key?12:03
pradnyait shows correctly12:04
sdagueok, then I'm out of helpfulness. I don't actually have enough access to look at your account12:04
sdagueneed to wait for fungi to wake up12:04
*** yamahata__ has quit IRC12:04
openstackgerritA change was merged to openstack-infra/config: stop indexing libvirt logs  https://review.openstack.org/7958612:06
pradnyacan you look at logs http://paste.openstack.org/show/73132/12:06
*** aysyd has joined #openstack-infra12:07
*** yamahata has quit IRC12:09
pradnyaneed help for tempest submission12:09
pradnyadebug1: Authentications that can continue: publickey debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey)12:10
pradnyagetting error while git "review -s" : Permission denied12:11
pradnyasuccessfuly generated ssh key and uploaded to launchpad gerrit12:11
*** dims_ has quit IRC12:12
*** JuanManuelOlle has joined #openstack-infra12:14
SergeyLukjanovsdague, looks like we have a problem with reverting sqla 0.912:14
fungiwhy do we need fungi?12:14
SergeyLukjanovfungi, morning!12:14
*** dims_ has joined #openstack-infra12:14
fungiahh, stop indexing libvirt logs12:14
pradnyaHi Fungi12:16
pradnyaneed help in submiting test case in tempest12:16
pradnyadebug1: Authentications that can continue: publickey debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey)12:16
fungipradnya: most likely cause is that your gerrit username is not what you're trying to authenticate as. that seems to trip up a lot of people12:17
*** adalbas has joined #openstack-infra12:17
fungipradnya: oh, i see you already said it was the same further up12:18
fungipradnya: second most common problem is that people add their ssh keys with linebreaks in them. that will also cause problems12:18
fungithe entire key need to be one long, unbroken line12:18
pradnyadebug logs: http://paste.openstack.org/show/73132/12:19
fungisdague: should https://review.openstack.org/79586 go in before or after https://review.openstack.org/79540 ?12:19
sdaguefungi: if we stop copying the libvirt log, then we give up on the bug in question12:20
fungisdague: so you're suggesting 79586 instead of 7954012:21
sdagueI get that the logs are big, but danpb basically said - Bug 1254872  is not debugable without them12:21
sdagueyeh, just drop indexing them12:21
*** david-lyle has quit IRC12:21
fungiokay, fair enough12:21
SergeyLukjanovfungi, sdague, but what's about free space on logs.o.o?12:23
fungiSergeyLukjanov: well, once i add one more 1tb volume, that server is tapped out. rackspace won't go above 1tb on cinder volumes and their xen deployment prevents more than 14 additional virtual block devices besides the two which the server gets by default12:24
fungipradnya: [2014-03-11 10:57:43,691 +0000] 9e8fa436 Hemanth - AUTH FAILURE FROM no-matching-key12:24
fungipradnya: that's from the server12:24
fungipradnya: i'l look around in teh database to make sure things on this end match what you're seeing in the web interface for your account12:25
sdagueSergeyLukjanov: my hope is we can get some resolution quicker and turn them off12:25
SergeyLukjanovsdague, cool, just to clarify12:25
pradnyafungi : ok.any work around. We are stuck with this for 4 days. It would be of great help if I get any pointers12:26
sdaguefungi: is the logstash drop real?12:29
fungipradnya: when you go to https://review.openstack.org/#/settings/ it says Account ID 10366 right?12:29
*** wchrisj has joined #openstack-infra12:29
sdagueor is that synthetic because something restarted?12:29
pradnyafungi: yes12:29
fungisdague: no, the workers are dropping like flies because they're oom'ing when trying to index libvirt logs12:29
*** miqui has quit IRC12:29
*** wchrisj has quit IRC12:29
sdaguefungi: ok12:30
sdaguefungi: no, I mean the queue dropped right now12:30
sdaguea lot12:30
*** mburned_out is now known as mburned12:30
fungisdague: oh, that... maybe a bad snmp sample, or that might be where they restarted for the config change?12:30
*** rfolco has joined #openstack-infra12:31
openstackgerritshihanzhang proposed a change to openstack/requirements: Upgrade requirements to alembic  https://review.openstack.org/7959312:32
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: update to not use libvirt log  https://review.openstack.org/7959412:34
fungipradnya: does the output of 'ssh-keygen -y -f /home/Hemanth/.ssh/id_rsa' match what was added to https://review.openstack.org/#/settings/ssh-keys (except for the comment field at the end)?12:35
sdaguefungi: so the config change restart means we just end up dumping all the intermediate results12:35
fungisdague: i'm not sure12:36
sdaguefungi: honestly, it looks like this bug was the only one that ever needed libvirt logs anyway12:36
sdagueand the fingerprint worked fine on screen-n-cpu12:36
fungisdague: yeah, we started archiving those logs specifically because of that bug12:37
sdagueI just want to get our requirements reverted so we can actually test with global requirements again12:38
sdaguefungi: if we see a reset, can you promote - 79454,112:38
fungisdague: is that the requirements backoff?12:38
pradnyafungi: id_rsa is having public and private. We have uploaded public key in gerrit.12:38
sdaguefungi: yeh, it's a revert of the allow for 0.912:39
*** e0ne_ has joined #openstack-infra12:39
sdaguebecause it breaks ceilometer's db migrations12:39
sdaguewhich I think I've actually got the fix for12:39
fungipradnya: right, but 'ssh-keygen -y -f /home/Hemanth/.ssh/id_rsa' will tell you what the private key corresponding to /home/Hemanth/.ssh/id_rsa should be, in case the one in /home/Hemanth/.ssh/id_rsa.pub is actually incorrect for some reason12:39
sdaguehowever, we should back that off, fix devstack so it actually enforces g-r (which we stopped doing), land the ceilo fix, then try the g-r update again (so it can be tested for real this time)12:40
*** heyongli has joined #openstack-infra12:40
sdagueso I've got the first 3 patches all in flight12:40
fungisdague: awesome12:40
sdagueif I had cross project dependencies I could even tell you if #4 would work :)12:40
*** e0ne has quit IRC12:41
sdaguefungi: we just reset (all but the top item), want to do a promote?12:42
sdaguewe can lose the time on the top item12:42
fungisdague: well, if i do that cinder change is also going to get retested12:42
sdagueoh, right12:43
sdagueand they failed on unit tests12:43
sdaguenever mind12:43
pradnyafungi: I have pasted both keys : http://paste.openstack.org/show/73139/12:43
*** nosnos has quit IRC12:44
*** CaptTofu has quit IRC12:44
fungipradnya: is that first line the output from 'ssh-keygen -y -f /home/Hemanth/.ssh/id_rsa' or just the contents of /home/Hemanth/.ssh/id_rsa.pub12:44
sdaguefungi: any idea why the logstash workers are still queuing up - http://graphite.openstack.org/render/?from=-24hours&height=1024&until=now&width=1600&bgcolor=ffffff&fgcolor=000000&target=color(alias(stats.gauges.logstash.geard.queue.running,%20%27Running%27),%20%27blue%27)&target=color(alias(stats.gauges.logstash.geard.queue.waiting,%20%27Waiting%27),%20%27red%27)&target=color(alias(stats.gauges.logstash.geard.queue.total,%20%27Total%20Jobs%27),%20%278812:45
sdagueI would have expected after dropping the libvirt logs we'd be ok12:45
fungisdague: i wouldn't be surprised if there's a brief increase for some backlog which built up while they were restarting, but it's possible they're just climbing again12:46
GheRiverois there any plan to release a new version of pypi-mirror?12:47
fungiGheRivero: if you want one, i'll stick it on the agenda for today's meeting12:47
fungiGheRivero: looks like there are a couple non-cosmetic patches since 1.0.0 so it may be time for 1.1.012:48
GheRiverofungi: I could be worth it. The one actually released doesn-t support  --allow-external sources, so it's  useless :)12:49
fungiGheRivero: well, the d24f19b commit just hard-codes a whitelist for netifaces, psutil, pysendfile and pytidylib but as long as those are all you need, then i guess that helps12:50
pradnyafungi: Thanks we have corrected ssh key in gerrit manually and now its going forward12:50
fungiGheRivero: we hoped it would be temporary, but there do seem to be some holdouts still12:51
fungipradnya: you're not getting the authentication error any longer? excellent. i'm glad that exercise was worth trying then12:51
*** jhesketh has quit IRC12:51
*** mugsie has joined #openstack-infra12:52
fungisometimes people do, for various reasons, end up with public and private keys which aren't from the same set and don't realize it12:52
pradnyafungi: yes we are not getting authentication error any more. We were stuck here for 4 days. thanks a ton12:52
fungipradnya: you're welcome. that's a weird issue to diagnose and a lot of people don't think to check it12:52
*** dkliban has joined #openstack-infra12:53
*** rcarrillocruz1 has joined #openstack-infra12:53
fungisdague: it looks like that queue bump may be cresting now12:53
sdaguefungi: great12:53
pradnyafungi: true12:53
*** rcarrillocruz has quit IRC12:54
*** dkranz has joined #openstack-infra12:55
*** sarob_ has joined #openstack-infra12:55
openstackgerritA change was merged to openstack-infra/elastic-recheck: update to not use libvirt log  https://review.openstack.org/7959412:57
*** mwagner_lap has quit IRC13:00
*** sarob_ has quit IRC13:00
*** afazekas has quit IRC13:04
*** alexpilotti has joined #openstack-infra13:05
*** dprince has joined #openstack-infra13:05
*** CaptTofu has joined #openstack-infra13:07
*** yamahata__ has joined #openstack-infra13:11
*** CaptTofu has quit IRC13:11
*** jgallard has joined #openstack-infra13:13
*** hashar has joined #openstack-infra13:14
*** yamahata__ has quit IRC13:15
*** mriedem has joined #openstack-infra13:15
*** yamahata__ has joined #openstack-infra13:16
*** afazekas has joined #openstack-infra13:17
*** thuc has joined #openstack-infra13:18
*** julim has joined #openstack-infra13:18
sdaguefungi: the worker queue still looks like linear growth13:20
sdagueany chance you could go look. It would be good to know if we need to do something other than drop the libvirt indexing here13:20
sdagueespecially if there were queued events that are just killing things13:21
*** yamahata__ has quit IRC13:21
*** yamahata__ has joined #openstack-infra13:22
fungii'll see if i can figure out what jeblair and clarkb were looking at last night13:22
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Only public user fields in unauthorized requests  https://review.openstack.org/7858613:24
*** plaurin_ has joined #openstack-infra13:25
*** lukego has joined #openstack-infra13:25
lukegoHowdy! I'm testing 3rd party CI with the "Tail-f Jenkins" account on the openstack-dev/sandbox project using Jay Pipes' scripst. I don't see any votes coming from my side. Is that because the account is non-voting even for the sandbox repo?13:26
*** yamahata has joined #openstack-infra13:26
*** yamahata__ has quit IRC13:27
*** yamahata__ has joined #openstack-infra13:28
*** johnthetubaguy has joined #openstack-infra13:28
*** wchrisj has joined #openstack-infra13:28
*** yamahata__ has quit IRC13:30
*** yamahata__ has joined #openstack-infra13:31
fungilukego: the "Tail-f NCS Jenkins" account is listed in https://review.openstack.org/#/admin/groups/270,members so it should be capable of leaving +/-1 vrif votes on changes for the openstack-dev/sandbox project13:32
lukegoOK. Could be something on my side then. Thanks!13:33
fungilukego: have you tried (or observed) uploading a new change or a new patchset for an existing change on that project?13:33
lukegoDo you have any tips for how to check? I was expecting votes. have looked in every log I can think of. I do see the tests running and just not the votes going up13:33
lukegoI see the "Starting jobs..." update from my CI but nothing more13:33
lukegoNow might be a good time to setup the public logging :-)13:34
fungilukego: hmm... i'm not sure where to look for error logs from the gerrit-jenkins trigger plugin since i don't use it myself13:34
*** yamahata__ has quit IRC13:34
fungilukego: jaypipes or others who are using it might have better suggestions on what to look at13:34
plaurin_Hello everyone, I want to implement Openstack CI with Atlassian Bamboo. After some searching, I did not found a lot of information.. I found about the run_test.sh files though. Any guidelines? Is that script for 3rd party CI?13:35
lukegoI'll see how far I can get before the sun reaches their timezone13:35
*** yamahata__ has joined #openstack-infra13:35
*** afazekas has quit IRC13:36
*** mfer has joined #openstack-infra13:37
fungiplaurin_: i have no idea what an atlassian bamboo is... we do have some basic documentation on doing third-party integration testing, but without knowing what you're trying to test or which run_test.sh files you're talking about (tempest? nova? what?) it's hard to know where to send you13:38
plaurin_I'm working with Horizon for now13:40
*** mrodden1 has quit IRC13:40
*** thuc has quit IRC13:41
*** yamahata has quit IRC13:41
*** yamahata has joined #openstack-infra13:41
fungiplaurin_: so are you trying to run horizon unit/functional tests or are you trying to perform integration testing between horizon and other openstack software?13:42
*** bogdando has joined #openstack-infra13:42
plaurin_Unit tests/Func.13:43
*** zns has joined #openstack-infra13:43
*** yamahata__ has quit IRC13:43
sdagueplaurin_: honestly, you are going to need to do a ton of work to get bamboo integrated here13:46
sdagueis there really a reason for that decision?13:46
*** yamahata__ has joined #openstack-infra13:47
pradnyaHello Fungi13:48
pradnyaI am getting error while submitting the code. for "git review", i am getting error pasted at http://paste.openstack.org/show/73140/13:48
plaurin_A ton of work! Oh :P We use Bamboo for most of our work but maybe I will use Jenkins for openstack..13:49
pradnyaError is for email address that is not matching with the registered email id in gerrit. eg. remote: ERROR:  In commit 7b3c9c46a3406ad0c0b1886a2a550ba956118b81 remote: ERROR:  committer email address Hemanth@localhost.(none) remote: ERROR:  does not match your user account.13:49
fungipradnya: yes, you need to configure git to know what e-mail address it should associate with your commits13:50
*** jcoufal has quit IRC13:50
pradnyaI have set hemanth.n@tcs.com as email-address13:50
pradnyaboth in gerrit and launchpad13:50
*** yamahata__ has quit IRC13:50
fungipradnya: you need to set it in your global git config on your workstation13:50
sdagueplaurin_: so I haven't used bamboo in 7 years, however, realize that *everyone* doing openstack work, all 30 3rd party ci systems, are using jenkins13:51
*** bada has joined #openstack-infra13:51
sdagueso you'd be out on your own figuring out things from scratch13:51
sdagueincluding the reporting pieces13:51
fungisdague: (well, except for turbo-hipster)13:51
pradnyafungi: I have run " git config --global user.name "Hemanth"13:51
*** yamahata__ has joined #openstack-infra13:51
*** malini is now known as malini_afk13:52
pradnyaalso git config --global user.email hemanth.n@tcs.com13:52
plaurin_okay, I am surprised, thanks for the info13:52
fungipradnya: you probably set that after you committed the change you're trying to submit for review. if so, git commit --amend first13:52
pradnyaafter that also i am getting the above error13:53
*** smarcet has joined #openstack-infra13:53
*** wenlock has joined #openstack-infra13:53
fungipradnya: what gerrit's complaining about there is that a commit you're trying to submit for review was made by "Hemanth@localhost"13:54
fungipradnya: if you run 'git log' you'll probably see the same thing13:54
sdaguefungi: sure, it's running completely different kinds of tests though. So was ground up built for our system.13:54
fungisdague: agreed13:54
pradnyaokay. next step would be "git commit --amend" and then again running above command for config and review. Is that fine?13:55
fungipradnya: you probably shouldn't have to re-run the git config commands13:55
sdaguepradnya: you need --reset-author13:55
fungipradnya: sdague is right, actually. just a --amend won't change the author13:56
* fungi forgot about that13:56
fungiusually if you're running git commit for the first time on a system it will provide output telling you these things13:56
fungiand i tend to just cut and paste what it says13:56
*** mrodden has joined #openstack-infra13:56
pradnyaI am giving the command "git commit --amend --reset-author "hemanth.n@tcs.com"...13:57
*** zns has quit IRC13:57
pradnyais that fine?13:57
fungipradnya: you don't have to specify your e-mail address at the end of it. it should figure out which one to use from your global config13:58
*** yamahata__ has quit IRC13:59
fungithis is just fixing the commit which you made prior to setting up your git configuration correctly13:59
*** esker has joined #openstack-infra14:00
fungisince the commit predates your configuration14:00
*** zns has joined #openstack-infra14:00
*** heyongli has quit IRC14:01
*** pdmars has joined #openstack-infra14:01
*** fcarpenter has joined #openstack-infra14:02
pradnyaFungi and sdague: Thanka a lot :)14:03
pradnyaI have submitted the test case14:04
*** jgrimm has joined #openstack-infra14:04
fungipradnya: congratulations! your first commit is always the hardest. it gets a lot easier from this point on14:04
russellbfungi: can we move gantt and python-ganttclient to stackforge in the upcoming rename window?14:05
*** yassine has quit IRC14:05
*** yamahata__ has joined #openstack-infra14:05
*** sarob_ has joined #openstack-infra14:05
fungirussellb: do you have a change up for openstack/governance already?14:06
fungiwe should probably get that proposed too14:06
*** afazekas has joined #openstack-infra14:08
fungirussellb: okay, fair enough. i'm a little worried about moving a project out of the openstack namespace while it's still officially considered part of an official program, but as long as both happen in close proximity i guess that's fine. i'll bring it up on the agenda for today's team meeting14:08
*** pblaho has quit IRC14:08
russellboh, you'd rather merge the governance change first?14:08
russellbthat's fine14:08
*** gokrokve has joined #openstack-infra14:09
fungirussellb: keep in mind that melange and python-melangeclient are still in the openstack namespace14:09
fungirussellb: i don't see that causing anyone to think they have to review changes for them14:09
*** yamahata__ has quit IRC14:09
openstackgerritZiad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools  https://review.openstack.org/7905414:10
*** sarob_ has quit IRC14:10
*** alexpilotti has quit IRC14:10
fungiwhat makes them officially part of your program or not is your decision to include or not include it in the list for the governance repo14:10
russellbfungi: maybe i'm being too paranoid about what being under openstack/ says to people14:10
russellband true14:11
*** fcarpenter has quit IRC14:11
*** dkranz has quit IRC14:11
*** resker has joined #openstack-infra14:12
russellbfungi: i had gone neutral on the rename, and then the thread went this way: http://lists.openstack.org/pipermail/openstack-infra/2014-February/000879.html14:12
* fungi is woefully behind on ml reading14:13
russellbwe all are14:13
sdaguerussellb: we still have openstack/melange so I wouldn't be too concerned about what being under openstack/ means14:14
*** esker has quit IRC14:14
russellbi don't think we should have that either, heh14:14
russellbanyway, i don't feel *that* strongly about it14:14
sdagueand, honestly, if gantt's going to be a doover anyway, it probably doesn't need all the gerrit history preserved14:15
fungii think these cosmetic namespace groupings are causing more confusion than they solve, personally14:15
*** pradnya has quit IRC14:15
sdagueand could just reimport as stackforge instead of the outage rename14:15
*** changbl has joined #openstack-infra14:15
fungithough they would need a new project name since we have a lot of systems which rely on just the part afterthe slash14:16
sdagueI think that's the real question in my mind. it sounds like gantts going to end up as a do-over.14:16
sdagueso I'm not sure why we're preserving anything14:16
*** ociuhandu has joined #openstack-infra14:16
russellbpersonally i'm not sure why it's being kept14:16
russellbbut those working on it want to play with it some more14:16
russellbso *shrug* doesn't hurt anything14:16
sdaguewell, except it generates more work for the infra team, because someone wants to play :)14:17
*** atiwari has joined #openstack-infra14:17
sdagueI feel very similar to the kite discussion14:17
sdagueit has not purpose, delete it.14:17
fungiwell, deleting things from gerrit doesn't really work14:17
*** pcrews has joined #openstack-infra14:18
*** resker has quit IRC14:18
sdaguefungi: ok, that's news to me14:18
*** wenlock has quit IRC14:18
*** yassine has joined #openstack-infra14:18
SergeyLukjanovwhat's the melange project?14:19
*** alexpilotti has joined #openstack-infra14:19
sdaguesomething that predates me14:19
fungiit can be basically abandoned and left there once nobody cares about it any longer, and if we really want someone can merge a commit to master which removes all the other files and puts a deprecation notice in the readme or something14:19
fungiSergeyLukjanov: sdague: description='IPAM mangement service for Openstack' http://git.openstack.org/cgit/openstack/melange/tree/setup.py#n7814:21
fungieven the name was redundant! ;)14:21
fungier, the description14:21
fungi"ipam management"14:21
fungiisn't management what the m in ipam stands for?14:21
*** malini_afk is now known as malini14:22
*** zhiwei has quit IRC14:23
fungisdague: the logstash queue is caught back up now. none of the jenkins log workers were running and had stale pidfiles left behind14:23
fungi[Tue Mar 11 09:45:06 2014] Out of memory: Kill process 8594 (log-gearman-wor) score 688 or sacrifice child14:24
fungifrom dmesg on one of them. but they all did that looks like14:24
*** dkranz has joined #openstack-infra14:24
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews  https://review.openstack.org/7928014:25
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port  https://review.openstack.org/7928114:25
*** david-lyle has joined #openstack-infra14:26
lukegoI think my dsvm-tempest-full is stuck. any ideas for troubleshooting? The last message in Jenkins console was 30 minutes ago: Running devstack ... this takes 5 - 8 minutes (logs in logs/devstacklog.txt.gz)14:28
*** krotscheck has joined #openstack-infra14:29
fungilukego: you could log into that slave and see what's in the devstack log14:29
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews  https://review.openstack.org/7928014:29
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port  https://review.openstack.org/7928114:29
fungilukego: and connect to the screen session and see what the various services are doing14:30
*** JuanManuelOlle has quit IRC14:30
*** afazekas has quit IRC14:32
lukegodevstack log says:14:32
lukego2014-03-11 15:15:15.437 | + bash14:32
lukego2014-03-11 15:15:15.438 | ++ sudo virsh list --all14:32
lukego2014-03-11 15:15:15.439 | ++ grep instance-14:32
lukego2014-03-11 15:15:15.440 | ++ sed 's/.*\(instance-[0-9a-fA-F]*\).*/\1/g'14:32
lukego(sorry - is that too much to paste at once here?)14:32
fungilukego: we have a paste server at http://paste.openstack.org/ if you want to add more detail14:33
fungilukego: also, for devstack troubleshooting, you might have more luck asking in #openstack-qa14:33
*** markmcclain has joined #openstack-infra14:33
*** JuanManuelOlle has joined #openstack-infra14:35
*** wchrisj has quit IRC14:38
*** jcoufal has joined #openstack-infra14:40
*** GnomeKing has quit IRC14:40
*** lcheng has joined #openstack-infra14:42
*** rcarrillocruz1 has quit IRC14:42
*** reed has joined #openstack-infra14:42
*** mgagne has joined #openstack-infra14:42
*** esker has joined #openstack-infra14:42
*** rcarrillocruz has joined #openstack-infra14:43
*** zns has quit IRC14:44
jeblairfungi: good morning14:46
fungimorning jeblair14:46
*** mgagne has quit IRC14:46
*** esker has quit IRC14:47
*** wenlock has joined #openstack-infra14:47
*** afazekas has joined #openstack-infra14:47
fungiyour change last night ran into an integration test failure and didn't merge, but sdague had an alternate change to just stop indexing the file but keep collecting it for now so they can continue to troubleshoot libvirt upgrade testing14:47
*** plaurin_ has quit IRC14:48
fungialso all the logstash worker processes oom'd overnight, so i've restarted them following your paste this morning after sdague's change landed14:48
sdaguefungi: it's not even libvirt upgrade testing14:48
sdagueit's actually existing libvirt bugs14:48
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews  https://review.openstack.org/7928014:48
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port  https://review.openstack.org/7928114:48
fungijeblair: oh, and i had a (completely unsupportable) theory about why it's only openstack-manuals jobs we see hanging... as of a few weeks ago we now try to start 17 jobs for them simultaneously in the post pipeline, but frequently only end up with 16 which run... is there maybe a gearman protocol limit coming into play there?14:51
jeblairfungi, sdague: yes, we considered that, but we also need to not run out of space on logs.o.o14:52
fungi16 just seems like a suspiciously non-arbitrary number14:52
jeblairfungi: i think devstack-gate starts more14:52
fungijeblair: okay, fair enough14:52
jeblairsdague: how long do you want to have the system copying the libvirt logs14:52
sdaguejeblair: sure. however I was told by danpb that we didn't have sufficient logs to actually debug the libvirt issue which is our top ER bug14:52
jeblairsdague: because i want to help get it solved, but it's also unsustainable to be archiving 1M line log files14:53
*** Daisy_ has joined #openstack-infra14:53
sdaguejeblair: hopefully only a few days, I'm hoping that we can get enough data by then to get a fix14:53
jeblairfungi: did you get the message i left for you?14:53
SergeyLukjanovjeblair, morning14:53
jeblairfungi: here? http://paste.openstack.org/show/73108/14:54
sdaguejeblair: the biggest thing now is for es / er to recover so we can actually detect one of these fails again14:54
jeblairsdague: that was the intent14:54
sdaguethen point the libvirt folks at them14:54
sdaguejeblair: yep14:54
fungijeblair: that's what i meant when i said i restarted the workers following your paste14:55
jeblairfungi: ah cool14:55
fungijeblair: sdague made a compelling argument for not dumping the libvirt logs completely, and SergeyLukjanov had already emergency-approved it, so i didn't reverify the change14:55
fungibut did otherwise follow the restart steps14:55
*** thedodd has joined #openstack-infra14:56
jeblairsdague: so, shortly, when we no longer need the logs, how should we reduce the verbosity?14:56
jeblairsdague: is it just remove all the lines in your devstack change?14:56
sdaguejeblair: yes14:57
jeblairrussellb: is there ever going to be a gantt?14:57
*** dizquierdo_afk is now known as dizquierdo14:57
*** pdmars has quit IRC14:57
sdagueprobably will turn if off by making it a conditional that defaults false, so in the future we could turn it back on with d-g for some window easily if we felt we needed it14:58
jeblairrussellb: because i think it's silly to move it to stackforge and then move it back.15:00
*** caleb_ has joined #openstack-infra15:01
*** blamar has joined #openstack-infra15:02
*** wchrisj has joined #openstack-infra15:02
*** sparkycollier has joined #openstack-infra15:03
*** pdmars has joined #openstack-infra15:04
*** jcoufal has quit IRC15:06
*** jcoufal has joined #openstack-infra15:07
*** rlandy has quit IRC15:07
*** johnthetubaguy1 has joined #openstack-infra15:07
*** fungi has quit IRC15:07
*** rlandy has joined #openstack-infra15:08
russellbjeblair: maybe15:09
russellbjeblair: it's fine, if it's a pain for you guys, let's just leave it alone, and we'll remove it in the governance repo15:09
russellbjeblair: if there is, it will be a do-over from the current repo, if that matters to you15:09
*** johnthetubaguy has quit IRC15:10
jeblairrussellb: so i think the restrictions we have to work with are (1) repos have indepedent names even across org boundaries (so no openstack/gantt _and_ stackforge/gantt at the same time)15:10
jeblairrussellb: and (2) nothing is ever deleted15:10
*** mkoderer has quit IRC15:11
*** mgagne has joined #openstack-infra15:11
jeblairrussellb: so that probably means either a bit of git-work on gantt to effect the do-over, or a rename to something not named gantt for the current repo at some point15:12
jeblairrussellb: (we did completely replace keystone with some git magic, so it's entirely possible)15:12
*** jcoufal has quit IRC15:12
russellbwell i don't expect that to be a near term thing15:12
russellblike, not juno timeframe15:12
*** jcoufal has joined #openstack-infra15:13
jeblairrussellb: okay, that sounds far enough off and vague enough that renaming it isn't unreasonable15:14
*** rcarrillocruz1 has joined #openstack-infra15:19
*** dcramer__ has joined #openstack-infra15:21
*** rcarrillocruz has quit IRC15:21
*** jcoufal has quit IRC15:25
*** zns has joined #openstack-infra15:26
*** thuc has joined #openstack-infra15:27
*** sparkycollier has quit IRC15:30
*** fcarpenter has joined #openstack-infra15:30
*** fcarpenter has left #openstack-infra15:30
*** gokrokve has quit IRC15:33
*** miqui has joined #openstack-infra15:33
*** gokrokve has joined #openstack-infra15:33
*** gokrokve has quit IRC15:37
*** harlowja_away is now known as harlowja15:38
*** Daisy_ has quit IRC15:38
*** johnthetubaguy1 has quit IRC15:40
*** krotscheck has quit IRC15:41
*** johnthetubaguy has joined #openstack-infra15:42
*** harlowja has quit IRC15:43
openstackgerritA change was merged to openstack/requirements: Revert "Switch to SQLAlchemy 0.9"  https://review.openstack.org/7945415:43
*** jcoufal-mobile has joined #openstack-infra15:45
*** fungi has joined #openstack-infra15:46
fungifun. my connection to the server was dead a good 45 minutes without the client attempting to reconnect15:47
fungi(and yet i never lost my connection to oftc)15:48
*** harlowja has joined #openstack-infra15:48
*** sarob_ has joined #openstack-infra15:49
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Creating a new story now requires authentication.  https://review.openstack.org/7939015:50
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Fixed padding on modals.  https://review.openstack.org/7939315:50
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Clean-up the address-bar after auth  https://review.openstack.org/7895315:50
*** gyee has joined #openstack-infra15:50
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Added UI switches for authentication state.  https://review.openstack.org/7947815:50
*** zhiyan is now known as zhiyan_15:51
openstackgerritA change was merged to openstack-infra/storyboard: Only public user fields in unauthorized requests  https://review.openstack.org/7858615:51
*** khyati has joined #openstack-infra15:53
openstackgerritA change was merged to openstack-infra/config: Revoke sudo permissions before infra jobs  https://review.openstack.org/7277815:54
david-lyleI would like to add an additional Horizon gate job to check python 2.7 combined with django 1.5, what repo do I need to be looking at?15:54
sdaguejeblair / fungi: is there an easy way to put a block on requirements changes until we actually fix the testing of them?15:54
*** bogdando has quit IRC15:55
fungisdague: depends on what you mean by easy. we can revoke approval rights on the acl for the requirements project...15:55
jeblairsdague: link to change that fixes testing?15:56
fungidavid-lyle: is it in the global requirements list yet?15:56
david-lyledjango 1.6 is, want to make sure we maintain support for 1.515:56
sdaguejeblair: I believe this does - https://review.openstack.org/#/c/77815/15:57
david-lyleI added a tox env to Horizon to test py 2.7 and django 1.5 combo15:57
fungidavid-lyle: got it... the tox.ini in horizon i believe already has a section for 1.4 so you'll need to add15:57
fungiyou beat me to it15:57
*** gokrokve has joined #openstack-infra15:57
david-lylebut somewhere I need to tell the gate to run that tox env, no?15:57
AJaegerfungi, the openstack-manuals changes are still in the post gate at http://status.openstack.org/zuul/ - since 49 hours now. Can't we just remove them from the queue?15:58
sdaguefungi: revoke approval is probably good15:58
jeblairAJaeger: no way to do that15:58
sdagueI'm just concerned we're chasing to get things testing again15:58
david-lyleOr does that happen in patches post tox.ini merge?15:58
sdaguethe approvers on requirements are often not very plugged into the state of the world15:58
fungidavid-lyle: in that case, in the openstack-infra/config repo you need to add the job definition in modules/openstack_project/files/jenkins_job_builder/config/horizon.yaml15:58
sdaguefor better or worse15:58
fungidavid-lyle: and then entries in modules/openstack_project/files/zuul/layout.yaml15:59
AJaegerjeblair, is there a way to fix these?15:59
fungiAJaeger: we would have to restart zuul15:59
AJaegerjeblair, note that the post gate just publishes all guides, so each of the jobs does the same, and removing all of them except the latest would not cause any problems on the endresult...15:59
fungiAJaeger: we don't have a feature implemented to remove pending jobs directly right now15:59
*** jcoufal has joined #openstack-infra16:00
jeblairAJaeger: i know.  the ones that have run have already published, so the only effect of this is that the status screen looks messy.16:00
david-lylefungi: thank you16:00
AJaegerjeblair, the config-reference is not getting published, everything else is.16:01
sdaguejeblair: so I'm about to go run outside for a bit. I'm hoping the change moves through check -> gate fine at this point. Then I'll check in later on that one.16:01
AJaegeroh, content looks updated today - now I'm confused ;)16:01
jeblairAJaeger: the config reference ran for 6294e8b16:02
sdaguethen we can actually be testing global requirements again, which would be goodness16:02
AJaegerjeblair, ok, I'll live with the mess for now ;)16:02
openstackgerritA change was merged to openstack-infra/zuul: Cancel a build even if not found  https://review.openstack.org/7939116:02
jeblairfungi, AJaeger: ^ that's why they are stuck16:03
*** SumitNaiksatam has quit IRC16:03
*** johnthetubaguy1 has joined #openstack-infra16:04
fungijeblair: weird. if those were converted to non-maven jobs in the change that i approved, then i'm sure i deleted them with jjb on all the masters16:05
fungichecking my shell history16:05
*** johnthetubaguy has quit IRC16:05
*** weiwei has quit IRC16:06
jeblairfungi: i think it's 2 things -- those jobs are set to concurrent=false and then probably a jenkins/nodepool/gearman-plugin thing happened and one got scheduled on a dead node16:06
jeblairso the others are waiting16:06
fungiohhh... so they are supposed to still be maven jobs?16:06
jeblairi think at this point with multiple jenkins masters, we can't make any guarantees about nonconcurrency, so we should just remove that16:07
jeblairfungi: no, maven/not-maven doesn't matter for this16:07
fungigood point. i'll propose a patch to strip concurrency=1 from anything which runs on an untrusted slave16:07
fungipresumably we still want it for the jobs which run on jenkins.o.o though16:08
AJaegerfungi, yes, they are still maven jobs. Haven't converted them yet.16:08
fungibut yeah, i see them sitting in the build queue on those masters as well16:08
jeblairfungi: oh, i just wrote that16:09
fungiall the better16:09
AJaegerfungi, I plan to convert them later today to non-maven and have one question: Do you prefer one large job or smaller ones?16:09
jeblairfungi: well, i removed it from all jobs16:09
jeblairfungi: you think we should keep them on the trusted slaves for now?16:09
jeblairfungi: actually, it's the same anyway16:09
AJaegerI would do one publishing job that all maven jobs16:09
*** bogdando has joined #openstack-infra16:09
jeblairfungi: there's only one instance of each trusted slave so they can't run concurrently anyway16:10
jeblairfungi: so i'll remove them everywhere16:10
fungiAJaeger: whichever is more convenient for you16:10
fungijeblair: great point. we only have one executor on each of those trusted slaves anyway so they'll still queue up16:11
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Remove concurrency restriction from all jobs  https://review.openstack.org/7965916:11
AJaegerfungi, ok, will try going with one as I expect that to be the easiest16:11
jeblairthere is a small chance that when that change is approved, some or all of the jobs zuul is waiting on will run.16:12
fungionce jjb reruns on those masters anyway16:12
AJaegerjeblair, what does the concurrency say? Does it allow running the same post job in parallel for two changes - or only different post jobs for the same change?16:13
fungiAJaeger: it's supposed to prevent two copies of the same job from running at the same time, but only applies locally for each jenkins master (we don't have any cross-master coordination which could enforce it currently)16:14
openstackgerritAndrea Frittoli  proposed a change to openstack-infra/config: Add experimental keystonev3 pipeline  https://review.openstack.org/7931416:14
AJaegerfungi, those jobs run on different VMs? In that case there's no problem for the manual jobs with jeblair's change16:15
fungiAJaeger: so while jenkins01 wouldn't run two of that job simultaneously, there's nothing stopping jenkins02 from running the same job for a different change at the same time16:15
fungiAJaeger: yes, all those jobs run on different vms. except for a handful of jobs which we have pinned to specific trusted vms and those have only one executor apiece so can't run more than one job of any kind at the same time anyway16:16
*** vkozhukalov has joined #openstack-infra16:16
*** eharney has quit IRC16:16
*** wchrisj has quit IRC16:17
*** afazekas has quit IRC16:17
*** wchrisj has joined #openstack-infra16:17
AJaegerok, then the change is fine from the manuals side. Thanks for the explanation.16:18
openstackgerritDoug Hellmann proposed a change to openstack/requirements: pin nose to 1.3.0  https://review.openstack.org/7966116:18
*** weiwei has joined #openstack-infra16:19
AJaegerit only means I need to rebase https://review.openstack.org/#/c/77446/ ;) will do later...16:19
jeblairAJaeger: sorry :(  it's for a good cause though.  :)16:19
fungiAJaeger: the biggest risk right now from a manuals concurrency standpoint which i can see is that it's possible for a two changes to merge at roughly the same time for the same document and then finish in a different order than they merged, which leaves the manual published with slightly outdated content. but that's already the case today16:20
AJaegerjeblair, no problems. Such things happen....16:20
jeblairfungi, AJaeger: yes, to fix this correctly, we'd need some changes to zuul16:20
jeblairbut most of the time it's going to work right.  and when it is wrong, it won't be wrong for long.16:21
AJaegerfungi, I hope that we publish often enough ;)16:21
AJaegerjeblair, exactly16:21
fungiyep, i'm sure it's happened, but i've never heard anyone mention that they've noticed it16:21
AJaegerOk, will need to cycle home now and send patches later tonight. Thanks for working on the good cause ;)16:21
*** e0ne has joined #openstack-infra16:27
*** zigo has joined #openstack-infra16:29
*** e0ne_ has quit IRC16:30
fungiclarkb: jeblair: was https://launchpad.net/bugs/1290669 fallout from the sudoers changes?16:30
*** vhoward has joined #openstack-infra16:31
jeblairfungi: yes16:31
*** SumitNaiksatam has joined #openstack-infra16:31
*** markwash has joined #openstack-infra16:33
*** talluri has joined #openstack-infra16:34
*** dcramer__ has quit IRC16:34
*** jcoufal-mobile has quit IRC16:36
*** jcoufal-mobile has joined #openstack-infra16:36
*** jcoufal-mobile has quit IRC16:37
*** rcarrillocruz has joined #openstack-infra16:38
*** rcarrillocruz1 has quit IRC16:38
*** achuprin has quit IRC16:39
pabelangerjeblair, recommendations for zuul talks? I know you have a few floating around16:41
jeblairpabelanger: http://docs.openstack.org/infra/publications/ are the slides...16:41
*** krotscheck has joined #openstack-infra16:41
jeblairpabelanger: i hear there are recordings on youtube16:42
jeblairpabelanger: i think this is the most recent i've done: http://www.youtube.com/watch?v=xTPILCPRGeg16:43
*** CaptTofu has joined #openstack-infra16:43
*** CaptTofu has quit IRC16:43
*** markmcclain has quit IRC16:43
*** thuc has quit IRC16:44
*** thuc has joined #openstack-infra16:44
*** Ryan_Lane has joined #openstack-infra16:44
*** nicedice has joined #openstack-infra16:44
*** CaptTofu has joined #openstack-infra16:45
pabelangerjeblair, great, thank you16:45
jeblairpabelanger: ttx did a good talk too that was a little broader (not as deep into zuul, but put it into context of the whole system): https://fosdem.org/2014/schedule/event/openstack_testing_automation/16:46
dims_clarkb, where can i hang out to bug libvirt/uca folks?16:46
jeblairi can't find the video for that16:46
*** aysyd has quit IRC16:47
jeblairapparently they haven't transcoded it yet16:47
fungiwell, you did say that the concept of recording and publishing sessions was sort of new to the fosdem disorganizers16:51
jeblairsaturday is available!16:52
andreafsdague: ping16:52
openstackgerritA change was merged to openstack-infra/config: OpenstackID Documentation  https://review.openstack.org/6962016:56
*** afazekas has joined #openstack-infra16:56
*** bhuvan has joined #openstack-infra16:56
*** bhuvan_ has joined #openstack-infra16:56
openstackgerritA change was merged to openstack-infra/zuul: Shut down the gearman server last in tests  https://review.openstack.org/7947216:58
pleia2bug day time! https://etherpad.openstack.org/p/cibugreview-march201417:00
*** david-lyle has quit IRC17:01
*** jcoufal has quit IRC17:04
*** sabari2 has joined #openstack-infra17:04
*** lcheng has quit IRC17:04
*** caleb_ has quit IRC17:04
*** ociuhandu has quit IRC17:04
*** esker has joined #openstack-infra17:05
*** zns has quit IRC17:06
fungii'm working on triaging the ones in new status starting from the top17:06
pleia2thanks fungi17:07
andreafHi - I'm setting up an experimental check for a keystone v3 based test run - I've never done changes to layout / jobs before, so I'd very much like feedback reviews - https://review.openstack.org/#/c/79212/, https://review.openstack.org/#/c/79213/17:08
*** yassine has quit IRC17:09
jeblairpleia2: i'm in!17:09
jeblairclarkb: just in time for bug day!17:10
clarkbjeblair: yup17:10
clarkbto be fair this is slightly early for me on a UTC clock >_>17:10
pleia2jeblair: thanks for the fedora review, I think there was a reason we didn't just pull in devstack script but looking at it now that might be gone17:10
*** khyati has quit IRC17:10
mordreddhellmann: ping17:11
jeblairpleia2: cool; the -1 was mostly for the script name and confusion there; i wouldn't block it on the devstack reuse, but would love it if things can converge17:11
*** e0ne has quit IRC17:11
jeblairmordred: oh good you're here in time for bug day too! :)17:11
*** sdake has joined #openstack-infra17:11
mordredjeblair: I'm only partially here- my dad has decided to make a last-minute visit so I'm scrambling a bit17:12
clarkbdims_: zul and jamespage hang out in here usually17:12
jeblairfungi: i'll work from the bottom up on that list17:12
fungijeblair: great, thanks17:13
openstackgerritPaul Michali proposed a change to openstack/requirements:     Add httmock to test-requirements.  https://review.openstack.org/7968417:13
jeblairsdague: can you please tag gate failure bugs with 'gate-failure'?17:14
*** lcheng has joined #openstack-infra17:15
clarkbsdague: jeblair: I honeslty don't see how having all of that extra logging helps. If ES can't keep up you'd need a hadoop laying around17:16
openstackgerritRicardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases on IRC Closes-Bug: #1227985 Change-Id: I9b80b465d9d6b0e2793286cc8229b82b4389a08d  https://review.openstack.org/7920517:16
*** dcramer__ has joined #openstack-infra17:16
*** sarob_ has quit IRC17:17
*** sarob_ has joined #openstack-infra17:18
SergeyLukjanovclarkb, IIRC ES isn't too bad, probably, it'll be hard to replace it with other tools to have better performance17:18
SergeyLukjanovjeblair, clarkb, fungi, pleia2, how can I help with bugs triage?17:18
*** rms_13 has joined #openstack-infra17:18
jeblairSergeyLukjanov: go through the list here: https://etherpad.openstack.org/p/cibugreview-march201417:19
rms_13sdague: ping17:19
pleia2SergeyLukjanov: today tends to be a good reminder to update any of your own bugs, and review the bug list to see if there is anything you should fix yourself - particularly the higher priority ones17:19
clarkbSergeyLukjanov: hop on the etherpad (which I am about to do), and start plugging away at the list. usually we go through the old bugs to make sure they are up to date and also go through the new list to categorize and prioritize17:19
SergeyLukjanovI can start from the end of list assuming that I'll be able to evaluate low prio issues :)17:19
jeblairSergeyLukjanov: if you examine a bug listed in the etherpad, make a note in the etherpad17:20
clarkbby the way, holger responded to my pull request and bug for tox17:20
rms_13Can a devstack-core member take a look at https://review.openstack.org/#/c/78042/ please?17:21
jeblairSergeyLukjanov: i updated the top 2 bugs as examples17:21
*** nati_ueno has joined #openstack-infra17:21
*** miqui has quit IRC17:21
*** SumitNaiksatam has quit IRC17:21
SergeyLukjanovjeblair, are we using some mark to mark that issue is now under review?17:21
jeblairSergeyLukjanov: you might not want to worry about the low priority ones.  we won't get to everything, so it'd be better to skip those.17:21
fungirms_13: devstack development tends to mostly get discussed in the #openstack-qa channel i think17:22
fungiyou might have better luck there17:22
SergeyLukjanovjeblair, ok17:22
rms_13fungi: ok. Saw few names up here so took a chance :)17:23
*** SumitNaiksatam has joined #openstack-infra17:25
*** caleb_ has joined #openstack-infra17:25
*** pbelanyi has quit IRC17:26
*** lcheng has quit IRC17:26
openstackgerritRicardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases on IRC  https://review.openstack.org/7920517:26
*** JuanManuelOlle has quit IRC17:26
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove false positives from query 1240256  https://review.openstack.org/7950317:27
openstackgerritA change was merged to openstack/requirements: Correct iso8601 parseing invalid year num  https://review.openstack.org/7703317:27
*** JuanManuelOlle has joined #openstack-infra17:28
*** lcheng has joined #openstack-infra17:28
jeblairfungi: can you triage https://bugs.launchpad.net/openstack-ci/+bug/1287975 ? it seems to have something to do with py3k; you might have more info handy (and possibly a dup bug)17:28
fungijeblair: sure17:29
*** marun has quit IRC17:30
*** marun has joined #openstack-infra17:31
*** markmc has quit IRC17:31
fungijeblair: one of the bugs i triaged was about the zuul unit test issues i think you just solved, so i corrected the lp project and assigned it to you for confirmation17:32
jeblairfungi: cool, thx17:32
clarkbpleia2: mordred https://bugs.launchpad.net/openstack-ci/+bug/1082795 is that something we need to track under openstack-ci?17:32
SergeyLukjanovclarkb, re jjb @ py26 https://review.openstack.org/#/c/71424/ looks ok for me, could I approve it or should we wait for someone else from jjb team?17:33
clarkbSergeyLukjanov: you can approve17:34
SergeyLukjanovclarkb, should I remove it from the etherpad when it'll be landed?17:35
*** zns has joined #openstack-infra17:35
*** sarob_ has quit IRC17:36
SergeyLukjanovoh, I see all savanna jobs failed with http://logs.openstack.org/47/79347/2/gate/gate-tempest-dsvm-savanna-full/e9f3dff/console.html#_2014-03-11_17_23_07_85217:36
clarkbSergeyLukjanov: you can mark it as fix released on the etherpad then next bug day it won't show up17:36
*** sarob_ has joined #openstack-infra17:36
*** markmcclain has joined #openstack-infra17:38
bknudsonjenkins is posting updates to the requirements faster than they can be checked, reviewed, validated, merged.17:39
*** rcarrillocruz1 has joined #openstack-infra17:39
*** khyati_ has joined #openstack-infra17:39
*** rcarrillocruz has quit IRC17:39
SergeyLukjanovclarkb, ok, thx, got it17:39
clarkbbknudson: that should be ok, it is updated the existing changes right?17:40
jeblairclarkb: it's kind of annoying when it does that while it's in the gate.  :(17:40
bknudsonclarkb: yes, it's updating it's existing change, so it loses the vote17:40
*** rcarrillocruz has joined #openstack-infra17:40
clarkbjeblair: yeah but that is a different problem17:41
jeblairwell, bknudson said "validated, merged" so i assume it was at least a component17:41
*** melwitt has joined #openstack-infra17:42
fungibknudson: sounds like a sort of review which should be prioritized if a project is having trouble keeping up with global changes to their requirements. also participating in reviewing changes proposed to the openstack/requirements project's global-requirements.txt which have impact on a project of interest may allow for getting ahead of potentially detrimental proposals17:43
dims_clarkb, thanks17:43
*** rcarrillocruz1 has quit IRC17:43
clarkbhttps://bugs.launchpad.net/openstack-ci/+bug/1280008 is a requirements bug (asserting that packages are missing) is that something we want in openstack-ci's bug list?17:44
bknudsonthe sqlachemy 0.9 change caused keystone docs to fail to build, but that was reverted... haven't had a chance to look at it.17:44
clarkbI don't know that there is a better location for that17:44
*** esker has quit IRC17:45
pleia2clarkb: so I think mordred agreed that we'd make testing happen in -infra once nova baremetal went in, but now that things are moving to ironic which isn't graduated yet I'm not really sure where this all landed17:45
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Logging in with a pre-existing user saves details  https://review.openstack.org/7970517:45
jeblairclarkb: that looks like a taskflow bug17:45
clarkbjeblair: wfm17:45
jogofungi: as per youre comments last night https://review.openstack.org/#/c/79519/17:45
fungibknudson: yes, there were some recent changes to devstack which broke requirements change vetting and allowed that through when it shouldn't have gotten. i think sdague is very closed to having that solved17:45
jeblairclarkb: i'm not sure requirements can have bugs.  :)17:45
*** esker has joined #openstack-infra17:45
clarkbjeblair: nice17:45
clarkbthis is the way it will be no bugs about it17:46
clarkbpleia2: I think more importantly there isn't much we can do to facilitate that17:46
*** sabari2 has quit IRC17:46
jeblair(i mean, mirror or requirements enforcement scripts are infra bugs)17:46
*** zns has quit IRC17:46
clarkbeither they write tests to run against tripleo baremetal or they do devstack stuff to have fake baremetal17:46
fungijogo: okay. i'll add it to the agenda for today's 19:00 infra meeting17:46
jogoI added something last night, but it was very vague17:47
sdaguefungi: so the fix is in the gate17:47
sdagueif we get a reset we should bump it17:47
pleia2clarkb: yeah17:47
fungisdague: okay. if we're lucky it'll just merge in an hour17:47
sdaguethen I'll re propose the sqla thing, and we'll see how it goes17:48
*** hashar has quit IRC17:48
*** freyes has joined #openstack-infra17:48
sdaguethen I need to come up with a clever way of actually testing this17:49
sdagueso we never get in this situation again17:49
jeblairclarkb: https://bugs.launchpad.net/openstack-ci/+bug/128559317:49
jeblairclarkb: seems similar to horizon issue17:49
*** markwash has quit IRC17:49
jeblairclarkb: is there a bug or review for that?17:49
*** mfer has quit IRC17:50
*** johnthetubaguy1 has quit IRC17:50
clarkbI am not familiar with the horizon issue. Do you mean keystone port 35357 issue?17:50
*** esker has quit IRC17:50
jeblairclarkb: yes!17:51
jeblairclarkb: oh, what's the lower bound for the ephemeral port range?17:52
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: make command parsing compatible with python 2.6  https://review.openstack.org/7142417:52
nibalizerso daylight savings time is confusing me, is the meeting in 10 minutes ?17:52
clarkbjeblair: there is a bug for that against devstack and keystone because it is related to the use of a port in that port range17:52
clarkbjeblair: 9000 is way outside of that range so should be fine17:52
nibalizeror an hour and 10 minutes?17:52
clarkbdifferent bug17:52
clarkbnibalizer: an hour17:52
*** johnthetubaguy has joined #openstack-infra17:52
nibalizerah k17:52
funginibalizer: one hour and 10 minutes (19)) utc regardless of dst)17:52
nibalizerokay cool17:52
clarkbdevananda: areyou around? https://bugs.launchpad.net/openstack-ci/+bug/128138517:52
clarkbdevananda: pyghmi is a stackforge project17:53
devanandaclarkb: looking17:53
clarkbdevananda: pyghmi is more than welcome to do the things you have proposed -infra do for it17:53
clarkbbut that isn't an infra bug17:53
fungiclarkb: i think what makes it an infra bug is that it would be closed with the approval of https://review.openstack.org/74499 (which changes infra config)17:54
devanandaclarkb: I agree that its not a bug in infra, but ^17:55
devanandaclarkb: also a related question came up last week at the sprint17:55
clarkbfungi: devananda right but we try to have a separation of concerns there17:55
jeblairand we don't require bugs for patches17:55
clarkbwe are not responsible for stackforge project testing17:55
jamespageclarkb, yes17:55
*** talluri has quit IRC17:55
clarkbjamespage: oh hi, I think dims_ has questions for you17:56
devanandaclarkb: should we (ironic and other projects) add things to requirements.txt if they are required for third-party drivers?17:56
fungiclarkb: yeah, not an infra bug, but people filing bugs because they want to see them closed by patches. i agree it's unnecessarily pedantic17:56
*** sweston has quit IRC17:56
clarkbjamespage: dims_ has a devstack change that enables UCA so that he can test it in a gate environment17:56
*** talluri has joined #openstack-infra17:56
*** david-lyle has joined #openstack-infra17:56
clarkbdevananda: if you are asserting you need them then yes17:56
jamespageclarkb, ack17:56
jogofungi: https://bugs.launchpad.net/bugs/129063717:57
devanandaclarkb: where "need" is "support for this optional driver" - yes. where "need" is "core functionality requires it" - no.17:57
jogothat wasn't meant to be a duplicate17:57
fungijogo: so what is the actual bug there?17:57
jogonew kernel doesn't like us17:57
devanandaclarkb: so i'm seekign some guidance on whether external libs for third-party drivers should go into requirements or not17:57
jogofun notsure how to resolve that though17:58
jogofungi: *17:58
*** esker has joined #openstack-infra17:58
fungijogo: um... open a bug against ubuntu's kernel package or against libvirt?17:58
jogobut a noticable spike in failures17:58
jogofungi: I think that would be the right thing17:58
jogonot even sure how to do that17:58
fungijogo: you already have a bug open for the failures. and increase in those failures doesn't necessarily require a new bug17:58
clarkbdevananda: we use requiremetns for two things today. Building the pypi mirror and enforcing the requirements you list in your project. If you need the thing to be in your requirements list then you need it on the mirror which means it shoudl go in global requirements17:59
jogofungi: well this one maps identically to a change in the kernel we use17:59
*** weiwei has quit IRC17:59
jogoanyway how do I add this to ubuntu kernel?17:59
jogoor libvirt17:59
fungijogo: it's possible other things upgraded at the same time that the kernel did, so that's conjecture. but "here's something we noticed which increases the incidence of this bug" is potentially useful information to the bug which is happening more often now18:00
*** amcrn has quit IRC18:00
clarkbdtroyer: https://bugs.launchpad.net/openstack-ci/+bug/1282772 is that only a grenade bug or do we need to do something from an infra side? I will mark it incomplete as it is missing info about what infra can do to help18:00
fungijogo: ubuntu and libvirt both use launchpad for their bugs, i believe18:00
*** talluri has quit IRC18:00
*** jnoller has joined #openstack-infra18:00
*** sweston has joined #openstack-infra18:00
*** derekh has quit IRC18:00
*** esker has quit IRC18:01
dims_jamespage, i still see 1.2.1-0ubuntu5~cloud0 which was built on Feb 12 on precise-proposed/icehouse - wanted to find out when we could see a fresh update?18:01
jogolibvirt doesn't18:01
jogowhat is the name of ubuntu's kernel group?18:01
*** esker has joined #openstack-infra18:01
jogowendar or other ex Canonical folks?18:01
jamespagedims_, prob later this week as I pushed b3 out to proposed/updates18:01
fungijogo: i guess the question i have is whether you think fixing the bug from the libvirt or nova side is going to cause it to not be fixed because it happens more often under a specific kernel version18:01
jogofungi: not really sure, I do know that the timing is suspicious and the spike in in failures is bad18:02
dims_jamespage, does b3 have this? https://bugs.launchpad.net/nova/+bug/1228977/comments/4018:02
jogodims_: ^^18:02
clarkbjeblair: fungi: devananda: it wasn't clear to me from the above if I shouldn't mark that pyhgmi test thing as invalid in openstack-ci18:02
clarkbfungi: I think you were arguing we should keep it and jeblair was on board with marking it invalid18:03
jogofungi: and want to get one of the ubuntu kernel folks to glance at this ideally18:03
*** ryanpetrello has joined #openstack-infra18:03
jamespagedims_, can you check the version in trusty development - that's what I'll land this week18:03
fungiclarkb: if we had a "pedandic and unnecessary, track your infra config changes in your own project not ours" status for bugs i'd say that would fit. we could just on policy start closing bugs people open against openstack-ci purely because they're planning to make a config change18:03
dims_jamespage, unfortunately i can test only precise18:04
jamespagedims_, it should be 1.2.218:04
*** johnthetubaguy has quit IRC18:04
*** johnthetubaguy has joined #openstack-infra18:04
*** zns has joined #openstack-infra18:05
clarkbfungi: I want to say https://bugs.launchpad.net/openstack-ci/+bug/1282795 was something you dug into but couldn't find anything out of the ordinary?18:05
fungijogo: probably best to get 1254872 brought to the attention of the ubuntu kernel packagers then and let them know that we're seeing it more often under the new kernel, but without knowing the underlying cause i'm still unconvinced that an increase in occurrence of a bug is a separate bug18:05
jogojamespage: you work for cannonical right? ^18:06
*** esker has quit IRC18:06
*** mfer has joined #openstack-infra18:06
*** jamielennox|away is now known as jamielennox18:06
jamespagejogo, yes18:07
jogohttps://bugs.launchpad.net/bugs/1254872 is hurting us a lot more under the new .60 kernel18:07
jogonot sure who to pass that info onto perhaps you would know18:07
jogojamespage: http://status.openstack.org/elastic-recheck/18:07
*** jgallard has quit IRC18:08
dims_jamespage, thanks for the info. please do ping me when you push 1.2.2 to precise.18:08
jeblairclarkb: i think you should just remove openstackci from the list of affected projects18:09
jamespagedims_, will do - just doing a vUDS session right now18:09
fungiclarkb: i simply couldn't fathom how 1282795 could happen since tox is a python program, it's complaining "InterpreterNotFound: python2.6" well into execution on centos, and that's the only python interpreter centos has (so tox should never have been able to run). i don't think i was able to catch one where the failure happened and log into it to look for the interpreter existing though18:09
clarkbjeblair: wfm18:10
clarkbfungi: :/ that is a fun one18:10
clarkbfungi: I will let you update the bug then as you are most familiar with it18:10
fungiclarkb: can do18:11
*** weiwei has joined #openstack-infra18:11
*** jlibosva has quit IRC18:12
*** sabari2 has joined #openstack-infra18:13
clarkbgreat thanks18:13
jogojamespage: looks like  0.9.8 is buggy we should at least use 0.9.918:13
jeblairI'm typing "18:13
jeblairThis is a real test failure, not an infrastructure bug.18:13
jeblair" a lot.18:13
reedjeblair, fungi: can either of you open port 5901 on activity.openstack.org? we need it for profiling18:13
dims_jogo, even with jamespage 's 1.2.2 - we'll need at least one more fix from Daniel :(18:13
clarkbjeblair: :/18:13
jogodims_: so I see this as a two prong effort18:14
jogoget 1.2.2 working18:14
jogoand get 0.9x working a little better18:14
dims_jogo, ack18:14
jogoI want to move to 1.2.2 in gate18:14
jogobut I also want 0.9.x working in the meantime18:14
clarkbjeblair: gah we are racing18:14
jogosounds like danpb made headway on https://bugs.launchpad.net/nova/+bug/1254872 and things it just needs 0.9.918:15
fungireed: i don't believe any of is have a login to activity.o.o? at least i don't seem to18:15
jeblairclarkb: oh, are you on new bugs too?18:15
jogodims_: so not sure if you know who to ask to make that happen18:15
clarkbjeblair: ya but I think I mostly managed to avoid you :)18:15
reedfungi, can i do it from iptables? I thought it was a config panel thing on rackspace18:15
clarkbjeblair: I will let you finish18:16
* clarkb goes back to older bugs18:16
*** mriedem has quit IRC18:16
fungireed: i don't think we use security zone configuration on rackspace, but maybe someone did on that one since it's not controlled by infra. i'll check18:16
*** sabari3 has joined #openstack-infra18:17
clarkbit should be via iptables18:17
*** sabari2 has quit IRC18:17
*** andreaf has quit IRC18:17
clarkbwe manage firewalls that way rather than using security groups18:17
reedclarkb, thanks, I'll go check iptables18:18
*** ildikov_ has quit IRC18:19
jeblairclarkb, fungi: does this ring a bell? https://bugs.launchpad.net/openstack-ci/+bug/128226218:20
reeduhm ... I thought I had access to that machine18:21
clarkbjeblair: no, but I am reading logs for it now18:21
jamespagedims_, jogo - give me 20 mins and we can talk better18:21
clarkbwhich have purple links18:21
clarkbso I have read them before18:21
dims_jamespage, thx18:21
fungiclarkb: jeblair: was this maybe around the time we did the tempest account split and had to do several iterations of sudo -u/chown patches to get things to be writeable by the right accounts?18:23
clarkbfungi: I bet that is it18:23
jeblairokay, cool, i'll mark it fixed released18:23
*** Ajaeger1 has joined #openstack-infra18:23
jogojamespage: I need to go AFK for a a little but everything should be in bug 125487218:24
*** dizquierdo has quit IRC18:25
fungireed: i'm not seeing anything obvious in the rackspace dashboard to imply that we've configured them to do any network filtering of that server18:26
*** SumitNaiksatam_ has joined #openstack-infra18:27
reedfungi, thanks, i'll have sanjiva check iptables himself then18:27
*** SumitNaiksatam has quit IRC18:27
*** rcarrillocruz1 has joined #openstack-infra18:28
*** vhoward has left #openstack-infra18:29
*** rcarrillocruz has quit IRC18:29
*** SumitNaiksatam_ is now known as SumitNaiksatam18:29
*** jooools has quit IRC18:29
*** VijayTripathi has joined #openstack-infra18:30
*** jpich has quit IRC18:31
*** arborism has joined #openstack-infra18:33
*** dcramer__ has quit IRC18:33
*** mriedem has joined #openstack-infra18:34
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Replaced first_name and last_name with full_name.  https://review.openstack.org/7972018:34
*** arborism is now known as amcrn18:34
*** khyati_ has quit IRC18:34
jeblairfungi: we're done!18:37
*** thuc has quit IRC18:37
fungiwell, you're done. i'm still triaging the last one, but yay!18:37
*** thuc has joined #openstack-infra18:37
jeblairfungi: i might have already done it18:38
*** esker has joined #openstack-infra18:38
fungioh, so you did18:38
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Consume full_name rather than first_name and last_name  https://review.openstack.org/7972318:38
*** johnthetubaguy has quit IRC18:39
fungii tried to obtain a lock on it by setting to triaged before i started working on it. i think that one's a duplicate, just trying to hunt down the original18:39
jeblairfungi: oops, sorry18:40
*** thuc_ has joined #openstack-infra18:40
*** thuc_ has quit IRC18:40
fungino worries!18:41
*** thuc has quit IRC18:41
jamespagejogo, looking now18:41
*** thuc has joined #openstack-infra18:41
clarkbjeblair: https://bugs.launchpad.net/openstack-ci/+bug/1278210 may interest you18:42
clarkband is https://bugs.launchpad.net/openstack-ci/+bug/1281320 the thing affecting doc jobs?18:42
jeblairclarkb: a fix to 320 would help the doc job situation18:44
jeblairclarkb: (though the root cause of that is now understood to be related to concurrent builds and jenkins+nodepool+gearman-plugin)18:44
jeblairclarkb: so fixing 320 would help mask the problem.18:45
jeblairclarkb: fix for docs situation: https://review.openstack.org/#/c/79659/18:45
jeblairclarkb: the actual problem: https://jenkins05.openstack.org/job/openstack-user-guide-admin/18:45
clarkbjeblair: fungi SergeyLukjanov mordred https://bugs.launchpad.net/openstack-ci/+bug/1273283 is that fixable now? we are using single use slaves for all of the major testing18:45
clarkbjeblair: looking18:45
*** freyes has quit IRC18:46
SergeyLukjanovclarkb, but we still have proposal node18:46
jeblairSergeyLukjanov: we don't cancel any jobs that run on the long-running nodes18:46
jamespagedims_, which version of libvirt are you running in gate? the 1.1.1 from the Havana CA or the 0.9.8 from 12.04 base?18:46
jeblairSergeyLukjanov: and canceling jobs was what triggered that error18:47
dims_james 0.9.818:47
SergeyLukjanovjeblair, oh, /me forget that it was because of cancelation...18:47
dims_jamespage, 0.9.818:47
*** dcramer__ has joined #openstack-infra18:47
jamespagedims_, OK18:48
dims_jamespage, 1.1.1 from Havana was bad because it had locking issues which Daniel fixed. i was able to validate his changes worked.18:48
jeblairclarkb: logstash graph is interesting.  32 workers are still online.18:48
dims_jamespage, by building his changes straight from libvirt git18:49
*** mkoderer has joined #openstack-infra18:49
jeblairclarkb: (i haven't checked to see if they are doing anything)18:49
jamespagedims_, we can fix this stuff btw; but need to know :-)18:49
*** JuanManuelOlle has left #openstack-infra18:49
jamespagedims_, if there are commits that we can take and SRU that's great18:49
clarkbjeblair: they appear to be doing stuff, I just refreshed a kibana saerch and have data18:49
jamespagedims_, so bug 1254872 is related to the 0.9.8 issue right?18:49
sdaguejamespage: danpb has a pretty deep analysis, unfortunately I don't think he put it out to the list yet18:50
* clarkb looks at cacti18:50
jeblairclarkb: http://paste.openstack.org/show/73175/18:50
dims_jamespage, yes, so we tried to upgrade to precise-proposed and saw the stuff mentioned in here - https://bugs.launchpad.net/nova/+bug/122897718:51
clarkbjeblair: I have a graph similar to that, I use the one on zuul status but change the time to -2hours18:51
dhellmannmordred: just saw your ping, if you're still around let me know18:51
openstackgerritA change was merged to openstack-infra/config: Remove concurrency restriction from all jobs  https://review.openstack.org/7965918:52
dims_jamespage, so at this point we need 1.2.2 + Daniel's fix mentioned here (https://bugs.launchpad.net/nova/+bug/1228977/comments/40)18:52
*** SumitNaiksatam has quit IRC18:52
clarkbjeblair: I am going to look at numbers and see if moving to a third worker on each node is feasible18:53
jamespagedims_, OK - so hallyn is working on fixing up the 1.1.1 in proposed; once its in there I can push it out to havana-proposed18:53
jeblairclarkb: all 32 are active18:53
jeblairclarkb: so i think we're just right at the inflection point18:53
dims_jamespage, you want to leave icehouse-proposed for 1.2.x?18:54
sdaguejeblair / fungi: we keep not getting a nice reset point in the gate, however we get a lot of resets today. So I'm wondering if we could do a promote of 77815,2 anyway18:54
jamespagedims_, icehouse-proposed will get the 1.2.2 that's in trusty18:54
jamespageif that needs a patch then we'll get that sorted18:54
jamespagelemme check with hallyn18:54
clarkbjeblair: I agree. free shows that most nodes have between 1.8GB and 2GB of cached + free memory. I think that is plenty of room to run a third worker18:55
clarkbjeblair: what do you think? should I start up -C on all of those nodes?18:55
dims_jamespage, that will then need a fix for this as well https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1283179 in addition to Daniels fix https://www.redhat.com/archives/libvir-list/2014-March/msg00501.html which was post 1.2.218:55
fungisdague: i've seen the same cinder unit test mismatch error in the gate a couple times now... i think maybe a racy change must have slipped in on them18:56
jeblairclarkb: wfm; i assume it was libvirt that killed them all before18:56
fungisdague: is that known already?18:56
SergeyLukjanovfolks, have you seen the d-g failures with "ImageKilledException: Image 45a63b5f-9c6d-48e2-b203-de4bf2d2a3cf 'killed' while waiting for 'active'"?18:56
clarkbjeblair: yup I think so18:56
dims_jamespage, sounds like a good plan 1.1.1 on precise/havana and 1.2.x on precise/icehouse18:56
*** dcramer__ has quit IRC18:56
jamespagedims_, we can't bump havana - that drives of saucy which is released so SRU only - we can cherry pick the fix tho18:57
clarkbjamespage: done18:57
clarkbgah jeblair ^18:57
*** hallyn has joined #openstack-infra18:57
jamespagehey hallyn18:57
clarkbjamespage: dims_ 1.1.1 is horribly broken18:57
sdaguefungi: I don't know18:57
sdagueI was basically spending all my attention on getting gr back to sanity18:58
sdaguebecause I consider this a giant hole in our system right now18:58
jamespageclarkb, I'm sure hallyn is doing his best to unbreak it18:58
fungisdague: the main reason to promote 77815,2 would be if there were requirements changes ahead of it though, right?18:58
clarkbjamespage: I think danpb's patch for 1.2.2 is the fix18:58
*** khyati_ has joined #openstack-infra18:58
clarkbbut /me defers to people that actually grok libvirt18:59
sdaguefungi: the main reason is so we can actually test any of the requirements outstanding18:59
sdaguewhich I'd like to be able to do in this work day to start getting results back18:59
jamespagehallyn, I was just discussing 1.2.2 in the context of bug 122897718:59
sdagueespecially the sqla one18:59
jamespagewith dims_18:59
hallynjamespage: when i have a free moment i'll push the patch to 1.2.218:59
jamespagehallyn, thanks - much appreciated19:00
clarkbmeeting time19:00
jeblairfungi, sdague: i think since it fixes a gating hole, it's okay to promote19:00
jamespagedims_, pending a few MIR's for Ubuntu trusty I hope to have b3 plus updated dependencies in the CA by the end of this week - I'll keep you updated19:00
hallynfor the precise version, i will push a package with the patch dberrange suggested;  if that doesn't suffice i can look deeper19:00
fungijeblair: okay, will do (even though it's not a hole for any of the changes ahead of it in the gate)19:00
*** jolyonbrown has joined #openstack-infra19:00
jamespagehallyn, +119:01
dims_hallyn, yep, that will do19:01
dims_hallyn, jamespage thanks!19:02
jamespagedims_, np19:02
jamespagedims_, please keep slinging stuff like this at us - need to know :-)19:02
dims_jamespage, yep will do19:03
*** rcarrillocruz has joined #openstack-infra19:04
*** gyee has quit IRC19:04
*** harlowja has quit IRC19:04
jamespagedims_, you might find these reports useful btw19:05
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Replaced first_name and last_name with full_name.  https://review.openstack.org/7972019:05
*** gyee has joined #openstack-infra19:05
dims_jamespage, very cool. thx19:06
*** rcarrillocruz1 has quit IRC19:06
jogodims_ jamespage hallyn: thanks for looking into the libvirt stuff19:07
*** yolanda has quit IRC19:08
*** harlowja has joined #openstack-infra19:08
jogoany change of replacing 0.9.8 with 0.9.11 as well?19:08
*** cosmogon has joined #openstack-infra19:08
adam_gwhile we are discussing libvirt issues WRT the gate, i've hit a wall trying to use newer libvirt 1.1.1 (cloud-havana or saucy) on RAX: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/124802519:09
openstackgerritPaul Michali proposed a change to openstack/requirements:     Add httmock to test-requirements.  https://review.openstack.org/7529619:09
*** cosmogon has quit IRC19:09
adam_gAFAIK a newer libvirt will eventually be required to get ironic tested there19:09
clarkbfungi: and for my last bug that I started before the meeting https://bugs.launchpad.net/openstack-ci/+bug/1266616 what do you think about my comments there?19:09
fungiclarkb: it's possible--do you have a link to a git bug about trying ipv6 directly rather than leaving it up to the tcp/ip stack on the system?19:11
clarkbfungi: I don't but I want to say I have seen that when I ran precise19:11
dims_adam_g, yes i have a variant of that bug open as well https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/128317919:11
clarkbfungi: I can certainly dig through the git change log19:11
*** skraynev is now known as skraynev_afk19:12
dims_adam_g, yes, i had to nuke the offending shared objects to get the ones i needed working19:12
*** waluta has joined #openstack-infra19:16
dims_jamespage, is the source tree for say 1.1.1-0ubuntu8.5~cloud0 is publicly available?19:16
jamespagedims_, always yes19:16
*** gyee has quit IRC19:16
jamespagedims_, apt-get source libvirt19:16
jamespage(of course you will have to be running on precise with the CA enabled to get that version)19:17
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add oslotest jobs for oslo.config  https://review.openstack.org/7665419:17
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add oslo.test integration test  https://review.openstack.org/7638119:17
dims_jamespage, ah.  was thinking of a git repo somewhere where i could switch around versions and view commit history etc.19:18
jamespagedims_, lp:ubuntu/saucy/libvirt might be up to date - but our package importer always seems to have challenges with libvirt for some reason19:18
dims_jamespage, thanks again19:19
jamespagedims_, you can always retrieve specific versions - LP keeps a full history for the main Ubuntu archive19:19
jamespageand CA is just a backport of that19:19
jamespagedims_, pull-lp-source in ubuntu-dev-tools is your friend19:20
jamespagedims_, pull-lp-source libvirt saucy19:20
jamespagefor example19:20
jamespagebut you can also pull specific versions19:20
jogojamespage: ping19:21
jamespagejogo, hello again19:21
jogoback, so libvirt 0.9.819:21
dhellmannclarkb: I think ^^ addresses the comments you left on the review for the oslo.test/oslo.config cross-test job19:21
jogoalthough I am more interested in getting libvirt 1.x working it looks like libvirt 0.9.8 is broken for us and a fix was added in 0.9.819:22
jamespagejogo, hallyn is going to cherry pick the commit and SRU it19:22
jogojamespage: oh sweet!19:22
jamespagejogo, we don't have a micro release exception for libvirt (not sure why) so we can't just bump it19:23
jamespagebut hopefully that will fix it up19:23
hallynwould someone care to put a detailed test case in that bug?19:23
jogothat should help we have seen 381 failures in the past 14 days on this one19:23
hallyni'm waiting on my test build to finish atm...19:23
jogohallyn: its a race condition, and I am not sure how to make it fail 100% of the  time19:23
jogobut our gate system does good job of hitting it19:23
fungido we know how to consistently trigger that failure mode?19:23
fungithat's what i thoughth19:23
*** waluta has quit IRC19:24
jogofungi: I don't, perhaps danpb does but he said its race19:24
jamespagejogo, I know its cheeky but could you point some gate nodes at the proposed packages and see if the failure rate drops?19:24
jogowhich makessense19:24
hallynjogo: 50% of the time would be sufficient :)19:24
*** sarob_ has quit IRC19:24
jogojamespage I like that idea19:24
jogofungi: ^ thoughts19:24
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Change operations-guide and openstack-manuals jobs to not use maven  https://review.openstack.org/7744619:24
sdaguejogo: we'd have to build a whole new nodepool image19:24
hallynok i'll post debs as soon as they're built19:24
sdagueand class19:25
*** sarob_ has joined #openstack-infra19:25
fungijogo: i don't really know anything about the bug, was just asking because i got the impression we didn't know how to consistently reproduce it19:25
jogofungi: correct we don't know how to19:25
jogoexcept we saw it 55 times in last 24 hours19:25
Ajaeger1fungi, clarkb: So, for stable/havana and master the openstack projects do not use maven anymore with the above patch.19:26
fungiAjaeger1: awesome!19:26
jogosdague: this may be useful in the future for other race conditions like this19:26
dims_hallyn, jamespage - "precise-proposed"?19:26
*** pblaho has joined #openstack-infra19:26
*** SpamapS has quit IRC19:26
sdaguehallyn / jogo: so here is an excerpt from what danpb sent me just before he went to bed - http://paste.openstack.org/show/73178/19:27
sdagueI was hoping to get him to just stick the whole thing on the list, but given he's asleep, and you guys are chasing this, there you go19:27
jogosdague: he put that in the bug19:27
jamespagedims_, yes19:27
*** bhuvan_ has quit IRC19:28
*** bhuvan has quit IRC19:28
Ajaeger1fungi: stable/grizzly still has maven but they will soon be obsolete. If you want me to change those as well, tell me and I work on getting our tools working with grizzly. I prefer to spend my time elsewhere ;)19:28
Ajaeger1but if these grizzly jobs block you, tell me.19:29
fungiAjaeger1: no, technically we were supposed to eol grizzly last thursday19:29
*** SpamapS_ has joined #openstack-infra19:29
fungiwe're just behind on pulling the trigger there19:29
*** sarob_ has quit IRC19:29
*** rcarrillocruz1 has joined #openstack-infra19:30
*** SpamapS_ is now known as SpamapS19:30
*** jp_at_hp has quit IRC19:30
hallyndims_: well i can just post them someplace for you to wget for a first test19:31
*** e0ne has joined #openstack-infra19:31
*** rcarrillocruz has quit IRC19:31
jogojamespage: so as for using OpenStack CI as a test for the patch, you would have to convince fungi and sdague that its worth it19:33
dims_hallyn, y let me know where i can wget from as precise-updates seem to have 0.9.8-2ubuntu17.17 (looking at http://us.archive.ubuntu.com/ubuntu/dists/precise-updates/main/binary-i386/)19:33
jogoI like the idea, but not sure how much overhead it will involve19:33
dims_jogo, no need, i can adapt my UCA patch to wget/dpkg19:33
jamespagejogo, that's ok - I appreciate it was a bit cheeky :-)19:33
jamespagejogo, we do need some sort of test case for the SRU tho19:33
*** markmcclain has quit IRC19:34
*** jgrimm has quit IRC19:34
*** markmcclain has joined #openstack-infra19:34
jogodims_: that may work19:34
clarkbjogo: teh test wouldn't have to merge19:34
clarkbyou can just push it19:34
clarkblet it run and done19:34
hallynsdague: yes, taht's the commit i'm pushing19:34
sdaguehallyn: great, thanks19:34
jogoclarkb: well it depends on how many runs we need ... but we can just babysit it19:34
dims_jamespage, hallyn just drop me a note or ping me when i can wget 0.9.9 from a public location19:35
*** malini is now known as malini_afk19:36
*** bhuvan has joined #openstack-infra19:37
*** bhuvan_ has joined #openstack-infra19:37
jogodims_: thanks again for doing all this19:37
*** khyati_ has quit IRC19:37
*** amcrn has quit IRC19:38
sdaguejamespage: part of the problem is the repeat scenario includes our volume19:39
sdagueso set up tests that create 50k guests in a day, you'll get 12 fails19:39
jogoclarkb: what was the fix for ES being backed up19:40
*** VijayTripathi1 has joined #openstack-infra19:40
clarkbjogo: we stopped archiving libvirt logs19:40
*** sarob_ has joined #openstack-infra19:40
*** amcrn has joined #openstack-infra19:40
*** moted has quit IRC19:40
*** sarob_ has quit IRC19:41
fungiand soon need to stop collecting/producing libvirt logs so the logserver doesn't fall over too19:41
clarkbfungi: that was the fix19:41
*** moted has joined #openstack-infra19:41
clarkbdevstack-gate stopped copying them for us19:41
clarkbbecause 30MB * 10k tests a day is a lot of gigabytes19:41
jogoclarkb: so I will abandon my patch  https://review.openstack.org/#/c/79535/19:42
fungiclarkb: no, actually that fix failed to merge, and sdague proposed an alternative to stop indexing them instead19:42
clarkbfungi: oh so sdague did jogo's patch?19:42
clarkbfungi: we should've pushed the d-g change through for the reason above19:42
*** bhuvan_ has quit IRC19:43
*** bhuvan has quit IRC19:43
sdagueclarkb: yes19:43
*** lottosport has joined #openstack-infra19:43
*** VijayTripathi has quit IRC19:43
sdagueclarkb: so if you pushed that through, we never would have gotten any progress on this bug19:43
sdaguewe needed the enhanced logs for a reason19:43
clarkbsdague: right see my comment before19:43
clarkbthere is far too much data to do anything useful with it19:43
clarkbwe need a more targetted approach imo19:44
sdaguein ES, for sure19:44
clarkbI don't think danpb needs 30MB compressed libvirt logs for every tempest run19:44
sdagueclarkb: what do you propose for more targetted approach?19:44
clarkbsdague: prop up a change that gives you more log data, run it experimentally or recheck it19:45
*** lukego has quit IRC19:45
clarkbthat also allows you to tweak things on the fly19:45
sdagueclarkb: if we had a way to "recheck 50x" on a change, that would be fine19:45
hallynprecise compile seems to be taking longer than the trustyone did19:45
*** dprince has quit IRC19:46
clarkbsdague: right but instead of 50x we are doing 10kx19:46
*** lottosport has quit IRC19:46
sdagueclarkb: yeh, because some of the failures turned out not to be useful in getting to the bottom of things19:46
sdaguewe turned this on for 24hrs, got enough data to analyze something, and now I can go and put a conditional around this19:47
*** e0ne has quit IRC19:47
*** morganfainberg_Z is now known as morganfainberg19:48
clarkbsdague: it just occurred to me that we could do recheck 50x talking direclty to gearman19:49
*** rcarrillocruz has joined #openstack-infra19:49
clarkbsdague: its not something anyone could do so not easy, but doable19:49
sdaguewe have definitely been in need of that in the past in patches we suspected would show or fix races19:50
sdagueand the solution today has been have a developer go back an "recheck no bug" 10 times19:50
sdaguewhen they remembered19:50
jogosdague: we are ready to turn this off libvirt debug in gate?19:50
sdagueover 4 days19:50
clarkbsdague: you only need about 5 rechecks19:50
sdaguejogo: I think so, I'm going to work on a devstack fix19:50
sdagueclarkb: depends on the bug19:50
*** rcarrillocruz1 has quit IRC19:50
jogosdague: cool19:51
*** bhuvan has joined #openstack-infra19:52
*** bhuvan_ has joined #openstack-infra19:52
*** zns has quit IRC19:53
jogosdague: and we want to turn libvirt indexing back on?19:53
*** yassine has joined #openstack-infra19:53
sdaguejogo: honestly, I don't think so19:54
sdaguethis was the only bug that used it19:54
sdagueand it didn't need to19:54
sdaguethe same line was in n-cpu19:54
jogoI added it originally thinking it would help give us abetter fingprint19:55
jogobut it didn't19:55
*** sweston has quit IRC19:57
SergeyLukjanovbtw, http://gabrielecirulli.github.io/2048/ funny game19:57
*** bhuvan_ has quit IRC19:59
*** bhuvan has quit IRC19:59
*** markmc has joined #openstack-infra19:59
*** pblaho has quit IRC20:01
pleia2morganfainberg: lmk when you have some time to chat about monitoring, I'm around all afternoon20:02
*** jnoller has quit IRC20:02
pleia2speaking of being around, I'm visiting family next week in Maine, will still be working but will be on east coast time and will probably pop out for "lunch with mom" and "buy nephew more toys" here and there20:02
*** mriedem1 has joined #openstack-infra20:02
SlickNikhey guys20:03
*** pdmars has quit IRC20:03
*** mriedem has quit IRC20:03
clarkbpleia2: have fun20:03
clarkbSlickNik: hi20:03
SlickNikI had a question regarding openstack/requirements and trove.20:03
*** mwagner_lap has joined #openstack-infra20:03
fungipleia2: which part of maine? portland area?20:03
clarkbSlickNik: go for it20:03
fungipleia2: i do love to go there, though it's been a few years since my last visit20:04
morganfainbergpleia2, hi20:04
pleia2fungi: the charming town of wilton (more central Maine than Portland, Portland is where I grew up though)20:04
SlickNikToday, trove's got 3 packages that exist as part of its requirements that are not in global-requirements.20:04
pleia2morganfainberg: hey20:04
morganfainbergpleia2, i;m just getting to work, bad sleep schedule due to issues with a maintenance last week :P20:04
fungipleia2: oh, i vaguely know where wilton is. you can't get there from here though20:04
morganfainbergpleia2, so give me a bit to get settled and i'm up for chatting20:04
SlickNikpexpect in requirements.txt; and wsgi_intercept and mockito in test_requirements.txt20:04
pleia2morganfainberg: no worries, I'm going to grab lunch now anyway20:04
morganfainberger, :)20:04
morganfainbergpleia2, good idea!20:05
pleia2fungi: hehe, ayup20:05
*** bhuvan has joined #openstack-infra20:05
*** bhuvan_ has joined #openstack-infra20:05
clarkbSlickNik: you will need to get them into global requirements if appropriate or remove them as dependencies if inappropriate20:05
clarkbSlickNik: guessing that adding them isfine20:05
*** wenlock has quit IRC20:05
clarkbI need to grab lunch20:06
* clarkb afks20:06
*** denis_makogon_ has joined #openstack-infra20:06
SlickNikclarkb: Thanks! That's what I thought. I'll push up a patchset to global requirements to address this. Just wanted to clarify if there's some other "process" involved (or special incantation that I need to recite).20:07
*** lukego has joined #openstack-infra20:09
*** smarcet has left #openstack-infra20:10
*** bhuvan_ has quit IRC20:11
*** bhuvan has quit IRC20:11
*** denis_makogon_ is now known as denis_makogon20:11
*** dstanek is now known as dstanek_afk20:12
*** vkozhukalov has quit IRC20:13
*** rcarrillocruz has quit IRC20:13
*** jolyonbrown has quit IRC20:14
*** zns has joined #openstack-infra20:15
*** zzelle has joined #openstack-infra20:16
*** rcarrillocruz has joined #openstack-infra20:16
*** hashar has joined #openstack-infra20:17
*** aysyd has joined #openstack-infra20:17
*** bhuvan has joined #openstack-infra20:18
*** bhuvan_ has joined #openstack-infra20:18
*** gyee has joined #openstack-infra20:18
*** mbacchi has quit IRC20:19
*** arosen has joined #openstack-infra20:19
*** mbacchi has joined #openstack-infra20:20
*** andreaf has joined #openstack-infra20:20
*** andreaf has quit IRC20:21
*** sweston has joined #openstack-infra20:22
openstackgerritZiad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools  https://review.openstack.org/7905420:22
*** bhuvan__ has joined #openstack-infra20:23
*** bhuvan___ has joined #openstack-infra20:23
*** bhuvan_ has quit IRC20:24
*** bhuvan has quit IRC20:24
*** weshay has quit IRC20:26
*** rlandy has quit IRC20:27
fungipleia2: any objections to resolving https://launchpad.net/bugs/995592 ? at this point i've deleted the last of the untrusted static jenkins slaves, so i think the original need in that bug is solved (even if you're still working separately on being able to introduce fedora nodepool nodes, that seems like a separate project/bug at this point)20:28
*** sweston has quit IRC20:30
*** rcarrillocruz has quit IRC20:31
openstackgerritSlickNik proposed a change to openstack/requirements: Updating global_requirements for trove  https://review.openstack.org/7975620:33
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints  https://review.openstack.org/7975720:37
*** khyati_ has joined #openstack-infra20:39
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints  https://review.openstack.org/7975720:39
*** julim has quit IRC20:40
*** mrda_away is now known as mrda20:41
*** lcestari has quit IRC20:42
*** jgrimm has joined #openstack-infra20:42
fungiclarkb: jeblair: the combined work on graphite and logstash for job statistics means https://launchpad.net/bugs/1172391 is done, yeah?20:43
*** adalbas has quit IRC20:44
jeblairi think the idea there was to see like "is nova pegging the cpu" or something...20:44
fungiahh, got it20:44
jeblairthough we have something in devstack-gate doing that now...20:44
fungiso actually something like getting the sar data we currently archive into a more queryable system?20:45
jeblairsar, that's it20:45
jeblairsdague was looking at that recently too20:45
fungior is the fact that we can look at the sar report files individually enough for this?20:45
sdaguejeblair: so we dumped sar for dstat in devstack20:45
fungirighth that20:45
fungimuch more better sar20:45
jeblairsdague: is there more work needing to be done to make that useful?20:45
sdaguejeblair: it's useful on it's own now20:46
sdagueI have some ideas to be able to get us a little more data on the state of the world with dstat plugins20:46
sdaguewhich you can write in python20:46
fungijust wondering whether that bug should be closed if we've got what we need20:46
sdaguewhat bug is this?20:46
fungior if we really need fancier queryable aggregate data for devstack performance20:46
fungisdague: https://launchpad.net/bugs/117239120:47
*** mwagner_lap has quit IRC20:47
sdagueyeh, I think dstat is probably sufficient at this point20:47
sdaguethat seemed to get us what we needed to make the # of testr workers decision20:48
fungisdague: thanks. i'll consider you a primary stakeholder for that bug20:48
fungifix released then20:48
sdagueso I'd call that closed20:48
*** mark0n has joined #openstack-infra20:49
fungijeblair: any reason we might still want to move forward with https://launchpad.net/bugs/1172401 or is the automated rechecking plus whatever fun comes in via new gerrit enough?20:50
jeblairfungi: i lean toward the latter20:50
*** SumitNaiksatam has joined #openstack-infra20:51
*** amcrn has quit IRC20:51
fungijeblair: me too. i'll mark it fix released20:51
*** sarob_ has joined #openstack-infra20:51
mark0nhi, anybody familiar with Jenkins Job Builder? I'd like to clone two upstream Git repos into two branches of the same repo in the working directory. This is easy with the Jenkins web front-end but I didn't find a way to do this with JJB. Any ideas?20:52
*** freyes has joined #openstack-infra20:52
*** arborism has joined #openstack-infra20:53
mark0nfrom the web front-end I don't need to use MultiSCM but just the normal Git plugin to accomplish that20:54
*** lcestari has joined #openstack-infra20:55
*** bhuvan has joined #openstack-infra20:58
*** bhuvan_ has joined #openstack-infra20:58
*** rfolco has quit IRC20:59
*** melwitt has quit IRC21:00
*** melwitt has joined #openstack-infra21:00
fungipleia2: do you want to pass the torch on https://launchpad.net/bugs/1218631 to nibalizer and let him stick a status update in there when he has time?21:00
*** SumitNaiksatam has quit IRC21:00
*** e0ne has joined #openstack-infra21:00
*** markwash has joined #openstack-infra21:01
fungimark0n: we don't really use much in the way of jenkins plugins, and don't use the git plugin at all, so not sure what to suggest there (we just run a shell builder which does a git clone/clean/reset as needed before running the job)21:01
*** bhuvan___ has quit IRC21:02
*** bhuvan__ has quit IRC21:02
*** bhuvan has quit IRC21:03
*** bhuvan has joined #openstack-infra21:03
*** bhuvan__ has joined #openstack-infra21:03
*** bhuvan_ has quit IRC21:03
mark0nfungi: do you think it makes sense to try the openstack-infra mailing list?21:04
afazekasmark0n: I wonder isn't enough to create two - git: entry with different 'branches:' and 'basedir:' ?21:04
dolphmjeblair: before we play tennis with this bug, it does look like an -infra bug to me, but very low impact https://bugs.launchpad.net/keystone/+bug/128457921:05
*** sabari3 has left #openstack-infra21:05
jeblairdolphm: oh, maybe a devstack bug then21:06
jeblairdolphm: oh, no, because of a network error between the host and pypi?21:07
jeblairdolphm: i'll take it back and mark it wontfix, thanks.21:08
dolphmjeblair: that's what it looks like to me -- dogpile looks to be the first python dep not in /var/cache21:08
fungimark0n: possibly. there are some jjb users who subscribe to that but may not be around in irc right this moment who might use that plugin in other environments21:08
*** jeckersb is now known as jeckersb_gone21:08
mark0nafazekas: I tried that, but as soon as there are two "git:" entries, JJB generates XML code that requires MultiSCM and I didn't manage to clone into two branches of the _same_ repo with MultiSCM21:08
fungidolphm: jeblair: i want to say we already have at least one bug open on "Cannot fetch index base URL http://pypi.openstack.org/openstack/" so probably better as a duplicate21:09
* fungi needs to skedaddle to dinner... back later this evening21:10
clarkbmark0n: it would probably help to grab the xml jenkins generates and compare to the xml jjb generates21:10
clarkbmark0n: then we can convince jjb to do the right thing21:11
*** jeckersb_gone is now known as jeckersb21:13
nibalizerfungi: updated21:16
*** e0ne has quit IRC21:16
mark0nthe ultimate goal is to copy a branch of a Github repo to a branch in our company repo. With the web front-end I just configured two remotes and pushed to one of them (emty build section).21:16
*** e0ne has joined #openstack-infra21:16
*** jgriffith has quit IRC21:19
*** mailforjob has joined #openstack-infra21:20
*** jgriffith has joined #openstack-infra21:23
*** mailforjob has quit IRC21:24
*** StevenK_ has joined #openstack-infra21:25
*** HenryG has quit IRC21:28
*** markwash has quit IRC21:28
*** harlowja has quit IRC21:28
*** zzelle has quit IRC21:28
*** weiwei has quit IRC21:28
*** nati_ueno has quit IRC21:28
*** alexpilotti has quit IRC21:28
*** jungleboyj has quit IRC21:28
*** mkerrin has quit IRC21:28
*** sdake_ has quit IRC21:28
*** sandywalsh has quit IRC21:28
*** matrohon has quit IRC21:28
*** rwsu has quit IRC21:28
*** salv-orlando has quit IRC21:28
*** melwitt has quit IRC21:28
*** lcestari has quit IRC21:28
*** sarob_ has quit IRC21:28
*** mbacchi has quit IRC21:28
*** aysyd has quit IRC21:28
*** yassine has quit IRC21:28
*** moted has quit IRC21:28
*** mkoderer has quit IRC21:28
*** lcheng has quit IRC21:28
*** mgagne has quit IRC21:28
*** atiwari has quit IRC21:28
*** dims_ has quit IRC21:28
*** spredzy has quit IRC21:28
*** skraynev_afk has quit IRC21:28
*** luis_ has quit IRC21:28
*** Mithrandir has quit IRC21:28
*** ianw has quit IRC21:28
*** dstufft has quit IRC21:28
*** homeless has quit IRC21:28
*** pfallenop has quit IRC21:28
*** dkehn_ has quit IRC21:28
*** amrith has quit IRC21:28
*** jd__ has quit IRC21:28
*** jamespage has quit IRC21:28
*** tteggel has quit IRC21:28
*** EntropyWorks has quit IRC21:28
*** annegentle has quit IRC21:28
*** dansmith has quit IRC21:28
*** rainya has quit IRC21:28
*** stevebaker has quit IRC21:28
*** dragondm has quit IRC21:28
*** tristanC has quit IRC21:28
*** bookwar has quit IRC21:28
*** zhiyan_ has quit IRC21:28
*** wayneeseguin has quit IRC21:28
*** tnurlygayanov has quit IRC21:28
*** bhuvan__ has quit IRC21:28
*** bhuvan has quit IRC21:28
*** jgrimm has quit IRC21:28
*** gyee has quit IRC21:28
*** zns has quit IRC21:28
*** mriedem1 has quit IRC21:28
*** nicedice has quit IRC21:28
*** wchrisj has quit IRC21:28
*** mrodden has quit IRC21:28
*** ArxCruz has quit IRC21:28
*** julienvey has quit IRC21:28
*** antonym has quit IRC21:28
*** bknudson has quit IRC21:28
*** mrda has quit IRC21:28
*** JoshNang has quit IRC21:28
*** simonmcc has quit IRC21:28
*** dangers_away has quit IRC21:28
*** loquacities has quit IRC21:28
*** nibalizer has quit IRC21:28
*** sdague has quit IRC21:28
*** Alex_Gaynor has quit IRC21:28
*** asadoughi has quit IRC21:28
*** cyeoh has quit IRC21:28
*** shortstop has quit IRC21:28
*** denis_makogon has quit IRC21:28
*** markmc has quit IRC21:28
*** VijayTripathi1 has quit IRC21:28
*** mfer has quit IRC21:28
*** david-lyle has quit IRC21:28
*** caleb_ has quit IRC21:28
*** CaptTofu has quit IRC21:28
*** reed has quit IRC21:28
*** dkranz has quit IRC21:28
*** pcrews has quit IRC21:28
*** yamahata has quit IRC21:28
*** lcostantino has quit IRC21:28
*** dripton has quit IRC21:28
*** cody-somerville has quit IRC21:28
*** AJaeger has quit IRC21:28
*** chandan_kumar has quit IRC21:28
*** mtreinish has quit IRC21:28
*** esmute has quit IRC21:28
*** Kiall has quit IRC21:28
*** ilyashakhat has quit IRC21:28
*** vipul has quit IRC21:28
*** saschpe has quit IRC21:28
*** sbadia has quit IRC21:28
*** dkehn has quit IRC21:28
*** dhellmann has quit IRC21:28
*** dachary has quit IRC21:28
*** saper has quit IRC21:28
*** morgabra has quit IRC21:28
*** gaelL_ has quit IRC21:28
*** maurosr has quit IRC21:28
*** Shrews has quit IRC21:28
*** sahumada has quit IRC21:28
*** clarkb has quit IRC21:28
*** kevinbenton has quit IRC21:28
*** kmartin has quit IRC21:28
*** e0ne has quit IRC21:28
*** arborism has quit IRC21:28
*** hashar has quit IRC21:28
*** lukego has quit IRC21:28
*** ryanpetrello has quit IRC21:28
*** marun has quit IRC21:28
*** afazekas has quit IRC21:28
*** Ryan_Lane has quit IRC21:28
*** blamar has quit IRC21:28
*** changbl has quit IRC21:28
*** unicell has quit IRC21:28
*** vponomaryov has quit IRC21:28
*** pete5 has quit IRC21:28
*** gmoro has quit IRC21:28
*** juice has quit IRC21:28
*** davidlenwell has quit IRC21:28
*** AaronGreen has quit IRC21:28
*** mattoliverau has quit IRC21:28
*** jaypipes has quit IRC21:28
*** kashyap has quit IRC21:28
*** medieval1 has quit IRC21:28
*** yamahata_ has quit IRC21:28
*** jamielennox has quit IRC21:28
*** flaper87 has quit IRC21:28
*** bradm has quit IRC21:28
*** koolhead17 has quit IRC21:28
*** cp16net has quit IRC21:28
*** locke105 has quit IRC21:28
*** freyes has quit IRC21:28
*** khyati_ has quit IRC21:28
*** arosen has quit IRC21:28
*** Ajaeger1 has quit IRC21:28
*** thedodd has quit IRC21:28
*** bada has quit IRC21:28
*** rossella_s has quit IRC21:28
*** hogepodge has quit IRC21:28
*** Hefeweizen has quit IRC21:28
*** dstanek_afk has quit IRC21:28
*** jeblair has quit IRC21:28
*** openstackgerrit has quit IRC21:28
*** jpeeler has quit IRC21:28
*** jeremyb has quit IRC21:28
*** soren has quit IRC21:28
*** gema has quit IRC21:28
*** alaski has quit IRC21:28
*** NikitaKonovalov has quit IRC21:28
*** michchap has quit IRC21:28
*** westmaas has quit IRC21:28
*** DuncanT- has quit IRC21:28
*** SergeyLukjanov has quit IRC21:28
*** ruhe has quit IRC21:28
*** DinaBelova has quit IRC21:28
*** ogelbukh has quit IRC21:28
*** jasond` has quit IRC21:28
*** gothicmindfood has quit IRC21:28
*** notmyname has quit IRC21:28
*** pleia2 has quit IRC21:28
*** comstud has quit IRC21:28
*** Vivek has quit IRC21:28
*** niska has quit IRC21:28
*** samalba has quit IRC21:28
*** geekinutah has quit IRC21:28
*** rahmu has quit IRC21:28
*** plomakin_ has quit IRC21:28
*** rdwrer has quit IRC21:28
*** dosaboy has quit IRC21:28
*** vladan has quit IRC21:28
*** spiffxp_ has quit IRC21:28
*** jcooley has quit IRC21:28
*** belliott has quit IRC21:28
*** adarazs_ has quit IRC21:28
*** jogo has quit IRC21:28
*** guitarzan has quit IRC21:28
*** mburned has quit IRC21:28
*** aude has quit IRC21:28
*** Steap has quit IRC21:28
*** gilliard has quit IRC21:28
*** lifeless has quit IRC21:28
*** mika has quit IRC21:28
*** jesusaurus has quit IRC21:28
*** primeministerp has quit IRC21:28
*** gpocentek has quit IRC21:28
*** mordred has quit IRC21:28
*** ihrachys has quit IRC21:28
*** dtroyer has quit IRC21:28
*** Daviey has quit IRC21:28
*** markmcclain has quit IRC21:28
*** dolphm has quit IRC21:28
*** greghaynes has quit IRC21:28
*** pabelanger has quit IRC21:28
*** sileht has quit IRC21:28
*** boris-42 has quit IRC21:28
*** Adri2000 has quit IRC21:28
*** StevenK has quit IRC21:28
*** JayF has quit IRC21:28
*** EmilienM has quit IRC21:28
*** chmouel has quit IRC21:29
*** jroll has quit IRC21:29
*** pvo has quit IRC21:29
*** Guest34377 has quit IRC21:29
*** wendar has quit IRC21:29
*** jbryce has quit IRC21:29
*** russellb has quit IRC21:29
*** jlk has quit IRC21:29
*** adam_g has quit IRC21:29
*** ttx has quit IRC21:29
*** gondoi has quit IRC21:29
*** msuriar has quit IRC21:29
*** mikal has quit IRC21:29
*** jeckersb has quit IRC21:29
*** hughsaunders has quit IRC21:29
*** isviridov has quit IRC21:29
*** fbo_away has quit IRC21:29
*** ChanServ has quit IRC21:29
*** morganfainberg has quit IRC21:29
*** mark0n has quit IRC21:29
*** rms_13 has quit IRC21:29
*** psedlak has quit IRC21:29
*** Ng has quit IRC21:29
*** ekarlso has quit IRC21:29
*** Hunner has quit IRC21:29
*** bodepd has quit IRC21:29
*** zaro has quit IRC21:29
*** malini_afk has quit IRC21:29
*** bnemec has quit IRC21:29
*** therve has quit IRC21:29
*** anteaya has quit IRC21:29
*** tru_tru has quit IRC21:29
*** tchaypo has quit IRC21:29
*** jomara has quit IRC21:29
*** portante has quit IRC21:29
*** hdd has quit IRC21:29
*** hub_cap has quit IRC21:29
*** phschwartz has quit IRC21:29
*** med_ has quit IRC21:29
*** shardy has quit IRC21:29
*** NobodyCam has quit IRC21:29
*** devananda has quit IRC21:29
*** SlickNik has quit IRC21:29
*** gondoi_ has joined #openstack-infra21:29
*** gondoi_ is now known as gondoi21:29
*** pabelanger_ has joined #openstack-infra21:29
*** jeckersb has joined #openstack-infra21:29
*** krotscheck has quit IRC21:30
*** sileht has joined #openstack-infra21:30
*** Adri2000 has joined #openstack-infra21:30
*** krotscheck has joined #openstack-infra21:32
*** pabelanger_ is now known as pabelanger21:34
*** krotscheck has quit IRC21:35
*** zhiyan_ has joined #openstack-infra21:38
*** yassine has joined #openstack-infra21:38
*** lcheng has joined #openstack-infra21:38
*** atiwari has joined #openstack-infra21:38
*** jd__ has joined #openstack-infra21:38
*** jd__ is now known as 23LAAONZW21:41
*** chmouel has joined #openstack-infra21:41
*** wayneeseguin has joined #openstack-infra21:41
*** melwitt has joined #openstack-infra21:41
*** homeless has joined #openstack-infra21:41
*** EntropyWorks has joined #openstack-infra21:41
*** wendar has joined #openstack-infra21:41
*** jbryce has joined #openstack-infra21:41
*** bookwar has joined #openstack-infra21:41
*** dansmith has joined #openstack-infra21:41
*** russellb has joined #openstack-infra21:41
*** pvo has joined #openstack-infra21:41
*** tteggel has joined #openstack-infra21:41
*** skraynev_afk has joined #openstack-infra21:41
*** dims_ has joined #openstack-infra21:41
*** sarob has joined #openstack-infra21:41
*** jroll has joined #openstack-infra21:41
*** luis_ has joined #openstack-infra21:41
*** dkehn_ has joined #openstack-infra21:41
*** amrith has joined #openstack-infra21:41
*** ianw has joined #openstack-infra21:41
*** mbacchi has joined #openstack-infra21:41
*** aysyd has joined #openstack-infra21:41
*** moted has joined #openstack-infra21:41
*** mgagne has joined #openstack-infra21:41
*** mkoderer has joined #openstack-infra21:41
*** tristanC has joined #openstack-infra21:41
*** Guest96737 has joined #openstack-infra21:41
*** jlk has joined #openstack-infra21:41
*** spredzy has joined #openstack-infra21:41
*** annegentle has joined #openstack-infra21:41
*** adam_g has joined #openstack-infra21:41
*** stevebaker has joined #openstack-infra21:41
*** ttx has joined #openstack-infra21:41
*** pfallenop has joined #openstack-infra21:41
*** ociuhandu has joined #openstack-infra21:41
*** krotscheck_ has joined #openstack-infra21:41
*** isviridov has joined #openstack-infra21:41
*** greghaynes has joined #openstack-infra21:41
*** sandywalsh_ has joined #openstack-infra21:41
*** hughsaunders has joined #openstack-infra21:41
*** HenryG_ has joined #openstack-infra21:41
*** dolphm has joined #openstack-infra21:41
*** fbo has joined #openstack-infra21:41
*** EmilienM has joined #openstack-infra21:41
*** jooools has joined #openstack-infra21:41
*** derekh has joined #openstack-infra21:41
*** boris-42 has joined #openstack-infra21:41
*** mikal has joined #openstack-infra21:41
*** JayF has joined #openstack-infra21:41
*** msuriar_ has joined #openstack-infra21:41
*** markmcclain1 has joined #openstack-infra21:41
*** rpodolyaka has joined #openstack-infra21:41
*** weshay has joined #openstack-infra21:41
*** ildikov_ has joined #openstack-infra21:41
*** mriedem has joined #openstack-infra21:41
*** e0ne has joined #openstack-infra21:41
*** arborism has joined #openstack-infra21:41
*** freyes has joined #openstack-infra21:41
*** jgrimm has joined #openstack-infra21:41
*** khyati_ has joined #openstack-infra21:41
*** arosen has joined #openstack-infra21:41
*** gyee has joined #openstack-infra21:41
*** hashar has joined #openstack-infra21:41
*** zns has joined #openstack-infra21:41
*** lukego has joined #openstack-infra21:41
*** denis_makogon has joined #openstack-infra21:41
*** markmc has joined #openstack-infra21:41
*** VijayTripathi1 has joined #openstack-infra21:41
*** Ajaeger1 has joined #openstack-infra21:41
*** ryanpetrello has joined #openstack-infra21:41
*** david-lyle has joined #openstack-infra21:41
*** marun has joined #openstack-infra21:41
*** caleb_ has joined #openstack-infra21:41
*** rms_13 has joined #openstack-infra21:41
*** afazekas has joined #openstack-infra21:41
*** CaptTofu has joined #openstack-infra21:41
*** nicedice has joined #openstack-infra21:41
*** Ryan_Lane has joined #openstack-infra21:41
*** wchrisj has joined #openstack-infra21:41
*** blamar has joined #openstack-infra21:41
*** thedodd has joined #openstack-infra21:41
*** reed has joined #openstack-infra21:41
*** pcrews has joined #openstack-infra21:41
*** changbl has joined #openstack-infra21:41
*** mrodden has joined #openstack-infra21:41
*** bada has joined #openstack-infra21:41
*** yamahata has joined #openstack-infra21:41
*** ArxCruz has joined #openstack-infra21:41
*** unicell has joined #openstack-infra21:41
*** julienvey has joined #openstack-infra21:41
*** vponomaryov has joined #openstack-infra21:41
*** rossella_s has joined #openstack-infra21:41
*** kashyap has joined #openstack-infra21:41
*** dripton has joined #openstack-infra21:41
*** antonym has joined #openstack-infra21:41
*** bknudson has joined #openstack-infra21:41
*** pete5 has joined #openstack-infra21:41
*** hogepodge has joined #openstack-infra21:41
*** gmoro has joined #openstack-infra21:41
*** Hefeweizen has joined #openstack-infra21:41
*** cody-somerville has joined #openstack-infra21:41
*** juice has joined #openstack-infra21:41
*** davidlenwell has joined #openstack-infra21:41
*** AaronGreen has joined #openstack-infra21:41
*** AJaeger has joined #openstack-infra21:41
*** mattoliverau has joined #openstack-infra21:41
*** chandan_kumar has joined #openstack-infra21:41
*** mtreinish has joined #openstack-infra21:41
*** jaypipes has joined #openstack-infra21:41
*** mrda has joined #openstack-infra21:41
*** JoshNang has joined #openstack-infra21:41
*** simonmcc has joined #openstack-infra21:41
*** dangers_away has joined #openstack-infra21:41
*** dstanek_afk has joined #openstack-infra21:41
*** psedlak has joined #openstack-infra21:41
*** loquacities has joined #openstack-infra21:41
*** jeblair has joined #openstack-infra21:41
*** nibalizer has joined #openstack-infra21:41
*** esmute has joined #openstack-infra21:41
*** openstackgerrit has joined #openstack-infra21:41
*** jpeeler has joined #openstack-infra21:41
*** medieval1 has joined #openstack-infra21:41
*** sdague has joined #openstack-infra21:41
*** Alex_Gaynor has joined #openstack-infra21:41
*** shortstop has joined #openstack-infra21:41
*** cyeoh has joined #openstack-infra21:41
*** asadoughi has joined #openstack-infra21:41
*** jeremyb has joined #openstack-infra21:41
*** ilyashakhat has joined #openstack-infra21:41
*** vipul has joined #openstack-infra21:41
*** soren has joined #openstack-infra21:41
*** saschpe has joined #openstack-infra21:41
*** yamahata_ has joined #openstack-infra21:41
*** sbadia has joined #openstack-infra21:41
*** dkehn has joined #openstack-infra21:41
*** dhellmann has joined #openstack-infra21:41
*** jamielennox has joined #openstack-infra21:41
*** gema has joined #openstack-infra21:41
*** flaper87 has joined #openstack-infra21:41
*** dachary has joined #openstack-infra21:41
*** saper has joined #openstack-infra21:41
*** Ng has joined #openstack-infra21:41
*** morgabra has joined #openstack-infra21:41
*** malini_afk has joined #openstack-infra21:41
*** jomara has joined #openstack-infra21:41
*** bradm has joined #openstack-infra21:41
*** gaelL_ has joined #openstack-infra21:41
*** maurosr has joined #openstack-infra21:41
*** alaski has joined #openstack-infra21:41
*** Shrews has joined #openstack-infra21:41
*** NikitaKonovalov has joined #openstack-infra21:41
*** ekarlso has joined #openstack-infra21:41
*** gilliard has joined #openstack-infra21:41
*** michchap has joined #openstack-infra21:41
*** sahumada has joined #openstack-infra21:41
*** clarkb has joined #openstack-infra21:41
*** koolhead17 has joined #openstack-infra21:41
*** westmaas has joined #openstack-infra21:41
*** kevinbenton has joined #openstack-infra21:41
*** DuncanT- has joined #openstack-infra21:41
*** kmartin has joined #openstack-infra21:41
*** SergeyLukjanov has joined #openstack-infra21:41
*** ruhe has joined #openstack-infra21:41
*** DinaBelova has joined #openstack-infra21:41
*** Hunner has joined #openstack-infra21:41
*** ogelbukh has joined #openstack-infra21:41
*** bodepd has joined #openstack-infra21:41
*** zaro has joined #openstack-infra21:41
*** jasond` has joined #openstack-infra21:41
*** gothicmindfood has joined #openstack-infra21:41
*** notmyname has joined #openstack-infra21:41
*** pleia2 has joined #openstack-infra21:41
*** comstud has joined #openstack-infra21:41
*** Vivek has joined #openstack-infra21:41
*** niska has joined #openstack-infra21:41
*** samalba has joined #openstack-infra21:41
*** geekinutah has joined #openstack-infra21:41
*** bnemec has joined #openstack-infra21:41
*** therve has joined #openstack-infra21:41
*** rdwrer has joined #openstack-infra21:41
*** cp16net has joined #openstack-infra21:41
*** phschwartz has joined #openstack-infra21:41
*** locke105 has joined #openstack-infra21:41
*** Steap has joined #openstack-infra21:41
*** rahmu has joined #openstack-infra21:41
*** plomakin_ has joined #openstack-infra21:41
*** dosaboy has joined #openstack-infra21:41
*** tchaypo has joined #openstack-infra21:41
*** vladan has joined #openstack-infra21:41
*** spiffxp_ has joined #openstack-infra21:41
*** belliott has joined #openstack-infra21:41
*** jcooley has joined #openstack-infra21:41
*** adarazs_ has joined #openstack-infra21:41
*** jogo has joined #openstack-infra21:41
*** guitarzan has joined #openstack-infra21:41
*** mburned_out has joined #openstack-infra21:41
*** anteaya has joined #openstack-infra21:41
*** tru_tru has joined #openstack-infra21:41
*** aude has joined #openstack-infra21:41
*** portante has joined #openstack-infra21:41
*** lifeless has joined #openstack-infra21:41
*** mika has joined #openstack-infra21:41
*** jesusaurus has joined #openstack-infra21:41
*** primeministerp has joined #openstack-infra21:41
*** gpocentek has joined #openstack-infra21:41
*** mordred has joined #openstack-infra21:41
*** ihrachys has joined #openstack-infra21:41
*** Daviey has joined #openstack-infra21:41
*** dtroyer has joined #openstack-infra21:41
*** hdd has joined #openstack-infra21:41
*** hub_cap has joined #openstack-infra21:41
*** med_ has joined #openstack-infra21:41
*** morganfainberg has joined #openstack-infra21:41
*** ChanServ has joined #openstack-infra21:41
*** shardy has joined #openstack-infra21:41
*** NobodyCam has joined #openstack-infra21:41
*** devananda has joined #openstack-infra21:41
*** SlickNik has joined #openstack-infra21:41
*** dickson.freenode.net sets mode: +o ChanServ21:41
*** rainya has joined #openstack-infra21:41
*** jeckersb has quit IRC21:41
*** jeckersb has joined #openstack-infra21:41
*** atiwari has quit IRC21:41
*** atiwari has joined #openstack-infra21:41
clarkbafazekas: does `ssh -p29418 review.openstack.org gerrit ls-projects` work for you? that is a cheap way to test ssh connectivity21:41
*** bhuvan has joined #openstack-infra21:41
*** bhuvan has quit IRC21:41
*** bhuvan has joined #openstack-infra21:41
clarkbthat works for me and the web UI seems to work too21:41
*** pabelanger has quit IRC21:41
*** pabelanger has joined #openstack-infra21:41
*** 23LAAONZW is now known as jd__`21:41
*** jhesketh_ has joined #openstack-infra21:41
*** kiall has joined #openstack-infra21:42
*** Mithrandir has joined #openstack-infra21:42
*** Adri2000 is now known as Guest6828321:42
*** jd__` is now known as jd__21:42
*** krotscheck_ is now known as krotscheck21:42
*** dragondm has joined #openstack-infra21:42
*** dkliban has quit IRC21:42
*** nicedice_ has joined #openstack-infra21:44
*** mriedem has quit IRC21:44
*** mbacchi has quit IRC21:45
*** harlowja has joined #openstack-infra21:45
*** matrohon has joined #openstack-infra21:45
afazekasclarkb: the same issue21:45
*** sdake_ has joined #openstack-infra21:46
clarkbafazekas: I think it must be a connectivity thing. what does `mtr review.openstack.org` show21:46
*** bhuvan_ has joined #openstack-infra21:46
*** rwsu has joined #openstack-infra21:46
pleia2clarkb: I seem to recall you being involved with the translations stuff, it's on our priorities list, what on our end do we need to move this forward?21:46
clarkbpleia2: context? I think translation stuff is moving21:47
afazekasclarkb: I have connection, I got data in both direction, than it stops working in select(2)21:47
clarkbafazekas: huh21:47
*** nati_ueno has joined #openstack-infra21:47
pleia2clarkb: looking at our https://etherpad.openstack.org/p/infrastructure-priorities and trying to find something to work on (tripleo stuff is winding down for me now that things are coming together)21:47
*** jerryz has joined #openstack-infra21:48
clarkbfungi: https://review.openstack.org/#/c/75213/ do we want to try pushing that through to help rax pypy nodes?21:48
*** nicedice has quit IRC21:48
clarkbpleia2: ah right those translation tools. I think we need to swing around with the translation folks again and see if there is some consensus on a tool that is open21:48
*** dstufft has joined #openstack-infra21:49
clarkbpleia2: that was a havana session with a few strong opinions :) then once we know what we need to be running spin up a server for that21:49
clarkbwith puppeting and things21:49
clarkbpleia2: jeblair already has a dev pootle server but ryan_lane had people convinced we should look at the mediawiki related thing too21:49
pleia2clarkb: yeah, just saw it on our prioirties list and wondered if there was something I could do there21:50
pleia2like, look at mediawiki things or something21:50
*** freyes has quit IRC21:50
clarkbpleia2: ya that probably wouldn't be terrible also checking with daisy to see if they did any evaluation on their side whcih I thought they said they would during the summit21:50
*** markmcclain1 has quit IRC21:51
pleia2the translations folks seem happy to continue to use tranxifex, in spite of proprietaryness, so I think we'll need to nudge them21:51
pleia2clarkb: ok, I can follow up with daisy then21:51
*** ryanpetrello has quit IRC21:51
clarkbright, nudging is probably necessary21:51
*** mrodden has quit IRC21:51
*** tnurlygayanov has joined #openstack-infra21:52
clarkbdid we lose fungi and jeblair in the splits?21:52
*** Ajaeger1 has quit IRC21:53
mattoliverauphew reading scroll back takes a while when you join at the end of the US day :P So bug day.. is there anything I can help with, now that I'm back from Rackspace international trips?21:53
clarkbmattoliverau: we are currently working through https://etherpad.openstack.org/p/cibugreview-march2014 updating bugs as appropriate21:54
clarkbmattoliverau: always good to update bugs assigned to you, and if you have questions comments about other bugs feel free to bring them up21:54
mattoliverauCools, time to start reading then :)21:55
sdagueclarkb: https://review.openstack.org/7978421:55
sdaguesorry, it was an irc meeting afternoon21:55
sdaguethat will turn the logs back to sanity by default21:56
clarkbsdague: whats with trueorfalse instead of using the variable with default value as on the previous line?21:57
sdaguebecause it coalleses everything into true or false in a nice way21:57
*** e0ne has quit IRC21:57
sdagueregardless of the value provided21:57
*** markmc has quit IRC21:57
clarkbgotcha so 0 1 True false yes no get normalized?21:58
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews  https://review.openstack.org/7928021:59
sdagueanyway, going to dinner, but back on from hackerspace later. So I'll make sure that moves through the gate tonight21:59
clarkbsdague: awesome thank you, +1 from me21:59
*** alexandra has joined #openstack-infra21:59
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port  https://review.openstack.org/7928122:00
*** ryanpetrello has joined #openstack-infra22:00
jeblairclarkb, pleia2: yeah, i think checking with daisy is the next step22:00
jeblairpleia2: thanks for picking that up, that would be really helpful22:01
*** hashar has quit IRC22:01
clarkbjeblair: https://bugs.launchpad.net/openstack-ci/+bug/1240065 is the bug similar to what I saw with zuul mergers right? I think it might be beneficial to add the sigusr2 handler to statusbot to help debug. What do you think?22:02
pleia2jeblair: sure thing, translation tools are interesting to me anyway so it seemed like a good fit22:03
*** weiwei has joined #openstack-infra22:03
jeblairclarkb: afaik all known statusbot issues should be resolved; that is, once we merge some changes and get it running again22:03
*** sarob has quit IRC22:03
clarkbjeblair: gotcha22:03
jeblairclarkb: maybe there will be more once we run it22:04
jeblairspeaking of which, having it running tomorrow morning would be really nice22:04
clarkb:) I will leave that bug as is for now until we get it running again22:04
*** sarob has joined #openstack-infra22:04
*** bhuvan_ has quit IRC22:04
* jeblair merges some statusbot changes22:04
jerryzclarkb: is there a delay for the gerrit refs to be synced to git mirrors? now my gerrit triggered jobs can not get the refs from git.openstack.org22:05
*** bhuvan has quit IRC22:05
clarkbjerryz: yes there is a delay22:05
clarkbgerrit has to push changes out to the replicas after they happen22:05
*** ryanpetrello has quit IRC22:05
*** bhuvan has joined #openstack-infra22:06
*** bhuvan_ has joined #openstack-infra22:06
jeblairjerryz: best solutions would be to either use zuul, or sleep and retry from git.o.o22:06
jerryzclarkb: if the delay is so bad, i guess it is the same for zuul refs. how openstack jenkins jobs can get zuul refs from git.openstack.org instantly?22:06
jeblairjerryz: (those are better than having a lot of jobs hit gerrit directly)22:06
jeblairjerryz: they get refs from zuul mergers, and the mergers get them from gerrit22:07
*** jooools has quit IRC22:07
jeblairjerryz: so it's one fetch to get it to the merger, then hundreds of fetches from the workers hit the mergers instead of gerrit22:07
jeblairjerryz: so most workers (a) clone from a local cache or git.o.o; (b) update the repo to current state from git.o.o; (c) fetch changes to test from zuul merger22:08
*** arosen has quit IRC22:08
clarkbpleia2: jeblair https://bugs.launchpad.net/openstack-ci/+bug/1257964 that was completed around LCA right?22:08
clarkbiirc you guys sat in an airport and did that22:09
*** sarob has quit IRC22:09
*** arosen has joined #openstack-infra22:09
jeblairclarkb: i think so; if pleia2 confirms, we can delete the openstack-ci/publications repo22:09
*** ociuhandu has quit IRC22:09
jeblairthen close the bug22:10
*** StevenK_ is now known as StevenK22:10
pleia2clarkb: ah, yes! I can close22:10
clarkbpleia2: I think jeblair wants the old repo deleted first. I will let him do that as he is most familiar with it at this point22:11
jeblairi'm doing it right now22:11
pleia2ok, closing22:11
pleia2thanks jeblair22:11
*** lukego has quit IRC22:11
clarkbman github is slow today22:13
jeblairi just got a tcp rst from it22:13
clarkbjeblair: I would like to close https://bugs.launchpad.net/openstack-ci/+bug/1260983 but there is an upstream scp plugin patch that needs merging in github22:13
clarkbjeblair: do you want to keep out bug open as a reminder to fix upstream?22:13
jeblairclarkb: are we running that patch in prod?22:14
clarkbjeblair: yes22:14
jeblairjeblair merged commit 7ce951d into jenkinsci:master from zaro0508:fix-FileNotFound a month ago22:14
jeblairJames E. Blair jeblair closed this a month ago22:14
clarkbthough it may not be necessary anymore with the downgrade22:14
jeblairclarkb: ^22:15
clarkbjeblair: great, I was going to check that but github...22:15
clarkbjeblair: I am closing our bug now thanks22:15
clarkbreleasestatus is CD right? I think we can close https://bugs.launchpad.net/openstack-ci/+bug/126289122:17
*** salv-orlando has joined #openstack-infra22:18
jeblairclarkb: i think so22:18
pleia2clarkb: I think ttx still manages that himself, so.. probably22:18
* clarkb investigates22:18
pleia2it's on our list to move to us22:18
jerryzjeblair:  merger is in zuul, right?  workers get the code base from git.o.o and changes from zuul mergers. but i can't find where devstack code references zuul merger. i can only see a GIT_BASE as remote22:18
*** alexandra is now known as alex-backinbed22:19
jeblairpleia2: there's a whole module for it in puppet22:19
pleia2jeblair: oh, maybe I'm thinking of something else then :\22:19
clarkbyup its on status.o.o running via puppte and I have confirmed that the commit to fix it is there22:19
*** flaper87 is now known as flaper87|afk22:19
jeblairjerryz: yes, merger is in zuul; devstack-gate handles the repo setup for us, and it doesn't reference zuul directly, it uses the value of ZUUL_URL to get the merger to fetch from22:19
clarkbI have marked it fix released22:20
clarkbwell when I don't get a timeout from lp I will do that22:20
*** mrodden has joined #openstack-infra22:22
openstackgerritA change was merged to openstack-infra/statusbot: Handle lots of channels better  https://review.openstack.org/7903422:24
*** enikanorov_ has quit IRC22:24
openstackgerritA change was merged to openstack-infra/statusbot: Support SSL  https://review.openstack.org/7904722:24
*** yassine has quit IRC22:24
*** mrodden1 has joined #openstack-infra22:24
openstackgerritA change was merged to openstack-infra/zuul: Make zuul-merger default pidfile match init script  https://review.openstack.org/7910922:24
clarkboh I had intended on reading 79047 since I suggested it...22:24
* clarkb does that now22:24
clarkbnice and simple22:25
*** dizquierdo has joined #openstack-infra22:25
*** afazekas has quit IRC22:26
*** denis_makogon has quit IRC22:26
harlowjaarg, stop breaking github people, lol22:27
*** mrodden has quit IRC22:27
clarkbharlowja: we don't use github22:27
clarkbso someone else broke it22:27
harlowjai know ;)22:27
*** rossella_s has quit IRC22:27
harlowjato much pushing off openstack code to github22:27
harlowjaoverflowed the tubes22:27
jogoclarkb: where is the code that generates '[SCP] Trying to create /srv/static/logs/61/79661/1/gate/gate-tempest-dsvm-postgres-full/b7c0133'22:29
jogothat log msg22:29
jogoI was thinking of adding a link to the logs.o.o location in there22:29
jogoto make my life easier22:29
clarkbjogo: in the jenkins scp plugin22:29
*** dims_ has quit IRC22:30
jogoclarkb: ohh thats not our code directly?22:30
jeblairjogo: there's supposed to be a link there already due to jjb22:30
*** andreaf has joined #openstack-infra22:31
jeblairjogo: 2014-03-11 21:25:35.915 | Detailed logs: http://logs.openstack.org/49/78749/4/check/check-tempest-dsvm-full/c72a6ab/22:31
hallynlibvirt build keeps hanging on make check...  trying on another box.22:31
jeblairjogo: from https://jenkins01.openstack.org/job/check-tempest-dsvm-full/10481/console22:31
clarkbjeblair: I marked https://bugs.launchpad.net/openstack-ci/+bug/1270288 fix released and assigned it to you22:31
jogojeblair: I didn't see that link in https://jenkins01.openstack.org/job/gate-tempest-dsvm-postgres-full/8289/console22:32
clarkbbecause it failed and jenkins short circuits22:32
clarkbwe could put that shell blob first22:32
clarkband link at the beginning rather than the end22:32
jeblairclarkb: not a bad idea22:33
jeblairjogo: we'll have more control over that kind of output when we move logs to swift (because of the way the log upload will work)22:34
jogojeblair: woot to swift, but in the meantime I like clarkbs idea22:34
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool  https://review.openstack.org/7978922:35
jogoclarkb: do I just need to move the link-logs builder before the shell one?22:35
* pleia2 lost change-id22:35
clarkbjeblair: fungi https://bugs.launchpad.net/openstack-ci/+bug/1270321 I seem to briefly remember what happened there but zuul supports modern statsd now so whatever22:35
clarkbjogo: yup22:35
clarkbJJB creates them in order22:35
jogoclarkb: patch coming soon22:35
clarkband jenkins respects that22:35
* jogo likes how things in ifra do what you think they do22:35
jeblairclarkb: wfm22:36
*** Sukhdev has joined #openstack-infra22:36
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool  https://review.openstack.org/6951022:36
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Ensure there's always an http server object  https://review.openstack.org/7979222:37
jeblairclarkb: that fixes the nondeterministic zuul test error seen in your pidfile change22:38
clarkbjeblair: woot /me reviews22:38
*** thuc has quit IRC22:39
jogoclarkb: should link-logs go before or after devstack-checkout22:39
jogoI was going with after, but didn't put much thought into it22:39
*** thuc has joined #openstack-infra22:40
clarkbjeblair: https://bugs.launchpad.net/openstack-ci/+bug/1289432 is that related to the gear patch you had questions about?22:40
jeblairjogo: maybe before -- make it the first thing in every job?22:40
clarkbjogo: jeblair ya first thing will make it easier to find22:40
jeblairclarkb: re gear: i don't _think_ so...22:42
clarkbfungi: https://review.openstack.org/#/c/71323/ was tested with manage projects on Friday22:43
clarkbfungi: it seems to have worked well, but we didn't test any of the other jeepyb commands. Could you review that an possibly approve if you think it is safe?22:43
*** yamahata has quit IRC22:43
clarkbfungi: mostly looking for an opinion on whether or not we should do more testing before approval happens22:43
clarkbmordred: https://review.openstack.org/#/c/77311/ needs some love22:44
*** openstackstatus has joined #openstack-infra22:44
jeblairstatusbot is joining...22:44
*** thuc has quit IRC22:44
*** ociuhandu has joined #openstack-infra22:45
*** dims_ has joined #openstack-infra22:45
jeblairso watching the statusbot debug log is like reading all of the openstack channels simultaneously...22:46
jeblairi mean, not like it... it is it.22:46
clarkbjeblair: :)22:46
jeblairit's actually much less of a firehose right now than i expected.22:47
*** wchrisj has quit IRC22:48
reedif you don't mind, can you please move your comments into another column or into comments so I can do the sums?22:48
*** jhesketh__ has joined #openstack-infra22:48
reeddoh :)22:48
reedprevious message is a decoy, to confuse spies22:49
openstackgerritJoe Gordon proposed a change to openstack-infra/config: run link-logs builder first  https://review.openstack.org/7979422:49
* jeblair moves his comments22:49
jogoclarkb jeblair: I think that should do it22:49
*** CaptTofu has quit IRC22:51
openstackgerritJoshua Harlow proposed a change to openstack/requirements: Update the keyring dependency  https://review.openstack.org/7979622:51
*** arborism is now known as amcrn22:55
hallynjamespage: http://people.canonical.com/~serge/libvirt-0.9.8-2ubuntu1822:56
jeblairfungi: should we work on prep for tomorrow?22:56
hallyndims_: sdague: ^22:56
*** melwitt has quit IRC22:58
*** jeckersb is now known as jeckersb_gone22:58
*** che-arne has joined #openstack-infra22:58
*** changbl has quit IRC22:59
*** alex-backinbed is now known as Alexandra22:59
*** thuc has joined #openstack-infra23:01
*** thuc has quit IRC23:01
*** thuc has joined #openstack-infra23:02
*** mgagne has quit IRC23:02
*** ArxCruz has quit IRC23:03
*** jungleboyj has joined #openstack-infra23:05
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews  https://review.openstack.org/7928023:06
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port  https://review.openstack.org/7928123:06
clarkbdtroyer: Alex_Gaynor jnoller etc https://bugs.launchpad.net/openstack-ci/+bug/1272598 who should be in the openstacksdk-ptl group? Would be great to get that bug closed23:07
clarkbbecause yay bug day23:07
clarkbpleia2: what do you do to get bug numbers? would be curious what they look like now compared to the numbers on the ehterpad23:08
*** david-lyle has quit IRC23:08
openstackgerritJamie Lennox proposed a change to openstack-infra/config: Add the kite (KDS) project and kiteclient project to gerrit/jenkins  https://review.openstack.org/7307423:08
*** rpodolyaka has quit IRC23:09
pleia2clarkb: I wrote this script to gather everything: https://github.com/pleia2/openstack-infra-scripts/blob/master/infra_bugday.py23:10
dims_hallyn, thanks will give it a whirl23:10
pleia2clarkb: grabbing the whole bug link, which includes the number23:10
*** caleb_ has quit IRC23:10
pleia2it's probably something like task.number though23:10
pleia2clarkb: ooh wait, I know what you're saying23:11
pleia2clarkb: right hand side of https://bugs.launchpad.net/openstack-ci/23:11
clarkbpleia2: I am confused :) I just want to see post bug day stats. Since we seem to be winding down23:11
clarkbpleia2: thanks23:11
pleia2I will post them to the list too, and on my blog to the planet23:11
*** rpodolyaka has joined #openstack-infra23:12
*** thuc has quit IRC23:12
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Add tox.ini, setup.py and do syntax cleanup  https://review.openstack.org/7980023:13
pleia2clarkb: we did very well today :)23:14
jeblairhopefully the next bug day will involve storyboard23:14
*** thedodd has quit IRC23:15
SlickNikHere's the requirements patch that I was mentioning earlier: https://review.openstack.org/#/c/79756/23:16
SlickNikclarkb / other infra folks, please take a look when you get a chance.Thanks!23:16
clarkbSlickNik: you actually want requirements core to look at it :)23:16
clarkbpleia2: we did? yay23:16
SlickNikclarkb: Oh, I didn't realize there was a separate requirements-core. :)23:17
jogoSergeyLukjanov: it loosk like savanna patches are all failing23:17
jogoBug 1257885 - horizon does not work with django 1.623:18
jogoall hits are in savanna jobs23:18
*** che-arne has quit IRC23:20
*** andreaf has quit IRC23:21
* fungi is back now for the evening23:28
fungiclarkb: we can push 75213 through, sure, the patch itself seems like a fine idea. but i already tried it on the current round of problem and it didn't make any difference. seems actually not at all dependent on the locally-installed bits of the system and entirely happening within the virtualenv anyway23:28
pleia2I think this is my favorite bug day post photo yet http://princessleia.com/journal/?p=927723:29
pleia2all bugs wrapped up nicely!23:29
jeblairpleia2: thanks!23:29
jeblairclarkb: what's the state of the idea of a create-db-per-test fixture for independent parallel tests?23:30
jeblairclarkb: specifically for mysql23:30
*** mriedem has joined #openstack-infra23:31
clarkbjeblair: its basically done. On single use slaves we give the openstack_citest users ALL privs23:31
jeblairclarkb: is there code to actually use it?23:31
clarkbjeblair: so tests can set up dbs per whatever23:31
*** lcheng has quit IRC23:31
clarkbjeblair: that is project specific. I know taskflow and storyboard take advantage of the permissions, but sure of any other projects23:31
jeblairclarkb: pretty sure storyboard doesn't since it's set up to log in as "openstack_cifail"23:32
clarkbjeblair: well storyboard devs yelled at me for the feature :)23:32
clarkbI thought that was because they were running into problems with existing tests23:32
jeblairclarkb: anyway, i'm trying to figure out the curernt state and help storyboard fix that23:33
jeblairclarkb: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/migration/test_migrations_base.py#n14023:33
jeblairthere's where the login is set...23:33
clarkbjeblair: gotcha, so any project logging in at openstack_citest can create and use however many schemas they want23:33
fungijeblair: clarkb: the only reason the log link was at the end of jobs originally was to support jenkins smtp reporter for periodic jobs since it only quoted the end of the log in the e-mail. completely obsolete reason now23:33
clarkbjeblair: I would probably do schema per tests because its easier to reason about but you could do schema per test process23:33
jeblairclarkb: can you walk me through how schema-per-anything works in the context of that file?23:34
jeblairclarkb: i agree it should be schema per test23:34
jeblairthough it seems like it should be in this file http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/db_fixture.py23:35
clarkbjeblair: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/migration/test_migrations_base.py#n269 in methods like that you would drop and create a DB particluar to that test, probably based on test name23:35
clarkbjeblair: yes, using a proper fixture is probbly better /me looks at db_fixture23:35
clarkbjeblair: I would add an argument to that fixture called schema_name or similar. Then in setUp do an if present DROP finally create schema_name23:36
jeblairclarkb: okay, i'm trying to reconcile what i'm reading with the idea that this feature is supposed to already exist in storyboard... but that create line looks like it would just "create databasy storyboard"23:36
clarkbjeblair: then that schema will be available to whichver test is using that fixture23:36
fungiclarkb: i think i'm okay with 71323, though close-pull-requests is still broken on a completely separate issue i've not been able to track down yet (either a change in the github api or a change to our openstack-gerrit account is making its orgs list empty)23:36
clarkbjeblair: it may be that storyboard wanted the feature not for DB per test but so that it could use DB with different name than provided23:37
clarkbfungi: ya I was looking at that. I can't figure out how it would be a pygithub problem23:37
fungiclarkb: so 71323 says "Closes-Bug: 1275824" but it really doesn't at all23:37
clarkbfungi: oh because the errors changed...23:38
fungijeblair: i'm getting started on prep for tomorrow once i finish catching up on dinnertime scrollback in here. then hopefully finally getting started on my delinquent track chair duties before i pass out for the night23:39
clarkbfungi: http://developer.github.com/v3/orgs/ is the documented api for that. https://github.com/jacquev6/PyGithub/blob/master/github/AuthenticatedUser.py#L824 is the code we should be running23:39
clarkbfungi: one of us may need to run it under pdb to figure out what is happening23:39
openstackgerritJamie Lennox proposed a change to openstack-infra/config: Add the kite (KDS) project and kiteclient project to gerrit/jenkins  https://review.openstack.org/7307423:39
jeblairclarkb: yeah, this file: http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/tests/db/migration/test_migrations.py#n4823:40
jeblairclarkb: is going to cause it to just "create database openstack_citest".23:40
jeblairwhich i'm betting isn't going to work.23:40
jeblairand also is not schema-per-anything23:40
nibalizerso i would like to give a talk to PuppetConf about what we do here, and my part in it, which to date hasn't been much, but should be lots of cool stuff by september23:41
clarkbjeblair: that should actually work23:41
nibalizerand the CFP closes this week so its kinda now or never23:41
clarkbjeblair: as long as no other tests try to use that schema23:41
nibalizerhow would you all feel about that?23:41
jeblairnibalizer: i feel great about it23:41
clarkbjeblair: so much more robust to do schema per test using the db fixture and stop worrying23:41
*** mgagne has joined #openstack-infra23:41
nibalizerits possible i could turn the larger puppet community on to this project and get more drive-bys and maybe a consistent contributor or two23:42
jeblairnibalizer: feel free to look at slides we've done from previous talks: http://docs.openstack.org/infra/publications/23:42
jeblairnibalizer: in particular "Code Review for Systems Administrators" may be of interest...23:42
nibalizerokay great!23:42
jeblairnibalizer: also " PuppetConf, 2012: Collaboratively managing the OpenStack project with Puppet"23:42
jeblairnibalizer: and you're gonna like this....23:42
nibalizeroh yea!23:42
nibalizeroh well if this has already been covered maybe i dont need to give this talk23:42
openstackgerritA change was merged to openstack-infra/zuul: Ensure there's always an http server object  https://review.openstack.org/7979223:42
jeblairnibalizer: that was 2 years ago!23:43
clarkband much has changed23:43
jeblairnibalizer: so much has (and will) have change by then!23:43
jeblairnibalizer: http://git.openstack.org/cgit/openstack-infra/publications/23:43
sdaguecool, looks like the libvirt debug change is in the gate23:43
jeblairnibalizer: feel free to _use_ those slides, and submit changes, or if you make a new set, push them as well.  :)23:44
jeblairnibalizer: they all go through code review in gerrit.  :)23:44
jeblairnibalizer: http://git.openstack.org/cgit/openstack-infra/publications/tree/README.rst23:44
*** thuc has joined #openstack-infra23:44
*** thuc has quit IRC23:44
nibalizerhaha oh man thats awesome23:44
*** thuc has joined #openstack-infra23:45
jeblairnibalizer: the repo gets automatically published in the usual way :)23:45
fungiclarkb: i've been running close-pull-requests as the github user on review-dev with various iterations of debugging in modified copies of the script over past weeks as i get a spare moment here or there. the get_groups() api call literally gives us an empty list for that account these days. i even logged into the github webui with that account and used the restful api calls for it and get the same23:45
fungibehavior. i think maybe they stopped returning orgs for groups to which your account belongs? we might be better off iterating over projects.yaml to decide what projects to act on instead anyway23:45
jeblairfungi: ++23:45
*** mwagner_lap has joined #openstack-infra23:45
nibalizeroh yes this is going to be sweet23:46
jeblairclarkb: so does a db fixture that does what we actually want exist?23:46
clarkbjeblair: I don't think so. Openstack went the route of global locking isntead of fixing this properly23:46
*** mgagne has quit IRC23:46
nibalizerjeblair: clarkb so do I have general approval to do this then? how many of you need to approve before I can fire off my proposal?23:46
clarkbjeblair: so we didn't realize it was an issue until taskflow and storyboard complained after we thought we fixed it23:46
clarkbnibalizer: go for it. We are pretty laid back I don't think you need much approval23:47
fungiclarkb: er, i just re-read what i wrote, and meant to say i've been running close-pull-requests as the github user on review.o.o (not review-dev)23:47
clarkbfungi: interesting, so github changed thair api23:47
nibalizerokay thanks!23:47
clarkbfungi: so you think we should just assume the org exists if it is in projects.yaml? I think that is fair23:47
jeblairkrotscheck, mordred: we just spent like 100 lines talking about storyboard and databases ^23:47
fungier, and it's not get_groups() (i was misremembering) it's get_orgs()23:47
fungiclarkb: yeah, that's what we probably need to have it do23:48
clarkbfungi: ya get_orgs(). Using projects.yaml that way sounds fine to me23:48
jeblairkrotscheck, mordred, clarkb: i think the short version is that what's in storyboard should work if we just unbreak it, but it's not what we really want (which is a schema-per-test fixture)23:48
clarkbjeblair: agreed23:48
fungibecause github is too much of a black box. i poured over their current api documentation and honestly can't figure out why it's not still working the way we have it written23:48
clarkbfungi: agreed, reading their docs implies it should work as is23:49
fungiand it did! up until a few months ago when it didn't any longer23:49
clarkbfungi: does GET /users/:user/orgs form work any better?23:50
clarkbfungi: we could ghub.get_user("nameofgithubuser").get_orgs() to make that work I think23:50
clarkbbut maybe not23:50
fungiclarkb: nope. it was giving me a webpage back with []23:50
fungieven when authenticated23:50
*** jhesketh__ has quit IRC23:51
fungii didn't get as far as stuffing that form into the python, but i'll give it a shot just in case23:51
fungibut the web api was giving me back an empty array23:51
*** jhesketh__ has joined #openstack-infra23:51
*** thuc has quit IRC23:52
*** zul has joined #openstack-infra23:52
*** zul has quit IRC23:52
clarkbjeblair: fungi https://bugs.launchpad.net/openstack-ci/+bug/1082759 any news on that front?23:52
clarkbit is a bit saddening that we talk about it at every summit I have been to but haven't been able to do much more than retarget to the next release23:52
*** bhuvan_ has quit IRC23:52
*** bhuvan has quit IRC23:52
fungiclarkb: supposedly the openstackid project is a prerequisite for that (though i've yet to hear an adequate explanation as to why) so i've been trying to prioritize reviews for it when they show up23:53
fungiin hopes that once it's running, the www site code will just magically fall from the sky and plop down into a public git repo or something23:54
clarkbfungi: I don't see anything on http://developer.github.com/changes/ indicated the /user/orgs api changed at all23:54
fungiclarkb: nor i. i checked there in hopes that it did23:55
*** dizquierdo has quit IRC23:55
jogoclarkb: https://review.openstack.org/#/c/78612/1/modules/openstack_project/files/zuul/layout.yaml23:55
jogoI am confused23:55
jogoline 938, why 2x check-tempest-dsvm-neutron-pg23:56
fungiclarkb: yeah, i just tried list(ghub.get_user("openstack-gerrit").get_orgs()) and it too returns []23:56
clarkbfungi: :( I say screw it use projects.yaml23:56
fungijogo: so that it runs twice23:57
fungiclarkb: yeah, it's not worth my time to keep guessing further23:57
*** caleb_ has joined #openstack-infra23:57
clarkbfungi: https://bugs.launchpad.net/openstack-ci/+bug/1253774 we never fixed that did we? I can probably whip up a nodepool change that does that quickly23:57
*** bhuvan has joined #openstack-infra23:57
*** bhuvan_ has joined #openstack-infra23:57
clarkbjogo: because sdague wanted more numbers23:57
jogowhy 2x? salv-orlando was wondering as was I23:57
fungiand an increased chance that nondeterminism in those jobs would block changes from the projects most likely to cause them23:58
sdaguejogo: because having an integrated job be able to pass code running < 4 devstack runs is bad23:58
jogosdague: ahhh23:58
sdagueand leads to problems because it's too easy to land a coin flip race bug23:58
jogonow it makes sense23:58
sdaguewhich neutron used to do a lot23:58
salv-orlandoI suspected it was by design. I just wanted to be sure I was not seeing double because I was drunk ;)23:58
*** malini_afk is now known as malini23:59
fungisalv-orlando: i just always divide by two on basic principle23:59
sdaguewe invented other jobs at one point, to help there, but then zuul let us define doubles23:59
sdagueso that was just simpler23:59
sdaguealso, I agree, it should be explained in the config somewhere, because it's completely tribal knowledge23:59
salv-orlandoI mean from the neutron perspective I think it's pretty much the same. We already executed 4 devstack runs per patch.23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!