Tuesday, 2014-02-04

lifelesswenlock: no00:00
lifelesswenlock: what are you trying to figure out :)00:00
clarkbwenlock: if ubuntu counts as debian00:00
clarkbwe do have centos slaves too00:00
wenlocklifeless, trying to figure out how to properly start zuul... and deal with it when the server is rebooted00:01
fungiwenlock: what does that have to do with jenkins slave platforms>00:01
fungi(we don't stop and start zuul on slaves)00:01
fungi(we don't even install zuul on slaves)00:02
wenlockfungi: nothing... sorry... was thinking about a different problem00:02
wenlockfungi, correct00:02
wenlockfungi, we combined our slave and master00:02
clarkbwenlock: in jeblair's opinion the proper way is by hand00:02
wenlockfungi, got confused00:02
lifelesswenlock: ok, so what are you trying to figure out by asking about debian slaves?00:02
*** virmitio has left #openstack-infra00:02
wenlockfungi, whole start stop mechanisum on if a server like the master is rebooted00:03
fungiclarkb: nodepool deletes seem to be happening fine still. only 8 nodes in total in that state now00:03
clarkbfungi: great00:03
wenlockfungi, which is correct, not a slave problem00:03
*** boris-42_ has quit IRC00:03
fungiwenlock: sudo update-rc.d zuul defaults00:04
*** ekarlso- has joined #openstack-infra00:04
wenlockfungi, perfect00:04
fungishould make it start at priority 20 on runlevels 2,3,4,5 and stop on 0,1,600:04
wenlockfungi, im going to give that a shot00:05
clarkbfungi: wenlock: or just add a service def for it that makes the service enabled but not running00:05
clarkband add a toggle for that in puppet so that jeblair can have it manual and others can have it automagical00:05
fungiyeah, if you want puppet to configure it, then what clarkb said00:06
*** markwash has joined #openstack-infra00:06
wenlockclarkb:.... trying to understand... why would you try not starting automatically... ?00:06
wenlockfungi.... oooh00:06
fungiwenlock: if you stopped it on purpose00:06
wenlockfungi, yep, another good reason00:06
fungihaving it suddenly come back on because puppet||reboot could make badness00:06
*** boris-4__ has quit IRC00:07
openstackgerritKhai Do proposed a change to openstack-infra/config: add pipelines and update comment filter for gerrit 2.8  https://review.openstack.org/7086500:08
zaroclarkb: ^00:09
*** gokrokve has joined #openstack-infra00:09
*** yamahata has joined #openstack-infra00:10
wenlockcarlkb, lifeless, fungi: thanks for clearing that up for me :D00:10
*** marun has joined #openstack-infra00:10
wenlockclarkb: changing the subject... how is it going with reviews for today, i'd like another on https://review.openstack.org/#/c/51425/00:10
wenlocki know it has evil ruby in it :<00:11
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only  https://review.openstack.org/7022300:11
jog0clarkb: what did I miss this time ^00:11
clarkbzaro: approved00:11
*** matsuhashi has joined #openstack-infra00:12
zaroclarkb: thanks!00:13
jog0http://status.openstack.org/zuul/ looks strangly quit -- a lot of queued jobs00:14
*** marun has quit IRC00:15
jog0very few running00:15
clarkbjog0: you should probably offset the two crons00:15
*** alexpilotti has quit IRC00:15
clarkbjog0: either with like a random dely or with a different cron spec00:15
jog0how do I do that?00:15
clarkbjog0: if you grep for RANDOM in the infra/config repo you will see other crons that do that00:16
clarkbjog0: yes that does look dodd00:16
*** david-lyle has quit IRC00:16
*** pcrews has quit IRC00:16
clarkbfungi: ^ is nodepool still happy?00:16
zaroclarkb: puppet agent is running on zuul-dev right?00:17
clarkbzaro: uh maybe, I will double check00:17
clarkbzaro: nope, starting it now00:18
jog0clarkb: hah sleep command => $((RANDOM\%600)) && ...'00:18
*** thuc has quit IRC00:18
fungijog0: queuing tripleo jobs, or other jobs?00:18
clarkbfungi: other jobs00:18
*** thuc has joined #openstack-infra00:19
clarkbreconfigure managers is failing due to tripleo00:19
*** david-lyle has joined #openstack-infra00:20
*** eharney has quit IRC00:20
*** mriedem has joined #openstack-infra00:20
fungianother place where the code assumes providers will always be online?00:20
fungii can delete the tripleo cloud back out of the config and restart nodepool again00:21
clarkbfungi: possibly, I think we should try that00:21
clarkbfungi: I don't see the typical node allocation debug messages00:21
fungii'll stop puppet this time00:21
fungiso it won't be undone00:21
jog0clarkb: how about doing     minute      => '*/16',00:21
jog0so one job runs every 15 min, the othe revery 1600:22
clarkbjog0: that would work as well but maybe it less clear when the job will run00:22
jog0clarkb: or i can do00:23
jog0command => 'sleep $((RANDOM\%60+90)) &&00:23
jog0but don't know what that means00:23
*** thuc has quit IRC00:23
funginodepool.yaml edited to remove tripleo provider/images/nodes and nodepoold restarting now00:23
clarkbthat means sleep for at least 90 seconds with an additional random 0-59 second sleep00:23
jog0oh perfect00:24
*** dcramer_ has joined #openstack-infra00:25
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Generate two elastic-recheck-graph json files all, and gate only  https://review.openstack.org/7022300:25
jog0clarkb: hopefully I did it right this time00:26
*** jaypipes has quit IRC00:27
fungithere seems to be an inverse correlation between the times when i need nodepoold's graceful restart to return quickly, and the times when it actually does00:29
clarkbit knows00:29
*** yamahata has quit IRC00:29
*** mikal_ is now known as mikal00:29
fungiif this takes much longer, i'll just kill it and clean up any mess it leaves behind00:32
*** jaypipes has joined #openstack-infra00:32
*** nosnos has joined #openstack-infra00:33
*** yamahata has joined #openstack-infra00:33
fungii think it's waiting for that image build to complete00:34
lifelessfungi: yeah, it doesn't know how to suspend-restore that00:35
lifelesswe should fix that00:35
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1270212  https://review.openstack.org/7038600:38
*** masayukig has joined #openstack-infra00:39
*** thuc has joined #openstack-infra00:39
*** yamahata has quit IRC00:43
*** kraman has joined #openstack-infra00:43
*** nosnos_ has joined #openstack-infra00:44
*** masayukig has quit IRC00:44
clarkbfungi: how are we doing?00:46
lifelessanteaya: devananda: markmcclain: https://review.openstack.org/#/c/70549/ has landed; neutron in theory is no longer wedged.00:47
*** nosnos has quit IRC00:47
markmcclainlifeless: thanks00:48
devanandalifeless: ack. i have several rechecks going in ironic to confirm if we're unwedged00:48
fungiclarkb: i marked all the building images and nodes for deletion and then killed and restarted nodepoold00:48
*** yamahata has joined #openstack-infra00:48
markmcclainsame here I've issued a recheck on a critical review00:48
clarkbfungi: I see allocation requests now in debug log which should mean we will get nodes00:48
*** oubiwann_ has joined #openstack-infra00:48
fungii'll keep an eye out for unusually old ready/used/delete nodes here in a little while and delete them if they stick around00:49
*** ekarlso has quit IRC00:50
lifelessmarkmcclain: OOI which review?00:51
*** dcramer_ has quit IRC00:51
*** weshay has joined #openstack-infra00:52
clarkbfungi: 273 nodes building00:52
clarkbfungi: did we lose that image build in the restart?00:52
fungiclarkb: yeah, its started again now00:52
lifelessmarkmcclain: fingers crossed00:53
*** nosnos_ has quit IRC00:54
*** nosnos has joined #openstack-infra00:54
*** gokrokve has quit IRC00:55
*** david_lyle_ has joined #openstack-infra00:55
*** CaptTofu has joined #openstack-infra00:56
*** gyee has quit IRC00:59
*** david-lyle has quit IRC00:59
*** mrodden has quit IRC01:00
clarkbfungi: it looks like zuul and nodepool are almost completely caught up01:03
*** dcramer_ has joined #openstack-infra01:03
*** atiwari has quit IRC01:03
*** thuc has quit IRC01:10
*** thuc has joined #openstack-infra01:10
clarkbtripleo being the only the only oustanding nodepool/zuul trouble01:11
*** wenlock has quit IRC01:13
*** thuc has quit IRC01:14
openstackgerritClark Boylan proposed a change to openstack-infra/config: Make requirement update proposals more robust.  https://review.openstack.org/6996901:15
*** mrodden has joined #openstack-infra01:15
markmcclainlifeless: no luck… https://jenkins01.openstack.org/job/check-tempest-dsvm-neutron-pg/910/console01:20
lifelessmarkmcclain: thats what the others were doing?01:22
lifelessmarkmcclain: or is this a different pathology?01:22
markmcclainguestfs failures01:22
clarkblooks the same01:22
lifelessso thats bong01:22
clarkblifeless: perhaps the libvirt driver is buggy?01:23
* clarkb looks at it again01:23
lifeless2014-02-04 00:57:07.087 | 2014-02-04 00:57:07 + iniset /etc/nova/nova.conf libvirt inject_partition -201:23
clarkbya I think the value was set properly01:23
*** amcrn has quit IRC01:24
lifelessI can't see any override of it either01:25
*** mestery has joined #openstack-infra01:25
clarkbbut according to the logs it somehow fell into the block after elif inject_files and CONF.libvirt.inject_partition != -2:01:25
*** mestery has quit IRC01:26
*** sarob has quit IRC01:26
*** mestery has joined #openstack-infra01:26
lifelessif (            if any((key, net, metadata, admin_pass, files)):01:27
lifelessbut thats guarded bu01:27
lifeless        elif inject_files and CONF.libvirt.inject_partition != -2:01:27
lifelessgit grep inject_partition in devstack only has one hit, the setting line01:28
clarkblifeless: is it possible that the file is overridden after the ini set?01:28
*** marun has joined #openstack-infra01:28
clarkblifeless: yup that is the problem :/01:29
clarkbsee the location of create_nova_conf01:29
lifelessjust spotted that01:29
lifelessthis is a broader bug01:31
lifelessthat whole function is toast01:31
clarkbthe cgruop stuff is probably ok01:32
clarkbnow I feel bad saying that hte devstack change would override the nova changes01:33
clarkbI was very wrong about that :001:33
*** marun has quit IRC01:33
lifelesswe thought it would :P01:34
lifelesschmouel: dtroyer: sdague: https://review.openstack.org/7088501:34
lifelesschmouel: dtroyer sdague: russell's change was broken, my change was broken. This fixes the breakage so it should actually take effect.01:35
lifelessOf course, that may cause it to break in other ways :/01:35
*** masayukig has joined #openstack-infra01:40
*** david-lyle has joined #openstack-infra01:40
*** david_lyle_ has quit IRC01:44
*** masayukig has quit IRC01:45
*** ryanpetrello has quit IRC01:46
*** mestery has quit IRC01:47
*** gokrokve has joined #openstack-infra01:47
*** pcrews has joined #openstack-infra01:49
*** sandywalsh_ has quit IRC01:50
*** ken1ohmichi has joined #openstack-infra01:50
*** jecarey has joined #openstack-infra01:50
openstackgerritCyril Roelandt proposed a change to openstack/requirements: HTTPretty: bump to 0.8.0  https://review.openstack.org/7088601:53
*** markmcclain1 has joined #openstack-infra01:54
*** markmcclain has quit IRC01:54
*** markmcclain1 has quit IRC01:56
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters.  https://review.openstack.org/7063601:59
*** nati_uen_ has joined #openstack-infra01:59
*** nati_uen_ has quit IRC01:59
*** gokrokve has quit IRC02:00
*** nati_uen_ has joined #openstack-infra02:00
*** markmcclain has joined #openstack-infra02:01
*** CaptTofu has quit IRC02:02
*** nati_ueno has quit IRC02:02
*** russellb has joined #openstack-infra02:02
*** ryanpetrello has joined #openstack-infra02:02
*** CaptTofu has joined #openstack-infra02:03
*** gokrokve has joined #openstack-infra02:03
*** shivh has left #openstack-infra02:03
*** sarob has joined #openstack-infra02:04
*** sandywalsh_ has joined #openstack-infra02:06
*** masayukig has joined #openstack-infra02:09
portantejog0: you around?02:11
portanteI'd like to chat about that path I should take for those two e-r checks I have in review02:11
jog0portante: yup02:12
jog0portante: pulling 'em up now02:12
jog0so https://review.openstack.org/#/c/69702/2/queries/1270608.yaml02:13
portanteso for the IPv6 one, should I just fix up the duplicate, and then open another bug for the wider Network is unreachable match?02:13
jog0I want to see an example02:13
portanteyes, I'll post the two logstash queries for that02:13
jog0portante: yeah, although I *thought* I opened a bug for the unable to install pip part02:13
jog0but perhaps I imagined it02:13
portantei ran them, then yanked down the export with build_uuid only and uniq'd them to show that the new is a complete superset of the old02:14
*** crank has quit IRC02:14
jog0so we should still do the rename for https://review.openstack.org/#/c/69670/02:14
jog0so if you want to propose a patch to just do that I will +A right now02:14
portantejog0: can you find that bug for me?02:14
portanteokay, I'll do that now, sec02:14
*** crank has joined #openstack-infra02:14
jog0portante: sure let me look02:14
*** oubiwan__ has joined #openstack-infra02:15
portanteI'll propose a new patch for the straight rename, sec02:16
*** oubiwann has quit IRC02:16
*** oubiwann_ has quit IRC02:16
*** oubiwann has joined #openstack-infra02:18
*** melwitt has quit IRC02:18
jog0portante: https://bugs.launchpad.net/openstack-ci/+bug/127071002:18
portantegotit, thanks, jog002:18
openstackgerritPeter Portante proposed a change to openstack-infra/elastic-recheck: Update fingerprint to match opened bug 1266616  https://review.openstack.org/7088702:21
portantethat is the straight rename02:21
portanteSo bug 1270710 does not talk about the Network is unreachable02:23
portantedo you want to keep that out of it and just open another bug to cover the more generic NiU?02:23
* portante workings on the log stash queries for the other one out for review02:24
jog0portante: hmm yeah that was a timeout02:24
lifelessclarkb: so 70885 passed everything (except bash8 which i"m fixing)02:24
lifelessclarkb: but I think all the relevant people are asleep02:24
jog0but my hunch is its the same underlying issue (but I could be very wrong)02:24
*** sarob has quit IRC02:25
openstackgerritA change was merged to openstack-infra/elastic-recheck: Update fingerprint to match opened bug 1266616  https://review.openstack.org/7088702:25
*** sarob has joined #openstack-infra02:25
clarkblifeless: :( ya sdague is in korea and dtroyer probably hit his EOD and jeblair is flying back to the US (I think)02:25
clarkbfungi: you still about?02:25
clarkbchmouel: vishy: either of you guys able to review and possibly approve a devstack change?02:26
fungiclarkb: yep. mass deleting the cruft nodes from before the restart now (any in a used state for more than 3 hours)02:27
*** oubiwan__ has quit IRC02:27
portantejog0: do you want to go with your hunch and update the bug to talk about the Network is unreachable?02:27
portantethose are two different patterns, so it might be worth while keeping them separate until we can confirm the actually share a root cause02:27
*** oubiwann_ has joined #openstack-infra02:28
clarkbfungi: 70885 is something we should probably try to get in, but there is a lack of devstack cores around. I have a feeling we can get away with the current broken state until tomorrow morning though02:28
clarkbconsidering it has been broken for a little bit now02:28
*** david-lyle has quit IRC02:29
*** sarob has quit IRC02:30
jog0portante: keep 'em different for now02:30
lifelessclarkb: what I'd really like is to see the neutron/ironic jobs that fail tested against that change02:30
lifelessclarkb: could we perhaps add them as non-voting-on-devstack but run-on-devstack ?02:31
lifelessclarkb: then we don't need to wait for it to land to know...02:31
fungicheck experimental--i think they're there already02:31
* lifeless waits for browser to come back from sync()02:32
lifelessthis may take a while02:32
lifelessfungi: I don't think so02:33
lifelessfungi: only one jjob02:33
lifelessfungi: neutron-heat-slow02:33
lifelesswhich doesn't say isolated...02:33
lifelessand the definition doesn't have DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION=1 set02:34
lifelessso it won't tell us02:34
fungioh, i must have been thinking about tempest02:34
clarkbI thought ironic already does non voting against devstack?02:34
clarkbdevananda: ^ did we not add it there?02:34
clarkbI am completely on board with adding those jobs to devstack check02:35
lifelessclarkb: ironic is there02:35
portantejog0: regarding the iSCSI one, I have just posted the two log stash queries as inline comments so you can compare them yourself to see what I am talking about02:35
lifelessclarkb: and a devstack-dsvm-neutron02:35
portanteI could be off on this02:35
lifelessI don't know what the config of that is yet02:35
* lifeless pokes02:35
clarkblifeless: what do we need to do then?02:35
*** ryanpetrello has quit IRC02:35
portantejog0: I'll file a separate bug for the generic "Network is unreachable" under openstack-ci and add an e-r query for it02:36
fungiclarkb: still need to test the neutron-isolated job against it02:36
lifelessclarkb: pipeline}-devstack-dsvm-neutron{branch-designator doesn't have tenant isolation on either02:37
lifelessat least, in my copy of /config02:38
clarkbit doesn't the isolated jobs are different02:39
lifelesshow do you make something vote in one place but not another ?02:39
clarkbbut do we need the extra jobs to see if this fixes the problem?02:39
lifelesswell there are two cases we want to fix02:39
lifelessironic and neutron02:39
lifelessit's possible that this will fix one and not the other02:39
lifelessonly one of the cases are represented in jobs on devstack today.02:40
clarkblifeless: you can't directly. Instead we overload branch-designator to create a second job with an identical config and make that job nonvoting. the ironic -nv jobs do this02:40
clarkblifeless: oh are we using config drive on the normal neutron tests?02:40
*** rakhmerov has quit IRC02:40
clarkbI thought the neutron postgres test was using file injection and failing too02:40
lifelessclarkb: I don't know the details, but the normal ones pass02:40
*** rakhmerov has joined #openstack-infra02:40
*** rakhmerov has quit IRC02:40
lifelessthe specific interaction cared about only shows up in -isolated- jobs02:40
*** masayuki_ has joined #openstack-infra02:40
lifelessthe devstack-dsvm job did fail, looking for a cause02:41
lifelessfailed on floating_ips02:41
lifelesswhich is the thing I was asking anteaya about the other day02:41
lifeless(I still haven't filed a bug for that, but I believe its disabled in neutron gate tests as known broke)02:42
*** oubiwann_ has quit IRC02:43
*** oubiwann_ has joined #openstack-infra02:44
*** masayuki_ has quit IRC02:45
openstackgerritlifeless proposed a change to openstack-infra/config: Add Neutron isolated tenants to ds/experimental  https://review.openstack.org/7088902:46
lifelessclarkb: something like that perhaps02:46
lifelessclarkb: season to taste02:46
clarkbthat works02:46
lifelessif we can get that live on zuul a check experimental should tell us what we need02:47
lifelessassuming I understood everything :P02:48
*** nati_ueno has joined #openstack-infra02:48
clarkbI approved it02:48
*** nati_ueno has quit IRC02:49
openstackgerritA change was merged to openstack-infra/config: Add Neutron isolated tenants to ds/experimental  https://review.openstack.org/7088902:49
*** nati_ueno has joined #openstack-infra02:50
markmcclainlifeless: you beat me too it, I was about to suggest we add the isolated job02:50
* markmcclain is low on battery will check in after landing02:50
*** nati_uen_ has quit IRC02:51
lifelessclarkb: let me know when to do check experimental02:53
openstackgerritPeter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1275994  https://review.openstack.org/7089202:54
clarkblifeless: 10 after the hour is safest, there is work to add the last layout reload time to the zuul status page which will make this easier in the future02:55
portantejog0: above is the new e-r tracking the new bug, and I put references between the bugs so we can keep them associated02:55
jog0portante: thanks02:55
jog0I'll review in the AM02:55
portantek, 'night02:55
*** markmcclain has quit IRC02:56
*** markwash has quit IRC02:59
*** oubiwann_ has quit IRC03:00
*** dkehn_ has joined #openstack-infra03:03
*** nati_ueno has quit IRC03:11
*** russellb has quit IRC03:12
*** boris-42_ has joined #openstack-infra03:12
*** nati_ueno has joined #openstack-infra03:12
*** nati_ueno has quit IRC03:13
*** nati_ueno has joined #openstack-infra03:13
openstackgerritThomas Goirand proposed a change to openstack/requirements: Allow webob 1.3  https://review.openstack.org/7074103:14
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client  https://review.openstack.org/7089703:19
lifelessdevananda: your ironic run still failed, but not due to guestfs anymore03:19
lifelessdevananda: with 7088503:20
*** krotscheck has quit IRC03:21
jog0lifeless: http://status.openstack.org/zuul/ tripleo jobs are backed up03:24
lifelessjog0: I know03:25
lifelessjog0: see waaay back backscroll03:25
lifelessjog0: look for muppet.03:25
jog0lifeless: tripleo weding nodepool and whatnot03:25
*** gokrokve has quit IRC03:26
*** sarob has joined #openstack-infra03:26
* lifeless goes to pick up C03:26
*** gokrokve has joined #openstack-infra03:26
*** gokrokve has quit IRC03:30
*** malini is now known as malini_afk03:32
*** gokrokve has joined #openstack-infra03:32
*** sarob has quit IRC03:32
*** emagana has quit IRC03:39
*** gokrokve has quit IRC03:40
*** jecarey has quit IRC03:41
*** masayuki_ has joined #openstack-infra03:41
*** masayuki_ has quit IRC03:46
*** david-lyle has joined #openstack-infra03:57
*** dcramer_ has quit IRC04:01
*** sandywalsh_ has quit IRC04:02
*** gokrokve has joined #openstack-infra04:11
*** gokrokve_ has joined #openstack-infra04:13
*** harlowja is now known as harlowja_away04:13
*** harlowja_away is now known as harlowja04:15
*** koolhead17 has quit IRC04:16
*** gokrokve has quit IRC04:16
*** sandywalsh_ has joined #openstack-infra04:17
*** gokrokve_ has quit IRC04:19
*** sarob has joined #openstack-infra04:21
*** pcrews has quit IRC04:23
*** jgriffith has quit IRC04:24
*** mriedem has quit IRC04:24
*** AlexF has joined #openstack-infra04:26
*** jgriffith has joined #openstack-infra04:29
*** koolhead17 has joined #openstack-infra04:31
*** CaptTofu has quit IRC04:32
*** gokrokve has joined #openstack-infra04:32
*** AlexF has quit IRC04:37
*** ryanpetrello has joined #openstack-infra04:39
*** ryanpetrello has quit IRC04:40
*** masayuki_ has joined #openstack-infra04:42
*** nati_ueno has quit IRC04:44
StevenKThat's one gate reset too many. Can someone promote 70891,2 to unblock the nova gate?04:46
*** masayuki_ has quit IRC04:47
*** emagana has joined #openstack-infra04:49
*** emagana has quit IRC04:55
*** thuc has joined #openstack-infra04:58
*** ryanpetrello has joined #openstack-infra05:00
*** amcrn has joined #openstack-infra05:02
lifelessclarkb: fungi: ^05:03
*** sarob has quit IRC05:04
*** DinaBelova_ is now known as DinaBelova05:04
fungiStevenK: lifeless: done05:04
*** sarob has joined #openstack-infra05:04
StevenKfungi: Thanks!05:05
*** sarob_ has joined #openstack-infra05:07
*** sarob has quit IRC05:09
*** dims has quit IRC05:11
*** sarob has joined #openstack-infra05:13
*** dims has joined #openstack-infra05:13
*** sarob_ has quit IRC05:16
*** thuc has quit IRC05:19
*** slong has joined #openstack-infra05:19
*** thuc has joined #openstack-infra05:19
*** slong_ has quit IRC05:20
*** coolsvap has joined #openstack-infra05:21
*** dkliban has quit IRC05:22
*** amotoki has joined #openstack-infra05:22
*** thuc has quit IRC05:23
*** gokrokve has quit IRC05:24
*** Ryan_Lane has joined #openstack-infra05:24
*** rakhmerov has joined #openstack-infra05:25
*** rakhmerov has joined #openstack-infra05:25
*** markmcclain has joined #openstack-infra05:28
*** sarob has quit IRC05:32
*** sarob has joined #openstack-infra05:32
*** gokrokve has joined #openstack-infra05:33
zaroclarkb: so i see you already tried recheck on review-dev.  seems like it's not working, do you happen to know whether the project-dev.yaml got updated?05:34
clarkbzaro: I havent tried since you patched the layout file05:35
zaroclarkb: ohh yeah, didn't notice your rechecks were from a few days ago.  i just tried but zuul didn't trigger another job run.05:36
clarkbzaro: Oh I bet we need to make the regex multiline05:37
clarkbbecause of the newlines05:37
*** sarob has quit IRC05:37
*** gokrokve has quit IRC05:38
zaroclarkb: actually it didn't get merged.  crap, i made that change dependent on https://review.openstack.org/#/c/6034805:38
*** markwash has joined #openstack-infra05:40
zarothat was a mistake making it dependent on 60348.  what would you suggest i do?05:40
clarkbremove the dependdncy05:41
*** rwsu has quit IRC05:41
zaroyou mean push new patch without the dependency?05:41
* zaro tries that05:42
*** masayuki_ has joined #openstack-infra05:43
openstackgerritKhai Do proposed a change to openstack-infra/config: add pipelines and update comment filter for gerrit 2.8  https://review.openstack.org/7086505:47
*** masayuki_ has quit IRC05:47
zaroclarkb: ^05:48
*** slong has quit IRC05:50
*** AlexF has joined #openstack-infra05:54
*** AlexF has quit IRC05:59
*** AlexF has joined #openstack-infra05:59
*** nosnos_ has joined #openstack-infra06:00
*** ryanpetrello has quit IRC06:00
*** markwash has quit IRC06:02
*** nosnos has quit IRC06:03
*** sarob has joined #openstack-infra06:03
*** jcoufal has joined #openstack-infra06:04
*** AlexF has quit IRC06:04
*** VijayT has quit IRC06:07
openstackgerritBill Maxwell proposed a change to openstack-infra/jenkins-job-builder: added logstash plugin support and fixed test typo  https://review.openstack.org/7091706:07
*** AlexF has joined #openstack-infra06:08
*** hdd has quit IRC06:10
*** pcrews has joined #openstack-infra06:16
zaroclarkb: it was a trivial rebase, no approval this time?06:20
*** e0ne has joined #openstack-infra06:23
*** AlexF has quit IRC06:24
clarkbzaro trivial rebase does not preserve approvals. I can push it through in the morning06:24
*** jcooley_ has joined #openstack-infra06:26
*** e0ne has quit IRC06:27
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs  https://review.openstack.org/7059406:29
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs  https://review.openstack.org/7059406:30
*** Ryan_Lane1 has joined #openstack-infra06:30
*** sarob has quit IRC06:31
*** resker has joined #openstack-infra06:31
*** pcrews has quit IRC06:32
*** Shrews_ has joined #openstack-infra06:32
*** coolsvap_away has joined #openstack-infra06:32
*** matsuhashi has quit IRC06:33
*** gokrokve has joined #openstack-infra06:33
*** matsuhashi has joined #openstack-infra06:33
*** branen_ has joined #openstack-infra06:33
*** SpamapS_ has joined #openstack-infra06:33
*** soren has joined #openstack-infra06:34
*** soren has quit IRC06:34
*** soren has joined #openstack-infra06:34
*** masayuki_ has joined #openstack-infra06:35
*** flaper87l has joined #openstack-infra06:35
*** flaper87l is now known as flaper8706:35
lifelessfungi: / clarkb: are you going to reenable tripleo cloudbefore you sleep ?06:35
*** jd__` has joined #openstack-infra06:37
*** gokrokve has quit IRC06:38
*** Ryan_Lane has quit IRC06:38
*** coolsvap has quit IRC06:38
*** dims has quit IRC06:38
*** oubiwann has quit IRC06:38
*** masayukig has quit IRC06:38
*** mattoliverau has quit IRC06:38
*** jhesketh__ has quit IRC06:38
*** primeministerp has quit IRC06:38
*** SpamapS has quit IRC06:38
*** soren_ has quit IRC06:38
*** w_ has quit IRC06:38
*** Shrews has quit IRC06:38
*** bodepd has quit IRC06:38
*** esker has quit IRC06:38
*** branen has quit IRC06:38
*** zul has quit IRC06:38
*** med_ has quit IRC06:38
*** katyafervent has quit IRC06:38
*** flaper87|afk has quit IRC06:38
*** jd__ has quit IRC06:38
*** krtaylor has quit IRC06:38
*** mordred has quit IRC06:38
*** matrohon has quit IRC06:38
*** jd__` is now known as jd__06:38
*** primeministerp has joined #openstack-infra06:40
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs  https://review.openstack.org/7059406:42
*** mrda is now known as mrda_away06:43
*** masayukig has joined #openstack-infra06:44
*** harlowja is now known as harlowja_away06:44
*** katyafervent has joined #openstack-infra06:44
*** denis_makogon_ has joined #openstack-infra06:45
*** mattoliverau has joined #openstack-infra06:45
*** zul has joined #openstack-infra06:46
juiceif anyone has time to review this patch that will help trove gate get up and running, we'd appreciate it https://review.openstack.org/#/c/70905/06:47
*** masayukig has quit IRC06:48
*** skraynev_ is now known as skraynev06:50
*** bhuvan has joined #openstack-infra06:51
*** mordred has joined #openstack-infra06:54
*** oubiwann has joined #openstack-infra06:54
*** jhesketh_ has joined #openstack-infra06:55
*** bodepd has joined #openstack-infra06:56
*** VijayT has joined #openstack-infra06:57
*** krtaylor has joined #openstack-infra06:59
*** olaph has joined #openstack-infra07:01
*** bhuvan has quit IRC07:02
*** coolsvap_away has quit IRC07:04
*** bhuvan has joined #openstack-infra07:04
*** jkt_ is now known as jkt07:05
*** dims has joined #openstack-infra07:08
*** odyssey4me has joined #openstack-infra07:08
*** markmcclain has quit IRC07:10
*** VijayT has quit IRC07:12
*** jcooley_ has quit IRC07:12
*** saju_m has joined #openstack-infra07:12
*** bhuvan has quit IRC07:15
*** lcheng_ has joined #openstack-infra07:16
openstackgerritLukas Bednar proposed a change to openstack-infra/jenkins-job-builder: scms: MultiSCM is generated for scms hidden in macro  https://review.openstack.org/6982707:18
*** slong has joined #openstack-infra07:19
*** flaper87 has quit IRC07:21
*** flaper87 has joined #openstack-infra07:21
*** yamahata has quit IRC07:23
*** coolsvap has joined #openstack-infra07:26
*** dstufft is now known as dstufft-gone07:26
*** talluri has joined #openstack-infra07:27
*** yolanda has joined #openstack-infra07:28
*** AlexF has joined #openstack-infra07:32
*** gokrokve has joined #openstack-infra07:33
*** gokrokve has quit IRC07:38
*** AlexF has quit IRC07:41
*** rwsu has joined #openstack-infra07:42
*** sarob has joined #openstack-infra07:43
*** masayukig has joined #openstack-infra07:44
*** lcheng_ has quit IRC07:47
*** amcrn has quit IRC07:47
*** sarob has quit IRC07:48
*** jcooley_ has joined #openstack-infra07:48
*** pblaho has joined #openstack-infra07:48
*** masayukig has quit IRC07:48
*** slong has quit IRC07:49
*** rwsu has quit IRC07:51
*** rwsu has joined #openstack-infra07:53
*** jcooley_ has quit IRC07:56
*** e0ne has joined #openstack-infra07:59
*** afazekas has joined #openstack-infra08:01
*** kruskakli has joined #openstack-infra08:04
*** ken1ohmichi has left #openstack-infra08:05
*** afazekas has quit IRC08:07
*** denis_makogon_ has quit IRC08:09
*** sarob has joined #openstack-infra08:13
*** che-arne has quit IRC08:14
openstackgerritSteven Hardy proposed a change to openstack/requirements: Update python-keystoneclient to 0.5.0  https://review.openstack.org/7092908:15
*** sarob has quit IRC08:18
*** afazekas has joined #openstack-infra08:23
*** fbo is now known as fbo_away08:32
*** gokrokve has joined #openstack-infra08:33
*** pblaho has quit IRC08:34
*** pblaho has joined #openstack-infra08:35
*** coolsvap has quit IRC08:36
*** gokrokve has quit IRC08:37
*** ogelbukh1 is now known as ogelbukh08:42
*** yamahata has joined #openstack-infra08:43
*** masayukig has joined #openstack-infra08:45
*** hashar has joined #openstack-infra08:46
*** masayukig has quit IRC08:50
*** ilyashakhat has joined #openstack-infra08:55
*** jpich has joined #openstack-infra08:56
*** slong has joined #openstack-infra09:02
*** jhesketh_ has quit IRC09:02
*** jhesketh has quit IRC09:02
*** bhuvan has joined #openstack-infra09:03
chmouelclarkb: sure, is it still needed?09:04
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add replication key for gerrit from hiera  https://review.openstack.org/7093309:05
chmouelclarkb: ok i have approved 7088509:06
*** markmc has joined #openstack-infra09:13
*** sarob has joined #openstack-infra09:13
*** jhesketh has joined #openstack-infra09:14
*** jhesketh_ has joined #openstack-infra09:15
openstackgerritFei Long Wang proposed a change to openstack/requirements: Depending on python-swiftclient>=1.6  https://review.openstack.org/7093509:16
*** sarob has quit IRC09:18
*** masayuki_ has quit IRC09:20
*** fbo_away is now known as fbo09:22
*** coolsvap has joined #openstack-infra09:23
*** johnthetubaguy has joined #openstack-infra09:24
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to test_remote  https://review.openstack.org/7093609:27
*** jooools has joined #openstack-infra09:28
*** johnthetubaguy has quit IRC09:31
*** johnthetubaguy has joined #openstack-infra09:32
*** gokrokve has joined #openstack-infra09:33
*** Ryan_Lane1 has quit IRC09:36
*** gokrokve has quit IRC09:38
*** e0ne has quit IRC09:46
*** masayukig has joined #openstack-infra09:46
*** e0ne has joined #openstack-infra09:46
*** katyafervent is now known as katyafervent_awa09:49
*** reed has joined #openstack-infra09:50
*** e0ne_ has joined #openstack-infra09:50
*** e0ne has quit IRC09:50
*** masayukig has quit IRC09:51
*** matrohon has joined #openstack-infra09:51
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot.  https://review.openstack.org/7067209:52
*** BobBallAway is now known as BobBall09:52
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot.  https://review.openstack.org/7067209:59
*** e0ne_ has quit IRC10:02
*** e0ne has joined #openstack-infra10:02
*** e0ne has quit IRC10:06
*** e0ne has joined #openstack-infra10:07
*** VijayT has joined #openstack-infra10:09
*** e0ne has quit IRC10:10
*** e0ne has joined #openstack-infra10:10
*** sarob has joined #openstack-infra10:13
*** VijayT has quit IRC10:14
*** jcooley_ has joined #openstack-infra10:17
*** sarob has quit IRC10:18
*** Adri2000_ is now known as Adri200010:21
*** lcestari has joined #openstack-infra10:22
*** metabro has quit IRC10:22
*** jcooley_ has quit IRC10:23
*** coolsvap has quit IRC10:30
*** gokrokve has joined #openstack-infra10:33
openstackgerritAlexandre Levine proposed a change to openstack-infra/config: Removing python26 jobs for gce-api project  https://review.openstack.org/7094410:35
mordredjust so that everyone knows ...10:35
mordredI'd really like it if I could configure my phone to turn on usb tethering every time it's plugged in somewhere10:36
*** gokrokve has quit IRC10:37
bogdandofungi, hi. sorry for delayed reply. >>> <fungi> bogdando: if you have a particular change/bug combination which you believe meets those criteria, i can take a look and try to determine what happened10:43
*** boris-42_ has quit IRC10:43
bogdandofungi, please take a look onto https://bugs.launchpad.net/fuel/+bug/1271176 or later bugs from Fuel project10:43
bogdandofungi, I've had to change status manually after status tracking were lost10:44
*** slong has quit IRC10:44
*** dizquierdo has joined #openstack-infra10:46
*** masayukig has joined #openstack-infra10:46
*** cody-somerville has joined #openstack-infra10:46
*** e0ne has quit IRC10:50
*** e0ne has joined #openstack-infra10:50
*** masayukig has quit IRC10:51
*** che-arne has joined #openstack-infra10:52
*** hashar has quit IRC10:58
*** russellb has joined #openstack-infra11:00
*** DinaBelova is now known as DinaBelova_11:01
*** CaptTofu has joined #openstack-infra11:02
*** DinaBelova_ is now known as DinaBelova11:02
*** CaptTofu has quit IRC11:07
*** DinaBelova is now known as DinaBelova_11:10
*** e0ne has quit IRC11:10
*** jcooley_ has joined #openstack-infra11:13
*** sarob has joined #openstack-infra11:13
*** andre__ has quit IRC11:14
*** sarob_ has joined #openstack-infra11:16
*** sarob has quit IRC11:18
*** jcooley_ has quit IRC11:19
*** sarob_ has quit IRC11:20
*** jcoufal has quit IRC11:26
*** andre__ has joined #openstack-infra11:29
*** david-lyle has quit IRC11:29
*** nosnos_ has quit IRC11:30
*** CaptTofu has joined #openstack-infra11:32
*** gokrokve has joined #openstack-infra11:33
openstackgerritJulien Danjou proposed a change to openstack/requirements: pbr: update to 0.6  https://review.openstack.org/7095511:35
*** gokrokve has quit IRC11:38
*** amotoki has quit IRC11:38
*** amotoki has joined #openstack-infra11:38
*** masayukig has joined #openstack-infra11:47
*** rfolco has joined #openstack-infra11:48
*** masayukig has quit IRC11:52
*** dizquierdo is now known as dizquierdo_afk11:58
*** miqui has quit IRC12:04
*** yamahata has quit IRC12:04
*** malini_afk is now known as malini12:06
*** yamahata has joined #openstack-infra12:09
*** sarob has joined #openstack-infra12:13
*** sandywalsh_ has quit IRC12:15
*** esker has joined #openstack-infra12:17
*** e0ne has joined #openstack-infra12:17
*** sarob has quit IRC12:18
*** resker has quit IRC12:20
*** yassine has joined #openstack-infra12:20
*** exz has joined #openstack-infra12:22
*** CaptTofu has quit IRC12:22
*** e0ne_ has joined #openstack-infra12:24
*** e0ne has quit IRC12:24
*** bhuvan has quit IRC12:28
*** sandywalsh_ has joined #openstack-infra12:28
*** e0ne has joined #openstack-infra12:29
openstackgerritA change was merged to openstack/requirements: Allow webob 1.3  https://review.openstack.org/7074112:29
*** e0ne__ has joined #openstack-infra12:30
*** e0ne___ has joined #openstack-infra12:31
*** e0ne__ has quit IRC12:31
*** boris-42_ has joined #openstack-infra12:32
*** e0ne_ has quit IRC12:32
*** e0ne___ has quit IRC12:32
*** masayukig has joined #openstack-infra12:32
*** e0ne_ has joined #openstack-infra12:32
*** gokrokve has joined #openstack-infra12:33
*** vogxn has joined #openstack-infra12:34
*** e0ne has quit IRC12:34
*** matsuhashi has quit IRC12:36
*** masayukig has quit IRC12:36
*** matsuhashi has joined #openstack-infra12:36
*** yamahata has quit IRC12:37
*** gokrokve has quit IRC12:37
*** DinaBelova_ is now known as DinaBelova12:38
*** matsuhashi has quit IRC12:41
*** bhuvan has joined #openstack-infra12:46
*** smarcet has joined #openstack-infra12:46
*** hashar has joined #openstack-infra12:49
*** yamahata has joined #openstack-infra12:51
*** DinaBelova is now known as DinaBelova_12:52
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add xtrigger.  https://review.openstack.org/7097112:53
*** DinaBelova_ is now known as DinaBelova12:55
*** DinaBelova is now known as DinaBelova_12:55
*** weshay has quit IRC12:56
openstackgerritsebastian marcet proposed a change to openstack-infra/config: Implements:  blueprint openid-oauth2-infra-implementation-puppet-script [smarcet] - #5478 - Clean up puppet Scripts  https://review.openstack.org/6963612:56
*** markmcclain has joined #openstack-infra12:59
*** DinaBelova_ is now known as DinaBelova13:01
*** yamahata has quit IRC13:03
*** e0ne has joined #openstack-infra13:05
*** russellb_ has joined #openstack-infra13:06
*** russellb_ has quit IRC13:07
*** yamahata has joined #openstack-infra13:07
*** jhesketh_ has quit IRC13:07
*** jhesketh has quit IRC13:08
openstackgerritsebastian marcet proposed a change to openstack-infra/config: Implements: blueprint openid-oauth2-infra-implementation-documentation  https://review.openstack.org/6962013:08
*** AlexF has joined #openstack-infra13:08
*** e0ne_ has quit IRC13:08
*** DinaBelova is now known as DinaBelova_13:10
*** e0ne_ has joined #openstack-infra13:13
*** sarob has joined #openstack-infra13:13
*** bhuvan has quit IRC13:14
*** e0ne has quit IRC13:16
*** sarob has quit IRC13:17
*** alexpilotti has joined #openstack-infra13:18
*** bhuvan has joined #openstack-infra13:18
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Merger: log non Git exceptions on merge  https://review.openstack.org/7097713:20
*** jcoufal has joined #openstack-infra13:21
*** CaptTofu has joined #openstack-infra13:22
*** yamahata has quit IRC13:26
*** rainya has joined #openstack-infra13:27
*** annegentle has joined #openstack-infra13:27
*** pvo has joined #openstack-infra13:27
*** jbryce has joined #openstack-infra13:27
*** lillie has joined #openstack-infra13:27
*** smarcet has quit IRC13:28
*** lillie is now known as Guest4680613:28
*** talluri has quit IRC13:29
*** talluri has joined #openstack-infra13:30
*** weshay has joined #openstack-infra13:32
*** yamahata has joined #openstack-infra13:32
*** masayukig has joined #openstack-infra13:33
*** gokrokve has joined #openstack-infra13:33
*** talluri has quit IRC13:34
*** miqui has joined #openstack-infra13:36
*** masayukig has quit IRC13:37
*** gokrokve has quit IRC13:38
*** saju_m has quit IRC13:41
*** smarcet has joined #openstack-infra13:42
*** ekarlso- has quit IRC13:45
*** ekarlso has joined #openstack-infra13:45
*** yamahata has quit IRC13:46
*** ryanpetrello has joined #openstack-infra13:46
chmouelmordred: should be possible with tasker ? (if a plugin detect what usb host it connected to or via gps location)13:46
*** yamahata has joined #openstack-infra13:50
*** leifmadsen has quit IRC13:50
*** leifmadsen has joined #openstack-infra13:50
openstackgerritVictor Stinner proposed a change to openstack/requirements: Add a new dependency: trollius  https://review.openstack.org/7098313:53
*** yamahata has quit IRC13:55
*** prad_ has joined #openstack-infra13:55
*** rossella_s has joined #openstack-infra13:56
*** thomasem has joined #openstack-infra13:56
*** yamahata has joined #openstack-infra13:59
*** dims has quit IRC14:03
*** dkranz has quit IRC14:05
*** dprince has joined #openstack-infra14:05
*** beagles is now known as beagles_brb14:06
*** Shrews_ is now known as Shrews14:06
*** mattoliverau has quit IRC14:09
*** hashar has quit IRC14:10
*** hashar has joined #openstack-infra14:10
*** sarob has joined #openstack-infra14:13
*** malini is now known as malini_afk14:13
*** mriedem has joined #openstack-infra14:14
*** markmcclain has quit IRC14:15
*** vogxn has quit IRC14:15
*** vogxn has joined #openstack-infra14:16
*** sarob has quit IRC14:18
*** yamahata has quit IRC14:18
*** yamahata has joined #openstack-infra14:19
fungilifeless: is the tripleo cloud back online? if so i'll start puppet back up on the nodepool server14:19
openstackgerritafazekas proposed a change to openstack-infra/elastic-recheck: Adding query for bug 1275500  https://review.openstack.org/7075614:20
*** hashar_ has joined #openstack-infra14:20
*** thuc has joined #openstack-infra14:20
*** hashar has quit IRC14:21
*** pblaho has quit IRC14:22
*** hashar_ is now known as hashar14:23
*** changbl has quit IRC14:23
*** prad_ has quit IRC14:24
*** andreaf has joined #openstack-infra14:29
*** vogxn has quit IRC14:29
*** oubiwann_ has joined #openstack-infra14:32
fungibogdando: looks like we have a bug with the update script, probably resulting from recent changes to the library we use to interpret our projects lists14:32
fungimordred: http://paste.openstack.org/show/62443 (any idea what we missed?)14:33
fungii'll go ahead and open a bug14:33
*** gokrokve has joined #openstack-infra14:33
*** kraman has quit IRC14:33
*** masayukig has joined #openstack-infra14:33
*** amotoki has quit IRC14:35
fungifiled bug 1276180 for this14:36
*** masayukig has quit IRC14:38
*** gokrokve has quit IRC14:38
*** thuc has quit IRC14:39
*** thuc has joined #openstack-infra14:40
*** medberry has joined #openstack-infra14:40
*** medberry has quit IRC14:40
*** medberry has joined #openstack-infra14:40
*** coolsvap has joined #openstack-infra14:42
*** thuc has quit IRC14:44
*** wchrisj_ has quit IRC14:48
*** malini_afk is now known as malini14:48
*** jcooley_ has joined #openstack-infra14:51
*** jecarey has joined #openstack-infra14:53
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Toplevel collection typecheck.  https://review.openstack.org/7099714:54
*** DinaBelova_ is now known as DinaBelova14:54
*** boris-42_ has quit IRC14:54
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add xtrigger.  https://review.openstack.org/7097114:54
*** jcooley_ has quit IRC14:57
bogdandofungi, thank you for the fast update14:57
fungibogdando: now let's see whether i fix it before someone else beats me to it ;)14:58
*** mfer has joined #openstack-infra15:00
*** dkliban has joined #openstack-infra15:02
*** jasondotstar has joined #openstack-infra15:04
*** matrohon has quit IRC15:05
*** dcramer_ has joined #openstack-infra15:08
jeblairfungi: good morning!15:10
fungiit's a jeblair! [and the crowd roars]15:11
*** krotscheck has joined #openstack-infra15:12
jeblairfungi: let me know if there's anything urgent that needs my attention15:12
mordredchmouel: thanks - I will investigate taht15:13
chmouelmordred: i think lifehacker had a long series about that app and how to automate everything on an android phone with it15:13
*** sarob has joined #openstack-infra15:13
mordredjeblair: you did not contract my illness did you?15:14
mordredjeblair: gothicmindfood and krotscheck each got a little I think15:14
mordredchmouel: I think I remember a friend of mine trying to tell me it was awesome at some point15:14
krotscheck.... where "a little" means a fever of 102 for going on 2 days.15:15
fungijeblair: nothing super urgent. we have puppet disabled on nodepool again because another extended tripleo provider outage exposed additional bugs we haven't sorted out yet (in short, restarting nodepool with one configured provider completely offline was causing reconfigure managers to fail. i'm waiting for confirmation that the provider is operational again before firing puppet back up15:15
jeblairmordred, krotscheck: oh no!  i'm sorry to hear that!15:15
krotscheckjeblair, mordred: Did y'all get a chance to lok at the UI mocks for the MVP yet?15:16
mordredkrotscheck: was just about to - and yes, it sounds like you got what I got.15:16
*** dkranz has joined #openstack-infra15:16
mordredkrotscheck: it is not pleasant is it?15:16
jeblairkrotscheck: not yet; i'm just rejoining the land of the connected15:17
krotscheckHere's the link: http://docs-draft.openstack.org/97/70897/1/check/gate-storyboard-webclient-js-unittests/32bcc1d/dist/15:17
krotscheckjcoufal ^15:17
*** AlexF has quit IRC15:17
*** sarob has quit IRC15:18
krotscheckmordred: It's REALLY not pleasant. The fever seems to be dropping though.15:18
jeblairfungi: ack thanks.  yeah, it's probably time for nodepool to grow up a bit.15:18
mordredkrotscheck: I'm assuming you also have the chest, nose and aching?15:18
*** hashar has quit IRC15:19
krotscheckmordred: No, mine's all sinus, all the time.15:19
jeblairmordred: i believe you have described "the flu".15:19
*** hashar has joined #openstack-infra15:19
jcoufalkrotscheck: nice job, I'll have a look on that (maybe not today but tomorrow morning)15:19
jcoufalkrotscheck: take a rest...15:19
mordredjeblair: well, less GI than I'd normally expect with fly15:20
jeblairkrotscheck: sinus+fever is usually suspected to be bacterial so you can probably get antibiotics if you see a doc15:21
*** jnoller has joined #openstack-infra15:22
krotscheckjeblair: I think my snot's the wrong color for that....15:22
* krotscheck continues down the TMI train15:22
*** dizquierdo_afk is now known as dizquierdo15:22
krotscheckjcoufal: When you do take a look, check the mobile sizing as well.15:23
dansmithI have a weird PEP8 failure complaining that I didn't update the nova.conf.sample, but I didn't touch anything that would warrant it:15:23
*** eharney has joined #openstack-infra15:23
fungidansmith: last time this cropped up, it was from an oslo sync which merged ahead of the failing changes15:24
jcoufalkrotscheck: absolutely15:24
dansmithfungi: how could that happen?15:24
jeblairkrotscheck: color isn't a great indicator; usually anything non-clear is indicative.  (/me just had viral(probably) sinusitis and so is rather familiar with this from the negative side)15:25
krotscheckSo it's YOUR fault!15:25
fungidansmith: i'm not quite sure what methods the api sample comparison routine checks, so i don't really know... that was just the explanation i saw in the post-mortem15:26
fungidansmith: have you tried regenerating the samples to see what the diff ends up being?15:26
jeblairkrotscheck: yes, my viral sinusitis from 2 weeks ago caused your bacterial sinusitis.  :)  i'm just a walking bio-lab.15:26
dansmithfungi: okay, this is unrelated to api samples, this is the nova.conf sample15:26
dansmithfungi: and I don't see how something could have merged that introduced a change and didn't update the sample, as this should have caught them15:26
markmcdansmith, I'd say that's coming from keystone15:27
jeblairkrotscheck: blame mordred's flu too15:27
fungiahh... could changes to oslo.config affect that?15:27
dhellmanndansmith: see https://bugs.launchpad.net/nova/+bug/126861415:27
jeblairi'm going to let the people talking about work talk about work now.  :)15:27
dansmithdhellmann: ah, thanks15:27
markmcright, when we extract options from a library15:27
dhellmanndansmith: that example was a change in keystoneclient15:27
markmcand the library adds new options15:27
markmcnova is now taking options from both keystoneclient and oslo.messaging15:28
markmcadding options to either will break nova15:28
dansmithyep, I got it now15:28
dhellmannand that list is going to grow as we graduate more libraries15:28
dhellmannso we need to rethink this particular test15:28
dansmithsee, this is why I didn't even check the rechecks list.. in my early-morning pre-coffee stuper, I thought "for sure I have a unique bug here!"15:28
mordredwe MAY want to use zuuls ability to do coordinated gating15:29
markmccheck should probably check that this particular commit changes the sample file15:29
markmcas opposed to checking whether the file is out of date15:29
fungimaybe the sample config generation wants to become a patch proposal job?15:29
*** gokrokve has joined #openstack-infra15:29
mordredso that for specific things where there is potential for multi-project gating, we do them ina  combined context15:29
dhellmannfungi: that was proposed, as was building the conf file as part of packaging15:29
dhellmannit feels weird to be checking in a file we generate15:30
*** bhuvan has quit IRC15:30
markmcmordred, so, if you add a config option to oslo.messaging you need to simultaneously submit patches to all projects which use it?15:30
*** gokrokve_ has joined #openstack-infra15:30
markmcdhellmann, yeah, it does15:30
mordredmarkmc: ah - no, nevermind15:30
mordredmarkmc: I shall now shut up15:30
markmcmordred, heh, np :)15:30
dhellmannhow is this file used? is it just for docs, or do real people use it as the basis for deployments?15:31
*** bhuvan has joined #openstack-infra15:31
markmcdhellmann, for RDO, AFAIK we make it the default config file installed15:32
markmcdhellmann, distro defaults (like "use qpid", or "keep state in /var/lib") goes in /usr/share/nova/nova-dist.conf15:32
dhellmannI don't know enough about distro packaging -- could we just provide a script to generate the file and not ship the file itself?15:33
dhellmanni.e., "install all of these packages and run foo.sh"15:33
*** pcrews has joined #openstack-infra15:33
*** gokrokve has quit IRC15:33
markmcyeah, we can do that15:33
*** hashar has quit IRC15:33
markmcso long as we keep a lid on the dependencies required to generate the sample file15:34
*** david-lyle has joined #openstack-infra15:34
*** kraman has joined #openstack-infra15:34
markmce.g. allowing the config file to be generated without redis installed15:34
dhellmannok, there's a mailing list thread on this ("pep8 gating fails due to tools/config/check_uptodate.sh") so maybe we should discuss it there15:34
dhellmannyeah, that's already supposed to be a requirement for plugins15:34
*** masayukig has joined #openstack-infra15:34
dhellmannand we can move the generate script into oslo.config so apps only need to provide a wrapper to pass all the right args15:34
*** jgrimm_ has joined #openstack-infra15:35
*** wenlock has joined #openstack-infra15:36
openstackgerritMauro Rodrigues proposed a change to openstack-infra/devstack-gate: Remove exercises tests from grenade runs  https://review.openstack.org/7101015:39
*** masayukig has quit IRC15:39
dhellmannmarkmc: ack15:40
*** hashar has joined #openstack-infra15:45
*** jcooley_ has joined #openstack-infra15:46
*** krotscheck has quit IRC15:50
*** krotscheck has joined #openstack-infra15:50
*** DinaBelova is now known as DinaBelova_15:51
*** beagles_brb is now known as beagles15:51
*** jcooley_ has quit IRC15:51
*** amotoki has joined #openstack-infra15:53
*** DinaBelova_ is now known as DinaBelova15:53
*** oubiwann_ has quit IRC15:54
openstackgerritA change was merged to openstack/requirements: HTTPretty: bump to 0.8.0  https://review.openstack.org/7088615:55
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot.  https://review.openstack.org/7067215:57
*** CaptTofu has quit IRC15:57
*** oubiwann_ has joined #openstack-infra15:58
*** atiwari has joined #openstack-infra16:01
*** CaptTofu has joined #openstack-infra16:02
*** mrodden has quit IRC16:03
fungijeblair: speaking of nodepool fun, that ^ patch is a workaround to an interesting paramiko side-effect16:03
openstackgerritMalini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration  https://review.openstack.org/6514016:05
*** krotscheck is now known as krotscheck_sick16:07
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Fix provider.keypair so it loads the image's private key which enables SSH in on first boot.  https://review.openstack.org/7067216:08
*** bhuvan has quit IRC16:09
*** jcooley_ has joined #openstack-infra16:09
*** bhuvan has joined #openstack-infra16:10
devanandag'morning, folks16:12
openstackgerritMatt Farina proposed a change to openstack-infra/gitdm: Adding Matt Farina.  https://review.openstack.org/7042216:13
*** emagana has joined #openstack-infra16:13
fungidevananda: i don't suppose you happen to have any status information for the tripleo-ci cloud lifeless was rebuilding yesterday?16:13
devanandafungi: i do not. however, ironic's gate is still broken16:13
devanandafungi: i heard that folks landed some sort of fix overnight. it seems to have merely changed the way our gate is broken16:14
fungidevananda: oh, did the devstack patch which corrected the previous patch make it in? last i saw they were trying to find devstack cores still awake to approve it16:14
devanandafungi: i believe so16:15
malinifungi: hello again!!16:15
devanandai haven't had any coffee yet, and without its super powers, i can not even find where the failure in https://review.openstack.org/#/c/70869/ is16:15
fungidevananda: yeah, https://review.openstack.org/70885 is the one i was thinking of, and it's merged as of 12:30 utc16:16
fungimalini: howdy16:16
*** mrodden has joined #openstack-infra16:16
malinifungi: good :) I have another patch set ready for you https://review.openstack.org/#/c/65140/5/16:16
maliniCan you chk when you get a chace?16:16
*** bhuvan has quit IRC16:17
fungimalini: i think you still had a problem with the rebase on that change16:20
fungisee yje review comment i just added16:20
*** prad_ has joined #openstack-infra16:20
fungidevananda: i see the non-voting and experimental checks which ran for 70885 didn't fix the issue (see various jenkins job results in the comment history)16:22
*** pcrews has quit IRC16:22
malinifungi: I am really confused on tht one as  well16:24
maliniI checked the repo to make sure that the ironic stuff is already there16:24
malinilet me get you the repo lines16:24
*** katyafervent_awa is now known as katyafervent16:25
devanandafungi: http://logs.openstack.org/85/70885/1/check/check-tempest-dsvm-ironic-nv/170477a/logs/screen-n-cpu.txt.gz?level=AUDIT16:25
devanandafungi: looks like n-cpu didn't even DO much16:25
devanandafungi: i've been digging through the screen logs and dont see any errors, except the omission of nova doign any work16:26
*** emagana has quit IRC16:26
*** miqui has quit IRC16:29
jnollerfungi: did we ever see performance results?16:30
*** emagana has joined #openstack-infra16:31
fungijnoller: i haven't. i think we need someone to put together a proper statistical analysis. also i noticed yesterday that over the weekend e stopped being able to build new images from that base image... haven't had time to dig into the logs yet and find out why16:32
*** max_lobur has quit IRC16:33
malinifungi: the only possibility I can think of is the ironic stuff was there when I originally submitted the patchset & got removed from the repo since then..See an older patch here https://review.openstack.org/#/c/65140/3/modules/openstack_project/files/zuul/layout.yaml16:33
*** miqui has joined #openstack-infra16:33
malinifungi: does tht imply rebase doesn't remove stuff ? or I am doing something weird16:34
*** matrohon has joined #openstack-infra16:34
*** bookwar has quit IRC16:35
*** max_lobur has joined #openstack-infra16:35
fungimalini: it means at some point when you rebased you probably misinterpreted one of the conflict resolution chunks and left those lines in when the rebase said they should no longer be in master16:35
*** masayukig has joined #openstack-infra16:35
fungimalini: you could just amend your patch to take those lines out of it and will probably be fine16:35
malinifungi: tht is very much possible..Checking now16:35
sdaguejnoller: if you pull the latest - https://github.com/sdague/tempest-timings and run scrape.py you'll get the latest results. My understanding is check-rax-iad is the new image16:36
sdaguethat also calculates mean and std deviation timings on all the clouds16:37
openstackgerritMalini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration  https://review.openstack.org/6514016:37
sdaguejnoller, fungi: http://dague.net/testing/check-tempest-dsvm-full.png and http://dague.net/testing/check-tempest-dsvm-postgres-full.png16:38
*** thuc has joined #openstack-infra16:39
*** masayukig has quit IRC16:39
*** thuc_ has joined #openstack-infra16:40
fungisdague: jnoller: lower mean by about 15-20% and std stayed about the same i guess16:40
*** jcooley_ has quit IRC16:40
sdaguefungi: yeh, that's my take16:41
*** emagana has quit IRC16:41
*** markwash has joined #openstack-infra16:41
fungiso it doesn't seem to address the variability in runtimes we're seeing16:41
sdaguefungi: correct16:41
fungithough it is a bit faster, at least16:41
jnollerYou're using perf1-1 images, right?16:42
sdaguethat level of variability in rackspace cloud has been there forever though16:42
jnolleroh wait, 8gb16:42
sdagueit's not new16:42
*** markmcclain has joined #openstack-infra16:42
sdagueI think my first version of this scraper was over a year ago, and the results at the time were basically the same.16:43
jnollerthere's too much variability for my taste.16:43
jnollerI think I need to talk to pvo and gabe16:43
*** thuc has quit IRC16:43
markmcclainsdague: have a bit of time to talk about lingering config drive/metadata service/key injection issues?16:43
jnollerneed to think of a good summary though16:44
sdaguejnoller: so the interesting thing about the variability, if you look at the graphs, is it's not continuous16:44
sdagueit's sort of clusters around a few different values16:44
sdaguewhich I haven't done anything to pull those out with real math yet16:44
jnollerwestmaas_ ^^ see above when you're back16:44
jeblairsdague: i assume hpcloud-b has very little data (we run _very_ few jobs on that, each graph probably has 2 points) so the std for that probably isn't supportable, but is the mean close enough to az1-3 to expand the region-b quota to get more data?16:44
sdaguebut my expectation is that what's really going on is we're exposing some non homogeneous part of the rax infrastructure, and we can actually see that difference in the bulk data16:45
sdaguejeblair: sure, or just ignore region-b16:46
westmaas_faster average, still variable?16:46
westmaas_and this is just at runtime right? cause you pre-build instances?16:46
jnollerwestmaas_: see http://dague.net/testing/check-tempest-dsvm-full.png http://dague.net/testing/check-tempest-dsvm-postgres-full.png16:46
jeblairsdague: ignore?  that's the opposite of use more... what do you mean?16:46
fungijeblair: we increased the number of jobs we're running on hpcloud-region-b to be the same as for the rax-iad check nodes... right now we have a min 1 with equal weighting between the two providers (to avoid having more than one rax-iad provider configured)16:46
sdaguewestmaas_: yeh, graphs above, as well updated scraper script16:46
*** SergeyLukjanov is now known as SergeyLukjanov_a16:46
sdaguejeblair: I'm saying, don't care about the region-b data16:46
* fungi gets some quick node counts16:47
sdaguerealistically, we could collect more data on it16:47
jeblairsdague: oh, well, it's the future and az1-3 are deprecated so we should probably be somewhat interested in it.  :)16:47
sdaguebut my eyeball over time is that the hp azs are pretty equivalent16:47
sdagueok, so sure, run more on it to see how it reacts16:47
westmaas_we have some different manufacturers in different DCs...is that that straight line the HP region b or the rax ord?16:47
jnollerwestmaas_: the black line is perf 8gb in IAD16:48
westmaas_or too suspiciously straight16:48
devanandafungi: i see the failure that seems to be hitting our gate affecting others too16:48
fungijeblair: oh, nevermind. the max for hpcloud-region-b keeps us from adding more than one node at a time16:48
sdaguewestmaas_: so realize it's a line graph... so less than idea. I'd look at the mean / std for most of the bits16:48
westmaas_jnoller: yeah we have both manufacturers in iad :)16:48
devanandafungi: logstash search for: virSecurityDACRestoreSecurityFileLabel  AND build_status:"FAILURE"16:48
jeblairwestmaas_: i'm assuming region b is the straight line due to the low std16:48
sdaguejeblair: yeh, it's probably 2 data points total16:48
westmaas_sdague: yeah, k16:48
sdagueso will always be a straight line16:49
devanandafungi: actually, ignore that16:49
devanandared herring16:49
*** bhuvan has joined #openstack-infra16:49
westmaas_ok - I'm dealing with some other issues today, but will circle back on this with our perf testers16:49
sdaguewestmaas_: https://github.com/sdague/tempest-timings - ./scrape.py -s    will dump the timing data out to json as well16:49
westmaas_thanks for the help guys, really appreciate it16:49
westmaas_we'll get this better16:49
sdagueso you can look at actually number instead of silly graphs16:50
*** pcrews has joined #openstack-infra16:50
*** marun has joined #openstack-infra16:50
sdaguemarkmcclain: so honestly, I'm not entirely current on the issue, and need to actually try to get back to sleep (it's a very late/early local time in .kr right now) :)16:51
*** mestery has joined #openstack-infra16:51
*** mestery has quit IRC16:51
*** AlexF has joined #openstack-infra16:51
*** zehicle_at_dell has joined #openstack-infra16:52
*** dprince has quit IRC16:52
*** bookwar has joined #openstack-infra16:52
*** rcleere has joined #openstack-infra16:52
*** mestery has joined #openstack-infra16:53
markmcclainsdague: ok.. you can ping me when you're in awake hours (in PST this week)16:53
*** CaptTofu has quit IRC16:53
markmcclain*I'm PST this wk16:53
*** thedodd has joined #openstack-infra16:54
*** Ryan_Lane has joined #openstack-infra16:55
*** afazekas has quit IRC16:55
*** smurugesan has joined #openstack-infra16:56
*** boris-42_ has joined #openstack-infra16:57
devanandafungi: ok - really fouind it this time. n-cpu seems to just stop without any log message, then all subsequent attempts to schedule fail with "Filter ComputeFilter returned 0 hosts"16:58
zaropleia2: is it bugaday?16:58
devanandafungi: logstask search for: build_status:"FAILURE" AND filename:"logs/screen-n-sch.txt" AND "Filter ComputeFilter returned 0 hosts"16:59
*** zehicle_at_dell has quit IRC16:59
fungimordred: i opened https://launchpad.net/bugs/1276242 just now--you might be interested... it's pypi-mirror/wheel related16:59
*** zehicle_at_dell has joined #openstack-infra16:59
*** e0ne_ has quit IRC16:59
pleia2zaro: it is! starting now :)17:00
pleia2happy bug day17:00
pleia2etherpad here: https://etherpad.openstack.org/p/cibugreview-february201417:00
*** zehicle_at_dell has quit IRC17:00
devanandafungi: can we push https://review.openstack.org/#/c/70544/ through now?17:00
*** AlexF has quit IRC17:00
*** zehicle_at_dell has joined #openstack-infra17:00
*** boris-42_ is now known as boris-4217:00
zarofungi: you think you can do quick approval here to unblock my testing?  https://review.openstack.org/7086517:00
devanandafungi: this is, afaict, the only substantial difference between ironic's d-g and nova's, and our gate has been wedged since friday night ...17:01
*** CaptTofu has joined #openstack-infra17:01
devanandaclarkb: or if you're around ^17:01
fungidevananda: i have no problem with it... it's mostly between ironic and qa. i can review it on its correctness as a job configuration patch17:02
*** jcoufal_ has joined #openstack-infra17:02
*** jcoufal has quit IRC17:02
jeblairzaro: when you have a minute, can you look into combining the WIP and Approval functions into one column?17:02
jeblairzaro: so instead of having a "WIP" column and an "Approved" column, you have a "Workflow" column with -1=WIP and +1=Approved...17:02
*** mestery has quit IRC17:02
*** dizquierdo has quit IRC17:03
*** rockstar has joined #openstack-infra17:03
*** rockstar has joined #openstack-infra17:03
jeblairzaro: (with the usual "at least one +1 and also no -1 votes" requirement)17:03
*** mestery has joined #openstack-infra17:03
*** boris-42 is now known as boris-42_17:04
*** mestery has quit IRC17:05
*** markmc has quit IRC17:05
*** mestery has joined #openstack-infra17:06
zarojeblair: ok.  i'll take a look.17:06
*** hcc is now known as hdd17:07
reedjeblair, fungi: smarcet fixed the comments made by Marton, please check https://review.openstack.org/#/c/69636/17:08
*** slong has joined #openstack-infra17:09
*** gyee has joined #openstack-infra17:10
*** mestery_ has joined #openstack-infra17:10
*** gyee has quit IRC17:11
*** mestery_ has quit IRC17:11
*** mestery has quit IRC17:11
*** mestery has joined #openstack-infra17:12
openstackgerritA change was merged to openstack-infra/config: add pipelines and update comment filter for gerrit 2.8  https://review.openstack.org/7086517:12
*** gyee has joined #openstack-infra17:13
*** sarob has joined #openstack-infra17:13
*** rakhmerov has quit IRC17:15
pleia2jeblair, clarkb - bug day :) https://etherpad.openstack.org/p/cibugreview-february201417:16
*** sarob has quit IRC17:20
zarofungi: thank for quick aprv.17:21
*** AlexF has joined #openstack-infra17:21
*** talluri has joined #openstack-infra17:23
*** salv-orlando has quit IRC17:23
devanandajeblair: have a moment to help me try to unwedge Ironic's gate? https://review.openstack.org/#/c/70544/2 needs a +A17:25
*** jasondotstar has quit IRC17:25
*** salv-orlando has joined #openstack-infra17:26
*** pcrews has quit IRC17:26
*** oubiwann_ has quit IRC17:28
lifelessfungi: it's not, because you had it offline I didn't kill myself getting it up before bed17:29
lifelessfungi: I will be back on it after my morning-of-hell-meetings17:29
fungilifeless: okay--we didn't want to add it back to the nodepool config until it was up and running again, so stalemate on that last night i guess17:30
*** AlexF has quit IRC17:31
*** mestery_ has joined #openstack-infra17:34
*** jamesmcarthur has joined #openstack-infra17:34
*** amotoki has quit IRC17:34
*** jcooley_ has joined #openstack-infra17:34
jamesmcarthurHey guys. Just a heads up that we are about to do a deployment on OpenStack.org. Any services/graphics that are in use from this domain may see a temporary outage. We're expecting about 5-10 minutes of down time during this maintenance window.17:35
fungijamesmcarthur: thanks! i saw toddmorey's e-mail yesterday too. good luck!17:36
*** masayukig has joined #openstack-infra17:36
*** rakhmerov has joined #openstack-infra17:36
*** AlexF has joined #openstack-infra17:36
jamesmcarthurthank you sir!17:37
jeblairpleia2: ack17:37
*** mestery has quit IRC17:38
*** moted has joined #openstack-infra17:38
*** jamesmcarthur has left #openstack-infra17:38
*** masayukig has quit IRC17:41
*** sarob has joined #openstack-infra17:41
*** sandywalsh_ has quit IRC17:42
*** sandywalsh has joined #openstack-infra17:43
openstackgerritA change was merged to openstack-infra/config: Make Ironic tests stop using file injection  https://review.openstack.org/7054417:43
*** SpamapS_ is now known as SpamapS17:43
*** jasondotstar has joined #openstack-infra17:44
*** jcooley_ has quit IRC17:44
jeblairgrr, i thought we didn't link to openstack.org for any of the css, etc for status.o.o17:44
*** AlexF has quit IRC17:44
*** Ajaeger has joined #openstack-infra17:45
fungii thought we decided we shouldn't, but don't recall whether it was fixed17:45
*** fbo has quit IRC17:46
*** msuriar has quit IRC17:46
*** cp16net has quit IRC17:46
pleia2anteaya: is there work that can be done without Hunner's puppet board configs for apache? I'm happy to keep working on this, I just thought we were waiting on him17:46
*** sarob has quit IRC17:46
pleia2(not wanting to reinvent the wheel and all)17:46
anteayaI don't know what to do without him, pleia217:47
fungiyep, i cound at least 5 css files linked from www.o.o in the status.o.o page17:47
anteayahe had set up the example locally and stalled on getting it into our config17:47
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Reduce rax-iad usage by 50%  https://review.openstack.org/7104217:47
anteayaon the db issue, if I remember correctly17:47
anteayasince it requires puppetdb and we don't have that in our config17:47
jeblairfungi: pls spdy aprv https://review.openstack.org/71042 ?17:47
jeblairantonym: https://review.openstack.org/7104217:48
pleia2anteaya: ok, I updated your comment in the etherpad, it's not really stalled on us ;)17:48
anteayalast I talked to him, he was going to talk to his manager to get some time for this17:48
pleia2ok, thanks17:48
anteayapleia2: thank you, yes not stalled on us, sorry my comment left that impression17:48
openstackgerritmark mcclain proposed a change to openstack-infra/config: Enable config drive on all Neutron jobs  https://review.openstack.org/7104317:48
*** cp16net has joined #openstack-infra17:48
*** burt1 has joined #openstack-infra17:49
*** msuriar has joined #openstack-infra17:49
*** sarob has joined #openstack-infra17:49
devanandamarkmcclain: heh :)17:50
fungijeblair: i'll hand-apply 71042 on nodepool.o.o now, since puppet is intentionally off there while the tripleo-ci cloud continues its restoration17:50
jeblairfungi: ok, thx17:50
markmcclaindevananda: yeah I'd like to get patches merging again17:51
*** AlexF has joined #openstack-infra17:51
devanandamarkmcclain: nearly the same patch for ironic's tests just went through, i'm watching to see if that fixes it17:52
openstackgerritA change was merged to openstack-infra/config: Reduce rax-iad usage by 50%  https://review.openstack.org/7104217:52
antonymjeblair: thanks!17:52
*** amcrn has joined #openstack-infra17:53
AjaegerHi infra team, looking at one of the bugs on etherpad, I can tackle it if I get some advise: https://bugs.launchpad.net/openstack-ci/+bug/1216153 . The job is at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/manuals.yaml#n111 - and I have one question: Tom proposes to use generatedocbook from openstack-doc-tools instead of calling ./tools/generatedocbook directly. H17:53
*** esker has quit IRC17:54
anteayapleia2: though I will say in fairness to Hunner that I haven't been around to talk to him about it17:54
*** salv-orlando_ has joined #openstack-infra17:54
*** esker has joined #openstack-infra17:54
*** dprince has joined #openstack-infra17:54
jeblairAjaeger: you were cut off at "./tools/generatedocbook directly. H"17:54
*** gokrokve_ has quit IRC17:54
Ajaegerjeblair: oh, let me repaste again17:55
*** fbo has joined #openstack-infra17:55
AjaegerTom proposes to use generatedocbook from openstack-doc-tools instead of calling ./tools/generatedocbook directly. How can we do this best?17:55
jeblairAjaeger: irc has a 512 char limit iirc17:55
*** oubiwann_ has joined #openstack-infra17:55
Ajaegerjeblair: Sorry, didn't count ;)17:55
*** resker has joined #openstack-infra17:55
*** gokrokve_ has joined #openstack-infra17:55
fungijudging from the test nodes graph, nodepool doesn't seem to be deleting nodes as quickly following the merger of the aggressive deletes patch series last night17:56
fungiclarkb: ^17:56
*** salv-orlando_ has quit IRC17:56
*** esker has quit IRC17:57
fungii was mostly paying attention to things which slipped past the 3 hour mark without being culled, but there's a bunch between 2 and 3 hours in delete17:57
*** salv-orlando_ has joined #openstack-infra17:57
jeblairAjaeger: i think the best plan would be: change that to a freestyle project; use virtualenv (possibly via tox) to pip install openstack-doc-tools; run the tool; then "mvn whatever..."17:57
*** salv-orlando has quit IRC17:57
*** salv-orlando_ is now known as salv-orlando17:57
*** Guest54041 has joined #openstack-infra17:58
jeblairAjaeger: (strictly speaking, you could probably do that without changing the maven aspect, and just make a pre-builder that did the virtualenv bit)17:58
*** Guest54041 has quit IRC17:58
jeblairAjaeger: (but we are going to need to stop using maven jobs in jenkins soon anyway, so i'm trying to trick you into doing that at the same time while you're working on the job) :)17:58
*** harlowja_away is now known as harlowja17:59
*** elasticio has joined #openstack-infra17:59
* mordred likes when jeblair is tricky17:59
*** elasticio_ has joined #openstack-infra17:59
jeblairmordred: the best part is he'll never suspect ;)17:59
*** elasticio is now known as Guest757517:59
mordredjeblair: nice one!17:59
*** Guest7575 has quit IRC17:59
fungitricksy jeblair. we hates maven, we does17:59
*** elasticio_ is now known as Guest5828917:59
*** Guest58289 has quit IRC17:59
anteayamarkmcclain: before I vote on 71043 do we have any proof this will work? or just revert if it doesn't?17:59
*** AlexF has quit IRC17:59
Ajaegerjeblair: If you want me to convert all maven jobs, just tell me the whole story - the openstack-doc-test tool calls maven directly already. We can replace the maven stuff with jobs that publish.17:59
Ajaegerjeblair: have a look at https://review.openstack.org/#/c/70594/.18:00
AjaegerI can change that doc-publish job to build *all* books in one go and we then publish from publish-docs dir.18:00
*** KurtMartin is now known as kmartin18:00
*** 65MAAIZIK has joined #openstack-infra18:00
*** elasticio__ has joined #openstack-infra18:00
Ajaegerthe checkbuild tox test calls maven already...18:00
*** elasticio__ is now known as Guest518718:01
AjaegerIs that a direction we should go?18:01
*** morganfainberg_Z is now known as morganfainberg18:01
AjaegerIn that case, I could change the translation job as a first test...18:01
*** hashar has quit IRC18:01
jeblairAjaeger: awesome; yes -- we need to get rid of maven-based jenkins jobs because we want to get rid of jenkins (calling maven directly from shell is obviously fine, we don't care about that)18:02
*** Guest5187 has quit IRC18:02
markmcclainanteaya: when looking at https://review.openstack.org/#/c/67537/18:02
*** 65MAAIZIK has quit IRC18:02
markmcclainyou'll notice that the tests with config_drive enabled pass18:02
markmcclainand do not trigger the nova-compute bug which the nova team is currently working on18:02
* anteaya clicks to notice18:02
Ajaegerjeblair: can you review 70594 and tell me whether that logic for publishing is fine, please? Based on the review for that, I handle bug 1216153 ... and then do the other maven jobs...18:03
Ajaeger(the Documentation team related ones only)18:03
jeblairAjaeger: sure; i'm just back from a bit of traveling, so i have some catching up to do, but i starred that one and will try to look at it early on in my reviewing to keep you from being blocked too long18:04
clarkbfungi: as quickly compared to before the merge?18:04
*** nati_ueno has joined #openstack-infra18:04
mordredjeblair: I need to run out - I may  not be able to make it back by the infra meeting - but I'm going to try my best18:04
fungiclarkb: far, far less quickly18:04
clarkbfungi: ah yes, I see the vast regions of purple18:05
*** max_lobur is now known as max_lobur_afk18:05
*** apevec has joined #openstack-infra18:05
*** SergeyLukjanov_a is now known as SergeyLukjanov18:05
Ajaegerjeblair: ok.18:05
jeblairpleia2, clarkb, fungi: i will try to work on triaging18:06
fungijeblair: i've started triaging as well18:06
fungiclarkb: do you think 67986 is likely to help?18:06
jeblairfungi: ok.  are you starting from the top or bottom? :)18:06
apevechi, I have grenade g->h issue: http://logs.openstack.org/49/66149/2/check/check-grenade-dsvm/50595f9/logs/old/screen-c-vol.txt.gz18:06
fungiclarkb: i haven't looked at the logs yet to see whether it's throwing exceptions when deleting18:07
clarkbfungi: it may18:07
fungijeblair: at the top, but i can switch18:07
apevec"old" Grizzly fails with pkg_resources.DistributionNotFound: SQLAlchemy>=0.7.3,<=0.7.918:07
jeblairfungi: no prob, i'll start at bottom18:07
clarkbsince I believe the current deletion is do a delete not blocking, if that fails then we wait18:07
apevecpip freeze shows SQLAlchemy==0.7.1018:07
clarkbfungi: I can approve that change and see if it helps18:07
reedthanks fungi18:07
* reed away european time18:07
fungiclarkb: well, we'd need to hand-patch it onto the nodepool server18:07
apevecsdague, ^^^ are old and new sharing the same venv in grande?18:07
*** slong has quit IRC18:07
openstackgerritPhilippe Godin proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin.  https://review.openstack.org/6960818:08
clarkbfungi: oh right, I will approve and hand patch then18:08
*** e0ne has joined #openstack-infra18:08
clarkbthen swing around and do bugs with everony else18:08
fungiclarkb: tripleo cloud is still offline, but lifeless hopes to finish restoring it to service after his day-o-meetins18:08
anteayamarkmcclain: okay let's see what this does, +1'ing18:09
clarkbfungi: done18:09
markmcclainanteaya: dansmith may have fixed the bug on the nova side18:09
markmcclainwaiting for the experimental checks to complete on his patch18:09
anteayayay dansmith18:09
anteayapatch #?18:09
clarkbdevananda: fungi: I thought lifeless actually fixed devstack18:10
lifelessclarkb: I did18:10
lifelessand dansmith has just fixed nova (pending confirmation by tempest)18:11
clarkbgreat, then we shouldn't need to change the ironic jobs (unless something else cascaded into broken)18:11
dansmithlifeless: hold on there, cowboy :)18:11
lifelesssee 71046,118:11
dansmithlifeless: dansmith has just proposed a hopeful fix :)18:11
lifelessclarkb: fixing ironic is fairly important; I'm not suggesting not merging deva's thing18:11
clarkblifeless: I am18:11
clarkblifeless: if the thing is fixed why would we 'fix' it somewhere else?18:12
lifelessdansmith: I did say pending confirmation :)18:12
lifelessclarkb: because as I said for devstack fixing the thing in front of us might not fix ironic18:12
dansmithlifeless: it wasn't enough speculative-sounding for me :D18:12
lifelessclarkb: we don't know how deep the rabbit hole goes18:12
clarkbbut we won't know either unless we test it18:13
lifelessclarkb: but we do know that Ironic doesn't care about any of it for now, it just wants neutron running for  it's api tests until it gets it's own nova driver in the mix18:13
clarkbwhich devananda's change prevents18:13
devanandalifeless: the hopeful fix for ironic (stop disabling configdrive) has merged18:13
lifelessdevananda: cool18:13
lifelessclarkb: I thought deva's change was in ironic context only?18:13
devanandaalso, fwiw, we arent starting up neutron in ironic's d-g env at the moment18:13
devanandalifeless: it should be only in ironic tests, yes18:14
clarkblifeless: it is but as you said yesterday we need to test it and neutron and neutron isolated18:14
clarkbas there may be other problems18:14
lifelessclarkb: yes, but they are separate concerns - no need to hold ironic's jobs hostage18:14
*** sarob has quit IRC18:14
lifelessclarkb: my concern there was only ever that I thought config drive on forced metadata off, which it does not.18:14
*** sarob has joined #openstack-infra18:15
lifelessit just means that vm's don't *talk* to metadata because they have config drive locally.18:15
lifelessclarkb: and that matters in the holistic what-are-we-testing discussion18:15
openstackgerritA change was merged to openstack-infra/config: Cleanup old servers every minute.  https://review.openstack.org/6798618:15
lifelessclarkb: but ironic doesn't count *at all* in that space, yet, not for nova.18:15
Ajaegerclarkb: I need to pass an extra parameter to tox. So, instead of adding lots of "build-lang-$LANG" entries to tox.ini and calling these from run-tox.sh, I'd like to run "tox -e build-lang -- $LANG". Is that directly possible?18:15
*** coolsvap is now known as coolsvap_away18:16
jeblairAjaeger: you can pass the extra command line args given to tox on to the program it runs; there should be some examples in the python tox.ini files18:17
jeblairAjaeger: the 'venv' env comes to mind18:18
*** talluri has quit IRC18:18
clarkband if you need that in the jenkins jobs we can probalby have run-unittests do a shift then pass $@ to tox18:18
clarkblifeless: I understand but yesterday you were arguing for testing that the fix fixed everything, we will now not actually test that18:19
lifelessclarkb: I was arguing for making sure it fixed things before we merged it yes18:19
lifelessclarkb: I didn't want to run of saying 'all good, next in queue!' if that wasn't the case18:20
*** nicedice has joined #openstack-infra18:20
lifelessclarkb: it's still entirely possible ironic will have other issues; which will lurk for a while :( but deva needs to land stuff - cons and pros.18:20
*** oubiwann_ has quit IRC18:22
*** AlexF has joined #openstack-infra18:22
*** oubiwann_ has joined #openstack-infra18:22
clarkbfungi: jeblair: pleia2: where should I jump on bugs?18:23
*** pcrews has joined #openstack-infra18:23
jeblairclarkb: fungi and i are triaging new ones; you want to scan the list and look for any we could close or drop?18:24
clarkbfungi: delete state nodes fell from 281 to 227 over the last few minutes18:24
*** jcooley_ has joined #openstack-infra18:24
clarkbjeblair: yup18:24
devanandaclarkb: roughly how long before the merge to https://review.openstack.org/#/c/70544/ trickles down into test runs? ~ an hour, ya?18:24
clarkbdevananda: 10-15 minutes after the */15 cron18:25
fungiclarkb: jeblair and i are going through new bugs from both ends, so you could either start grabbing some out of the middle or dig into the ones in teh etherpad you know merit status updates18:25
fungiclarkb: excellent18:25
clarkbmordred: want to add anything to https://bugs.launchpad.net/openstack-ci/+bug/1242569 ?18:25
clarkbfungi has it up to date, not sure if there are additional things you want to add18:26
*** jcooley_ has quit IRC18:26
fungiis the log error checker script in infra/config as a slave script, or in devstack, tempest, elsewhere?18:26
clarkbfungi: tempest18:27
clarkbunder tools/ iirc18:27
fungiclarkb: thanks (one more bug i can move to another project)18:27
*** dkehn has quit IRC18:27
clarkbfungi: for https://bugs.launchpad.net/openstack-ci/+bug/1010621, once I backup eavesdrop any chance you want to do some of the puppetmaster work?18:27
fungiclarkb: sure18:28
clarkbfungi: great (I should hurry up and do eavesdrop, it should be easy)18:28
*** dkehn has joined #openstack-infra18:28
*** thuc_ has quit IRC18:28
*** thuc has joined #openstack-infra18:29
clarkbfungi: zaro: for https://bugs.launchpad.net/openstack-ci/+bug/1267364 I feel like we should take the time to report that upstream since upstream claims to have fixed it but it clearly isn't fixed18:30
clarkbat the very least poke the bug that indicated 1.542 had the fix and say 1.543 is not fixed18:30
*** jooools has quit IRC18:30
fungiclarkb: unfortunately since that report i've not caught another failure on a slave assigned to one of the upgraded jenkins masters quickly enough to catch the console log before it gets expired18:31
*** wendar_ is now known as wendar18:31
clarkbfungi: I bet having single use slaves makes it hard18:31
clarkbI wonder if those tracebacks show up in the jenkins /var/log/ log somewhere18:32
fungialso since adding more masters and getting a handle on the slave churn, the count of reports has dropped off (i don't even know whether we're still seeing that particular traceback any longer)18:32
*** e0ne_ has joined #openstack-infra18:33
*** thuc has quit IRC18:33
*** changbl has joined #openstack-infra18:34
clarkbfungi: down to 86 slaves in the delete state18:34
fungiclarkb: though the ready node count looks to have jumped up18:34
*** e0ne has quit IRC18:35
*** gokrokve_ has quit IRC18:35
*** gokrokve has joined #openstack-infra18:36
*** thuc has joined #openstack-infra18:36
*** masayukig has joined #openstack-infra18:36
*** che-arne has quit IRC18:37
*** mestery_ has quit IRC18:38
*** dprince has quit IRC18:39
*** yassine has quit IRC18:39
clarkbpleia2: https://bugs.launchpad.net/openstack-ci/+bug/1082795 any chance you can upgrade that with the state of latest and greatest tripleo testing18:39
zarofungi, clarkb : so could that be that we had too many connected slaves to the master?18:39
*** jpich has quit IRC18:40
clarkbjeblair: https://review.openstack.org/#/c/69442/ fixes a 'high' bug. Didn't note that in the commit message because I didn't know there was a preexisting high bug for the problem. I can update the commit message if you like, but that should probably get eyes on it if possible18:40
fungizaro: well, we've seen various slave agent communication failures in the past when we scaled up either the number of slaves or possibly rate at which we were adding and removing them18:40
clarkbzaro: sure, but its still a bug in jenkins18:40
*** markmcclain has quit IRC18:40
pleia2clarkb: sure thing18:41
*** masayukig has quit IRC18:41
*** gokrokve has quit IRC18:41
clarkb'too many slaves' is not something we should be saying about a jenkins with 100 slaves18:41
*** shivh has joined #openstack-infra18:41
*** sdake has quit IRC18:42
clarkbfungi: re six not getting latest versions installed, do we know what is happening there? is mirror build failing before it gets to six?18:42
*** sdake has joined #openstack-infra18:43
*** sdake has quit IRC18:43
*** sdake has joined #openstack-infra18:43
fungiclarkb: that's my suspicion. i only found time to document the symptom, but not yet time to start pouring over the logs18:44
jeblairzaro, clarkb, fungi: i don't believe the noclassdeffound error is correctible18:44
fungii've got it up in a tab and will try to unearth more details if nobody beats me to it18:44
*** dims has joined #openstack-infra18:44
clarkbjeblair: at the very least jenkins should mark the node offline and reschedule the job that would have failed on a different node18:45
jeblairzaro, clarkb, fungi: i mean, it may be worth mentioning on the upstream bug, but i think at this point we have found that if a job is aborted at just the right time, there is no way in java to correct that.18:45
jeblairclarkb: yeah, or perhaps try to detect it and re-cycle the node (offline-online)18:45
clarkbmordred: https://bugs.launchpad.net/openstack-ci/+bug/1254275 is that something that is still a problem?18:46
jeblairclarkb: i sort of doubt it would be a priority for them because most people don't abort jobs at scale.  :)18:46
fungiworse, it doesn't offline the node at all, and just keeps spamming it with more jobs18:46
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang}  https://review.openstack.org/7105618:46
Ajaegerclarkb, jeblair: I'm calling tox now directly in the change above so that you see what I want to do.18:47
AjaegerYour hints with run-unittests and venv didn't help me ;(18:48
clarkbjeblair: for https://bugs.launchpad.net/openstack-ci/+bug/1270319 you indicated it was a one line fix in geard, can I assign that bug to you?18:49
jeblairclarkb: sure18:49
*** VijayT has joined #openstack-infra18:49
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang}  https://review.openstack.org/7105618:50
clarkbhttps://review.openstack.org/#/c/69969/ is another fix for a high bug18:50
*** thedodd has quit IRC18:51
jeblairi'm retargeting clouddocs-maven-plugin bugs to openstack-manuals since that tool is not part of openstack-infra18:51
zarojeblair: i believe you have the power to do something about these? https://bugs.launchpad.net/openstack-ci/+bug/1247866 and https://bugs.launchpad.net/openstack-ci/+bug/126098318:52
jeblairAjaeger: sorry; i'll try to be more helpful when i look more closely at your changes18:52
clarkbpleia2: https://bugs.launchpad.net/openstack-ci/+bug/995592 is that something I should assign to you?18:52
clarkbjeblair: ++18:53
Ajaegerjeblair: no problem at all. Look closely and tell me what needs to be changed. I implement the tools support for this now ;) Thanks!18:53
clarkbzaro: pleia2 has been working on getting fedora to work with nodepool recently and making nodepool work with centos should be very similar. Once we have that we can kill the static slaves I think18:53
pleia2clarkb: ah yes, I'll grab that one18:53
clarkbpleia2: thanks18:54
*** jcooley_ has joined #openstack-infra18:55
*** freyes has joined #openstack-infra18:55
*** gokrokve has joined #openstack-infra18:55
*** andreaf has quit IRC18:55
*** sdake has quit IRC18:55
*** melwitt has joined #openstack-infra18:56
jeblairzaro: scp-plugin pr merged18:57
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang}  https://review.openstack.org/7105618:58
*** gokrokve has quit IRC19:00
*** markmcclain1 has joined #openstack-infra19:00
pleia2meeting time19:01
*** markmcclain1 has quit IRC19:01
*** bhuvan has quit IRC19:02
*** bhuvan_ has joined #openstack-infra19:02
*** jnoller has quit IRC19:03
*** mestery has joined #openstack-infra19:04
*** jcoufal_ is now known as jcoufal19:06
*** markmcclain has joined #openstack-infra19:06
*** gokrokve has joined #openstack-infra19:06
*** dprince has joined #openstack-infra19:06
*** e0ne_ has quit IRC19:09
*** AlexF has quit IRC19:09
*** gokrokve has quit IRC19:11
*** rossella_s has quit IRC19:13
*** salv-orlando has quit IRC19:13
*** gokrokve has joined #openstack-infra19:14
*** salv-orlando has joined #openstack-infra19:14
*** jcooley_ has quit IRC19:17
*** reed has quit IRC19:18
*** reed has joined #openstack-infra19:19
*** sdake has joined #openstack-infra19:19
*** sdake has quit IRC19:19
*** sdake has joined #openstack-infra19:19
*** mriedem has quit IRC19:19
*** smarcet has left #openstack-infra19:20
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang}  https://review.openstack.org/7105619:22
*** FallenPegasus has joined #openstack-infra19:23
jeblairmikal: do you know if matthewoliver is/will be around?19:24
SlickNikdtroyer: around?19:26
SlickNikHad a questions for you.19:26
*** rakhmerov has quit IRC19:26
*** mriedem has joined #openstack-infra19:26
*** jasondotstar has quit IRC19:27
*** pblaho has joined #openstack-infra19:27
SlickNikSo to install swift via devstack, it's preferable to explicitly add the swift services to ENABLED_SERVICES, right?19:27
dtroyerSlickNik: yes19:28
SlickNikToday, the trove scripts add "ENABLED_SERVICES += swift" which seems to install all 3 swift services, but I don't think that's good practice.19:29
SlickNikespecially if you consider multi-node devstack installs.19:29
dtroyerSlickNik: right…just been talking to juice and hub_cap about that… I think we're on the same page now19:29
*** rakhmerov has joined #openstack-infra19:30
SlickNikCool, I'll make a change to the trove CI scripts to install the services explicitly. Much thanks for the confirmation!19:30
SergeyLukjanovdevananda, clarkb, I'm writing about https://review.openstack.org/#/c/70001/ (Remove Neutron from Ironic tempest test pipelines)19:30
juiceslicknik - i already have a patch up there19:31
SlickNikjuice: Awesome. Thanks!19:32
*** dkliban has quit IRC19:32
SergeyLukjanovdevananda, clarkb, I think that after removing neutron SMOKE_SERIAL couldbe disabled too19:32
SlickNikjuice: link?19:32
*** dkliban has joined #openstack-infra19:32
anteayaSlickNik: join us in -meeting won't you?19:33
devanandaSergeyLukjanov: when I checked with clarkb on that, my understanding was we'd have to add TEMPEST_FULL to replace it, which also seemed unnecessary19:33
*** AlexF has joined #openstack-infra19:33
SergeyLukjanovdevananda, AFAIK SMOKE_SERIAL == run smoke tests in 1 thread19:34
*** johnthetubaguy has quit IRC19:34
SergeyLukjanovdevananda, anyway I don't think that it's really important19:35
*** jcooley_ has joined #openstack-infra19:36
*** AlexF has quit IRC19:36
hub_capdtroyer: thx for the clarification sir19:37
*** masayukig has joined #openstack-infra19:37
*** pblaho has quit IRC19:37
shivhfungi: hi19:40
fungishivh: hi--what's up?19:41
*** masayukig has quit IRC19:42
*** sdake has quit IRC19:42
openstackgerritKhai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond  https://review.openstack.org/6034819:42
shivhi dont seem to have voting enabled for the account (shivharis)19:42
shivhthis account is exclusively for CI19:43
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang}  https://review.openstack.org/7105619:43
shivhfungi: i dont seem to have voting enabled for the account (shivharis)19:43
*** sdake has joined #openstack-infra19:43
*** sdake has quit IRC19:44
*** sdake has joined #openstack-infra19:44
*** sdake has quit IRC19:44
*** sdake has joined #openstack-infra19:44
*** sdake has quit IRC19:44
*** sdake has joined #openstack-infra19:44
*** jcooley_ has quit IRC19:44
annegentleSergeyLukjanov: I can chat names over here if you wanna19:45
fungishivh: contributor accounts are not allowed to place verify votes. you need to request a service account for that19:46
SergeyLukjanovannegentle, It'll be awesome! some details: http://lists.openstack.org/pipermail/openstack-dev/2014-January/025905.html19:46
annegentleSergeyLukjanov: oh okay found it19:46
*** AlexF has joined #openstack-infra19:46
openstackgerritKhai Do proposed a change to openstack-infra/config: Add new job to build the javamelody gerrit plugin  https://review.openstack.org/7001419:47
*** AlexF has quit IRC19:47
openstackgerritKhai Do proposed a change to openstack-infra/config: update All-Project configuration for gerrit 2.8  https://review.openstack.org/7081819:49
*** salv-orlando has quit IRC19:49
*** jcooley_ has joined #openstack-infra19:49
*** salv-orlando has joined #openstack-infra19:49
*** fbo is now known as fbo_away19:52
shivhfungi: where should i request a service a/c19:54
fungishivh: http://ci.openstack.org/third_party.html19:54
shivhfungi: thanks.19:54
*** thuc has quit IRC19:59
*** thuc has joined #openstack-infra19:59
mikaljeblair: he should be around today, but perhaps not for another hour19:59
*** zehicle_at_dell has quit IRC20:00
*** zehicle_at_dell has joined #openstack-infra20:00
pleia2ok, back to bug day? ;)20:00
*** markmc has joined #openstack-infra20:00
annegentlewhat is the latest on moodle, curious20:00
clarkbfungi: lifeless: for nodepool, it doesn't look like we kick off a delete immediately after the delete state is set20:01
*** bhuvan_ has quit IRC20:01
*** sparkycollier has joined #openstack-infra20:01
*** emagana has joined #openstack-infra20:01
clarkbthe nodeCompleteThread sets teh state to DELETE and is done20:01
clarkbI bet that is why deletes are taking so long20:01
jeblairclarkb: can " Jenkins SCP Plugin fix for Elastic Search (clarkb, zaro) " be removed from the agenda?20:02
clarkbjeblair: yes20:02
clarkbwe should immediately fire off a non blocking delete of the node20:02
jeblairclarkb: k thx; i thought that's what we talked about, just wanted to make sure20:02
clarkbI am going to go grab lunch now then back to bugs20:02
fungiannegentle: good question--i don't know20:03
fungiannegentle: dguitarbite and sarob wanted to toss that into the agenda at the last moment, but we were already runnign a packed meeting20:03
*** jcooley_ has quit IRC20:03
*** thuc has quit IRC20:03
lifelessclarkb: thats correct; the code consolidated that to avoid race conditions and now deletes them all together20:04
lifelessclarkb: when you say 'takes so long' - how long is it taking ?20:04
jeblairfungi, clarkb: btw on the plane i wrote the change to split out the zuul merger; i want to revise it a bit more but that should be up for review this week.  but it passes tests.  :)20:05
*** CaptTofu has quit IRC20:06
SergeyLukjanovjeblair, fungi, clarkb, could you please take a look on enabling savanna's d-g jobs? https://review.openstack.org/#/c/68066/20:07
*** jcooley_ has joined #openstack-infra20:07
fungiclarkb: that may have been the result of https://review.openstack.org/#/c/67985/2..3/nodepool/provider_manager.py ?20:08
*** sparkycollier has quit IRC20:08
jeblairSergeyLukjanov: i moved it to the top of my list (but i can't look at it right now)20:08
fungiclarkb: though now that i look at it, i don't think so after all20:09
SergeyLukjanovjeblair, thank you!20:09
*** jasondotstar has joined #openstack-infra20:09
*** rfolco has quit IRC20:11
*** kraman1 has joined #openstack-infra20:13
*** kraman has quit IRC20:13
*** thedodd has joined #openstack-infra20:14
fungijeblair: i'm looking forward to the new, more scalable, more better zuul. can't wait to see the patch20:14
*** AlexF has joined #openstack-infra20:15
openstackgerritBrant Knudson proposed a change to openstack/requirements: Update keystoneclient to 0.5.0  https://review.openstack.org/7108520:16
*** hashar has joined #openstack-infra20:17
clarkblifeless: see graph on zuul status page20:17
*** odyssey4me has quit IRC20:18
*** emagana has quit IRC20:20
*** sparkycollier has joined #openstack-infra20:22
fungiclarkb: zaro: did we have a bug filed for the truncated console log/scp publisher issue you fixed?20:23
*** mestery has quit IRC20:23
*** UtahDave has joined #openstack-infra20:25
*** emagana has joined #openstack-infra20:27
*** e0ne has joined #openstack-infra20:28
*** malini is now known as malini_afk20:29
*** _david_ has joined #openstack-infra20:33
*** FallenPegasus has quit IRC20:33
*** thuc has joined #openstack-infra20:35
*** atiwari has quit IRC20:35
wenlockhi, was looking for a possibility of a review today https://review.openstack.org/#/c/51425  , clarkb, hoping your day is looking better20:36
*** rakhmerov has quit IRC20:37
*** gokrokve has quit IRC20:37
*** masayukig has joined #openstack-infra20:38
*** gokrokve has joined #openstack-infra20:38
*** markmcclain has quit IRC20:38
*** rakhmerov has joined #openstack-infra20:39
fungijog0: have you seen https://launchpad.net/bugs/1262827 ?20:40
*** denis_makogon_ has joined #openstack-infra20:41
*** weshay has quit IRC20:42
lifelessclarkb: lets see20:42
*** masayukig has quit IRC20:42
*** gokrokve has quit IRC20:43
lifelessclarkb: whats the delete cron job schedule in config ?20:44
*** slong has joined #openstack-infra20:44
*** denis_makogon_ is now known as denis_makogon20:44
lifelessclarkb: -deployed that is-20:44
*** salv-orlando has quit IRC20:44
*** salv-orlando has joined #openstack-infra20:44
*** markmcclain has joined #openstack-infra20:45
*** weshay has joined #openstack-infra20:46
jog0fungi: looking now20:48
*** sdake has quit IRC20:48
jog0fungi: that has been fixed20:49
fungijog0: i'm slightly hesitant to triage elastic-recheck bugs since i haven't been doing much devel/review on it20:49
zarofungi: https://bugs.launchpad.net/openstack-ci/+bug/126098320:49
fungijog0: excellent. please update with details if you have a moment20:49
clarkblifeless: */120:49
clarkblifeless: I made the change by hand20:49
fungizaro: thanks! i found (at least) one untriaged dupe of it20:49
*** sdake has joined #openstack-infra20:50
*** sdake has quit IRC20:50
*** sdake has joined #openstack-infra20:50
lifelessclarkb: so every 60s it should be picking up all the new deletes and adding them20:50
lifelessclarkb: is that not happening?20:50
jog0fungi: I am happy to triage any e-r bugs as they get reported20:51
clarkblifeless: which log message should I be looking for?20:52
clarkblifeless: 'Deleting node id: .*' ?20:53
*** thuc has quit IRC20:53
fungijog0: i'm trying to tag them with elastic-recheck as i come across them20:53
lifelessclarkb: Starting periodic cleanup for provider20:53
lifelessclarkb: Finished periodic cleanup for provider20:53
lifelessclarkb: get me a good section of that - say an hours worth20:53
clarkblifeless: k20:53
*** thuc has joined #openstack-infra20:53
jog0fungi: cool, is there a way I can just subscribe to that tag?20:53
lifelessit may be that we're blocking in there inappropriately still, which would delay submitting the deletes to the provider20:54
* jog0 is enjoying eavesdropping on TC today20:54
clarkblifeless: how much context do you want/need?20:56
clarkbI have the starts and finishes if that is all you want ot start with20:56
*** AlexF has quit IRC20:57
dolphmhow long of a delay should be expected between submitting a patch to gerrit with a Closes-Bug, and seeing a comment from openstack-hudson revising the bug status? (i.e. https://bugs.launchpad.net/keystone/+bug/1245629/comments/11 )20:57
fungijog0: i believe so... looks like http://askubuntu.com/questions/85410/how-do-i-subscribe-to-a-tag-in-launchpad20:57
clarkbdolphm: not very long, minute or two probably20:57
clarkbdolphm: however there are cases where it will not spam the bug20:58
dolphmclarkb: how about this one https://bugs.launchpad.net/keystone/+bug/1276325 https://review.openstack.org/#/c/71096/20:58
fungidolphm: clarkb: it's broken https://launchpad.net/bugs/127618020:58
*** thuc has quit IRC20:58
fungifiled this morning, haven't had time to dig deeper yet20:58
dolphmoh there we go20:58
*** atiwari has joined #openstack-infra20:58
lifelessclarkb: lets start with that20:58
fungitrimmed those out of the gerrit error.log20:58
*** nati_ueno has quit IRC20:59
jog0fungi: neato that was easy20:59
*** sparkycollier has quit IRC21:00
fungijog0: i made elastic-recheck an official bug tag for openstack-ci so that should help wrangle those21:00
*** sparkycollier has joined #openstack-infra21:00
*** nati_ueno has joined #openstack-infra21:00
jog0fungi: any reason to use fix commited vs fix released21:01
fungijog0: if the project the corresponding patch went into does releases, then fix committed when the fix is committed, and fix released once the next release containing that commit is cut21:02
*** sdake has quit IRC21:02
fungiif there are never releases, then just straight to fix released21:02
clarkbfungi: we seem to have removed the --- doc separator from our yaml projects file21:03
*** dizquierdo has joined #openstack-infra21:03
fungiclarkb: right, i suspect https://launchpad.net/bugs/1275824 is also in the same boat21:04
clarkbfungi: ya21:05
jeblairfungi: what does 'elastic-recheck' tag mean?21:05
jeblairfungi: and how does it compare to 'gate-failure'?21:06
fungibits of jeepyb need to do a better job of autodetecting whether the config and project registry are split out or not. i originally had thought this would get done with config format versioning, but instead i guess we're just guessing based on file existence21:06
SergeyLukjanovclarkb, you've mentioned that -check slaves could be used for -nv, what's the difference? performance?21:06
jeblairfungi: oh, bugs on the elastic-recheck program?21:06
jeblairheh, i mean 'project'.  :)21:06
clarkbSergeyLukjanov: yes, we run extra clouds for those slaves21:07
clarkbSergeyLukjanov: and is the reason check vs gate is a thing in the job names21:07
*** gokrokve has joined #openstack-infra21:08
clarkbfungi: ya, I will propose the --- back into the file then we can work on fixing jeepyb. Just testing that yaml does the right thing if [0] is nil21:08
fungijeblair: elastic-recheck is for bugs in elastic recheck itself21:08
fungi"elastic recheck did this thing i did not expect"21:08
fungiet cetera21:08
* fungi nods21:08
*** resker has quit IRC21:08
*** zehicle_at_dell has quit IRC21:08
fungiif we get too many bugs about elastic-recheck reported against openstack-ci, we could always just add a new launchpad project (or storyboard group!) for them21:08
*** esker has joined #openstack-infra21:08
*** mrda_away is now known as mrda21:09
fungiclarkb: at least manage-projects will go toes up in that case. it looks to see if the ini file exists and then expects the yaml projects list to be in the first (only) document of the yaml file21:09
*** sparkycollier has quit IRC21:09
SergeyLukjanovclarkb, ok, got it, it should be used only for check/non-voting jobs?21:09
clarkbfungi: gah nevermind then, I will propose that fix to jeepyb instead21:09
jeblairfungi: ack. yep.21:09
clarkbSergeyLukjanov: yes21:09
*** gokrokve_ has joined #openstack-infra21:10
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Enable tempest/savanna gate tests  https://review.openstack.org/6806621:11
fungiclarkb: i've updated the bug with a quick redux of what manage-projects is doing to make that determination21:12
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Allow devstack-precise-check for non-voting jobs  https://review.openstack.org/7110321:13
SergeyLukjanovclarkb, the same for ironic ^^21:13
*** gokrokve has quit IRC21:13
*** oubiwann_ has quit IRC21:13
*** gokrokve_ has quit IRC21:14
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Remove tripleo check jobs from devstack-gate  https://review.openstack.org/7110421:15
jeblairclarkb, fungi: can you review/aprv ^ soon please?21:16
fungijeblair: already +2'd21:16
*** mattoliverau has joined #openstack-infra21:17
*** sdake has joined #openstack-infra21:17
*** sdake has quit IRC21:17
*** sdake has joined #openstack-infra21:17
*** dkliban has quit IRC21:17
fungii discussed the possibility with lifeless yesterday as well21:17
anteayamorning mattoliverau, jeblair was looking for you21:17
mattoliverauGood morning anteaya. was he21:17
mattoliveraumorning jeblair21:18
*** sdake has quit IRC21:19
*** lcestari has quit IRC21:19
*** sdake has joined #openstack-infra21:19
*** sdake has quit IRC21:19
*** sdake has joined #openstack-infra21:19
openstackgerritA change was merged to openstack-infra/config: Remove tripleo check jobs from devstack-gate  https://review.openstack.org/7110421:19
*** emagana has quit IRC21:20
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Enable tempest/savanna gate tests  https://review.openstack.org/6806621:20
jeblairfungi: yeah, it's problematic now, but would be even more so if we require check jobs for gate enqueing21:20
jeblairmattoliverau: hey! i have an idea for something you could work on if you have some time; fairly self-contained, mostly puppet, touches a bit more of the infra than your last change21:21
fungifull agreement21:21
*** sarob has quit IRC21:22
mattoliveraujeblair: great! what would you like?21:22
*** alexpilotti has quit IRC21:22
*** CaptTofu has joined #openstack-infra21:22
jeblairmattoliverau: there's this program called nodepool and it's running on its own server (called nodepool.o.o), which is pretty much a black box.  one of the things it does is build vm images for the machines that are used to run tests21:22
jeblairmattoliverau: it logs those image builds to files, but they are inaccessible without logging into the server21:23
jeblairmattoliverau: it would be cool if they were logged to a place that could be served by apache21:23
jeblairmattoliverau: so anyone could see the logs21:23
jeblairfungi, clarkb: i'm pretty sure nothing in the image build logs is security sensitive... right?21:23
fungii take it we're comfortable that exceptions in nodepool won't leak provider credentials?21:24
*** freyes has quit IRC21:24
fungiwe also get a full dump of the puppet apply stdout/stderr in the image log... could that include diffs of sensitive files?21:24
jeblairfungi: i believe that the image logs only log the output from running the scripts; and it could contain sensitive info if there were any, but an easier way to get sensitive info off a devstack host would be to write a change to 'cat' it and submit to gerrit21:25
*** thuc has joined #openstack-infra21:25
jeblairfungi: so for that purpose, we don't put any sensitive info on those hosts21:25
fungijeblair: very, very true21:26
* fungi is trying to think up any other devil's advocate concerns21:26
*** Ajaeger has quit IRC21:26
fungii'm skimming the image log and nothing is jumping out at me screaming "not on the web!"21:27
openstackgerritA change was merged to openstack-infra/elastic-recheck: Adding query for bug 1275500  https://review.openstack.org/7075621:27
mattoliveraujeblair: Ok, consider it done :) I'll take a look, but if you do find something sensitive just let me know, or deny the patch when it comes in ;)21:27
fungithough ick... tons of embedded carriage returns without newlines21:27
*** miqui has quit IRC21:28
fungiseems to be progress output from some tools which think they're in an interactive shell21:28
fungiwonder if that's a new side effect of the pty patch21:28
*** emagana has joined #openstack-infra21:28
jeblairmattoliverau: a follow-up patch would be to improve the logging for nodepool; that's mostly python hacking.  let me know if you're interested.21:29
clarkbfungi: so this is extra fun because projects.py seems to change the type of the projects list...21:29
mattoliveraujeblair: sure am :)21:29
mattoliveraujeblair: thanks for thinking of me21:29
clarkbfungi: they are in an interactive shell :)21:30
jeblairmattoliverau: btw, i think this is going to be really useful -- we want the infra to be as accessible to non-root people as possible, and this will help that a lot, including tripleo folks who are working on image bootstrapping right now21:30
clarkbbecause sudo21:30
*** amcrn has quit IRC21:30
*** sarob has joined #openstack-infra21:30
*** changbl has quit IRC21:31
*** jcooley_ has quit IRC21:31
*** DinaBelova is now known as DinaBelova_21:32
mattoliveraugreat, well I'd better get on it then :)21:34
*** emagana has quit IRC21:34
openstackgerritClark Boylan proposed a change to openstack-infra/jeepyb: Determine yaml format when loading yaml configs.  https://review.openstack.org/7111021:36
clarkbfungi: ^21:36
clarkbok /me goes back to poking bugs21:37
markmcclainthe gate reset… would it possible to promote https://review.openstack.org/#/c/71046/ to the top incase it happens again21:38
*** masayukig has joined #openstack-infra21:39
*** mestery has joined #openstack-infra21:39
clarkbmarkmcclain: sure, did we just reset now?21:39
*** mestery has quit IRC21:40
fungimarkmcclain: clarkb looks like the nova job is reporting its fail right this moment21:40
fungiand there it went21:40
fungiso we should be safe to promote21:40
*** mestery has joined #openstack-infra21:40
fungiclarkb: i've got the command line ready in an ssh session on zuul21:41
clarkbfungi: go for it21:41
fungijust want to avoid both doing it21:41
fungiit's at the top now21:41
fungimarkmcclain: ^21:41
clarkbjeblair: https://bugs.launchpad.net/openstack-ci/+bug/1172384 is that something we can close?21:42
mgagneIn gerrit, must all change topics be unique across the branch? Can't I provide the same topic twice if the first one is closed/merged?21:42
markmcclainfungi: thank you21:42
clarkbmgagne: you should be able to reuse topics21:42
*** masayukig has quit IRC21:43
jeblairclarkb: sure, there's more work there but i think we can close it21:43
jeblairclarkb: (i closed it)21:43
clarkbjeblair: ya definitely more work, but it is flagging things today21:43
annegentleoh yeah, so moodle. where / what domain/ subdomain?21:44
mgagneclarkb: thanks, I'll check what's going on with my issue21:44
annegentleoh, sorry fungi already answered that we don't really know yet :)21:44
clarkbhttp://logs.openstack.org/10/71110/1/check/gate-jeepyb-pep8/2536094/console.html that has me stumped.21:44
*** dkliban_afk has joined #openstack-infra21:44
jeblairannegentle: can you catch me up on your question?  i've been away21:45
mgagneclarkb: I reverted a backport and I'm trying to cherry-pick it again. Gerrit won't let me as it would submit the same Change-Id twice21:45
*** dprince has quit IRC21:45
*** sarob has quit IRC21:45
clarkbmgagne: change ids must be different21:45
clarkbtopics don't need to be21:45
mgagneclarkb: ok, so I change to generate a new one right?21:46
clarkbmgagne: yup21:46
mgagneclarkb: ok, thanks21:46
fungijeblair: dguitarbite added a last-minute agenda item about moodle app integration for the training manuals \to the meeting today but wasn't around at the end and we wouldn't have had time to discuss it21:47
fungiapparently sarob was also mentioned on it but currently isn't around either21:48
fungino idea what "Request for Moodle App Integration to infra for Training-Manuals" means (and i'm doing my best not to ponder the possibilities)21:49
clarkbI see the problem with jeepyb pep8 checks. pbr in new hacking conflicts with pbr elsewhere. I am updating requirements for jeepyb21:50
*** gyee has quit IRC21:50
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1275256  https://review.openstack.org/7111821:51
openstackgerritClark Boylan proposed a change to openstack-infra/jeepyb: Fix pbr and hacking requirements.  https://review.openstack.org/7111921:51
openstackgerritClark Boylan proposed a change to openstack-infra/jeepyb: Determine yaml format when loading yaml configs.  https://review.openstack.org/7111021:51
clarkbI think that should get us sorted21:51
annegentlejeblair: fungi: clarkb: yeah adding moodle seems like a lot of work but want to hear what they need it for. probably quizzes? class graduation tracking? Not sure what other requirements they have?21:52
clarkbannegentle: but why is that a thing we would be involved in?21:53
annegentleclarkb: because they'd need it stood up, domain registered, etc. I guess21:53
fungiclarkb: maybe publishing stuff to moodle21:53
clarkbwait is the foundation certifying people direclty?21:53
annegentlethey're not really going through me for this21:53
annegentleas far as requirement gathering21:53
fungiclarkb: i don't think so, no21:53
annegentleclarkb: no idea21:53
*** jhesketh has joined #openstack-infra21:54
*** apevec has quit IRC21:54
fungiclarkb: i believe the foundation is helping the community settle on standards for certification tests21:54
anteayamorning jhesketh21:54
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1265495  https://review.openstack.org/7055621:55
mriedemjog0: ^21:55
*** sdake has quit IRC21:55
fungiclarkb: but as far as i know we're not going to be endorsing any one particular test provider/company and want the materials to be free as in speech21:55
jog0mriedem: mriedem I am having trouble  keeping up with you now21:56
clarkbwoot that fixed jeepyb tests21:56
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1275256  https://review.openstack.org/7111821:56
mriedemjog0: that's ok, i'm out of here in about 35 minutes anyway :)21:56
clarkbnow I can actually move on to more bug stuff21:56
*** nicedice has quit IRC21:56
*** nicedice has joined #openstack-infra21:57
jog0mriedem: heh, once your patches land hopefully our classifcation rate will jump http://status.openstack.org/elastic-recheck/data/uncategorized.html21:59
*** hashar has quit IRC21:59
mriedemoh yeah, that21:59
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1265495  https://review.openstack.org/7055622:01
*** masayukig has joined #openstack-infra22:01
*** markmc has quit IRC22:01
*** UtahDave has quit IRC22:03
*** e0ne has quit IRC22:04
*** david-lyle has quit IRC22:04
clarkbI am closing https://bugs.launchpad.net/openstack-ci/+bug/1172443 beause infra projects use testr and there is a bug with the other projects in it for tracking progress across most projects22:04
*** jasondotstar has quit IRC22:05
*** masayukig has quit IRC22:05
*** whoops has joined #openstack-infra22:08
clarkbfungi: https://review.openstack.org/#/c/53245/1 :(22:09
clarkbwe can close that bug but it got auto abandoned during summit and missed22:10
clarkbjeblair: ^ if you guys want to review that really quick it would be awesome. I am checking it for merge conflicts now22:10
*** jcooley_ has joined #openstack-infra22:10
clarkbit conflicts, fixing now22:12
mordredclarkb: I just -1'd 7115922:12
mordredclarkb: I just -1'd 71119  - there is no 7115922:12
mordredclarkb: I'm happy to just fix it instead though22:12
clarkbmordred: go for it22:13
fungiyeah, it'll need a rebase22:13
openstackgerritClark Boylan proposed a change to openstack-infra/config: Install graphiz on Jenkins slaves.  https://review.openstack.org/5324522:13
mordredclarkb: k. /me wants to at least attempt to be useful22:13
jeblairclarkb: i -1 53245 due to a typo22:13
jeblairclarkb: (which is also in the commit msg, btw)22:13
jeblairclarkb: but i did not -1 it because of the commit msg22:14
fungii had totally forgotten pbr needed to be first in the reqs lists as well22:14
clarkbjeblair: good catch22:14
clarkbjeblair: actually something else may have fixed that in the meantime22:15
clarkbauto abandoner makes me not happy right now22:15
clarkbjeblair: I will abandon 53245 and close the bug. A different change added graphviz to the slaves22:15
*** sdake has joined #openstack-infra22:16
fungii'm completely in favor of turning it off until we have new gerrit, and then looking into auto-wip22:16
*** jcooley_ has quit IRC22:16
*** _david_ has quit IRC22:16
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Determine yaml format when loading yaml configs.  https://review.openstack.org/7111022:17
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Fix pbr and hacking requirements.  https://review.openstack.org/7111922:17
mordredclarkb: I'm good with both of those now22:17
*** jcooley_ has joined #openstack-infra22:17
clarkbmordred: thanks22:17
anteayamordred: I'm helping you with the manage projects script22:18
anteayawhen are you available to work on it?22:18
* anteaya gets out her butterfly net22:19
mordredanteaya: heh.consider me unavailable this week22:20
anteayaso that means next Monday22:20
anteayadid I hear that correctly?22:20
*** CaptTofu has quit IRC22:23
clarkbmordred: https://bugs.launchpad.net/openstack-ci/+bug/1179007 how are we doing there?22:23
*** mestery has quit IRC22:24
mordredclarkb: I'd need to go explicitly verify things22:25
openstackgerritA change was merged to openstack-infra/jeepyb: Fix pbr and hacking requirements.  https://review.openstack.org/7111922:25
*** amcrn has joined #openstack-infra22:25
*** markmcclain has quit IRC22:26
clarkbmordred: and https://bugs.launchpad.net/openstack-ci/+bug/1179008 goes along with that.22:27
clarkbhow do people think about marking https://bugs.launchpad.net/openstack-ci/+bug/1182231 won't fix? I believe our clouds will have heat for us in the enearish future which is far better tha nrunning our own22:28
clarkbreed: https://bugs.launchpad.net/openstack-ci/+bug/1182690 is there an update for that? I don't know paul's nick22:28
jeblairzaro: can you look at https://bugs.launchpad.net/openstack-ci/+bug/1276293 ?22:28
jeblairzaro: i think i saw something related to that in review recently, but i don't recall the details22:29
clarkbreed: iirc the italian stuff happened? can we close that bug?22:29
zarojeblair: sure.  assigned to myself22:30
jeblairzaro: to be clear, i wasn't necessarily asking you to fix it, just thought you might know something about it22:30
zarojeblair: ohh.  then no.  i haven't noticed that it was failing.  i'll double check off master.22:31
lifelessclarkb: so it looks to me like it is aggressively cleaning up22:32
lifelessclarkb: can you also pull out 'Deleting node id:' and 'Exception deleting node id:' lines for me22:33
jeblairzaro: https://review.openstack.org/#/c/63579/ is the change i was thinking of22:33
clarkblifeless: yup22:33
zarojeblair: epy27 and epy26 both worked fine.22:33
lifelessclarkb: with the starting/finished periodic lines22:33
*** markmcclain has joined #openstack-infra22:33
fungisince that's the last untriaged bug, i'm going to take a quick dinner break before my next meeting22:34
jeblairfungi, clarkb, lifeless: i wonder if the full nodepool logs are safe to serve publicly, or is there a chance they could have creds?22:34
lifelessclarkb: from my quick scan there is at most 120s latency from delete to delete attempt [worst case] and I'd expect average of 45-50s22:34
zarojeblair: hmm, wondering if it's because the jjb cache.  maybe works on jenkins because jenkins build always has a fresh ENV.22:36
zaroi'll do a little test.22:36
*** dims has quit IRC22:39
clarkbmordred: https://bugs.launchpad.net/openstack-ci/+bug/122470522:40
*** dkranz has quit IRC22:41
lifelessjeblair: clarkb: so we're deleting nodes after ~12 seconds22:41
mordredclarkb: we need to get wheels uploading to pypi before we can fix that22:41
clarkbmordred: speaking of wheels I could not reproduce the wheel failures we saw when we enabled wheels22:42
*** mriedem has quit IRC22:42
clarkbalso I hate being mean but I think we should unpin tox22:42
lifelessjeblair: clarkb: whats worrying though is that some of them are much longer22:42
clarkbthere is no traction to fix that right now22:43
lifelesslike .2 of an hour22:43
clarkblifeless: yeah clearly there are nodes taking longer, possibly due to the provider side of things22:43
mordredclarkb: that's excellent22:43
mordredclarkb: what breaks when we do?22:43
clarkbmordred: posargs apparently? but that shouldn't break and pip does not allow external with new tox22:43
jeblairlifeless: yes, some providers take a long time; that's less worrying since it's previously observed behavior; it's definitely worring from the perspective of "openstack should be good"22:44
mordredclarkb: I'm in favor of unpinning - I looked at the new docs and couldn't figure out how our usages would be broken22:44
clarkbmordred: the virtualenv and pip upgrades are the biggest headache22:44
lifelessclarkb: jeblair: no, you misunderstand me22:44
clarkbmordred: it will break and the gate will grind to a halt I guaruntee it22:45
lifelessclarkb: jeblair: in that log file, the *first* occurence of the node id in the delete routine is when it's been in that state for many minutes.22:45
mordredclarkb: then we shouuld maybe not do it yet22:45
clarkbmordred: I would agree with you if we had fixed sqlalchemy by now22:45
lifelessUhm, I think I need to see what provider they are on too; prepping a patch for that22:46
*** mfer has quit IRC22:46
jeblairlifeless: oh yes i misunderstood, sorry.22:46
anteayamattoliverau: oh, you probably haven't participated in a bug day yet, today is bug day22:47
*** eharney has quit IRC22:47
anteayamattoliverau: during bug day someone (pleia2) creates an etherpad for the event and populates it with our bugs for ci: https://etherpad.openstack.org/p/cibugreview-february201422:47
dhellmannanteaya: is today a good day to ask about importing some git repos, or should I wait until tomorrow?22:47
*** jhesketh__ has joined #openstack-infra22:47
anteayadhellmann: asking is always good22:47
openstackgerritlifeless proposed a change to openstack-infra/nodepool: Also log provider name when debugging deletes.  https://review.openstack.org/7113722:48
anteayaexpecting them to be auto managed won't happen today, or apparently the rest of this week22:48
anteayabut right now mordred is hand creating them22:48
lifelessjeblair: if I can get ^ [untested] on prod and some data, we can see if its slow cleanup loops (in which case optimising that will be the next step) or if its something unexpected.22:48
anteayaso perhaps if a patch is merged, he can do that for you22:48
clarkbfungi: was there any additional breakage from new tox? I wasn't awake for most of it22:48
dhellmannanteaya: hmm, does mordred still have a fever? ;-)22:48
anteayamordred: do you still have a fever?22:49
dhellmannif you guys are working on tools, these repos can wait22:49
anteayawell you guys right now is me22:49
lifelessadding in an instant-try-delete-now will a) add race conditions with sqla you wanted to avoid and b) increase API pressure which will slow everything down by ratelimiting use in a more random fashion22:49
*** gokrokve has joined #openstack-infra22:49
clarkbfungi: I thought https://bugs.launchpad.net/openstack-ci/+bug/1263160 required non open software22:49
anteayaand I am slow at the best of times, and trying to come down off of neutron and refocus on -infra again22:49
dhellmannanteaya: I have a patch up and dims has another one, so they won't be forgotten22:49
anteayadhellmann: what is your timeline for the repos?22:49
lifelessthe whole point of grouping this up is to ensure we scan across all the deletes at once so that we don't block other threads (by the ratelimiting aspect) and thus starve deletes that could happen quickly, perhaps.22:50
anteayaif mordred gets them in by hand, that is good, if I beat him to it, even better22:50
dhellmannanteaya: soon is nice, no particular rush, yet22:50
dhellmannanteaya: I'll add you to the reviews22:50
anteayadhellmann: awesome thanaks22:51
zarojeblair: i could not repro, i updated the bug to get more info.  have assigned it to me so i'll keep an eye out.22:51
clarkbmordred: do you remember how we enabled wheels back when we neabled it? where did the pip flag to use wheels get enabled?22:52
clarkbmordred: I can only find the change to select-mirror to list distro specific mirror locations22:52
clarkbmordred: was that enabled in pbr?22:53
*** dims_ has joined #openstack-infra22:55
*** pcrews has quit IRC22:55
*** weshay has quit IRC22:55
*** thuc has quit IRC22:55
anteayamattoliverau: so we go through them, id the ones we are working on, update the etherpad and the bug report so the status is current, and those that are able to, triage bugs22:55
mattoliverauanteaya: no I have not. Cool so there is a big list of bugs.22:56
anteayamattoliverau: and we discuss the ones that need to be discussed and track down anyone we need to hear from22:56
*** thuc has joined #openstack-infra22:56
mattoliverauahh i see22:56
anteayamattoliverau: we have a bug day once per milestone22:56
anteayausually just after the milestone cut22:56
anteayaanyone can organize, pleia2 has been driving this ship for a bit, and doing a great job at it22:57
*** mfer has joined #openstack-infra22:57
*** mfer has quit IRC22:57
*** pcrews has joined #openstack-infra22:58
reedclarkb, the bug is still valid22:59
reedclarkb, paul's nick is rockstar22:59
reedclarkb, going to bed now, I'm in Europe22:59
*** thuc has quit IRC23:00
*** dizquierdo has quit IRC23:01
anteayamattoliverau: so next time we have a bug day chat with pleia2 about the date, she posts to the infra ml: http://lists.openstack.org/pipermail/openstack-infra/2014-January/000734.html23:01
*** harlowja is now known as harlowja_away23:01
anteayareed sleep well23:02
mattoliverauanteaya: ok, thanks for the info. got a team meeting, bbs.23:03
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/config: Create git repo and jenkins jobs for oslo.vmware  https://review.openstack.org/7076123:03
*** burt1 has quit IRC23:04
anteayaI'm about to make an aweful suggestion23:04
clarkbmordred: ya I am configuring my local pip.conf to use the wheel locations and I cannot get tox 1.6.1 to fail like it did back then23:05
anteayawhat would happen if we moved the infra meeting to 2200 utc on Tuesdays?23:05
anteayait would no longer conflict wtih tripleo23:05
clarkbmordred: also because we use deps in tox we won't actuall use wheels until pip 1.523:05
anteayaand it hopefully would allow mattoliverau to attend23:05
anteayaif he is infra, and he is, he should at least have a fighting chance to attend team meetings23:06
clarkbanteaya: does it conflict with anything else?23:06
*** krotscheck_sick has quit IRC23:06
anteayawell the -meeting channel is empty after project meeting23:07
anteayalet me check teh schedule23:07
clarkbanteaya: some meetings happen every other week23:07
clarkbI thought tuesday afternoons are completely booked23:07
fungiclarkb: on 1263160, possibly so... i was just breaking out individual bugs for the remaining non-nova servers23:07
anteayaclarkb: ah, let me look with an eye for alternate weeks23:08
clarkbfungi: mordred: I am trying really hard to make wheels fail for me locally running tox. No luck so far23:08
*** branen_ has quit IRC23:08
anteayaI just control-f'd through the meetings wikipage filtering on Tuesday23:10
clarkbI am guessing mordred went to sleep23:10
anteayanoone has 2200 utc that I saw23:11
clarkbanteaya: cool23:11
*** jgrimm_ has quit IRC23:11
*** masayukig has joined #openstack-infra23:11
*** prad_ has quit IRC23:12
anteayaI'll put something on next week's agenda about changing the meeting time23:12
clarkbfungi: are the only issues with new tox that virtualenv and pip are upgraded as well?23:13
anteayaI wonder if I should but a placeholder somewhere that we are considering this time slot23:13
clarkbfungi: I can't find anything in new tox itself that should actually be a problem23:13
anteayaso it doesn't get filled while we are discussing it23:13
*** harlowja_away is now known as harlowja23:13
clarkbthe posargs thing is weird I guess I debug that first23:14
*** dstanek has quit IRC23:14
*** jecarey has quit IRC23:14
clarkboh https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror that is amazing23:15
clarkbnow why does testr-args require the =23:15
*** masayukig has quit IRC23:15
mordredclarkb: I think the other problem with wheels may have had to do with the thing that was why pip 1.5 broke23:15
*** dcramer_ has quit IRC23:16
clarkbmordred: pip 1.5 broke beacuse it disabled externally hosted reqs23:16
clarkb1.4 didn't23:16
mordredthere was another thing too23:16
mordredwhich had to do witha bug in wheels and bootstrapping into setuptools wheels23:16
*** mestery has joined #openstack-infra23:17
clarkbmordred: where is the python setup.py test entrypoint installed? is that pbr?23:17
*** mestery has quit IRC23:17
mordredI _believe_ that the reason I was baffled by the wheels work before was because it was hitting that bug, but I thought it was a bug in our wheels work23:17
mordredwhile I _think_ it was an upstream bug23:17
mordredclarkb: yes. pbr23:17
*** mestery has joined #openstack-infra23:17
*** sarob has joined #openstack-infra23:18
clarkbmordred: why is the option testr-args= there?23:18
mordredit's not an entrypoint though really - when pbr is running setup.py, it injects commands more directly into setuptools23:18
mordredbecause ... uhm ...23:18
clarkbthe = is what breaks tox 1.723:18
*** zehicle_at_dell has joined #openstack-infra23:18
mordredwow. really?23:18
clarkbmordred: yes https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror23:18
clarkbwell I mean if we could put whitespace after the = it would be fine but that is just awkward23:18
clarkbinstead I think we want --testr-args '{posargs}'23:19
fungiclarkb: catching up... the issue we ran into with new tox is that it requires newer virtualenv/pip than we were ready for23:19
anteayaI have added it to the agenda and edited in a placeholder as well, which I will free up if we decide against the proposal23:19
mordredI think it's there so taht we could do something fancy ...23:20
clarkbmordred: fungi: I think untangling both wheels and tox is very important and in the future I think we should consider leaving things unpinned for longer23:20
*** thuc_ has joined #openstack-infra23:21
clarkbmordred: does -t work? is that what the 't' there means?23:21
fungiclarkb: i'm cool with tox unpinning... that last one cropped up while only i was around so i didn't have time to dig into that and everything else that was going on23:21
clarkbfungi: right I think we leave it unpinned so that others dig into it :)23:21
anteayamattoliverau: would Tuesdays at 2200 utc allow you to participate in an -infra team meeting?23:22
mordredclarkb: oh  - maybe?23:22
fungiwe'll need to unpin tox/virtualenv/pip together though23:22
anteayaI guess I should check with him23:22
mordredclarkb: yes. -t shoudl work23:22
clarkbmordred: so I think we remove the = from pbr and/or switch to using -t23:23
clarkbbut none of that matters until pip and virtualenv work23:23
mattoliverauanteaya: yeah, that works well for me!23:23
*** thomasem has quit IRC23:24
anteayawell so far it is proposed23:24
mordredclarkb: the = is part of setuptools parsing - we can't fix it23:24
mordredclarkb: so - I think that we should move to -t23:24
mordredand before that, we should do wheel uploads23:24
anteayamattoliverau: let's see what others have to say when it is discussed23:24
mordredthen fix the proejct that's still using direct urls23:24
mordredthen do pbr/pip/virtualenv23:25
anteayabut you will learn more the more you can participate23:25
mordredwe can do the -t fix everywhere and it shoudl work old and new, btw23:25
mattoliverauanteaya: yeah fair enough :)23:25
*** markmcclain has quit IRC23:25
anteayamattoliverau: :D23:25
pleia2zaro: want to flip the switch on bug 1276293 to "Incomplete" (that's the last of our "New" bugs!)23:25
*** rcleere has quit IRC23:26
fungimordred: i think we needed oslo.config to have an initial release to pypi too23:26
fungino, wasn't oslo.config23:27
*** sarob_ has joined #openstack-infra23:27
*** rfolco has joined #openstack-infra23:27
clarkbmordred: right so, I suggest unpinning beacuse otherwise none of that stuff will happen before icehouse23:27
fungioslo.messaging, but it seems to have released finally23:27
mordredoh - wait23:27
clarkbit is mean, but it is reality23:27
mordredthey've already released23:27
mordredwhich means can change the reqs23:28
fungiit's an alpha release, but at least it's uploaded23:28
mordredthat's all we need23:28
*** miqui has joined #openstack-infra23:28
fungiso yes, we need reqs changed and then projects can fix themselves to match23:28
*** sarob has quit IRC23:28
*** mriedem has joined #openstack-infra23:29
clarkbweren't there also some straggler reqs externally hosted that were not oslo.*?23:29
*** mriedem has quit IRC23:29
clarkbthe problem with this is these are all project things, and we end up fixing them because we pinned23:30
fungithere are, but the projects using them hopefully have their tox.ini files patched now23:30
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Map failed jobs to bugs in gerrit comment  https://review.openstack.org/7114823:30
fungii fixed it in pypi-mirror, and only found 3 total remaining23:30
clarkbfungi: that isn't too bad23:30
*** mriedem has joined #openstack-infra23:30
fungibut there are a handful of others not in global requirements for non-integrated projects too23:30
fungiwe may discover more of those once it's allowed to be broken longer23:31
fungisomeone might want to draft a quick "we're about to open the gates of the abyss and let the demons loose" e-mail to the dev list23:31
*** flaper87 is now known as flaper87|afk23:31
lifelessanteaya: devananda: https://review.openstack.org/#/c/71046/2 has merged23:31
zaropleia2: done23:31
lifelessthis should make neutron isolated happy with no file injection23:31
fungibecause i'm quite positive all integrated gating will be flat out broken while everyone gets caught up23:31
lifelessassuming no further hidden bugs23:32
clarkbfungi: that is what I am thinking, something along the lines of everyone using python setup.py test should s/testr-args=/-t / and be prepared to fix your external project dependencies23:32
clarkbfungi: we can continue to do the patch bomb thing but I think projects have become to reliant on that23:32
clarkbmordred shouldn't have to write 60 patches every cycle to fix silly project things23:32
jog0clarkb: if you have some time today https://review.openstack.org/#/q/status:open+branch:master+topic:graph-gate,n,z23:32
jog0if not, ignore me23:32
anteayalifeless: thank you, yes markmcclain is testing a patch to see how happy neutron is now23:33
fungiclarkb: the last round of those i did for some tox.ini fixup, half of them merged and the other half got randomly downvoted for various reasons and eventually abandoned because i didn't have time to address them all23:33
clarkbmordred: or do you prefer to do patch bombs?23:33
*** hogepodge has joined #openstack-infra23:33
mordredclarkb: I prefer to not do patch bombs23:33
clarkbfungi: right, and imo infra exists to run the tests. Nova being incompatible with tox pip and virtualenv isn't an infra problem23:34
mordredclarkb: I consider icehouse pretty successful because we've done almost no dev on pbr!23:34
anteayatranslations from transifex should be fairly benign: https://review.openstack.org/#/c/66501/23:34
clarkbfungi: we can still run the tests, they just fail :)23:34
fungimordred: stale pbr! the best kind!23:34
*** atiwari has quit IRC23:35
*** rfolco has quit IRC23:36
*** yamahata has quit IRC23:38
*** thuc_ has quit IRC23:38
*** yamahata has joined #openstack-infra23:40
*** andreaf has joined #openstack-infra23:42
clarkbI am drafting a ml thread starter now23:42
mordredclarkb: I just +2'd https://review.openstack.org/#/c/66072/ - but did not +A23:42
mordredclarkb: becase I'm going to sleep - and I thought that would be rude, but I think we're all generally in favor of it23:43
mordredfungi: ^^23:43
clarkbmordred: np before you go23:43
mordredclarkb: yah?23:43
clarkbmordred: do you think we should set a date for unpinning and give projects a list of things to do?23:43
*** andreaf has quit IRC23:43
clarkbor do we need to be nicer and unpin after these thigns have been done?23:43
anteayaclarkb: set a date23:43
anteayaotherwise you will wait for ever23:44
mordredbut be nice about it23:44
anteayaa nice date23:44
fungiset a date mainly so that sdague knows to be on vacation23:44
anteayaha ha ha23:44
anteayaor at least wake up after you, fungi23:44
fungii don't think he's capable of sleeping past 5am23:44
anteayasalv-orlando thought he lived in the middle of teh atlantic23:45
anteayabased upon his start work time23:45
fungisome day, i will live in the middle of the atlantic, but even then he'll probably still be getting up earlier than me23:46
*** thuc has joined #openstack-infra23:46
anteayabut you can tell us good stories23:46
anteayaof living on the waves23:46
StevenKThe Debian LDAP database contains lat/long for each developer -- years ago, a bunch of developers got mailed saying that the co-ordinates had been plotted, and they appeared to live in an ocean.23:47
StevenKOne of my workmates at the time was one of the lucky few to get such an e-mail, and I told him he should replied that he was living on a boat in the middle of the Pacific23:48
*** atiwari has joined #openstack-infra23:48
*** thuc_ has joined #openstack-infra23:53
anteayaI could live on a boat too23:53
anteayaAtlantic, Pacific, not picky just water23:53
StevenKanteaya: The Med is now feeling left out :-P23:54
*** markmcclain has joined #openstack-infra23:54
*** yamahata has quit IRC23:54
anteayayeah, I seem to manage that23:54
anteayaleaving someone/thing out23:54
clarkbgah the '{posargs}' replaces the quotes23:54
anteayanot sure I could do the Med23:54
clarkbhow are we supposed to use this thing?23:54
anteayatoo many problems from people trying to flee from the south23:55
clarkbactually tox is on crack23:55
fungiclarkb: that sounds mildly unsafe. though maybe i'm missing relevant context that makes it not a shell vulnerability waiting to happen23:55
*** thuc has quit IRC23:56
clarkbfungi: it means that -t {posargs} won't do what we want it23:56
*** whoops has quit IRC23:56
clarkbwe will get -t foo bar baz instead of -t 'foo bar baz'23:56
anteayaclarkb: careful, tox might become mayor ford and refuse to leave office after stripped of all power23:57
anteayayou don't know any canadians, but I will put money on you know who mayor ford is23:57
*** branen has joined #openstack-infra23:57
*** weshay has joined #openstack-infra23:58
clarkbI don't know how we can work around this23:58
clarkbI am going to comment on the bug upstream23:59
clarkbbut this is seriously insane23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!