Wednesday, 2014-11-26

*** wenlock has joined #openstack-infra00:00
asselini.e fixed the conflict00:00
clarkbI wonder if we can stop using that db connection entirely in manage projects00:00
clarkboh no because races nevermind00:00
clarkbanyways +200:00
*** banix has quit IRC00:00
swestonasselin: ok, glad you found it :-)00:00
fungiclarkb: we could switch it to sqlalchemy at some point which does a ping(True) behind the scenes if we got really bored00:01
*** doug-fish has quit IRC00:01
*** doug-fish has joined #openstack-infra00:02
*** Guest8043 has quit IRC00:02
*** shashankhegde has joined #openstack-infra00:04
*** teran has joined #openstack-infra00:04
clarkbfungi: ya it started as a hack and probably should consider less hacky methods00:07
*** teran_ has quit IRC00:07
*** achanda has quit IRC00:08
clarkbbut we haven't heard of any psycopg2 ping timeouts so it may be a non issue there00:09
clarkbwould potentially explain why they don't have a similar method too00:09
clarkbAJaeger: when I wake up tomorrow I would like to try and get python26 changes sorted before I start the long weekend00:13
clarkbAJaeger: I will do my best to start the day early :)00:13
clarkbthat way you are still around :)00:13
*** JayJ has quit IRC00:14
asselinsweston, interesting, I had to do this again, so something must be installing it and causing the conflict: sudo apt-get remove python-jenkins00:14
*** wenlock has quit IRC00:16
fungiclarkb: i'll be around to help with those for the first half of my day if needed00:16
*** emagana has joined #openstack-infra00:16
clarkbfungi: mostly I think we just need to rebase and rereview so that stuff is ready for end of weekend00:16
clarkbsince AJaeger hsa changes written iirc00:16
fungiand sent the reminder to the ml too00:17
swestonasselin: i'll bet you a million dollars it's puppet ;-)00:18
*** ChuckC_ has quit IRC00:18
JayFI have a change that's been up since Monday that has never had jenkins run on it, and although it's +A'd, doesn't appear to be merging00:19
JayF is the one I'm talking about00:19
fungiJayF: leave a recheck comment on it00:20
clarkbMonday the day of doom00:20
*** Sukhdev has joined #openstack-infra00:20
fungiJayF: it was likely uploaded while zuul was offline so it missed the patchset uploaded event00:20
swestonasselin: it's in the job_builder.pp file00:20
JayFCool. I avoid spamming a recheck in unless I know it's the real solution to the problem :)00:20
fungiJayF: recheck will cause it to run through check and, if jobs succeed, will automatically enqueue it into the gate since it's already approved00:20
asselinsweston, notice: /Stage[main]/Jenkins::Job_builder/Package[python-jenkins]/ensure: ensure changed 'purged' to 'present'00:21
hasharhello there!00:22
openstackgerritSpencer Krum proposed openstack-infra/system-config: Remove logic from puppet.conf
clarkbhashar: hello00:22
hasharI think i remember a spec / doc that proposed to run tests with the latest version of dependencies provided by Pypi , or at least test out with some libs different than the global requirements.  Does it sounds familiar to anyone?00:22
*** emagana has quit IRC00:23
*** HACKING-FACEBOOK has quit IRC00:23
swestonasselin: should be here: /opt/config/environment/modules/jenkins/manifests/job_builder.pp00:23
clarkbhashar: I seem to remeber the suggsetion sdague made about ways to get around upstreams beraking us00:23
*** emagana has joined #openstack-infra00:23
clarkbhashar: and locking down global requirements then running a job like you describe was one of the ideas thrown out there for doing that00:23
asselinsweston, don't have that00:24
clarkbhashar: but I don't think it got as far as a spec00:24
hasharclarkb: agh I have been searching for an half an hour and was wondering whether I dreamed about it :]00:24
JayFfungi: how long will it take to queue the chceks? I'm used to seeing a message pretty quickly after posting a recheck comment00:24
swestonasselin: you must have it somewhere00:25
*** ChuckC_ has joined #openstack-infra00:25
asselinsweston, looking00:25
clarkbhashar: if ou dig through the openstack-dev mail list you should find the thread00:25
clarkbhashar: it would be from the last couple weeks00:25
swestonasselin: puppet is hella annoying, but he's not magic00:25
*** sputnik13 has quit IRC00:26
*** achanda has joined #openstack-infra00:26
hasharclarkb: thanks, will dig in and if all fails will ask sean :]00:26
*** sputnik13 has joined #openstack-infra00:26
fungiJayF: the zuul status page shows it running in the check pipeline currently with 1 hour 18 minutes to go before it finishes and possibly enqueues into the gate. at that point zuul will update the change with comments about the check results and that it's being chucked into the gate pipeline (if all the voting ones pass)00:26
JayFfungi: did the "Check jobs starting..." gerrit messages get nuked then?00:27
*** emagana has quit IRC00:28
asselinsweston, but I see it in the manifest:
fungiJayF: yes, it was necessary noise we endured while running gerrit 2.4 but once we upgraded to 2.8 we were able to stop zuul from needing to leave that comment00:28
JayFfungi: aha, that's where my confusion was coming from then, I was looking for that message. I guess this always just happens in the background -- that I don't usually have to think about it is a testament to you folk :P00:29
fungiJayF: yeah the theory was that you should be able to just assume it happens automatically. except you know when we have a catastrophe and then we sort of undermine that expectation. hopefully that will be less and less frequent as time goes on00:30
*** dims_ has quit IRC00:30
swestonasselin: find / -name job_builder.pp -print00:31
swestonasselin: does that do anything for you00:31
asselinsweston, yes, but when I find it, then what?00:32
*** HACKING-FACEBOOK has joined #openstack-infra00:32
*** doug-fish has left #openstack-infra00:32
swestonthere will be a line in there that tells puppet to install the package,00:33
asselinsweston, yes, but why does it cause the conflict? <asselin> sweston, but I see it in the manifest:
asselinsweston, so the manifest on master is out of sync with the JJB on master?00:35
swestonasselin: If I remember correctly, it installs a dependency that overrides a library that jjb needs for pyaml parsing00:35
*** emagana has joined #openstack-infra00:35
*** emagana has quit IRC00:35
*** emagana has joined #openstack-infra00:35
swestonasselin: I might just remove it, install jenkins from source00:35
*** achanda has quit IRC00:37
asselinsweston, but it is installed from the source00:38
swestonasselin: the os packages often lag way being the current development pip bundles, it has always been a hot spot for issues00:38
asselinsweston, no, it's installed here from source:
asselinsweston, I think it's a real bug00:39
swestonasselin: jjb is, yes, but what about Jenkins?00:39
*** doug-fish1 has joined #openstack-infra00:39
asselinsweston, jenkins is fine :)00:39
*** julim has joined #openstack-infra00:40
asselinsweston, jenkins is java00:40
swestonasselin: didn't you say you fixed the jjb issue when you removed the python-jenkins package?00:41
asselinsweston, yes00:42
*** HACKING-FACEBOOK has quit IRC00:42
*** doug-fish2 has joined #openstack-infra00:42
*** doug-fish2 has quit IRC00:43
*** doug-fish1 has quit IRC00:44
swestonasselin: then I must not be following you, seems like the logical solution would be to remove it from the puppet manifest that is installing it00:44
*** yamamoto has joined #openstack-infra00:44
asselinsweston, yes, I am trying that00:44
swestonasselin: ok, then I am following you, carry on :-)00:45
*** koolhead17 has joined #openstack-infra00:45
asselinsweston, so when I delete it, the problem is fixed.00:46
*** erikwilson has joined #openstack-infra00:46
asselinsweston, I'll propose the solution, but I don't think that's correct still...because there's  a comment saying other packages may need it as well...00:46
*** erikwilson has left #openstack-infra00:47
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova libvirt bug 1372670
*** ZZelle_ has quit IRC00:47
uvirtbotLaunchpad bug 1372670 in nova "libvirtError: operation failed: cannot read cputime for domain" [High,Confirmed]
swestonasselin: let me dig through some logs, before you propose it00:48
asselinsweston, ok00:48
*** Ryan_Lane has joined #openstack-infra00:49
*** davideagnello has quit IRC00:50
*** alexpilotti has quit IRC00:51
*** david-lyle is now known as david-lyle_afk00:51
openstackgerritRamy Asselin proposed openstack-infra/puppet-jenkins: Update to direct storyboard link
*** NobodyCam has quit IRC00:56
*** mpavlase has quit IRC00:56
*** baoli has joined #openstack-infra00:56
*** achanda has joined #openstack-infra00:57
*** emagana has quit IRC00:57
*** NobodyCam has joined #openstack-infra00:58
*** juice has quit IRC00:58
*** shashankhegde has quit IRC00:58
*** emagana has joined #openstack-infra00:58
*** yamahata has joined #openstack-infra01:01
*** juice has joined #openstack-infra01:01
*** emagana has quit IRC01:02
asselinsweston, going home. I'll just submit it and abandon it later.01:03
openstackgerritRamy Asselin proposed openstack-infra/puppet-jenkins: Puppet-Jenkins not compatible with JJB
swestonasselin: ok, I am not finding it in my logs, too long ago01:04
asselinshould really be a bug01:05
*** stevemar has quit IRC01:05
*** davideagnello has joined #openstack-infra01:05
asselinbut I gotta go. good night01:05
swestonasselin: we might be able to isolate it in the future.  i'll keep my eye out for dependencies.  have a good night01:06
*** gokrokve has quit IRC01:06
zulmordred:  what did i  do?01:06
*** shashankhegde has joined #openstack-infra01:06
*** hashar has quit IRC01:07
*** salv-orlando has quit IRC01:07
*** teran_ has joined #openstack-infra01:08
*** yamamoto has quit IRC01:08
oomichihi, some patches cannot be kicked for the gate job even if getting approvals.  e.g) ,
jeblairmordred, clarkb, fungi: when i used an instance variable in the packet read function, i made it 25x slower.01:09
jeblaircaching it in a local var makes it better again01:09
fungijeblair: that's a substantial magnification01:09
clarkbjeblair: wow01:10
clarkboomichi: looking01:10
*** bo_sh has left #openstack-infra01:10
*** amotoki has joined #openstack-infra01:10
*** shashankhegde has quit IRC01:10
oomichiclarkb: thanks :)01:10
*** teran has quit IRC01:11
*** mmaglana has quit IRC01:14
clarkbjeblair: fungi: looks like did not go from check to gate properly01:18
clarkboomichi: I think if you recheck it again it will properl go to the gate. not currently sure hwy it didn't automatically01:19
*** baoli has quit IRC01:19
fungiclarkb: looks like it was probably in the gate pipeline when we had zuul going up and down01:20
oomichiclarkb: thanks, rechecks are done on both.01:20
fungijeblair: any chance that was left out of your list of changes to reenqueue in the gate?01:21
jeblairfungi: i think that was from just a few hours ago01:22
fungid'oh, you're right01:22
fungii got my dates screwed up01:22
*** mmaglana has joined #openstack-infra01:22
fungii'm looking in the zuul log now01:22
*** achanda has quit IRC01:22
clarkb2014-11-26 00:39:43,118 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added openstack/nova master 68560,19> I think that is where it should figure out that it can gate on it01:24
fungino, i was right on the dates... mikal approved patchset 19 a little over 24 hours ago01:26
clarkbfungi: but there was a reverify this afternoon01:27
clarkbthat came back +1 but did not enqueue to the gate01:27
jeblairdid we remove reverify?01:27
fungizuul would have been offline at the time. then it was rechecked and passed at 04:1501:27
clarkbjeblair: no it reran the jobs properly01:27
fungibut never enqueued when the check results came back at that point01:27
clarkbthe layout.yaml looks correct01:29
clarkbrequires jenkins to +1 or +2 and triggers on comment added +1 by jenkins01:29
clarkbor and require workflow +101:29
*** dimsum__ has joined #openstack-infra01:30
clarkboh I know01:30
*** baoli has joined #openstack-infra01:30
clarkbif you look at the jenkins comments only the really early one says +101:30
*** mmaglana has quit IRC01:30
clarkbI wonder if the events neglect to say +1 in the event if it isn't a state change01:30
clarkbbecause gerrit01:30
fungid'oh, right01:30
clarkband so we get an event that says "looks good" but without the +1 and zuul does nothing01:31
*** achanda has joined #openstack-infra01:31
clarkband its back in check not gate01:31
clarkbso I think what we need is to approve that change again01:31
clarkband maybe to cry about gerrit01:31
*** teran has joined #openstack-infra01:31
*** MaxV_ has quit IRC01:31
jeblairthe latest recheck should do it, right?01:31
clarkbjeblair: no01:31
clarkbbecause there is no state change01:32
jeblairer, why isn't that a trigger for gate?01:32
clarkbbecause gerrit only emits vote values if they change01:32
*** r-daneel has quit IRC01:32
clarkbyay gerrit01:33
jeblairoh, because we have no reverify now01:33
jeblairi thought we had recheck and reverify as triggers in both pipelines01:33
clarkboh ya we don't put those in gate they are in check because assumption is you reverify or recheck, get the +1 then that enqueues01:33
clarkbno just check because ^ but that appears to be a bad assumption01:33
*** otter768 has joined #openstack-infra01:34
*** teran_ has quit IRC01:34
*** achanda has quit IRC01:35
*** dimsum__ has quit IRC01:36
openstackgerritJay Faulkner proposed openstack-infra/project-config: Make agent_ssh check job voting for IPA
*** dimsum__ has joined #openstack-infra01:37
*** guest764 has quit IRC01:37
clarkbgit log -p is failing me01:37
jeblairyeah i'm having trouble too01:37
clarkbtrying to find the point where we removed reverify01:37
jeblairi think something broke when the file was moved01:38
JayFI changed the scope of to make the tempest job only vote for IPA, not Ironic, if others want to take a look I'd appreciate it.01:38
*** otter768 has quit IRC01:38
clarkbJayF: ya01:39
clarkber jeblair yup01:39
* JayF messing up your j<tab> sorted by recent nicks01:40
*** hdd has joined #openstack-infra01:40
*** guest764 has joined #openstack-infra01:40
clarkbJayF: indeed01:40
*** dimsum__ has quit IRC01:41
adam_gfungi, hmph so raising the oslo.vmware dependency to a higher line in requirements.txt  doesn't seem to help here :|
clarkbjeblair: that already had it removed from gate though01:42
clarkbso we need to look before that01:43
jeblairoh right01:43
clarkb,p,002ed22a0001b259 is what I am looking through01:43
clarkbits between that and somewhere01:43
clarkboh wait no its even earlier01:44
*** patrickeast has quit IRC01:44
jeblairclarkb, fungi: after addressing the instance variable issue, the client performance is back up to par, but i'm a bit unhappy with the server performance; i may give this another pass and make it simpler01:45
clarkbbefore even01:46
clarkbjeblair: hack away :) very interested to see the final result though01:46
jeblairand my best guess as to what was going on is very bad performance; i can't reproduce the same numbers we saw in production locally, but i can get some really bad ones :)01:46
*** shashankhegde has joined #openstack-infra01:48
clarkbjeblair: I think I found it01:48
fungithat does appear to be where we dropped it, yes01:50
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Add ssh key to root user
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Enable live block migration
*** wuhg has joined #openstack-infra01:52
*** dimsum__ has joined #openstack-infra01:53
*** marun has quit IRC01:55
*** mudassirlatif has quit IRC01:55
*** fandi has quit IRC01:55
*** mmaglana has joined #openstack-infra01:56
*** mmaglana has quit IRC01:58
*** ayoung-dad-mode is now known as ayoung02:01
adam_gfungi, you mentioned some concern about capping oslo libs in stable, care to expand? bumping oslo.vmware higher in the req list didn't help at all and i'm thinking we may need
fungiadam_g: because devstack doesn't (yet) know how to do that and ends up installing from the tip of master anyway. or did until recently but maybe sdague's changes make that possible now?02:04
adam_gfungi, the devstack changes to install from released versions got backported, which is kinda how we've ended up here02:04
*** emagana has joined #openstack-infra02:06
fungiadam_g: which dependency is it that was not getting upgraded even after reordering oslo.vmware to appear earlier in the requirements list?02:06
adam_gfungi, eventlet02:06
adam_gfungi, its still ending up with the dependency satisfied by the distro package (0.13.0)02:06
adam_gi cant seem to reproduce this locally, though02:07
*** emagana_ has joined #openstack-infra02:07
*** yaguang has joined #openstack-infra02:08
*** dannywilson has quit IRC02:08
fungiadam_g: so in this case it looks like keystone is being installed by devstack as the first thing to request eventlet and asks for eventlet>=0.13.002:08
*** yaguang_ has joined #openstack-infra02:09
*** dannywilson has joined #openstack-infra02:09
fungiswift does the same...02:09
fungithen glance02:10
fungicinder, nova...02:10
*** emagana has quit IRC02:10
*** Ark has joined #openstack-infra02:11
*** Ark is now known as Guest3413802:12
adam_gduring the glance install, prior to the eventlet pass, oslo.vmware 0.7.0 gets downloaded but its eventlet requirement seems to be ignored02:12
adam_gi believe thats the first occurence of oslo.vmware being hit02:12
*** yaguang has quit IRC02:12
fungiyeah, looking at the bit where it's installing glance, it installs oslo.vmware 0.7.0 but does not go directly to installing its requirements02:13
*** dannywilson has quit IRC02:13
adam_gi've spent some time reproducing what devstacks doing with different variations of a requirements.txt and can't reproduce it02:14
fungiaha, it _is_ breadth-first not depth-first02:14
fungilook down toward the end and you'll see it starting to install the transitive dependencies if needed02:14
*** nosnos has joined #openstack-infra02:14
fungiso the "list it earlier" trick only works for competing transitive dependencies02:15
funginot for a transitive dependency which needs to meet more specific criteria than a directly listed version of the same requirement02:15
adam_gfungi, hmmm02:15
fungiblah blah (from oslo.vmware>=0.2->glance==2014.1.4.dev6.gef77c79)02:16
fungidoesn't mention eventlet, presumably because it already checked eventlet earlier where it was listed as a requirement of glance02:16
fungiadam_g: so yeah, you can float an oslo.vmware version cap in stable global-requirements.txt and see if it passes tests i guess02:17
adam_g is in check queue now02:18
*** bo_sh1 has joined #openstack-infra02:18
fungisince pip is satisfying reqs breadth-first for projects, there's really no way with reordering you're going to be able to get a transitive req to trump a direct req i don't think02:18
*** banix has joined #openstack-infra02:18
adam_gunfortunately, the grenade job that would let a fix for this be tested pre-merge is busted on a devstack bug introduced in that backport i mentioned, and can't be fixed till this is resolved :|02:19
fungiquite the pity02:19
*** bo_sh1 has left #openstack-infra02:20
*** Sukhdev has quit IRC02:20
*** Sukhdev has joined #openstack-infra02:21
*** yamamoto has joined #openstack-infra02:22
*** asettle is now known as asettle-optometr02:25
*** asettle-optometr is now known as asettle-glasses02:25
*** NobodyCam has quit IRC02:26
*** NobodyCam has joined #openstack-infra02:27
*** MarkAtwood has quit IRC02:28
*** sdake_ has joined #openstack-infra02:31
*** sdake_ has quit IRC02:31
*** sdake_ has joined #openstack-infra02:31
*** MaxV has joined #openstack-infra02:32
*** vipul has quit IRC02:33
*** jerryz has joined #openstack-infra02:35
*** dmsimard_away is now known as dmsimard02:37
*** MaxV has quit IRC02:37
*** vipul has joined #openstack-infra02:37
*** MarkAtwood has joined #openstack-infra02:38
*** koolhead_ has joined #openstack-infra02:45
*** wenlock has joined #openstack-infra02:45
*** koolhead17 has quit IRC02:48
*** shashankhegde has quit IRC02:49
*** sputnik13 has quit IRC02:51
*** gokrokve has joined #openstack-infra02:52
*** gokrokve has quit IRC02:52
*** sputnik13 has joined #openstack-infra02:52
*** dimsum__ has quit IRC02:52
*** gokrokve has joined #openstack-infra02:52
*** gokrokve has quit IRC02:53
*** gokrokve has joined #openstack-infra02:53
*** banix has quit IRC02:54
*** gokrokve has quit IRC02:54
*** gokrokve has joined #openstack-infra02:55
*** guest764 has quit IRC02:58
*** sputnik13 has quit IRC02:58
*** gokrokve has quit IRC02:59
*** gokrokve has joined #openstack-infra02:59
*** gokrokve has quit IRC02:59
*** MarkAtwood has quit IRC02:59
*** gokrokve has joined #openstack-infra02:59
*** gokrokve has quit IRC03:00
*** gokrokve has joined #openstack-infra03:00
*** nosnos has quit IRC03:01
*** fifieldt has joined #openstack-infra03:01
*** Sukhdev has quit IRC03:01
adam_gfungi, ugh yeah-- ... transitive dependencies are skipped entirely if one by the same name exists in the top-level or has already been added by a previously processed requirement03:02
*** Ryan_Lane has quit IRC03:03
*** mmaglana has joined #openstack-infra03:03
*** hdd has quit IRC03:04
*** gokrokve has quit IRC03:05
*** gokrokve has joined #openstack-infra03:05
*** banix has joined #openstack-infra03:06
*** gokrokve has quit IRC03:06
*** gokrokve has joined #openstack-infra03:07
*** gokrokve has quit IRC03:07
*** gokrokve has joined #openstack-infra03:08
*** gokrokve has quit IRC03:08
*** gokrokve has joined #openstack-infra03:12
*** gokrokve has quit IRC03:17
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full
*** ryanpetrello has joined #openstack-infra03:18
*** dmsimard is now known as dmsimard_away03:21
*** harlowja is now known as harlowja_away03:29
*** wenlock has quit IRC03:30
*** hdd has joined #openstack-infra03:35
*** otter768 has joined #openstack-infra03:35
*** jerryz1 has joined #openstack-infra03:35
*** jerryz has quit IRC03:37
*** otter768 has quit IRC03:40
*** ryanpetrello has quit IRC03:41
*** sarob has joined #openstack-infra03:45
*** amcrn has quit IRC03:47
*** emagana_ has quit IRC03:48
*** emagana has joined #openstack-infra03:48
*** emagana has quit IRC03:49
*** sarob has quit IRC03:49
*** emagana has joined #openstack-infra03:50
*** unicell has joined #openstack-infra03:51
*** dimsum__ has joined #openstack-infra03:52
*** ivar-lazzaro has quit IRC03:53
*** dimsum__ has quit IRC03:57
*** ryanpetrello has joined #openstack-infra04:07
*** achanda has joined #openstack-infra04:12
*** nosnos has joined #openstack-infra04:16
*** MaxV has joined #openstack-infra04:22
*** ryanpetrello has quit IRC04:23
*** bhunter71 has quit IRC04:24
*** MaxV has quit IRC04:25
*** Sukhdev has joined #openstack-infra04:26
*** AlexF has joined #openstack-infra04:26
*** emagana has quit IRC04:28
openstackgerritIan Wienand proposed openstack-infra/puppet-httpd: Update Rakefile and module info
*** achanda has quit IRC04:30
*** dannywilson has joined #openstack-infra04:31
openstackgerritIan Wienand proposed openstack-infra/system-config: Add puppet-httpd to modules.env
*** banix has quit IRC04:34
*** weshay has quit IRC04:37
*** sabeen1 has joined #openstack-infra04:46
*** pcrews has quit IRC04:48
*** shashankhegde has joined #openstack-infra04:49
*** ryanpetrello has joined #openstack-infra04:49
*** zz_dimtruck is now known as dimtruck04:50
*** yongli has joined #openstack-infra04:51
*** yfried has joined #openstack-infra04:51
*** tnovacik has joined #openstack-infra04:54
*** ayoung is now known as ayoung_ZZzz_zzZZ04:59
*** AlexF has quit IRC05:04
*** harlowja_at_home has joined #openstack-infra05:08
*** hdd has quit IRC05:09
*** garyh has quit IRC05:09
*** rushiagr_away is now known as rushiagr05:10
*** harlowja_at_home has quit IRC05:12
*** stevemar has joined #openstack-infra05:13
*** asettle-glasses is now known as asettle05:20
*** AlexF has joined #openstack-infra05:21
*** emagana has joined #openstack-infra05:29
*** BharatK has joined #openstack-infra05:30
*** talluri has joined #openstack-infra05:30
*** jerryz1 has quit IRC05:32
*** yfried has quit IRC05:34
*** otter768 has joined #openstack-infra05:36
*** stevemar has quit IRC05:36
*** chandankumar has joined #openstack-infra05:37
*** Longgeek has joined #openstack-infra05:38
*** ujuc has joined #openstack-infra05:39
*** otter768 has quit IRC05:40
*** garyh has joined #openstack-infra05:41
*** michchap has quit IRC05:41
*** Longgeek has quit IRC05:44
*** Guest34138 has quit IRC05:47
*** dannywilson has quit IRC05:50
*** gokrokve has joined #openstack-infra05:50
*** gokrokve has quit IRC05:52
*** dannywilson has joined #openstack-infra05:58
*** rushiagr is now known as rushiagr_away05:59
*** tnovacik has quit IRC06:00
*** stevemar has joined #openstack-infra06:01
*** k4n0 has joined #openstack-infra06:03
*** ryanpetrello has quit IRC06:05
*** talluri has quit IRC06:15
*** Ark has joined #openstack-infra06:17
tchaypottx: we seem to have a conflict between the meetings page and the meetings ical feed as to the time of the tripleo meeting06:17
*** Ark is now known as Guest541606:18
*** Masahiro has joined #openstack-infra06:19
*** Masahiro has quit IRC06:19
*** Masahiro has joined #openstack-infra06:19
*** dimtruck is now known as zz_dimtruck06:21
*** Sukhdev has quit IRC06:22
*** Longgeek has joined #openstack-infra06:22
*** mrmartin has joined #openstack-infra06:23
*** mudassirlatif has joined #openstack-infra06:23
*** Guest5416 has quit IRC06:26
*** Masahiro has quit IRC06:26
*** Masahiro has joined #openstack-infra06:27
*** dannywilson has quit IRC06:28
*** Masahiro has quit IRC06:29
*** Masahiro has joined #openstack-infra06:29
*** yfried has joined #openstack-infra06:31
*** talluri has joined #openstack-infra06:33
tchaypottx: because the wiki page is wrong. That’s something I can fix, so I did.06:34
*** stevemar has quit IRC06:34
*** SumitNaiksatam has quit IRC06:38
*** SumitNaiksatam has joined #openstack-infra06:39
*** guest764 has joined #openstack-infra06:41
*** emagana has quit IRC06:43
*** emagana has joined #openstack-infra06:43
*** belmoreira has joined #openstack-infra06:46
*** emagana has quit IRC06:48
*** pcrews has joined #openstack-infra06:48
*** afazekas has joined #openstack-infra06:50
*** viglesias has quit IRC06:51
*** pcrews has quit IRC06:53
*** viglesias has joined #openstack-infra06:56
*** fandi has joined #openstack-infra06:59
*** yfried is now known as yfried|afk06:59
*** Masahiro has quit IRC07:02
*** Masahiro has joined #openstack-infra07:03
*** yfried|afk is now known as yfried07:05
*** Masahiro has quit IRC07:05
*** rushiagr_away is now known as rushiagr07:07
*** fifieldt has quit IRC07:16
*** ashaeron has joined #openstack-infra07:20
*** yfried is now known as yfried|afk07:21
*** pcrews has joined #openstack-infra07:23
*** Masahiro has joined #openstack-infra07:25
*** talluri has quit IRC07:27
*** ujuc has quit IRC07:30
*** koolhead_ has quit IRC07:30
*** yfried|afk is now known as yfried07:31
*** koolhead17 has joined #openstack-infra07:31
*** achanda has joined #openstack-infra07:31
*** pcrews has quit IRC07:31
*** koolhead17 has quit IRC07:35
*** camunoz has quit IRC07:35
*** achanda has quit IRC07:36
*** Masahiro has quit IRC07:36
*** otter768 has joined #openstack-infra07:36
*** ala_ has joined #openstack-infra07:37
*** e0ne has joined #openstack-infra07:38
*** ociuhandu has quit IRC07:38
*** e0ne has quit IRC07:38
*** teran_ has joined #openstack-infra07:38
*** ildikov has joined #openstack-infra07:39
*** alla__ has joined #openstack-infra07:41
*** zz_dimtruck is now known as dimtruck07:41
*** otter768 has quit IRC07:41
*** unicell1 has joined #openstack-infra07:41
*** teran has quit IRC07:41
*** unicell has quit IRC07:43
*** ala_ has quit IRC07:44
*** SumitNaiksatam has quit IRC07:46
*** sputnik13 has joined #openstack-infra07:47
*** AlexF has quit IRC07:47
*** camunoz has joined #openstack-infra07:49
*** SumitNaiksatam has joined #openstack-infra07:49
*** jaypipes has quit IRC07:50
*** HeOS has quit IRC07:50
*** dimtruck is now known as zz_dimtruck07:51
*** ryanpetrello has joined #openstack-infra07:51
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove python26 jobs from all stackforge projects
oomichiclarkb: FYI: I could kick the gate job on some patches after rebasing them for the latest master. maybe they got old.07:51
AJaegerclarkb, is the python 2.6 job, I've rebased and added a comment with one question.07:53
*** AlexF has joined #openstack-infra07:54
*** armax has quit IRC07:55
*** ryanpetrello has quit IRC07:56
*** salv-orlando has joined #openstack-infra08:02
*** ZZelle has quit IRC08:02
*** ZZelle has joined #openstack-infra08:03
*** amuller has joined #openstack-infra08:05
*** ildikov has quit IRC08:07
*** michchap has joined #openstack-infra08:11
*** dtantsur|afk is now known as dtantsur08:13
*** oomichi has quit IRC08:17
*** henditan has joined #openstack-infra08:17
*** yfried is now known as yfried|afk08:17
*** mudassirlatif has quit IRC08:17
*** yamahata has quit IRC08:18
*** Masahiro has joined #openstack-infra08:18
*** henditan has quit IRC08:19
*** markus_z has joined #openstack-infra08:21
*** arxcruz has joined #openstack-infra08:21
*** ildikov has joined #openstack-infra08:22
*** nadya has joined #openstack-infra08:26
*** nadya is now known as Guest2838108:27
*** koolhead17 has joined #openstack-infra08:28
*** jcoufal has joined #openstack-infra08:29
*** koolhead_ has joined #openstack-infra08:32
*** achanda has joined #openstack-infra08:33
*** andreykurilin_ has joined #openstack-infra08:35
*** koolhead17 has quit IRC08:35
*** yfried|afk is now known as yfried08:38
*** MaxV has joined #openstack-infra08:40
ttxtchaypo: thx! Should get reported on the meeting ical when Tony catches up08:40
*** talluri has joined #openstack-infra08:42
openstackgerritJoakim Löfgren proposed openstack-infra/jenkins-job-builder: Add DRY publisher
*** AlexF has quit IRC08:45
*** koolhead_ has quit IRC08:46
*** shashankhegde has quit IRC08:47
*** teran_ has quit IRC08:48
*** achanda has quit IRC08:48
*** mpavlase has joined #openstack-infra08:49
*** koolhead17 has joined #openstack-infra08:50
*** SumitNaiksatam has quit IRC08:51
*** SumitNaiksatam has joined #openstack-infra08:52
*** koolhead17 has quit IRC08:55
*** SumitNaiksatam has quit IRC08:56
*** berendt has joined #openstack-infra08:57
*** andreykurilin_ has quit IRC08:58
*** andreykurilin_ has joined #openstack-infra08:59
*** mpaolino has joined #openstack-infra08:59
*** jistr has joined #openstack-infra08:59
*** jpich has joined #openstack-infra08:59
*** jp_at_hp has joined #openstack-infra08:59
*** jp_at_hp has quit IRC08:59
*** jp_at_hp has joined #openstack-infra08:59
*** SumitNaiksatam has joined #openstack-infra09:00
*** HeOS has joined #openstack-infra09:06
openstackgerritJoakim Löfgren proposed openstack-infra/jenkins-job-builder: Refactor checkstyle publisher
openstackgerritNikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client
*** amotoki has quit IRC09:11
tchaypottx: it’s okay, the ical was correct already, I just made the wiki match the ical :)09:11
*** derekh has joined #openstack-infra09:12
*** Masahiro has quit IRC09:13
ttxtchaypo: oh ok. All good then09:14
*** e0ne has joined #openstack-infra09:16
*** bo_sh has joined #openstack-infra09:17
*** ihrachyshka has joined #openstack-infra09:19
*** vigneshvar has joined #openstack-infra09:20
*** Masahiro has joined #openstack-infra09:23
*** nfedotov has joined #openstack-infra09:23
*** sergsh_away is now known as sergsh09:25
*** dannywilson has joined #openstack-infra09:29
*** yamamoto has quit IRC09:32
*** sputnik13 has quit IRC09:33
*** dannywilson has quit IRC09:33
*** afazekas has quit IRC09:33
*** mpavlase has quit IRC09:35
*** otter768 has joined #openstack-infra09:37
*** Longgeek_ has joined #openstack-infra09:38
*** Longgeek has quit IRC09:39
openstackgerritJulien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack
*** otter768 has quit IRC09:42
*** jedimike has joined #openstack-infra09:42
*** darvon has quit IRC09:43
*** darvon has joined #openstack-infra09:43
*** pblaho has joined #openstack-infra09:43
*** yamahata has joined #openstack-infra09:45
*** BobBall_AWOL is now known as BobBall09:45
rcarrillocruzNikitaKonovalov: hi there, is there a python storyboard client code available yet?09:45
rcarrillocruzi'd be interested in get involved in that09:45
NikitaKonovalovrcarrillocruz: not yet09:46
NikitaKonovalovThere is a change to set up a repo09:46
NikitaKonovalovAs soon as it is merged, I'll upload initial commits09:47
*** Ark has joined #openstack-infra09:47
NikitaKonovalovThose will contain basic layout, oslo apiclient, and some base classes09:48
*** Ark is now known as Guest5539509:48
*** afazekas has joined #openstack-infra09:48
NikitaKonovalovI'm now working on setting up a manager for /users endpoint09:48
NikitaKonovalovAs soon as it is ready, all other endpoints support will not be a problem09:49
openstackgerritNikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client
*** andreykurilin_ has quit IRC09:52
*** Guest55395 has quit IRC09:53
*** jcoufal_ has joined #openstack-infra09:57
*** abhishek has joined #openstack-infra09:59
*** ociuhandu has joined #openstack-infra09:59
*** jcoufal has quit IRC10:00
openstackgerritNikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client
*** sasik has joined #openstack-infra10:02
*** sasik has left #openstack-infra10:02
*** ociuhandu_ has joined #openstack-infra10:03
*** ociuhandu has quit IRC10:04
*** ociuhandu_ is now known as ociuhandu10:04
abhishekhi, does any one has idea why check-grenade-dsvm is failing on stable/juno patches?10:05
*** rlandy has joined #openstack-infra10:05
*** afazekas has quit IRC10:07
openstackgerritJoakim Löfgren proposed openstack-infra/jenkins-job-builder: Refactor checkstyle publisher
openstackgerritJulien Danjou proposed openstack-infra/project-config: oslo: add missing Python 2.6 jobs
*** talluri has quit IRC10:18
*** talluri has joined #openstack-infra10:18
*** Guest28381 has quit IRC10:20
*** afazekas has joined #openstack-infra10:21
*** Masahiro has quit IRC10:25
*** andreykurilin_ has joined #openstack-infra10:27
*** mmaglana has quit IRC10:28
*** nadya_ has joined #openstack-infra10:29
*** rediskin has joined #openstack-infra10:31
*** koolhead17 has joined #openstack-infra10:31
*** KanagarajM has joined #openstack-infra10:34
*** ldnunes has joined #openstack-infra10:35
*** nadya_ has quit IRC10:38
abhishek hi, does any one has idea why check-grenade-dsvm is failing on stable/juno patches?
*** zz_johnthetubagu is now known as johnthetubaguy10:41
*** yamahata has quit IRC10:43
*** yamahata has joined #openstack-infra10:45
*** dizquierdo has joined #openstack-infra10:52
*** yfried is now known as yfried|afk10:55
*** jerryz has joined #openstack-infra10:58
*** adalbas has joined #openstack-infra10:58
*** rfolco has joined #openstack-infra11:01
*** maishsk has joined #openstack-infra11:02
maishskAnyone awake?11:02
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove python26 jobs from all stackforge projects
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python 2.6 for tooz
*** nadya_ has joined #openstack-infra11:04
*** ildikov has quit IRC11:05
*** rfolco has quit IRC11:05
*** koolhead17 has quit IRC11:06
AJaegerjd__, which oslo libs are missing python 2.6 gating?11:06
AJaegermaishsk, hi!11:07
*** koolhead17 has joined #openstack-infra11:07
maishskAJaeger: Hi11:07
maishskmaybe you can help me out11:07
*** rfolco has joined #openstack-infra11:07
AJaegermaishsk, let's give it a try. What's up?11:08
maishskSo here goes. I am looking to create a easy to deploy environment that will allow people to contribute to Openstack11:09
maishsk i.e. a quick way to get gerrit, git up and running11:09
maishskeasiest way to do this - is a container11:10
maishskin order to do this I will need a few things - username, email etc...11:10
maishskbut also an SSH key11:10
maishskthe public key is up on gerrit - because I pasted it there11:11
*** rfolco has quit IRC11:12
AJaegerand the private part of the key needs to be in your container - or home directory11:12
*** koolhead17 has quit IRC11:12
maishskso I have .ssh/ and .ssh/id_rsa I have11:12
AJaegerSo, .pub is the public one, id_rsa the private one.11:12
maishsknow the question - is there a way to programatically create a key on gerrit?11:13
maishsknot through the gui?11:13
maishskif so how?11:13
AJaegerCheck the upstream gerrit documentation at:
AJaegerI'm not aware of anything - but I'm not the gerrit expert, those are still asleep ;)11:14
AJaegerYou could write a script that asks for username etc and generates a key if none exists.11:15
maishskbut I assume I cannot do that from a remote location11:15
*** yfried|afk is now known as yfried11:15
maishskSo I have been looking here11:16
AJaegerI don't think so. And for the set-account command, you need to identify yourself first - so have already one ssh added11:16
AJaegerso, you should be able to upload with your gerrit credentials - good11:17
maishskI am having difficulty with finding the correct curl commands and the correct endpoint...11:17
AJaegermaishsk, better wait for fungi or clarkb to help you with that...11:18
AJaegergive them another 3 or 4 hours11:19
maishskwill catch them later when they decide to wake up .... :)11:19
*** rfolco has joined #openstack-infra11:24
*** dizquierdo has quit IRC11:24
*** bo_sh has left #openstack-infra11:25
*** yfried is now known as yfried|afk11:25
*** Masahiro has joined #openstack-infra11:26
*** sasik has joined #openstack-infra11:28
*** sasik has left #openstack-infra11:28
*** yfried|afk is now known as yfried11:28
*** mmaglana has joined #openstack-infra11:29
*** salv-orlando has quit IRC11:29
*** mpavlase has joined #openstack-infra11:30
*** Masahiro has quit IRC11:30
*** aysyd has joined #openstack-infra11:31
*** hdd has joined #openstack-infra11:35
*** dizquierdo has joined #openstack-infra11:36
*** mmaglana has quit IRC11:36
*** otter768 has joined #openstack-infra11:38
*** otter768 has quit IRC11:43
*** achuprin_ has quit IRC11:46
*** jamielennox is now known as jamielennox|away11:48
*** hdd has quit IRC11:49
*** dmakogon_ is now known as denis_makogon11:49
*** fandi has quit IRC11:50
yolandajeblair, clarkb, i have an issue with nodepool and dib, please ping me when you are available11:57
sdagueabhishek: because swift did not start -
*** achuprin_ has joined #openstack-infra12:03
ekarlso-is there anything wrong with devstack these days ?12:04
ekarlso-2014-11-26 12:01:05.108 31830 TRACE glance     os.makedirs(self._signing_dirname, stat.S_IRWXU)12:04
ekarlso-2014-11-26 12:01:05.108 31830 TRACE glance   File "/usr/lib/python2.7/", line 150, in makedirs12:04
ekarlso-2014-11-26 12:01:05.108 31830 TRACE glance     makedirs(head, mode)12:04
ekarlso-2014-11-26 12:01:05.108 31830 TRACE glance   File "/usr/lib/python2.7/", line 157, in makedirs12:04
ekarlso-2014-11-26 12:01:05.108 31830 TRACE glance     mkdir(name, mode)12:04
ekarlso-2014-11-26 12:01:05.108 31830 TRACE glance OSError: [Errno 13] Permission denied: '/var/cache/glance'12:04
ekarlso- is my local.conf12:04
*** fandi has joined #openstack-infra12:06
sdagueekarlso-: I'm sure there is plenty wrong, however it's running hundreds of successful gate jobs every day12:06
ekarlso-dunno why that happens then on a fresh trusty vm :|12:07
*** cnesa has joined #openstack-infra12:07
sdagueyour service list looks ancient12:09
ekarlso-sdague: what should they be ? :D12:09
*** baoli has quit IRC12:10
sdaguedon't know for all things, but don't assign to ENabled_sevices directly, just enable / disable what you need12:10
mrmartinsdague: I was looking for nginx examples in system-config and found only remains of a nginx-remove module. Do we have a policy to not use nginx for some reason?12:10
*** ildikov has joined #openstack-infra12:11
chmoueldevananda, sdague: just catched with my highlights, the proposal is up
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Sort taskflow properly
*** salv-orlando has joined #openstack-infra12:20
sdaguemrmartin: no particular policy per say, beyond the fact that the infrastructure runs on apache, and unless there is a really good reason, maintaining a second web server stack is something that should be avoided12:26
sdaguechmouel: yeh, did you see my follow up message on list ?12:26
chmouelsdake: i don't think so, is is the part about being able to have integrated projects have a /devstack/ dir ?12:27
mrmartinsdague, ok, beacuse I'm working on this askbot puppetization and the current site is running on nginx, but it is a good argument to use apache to avoid additional maintenance12:27
AJaegersdague, could you review , please?12:28
sdaguechmouel: yeh, I was thinking about the devstack side of the interface12:28
*** viktors has joined #openstack-infra12:28
*** KanagarajM has quit IRC12:28
chmouelsdague: yeah i mentton it in the spec,12:28
sdagueok, I didn't think it was in the first rev, I'll look again12:29
sdagueoh... look, more revs :)12:29
sdaguesorry, yesterday was a long and busy day12:29
chmouelsdague: ah no worries :)12:29
viktorshi folks! Does anybody knows the current state of the gate-tempest-dsvm-neutron-src-*-icehouse and gate-tempest-dsvm-src-*-icehousegate jobs ?12:29
chmouelsdague: it's just that part
sdaguethe icehouse jobs should have been fixed yesterday12:30
chmouelsdague: maybe i should enphasis that's the goal of this spec?12:30
chmouelemphasis even12:30
viktorssdague: I'm still observing issues with these jobs  =(12:30
sdaguechmouel: so, I think the difference that I was thinking was around the plug space12:31
sdagueviktors: url ?12:31
viktors'doc8' is not a global requirement but it should be,something went wrong12:31
sdagueyeh, oslo.db apparently added a new requirement when we weren't enforcing stable requirements12:32
silehtsdague, hi, I have seen the result of the meeting yesterday for pinnig oslolib/client version for stable branches, I have oslo.messaging gate blocked since stable jobs have been enabled since, perhaps we can merged now ?12:32
sdaguethat's a valid fail in the existing system12:32
*** rpodolyaka1 has joined #openstack-infra12:32
viktorssdague: so should I add this requirement to the stable requirements?12:33
sdaguesileht: it's going to need to be done systemically, not one lib at a time, or we're going to lose everything. I'd suggest coordinating with ttx and stable maint on getting that rolling12:33
sdagueviktors: no, this is the whole conversation around requirements yesterday12:33
AJaegersdague, doc8 is in global-requirements for trunk but perhaps not for icehouse?12:33
sdagueplease take a look at yesterday's project meeting12:33
sdagueAJaeger: yep12:34
silehtsdague, ok, thanks12:34
viktorssdague: ok, will look12:34
*** xchu has joined #openstack-infra12:36
chmouelsdake: feel free to comment on the specs, i think it wil be easier there :)12:37
*** xchu has quit IRC12:37
sdaguethe net of this, oslo libs are going to be blocked until this is resolved. In moving to a new testing scheme we temporarily allowed oslo libs to do a thing that they couldn't before in adding requirements that only exist in master, but that's going to break everything on release. So the capping scheme has to be implemented or the oslo libs need to roll back their dependency changes to release.12:38
openstackgerritThierry Carrez proposed openstack-infra/infra-specs: StoryBoard: story types specification
*** rushiagr is now known as rushiagr_away12:46
sdaguechmouel: ok, commented in the spec12:47
sdagueconsider my -1 advisory of "hey, what about this approach"12:47
sdagueit's not something I'm married to12:47
*** jcoufal_ has quit IRC12:47
*** hdd has joined #openstack-infra12:48
*** jcoufal has joined #openstack-infra12:48
chmouelsdague: no i like it :) i wasn't sure if allowing all XX-*.sh would make sense to be allowed but i guess that worked with for sysv-rc scripts for a while12:51
sdaguechmouel: yeh, until we write devstack-systemd ;)12:52
chmouellet me try to put a new version of the spec getting your suggestions in there,12:52
chmouelsdague: lol :)12:52
*** talluri has quit IRC12:54
*** koolhead17 has joined #openstack-infra12:54
*** baoli has joined #openstack-infra12:54
*** koolhead17 has quit IRC12:54
*** koolhead17 has joined #openstack-infra12:54
jkthmm, trying to run the test suite of turbo-hipster on centos 7, I had to modify the test-requirements.txt so that "six" and "unittest2" are listed before "testtools", otherwise I got this:
jktam I doing something wrong?12:59
jktand I don't see any dependencies listed at , but maybe I just don't know where to look13:00
*** banix has joined #openstack-infra13:01
*** xchu has joined #openstack-infra13:01
jktI mean, this is listed at , but I was hoping for something machine-readable13:02
sdaguejkt: what version of testtools?13:04
*** banix has quit IRC13:04
*** xchu has quit IRC13:04
sdaguejkt: is where the dependencies are listed13:04
openstackgerrityolanda.robla proposed openstack-infra/storyboard-webclient: Add more details on event dashboard
maishskAJaeger: is there a sandbox area where I can test upload a commit - I don't want to start polluting any repositories...13:06
AJaegermaishsk, sure, it's call sandbox ;)13:06
AJaegerPLease abandon patches afterwards so that the review queue is clean13:06
maishskI wonder why I did not think of that.... :)13:06
maishskof course...13:07
*** xchu has joined #openstack-infra13:10
*** Longgeek_ has quit IRC13:11
jktsdague: I do see that in my version of testtools, too (appears to be "1.5.0")13:11
openstackgerritThierry Carrez proposed openstack-infra/release-tools: spec2bp v2 -- support in-review specs
*** Masahiro has joined #openstack-infra13:14
*** xchu has quit IRC13:16
jktsdague: I'm gonna try it with upgraded pip because I've seen it doing *crazy* stuff here on centos713:16
sdaguewhat pip version?13:16
jktstuff like ignoring any version requirements as long as a wrong version is already in the download cache, etc13:17
jktok, no change here :(13:17
sdagueoh, yeh, so that's terrible13:19
*** yaguang_ has quit IRC13:19
sdaguewe are definitely at min req of 1.413:19
*** julim has quit IRC13:19
*** Masahiro has quit IRC13:19
sdagueexpect everything to be broken if you have < 1.413:20
*** nosnos has quit IRC13:20
*** mmaglana has joined #openstack-infra13:20
*** alexpilotti has joined #openstack-infra13:21
maishskAJaeger: it seems this actually works ...
*** bswartz has quit IRC13:22
*** jpich has quit IRC13:22
AJaegermaishsk, congrats!13:22
*** frank_zhou has joined #openstack-infra13:23
*** jpich has joined #openstack-infra13:23
*** mmaglana has quit IRC13:25
jktsdague: so I erased the systemwide pip, so that I now have pip from pip (1.5.6), but I still get this:
sdaguejkt: yeh, honestly, I don't know. I don't use centos and haven't tried to install turbohipster13:28
sdaguethe pbr < 0.6 seems suspicious to me13:28
*** BharatK has quit IRC13:29
*** vigneshvar_ has joined #openstack-infra13:31
*** weshay has joined #openstack-infra13:31
*** alexpilotti has quit IRC13:32
openstackgerritCedric Brandily proposed openstack-infra/git-review: get_remote_url(): honor any "url.<base>.insteadOf" config setting
*** fandi has quit IRC13:33
*** vigneshvar has quit IRC13:34
*** jaypipes has joined #openstack-infra13:36
*** mestery has quit IRC13:36
*** ildikov has quit IRC13:36
*** Longgeek has joined #openstack-infra13:37
*** baoli has quit IRC13:37
*** dizquierdo has quit IRC13:38
*** otter768 has joined #openstack-infra13:39
*** berendt has quit IRC13:40
*** BharatK has joined #openstack-infra13:41
*** ryanpetrello has joined #openstack-infra13:42
*** rushiagr_away is now known as rushiagr13:42
*** dizquierdo has joined #openstack-infra13:42
*** otter768 has quit IRC13:44
*** baoli has joined #openstack-infra13:44
mordredjkt, sdague: you are saying words that sound like I should look at something13:48
*** ildikov has joined #openstack-infra13:48
*** julim has joined #openstack-infra13:49
jktwhat I'm saying is that I have no idea why this is broken, mordred13:49
jktjhesketh: it seems that there's some non-determinism in the way turbo-hipster's tests are done, see
jktjhesketh: my torbo-hipster has some trivial patches, -- the disabling of jjb and mysql is there because I believe that they are not neeed in my setup (are they?)13:50
*** yamahata has quit IRC13:50
*** dimsum__ has joined #openstack-infra13:51
*** erw has quit IRC13:51
*** yamahata has joined #openstack-infra13:51
mordredjkt: can you also pip install unittest2 by hand real quick?13:52
*** erw has joined #openstack-infra13:52
mordredit looks like there is a bug somewhere that is not telling your system that unittest2 is now needed by testtools13:53
*** BharatK has quit IRC13:53
*** yfried is now known as yfried|afk13:53
jktmordred: while I was trying to fix this, I used yum to install it13:53
yolandamordred, you there? i have a problem with nodepool and dib image upload, can we talk about it?13:53
mordredeither way13:53
jktmordred: but because this is a virtual env, apparently, I belive that it doesn't make any difference what actually is in the system13:53
openstackgerritDmitry Teselkin proposed openstack-infra/nodepool: Add per-provider env vars
mordredah - yes, in a venv no, that will not have helped13:53
*** nadya_ has quit IRC13:54
jktmordred: for example, it went trying to compile PyYAML even though it's already present13:54
mordredso install it in to the venv with pip13:54
jktmordred: how do I do that?13:54
mordredwhat's the venv?13:54
mordredoh - you're just doing tox?13:54
jktI have no idea, I just run `tox` in a top-level dir of turbo-hipster13:54
fungidhellmann: sdague: just a heads up, i'm wondering if is likely to wedge oslo.vmware icehouse stable-compat jobs?13:54
*** che-arn8 has joined #openstack-infra13:54
*** che-arne has joined #openstack-infra13:54
*** che-arn8 has quit IRC13:54
mordreddo this: .tox/py27/bin/pip install unittest213:55
*** vigneshvar_ has quit IRC13:55
jktmordred: it's present now because I added it before. So I'll nuke the env, undo my local change, and try again13:56
fungijkt: if you pass -r to tox it will recreate its venv(s)13:56
sdaguefungi: yeh, possibly, like I said in the project meeting this whole thing is complicated enough to need a champion to navigate through it13:56
*** thingee has quit IRC13:57
*** fabo has quit IRC13:58
*** thingee has joined #openstack-infra13:59
jktmordred: I don't think it helped. I undid my changes to test-requirements.txt, destroyed the env, run `tox -e py27` which failed, as expected14:01
jktmordred: then I run `.tox/py27/bin/pip install unittest2` as requested, it installed stuff, and re-run `tox -e py27` again, which failed with the same error as before14:02
*** chandankumar has quit IRC14:02
sdaguejkt: what version of pip, what version of tox14:02
jktmordred: my assumption is that tox notices that the venv is not set up properly, and decided to wipe it out before proceeding14:02
*** hdd has quit IRC14:02
sdaguejkt: that is the wrong assumption14:02
sdaguetox does nothing to the venv if it exists14:02
sdagueevenif it is wrong14:02
jktsdague: tox 1.4.2, pip 1.5.614:02
mordredjkt: I'm trying locally14:03
sdagueso at least in the main projects, you need tox 1.614:03
*** mjturek has joined #openstack-infra14:04
jktsdague: how do you explain that "unittest2" is now absent from my ./.tox/py27/lib/python2.7/site-packages/ ?14:05
*** mjturek has left #openstack-infra14:05
*** mestery has joined #openstack-infra14:05
openstackgerritDmitry Teselkin proposed openstack-infra/system-config: Fix vhost configuration sections
jktsdague: OK, I just checked that running `tox -e py27` effectively removed unittest2 from there14:06
jktI'll try with tox from pip as opposed to my system's copy14:06
jktmaybe rhel7 is just too obsolete for this tuff already :(14:06
dteselkinI have two reviews with 2 +2, but not merged. Could someone take a look please?14:07
*** bswartz has joined #openstack-infra14:08
jktsdague, mordred: tox 1.8.1 fixes this issue14:09
*** thingee has quit IRC14:10
*** AJaeger has quit IRC14:11
mordredjkt: w00t!14:11
*** AJaeger has joined #openstack-infra14:11
*** AJaeger has joined #openstack-infra14:11
sdaguejkt: I wonder if this would fix it for you -
*** yfried|afk is now known as yfried14:13
*** tchaypo has quit IRC14:15
* jkt checks14:15
*** thingee has joined #openstack-infra14:15
dteselkinIt's and
*** tchaypo has joined #openstack-infra14:17
jktsdague: commented in there14:18
*** ameade_ has quit IRC14:18
jktis there a turbo-hipster channel? #turbo-hipster isn't it, apparently14:18
sdaguejkt: those folks mostly are here14:19
jktok, I just asked as openstackgerrit hasn't reported your change in here14:20
*** ameade_ has joined #openstack-infra14:20
*** thingee has quit IRC14:20
*** thingee has joined #openstack-infra14:21
*** ayoung_ZZzz_zzZZ is now known as ayoung14:21
*** frank_zhou has quit IRC14:21
*** dkliban_afk is now known as dkliban14:22
jktjhesketh: OK, I'm definitely seeing test flakiness even with just tests.test_worker_manager.TestWorkerServer.test_zuul_manager_started, see for an example of two executions14:28
jktjhesketh: both invocations reported "passed", but I do not terribly like the backtraces in there14:28
*** Ark has joined #openstack-infra14:30
*** Ark is now known as Guest6203614:30
*** yfried is now known as yfried|afk14:31
*** jgrimm is now known as zz_jgrimm14:32
jktseems that the gear server gets shutdown, interesting14:32
*** Longgeek has quit IRC14:34
*** Guest62036 has quit IRC14:34
*** nadya_ has joined #openstack-infra14:34
*** armax has joined #openstack-infra14:36
*** thingee has quit IRC14:36
*** r-daneel has joined #openstack-infra14:37
*** Longgeek has joined #openstack-infra14:40
*** mikeit has joined #openstack-infra14:42
*** xyang0 has joined #openstack-infra14:43
*** AlexF has joined #openstack-infra14:44
*** che-arne has quit IRC14:46
*** bhunter71 has joined #openstack-infra14:46
*** mikeit has quit IRC14:47
*** mikeit has joined #openstack-infra14:48
*** mestery has quit IRC14:48
*** kgiusti has joined #openstack-infra14:49
*** dkranz has joined #openstack-infra14:50
*** rediskin has left #openstack-infra14:54
*** AJaeger_ has joined #openstack-infra14:55
*** che-arne has joined #openstack-infra14:56
clarkbAJaeger thank you. I will review14:57
AJaeger_morning, clarkb!14:57
*** markmcclain has joined #openstack-infra14:58
clarkbgood morning/afternoon14:58
AJaeger_clarkb: there was only one request for python 2.6 - tooz. I've put this in as additional change.14:58
AJaeger_clarkb: 4pm for me ;)14:58
fungiwell, and jd__'s patch to add 25 back to a lot of the oslo libs they were missing from14:58
clarkbya I see tooz as the child change14:58
fungier, s/25/26/14:58
*** skolekonov has joined #openstack-infra14:58
AJaeger_fungi: yeah, exactly.14:58
clarkbfungi 2 libs and not really missing14:59
AJaeger_That patch to remove them was proposed before further discussions happened.14:59
clarkbits the special concurrenct or whatever test14:59
fungioh, right, most of it's just the new template14:59
AJaeger_and it was for 2 removed - and many more added now ;)14:59
AJaeger_fungi: yeah.14:59
*** thingee has joined #openstack-infra15:00
clarkbbut I havent made up my mind yet ...15:00
fungiit also sets oslo.log gating on py3315:01
fungithough it was already voting in check15:01
fungiso that's more just cleanup i guess15:01
clarkbthis isnt actually adding any new py26 jobs15:01
AJaeger_fungi: exactly15:02
*** mattfarina has joined #openstack-infra15:02
AJaeger_clarkb: it adds back the python jobs removed in my other patch...15:02
clarkboh I didnt see that15:02
fungioh, and oslo.log15:03
*** Masahiro has joined #openstack-infra15:03
clarkbso maybe we should revert the other change15:04
fungiso, yeah, adds py26 jobs for oslo.concurrency and oslo.log, adds py33 to the gate for oslo.log though that was already voting on check15:04
clarkband not have this largee hard to see what is really changing change15:04
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/system-config: Support filtering by owner information
fungiit's raining buckets here. i think someone turned the ocean upside down15:05
*** wuhg has quit IRC15:05
*** wuhg has joined #openstack-infra15:05
AJaeger_Note the two sets of changes - stackforge and oslo libs - are independend of each other, just related by topic15:06
fungiand i don't think they merge conflict15:06
AJaeger_fungi: they shouldn't merge conflict. And we can merge oslo now, just wait until Sunday with the stackforge one...15:07
*** Masahiro has quit IRC15:08
jd__is there a window for our tooz rename/migration to openstack/ ?15:08
clarkbpersonally I am -1 on that chnage15:08
jd__and hi btw, I'm rude :)15:08
clarkbwe should revert the change that was a problem15:08
clarkbbecause I could not understand what was actually fixed in the new change15:08
fungii'm fine with the patch as-is, though i agree separate patches to revert the removal, add teh missing py33 job and templatify py26 would be cleaner15:08
clarkbwhich may or may not be due to being awake at 7am15:09
fungiyeah, i'm already on coffee #2 so things may make more sense to me briefly than they should too15:09
*** tonytan4ever has joined #openstack-infra15:09
*** mmaglana has joined #openstack-infra15:09
AJaeger_jd__: we didn't schedule the tooz rename yet AFAIK15:09
*** thingee has quit IRC15:09
AJaeger_clarkb, fungi: Shall I propose a reversal of and jd__ is adjusting his change as pure cleanup?15:10
*** jcoufal has quit IRC15:10
clarkbAJaeger_: that would be my preference15:11
fungifine here15:11
fungihappy to re-review the decomposition15:11
jd__AJaeger_: I can git revert + my patch and git review send that?15:11
*** ildikov has quit IRC15:11
AJaeger_jd__: let's do the revert quickly, I'll propose it and we can merge quickly...15:12
fungijd__: separating out the py26 templating and the addition of the missing py33 gate job (switching to python3-jobs template for oslo.log) would also probably help15:12
*** AJaeger has quit IRC15:12
fungijd__: makes it more obvious that was intended and not an accident15:13
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Adjust python 2.6 jobs for oslo libraries"
*** AJaeger has joined #openstack-infra15:13
*** mmaglana has quit IRC15:14
AJaeger_clarkb, fungi: since we're discussing oslo - I suggest to merge this one as well, it's a pure rearranging of lines for clarity:
jktI don't think this test log is good,
*** thingee has joined #openstack-infra15:16
*** mestery has joined #openstack-infra15:16
fungiAJaeger_: it will likely merge-conflict with what jd__ is working on so we likely want to either ask him to rebase once it merges or rebase it on what he proposes15:16
clarkbjkt: it looks like the connection object is going away.15:18
clarkbjkt: possibly just a race between test cleanup and gear sending status updates?15:18
clarkbjkt: but I don't know enough about turbo hipsters tests to say15:18
jktclarkb: actually, I have multiple logs, and sometimes this pretty trivial test is getting stuck, that's why I'm looking into it15:18
fungijkt: yeah, that was after gear shutdown, so might just be ignored (though i agree not necessarily very clean)15:18
AJaeger_fungi: agreed. So, we would schedule the rename at next week's meeting? In that case, I suggest we rebase the tooz patch after mine. jd__ shall I do this for you?15:19
jktsuch as this one,
clarkbAJaeger_: ya we should schedule soon since we will between holidays15:19
clarkbif we wait too long it will be hard to schedule again15:20
fungiAJaeger_: clarkb: sounds good15:20
AJaeger_clarkb: so, discuss at next meeting?15:20
clarkbAJaeger_: yup15:20
fungii'm expecting to be mostly afk starting in a few hours, so i'm going to scramble to try and get a few things knocked out15:20
AJaeger_Then let me move this up on the agenda15:20
AJaeger_fungi: for how long?15:21
clarkbfungi: you driving into the mountains?15:21
fungiclarkb: no, have a lot of pies to bake this afternoon, then driving to visit christine's family for the next couple days15:21
*** Ark has joined #openstack-infra15:21
fungiAJaeger_: likely not around much through this weekend15:22
clarkbI am doing stuffing, turkey, and potatoes tomorrow morning15:22
*** Ark is now known as Guest1052915:22
clarkbAJaeger_: it is a US holiday on thursday and typically people have a 4 day weekend15:22
fungiit is the annual gluttony celebration15:22
fungirecovery usually takes a few days15:23
jd__AJaeger: it was already supposed to be scheduled yesterday…15:23
clarkbapparently they are getting snow days today in the NE so 5 day weekend for some15:23
jd__AJaeger: so yeah ASAP :)15:23
AJaeger_fungi: ah, thanks giving - I know about it, just feared you were gone for much longer...15:23
* AJaeger_ already appreciates that my US colleagues are on vacation :)15:24
*** ildikov has joined #openstack-infra15:24
mtreinishclarkb: yes, it's snowing...15:24
AJaeger_jd__: I just updated and moved it to the top15:24
jd__AJaeger: 👍15:25
openstackgerritJulien Danjou proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack
*** david-lyle_afk is now known as david-lyle15:28
*** dimsum__ has quit IRC15:28
*** stevemar has joined #openstack-infra15:28
AJaeger_jd__: I'll base 135215 on top of 137348. Ok?15:28
*** dimsum__ has joined #openstack-infra15:29
jd__AJaeger: ok15:29
sdagueclarkb: well, except for us work from home folks15:30
*** gokrokve has joined #openstack-infra15:30
sdagueit does mean I'll have a shoveling break at some point15:30
sdaguethough my snow blower is also supposed to arrive today15:30
*** Sukhdev has joined #openstack-infra15:31
*** marun has joined #openstack-infra15:31
clarkbsdague: I think you get to roll around in the snow even when you work from home15:31
*** radez is now known as radez_g0n315:32
clarkbryanpetrello: you around? so we can remove py26 from pecan15:32
*** dmsimard_away is now known as dmsimard15:32
clarkbryanpetrello: or if you prefer we can continue running those jobs for you. With the understanding that we really will remove them in about a year15:32
*** dmsimard is now known as dmsimard_away15:32
*** dmsimard_away is now known as dmsimard15:33
sdagueclarkb: - view out my office window15:33
clarkbAJaeger_: In your base change we should remove all of pecan's py26 tests and sqlalchemy migrate's py26 tests15:33
clarkbAJaeger_: we should probably add migrates back in a change since a lot of openstack that does py26 depends on it15:34
clarkbAJaeger_: then hear from ryanpetrello on whether or not he wants us to keep running those jobs15:34
clarkbAJaeger_: does that sound reasonable?15:34
AJaeger_clarkb: yes, sounds reasonable. But that's a different patch set. Will do.15:34
clarkbsdague: ya family in NJ said that school was cancelled yesterday before any snow had fallen15:34
clarkbAJaeger_: ya will need a new patch I think that is ok15:34
*** abhishek has quit IRC15:35
openstackgerritMerged openstack-infra/project-config: Revert "Adjust python 2.6 jobs for oslo libraries"
clarkbsdague: on this side of the country the leaves are finally turning15:36
sdagueclarkb: yeh, they are calling for 8 - 16" here, glad we are not traveling for turkey day15:36
clarkbit has been a weird fall15:36
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Move tooz from Stackforge to OpenStack
sdagueyay! snowblower delivery on it's way15:36
*** ddieterly has joined #openstack-infra15:36
sdagueso I won't be shoveling, I'll be blowing15:36
funginow you get to be that annoying neighbor who runs his snow blower at 5am15:37
clarkbfungi: pretty sure that is how you end up disappeared15:38
sdagueyeh, well last year we got a 16" snow storm while I was in korea on a trip. So my wife put her foot down that for me to get on a plane this winter, there would be a snow blower.15:38
fungithey find you sticking partway out of a melting drift at some point halfway into the spring thaw15:38
*** alexpilotti has joined #openstack-infra15:39
sdaguefungi: they all have them here, so no one cares if you run them early. Honestly, they aren't that loud.15:39
clarkbwe are supposed to get biblical amounts of rain15:39
mtreinishsdague: heh, they're delivering it in the nick of time15:39
clarkbI think one forecast called for a foot of rain in the seattle area. slightly less in portland15:39
sdaguemtreinish: yep15:39
fungiclarkb: we're getting similar here. the parking area and back yard are now under standing water15:40
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove python26 jobs from all stackforge projects
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-add gate-sqlalchemy-migrate-unittests-26sa07
*** otter768 has joined #openstack-infra15:40
AJaeger_clarkb: does these two jobs do what you proposed?15:40
AJaeger_sdague: that's a nice snowy view15:42
ryanpetrelloclarkb: doesn't matter to me15:42
ryanpetrelloif you all want them gone, that's fine w/ me15:42
clarkbryanpetrello: ok we will remove them then15:42
clarkbAJaeger_: +2 on the first one15:42
AJaeger_ryanpetrello: removed as part of  - thanks15:43
clarkbAJaeger_: for sqlalchemy migrate should we run a normal py26 job there?15:43
clarkbAJaeger_: I guess it wasn't there before15:43
sdaguefungi / clarkb - want to +A this - - current and last heat ptl are +1 on it15:43
*** BharatK has joined #openstack-infra15:43
AJaeger_clarkb: I guess we should, let me add one.15:44
*** yamamoto has joined #openstack-infra15:44
*** wenlock has joined #openstack-infra15:44
*** otter768 has quit IRC15:44
*** chandankumar has joined #openstack-infra15:45
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-add gate-sqlalchemy-migrate-unittests-26sa07
AJaeger_clarkb: added ^^^15:45
*** jcoufal has joined #openstack-infra15:46
clarkbAJaeger_: I am going to see if I canfigure out where those py26 jobs were removed15:47
AJaeger_jd__: if you split and update, we can approve quickly15:48
fungimbayer should probably also weigh in on it15:48
AJaeger_clarkb: I removed them as part of my first patch15:48
clarkbAJaeger_: oh right cool15:48
clarkbfungi: I don't think he cares15:48
*** bo_sh1 has joined #openstack-infra15:48
AJaeger_didn't remember ;(15:48
* AJaeger_ updates commit message15:48
clarkbalembic is the tool zzzeek cares about15:48
clarkbbut that stack looks good to me now15:48
jd__AJaeger_: doing15:48
fungiclarkb: good point. the sqlam core group is a grab-bag of volunteers anyway15:49
clarkbfungi: beat me to approving heat slow removal15:49
clarkbfungi: mostly just thinking it is an important piece of many projects particularly in the past as we migrate to alembic15:49
clarkbfungi: so trying to be good and run those tests15:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-add sqlalchemy-migrate python 2.6 jobs
fungiclarkb: yep, i'm totally +2 on keeping py26 jobs there15:50
jd__AJaeger_: I don't think I need to split, now it's just factorizing python26 jobs AFAICS15:50
AJaeger_jd__: you're adding one new job to the gate15:50
AJaeger_don't you?15:51
jd__AJaeger_: not anymore since you reverted15:51
clarkbya the revert fixed the py26 and py33 issues15:51
AJaeger_jd__: ah... Ok, I'll review carefully ;)15:51
jd__AJaeger_: so this patch isn't needed anymore unless you want to use a python26 template15:51
clarkbAJaeger_: ok stack looks good now. As long as we don't make any drastic changes over the rest of the week we should be ready15:51
openstackgerritMerged openstack-infra/project-config: remove heat-slow job and definition
AJaeger_clarkb, sdague, fungi could be approved now....15:51
*** JayJ has joined #openstack-infra15:52
AJaeger_clarkb: thanks for the reviews!15:52
clarkbjd__: the python26 template is a reasonable thing to have. might make it more readable that we actually want python26 on those projects too15:52
jd__ok I'll do that clarkb15:52
*** alla__ has quit IRC15:53
AJaeger_should we have a comment in front of the python26 job that these are deprecated?15:53
yolandaclarkb, mordred, do you have a sec to talk about nodepool upload problem?15:53
clarkbyolanda: sure15:53
clarkbAJaeger_: maybe?15:53
yolandaclarkb, so we detected a problem with our side15:53
yolandarealated to slow image uploads15:53
clarkbAJaeger_: and a note that they have until juno's end of life before removal?15:54
yolandaas our network is not so great, it can take over 2-3 hours to upload an image15:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python 2.6 for tooz
yolandaas it's using providermanager for the upload, it's blocking the rest of tasks15:54
fungiyolanda: oh, ew, that's no good15:54
yolandawe detected an outage today, nearly 3 hours without any new task being processed, until the image upload finished15:54
yolandaand i guess it has happened other times15:54
clarkbweren't we using a glance provider manager?15:55
AJaeger_fungi: regarding - since you have the -2 on it: Will you approve all changes once you're back on Monday?15:55
fungiAJaeger_: absolutely15:55
AJaeger_thanks, fungi15:55
yolandaso my idea was to split that, create a "UploadManager" similar to the ProviderManager, but only performing image uploads15:55
yolandawhat do you think? can you imagine a better approach?15:55
fungiAJaeger_: if you're around monday, ping me once i get online and i'll cram all those through first thing15:55
clarkboh its a glanceclient in the existing provider managers gotcha15:55
jeblairyolanda: or just have the upload command execute directly instead of going into the task manager queue15:56
*** pcrews has joined #openstack-infra15:56
AJaeger_fungi: ok, will try to remember ;)15:56
fungijeblair: as part of the image build thread?15:56
* AJaeger_ should be around15:56
jeblairfungi: yeah, so it will block that thread but nothing else15:56
yolandafungi, image build is another thread actually15:56
yolandathere are 2 steps: image build and image upload15:56
clarkbyolanda: I think nodepool tasks can run async15:56
clarkbyolanda: I would just do it that way15:56
*** tykeal has left #openstack-infra15:57
clarkboh I defer to jeblair15:57
yolandaok that approach is simpler15:58
yolandaimage uploads in provider are fine if they take 5 min, but in our case is quite crazy15:58
*** afazekas has quit IRC15:58
clarkbalso the spin out of image builder nodes would potentially alleviate this issue a bit if you can run those nodes closer to glance15:58
clarkbbut it doesn't fix the problem15:58
yolandacool, i'll file a change for you to review, is quite an urgent issue for us15:59
jeblairclarkb: good point15:59
*** sarob has joined #openstack-infra16:00
yolandayes, our nodepool is not in the same network as glance atm16:00
*** maishsk has quit IRC16:01
mtreinishfungi: if you've got a sec: should be pretty quick16:01
*** pblaho_ has joined #openstack-infra16:01
openstackgerritMerged openstack-infra/project-config: Sort taskflow properly
fungiyolanda: nor ours... they're separated by a gulf of internets (nodepool in rax, glance in hpcloud)16:02
dimsum__fungi: the 'owner' information in status.json finally kicked in (zuul restart?) - finally am able to filter the status page by name/email... thanks -
clarkbfungi: we haven't really had issues with uploads though16:02
clarkbfungi: the entire process is faster than a single snapshot build16:03
fungiclarkb: agreed. we have a fair amount of bandwidth and smallish images16:03
yolandaclarkb, i guess is problem with our connectivity16:03
*** dimsum__ is now known as dims16:03
fungidimsum__: yeah, it got restarted on monday16:03
yolandaand our images are over 5G16:03
clarkbyolanda: ya ours are just under 3GB16:03
*** mikeit has quit IRC16:04
*** pblaho has quit IRC16:04
sdaguefungi: oh, in my list of patches that I think are ready, is the pagination for merge-check (clarkb found a real bug)16:04
sdagueyou had previously +216:04
yolandaour speed can vary from 20 min to 3 hours, depends on proxy really... i've done manual uploads so quick and i had to abort others16:04
AJaeger_clarkb, fungi: I have still two patches to remove the docutils pin - and since weeks open. reviews for this housekeeping patches are welcome.16:04
*** ryanpetrello has quit IRC16:06
mtreinishfungi: thanks16:07
*** rhe00 has left #openstack-infra16:08
*** jerryz has quit IRC16:08
*** bogdando has quit IRC16:09
fungisdague: i'm cool with 134283 but was hoping jeblair might have time to look it over once he wasn't travelling16:09
*** mmaglana has joined #openstack-infra16:10
jeblairyeah, let me look at that since i spent some time with gerrit pagination recently16:10
*** bogdando has joined #openstack-infra16:11
openstackgerritMerged openstack-infra/system-config: Add support to log gearman client to filter on build-queue
*** JayJ has quit IRC16:12
*** Ark has joined #openstack-infra16:12
*** JayJ has joined #openstack-infra16:13
*** patrickeast has joined #openstack-infra16:13
*** Ark is now known as Guest863816:13
*** Guest10529 has quit IRC16:13
*** MarkAtwood has joined #openstack-infra16:13
openstackgerritJulien Danjou proposed openstack-infra/project-config: Use a template for python26 jobs
openstackgerritJulien Danjou proposed openstack-infra/project-config: Move pycadf in Oslo section
*** mmaglana has quit IRC16:15
*** nfedotov has quit IRC16:15
clarkbjeblair: will there be a gear change today with your stuff from yesterday? (I am mostly curious to see what you changed)16:16
jeblairclarkb: i hope so; will start on it in about 2 hours16:16
*** kgiusti has quit IRC16:16
*** pblaho_ is now known as pblaho16:16
*** zigo has quit IRC16:16
*** fabo has joined #openstack-infra16:17
fungii will try to pop back in at some point this evening to review gear improvements16:17
*** JayJ has quit IRC16:17
*** JayJ has joined #openstack-infra16:18
AJaeger_jd__: you templatized all the stackforge jobs as well ;(16:19
AJaeger_that conflicts with my other patches to remove them.16:19
jd__AJaeger_: is it wrong?16:19
jd__whoever goes first win! ;)16:19
sdaguesounds good to me16:19
AJaeger_jd__: see
*** amuller has quit IRC16:20
*** kim__ has joined #openstack-infra16:20
kim__Hi, What cause to "Merge Failed."16:20
AJaeger_jd__: I can rebase if you find two other cores to approve your patch ;)16:20
clarkbkim__: can you link an example?16:21
*** shashankhegde has joined #openstack-infra16:21
*** JayJ has quit IRC16:22
*** mmaglana has joined #openstack-infra16:22
*** JayJ has joined #openstack-infra16:23
*** mmaglana has quit IRC16:23
jd__AJaeger_: 2 cores? I don't have this amount of money! ;)16:23
kim__clarkb: I will describe the problem: When you run a new job of a new commit I noticce that there is a failed message in the gerrit: "Merge Failed." Even when the job passed or failed16:23
clarkbkim__: can you link to an example?16:24
clarkbkim__: there are several ways this can happen and iwthout a concrete example it is hard to sya what caused it16:24
*** ildikov has quit IRC16:24
openstackgerritMerged openstack-infra/system-config: Add a site for advisories
openstackgerritMerged openstack-infra/system-config: Enable SSL in
kim__clarkb: I found one... for example in this commit there are some Merge failed"16:25
clarkbkim__: is "This change depends on a change that failed to merge" the message you are curious about or "Merge Failed"16:26
*** zigo has joined #openstack-infra16:27
clarkbkim__: for the first things it means that a parent change of this change was unable to merge. since this change depends on that parent it cannot merge either16:27
clarkbkim__: for the second it means there is a merge conflict that could not be automatically resolved. You should rebase on the latest version of the target branch and resolve these conflicts manually16:27
*** sarob has quit IRC16:28
jeblairsdague: lgtm thanks16:29
*** gokrokve_ has joined #openstack-infra16:30
*** k4n0 has quit IRC16:30
kim__clarkb: I expect it to be automatically16:30
*** gokrokve has quit IRC16:31
kim__clarkb: I think that the issue can be resolved with the manager parameter in layout.conf16:31
*** ryanpetrello has joined #openstack-infra16:31
*** pblaho_ has joined #openstack-infra16:31
sdagueclarkb: ... I have a testr mystery if you have the ability to help deduce16:31
clarkbkim__: no, these are conflicts that are not able to be automatically resolved16:31
kim__clarkb: It should be manager: dependentPipelineManager..16:31
clarkbkim__: zuul attempts to do a git merge but runs into merge conflicts16:31
clarkbkim__: it is for the gate pipeline16:31
sdaguenova functional tests died in a very odd way that I can't reproduce -
openstackgerritMerged openstack-infra/zuul: implement gerrit pagination in simplequery
kim__clarkb: So..Who should fix this? THe commiter?16:32
clarkbkim__: yes16:33
*** jcoufal has quit IRC16:33
clarkbkim__: the message left by jenkins indicates the steps that should be taken16:33
*** JayJ has quit IRC16:34
openstackgerritJay Faulkner proposed openstack-infra/project-config: Make agent_ssh check job voting for IPA
clarkbsdague: the stderr thing about openssl.cnf seems pretty straightforward16:34
clarkbsdague: not sure this is a testr thing16:34
sdagueclarkb: well, except, it doesn't happen consistently16:34
sdagueand never locally16:34
sdagueand never in past runs of that job16:35
*** pblaho has quit IRC16:35
clarkbsdague: the first thing I would do is grab the subunit log for that run16:35
clarkbsdague: do testr load < subunitlog16:35
openstackgerritJay Faulkner proposed openstack-infra/project-config: Make agent_ssh check job voting for IPA
clarkbthen run testr run --analyze-isolation which does bisection to find conflicting tests16:35
*** sarob has joined #openstack-infra16:35
*** packet has joined #openstack-infra16:35
*** shashankhegde has quit IRC16:36
sdaguehmmm... no subunit -
clarkbyour job isn't copying the file16:36
*** packet has quit IRC16:37
*** kim__ has quit IRC16:38
clarkbsdague: it tries to but the files are not there16:38
clarkbmaybe a PWD thing?16:38
*** shashankhegde has joined #openstack-infra16:39
*** asselin_ has joined #openstack-infra16:43
*** gokrokve_ has quit IRC16:45
*** ryanpetrello_ has joined #openstack-infra16:45
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager
yolandajust initial proposal, i need to test in my instances ^16:48
*** ryanpetrello has quit IRC16:48
*** ryanpetrello_ is now known as ryanpetrello16:48
*** bo_sh1 has quit IRC16:48
*** pblaho_ has quit IRC16:49
*** bo_sh has joined #openstack-infra16:49
dteselkinDoes anyone knows anything about 'refs-updated' event from gerrit stream-events?16:50
dteselkinIt looks like sometimes gerrit sends *a lot* of such messages with 'failed' status.16:50
dteselkinAnd that kills my zuul :(16:51
clarkbdteselkin: what version of gerrit?16:51
*** emagana has joined #openstack-infra16:51
jeblairdteselkin: and i don't know what 'failed' means in this context: ref-updated happens when a change is merged or pushed to a ref (eg 'master' or some other branch); so i don't know why it would emit such an event in the case of any kind of failure16:52
*** belmoreira has quit IRC16:52
*** Masahiro has joined #openstack-infra16:52
clarkbdstufft: doesn't have a status field16:52
*** ildikov has joined #openstack-infra16:53
*** viktors is now known as viktors|afk16:53
dteselkinEvents like
clarkbdteselkin: ^16:54
dteselkinOh, sorry16:54
jeblairoh ref-replicated16:54
*** talluri has joined #openstack-infra16:54
clarkbthats new hence my gerrit version question :)16:55
jeblairdteselkin: i think that zuul should ignore that since it hasn't been programmed to respond to it; you might get a warning in the error log but should be harmless.  what affect is it having on your zuul?16:55
*** changbl has joined #openstack-infra16:56
jeblairi'm guessing you get "Received unrecognized event type '%s' from Gerrit." a lot16:56
*** Masahiro has quit IRC16:56
dteselkinjeblair, the problem is that there are *thousands* of such messages, and they flood internal queue16:57
dstufftclarkb: lol I was really confused there for a moment ;)16:57
jeblairdteselkin: ah, so they get enqueued into the trigger queue and take too much time to process?16:57
dteselkinYep, there are some, but then is just stops further processing16:57
*** talluri_ has joined #openstack-infra16:58
dteselkinYes, they are received by GerritWatcher16:58
dteselkinBut at some point getEvent stops fetching them16:58
jeblairdteselkin: that sounds like a bug :/  we've worked through thousands of events before without problem (it should work through them fairly quickly)16:59
dteselkinGerrit version 2.916:59
jeblairdteselkin: regardless, we have two ways to proceed -- figure out why it gets stuck, or ignore those events16:59
jeblairi lean toward the latter since they don't seem particularly useful17:00
*** talluri has quit IRC17:00
jeblair(and then if your zuul still gets stuck, we'll look into that subsequently)17:00
clarkbin theory you might trigger jobs based on those events17:00
clarkbbut it doesn't seem likely17:00
dteselkinI agree that it's better to ignorre them, as early as possible17:00
clarkbthinking of something like "Oh mirror is up to date let me go run thing against the mirror)17:00
*** asselin_ has quit IRC17:01
dteselkinIs it a good way to add a kind of filter to GerritWatcher?17:01
jeblairclarkb: i suppose we could ignore them now and add them back in later if needed?17:01
clarkbjeblair: yup that works too17:02
fungiperhaps zuul grows a configurable blacklist of event types for efficiency, which defaults to things we know to be potentially noisy and of little value17:02
fungiif it turns out people actually want the flexibility to tweak that17:02
jeblairclarkb: (and if we add it back, maybe with a few changes to make it more efficient to receive thousands of events with little impact :)17:02
*** mmaglana has joined #openstack-infra17:02
jeblairthere's already something that's nearly a whitelist in there...17:03
clarkbin the pipeline managers17:03
jeblairthere's a list of events and mapping to the user attribute (since it changes names sometimes)17:03
jeblairno i meant in the gerrit trigger itself17:03
clarkbwe list the things we are interested in17:04
*** arxcruz has quit IRC17:04
clarkbjeblair: right I am thinking that we could map managers onto that other mapping and ignore everything else17:04
*** dizquierdo has quit IRC17:04
jeblairclarkb: oh, thinking of maybe taking the list of interested events from the config file and generate a whitelist based on that?17:04
*** ashaeron has quit IRC17:04
jeblairthat could work; my idea was to take the existing list of events that the trigger knows about and instead of logging an error and continuing if it hits a new one, just ignore it17:05
jeblaira third option is to make a blacklist17:05
jeblairso we're still partially future-compatible, but ignore "annoying" events like ref-replicated17:05
dteselkinTwo lines from my log shows how fast events are coming17:05
*** ChuckC_ has quit IRC17:05
*** ChuckC has joined #openstack-infra17:06
*** chandankumar has quit IRC17:06
jeblairdteselkin: ah!  tens of thousands!  we've never had a queue that deep, so that's pretty much untested17:06
dteselkinThat because it's a garbage :)17:07
*** MaxV has quit IRC17:07
jeblairdteselkin: did you add that log message?17:08
dteselkinAn updated version ^^^17:08
jeblair(i can't find it in the source)17:08
*** patrickeast has quit IRC17:08
dteselkinYep, I added that to addEvent17:08
*** shashankhegde has quit IRC17:08
*** baoli has quit IRC17:08
dteselkinI'll show17:09
jeblairok, hrm.  i'd rather not add a filter to that class; i think it's the wrong level of abstraction (that actually should be in gerritlib)17:09
jeblairi guess it could be an argument to the constructor17:10
fungii like clarkb's suggestion17:10
fungidynamism based on expressed interest in the config makes for the most user-friendly option i think17:10
clarkbI thought about the gerritism of not sending the +1 in the event if the value doesn't change17:10
clarkbyou know how we can fix that? have jenkins reset votes when testing starts ....17:11
fungiclarkb: yeah except17:11
fungiclarkb: that leaves a comment, so now we're back to "starting check jobs"17:11
jeblairyeah, so if we do that, we'll need to pass the list all the way into the lib/gerrit class17:11
clarkbfungi: ya17:11
clarkbfungi: I don't like the solution but it should work and may explain why it was working until recently17:12
*** ryanpetrello has quit IRC17:12
fungiclarkb: alternatively, trigger the gate pipeline on any voteless comment left by jenkins and then rely on it querying the change to see if it already has a +1 present17:12
fungithat way it will still only get enqueued under the correct circumstances i think17:13
jeblairdteselkin: are you interested in hacking on a fix?17:13
dteselkinjeblair, yes, definitely17:13
*** yamamoto has quit IRC17:13
openstackgerritJulien Danjou proposed openstack-infra/project-config: Move pycadf in Oslo section
openstackgerritJulien Danjou proposed openstack-infra/project-config: Use a template for python26 jobs
clarkbfungi: ya I think that will require zuul code changes though17:14
jeblairdteselkin: it sounds like we think the best approach is to have trigger/ calculate the list of interesting events in the postConfig method, and then somehow pass that into the gerritWatcher in lib/gerrit17:15
jeblairdteselkin: note that will need to update while running to handle live reconfiguration17:15
dteselkinAnd check it somewhere before ?17:15
jeblairdteselkin: yep17:15
*** sergsh is now known as sergsh_away17:15
*** mpaolino has quit IRC17:16
dteselkinTake whitelist from config ?17:16
dteselkinOr hardcode them for now17:17
jeblairdteselkin: whitelist from config would be best i think17:17
jeblairdteselkin: the postConfig method gets called when the config changes17:17
jeblairdteselkin: so you can inspect it then to generate the whitelist17:17
dteselkinOk, thanks. Didn't know that yet.17:18
jeblairdteselkin: if you want a quick fix for your zuul, of course, adding something to ignore that event to is probably the thing :)17:18
*** dizquierdo has joined #openstack-infra17:18
dteselkinjeblair, well, it's better to have universal solution than my own fix :)17:19
*** bo_sh1 has joined #openstack-infra17:19
jeblairdteselkin: definitely; just didn't know how bad your fire was :)17:19
dteselkinOk, thanks a lot for help!17:19
*** mudassirlatif has joined #openstack-infra17:20
jeblairdteselkin: thanks for yours!  and running into this before we did!  :)17:20
*** vhoward has left #openstack-infra17:20
*** bo_sh has quit IRC17:21
*** Longgeek has quit IRC17:24
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager
openstackgerritSean Dague proposed openstack-infra/project-config: the src-neutron-icehouse jobs should only run smoke tests
*** rediskin has joined #openstack-infra17:27
*** atan8 has joined #openstack-infra17:27
*** e0ne has quit IRC17:28
*** marcusvrn has quit IRC17:29
*** chandankumar has joined #openstack-infra17:31
pleia2good morning17:33
*** Sukhdev has quit IRC17:34
*** zul has quit IRC17:34
*** HeOS has quit IRC17:34
*** andreykurilin_ has quit IRC17:34
clarkbjogo: so you have working migrations now on those multinode tests?17:35
*** zul has joined #openstack-infra17:35
*** ryanpetrello has joined #openstack-infra17:35
*** jpich has quit IRC17:35
*** AJaeger_ has quit IRC17:36
openstackgerritClark Boylan proposed openstack-infra/project-config: Add a ci-sandbox project.
*** mudassirlatif has quit IRC17:38
*** kgiusti has joined #openstack-infra17:38
*** chandankumar has quit IRC17:39
*** gokrokve has joined #openstack-infra17:39
*** baoli has joined #openstack-infra17:40
*** otter768 has joined #openstack-infra17:41
*** bdpayne has joined #openstack-infra17:41
*** unicell1 has quit IRC17:43
*** dannywilson has joined #openstack-infra17:44
*** dannywil_ has joined #openstack-infra17:44
*** otter768 has quit IRC17:45
*** talluri_ has quit IRC17:46
*** tonytan4ever has quit IRC17:47
*** dannywilson has quit IRC17:48
clarkbsdague: ok, your job uses which does none of the test runner specific actions to get things like subunit logs17:48
* clarkb writes a hacky thing that we can argue over17:50
*** jistr has quit IRC17:56
*** derekh has quit IRC17:58
*** markus_z has quit IRC17:58
*** patrickeast has joined #openstack-infra17:59
*** talluri has joined #openstack-infra18:00
*** nadya_ has quit IRC18:00
*** ryanpetrello_ has joined #openstack-infra18:01
*** dannywil_ has quit IRC18:03
jrollmordred, clarkb: for rax dib stuff, were you planning on using cloud-init to configure the network or?18:03
clarkbI would assume so but am happy to let mordred figure out the best options18:03
clarkbpersonally I think dhcp should be used18:03
clarkbsince it is 30 years old and was created to solve this problem18:03
*** harlowja_away is now known as harlowja18:04
jrollright, assuming people can't be convinced to make that work :/18:04
*** ryanpetrello has quit IRC18:04
*** ryanpetrello_ is now known as ryanpetrello18:04
*** BharatK has quit IRC18:04
jrollI'm working this from the configdrive route18:05
clarkbif configdrive then cloud init is what I would expect18:06
*** tonytan4ever has joined #openstack-infra18:06
jrollyeah, ok18:06
clarkbdhcp has the benefit of being pretty well understood everywhere. I know how to debug it but debugging cloud init and configdrive sounds like pain18:07
jrollyeah, agree18:08
fungiunfortunately we'll likely need to shoehorn in our own cloud-init so that we don't preinstall its python dependencies as distro packages18:09
jrollI just think it would be much harder to convince someone to change our network that much, then to dump some data in an iso :P18:09
jrollfungi: are we finally running things in virtualenvs?18:10
clarkbjroll: I have no idea what your network looks like but usually you only need like 2 firewall rules and a dhcp server18:10
*** dizquierdo has quit IRC18:10
fungiclarkb: or nova network or neutron which provide you targeted dnsmasq services18:10
jrollclarkb: across many cells across 7 regions/datacenters...18:11
*** bo_sh1 has quit IRC18:11
*** dkliban is now known as dkliban_brb18:11
fungijroll: no, pip installing some things system-wide which conflict with other distro-packaged python-based tools pulling in python libs from distro packages18:11
jrollclarkb: and I don't know if there's other things that for whatever reason depend on instances using statoc configs18:11
clarkbjroll: so I did the build entire datacenters without dhcp thing for a couple years18:11
jrollfungi: ah, right18:11
clarkbjroll: I was also the person stuck with fixing it18:11
clarkbjroll: and it was probably the most worthwhile thing I ever did in that job18:12
*** mmedvede has quit IRC18:12
jrollfungi: we have some builders for custom cloud-init packages, you might be able to fork them and do something similar
*** e0ne has joined #openstack-infra18:13
jrollclarkb: ha, I don't disagree that using dhcp may be better, I just would have no idea where to start talking to people here about doing that :)18:13
JayFNote that the repos ^ that points to has an OnMetal downstream patch in it18:13
*** mmedvede has joined #openstack-infra18:13
*** dtantsur is now known as dtantsur|afk18:13
JayFbut it's easy to remove and make it stock again18:13
mordredno cloud init18:13
mordredI have NO intent on using cloud init18:14
jrollmordred: how will you be configuring networking, then18:14
openstackgerritClark Boylan proposed openstack-infra/project-config: Make run-tox a super set of run-unittests
openstackgerritClark Boylan proposed openstack-infra/project-config: Switch to using everywhere.
openstackgerritClark Boylan proposed openstack-infra/project-config: Delete
fungimordred: initscript that adjusts interface configs early in boot?18:14
clarkbsdague: ^ that will fix you. its a bit more terrible than I like but it fixes this permanently18:14
*** jedimike has quit IRC18:14
JayFUgh, I don't love cloud-init but reinventing the wheel in small scripts over and over isn't always better :(18:14
jrollmordred: better question: do you want the networking data in JSON or debian-interfaces-file formats?18:15
jrollJayF: though they only need to support one or two distros18:15
jrollso not the worst idea18:15
*** baoli has quit IRC18:15
clarkbmordred: so I question that18:15
jrollmordred: (or both formats)18:15
clarkbbecause cloud init solves this problem18:15
clarkband is widely used etc18:15
mordredbut I'm on a phone ... will word vomit later about plan18:15
fungithough i will agree that cloud-init is no fun to debug18:15
mordreddon't care ...json is fine.18:16
jrollthanks, helpful18:16
fungiflemel json18:16
fungiconsume yaml18:16
mordredclarkb: will argue about it later18:17
clarkbmordred: I don't know that I need to argue but its the same thing with dhcp18:17
clarkbmordred: we need to stop being silly18:17
fungijroll: sorry, flamel18:17
clarkband use standard tooling that solves these problem18:17
jrolloh, neat18:17
fungijroll: it's a SpamapS special18:18
*** e0ne has quit IRC18:18
mordredI want dhcp, but can't have nice things18:18
jrollI'm going to provide y'all with json, you can flamel it if you like :P18:18
jeblair is alo cool18:18
*** harlowja has quit IRC18:18
fungijeblair: good point18:18
jeblair(and potentially useful here)18:18
jrollmordred: if you want a preview of the format or whatever, spin up an onmetal server, we do this today18:18
jrollit's in vendor_data18:19
*** harlowja has joined #openstack-infra18:19
mordredI'll look at it in a few, and thanks!!18:20
jrollthe nova spec to match is
jrollnp :)18:20
*** e0ne has joined #openstack-infra18:20
*** emagana has quit IRC18:23
*** emagana has joined #openstack-infra18:23
openstackgerritDarragh Bailey proposed openstack-dev/hacking: Add import check for try/except wrapped imports
jeblairclarkb, fungi: i have working code that is as fast as the current system.  only thing we lose is protection against starvation, but i'm not terribly worried about that18:24
SpamapSugh networks that resist DHCP are like drivers that won't use blinkers IMO .. wasting everyone's time and usually dangerous18:25
jeblairclarkb, fungi: (we don't have that now)18:25
clarkbjeblair: this is asyncio?18:25
jeblairclarkb: yeah18:25
clarkber ++18:25
clarkbjeblair: ya I was fine with it in the original change/patchsets18:25
mordredmtreinish: merged18:25
clarkbjeblair: I think it should be ok18:25
fungijeblair: awesome. sounds good enough to me18:25
jeblairgot some cleanup to do, should be able to push it up soon18:25
clarkbSpamapS: the best part is when you have humans manually configuring nodes and you get conflicting IPs or wrong masks or wrong gateways18:26
mtreinishmordred: you might want to also push a release with it18:26
clarkbSpamapS: debugging that is hard18:26
SpamapSclarkb: yeah it's just ridiculous18:26
mordredmtreinish: that's harder to do from my phone18:26
SpamapSAnd hell we have this whole service that gives you an API for controlling DHCP we call Neutron. There's just no reason to resist DHCP in this case other than "I had a bad experience one time somewhere"18:27
clarkbmordred: you need ubuntu phone so you can push git tags without crazyness off your phone18:27
*** mpavlase has quit IRC18:27
mtreinishmordred: heh, I have faith :)18:27
SpamapSBut yeah, can't have nice things. :-P18:27
mordredclarkb: uhm ....18:27
clarkbmordred: don't take me seriously though18:27
*** tonytan4ever has quit IRC18:27
clarkbhowever do they have that working on N5s yet?18:28
jrollSpamapS: it's a process, we might get there eventually18:28
mtreinishclarkb: I do things like that with connectbot and logging into one of my servers18:28
cody-somervilleCould I bother from fine folks for review on please?18:28
*** emagana has quit IRC18:28
jrollSpamapS: also, dnsmasq explicitly says "for small networks" on their home page... I wouldn't run a whole datacenter on it18:28
clarkbjroll: your neutron networks are typically not large18:29
morganfainbergjroll: where is your sense of adventure? :P18:29
clarkband you run a dnsmasq for portions of your deployment iirc18:29
clarkblike per hypervisor or some such18:29
*** jp_at_hp has quit IRC18:29
jrollclarkb: I'd imagine it would be dhcp for a whole cell18:29
*** e0ne has quit IRC18:29
clarkbcody-somerville: yolanda specifically said its not ready18:29
jrolland I guess it depends on your definition of large :P18:30
clarkbcody-somerville: has that changed?18:30
cody-somervilleyolanda: ^^18:30
yolandaclarkb, sorry, yes, it's ready now18:30
yolandai uploaded patchset 218:30
*** ryanpetrello_ has joined #openstack-infra18:30
SpamapSjroll: The source of that information _MAY_ have been used to many thousands of nodes as a small network. I'd be interested to hear what is "small" to them. :)18:30
*** nadya_ has joined #openstack-infra18:30
jrollSpamapS: right :)18:31
SpamapSand seriously, this is just push vs. pull all over again. :)18:31
jrollI'm not trying to defend our lack of dhcp, btw :)18:31
SpamapSit's the constant pendulum of distributed computing.. "Push is robust, and decentralized.." "Pull is responsive and self-healing."18:31
SpamapSjroll: yeah I can tell that you're just trying to educate us with the realities that you're faced with, and I appreciate that very much.18:32
fungiyolanda: i don't suppose we have unit test coverage over that operation?18:32
clarkbfungi: we do for everything until where the comment says skip if testing18:33
SpamapSclarkb: btw, what part of cloud-init handles networking? Having trouble finding it.18:33
jrollSpamapS: :)18:33
*** ryanpetrello has quit IRC18:33
*** ryanpetrello_ is now known as ryanpetrello18:33
yolandafungi, there is a test, but it doesn't arrive to the glance upload, it assumes fake-image and skips18:33
SpamapSoh I think I just did find it actually18:33
*** wuhg has quit IRC18:33
SpamapSand it looks like it only speaks /e/n/i format?18:34
*** emagana has joined #openstack-infra18:34
*** pradk has joined #openstack-infra18:35
fungiyolanda: right, seeing that. i think i'm okay with the change as it seems unlikely to break anything besides dib uploads anyway. i'm assuming you've functionally tested this, and if it turns out to be broken for us we'll spot it pretty quickly18:35
yolandafungi, yes, i've tested that from my side, it uploaded image as expected18:36
fungimost of it's just cut and paste from the provider manager to the updater18:36
clarkbjeblair: did you want to review before I approve?18:36
yolandayes, mostly to move that outer of taskmanager to avoid the blocking18:36
clarkbI am +2 going to give jeblair a chance to look since he suggested the fix18:39
clarkbbut can approve it in a bit18:39
*** nadya_ has quit IRC18:39
SpamapSclarkb: ok n/m I found it.. so 'network_config' gets set to /e/n/i format. Hm.18:39
*** andreykurilin_ has joined #openstack-infra18:39
clarkbdoes that work on rhel?18:39
*** e0ne has joined #openstack-infra18:40
jrollnetwork config is parsed from debian-interfaces format18:41
*** Masahiro has joined #openstack-infra18:41
jrolland written as whatever format the OS needs18:41
clarkboh its the input side that is /e/n/i18:41
*** harlowja_ has joined #openstack-infra18:41
jrollyeah, it's absolutely insane18:41
jrollwe have some patches to read from a json format a la
*** talluri has quit IRC18:42
*** talluri has joined #openstack-infra18:42
* jroll looks18:42
SpamapSclarkb: yes it works on rhel18:43
SpamapSFunny I wonder if dprince ever saw this code before he wrote os-net-config. :)18:43
*** ivar-lazzaro has joined #openstack-infra18:43
*** changbl has quit IRC18:44
SpamapSjroll: for your entertainment..
jrollsee e.g.
*** e0ne has quit IRC18:44
clarkbwow we have >1 thing to deal with no dhcp18:45
*** harlowja has quit IRC18:45
JayFSpamapS: I mean, I appreciate that tripleo has a custom thing to support that, but cloud-init is what most folks use to do tenant configuration18:45
jrollSpamapS: ha, interesting18:45
JayFand I don't think I'd want to add that as a dep on cloud-init18:45
fungiour project has become so large that we're reinventing the same wheels multiple times18:45
*** rushiagr is now known as rushiagr_away18:45
JayFfungi: I hate reinvented wheels. They never seem to be quite as round.18:45
clarkbthis might be a good time to go get breakfast18:46
* jroll looks at the clock18:46
*** Masahiro has quit IRC18:46
jrollsounds like brunch18:46
* jroll passes clarkb a bloody mary18:46
*** dannywilson has joined #openstack-infra18:48
*** dannywilson has joined #openstack-infra18:48
*** dkliban_brb is now known as dkliban18:50
*** Ryan_Lane has joined #openstack-infra18:50
*** ZZelle_ has joined #openstack-infra18:51
SpamapSclarkb: actually os-net-config is not to deal with no dhcp18:52
SpamapSclarkb: it is to deal with complex device configuration. Bridges, bonding, etc.18:52
jrollSpamapS: so wy not fix cloud-init to do that?18:52
SpamapSjroll: GPLv3 and CLA?18:52
fungiSpamapS: clearly you should have expanded the dhcp spec with additional option codes for those ;)18:53
SpamapSjroll: thats certainly why _I_ am prohibited from working on it. :)18:53
jrollSpamapS: I mean, GPL sucks, but is that enough reason to reinvent the wheel?18:53
* fungi wonders where the "gpl sucks" is coming from18:53
jrollat any rate, we did it for you :P18:53
SpamapSGPLv3 is explicitly forbidden for me without high level mgmt approval because of the patent grant clause.18:53
jrollfungi: s/sucks/not great/18:53
jrollor not preferred, I should say18:54
SpamapSGPL is fine18:54
fungi(i think gplv3 is plenty great for a lot of things, maybe affero clause gpl variants less so)18:54
SpamapSGPLv3 just has thorny bits for large orgs.18:54
openstackgerritJames E. Blair proposed openstack-infra/gear: Use non-blocking IO in server
SpamapSHeat makes it kind of weird to inject extra cloud-init stuff.18:55
SpamapSnot impossible, but kind of weird.18:55
jrollwe're working on that, too :P18:55
SpamapSAlso cloud-init is hard to update18:55
jrollwe're working on making nova do that, based on what neutron tells it about18:55
jeblairSpamapS: i'm going to disagree with you about GPLv3 having thorny bits for large orgs :)18:55
jeblairSpamapS: and i'm not sure how much you want to get into hp politics here, but it's actually really easy for you to work on gplv3 stuff18:56
SpamapSjeblair: large orgs that have a giant trove of patents to cling tightly to?18:56
jeblairSpamapS: yep. send a mail to fontana if you have any questions18:56
SpamapSjeblair: I have already tried once with Ansible and been told explicitly that I am not to patch it.18:56
jeblairSpamapS: i had a long chat with him recently and things are not as bad as we have been told18:56
jeblairSpamapS: when?18:57
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add debootstrap to the needed packages for dib
SpamapSjeblair: thats good to hear. Perhaps OSRB was just rubber stamping my Ansible stuff. :-P18:57
SpamapSjeblair: August?18:57
* SlickNik surfaces on account of SpamapS' colorful vocabulary 18:57
jeblairSpamapS: yeah, that's like forever ago.  :)  an email to fontana on this will be productive.18:57
SpamapShahaa nice18:58
clarkbjeblair: oh wow the diff there is pretty large18:58
clarkbjeblair: do you want to review before Iapprove it?18:58
SpamapSOk time to go expose myself to the ball of fire in the sky for a little while.18:58
fungitbm heads up the hp osrb right?18:58
*** yamahata has quit IRC18:59
fungimartin michlmayr18:59
jeblairclarkb: it's 100 lines less than the last patchset (30% less!)18:59
clarkbjeblair: :)18:59
clarkbI diffed against the old patchset19:00
clarkband that diff is large maybe I hsould just read it from scratch19:00
fungihe's tbm in #debian-devel19:00
*** johnthetubaguy is now known as zz_johnthetubagu19:01
jeblairclarkb: oh, yeah, so much got removed.  if you want the quick walkthrough of the inter-patch diff it's: don't use an instance variable in the read method, git rid of all the threads in the server, handle all server io in the poll loop19:01
*** thingee has quit IRC19:02
*** reed has quit IRC19:02
clarkbjeblair: any interest in throwing that on nodepool-dev.o.o then possibly cutting over to it before tomorrow?19:02
clarkbprobably not a great idea considering holiday but maybe no one will be around to notice if it breaks again19:02
*** teran has joined #openstack-infra19:03
*** talluri has quit IRC19:04
*** thingee has joined #openstack-infra19:04
jeblairclarkb: we could do that19:05
clarkbjeblair: looking at the read method you just set a reference to the instance variable and use that? and that helped that much that is interesting19:06
*** dmsimard is now known as dmsimard_away19:06
openstackgerritJames E. Blair proposed openstack-infra/gear: Use non-blocking IO in server
jeblairclarkb: 25x improvement.  i love it when you have to think like a c programmer to work in python :/19:07
clarkbya I guess that removes a couple layers of indirection for access19:07
jeblair(latest ps is pep8 fix)19:08
clarkbjeblair: when you set input_buffer to b'' why not use packet = b'' instead?19:08
clarkboh wait that doesn't do the right thing derp19:08
clarkbbecause python19:08
jeblairyeah, think in pointers.  :)19:08
openstackgerritElizabeth K. Joseph proposed openstack-infra/project-config: Create project for puppet-zanata module
clarkbjeblair: so how does that handle the case where you get the EAGAIN error when reading a normal packet19:09
clarkbjeblair: it appears that it would be dropped on the floor?19:09
jeblairclarkb: it's saved in self.input_buffer19:10
clarkb+= does the right thing?19:10
* clarkb tests19:10
clarkbmy test says it doesn't19:11
clarkbs1 = 'foo' ; s2 = s1 ; s2 += 'bar' ; print s1 ; print s219:11
clarkbbecause strings are immutable you get back a new reference from +=19:11
openstackgerritMerged openstack-infra/storyboard: Add details to event subscription
jeblairright; this briefly did a stint as a list (where that works)19:12
clarkbjeblair: I think if you do self.input_buffer = packet in a finally block that will cover this19:12
jeblairprobably the easiest fix19:12
*** TimO has joined #openstack-infra19:12
*** TimO is now known as Guest9573419:13
clarkbbecause the returns are handled19:13
*** andreykurilin_ has quit IRC19:13
clarkbso only exceptions need that19:13
jeblairclarkb: finally runs on returns too, so i'll actually just switch everything to packet19:14
jeblaireasier to read that way19:14
clarkbjeblair: ++19:14
swestonasselin: mmedvede the next version of the script is posted.  will be testing it again today.  now, all of the subtree splits are done on a single cloned merge repo.  this will be much easier for us to track and update the history on the entire repository.19:14
mmedvedesweston: hehe, wrong room19:15
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Remove stable compatibility jobs from capped libs
*** bdpayne_ has joined #openstack-infra19:15
*** banix has joined #openstack-infra19:16
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Remove stable compatibility jobs from capped libs
mmedvedesweston: that is cool. Were does it maintain the new repositories?19:16
Guest95734i was wondering what is the best way to find out/request that my dedicated service account can now be upgraded to be approved for voting in OpenStack?19:16
openstackgerritJames E. Blair proposed openstack-infra/gear: Use non-blocking IO in server
swestonmmedvede: right? embarrassing ..19:17
clarkbGuest95734: you should speak to the project you would like to report voting on. If they are happy to do that then we can update your permissions. there is a while blurb on this at
*** rmcall has joined #openstack-infra19:18
jeblairclarkb:  ^ stress tested that; shows no adverse effects19:18
*** thingee has quit IRC19:18
*** bdpayne has quit IRC19:19
asselinsweston, ok cool19:19
asselinsweston, yes, was wonder what meeting I'm missing :)19:19
swestonmmedvede: it's all in github, there's a new merge repository called puppet-modules, you can find it in my organization.19:19
*** yfried|afk is now known as yfried19:19
Guest95734thanks, and have a great Thanksgiving too!19:20
*** Guest95734 has quit IRC19:21
swestonasselin: hehe, yeah .. am a bit short on sleep today ...19:21
*** ddieterly has quit IRC19:21
jeblairclarkb, fungi: i'll let you mull on that for a bit while I check out that ball of fire SpamapS was on about, and if you like it we can maybe try it out after lunch.  or save it for next week.  :)19:21
clarkbI am all for trying it today19:22
fungijeblair: yep, already reading through it19:22
clarkbbut fungi is mostly afk iirc19:22
clarkbbut ya reading it now19:22
nibalizerclarkb: fungi jeblair so we don't have to deal with this now19:22
nibalizerbut 137233 basically means that the templaing we've got in place for puppet.conf is failsauce19:22
jeblairfungi's 'mostly afk' is more present than some people's active presence :)19:22
*** reed has joined #openstack-infra19:23
*** thingee has joined #openstack-infra19:23
nibalizerso my advice would be to get through thanksgiving weekend by 'not touching it' but then we need to look at the pupeptmaster's puppet.conf and figure out why its working at all19:23
* fungi insists he is not a bot19:23
jeblairnibalizer: sounds like a plan; want to wip that so we don't get too excited about merging it?19:24
funginibalizer: also thanks for the heads up to defer further jenga moves until after we're well-rested19:24
jeblair(i mean, it should be safe, but "should be safe" + "don't understand why it works" == double fail)19:25
clarkbjeblair: I am now trying to understand admin_request in the read method19:25
nibalizerya what it means is that currently we're doing if (float(undefined_variable) < 3.6 ): { stuff }19:26
*** MaxV has joined #openstack-infra19:26
swestonmostly afk, that's a good away message ... also like intermittently afk19:26
nibalizerand of course in ruby float(undefined_variable) = 0.0 because ruby19:26
clarkbjeblair: and it appears that we push onto the stack after we pop it?19:26
clarkbjeblair: I am not sure how that is working just yet19:26
openstackgerritElizabeth K. Joseph proposed openstack-infra/infra-specs: Add Migrate to Zanata spec
mmedvedesweston: this one?
swestonmmedvede: yup19:27
*** thingee has quit IRC19:28
clarkbjeblair: [].pop(0) is an error19:28
jeblairclarkb: where?19:28
*** bitblt has joined #openstack-infra19:28
*** banix has quit IRC19:28
clarkbjeblair: in readPacket()19:29
jeblairclarkb: as soon as we know we're working on an admin request, we pop; and if we have popped and get EAGAIN, we push19:29
clarkbjeblair: the check is if admin is None: if byte else: admin = True admin_request = self._getAdminRequest() which does a pop19:29
clarkbjeblair: right but what does the initial push?19:29
*** yfried is now known as yfried|afk19:29
*** changbl has joined #openstack-infra19:29
clarkbthat list is initialized to [] and I don't see a putAdminRequest happening before the getAdminRequst19:30
*** bitblt has left #openstack-infra19:30
jeblairclarkb: ah -- sending an admin request19:30
*** adalbas has quit IRC19:30
*** thingee has joined #openstack-infra19:30
clarkboh so that code only runs client side?19:31
*** AJaeger_ has joined #openstack-infra19:31
jeblairclarkb: yeah, it's different on client vs server19:31
jeblairclarkb: on client, this is handling a response to a previously sent admin request19:31
jeblairclarkb: thus it will pop from a previously externally populated list of admin requests that have been sent19:31
*** emagana has quit IRC19:31
clarkbjeblair: but the server won't see that as an admin_request in readPacket()?19:31
*** tnovacik has joined #openstack-infra19:32
jeblairclarkb: server side, _getAdminRequest instantiates a new admin request object19:32
*** emagana has joined #openstack-infra19:32
jeblairclarkb: i think there's a small bug here though19:32
jeblairin that if the server gets eagain while reading it, it will put the admin request on the stack19:32
clarkbI don't see where server side does that19:33
jeblairclarkb: the same read method19:33
clarkb_getAdminRequest just pops off that stack which should be empty19:33
clarkbjeblair: right but something must push onto the stack before we pop19:33
jeblairclarkb: server connections override _getAdminRequest19:33
*** AlexF has quit IRC19:34
openstackgerritAndrey Pavlov proposed openstack-infra/jenkins-job-builder: Add parameter to trigger_parameterized_builds
*** ddieterly has joined #openstack-infra19:34
jeblairclarkb: the server-side bug is a memory leak -- if it gets eagain, it will _putAdminRequest which will push it on a stack that is unused by the server; so i think the fix is a noop _putAdminRequest19:34
clarkboh I see my grep was failing me19:34
clarkbjeblair: so server side should override putAdminRequest as well19:35
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Remove stable compatibility jobs from capped libs
jeblairclarkb: so in short -- client side behavior should be pop/push from list (where previous submissions of admin requests go in the list); server side should ignore the list completely and only produce new admin request objects to return when they have been fully received19:35
clarkbdhellmann: ^ I think you do, you just want to limit the branches where those compat jobs run to the things they support19:36
clarkbjeblair: yup19:36
clarkbjeblair: are you going to push a new patchset with putServerRequest on server side overridden?19:36
dhellmannclarkb: ?19:36
clarkbdhellmann: you don't want to remove those jobs. you just want ot restrict where they run19:36
*** emagana has quit IRC19:36
clarkbdhellmann: so that you can do bugfixes19:36
*** koolhead17 has quit IRC19:36
*** banix has joined #openstack-infra19:36
jeblairclarkb: yep19:36
dhellmannclarkb: where would I want them to run, though?19:37
clarkbdhellmann: under the version caps19:37
*** koolhead17 has joined #openstack-infra19:37
dhellmannclarkb: those jobs currently test the master version of the library against the stable version of the apps -- that's no longer a supported configuration19:37
openstackgerritJames E. Blair proposed openstack-infra/gear: Use non-blocking IO in server
clarkbdhellmann: no the test the proposed change against that stable version of the apps19:38
clarkbdhellmann: this is still supported for things under the version caps19:38
clarkbsay<0.7.0 is your cap19:38
clarkband you discover 0.6.0 is broken19:38
clarkbsecurit vulnerability19:38
clarkbyou then push a patch that will become 0.6.1. you want to test that against say icehouse19:38
clarkbthat change doesn't happen on master it happens on another branch19:39
dhellmannyes, that's right19:39
clarkbyou want any of those changes to be tested against the server apps stable branches that they support19:39
dhellmannI think I'm content to use unit tests for that19:39
dhellmannsdague: ^^19:40
sdagueyeh, I'm assuming that these sec backports are going to be extremely small19:40
dhellmannand equally infrequent19:40
dhellmannif that proves not to be the case, we can add some jobs, but I'm not sure we need them at this point19:41
sdagueright, agreed19:41
*** ihrachyshka has quit IRC19:41
* clarkb notes that we have broken all the things a couple times in just the last week19:41
*** koolhead17 has quit IRC19:41
*** otter768 has joined #openstack-infra19:42
sdagueso what this really begs for is a thing that has been asked for from time to time19:42
*** emagana has joined #openstack-infra19:42
sdaguewhich is the ability to adhoc a job19:42
*** ihrachyshka has joined #openstack-infra19:43
sdaguelike when lifeless is getting ready to release new testr, and really wants to build a thing that tests vs. a branch of testr19:43
clarkbyou can do that19:44
sdaguebut it's close to snow blowing time so I'm not sure I have a fully formed thought there19:44
*** Ryan_Lane has quit IRC19:44
sdagueclarkb: how?19:44
clarkbits ugly19:44
clarkbyou push a devstack/d-g change that does it19:44
sdaguethat won't let you test against nova unit tests, if you know that's the thing you usually break19:44
*** Ryan_Lane has joined #openstack-infra19:45
clarkbsdague: so its a similar mechanism for nova unittests19:45
clarkbpush a change to nova that installs the version ofthe thing you want from git19:45
clarkbor bzr/svn/hg etc19:45
sdaguesure, but it's such a system level hack. I wonder if we could make it a simple thing.19:46
sdagueanyway, it's a half formed thought19:46
*** Sukhdev has joined #openstack-infra19:46
*** otter768 has quit IRC19:46
*** HeOS has joined #openstack-infra19:46
clarkbI think the trouble with making it a "simple" thing is that you would have to somehow bypass the mechanism I describe which is how we always install software19:46
clarkbIMO you want to test it as normal19:47
lifelessclarkb: or you make the mechanism aware19:49
lifelessclarkb: I think the success criteria needs to be that a single action is taken that runs all jobs that use that upstream thing, with the proposed upstream thing19:49
sdaguelifeless: aren't you supposed to be on holiday :)19:50
lifelessI think there are a number of related things needed to do that19:50
*** thingee has quit IRC19:50
lifelesssdague: I am19:50
lifelesssdague: someone said my name :)19:50
*** Ryan_Lane has quit IRC19:50
sdaguelifeless: well you have a valid use case here :)19:50
clarkbright but how do you know that some person over there wherever there is has a thing you should test19:51
clarkbyou don't really. So pushing a change to test it is reasonable19:51
lifelessFWIW the proposed? landed? plumbing in toci to make a localpypi index for things from git is one way of avoiding special cases in the code19:51
*** ecosta has joined #openstack-infra19:51
lifelessclarkb: pushing as a mechanism doesn't bother me19:51
lifelessclarkb: pushing 40 patches to test one thing does19:51
lifelessclarkb: because frankly, too much effort.19:52
clarkblifeless: the toci thing still needs to be told what the composition is19:52
clarkbwhich is the $patches19:52
clarkbif you want a meta project that has all the tests in the world against it that you can trigger via a single change that is probably doable19:52
sdagueclarkb: yeh, I think that would be the shape of it19:53
clarkbbut you still need to say here is a ref go test it19:53
sdaguebut that would be the format of the proposed patch19:53
clarkbjeblair: current gear patchset lgtm19:55
*** dims has quit IRC19:56
kevinbentonin our 3rd party CI system I ran into an issue with a mismatch between oslo.middleware and the global requirements on stable/icehouse tests19:57
kevinbentonand i made this change which seemed to fix it
kevinbentonbut it seems like that might not be the real issue19:57
*** thingee has joined #openstack-infra19:57
kevinbentonsince the OpenStack CI doesn't seem to be suffering the same issue19:57
dhellmannkevinbenton: see
krotscheckAny infra cores interested in helping clear out some minor puppet-storyboard patches?,n,z19:58
clarkbkrotscheck: the rspec job is expected to fail right?19:58
kevinbentondhellmann: ah, so that should supercede my patch then, right?19:58
*** emagana has quit IRC19:59
dhellmannclarkb: right now I'm trying to figure out how to make check-requirements-integration-dsvm branch-aware, but I think I don't understand enough about the preconditions for that job.19:59
krotscheckclarkb: Yep, nibalizer pointed that out last week.19:59
dhellmannkevinbenton: yeah, we don't want to pin things in devstack itself19:59
kevinbentondhellmann: sounds good. thanks for the fast response19:59
*** emagana has joined #openstack-infra19:59
sdagueok, need to go deal with snow for a while. Back in a bit.20:00
dhellmannkevinbenton: glad to help!20:00
clarkbdhellmann: it shold already be branch aware20:00
*** ryanpetrello has quit IRC20:00
clarkbdhellmann: maye I don't understand what you mean by branch aware but it should check stable/X against stable/X20:00
dhellmannclarkb: well, it's currently failing in stable/juno because it is checking the master version of requirements in oslo.messaging20:00
clarkbdhellmann: what should it be checking in oslo.messaging?20:01
*** radez_g0n3 is now known as radez20:01
dhellmannclarkb: stable/juno, or nothing20:01
dhellmannwell, when it's running against the stable/juno version of the global requirements it should check the stable/juno version of messaging20:01
clarkbthere is no stable/juno in oslo.messaging20:01
clarkbif there was I think it would work properly20:01
dhellmannthat job failed on this patch against
dhellmannah, ok20:02
kevinbentonclarkb: i thought the branches were explicit in stackrc?20:02
clarkbkevinbenton: thats completely separate20:02
*** ryanpetrello has joined #openstack-infra20:02
clarkbkevinbenton: this is just doing comparison of requirements files20:02
clarkbkevinbenton: and installing things into virtualenvs to ensure they at least install together20:03
*** emagana has quit IRC20:03
kevinbentonclarkb: oh okay. so that process checks branches as well?20:03
dhellmannclarkb: how do I make a new stable/juno branch? is that a gerrit thing?20:04
ecostahello, i am new here. Is there a reason for jenkins not running for a review, even after all of its dependencies are met?20:04
*** ihrachyshka has quit IRC20:04
*** nadya_ has joined #openstack-infra20:04
dhellmannclarkb: now that I think of it, I think ttx has always done that for me so I might not have permission20:04
clarkbdhellmann: ya in the gerrit gui under the project you can create it20:04
clarkbdhellmann: let me get you a link and ou can see if you have perms. if not I can do it for you20:04
clarkbecosta: can you link to the review?20:05
dhellmannclarkb: yeah, I don't see anything on that page that looks like a "make a branch" button20:05
fungithat change was definitely uploaded while zuul was offline20:06
clarkbdhellmann: there should be a create branch button near the bottom20:06
*** e0ne has joined #openstack-infra20:06
fungiecosta: a recheck comment should get it going20:06
clarkbdhellmann: if you give me a ref to base stable/juno on I can create that branch20:06
dhellmannclarkb: no, I think I'm not on the approved list20:06
dhellmannclarkb: 6ea3b12492e86f9e8d109fc3490cc4d3a0edd8b620:06
ecostaclarkb: fungi: ok, thanks!20:07
*** thingee has quit IRC20:07
clarkbdhellmann: done, you should see it if you refresh that page20:07
dhellmannclarkb: got it, thanks!20:07
*** aysyd has quit IRC20:07
*** AJaeger_ has quit IRC20:07
*** thingee has joined #openstack-infra20:08
dhellmannclarkb: I want to recheck my stable/juno requirements change now. Should I wait some period of time for that branch to be synced somewhere or is it safe to recheck right away?20:08
*** ihrachyshka has joined #openstack-infra20:08
*** wenlock1 has joined #openstack-infra20:09
fungiclarkb: jeblair: 128754 lgtm. not sure what the plan was for landing it in production so didn't approve20:09
clarkbdhellmann: that is a good question let me check the git mirror really quickly20:09
dhellmannclarkb: thanks20:09
clarkbdhellmann: it should sync almost immediately20:09
clarkbdhellmann: has it so you should be fine to recheck20:09
dhellmannclarkb: cool20:09
*** ryanpetrello has quit IRC20:10
dhellmannoh, bother, did the recheck detection get updated to allow extra comments or do I still have to just say "recheck" by itself?20:11
*** ldnunes has quit IRC20:12
fungidhellmann: as long as the first line starts with the word recheck it should take it20:12
*** hashar has joined #openstack-infra20:12
fungiadditional words after recheck are just ignored now20:12
dhellmannfungi: ok, that's what I thought but it didn't seem to work20:12
clarkbfungi: found a bug in the groups vhost stuff. patch incoming shortly20:12
dhellmannfungi: oh, weird, I somehow commented on an old changeset so that's why it didn't work20:13
*** thingee has quit IRC20:13
fungidhellmann: yep, has to be the most recent20:13
openstackgerritMerged openstack-infra/elastic-recheck: Add e-r query for bug 1372670
uvirtbotLaunchpad bug 1372670 in nova "libvirtError: operation failed: cannot read cputime for domain" [High,Confirmed]
dhellmannfungi: yeah, that makes sense; user error20:14
fungiclarkb: d'oh! yep, failed to start apache20:14
clarkbfungi: ya20:14
*** thingee has joined #openstack-infra20:14
fungiclarkb: SSLCertificateChainFile takes one argument, SSL Server CA Certificate Chain file (`/path/to/file' - PEM encoded)20:15
clarkbnibalizer: how do I compare against undef in an erb file?20:15
* fungi sighs20:15
*** mpavlase has joined #openstack-infra20:16
clarkbhrm I think I found a table in puppet docs20:16
*** e0ne has quit IRC20:17
*** ecosta has quit IRC20:18
*** denis_makogon has quit IRC20:19
*** dmakogon_ has joined #openstack-infra20:19
*** thingee has quit IRC20:19
*** ryanpetrello has joined #openstack-infra20:20
*** andreykurilin_ has joined #openstack-infra20:20
*** thingee has joined #openstack-infra20:21
openstackgerritClark Boylan proposed openstack-infra/system-config: Fix the setting of the chain file in drupal vhost
clarkbfungi: ^20:22
fungiclarkb: yep, looks right20:23
nibalizerclarkb: gj20:23
fungii totally missed there was no explicit default being set for it20:24
clarkbok I am going to eat and stuff then come back ready to do nodepool again20:24
fungii'm going to beat on the manage-projects fix by approving a few more project creation patches to see if i can get the bug to resurface20:25
*** sarob has quit IRC20:28
*** enikanorov_ has quit IRC20:28
*** enikanorov_ has joined #openstack-infra20:28
*** sarob has joined #openstack-infra20:29
*** Masahiro has joined #openstack-infra20:30
*** ociuhandu has quit IRC20:31
mrmartinfungi: hi. I guess the SSL patch had a little bug that killed apache :) I guess this line is wrong: because we are not checking the value against undef.20:31
fungimrmartin: see just moments ago where clarkb proposed a patch to fix that. should be merging shortly20:32
mrmartinoh perfect, I also found that meanwhile20:32
openstackgerritMerged openstack-infra/system-config: Fix the setting of the chain file in drupal vhost
*** sarob has quit IRC20:34
*** Masahiro has quit IRC20:34
*** AlexF has joined #openstack-infra20:34
fungimrmartin: ^ there it is. should be live in a few minutes20:35
*** baoli has joined #openstack-infra20:35
mrmartinfungi: it was super-fast20:36
*** rpodolyaka1 has quit IRC20:36
openstackgerritMerged openstack-infra/project-config: Create initial ansible deployment repository
*** andreykurilin_ has quit IRC20:37
fungiokay, hopefully i'll know shortly if that ^ succeeded in manage-projects20:37
*** nadya_ has quit IRC20:39
*** Bart has joined #openstack-infra20:42
*** Bart___ has quit IRC20:42
*** belmoreira has joined #openstack-infra20:42
*** lttrl has quit IRC20:48
openstackgerritSpencer Krum proposed openstack-infra/infra-specs: Spec to refactor puppet code in openstack_project
*** bswartz has quit IRC20:54
lifelessclarkb: sdague: meta-repo, aka requirements?20:55
sdagueclarkb: so, do you have thoughts on this way of cross testing that the python libs definition in devstack with project-config -
*** nadya_ has joined #openstack-infra20:56
sdaguelifeless: I think more meta than that. A little more like if project-config actually ran the proposed tests in your diff when you proposed to it20:57
clarkbwhich should be a lot more possible when we ditch jenkins20:58
clarkbsince each worker should have a mostly complete view of the world20:58
clarkbsdague: for cross testing like that I think you just run those tess on project-config changes and on devstack changes20:58
clarkbsdague: you can limit it to the layout file on the project-config side20:58
sdagueclarkb: so my test assumes that you update project-config first20:58
clarkbsdague: I think thats ok20:59
sdagueI'm ok with devstack wedging if someone does a bad in project-config20:59
sdagueas it's easy enough to disable (the tests being in tree)20:59
clarkboh in that case just run it against devtsack20:59
lifelesssdague: if requirements changes trigger everything today (they do, right?) and we want to know when they change that everything will work with the new dep (thats the goal right?) then I don't think we need more meta20:59
sdaguemostly the fact that I'm cloning20:59
lifelessOTOH I'm just gasbagging, so whatevs, whomever does it can make it work20:59
clarkblifeless: no requirements changes trigger very little today20:59
sdaguelifeless: they don't trigger everything, they trigger a small static set20:59
* dhellmann wishes this requirements check job had access to wheels for all of these things it is pip installing21:00
lifelessanyhow, running21:00
fungiclarkb: good news. stackforge/os-ansible-deployment seems to have created successfully, and it's pretty far down the layout.yaml so would have likely run into the timeout i think21:00
clarkbdhellmann: it does21:00
clarkbunless we ripped that out21:00
clarkbwhich we probably did rip out beacuse pypi-mirror is dead21:00
dhellmannclarkb: it's building pyyaml over and over, so I don't know21:00
*** AlexF has quit IRC21:01
*** baoli has quit IRC21:01
clarkbfungi: fyi I manually ran puppet on groups-dev and it seems to work now21:01
clarkbdhellmann: ya I think pypi-mirror getting ripped out meant no more wheels21:02
*** baoli has joined #openstack-infra21:02
sdagueclarkb: that's running in a devstack, right?21:02
fungiclarkb: mrmartin: yep, looks like it's working now21:03
jeblairclarkb, fungi: back21:03
*** AlexF has joined #openstack-infra21:03
clarkbsdague: it uses devstack to install the C deps21:03
clarkbsdague: but it shuts down devstack iirc21:03
sdagueclarkb: ok21:03
fungii'm disappearing for a bit, and then testing some more project creation changes when i get back21:03
sdagueI was thinking that adding working devpi install into devstack might simplify a bunch of things21:04
jeblairclarkb, fungi: etherpad for testing gear stuff:21:04
mrmartinseems to be:
clarkbjeblair: I am ready to try things in just a minute21:04
jeblairclarkb, fungi:
*** esker has joined #openstack-infra21:04
*** esker has quit IRC21:04
sdaguebut, anyway, I need to go run and get beer for tomorrow while we have a break in the snow.21:04
clarkbsdague: I dunno. I still contend devpi exists so that you can have your forks and private code in a pip installable place21:04
clarkbsdague: which really isn't a use case we have anywhere21:04
fungicool, when i return in a little while i'll catch this program already in progress ;)21:04
sdagueclarkb: you can also prepopulate it's cache with wheel builds21:04
*** esker has joined #openstack-infra21:05
clarkbsdague: we can do that on bandersnatch too21:05
clarkbor alongside bandersnatch21:05
sdagueok, so lets do that then :)21:05
*** atan8 has quit IRC21:05
dhellmannclarkb: could you make a stable/juno branch of oslo.vmware on commit 589ba439cf9be506ceafc8ad195fb4260b58a0d8 for me please?21:05
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Enable color support based on tty and config
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/nodepool: Add REST API to Nodepool
dhellmannclarkb: wait, wrong commit21:05
dhellmannclarkb: use 1661a0af8ce14edbfb00fed0a30a3b9135517d9321:06
jeblairclarkb: oh, the puppet dns change hasn't merged?21:06
*** emagana has joined #openstack-infra21:06
clarkbjeblair: not yet21:07
*** pc_m has joined #openstack-infra21:07
jeblairclarkb: let's merge that now, since the point of that was to make it easy to do this21:07
clarkbjeblair: ok you want to do that? fungi has +2'd it21:07
jeblairhave a link?21:07
clarkbits in the etherpad21:07
*** pc_m has quit IRC21:08
*** pc_m has joined #openstack-infra21:09
*** e0ne has joined #openstack-infra21:09
*** atan8 has joined #openstack-infra21:10
*** Sukhdev has quit IRC21:10
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support copyartifact build selector param
jeblairclarkb: okay, i think that's the preliminaries.  let's take a short intermission while that change works its way through and resume in a bit (should be like 20-30 mins)21:12
*** weshay has quit IRC21:12
*** e0ne has quit IRC21:13
*** achanda has joined #openstack-infra21:15
*** thingee has quit IRC21:15
*** radez is now known as radez_g0n321:17
openstackgerritMerged openstack-infra/system-config: Allow traffic from nodepool-dev
*** thingee has joined #openstack-infra21:17
*** Sukhdev has joined #openstack-infra21:17
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support copyartifact build selector param
*** andreykurilin_ has joined #openstack-infra21:19
*** achanda has quit IRC21:20
*** pradk has quit IRC21:23
dhellmannclarkb: it looks like you're in the middle of something with jeblair, but when you have a free minute would you let me know?21:23
*** baoli has quit IRC21:24
clarkbdhellmann: I am free for the next 10 minutes or so21:24
*** emagana has quit IRC21:24
*** atan8 has quit IRC21:24
*** baoli has joined #openstack-infra21:24
dhellmannclarkb: cool. I ran into another branch-related issue with that requirements job with oslo.vmware this time.21:24
*** emagana has joined #openstack-infra21:24
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Basic folder support
clarkboh that request ya I haven't created the branch yet21:25
clarkbdhellmann: you want stable/juno based on 1661a0af8ce14edbfb00fed0a30a3b9135517d93 ?21:25
dhellmannclarkb: yeah, I was actually going to ask about what I need to do to get permissions to make branches myself. Is that an infra thing, a release team thing, or something else?21:25
dhellmannclarkb: yes, that's the right tag21:25
dhellmannI suspect there will be a couple more of these, and I'd like to go back and retroactively branch all of the libs so we're consistent21:26
clarkbits a release managers thing21:26
dhellmannok, I'll talk to ttx about it next week then21:26
*** nadya_ has quit IRC21:26
clarkbso I think adam_g, apevec, sdague, and ttx can do it21:26
clarkbthen infra can because hax21:26
dhellmannI need to make sure he's aware of this new process thing for us anyway, since I think we decided to only create branches when we needed them, but now we'll need them at the end of each cycle21:26
*** bhunter71 has quit IRC21:27
*** atan8 has joined #openstack-infra21:27
clarkbbranch created21:27
*** emagana_ has joined #openstack-infra21:27
*** emagana has quit IRC21:27
dhellmannclarkb: on the wheel thing, it sounds like bandersnatch can do that for us somehow, is that right?21:28
clarkbdhellmann: not really. What I meant to say is we could host wheels like we used to alongside the bandersnatch mirror21:29
clarkbdhellmann: so we have we could also have
clarkbthen use something to fill that in21:29
clarkbmaybe resurrect a subset of pypi-mirror's features21:29
dhellmanna post-merge job on the requirements list, I would think21:29
clarkbI am sure mordred has opinions n this21:29
clarkbdhellmann: ya that would be one way of doing it but it would miss any upstream updates21:30
clarkbmore likely a daily cron or something that uses requireemnts as the base and builds wheels on trusty, precise, centos, fedora et al then rsyncs everywhere21:30
*** amitpp has joined #openstack-infra21:30
openstackgerritMerged openstack-infra/project-config: Ignore playground directories for translation
*** yfried|afk is now known as yfried21:31
*** emagana has joined #openstack-infra21:31
*** emagana_ has quit IRC21:31
*** weshay has joined #openstack-infra21:31
*** ryanpetrello has quit IRC21:32
*** Bart has quit IRC21:33
*** Bart has joined #openstack-infra21:33
jeblairclarkb: (afs)21:34
jeblair(which is next on my list once we unblock the dib stuff)21:34
*** rfolco has quit IRC21:34
jeblairclarkb: i don't see the firewall changes having taken effect on jenkins\d+21:36
openstackgerritMerged openstack-infra/project-config: Use storyboard for all infra puppet modules
*** JayJ has joined #openstack-infra21:36
clarkboh ya afs21:37
clarkbjeblair: hrm let me look at puppetboard21:37
clarkb doesn;t look like jenkins01 has reported back yet21:37
clarkbjeblair: so maybe ansible is still running21:37
*** thingee has quit IRC21:38
jeblairclarkb: it is; i was expecting something from the :20-:30 run21:38
jheskethHey, if anybody has a chance to create me an initial gerrit group member (!/story/2000029) that'd be awesome :-)21:38
clarkbjeblair: it runs every 15 minutes21:38
clarkbso the :17 merge was just too late21:39
nibalizercan i just say that the system-config and project-config split was the best idea ever21:39
*** kgiusti has quit IRC21:39
jeblairclarkb: oh i thought it was 1021:39
nibalizerevery time i mess with our hp version that is pre-split i am so amazed21:39
jeblairjhesketh: will do21:39
*** thingee has joined #openstack-infra21:39
clarkbjeblair: I got it :)21:39
jeblairclarkb, fungi: do we still want people to open bugs for initial group membership?21:40
jeblair(i don't)21:40
jheskethjeblair: Thanks :-)21:40
jeblairjhesketh: someone already did that21:40
jheskethyou guys are fast!21:41
clarkbjeblair: I personally prefer being pinged and just getting it done, maybe do email for cases when timezones don't overlap21:41
clarkbor ENOIRC21:41
*** yfried is now known as yfried|afk21:41
jeblairclarkb: okay, jenkins02 has the new iptables rule21:42
*** kgiusti has joined #openstack-infra21:42
clarkbhrm the rules say nodepool.o.o twice21:43
*** otter768 has joined #openstack-infra21:43
clarkbis it looking at the reverse record?21:43
* clarkb tests from the new trusty node21:43
jeblairclarkb: yeah, the actual rule is ip based21:43
clarkbI can telnet to jenkins01:8888 so it seems happy21:43
jeblairclarkb: so it does a forward lookup when setting the rule21:43
jeblairclarkb: and if you list rules, it does a reverse lookup to be pretty21:44
jeblairclarkb: sudo iptables -L -n21:44
jeblairclarkb: and you'll get the numeric version21:44
clarkblooks like all of the jenkinses have updated according to puppetboard21:44
jeblairclarkb: shall i do zuul while you do the nodepools?21:45
*** wenlock1 has quit IRC21:45
jeblairclarkb: ready?21:45
clarkbone sec21:46
*** Guest8638 has quit IRC21:46
clarkbok I am ready let me know when zuul is down and I will take down old nodepool21:46
jeblairclarkb: zuul is stopped21:46
clarkbold nodepool service is stopped21:47
clarkbif I start new nodepool it will wait until gear is up. will you run an out of band geard?21:47
clarkb(I am ready to start new nodepool service when you are)21:47
jeblairclarkb: geard is running21:47
clarkbok starting new nodepool now21:47
*** wenlock has quit IRC21:48
*** otter768 has quit IRC21:48
*** achanda has joined #openstack-infra21:48
clarkbit is doing stuff21:48
*** dkliban is now known as dkliban_biab21:48
jeblairclarkb: so actually we want to wait until all old nodepool nodes are deleted, right?21:49
jeblairclarkb: before starting zuul?21:49
clarkbjeblair: I don't think it is required21:49
clarkbit will help with quota limits though21:49
clarkbI am starting the deletions now21:49
*** achanda has quit IRC21:49
*** banix has quit IRC21:49
jeblairclarkb: we might start running jobs on them though21:49
*** jungleboyj has quit IRC21:49
clarkboh right21:50
jeblairclarkb: ah, so actually it's the ready nodes we care about21:50
jeblairclarkb: so i guess once all the old ready's are deleted we can start zuul21:50
clarkbthe deletion process is running with 10 "workers"21:50
*** mattfarina has quit IRC21:50
*** MaxV has quit IRC21:51
*** bhunter71 has joined #openstack-infra21:51
*** kgiusti has quit IRC21:51
*** ihrachyshka has quit IRC21:52
*** fandi has joined #openstack-infra21:53
*** matt__ is now known as mattoliverau21:53
clarkbjeblair: I spun up a second set of deletions to focus on ready nodes21:54
clarkbso those should go away relatively quickly21:54
jeblairnew nodepool got a TimeoutError, but i have geard running in a terminal and it does not seem to be backlogged21:54
clarkbjeblair: iirc old nodepool had one or two right after switching over. Maybe that is a separate issue?21:55
*** hdd has joined #openstack-infra21:56
jeblairclarkb: maybe?  the function list is taking only a few seconds, but maybe if a lot of nodes came on line around the same time it could get starved enough...21:57
*** dannywilson has quit IRC21:58
*** amitpp has quit IRC21:58
*** dannywilson has joined #openstack-infra21:59
jeblairclarkb: all the ready nodes are gone ;shall i start zuul?22:00
clarkbyup go for it22:00
*** melwitt has joined #openstack-infra22:01
jeblairclarkb: started and queue resumed22:02
*** thingee has quit IRC22:02
jeblairthere are some not_registered jobs22:02
*** patrickeast_ has joined #openstack-infra22:02
jeblairclarkb: so maybe we need to rebuild the devstack-precise image here?22:03
clarkblet me look22:03
clarkbwe are building one in iad right now22:04
clarkba node not image. and if iad on new nodepool has same problem as old nodepool then yes. let me go ahead and delete the existing image in iad and force it to rebuild22:04
jeblairclarkb: before you do that22:04
* clarkb waits22:04
jeblairclarkb: let's make sure that we are currently seeing build failures for it22:04
*** patrickeast has quit IRC22:04
*** patrickeast_ is now known as patrickeast22:04
*** thingee has joined #openstack-infra22:05
*** liam__ has joined #openstack-infra22:06
*** liam__ has quit IRC22:06
clarkbit is still going22:06
jeblairclarkb: we launched one in dfw before this and it failed with error state22:06
jeblairclarkb: so it's probably worth deleting the dfw image?22:06
clarkbjeblair: sure let me do that22:06
jeblairclarkb: but lets wait and see about iad22:06
ayoung  made it through check, but never ended up in a gate queue.  We have a current issue with stackforge stuff?22:08
*** AlexF has quit IRC22:09
clarkbayoung: it got caught in our nodepool service migration (which included a zuul restart)22:09
ayoungclarkb, should I do another reverify?  If so, what bug#?22:09
clarkbjeblair: the iad node just switched to delete22:10
clarkbayoung: no bug number22:10
ayoungreverify no bug22:10
clarkbwe stopped requiring bug numbers a while back22:10
ayoungwe can do that?22:10
clarkbayoung: just `recheck`22:10
clarkbthe bug numbers individuals supplied were far too inaccurate to be useful data22:10
ayoungits at +2.  Will recheck kick it into the merge queue?22:10
*** banix has joined #openstack-infra22:10
clarkbwe now focus on elastic recheck instead22:10
jeblairclarkb: it has a +1 from jenkins so i assume we'll need to yank the jenkins vote?22:10
clarkbjeblair: oh right22:11
clarkbor someone can reapprove22:11
jeblairi yanked it22:11
clarkbayoung: if you recheck it now it will do the correct thing.22:11
ayoungso just "recheck"  now?22:11
ayoungor "recheck no bug"22:12
ayoungor does it matter22:12
clarkbayoung: just `recheck`22:12
clarkbayoung: the bug stuff is no longer important (it is still accepted but not required)22:12
*** banix has quit IRC22:12
jeblairayoung: doesn't matter.  you could even do "recheck this got caught up in a service restart" too if you want :)22:12
ayoungFIRE IN THE... ahem.  Sorry.22:12
*** alexpilotti_ has joined #openstack-infra22:12
*** alexpilotti_ has quit IRC22:13
*** thingee has quit IRC22:13
ayoungwere there a  couple of those nodepool migration type events?  This happened twice to this patch.22:13
clarkbhrm now we are building no devstack-precise nodes22:15
clarkbjeblair: and we just had a timeout22:15
*** alexpilotti has quit IRC22:15
jeblairclarkb: i'll look into the timeout22:15
clarkbok looks like we just had allocations go through so its not 100% of the time22:16
clarkbgah its building on iad again and I didn't manage to delete images there22:16
* clarkb does that now22:16
clarkbayoung: yes we ran into trouble on monday22:17
ayoungthanks for the help22:17
*** thingee has joined #openstack-infra22:17
*** Masahiro has joined #openstack-infra22:18
*** hdd has quit IRC22:21
jeblairclarkb: it's currently taking 1-4 seconds to execute the 'status' command from nodepool-dev... i'm going to throw out the idea that maybe the 200 threads it's running may be contributing to starving it on the client side.22:21
clarkbjeblair: that could be22:22
clarkbespecially since they all sit on the same cpu22:22
*** banix has joined #openstack-infra22:22
jeblairi've got status running in a loop; if there's a geard performance problem, it should show up there around the same time as a timeouterror22:23
*** Masahiro has quit IRC22:23
clarkbValueError: invalid literal for int() with base 10: '0build:gate-xstatic-jsencrypt-python33:py3k-precise'22:23
clarkbthat looks like a nodepool bug22:23
jeblairclarkb: or a gear bug22:24
*** koolhead17 has joined #openstack-infra22:24
jeblairclarkb: if it reassembled a packet wrong22:24
*** hdd has joined #openstack-infra22:24
clarkbit only happened once22:24
*** pc_m has quit IRC22:24
*** rlandy has quit IRC22:25
*** banix has quit IRC22:25
*** esp has joined #openstack-infra22:26
clarkbother than not getting devstack-precise images to build it seems to be mostly working22:27
jeblairzuul looks pretty happy22:27
clarkbthere are allocation grants for devstack-precise but no images building. I think that means we are stuck in a rate limited queue22:28
jeblairanother timeouterror22:28
openstackgerritMichael Krotscheck proposed openstack-infra/infra-specs: StoryBoard Streaming API specification
*** mrmartin has quit IRC22:29
jeblairi just stopped my test program to update it :(  oops22:29
*** koolhead17 has quit IRC22:29
clarkbit looked like the admin requset ran right afterwards though22:29
clarkblike it failed then went through22:30
*** jungleboyj has joined #openstack-infra22:30
sdaguelooks like some unregistered fails in the gate now22:34
clarkbsdague: yup22:34
jeblairi've taken a note of them and will re-enqueue them when the images are available22:34
clarkbnodepool still refuses to actually build that allocation though22:35
jeblairclarkb: i started a new version of my test script that does the same parsing that nodepool does22:35
*** atan8 has quit IRC22:35
jogoclarkb: yes live migration is working now22:35
jeblairclarkb: yeah, it wants to build it on dfw; i'm not sure why it wants to do that considering there's no current image there22:35
jogoclarkb: (was on a plane for the last 5 hours)22:36
clarkbjeblair: huh I wonder if it sees the building image as valid enough22:36
hasharhey folks!22:37
clarkbthat image is almost done building though22:37
jeblairhashar: hi!22:37
clarkbso hopefully we get moving real soon now22:37
hasharis there a doc for storyboard that explains its markup?  I would like to paste a stacktrace with a <pre>  :]22:37
krotscheckhashar: There is no markup in storyboard.22:37
krotscheckYou are welcome to add it :)22:38
hasharkrotscheck: thanks for confirming at least!22:38
jeblairis there a story about that?22:38
clarkbdownloading the openvz image22:38
clarkbsdague: what can we do about killing that with fire?22:38
clarkbI don't think openvz is goin to happen. docker wins22:38
hasharzaro: I have filled a story about the Jenkins Gearman plugin maintaining executors in a deadlock!/story/2000030  . Was long overdue sorry.22:40
jeblairkrotscheck: erm, i just got logged out while doing a search22:40
hasharjeblair: while you are around, I found out zuul-cloner does not support being triggered by a ref-update event ( zuul.model.Ref ). It would complains about missing parameters such as ZUUL_BRANCH22:41
jeblairhashar: ah, we should test/fix that, as well as a timer event22:41
hasharjeblair: would get a patch for it soonish since it is blocking me.   Might impact whatever job you have22:41
jeblairi know i'll be running into the timer event issue myself soon (need that to replace devstack-gate)22:41
fungiback and catching up on what i've missed22:42
*** weshay has quit IRC22:42
*** banix has joined #openstack-infra22:42
hasharand I am not sure how to handle the branch fallback for a release tag22:43
hashari.e. you have branches 1.x and 2.x   , a tag for 1.0.1, we would want to fallback to 1.x probably22:43
clarkbhashar: a tag can discover what branches it is on22:44
sdagueclarkb: yeh... we can do some image cleanup in devstack post holiday, there are a few things we can clean up now I think22:44
clarkbhashar: maybe do that then pick the first one?22:44
jeblairhashar: in our use case, we have no use for a fallback22:44
sdaguedid the openvz one fail, or it's just annoying?22:44
clarkbsdague: its just annoying22:44
clarkbsdague: it may eventually fail but I am still witing on it22:44
jeblairclarkb: but in this case we're dealing with a project without the tag22:44
jeblairclarkb, hashar: so i think maybe either fallback to master or explicitly error may be the best options22:45
sdaguethough with the cirros images, maybe we should cache them in infra22:45
clarkbjeblair: oh right in other projects22:45
clarkbsdague: we do. this is that caching22:45
*** ociuhandu has joined #openstack-infra22:45
clarkbdib makes it much smarter and is why I am working very hard to make dib go22:45
*** alexpilotti has joined #openstack-infra22:45
hasharclarkb: if you have the magic git command to figure out the branch of a tag, I am willing to learn :]22:46
*** Ryan_Lane has joined #openstack-infra22:46
fungijeblair: clarkb: regarding bugs for group membership, i think i originally suggested it as a signaling mechanism for people who don't irc, but ultimately that may be roughly nobody (within an error margin of we don't care)22:47
*** EmilienM has quit IRC22:47
hasharat worth, we could have a python function that attempts to craft a branch name out of the tag (i.e. tag '1.0.1' -> branch '1.x' ) but that is awkward22:47
clarkbhashar: git branch --contains22:47
clarkbsays the google22:47
jeblairhashar: what is your use case for a fallback in ref-updated?22:47
*** EmilienM has joined #openstack-infra22:47
*** banix has quit IRC22:49
hasharjeblair: for mediawiki we maintain old versions. Currently we are preparing 1.24.0  out of branch REL1_2422:50
*** achanda has joined #openstack-infra22:50
hasharand we back port security patches to old version, for example 1.19.24  . So the tag 1.19.24 would need the repo to fallback to REL1_1922:50
*** stevemar has quit IRC22:50
clarkboh hey we are now on to a vmware image22:51
*** jamielennox|away is now known as jamielennox22:51
jeblairhashar: what does your ref-updated job do?22:51
clarkbwow that was fast now on to fedora22:51
hasharclarkb: yeah that works :-] thank you!22:52
clarkbimage is snapshotting now in rax-dfw22:53
clarkbI think22:53
fungiclarkb: jeblair: so after the last two nodepool cut-overs we similarly saw it repeatedly fail to build devstack-precise nodes in rax-iad22:53
*** dannywilson has quit IRC22:53
fungidifferent images each time too22:54
clarkbfungi: Ithink that was related to rax issues that are supposed to be fixed now according to johnthetubaguy22:54
bknudsonI think we're close on -- now getting NOT_REGISTERED22:54
clarkbor at least mitigated22:54
hasharjeblair: they create tarballs for the version released, so we probably don't need multiples repositories there22:54
hasharjeblair: and publish the documentation for the version. Ditto we might not need multiple repos22:54
fungiwe weren't getting image-related errors when i was digging into it though, just nova boot ending with the instance in error state22:55
*** belmoreira has quit IRC22:55
*** yfried|afk is now known as yfried22:55
*** achanda has quit IRC22:55
fungithe image issues were different... nova boot would respond with something like "requested image is not active"22:55
hasharjeblair: I am just thinking out loud sorry. Will follow up next week once I have some code / proof of concept22:55
fungiand i think those are the ones johnthetubaguy said should be solved now22:56
clarkbfungi: I got the impression it was all related but I may be wrong22:56
fungianyway, looks like i missed all the fun? any cleanup i can help with still?22:56
clarkbwell it looks like nodepool may not be booting enough nodes right now?22:56
*** Ark has joined #openstack-infra22:56
fungihaving a look to see if anything looks obviously wrong to me22:57
*** Ark is now known as Guest5107522:57
clarkbjeblair: ^22:57
fungi2014-11-26 22:28:46,686 ERROR nodepool.GearmanClient: Exception while listing functions22:57
fungithat was half an hour ago22:58
jeblairfungi: yeah, we've seen a couple TimeoutErrors and one error in parsing the output (!)22:58
*** banix has joined #openstack-infra22:58
jeblairi have a test running in a loop on the same host to try to catch it outside of nodepool22:58
fungiahh, yeah the misidentifed octal value22:58
fungithat one was pretty amusing22:59
*** dannywilson has joined #openstack-infra22:59
clarkbit seems to have gotten us into a place where we only boot min ready nodes22:59
hasharalso I found a rather nasty bug in Zuul that would make it keep changes in its queue indefinitely whenever it lost connection with Gearman22:59
fungii'm analyzing the demand details22:59
*** dannywilson has quit IRC22:59
hashar  <-- working fine for me for a few weeks now:]22:59
*** dannywilson has joined #openstack-infra23:00
jeblairclarkb, fungi: i suspect this may be related:
jeblairclarkb, fungi: probably we got stuck after that parse error23:00
*** Guest51075 has quit IRC23:01
fungihighly likely because i don't see it reporting demand info23:01
clarkbjeblair: I thought that was applied on nodepool-dev?23:01
jeblairhrm lemme check23:02
clarkbrax-dfw image is ready now23:02
jeblairclarkb: oh, so it is.  :/23:02
clarkbya its installed according to pip freeze23:02
fungibut yeah, the allocation requests show a demand of 0 for all labels23:02
jeblairclarkb: ah, but the exception happened elsewhere23:02
*** jungleboyj has quit IRC23:03
clarkbdevstack-precise node building on rax-dfw now23:03
*** jungleboyj has joined #openstack-infra23:03
jeblairclarkb, fungi: i'm going to disable puppet on nodepool and configure some more verbose logging23:04
fungijeblair: sound like a plan23:04
*** yfried is now known as yfried|afk23:05
jeblairrestarting nodepool23:05
*** ddieterly has quit IRC23:05
fungiyeah, i find no non-zero deficits after that parse error, but before it there was evidence of non-zero deficit calculations23:06
*** ddieterly has joined #openstack-infra23:06
*** MarkAtwood has quit IRC23:08
*** camunoz_ has joined #openstack-infra23:09
*** camunoz_ has quit IRC23:09
*** camunoz has quit IRC23:09
clarkbthe deficits still seem to be 023:10
*** SumitNaiksatam has quit IRC23:10
fungiafter some more careful parsing of the log, i do see a little (one here, one there) in the log after that traceback, but not the values i would expect23:10
*** SumitNaiksatam has joined #openstack-infra23:10
*** ddieterly has quit IRC23:11
*** kgiusti has joined #openstack-infra23:11
jeblairhrm, there must be something else going on...23:11
*** patrickeast has quit IRC23:11
jeblairthe zuul job queue is quite high23:11
clarkbsomething server side maybe?23:12
*** ryanpetrello has joined #openstack-infra23:12
clarkbwe have a devstack-precise node now fwiw23:14
clarkbzm01 definitely seems like its stuck23:14
fungi96 established gearman connections on zuul, according to netstat. that seems relatively high?23:14
clarkbfungi: there is one per jenkins slave23:15
jeblairfungi: there's one for each worker, so that's about right right now23:15
fungioh, got it23:15
*** dannywilson has quit IRC23:16
clarkblooks like zm01 stalled slightly before we restarted nodepool so I don't think that is related23:16
jeblairclarkb, fungi: okay, i'd like to restart zuul and zm01 with verbose debugging23:17
fungitcpdump shows no packets on its gearman connection (sniffing at both ends of the socket)23:17
*** Sukhdev has quit IRC23:17
jeblairsee what we can see there, and probably shortly thereafter, start the rollback process23:17
*** dannywilson has joined #openstack-infra23:17
clarkbjeblair: rgr23:18
fungiokie dokey23:18
funginow i see that socket with traffic on it23:19
jeblairjust restarted zm01 with debugging23:19
*** ryanpetrello has quit IRC23:19
jeblairstopped zuul23:19
openstackgerritMichael Krotscheck proposed openstack-infra/infra-specs: StoryBoard Permissions Specifications
fungiyep, different source port for the zm01 end obviously23:19
fungijeblair: worth noting, right after you restarted zm01, some merges were performed and a couple of jobs started23:20
fungiso that seemed to have dislodged whatever was hung23:20
clarkbwe are now building all the nodes23:20
jeblairfungi: yeah, i was hoping that it would just start one and stop again :/23:20
*** JayJ has quit IRC23:21
jeblairzuul is up; re-enquing23:21
fungiblammo. that's a lot of gearman ;)23:21
jeblairyeah, and i think of it as sparse :)23:22
*** andreykurilin_ has quit IRC23:22
fungiwell, this is a relatively slow time23:22
*** bswartz has joined #openstack-infra23:23
clarkbdid that unstick the other mergers? /me looks23:24
clarkbyup looks like 06 moved (I had checked it previously)23:24
jeblairhrm, 01 looks idle23:25
clarkbya 06 also looks like it may have gotten stuck23:25
clarkbso maybe we have the data you wanted?23:25
jeblairoh it just moved again23:26
jeblairi think the mergers may have worked through the queue and are actually just genuinely idle now23:26
*** gokrokve has quit IRC23:27
fungithough it looks like we still haven't gotten any devstack-precise workers to build, hence the not_registered23:27
clarkbfungi: we had one23:28
fungiwell, at least no new ones since the last zuul restart23:28
clarkboh right only unused nodes will register jobs23:28
clarkbwell we have some queued23:28
fungiin a variety of providers/regions now that there's a calculated non-minimum deficit23:29
fungiso, yeah, should be solved in short order23:29
fungino new gearmanclient timeouts in nodepool-dev's nodepoold yet23:33
clarkbI wonder if we are stuck again though23:33
clarkbthe status graph isn't a massive spike like I expected23:33
clarkband no nodes building23:33
*** jungleboyj has quit IRC23:33
jeblairclarkb: i think that just changed23:34
clarkboh wow yup23:34
fungithe graph may be behind23:34
fungi822 nodes in the list23:34
jeblairnodepool was behind at the time23:34
clarkbfungi: well I checked manually too23:34
clarkbbuilding list was empty23:34
clarkbthen like 20 seconds later it was massive23:34
fungiahh, i checked right after it skyrocketed i guess23:34
jeblairclarkb: okay now i'm starting to wonder if the mergers are stuck again23:35
clarkbjeblair: zm06 has not moved since ~14 minutes ago23:36
*** camunoz has joined #openstack-infra23:36
clarkbaccording to zuul staus list we have newer queued changes than that23:36
fungiand the pending count for merger:merge is rising23:36
jeblair2014-11-26 23:28:26,606 DEBUG gear.ServerConnection: Queuing packet to <gear.ServerConnection 0x1786950 name: Zuul Merger host: port: 44269>: <gear.Packet 0x7fc5b6115950 type: NOOP>23:36
jeblairyep, it wanted to send a noop that zm01 did not receive23:37
*** resker has joined #openstack-infra23:38
*** resker has quit IRC23:38
*** fandi has quit IRC23:38
*** camunoz is now known as camunoz_mtg23:39
*** hdd has quit IRC23:40
clarkbjeblair: should I start rollback prep?23:41
*** shashankhegde has joined #openstack-infra23:41
jeblairclarkb: i'd like another minute to see if i can narrow this down further, but we should do that soon23:42
*** banix has quit IRC23:42
*** esker has quit IRC23:42
clarkbsounds good23:42
*** SumitNaiksatam has quit IRC23:42
*** banix has joined #openstack-infra23:42
rediskinanteaya: hi there23:44
*** otter768 has joined #openstack-infra23:44
*** baoli has quit IRC23:44
rediskinanteaya: i heard you can help with gerrit account for third-party ci23:44
*** baoli has joined #openstack-infra23:45
rediskinanteaya: i sent email a week ago, but still no response T_T23:45
jeblairfungi: any chance you have a tcpdump between zuul and zm01?23:45
clarkbrediskin: we are currently trying to rework the process23:45
fungijeblair: not one from the time it hung23:45
fungii had stopped it prior to then, and restarted it after you commented23:46
clarkbrediskin: unfortunately we have been very busy this week dealin with fires and its prompted us to step up work on making this more self service23:46
rediskinclarkb: does this mean i will wait for a month? -_-23:46
fungijeblair: though if you want to try to retrigger the hang, i can set up to get full-size packets recorded at both ends23:46
clarkbrediskin: hopefully not that long, you could help us review the changes and otherwise participate in the planningaround that23:46
rediskinclarkb: ok, thanks23:47
jeblairfungi, clarkb: another run would help, but i think we should go ahead and roll back23:47
clarkbok I can stop new nodepool now23:47
clarkbanything you want to do before I do that?23:48
jeblairhopefully i can stare at that bit of code and guess at what might have gone wrong, maybe reproduce23:48
jeblairclarkb: nope, go ahead23:48
fungilet's do that for now i guess23:48
jeblairi'll stop zuul23:48
fungii can start deleting nodepool-dev nodes23:48
fungiready nodes first i guess23:49
*** otter768 has quit IRC23:49
clarkbfungi: I just did it concurrently23:49
clarkbit will fail to delete some nodes because they are already deleted but thats fine23:49
*** SumitNaiksatam has joined #openstack-infra23:49
clarkbI have the commands all ready though23:49
fungii'll do it in parallel but have it delete the ready nodes before the others23:49
jeblairi have enabled puppet and had it reset the config files on nodepool-dev, zuul, and zm0123:50
clarkbwhen the ready nodes drops to 0 I will start old nodepool23:50
jeblairi'll have a standalone geard in a minute23:51
*** rediskin has left #openstack-infra23:51
jeblairokay, gear has been downgraded on zuul to the locally patched version (with the 300 sec timeout) and geard is running standalone; ready for old nodepool23:53
jeblairrestarted zm01 with the old log config23:53
*** hashar has quit IRC23:54
clarkbwe are down to 28 ready nodes23:54
clarkbfrom ~8023:55
fungithese last 10 are pesky23:56
*** banix has quit IRC23:56
fungidfw and ord really don't like to take on api calls23:56
clarkbshould we just delete them from the jenkins masters?23:57
fungithey're starting to dwindle now23:57
*** marun has quit IRC23:57
fungithere we are23:58
fungiall gone23:59
fungiworking on the non-ready nodes now23:59
clarkbcool I am starting old nodepool now23:59
clarkbjeblair: ready?23:59

Generated by 2.14.0 by Marius Gedminas - find it at!