Thursday, 2014-06-05

*** maxbit has quit IRC00:02
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Install newer requests package for twine
fungijhesketh: ^00:03
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Move variables in manifests/site.pp to top of file
*** camunoz has quit IRC00:04
jheskethfungi: LGTM :-)00:05
*** signed8bit has quit IRC00:05
openstackgerritJoe Gordon proposed a change to openstack-dev/hacking: Make import checks properly detect blank lines and comments
*** maxbit has joined #openstack-infra00:06
*** salv-orlando has quit IRC00:07
*** Ivo has quit IRC00:07
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Add templated uploading jobs for doc8 project
*** signed8bit has joined #openstack-infra00:09
*** signed8bit has quit IRC00:10
*** praneshp_ has joined #openstack-infra00:10
*** wenlock has quit IRC00:10
*** signed8bit has joined #openstack-infra00:11
*** praneshp has quit IRC00:11
*** praneshp_ is now known as praneshp00:11
*** signed8bit has quit IRC00:11
*** dims__ has joined #openstack-infra00:12
*** signed8bit has joined #openstack-infra00:12
*** jaypipes has joined #openstack-infra00:13
*** mmaglana has quit IRC00:13
*** dims_ has quit IRC00:14
*** dims__ is now known as dims00:15
clarkbsdague: failed in the gate00:16
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Bug 1324932 is a duplicate of 1286635
uvirtbotLaunchpad bug 1324932 in openstack-ci "devstack setup fails with "E: There are problems and -y was used without --force-yes" (dup-of: 1286635)" [Undecided,New]
uvirtbotLaunchpad bug 1286635 in openstack-ci ""Packages cannot be authenticated" when doing an apt-get in the gate" [Undecided,Won't fix]
clarkbso did 9775700:16
lifelessclarkb: saddddface00:18
mtreinishclarkb: the 97757 fail looks like a new one, I'm working on a query00:19
openstackgerritJoshua Harlow proposed a change to openstack/requirements: Updated zake now that zake 0.0.20 is released
clarkbmtreinish: thanks00:19
*** signed8bit has quit IRC00:21
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Suppress the graph for bug 1260654
uvirtbotLaunchpad bug 1260654 in openstack-ci "Could not initialize class jenkins.model.Jenkins$MasterComputer" [Critical,Fix released]
*** apevec has quit IRC00:24
openstackgerritA change was merged to openstack-infra/config: Install newer requests package for twine
*** ramashri has joined #openstack-infra00:26
*** zhiyan_ is now known as zhiyan00:27
openstackgerritA change was merged to openstack-infra/elastic-recheck: Bug 1324932 is a duplicate of 1286635
uvirtbotLaunchpad bug 1324932 in openstack-ci "devstack setup fails with "E: There are problems and -y was used without --force-yes" (dup-of: 1286635)" [Undecided,New]
uvirtbotLaunchpad bug 1286635 in openstack-ci ""Packages cannot be authenticated" when doing an apt-get in the gate" [Undecided,Won't fix]
*** yamahata has joined #openstack-infra00:30
harlowjahi guys, is it possible to get checked, it has 2 +2's and then i had to rebase and adjust 1 line (nothing else changed)00:31
*** unicell has joined #openstack-infra00:33
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Use the correct name for apache
*** UtahDave has quit IRC00:36
*** maxbit has quit IRC00:36
*** maxbit has joined #openstack-infra00:36
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1304184
uvirtbotLaunchpad bug 1304184 in nova "instance stuck into rebuild state when nova-compute unexpected restart" [Undecided,In progress]
*** annegentle has quit IRC00:40
*** mwagner_lap has joined #openstack-infra00:40
*** jhesketh has quit IRC00:41
*** homeless has quit IRC00:46
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1304184
uvirtbotLaunchpad bug 1304184 in nova "instance stuck into rebuild state when nova-compute unexpected restart" [Undecided,In progress]
*** ramashri has quit IRC00:48
*** rwsu has quit IRC00:51
*** gokrokve has joined #openstack-infra00:54
*** aconrad has quit IRC01:00
*** jhesketh_ has joined #openstack-infra01:00
*** jhesketh_ has quit IRC01:01
*** jhesketh has joined #openstack-infra01:01
*** mrodden has quit IRC01:01
*** camunoz has joined #openstack-infra01:03
*** mriedem has joined #openstack-infra01:04
*** maxbit has quit IRC01:04
*** maxbit has joined #openstack-infra01:05
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1304184
uvirtbotLaunchpad bug 1304184 in nova "instance stuck into rebuild state when nova-compute unexpected restart" [Undecided,In progress]
*** amcrn has quit IRC01:07
*** asettle has quit IRC01:13
*** asettle has joined #openstack-infra01:14
*** maxbit has quit IRC01:15
*** maxbit has joined #openstack-infra01:16
*** yaguang has joined #openstack-infra01:19
*** maxbit has quit IRC01:24
*** maxbit has joined #openstack-infra01:25
*** aconrad has joined #openstack-infra01:25
*** crc32 has quit IRC01:25
*** yamahata has quit IRC01:26
*** yamahata has joined #openstack-infra01:27
jogocan that be bumped to top of gate?01:27
clarkbit was already bumped01:27
clarkband I need to make dinner01:27
jogoclarkb: it failed out once01:27
*** maxbit has quit IRC01:27
jogoclarkb: because of
clarkbright so I think we need to reprioritize01:28
clarkbget fixes for actual fails in first01:28
clarkbI doubt there are a lot of icehouse changes right now01:28
*** maxbit has joined #openstack-infra01:28
jogo has two patches for actual fails so far01:29
fungiwell, apevec is trying to cut 2014.1.1 tomorrow01:29
clarkboh is it that time already?01:29
clarkbthat seems early01:29
jogofungi: and this should be in 2014.1.1 right?01:29
clarkbfungi: feel free to promote but I can't do that right now01:29
clarkbI am dinnering01:29
clarkbbut if things keep failing the priority should be fixing the underlying issues01:30
fungijogo: that's more what i'm asking... did this get a freeze exception?01:30
clarkblike 97912 and the ntpdate thing01:30
jogofungi: hmm not sure I don't see any review from apevec01:31
fungijogo: i'm actually curious how there are non-stable-maint +2s on it01:31
* jogo tries to +2 it01:31
jogoseems I cannot01:31
jogoanyway I don't really see how cutting 2014.1.1 can block stablizing the gate01:32
jogothat doesn't make sense to me01:32
fungijogo: oh, they are in stable-maint-core01:33
fungier, openstack-stable-maint01:33
fungii forgot they were members01:33
fungibut it does look like all nova reviewers on that01:34
fungiprobably would be good to double-check since the call for testing has already gone out on that branch earlier today01:34
clarkbthe ntpdate change didn't work01:34
fungioh? what did it do?01:35
*** lcostantino has quit IRC01:35
*** rfolco has quit IRC01:36
clarkbfungi: I can reproduce that with `ntpdate You`01:36
anteayajogo: is he back?01:36
*** maxbit has quit IRC01:36
anteayajogo: he has been away01:36
jogoanteaya: didn't know he was gone01:36
clarkbfungi: so I think the grep for the ntp server is wrong01:36
jogoanteaya: looking for some devstack cores01:36
fungirussellb: if you're around, did get a stable freeze exception? call for testing already went out and point release was supposed to be tagged tomorrow i think01:36
clarkbfungi: maybe we hardcode debian ntp servers as the default then we can override using that variable01:36
anteayajogo: last week he was and the first of this, sdague said he should be back end of this week01:36
anteayajogo: I haven't seen him around though01:37
*** fbo_away is now known as fbo01:37
jogojeblair: ^01:37
jogoyour devstack core01:37
anteayajogo: you are hitting all the away folks01:37
jogoalong with jessie andrews and vish01:37
clarkbfungi: left in a comment on that change01:38
clarkbnow time for food01:38
*** gokrokve has quit IRC01:38
jogowhat devstack cores are around?01:38
fungiclarkb: hah. i think it's parsing a comment because it should grep for ^server01:38
fungijogo: jeblair is away for a week or more01:39
* jogo thinks we should make fungi an honerary devstack core01:39
anteayajogo: wow, sdague01:39
anteayayeah teh devstack core group needs some updating01:40
jogoanteaya: badly01:40
* fungi doesn't need more spinning plates01:40
anteayaare jesse and vish around?01:40
anteayaI rarely see them on irc01:40
anteayachmouel is but wrong time zone01:40
clarkbfungi: oh if its that simple then we can update the grep01:40
anteayaand garyk is active on reviews but I don't often see him about on irc01:41
clarkbjogo: he is already honorary everything core when he feels like it01:41
jogoclarkb: an offical honorary core in this case01:41
clarkbfungi: this may be a case for unofficial honorary approving, but I think we can let sdague self approve instead01:41
anteayaso hopefully dtroyer is coming to the qa/infra meetup and can bootstrap some folks on the path to devstack-core01:42
jogoanteaya: that reminds me I have to tell monty I am gonna go01:43
openstackgerritJeremy Stanley proposed a change to openstack-infra/devstack-gate: Force ntpdate before we start the timer
fungiclarkb: updated ^01:43
*** aysyd has quit IRC01:44
clarkbfungi: thanks, I can approve it once tests get beyond that first bit of d-g01:44
fungijogo: looking forward to seeing you there01:44
*** jcooley has quit IRC01:44
clarkbjogo: we have to tell monty? :P01:45
clarkbI hope he knows I am going01:45
anteayajogo: sign up:
anteayaclarkb: I'm just going to send him my expense report01:45
clarkbfungi: also good catch and woo comments01:45
anteayawell he was there when we were talking about it01:46
*** nosnos has joined #openstack-infra01:46
jogomtreinish: ping
jogoclarkb: well tell him is a code for book the tiket01:47
anteayathe golden tiket01:47
clarkbfungi: 5 Jun 01:47:21 ntpdate[1886]: the NTP socket is in use, exiting I was worried about that01:47
fungiclarkb: crap. so we need to stop and start ntp around it i guess01:48
clarkbeither that or convince ntpd to do a 4 hour update (which it won't do)01:48
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: [WIP] Modularized StoryBoard Module
jogojeblair: works for HP?01:48
anteayayeah I saw that too01:48
anteayathat also was my response01:48
anteayaif you look at the history corvus made that change01:49
anteayaso I don't understand01:49
clarkbfungi: not sure what the ntp appropriate response here is01:49
krotscheckmordred: Finally got around to making all the versions play happy ^01:49
jogoanteaya: I figured a cut and pase error?01:49
clarkbfungi: ntp hates me01:49
jogomtreinish: got some tempest patches for you01:49
anteayajogo: that makes sense01:49
krotscheckStill need to add more docs.01:49
*** jcooley has joined #openstack-infra01:49
jogofungi: ^01:49
anteayajogo: yeah cause it is the same as what mtreinish said01:49
jogomtreinish: oh you already +1ed it
clarkbfungi: maybe we need to prepend our ntpd init script with ntpdate?01:50
anteayawhereas we who work on monty's team all have a different name for it01:50
clarkbfungi: hax01:50
jogomtreinish: err
openstackgerritJeremy Stanley proposed a change to openstack-infra/devstack-gate: Force ntpdate before we start the timer
clarkbfungi is `service` something centos/fedora speak?01:52
fungiyeah, which is why i used it (though i don't know for sure that the servicename is the same, or even that ntpdate is installed on them by default)01:53
*** maxbit has joined #openstack-infra01:53
ianwis anyone with under-the-hood knowledge of hpcloud here?01:54
clarkbianw: tteggel and gilliard01:54
ianwi wonder if is related to the connection issues i've seen01:54
uvirtbotLaunchpad bug 1326539 in devstack "devstack setup fails to get pip" [Undecided,In progress]01:54
ianwdescribed here
ianwtteggel / gilliard : ^01:54
clarkbianw: could be but we shouldn't be talking to at all01:54
clarkbianw: so its a bug on our end for sure01:54
ianwclarkb: yeah, there's sdague's change out
ianwbut in the mean time, these timeouts seem semi-frequent01:55
ianwi was also seeing it to anyway01:56
jogofungi: so thoughts on bumping it has had 43 hits in last 24 hours01:56
jogoit almost got merged once before anyway01:56
jogoalso what do you think of approving ?01:57
*** melwitt has quit IRC01:57
clarkbianw: we cache the file01:57
clarkbianw: which is why that method is prefered01:57
clarkbfungi: ntpdate change lgtm01:58
clarkbfungi: did you want to check logs before I approve it?01:58
jogothat failure has 51 hits in 48 hours01:59
jogomessage:"err_trap" AND tags:"console"01:59
clarkbjogo: I think sdague should approve01:59
fungihrm... no experimental f20 jobs on d-g02:00
jogoclarkb: he is done for the day02:00
clarkbjogo: I know02:00
clarkbhe will be awake really early tomorrow02:00
jogohaha I thought we could have a present for him thats all02:00
clarkbI don't think ninja approving a change with no core votes is a present02:01
clarkbmore like invite to yell on ml02:01
jogoclarkb: fair enough02:01
jogobut the lack of devstack cores around is scary02:02
jogoalmost as bad as grenade02:02
jogoclarkb: so the problem with sdague approving his patch early in the morning, is what infra person will be around to bump it up to the top of the queue?02:03
clarkbjogo: SergeyLukjanov02:03
fungiclarkb: yeah, logs in progress for 97924 look right02:03
fungiand yeah, SergeyLukjanov already has lots of experience enqueuing and promoting changes in the gate, so should be fine02:04
clarkbjogo: honestly I don't think its that critical02:04
jogoclarkb: ahh excellent02:04
clarkbif it happens at 9am pst vs 5am pst meh02:04
jogoclarkb: well every fix to the gate we can do is a bug step forward02:05
fungiif it waits until 1300z (9am edt) i'll be around too02:05
jogobtw is there abug filed with rax about there mirror?02:05
fungiis it permanently broken?02:05
jogofungi: transiently02:06
jogoor to be more specific: apt-get update randmly fails02:06
jogoanyay dinner time o/02:06
clarkbme too02:08
ianwfungi: yeah, was thinking of adding f20 jobs to d-g02:08
clarkbI will probably pop in later to see how 97924 is doing02:08
clarkbianw: hopefully 97924 doesn't break you >_>02:08
clarkbianw: though that point in d-g things can fail without failing the job02:09
ianwfungi: get it checking on devstack first02:09
clarkbso we should be ok02:09
clarkbworst case time doesn't update but test will continue to run02:09
fungiright, worst case is no worse than previous except a few benign errors in the console log02:10
*** marcoemorais has quit IRC02:10
*** trinaths has joined #openstack-infra02:15
ianwfungi: benign errors are the worst though, because we have to find the problem all over again on fedora before realising it needs fixing.  anyway, just checking if ntp is installed by that point on f20...02:15
trinathsGood Morning All.02:16
*** fbo is now known as fbo_away02:16
fungiianw: it's entirely probable that we need to add ntpdate to the puppet manifests. i think it just happens to be on ubuntu images in our providers by chance02:17
*** dkranz has quit IRC02:17
ianwfungi / clarkb : ok, for a start it's "ntpd" not "ntp" on f2002:18
*** zz_gondoi is now known as gondoi02:19
ianwit's not in the cloud image i booted by default, but not sure how that translates to upstream02:20
ianwanyway, i'll propose the tests for upstream soonish and then i can do a fixup easy enough02:21
openstackgerritKen'ichi Ohmichi proposed a change to openstack/requirements: Add rfc3987 library
openstackgerritKen'ichi Ohmichi proposed a change to openstack/requirements: Add rfc3987 library
*** gondoi is now known as zz_gondoi02:30
*** Ryan_Lane has quit IRC02:30
*** pcrews has quit IRC02:32
mtreinishjogo: I actually have concerns on the second patch. I think it'll cause more problems02:32
mtreinishbecause we don't know how long the volumes will be lying around02:32
mtreinishinstead of falling apart because of slow deletes I think running out of space in the backing lvm volume is a possibility02:32
mtreinishand also leaking state between admin tests02:33
mtreinishI want to talk to sdague and jgriffith about it tomorrow to see if we can figure out a better fix02:33
*** trinaths has quit IRC02:33
*** signed8bit has joined #openstack-infra02:37
*** wenlock has joined #openstack-infra02:38
mtreinishand on the first patch mriedem brings up a good point about doing it for the v3 tests too. (even if it has a +2 from me already)02:38
mtreinishjogo: anyway I'm out for tonight, I'll be back tomorrow02:38
fungiianw: thanks! in part, the _existence_ of running ntp is the problem we're working around (having an instance boot with incorrect time isn't so bad if ntp doesn't then change it during the job)02:38
mordred_phonefungi: morning02:43
fungimordred_phone: landed?02:44
*** signed8bit has quit IRC02:45
mordred_phoneyup. taxiing02:45
mordred_phonethere was no power on this flight, so I did not accomplish work02:45
anteayamordred_phone: where are you?02:45
anteayapalo alto?02:45
mordred_phonejust landed in nyc02:46
openstackgerritJoshua Hesketh proposed a change to openstack-infra/config: Fix grab_console_log variable typo
anteayaI have to go to sleep02:48
anteayaI hope no more people get shot in Moncton02:48
jheskethclarkb, fungi: If you guys wouldn't mind merging this trivial patch I'd appreciate it (I'm poking at the swift pushing stuff)
*** dims has quit IRC02:53
*** dims has joined #openstack-infra02:55
clarkbjhesketh: done02:56
jheskeththanks clarkb :-)02:56
*** gokrokve has joined #openstack-infra03:02
openstackgerritA change was merged to openstack-infra/config: Fix grab_console_log variable typo
*** lcheng_ has joined #openstack-infra03:03
*** aconrad has quit IRC03:07
*** arnaud has quit IRC03:10
*** mriedem has quit IRC03:11
*** sballe has joined #openstack-infra03:11
*** asettle has quit IRC03:13
*** alexandra has joined #openstack-infra03:13
*** RuiChen has joined #openstack-infra03:15
*** alexandra is now known as asettle03:17
*** Ryan_Lane has joined #openstack-infra03:18
*** otherwiseguy has joined #openstack-infra03:23
RuiChenhi stackers, I have a feature need to be tested in libvirt 1.1.1+ and qemu1.5+, the current ci infrastructure support it?03:23
*** Ryan_Lane has quit IRC03:24
RuiChenif not, what I can do?03:24
clarkbRuiChen: not yet03:24
clarkbwhen we switch to trusty I think so03:24
clarkbI am hoping to have trusty slaves available by end of next week. Then we will have to test and so on so it may be a couple more weeks03:26
jesusaurusclarkb: what needs to happen to get trusty slaves going?03:26
clarkbjesusaurus: nodepool dib03:26
*** sballe has quit IRC03:26
mgagneclarkb: and puppet?03:26
clarkbwe want to switch to uploading glance images as the current image builds are clunky and don't scale03:27
clarkbmgagne: puppet is done03:27
mgagneclarkb: awesome03:27
clarkb(assuming we haven't regressed)03:27
*** sballe has joined #openstack-infra03:27
clarkbI was going to start working on it yesterday but things were still sideways03:29
clarkbfungi: oh we didn't restart nodepool today :(03:29
* clarkb will try to do that early tomorrow03:29
fungiclarkb: i'm still up for it, but it can wait too03:31
clarkbif you want to do it I can hang around but machine with keys is off03:32
clarkbI can turn it on if anything goes sideways or we can wiat03:32
fungii suppose i could just do it now... was there anything in particular we wanted to double-check?03:32
clarkbfungi: yes we need to make sure puppet is running and the version we want is installed03:32
RuiChenchangbl: thank you, I look forward to trusty :)03:32
clarkbfungi: then we need to check that we don't leak nodes again03:33
*** talluri has joined #openstack-infra03:33
*** jaypipes has quit IRC03:33
fungiclarkb: okay, so stop nodepool, delete cruft nodes, enable puppet agent, trigger puppet run from ci-puppetmaster, check installed nodepool version, start nodepool...03:35
fungimissing anything obvious?03:35
clarkbfungi: I think you can run puppet first and check the version before stopping03:36
clarkbotherwise lgtm03:36
fungiahh, good point03:36
fungistarting now03:36
*** dims_ has joined #openstack-infra03:39
*** dims has quit IRC03:39
clarkbbasically we want to make sure that jeblair's fix gets in for that node leakage and double check it worked03:40
*** cody-somerville has quit IRC03:42
fungiof course we've broken nodepool --version, but pip list sez 68860d9 is installed03:42
fungiwhich looks like the tip of master03:42
*** nosnos has quit IRC03:43
*** sballe has quit IRC03:43
fungino, wait03:43
fungiit's not03:43
clarkbya that must be the old broken version beacuse its not in my gitl og03:43
clarkbb6539f9cdd19a71d8e9e7dec49f21ac15876950b is what we want03:43
fungishould be b6539f9 yeah03:44
fungichecking logs03:44
clarkbfungi: I am going to guess the repo isn't on the master branch in /opt and puppet won't change it there to do an update03:45
clarkbfungi: I think if you git checkout master then run puppet it will be ok03:45
clarkbmake sure you do that as the owner of the repo though otherwise files may end up with funny perms03:46
fungi# On branch master03:46
clarkboh what if puppet is ff only03:46
clarkbmight need to just git reset --hard origin/master03:46
fungithe puppet log mentioned no problem updating to master and refreshing the nodepool install exec03:46
fungiwell, the repo in /opt matches the origin state, no problem03:47
fungibut pip reports a different installed nodepool version03:47
fungiis what i was trying to say03:47
clarkboh its one of those situations woo pip03:47
*** reed has quit IRC03:47
fungii wonder if the puppet exec doesn't work the way we think03:48
clarkbor if pip is confused03:48
fungioh, maybe03:48
fungiwith --version broken it's hard to tell03:49
clarkbfungi: well if nodepool is still up and running in its current state maybe we just deal with it tomorrow03:50
clarkb(it is pretty late for both of us more so you)03:50
fungidrwxr-sr-x 2 root staff 4096 Jun  5 03:40 /usr/local/lib/python2.7/dist-packages/nodepool-b6539f9-py2.7.egg-info03:51
fungiso maybe pip is confused, yes03:51
clarkbI think that is a case where pip uninstall ; rm -rf dist-packages/nodepool* ; pip install --upgrade cures the problem03:52
clarkbbut again maybe just deal with that tomorrow03:52
fungiagreed. puppet is still enabled... leave it that way, yeah?03:53
clarkbI think so, everything is up to date from a config and nodepool git perspective so pupept should be fine03:53
fungicool. have a good night!03:53
clarkbyou too03:53
openstackgerritJoshua Hesketh proposed a change to openstack-infra/config: Fetch graphitejs zuul dependency
ianwclarkb / fungi: is min-ram 30720 just supposed to be the default for hpcloud instances?03:58
fungiianw: it's the minimum flavor you need to get 8 cpus04:00
clarkbwhich after testing is what we hae found to be the sweet spot for performance to resources ratios04:00
clarkbpleia2: when did the e-r on status.o.o change merge?04:00
lifelessalso the point at which tempest doesn't kill devstack04:00
clarkbI completely missed that in the mess of broken clouds04:00
pleia2clarkb: yesterday I think04:00
*** basha has joined #openstack-infra04:00
clarkblifeless: well it worked fine on 4 cpu nodes04:00
ianwah cool, thanks04:00
fungiand roughly matches the performance we get from an 8g rax instance04:01
clarkblifeless: I have some rough unscientific numbers if you wan them for why this is how we do it04:01
lifelessclarkb: yes but see sdague's point about more cpu counts above ;>04:01
clarkbfungi: but still slower than 1.004:01
clarkblifeless: right but ^ 4vcpu 1.0 was still better04:01
lifelessclarkb: once I dig myself out of this mess, yes.04:01
*** RuiChen has quit IRC04:04
*** eharney has quit IRC04:04
*** harlowja is now known as harlowja_away04:07
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing
*** basha has quit IRC04:10
mordredlifeless: you haz mess?04:11
mordredkrotscheck: neat!04:11
*** lcheng_ has quit IRC04:12
lifelessmordred: yes, Ironic was being, uhm, ironic with me.04:13
clarkband 97757 failed04:13
clarkbmordred: you know if you aren't dead tired and ready to fall asleep04:13
clarkbmordred: 97924,4 and 97757,1 need promotion in the gate04:14
mordredlifeless: _excellent_04:14
clarkbin that order04:14
mordredclarkb: I'm actually not dead tired04:14
*** CSheedy has joined #openstack-infra04:14
clarkbI am not either but the machine with keys is over there off so that I don't work a whole lot04:14
clarkbthis strategy would work if I didn't have another computer over here with a connection to freenode04:15
mordredclarkb: should I wait for a reset? or just promote them?04:15
*** renlt has joined #openstack-infra04:15
clarkbprobably wait for reset as we may actually clear out ~20 changes in a half hour04:15
mordredok. I'll keep my eye on it04:15
mordredclarkb: I had a terrible idea, btw ...04:16
*** Sukhdev has joined #openstack-infra04:16
*** otherwiseguy has quit IRC04:16
mordredclarkb: what if we wrote a filter/function/something for zuul that would allow something to enqueue a change with a different pipeline config04:16
mordredclarkb: oh wait - nevermind04:17
*** yfried has quit IRC04:17
*** gokrokve has quit IRC04:17
mordredclarkb: thanks, it was helpful to try to write out the thought in my head04:17
clarkbok :)04:17
clarkbmordred: also looks like a small group of us may be leaning towards staying in darmstadt and not frankfurt04:18
clarkbto maximize sleep in time04:18
clarkband simplicity04:18
clarkbbut don't let me ruin your dreams of commuting every morning from frankfurt westin to darmstadt04:18
lifelesswhats in darmstadt04:19
clarkbinfra/qa mid cycle meetup04:19
openstackgerritA change was merged to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs
clarkbooh I wonder if darmstadt is situated in a ttx friendly location04:20
*** dims_ has quit IRC04:20
*** Hunner has quit IRC04:22
mordredclarkb: I think he does not want to join due to proximity to oscon04:23
*** trinaths has joined #openstack-infra04:23
* tchaypo learns where is ttx?04:24
* tchaypo glares at weechat04:24
tchaypolifeless: the other thing that's in darmstadt is a deutsche telekom office04:25
clarkbmordred: :( that is understandable I guess04:25
tchaypolifeless: not unrelatedly04:25
pleia2clarkb: staying in darmstadt is probably my preference as well, because yay sleep04:25
*** sballe has joined #openstack-infra04:25
clarkbpleia2: ya I hope to have 20 minutes tomorrow to book the trip and will look for hotel in darmstadt04:26
jogopleia2: won't jetleg mess that up anyway04:26
clarkbtchaypo: you guys have something there too?04:26
*** lcheng_ has joined #openstack-infra04:26
clarkbjogo: that is why every minute counts04:26
pleia2clarkb: cool, please lmk what you end up with04:26
jogoclarkb: haha04:27
clarkbadding ~1.5 hours of roundtrip every day commuting is time I could sleep04:27
pleia2jogo: I'm still a bit under the weather, so being able to return to hotel quickly/by myself is helpful :)04:27
clarkbor try to sleep04:27
clarkbpleia2: will do04:27
clarkblast I looked condor and lufthansa/united have direct flights from seatac but condor is cheaper and not united so I may giev them a try04:27
*** plars is now known as plars-away04:27
pleia2lufthansa is nice, that's what I'm taking to croatia next weekend04:28
clarkbthen I was going to try and do one of the two hotels on the wiki page04:28
pleia2but united co-chairs with them, so have to be careful :)04:28
pleia2(can tell when you book, *fine print)04:28
clarkbI will have to look closely then04:28
*** harlowja_at_home has joined #openstack-infra04:28
*** Longgeek has joined #openstack-infra04:28
tchaypoclarkb: no04:31
*** sballe has quit IRC04:32
*** CSheedy has quit IRC04:32
*** cody-somerville has joined #openstack-infra04:35
clarkbmordred: tminus 8 minutes (but no big deal if you have fallen asleep)04:35
mordredclarkb: I'm still here04:35
*** jcoufal has joined #openstack-infra04:36
*** matsuhashi has joined #openstack-infra04:39
*** harlowja_at_home has quit IRC04:40
clarkbmordred: now is a good time04:41
clarkbgate just reset04:41
*** nosnos has joined #openstack-infra04:41
mordredclarkb: going for it04:41
mordredclarkb: zuul promote 97924,4 97757,1 yeah?04:41
clarkbmordred: zuul promot --pipeline gate --changes 97924,4 97757,104:42
clarkbas root too04:42
clarkbso that it can read the zuul.confr04:42
clarkber zuul promote04:42
clarkbthat has an e at the end04:42
clarkbwoot thanks04:42
mordredyeah - i was mainly just making sure those were the changes04:42
mordredand the order04:42
clarkbyup thats the order04:43
clarkb97924 fixes the time in the future bug and 97757 fixes lifeless' bug04:43
*** renlt has quit IRC04:46
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add devstack-f20 to hpcloud
*** markwash has quit IRC04:56
SergeyLukjanovoh, my ...04:59
SergeyLukjanov>1000 msgs in scrollback04:59
SergeyLukjanovmorning/evening folks04:59
clarkbSergeyLukjanov: o/05:01
SergeyLukjanovanteaya, yeah, I like the proposed name pattern too05:01
clarkbSergeyLukjanov: they gate is a bit cranky. There is a devstack change that we semi expect sdague to merge early in the morning his time that may need promotion05:01
clarkbSergeyLukjanov: apparently he is the only devstack core that isn't on vacation and he wrote the change so we will see what happens :)05:02
SergeyLukjanovclarkb, okay, got it05:03
*** Longgeek has quit IRC05:06
*** dkehn has quit IRC05:06
*** Longgeek has joined #openstack-infra05:07
*** yfried has joined #openstack-infra05:07
*** basha has joined #openstack-infra05:07
*** dkehn has joined #openstack-infra05:08
*** marcoemorais has joined #openstack-infra05:17
*** marcoemorais1 has joined #openstack-infra05:19
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Fix up status page dependencies
openstackgerritJoshua Hesketh proposed a change to openstack-infra/config: Use the latest jquery on zuul
clarkbSergeyLukjanov: jhesketh if 97910 looks good to you guys I think that would help reduce some of the test slave pressure we have had the past few days and make our gate more robust05:21
*** marcoemorais has quit IRC05:21
SergeyLukjanovclarkb, ack05:22
*** hunner has joined #openstack-infra05:22
*** hunner is now known as Hunner05:22
jheskethclarkb: +205:22
openstackgerritA change was merged to openstack-infra/config: Remove markdown-docbook
SergeyLukjanovclarkb, +A05:23
clarkbcool thnaks05:23
clarkbjogo: ^ if you are still around05:23
clarkbbut that is two less tests per nova patch05:23
clarkbalso fyi I am going to approve 96907 which won't go into affect until after tomorrow morning (PDT) image builds05:26
clarkbso it should be safish :)05:26
clarkbthat should clean up a lot of the logs around sudo and be less confusing to people reading them05:28
openstackgerritA change was merged to openstack-infra/config: Bump LARGE_OPS number from 50=>100
*** marun_afk is now known as marun05:29
* SergeyLukjanov brb in 3 hours05:30
openstackgerritA change was merged to openstack-infra/config: Add the new hostname to /etc/hosts in prepare_node
openstackgerritA change was merged to openstack-infra/devstack-gate: Force ntpdate before we start the timer
clarkbthough 97757 failed again05:41
clarkbbut at least one gate bug killed now I can fall asleep05:41
*** Sukhdev has quit IRC05:41
mordredclarkb: woot05:44
lifelessmordred: so you'll handle 97757 ? :)05:47
*** mdenny has quit IRC05:51
*** arnaud has joined #openstack-infra05:51
*** mdenny has joined #openstack-infra05:51
*** miqui has quit IRC05:54
*** jcoufal has quit IRC05:56
*** oomichi has quit IRC06:02
*** gyee has quit IRC06:04
*** andreykurilin_ has joined #openstack-infra06:05
*** johnthetubaguy has quit IRC06:05
*** jraim has quit IRC06:05
*** mancdaz has quit IRC06:06
*** johnthetubaguy has joined #openstack-infra06:06
*** simonmcc has quit IRC06:06
*** jraim has joined #openstack-infra06:06
*** mancdaz has joined #openstack-infra06:08
*** simonmcc has joined #openstack-infra06:08
*** flaper87|afk is now known as flaper8706:08
mattoliverauNight all, I'm off for the day!06:09
*** srenatus has quit IRC06:11
*** srenatus has joined #openstack-infra06:11
mordred_phoneclarkb: I pushed up a new still not working nodepool patch06:13
*** marun is now known as marun_afk06:14
*** renlt has joined #openstack-infra06:16
*** trinaths has quit IRC06:20
clarkbjesusaurus: can you propose your hostname fix to d-g as we06:23
clarkb*as well06:23
clarkbmordred: cool06:23
*** harlowja_at_home has joined #openstack-infra06:27
harlowja_at_homeqq, whats up with the gate zuul job queue situation, is it getting better?06:27
*** Longgeek has quit IRC06:27
clarkbnot really. I think grenade/tempesr are having trouble06:28
harlowja_at_homehmmm, durn06:28
harlowja_at_homeout of curosity, are there less workers in the gate queue (interesting to see that only like 5 workers seem to be processing jobs?)06:29
* harlowja_at_home watching
harlowja_at_homeis there anyway to help to make it better?06:31
harlowja_at_home*excluding beating with stick, haha06:31
SlickNikCan I get some eyeballs on when folks get a chance? Thanks much!06:33
*** hashar has joined #openstack-infra06:33
*** rgerganov has joined #openstack-infra06:34
*** rdopiera has joined #openstack-infra06:38
*** tkelsey has joined #openstack-infra06:39
*** lttrl has quit IRC06:40
*** andreykurilin_ has quit IRC06:40
*** andreykurilin_ has joined #openstack-infra06:41
jesusaurusclarkb: i see
jesusaurusclarkb: what change were you thinking?06:44
*** srenatus has quit IRC06:44
*** srenatus has joined #openstack-infra06:44
clarkbhrm is that called early?06:48
clarkbI want to say maybe it isnt early enough to prevent sudo from complaining06:49
clarkbharlowja that is intentional. the gate operates on a window06:49
mordred_phoneclarkb: we really do need to write a bunch more about concepts and whys06:52
harlowja_at_home*for curious folks like me ;)06:52
*** lcheng_ has quit IRC06:52
jesusaurusmordred_phone: hopefully some of that will end up in infra-specs06:53
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Keep the hierarchy of tripleo files copied to log server
mordred_phonejesusaurus: ++06:53
mordred_phoneand infra-manuals06:54
*** freyes has joined #openstack-infra06:54
harlowja_at_homeclarkb, window being used so that only X number of changes can cause problems at the same time (if the window is small then its easier to identify what root caused an issue)06:54
*** andreykurilin_ has quit IRC06:56
*** Longgeek has joined #openstack-infra06:58
*** camunoz has quit IRC07:00
jesusaurusharlowja_at_home: well, a window to try to minimize the number of vms active (not that we can minimize the number of vms needed)07:00
*** freyes has quit IRC07:01
harlowja_at_homejesusaurus, gotcha, what happens if more vms become active? (bad things?)07:01
*** Longgeek has quit IRC07:02
jesusauruswell, some people have to pay more for that, but generally no cloud is limitless07:04
ttxclarkb: Germany is never a ttx-friendly location :)07:05
ttxI need to look up if i can join you all but it's a bit unlikely as i have a conference in Montpellier just befor07:06
*** AaronGr has quit IRC07:06
*** bradm has quit IRC07:07
*** wenlock has quit IRC07:11
*** AaronGr has joined #openstack-infra07:13
*** maxbit has quit IRC07:14
*** harlowja_at_home has quit IRC07:15
*** oomichi has joined #openstack-infra07:16
*** mattt has quit IRC07:17
*** bradm has joined #openstack-infra07:22
hasharat worth you could fly from montpellier to germany :D07:23
lifelessmordred_phone: ping07:24
*** jlibosva has joined #openstack-infra07:25
mordred_phonelifeless: I don't believe in ping07:25
*** jp_at_hp has joined #openstack-infra07:25
lifelessmordred_phone: 97757 needs to be requeued up the top -07:25
lifelessmordred_phone: its blocking ironic and tripleo checks07:25
lifelessmordred_phone: (its the one clarkb handed over to you :))07:26
mordred_phonelifeless: k. I'm at a bar. SergeyLukjanov , around?07:26
mordred_phoneharlowja_at_home: the window is adaptive to the gate health07:27
*** doude has joined #openstack-infra07:28
lifelessmordred_phone: at 1pm ? :)07:30
mordred_phonelifeless: at 3:30am07:30
lifelessmordred_phone: oh, you're stateside again?07:30
mordred_phonelifeless: this bar has the best deviled eggs07:31
mordred_phonelifeless: hell yes07:31
mordred_phonelifeless: in the NY cizzile07:31
hasharthey look good
hasharjhesketh: if you are around, I got a Zuul patch to loosen its dependencies to python-swiftclient ( )07:33
*** flaper87 is now known as flaper87|afk07:39
*** ihrachyshka has joined #openstack-infra07:42
*** arnaud has quit IRC07:42
*** Guest22618 has joined #openstack-infra07:44
*** mrda is now known as mrda-away07:44
*** skolekonov has joined #openstack-infra07:48
*** andreaf has joined #openstack-infra07:50
*** salv-orlando has joined #openstack-infra07:51
*** apevec has joined #openstack-infra07:52
*** jistr has joined #openstack-infra07:53
apevecugh, top of gate is >28h07:54
apevecnot a good start for 2014.1.1 release day07:54
*** Longgeek has joined #openstack-infra07:55
*** _nadya_ has joined #openstack-infra07:59
*** pelix has joined #openstack-infra08:01
*** tcammann has joined #openstack-infra08:04
*** e0ne has joined #openstack-infra08:06
*** Longgeek has quit IRC08:10
*** basha has quit IRC08:11
*** Longgeek has joined #openstack-infra08:11
*** dizquierdo has joined #openstack-infra08:13
*** Longgeek has quit IRC08:16
*** ominakov has joined #openstack-infra08:17
boris-42jhesketh hey there08:18
*** Longgeek has joined #openstack-infra08:19
*** e0ne has quit IRC08:19
*** viktors|afk has joined #openstack-infra08:19
ttxapevec: it certainly seems to be struggling a bit08:20
*** jcoufal has joined #openstack-infra08:27
*** jcoufal has quit IRC08:27
*** jcoufal has joined #openstack-infra08:28
mordredhashar: they're very tasty08:31
*** mkoderer has quit IRC08:32
apevecttx, btw please proof read relnote draft
apevectristanC, ^ please double check I included all OSSAs08:33
hasharmordred: i can imagine :)08:33
mordredhashar: the ones at ward iii come with a slice of olive and a slice of jalapeño on top08:34
* hashar looks up jalapeño 08:35
ttxapevec: on it08:35
openstackgerritChristian Berendt proposed a change to openstack-dev/hacking: Add H804: enforce title case in commit messages
*** trinaths has joined #openstack-infra08:36
hasharmordred: if I ever come to NYC, I will definitely attempt to meet you up in that bar to taste those devil eggsç08:36
ttxapevec: I'd say "80 bugs" rather than "80 launchpad bugs" -- otherwise looks good08:38
ttxapevec: also checked the OSSA list and it looks good08:38
*** prad has quit IRC08:44
apevecttx, looks like "launchpad bugs" was copied since
apevecbut no time like now to change the tradition08:45
*** viktors|afk is now known as viktors08:47
ttxapevec: I can update it if you aren't in the middle of an edit08:47
ttxoh, you fixed it already08:47
jheskethboris-42: Hey08:50
boris-42jhesketh could you explain your comment sorry08:50
apevecttx, yeah, was just doing wiki archaeology :)08:50
boris-42jhesketh I didn't get it..08:50
jheskethhashar: is that swiftclient ready to review?08:50
boris-42jhesketh this one
jheskethboris-42: the format of your gerrit ACL is wrong08:51
*** basha has joined #openstack-infra08:51
boris-42jhesketh I mean about tag08:51
jheskethboris-42: I don't follow sorry08:51
boris-42jhesketh this comment
boris-42jhesketh not first one08:52
jheskethoh right, sorry08:52
jheskethboris-42: there is a git tag on that repository08:52
*** hashar_ has joined #openstack-infra08:52
jheskethit will be imported into the cloned repository08:52
boris-42jhesketh so probably I should just remove it?08:52
*** hashar has quit IRC08:52
boris-42jhesketh with push --force08:52
jheskethtags generate releases from the tarball jobs08:52
jhesketh(I'm not sure if it would retroactively, I'd have to check)08:53
jheskethboris-42: well it depends if you want the tag or not. I'd say it probably wouldn't hurt to leave it, but I was point it out to you08:53
boris-42jhesketh I mean it's not a big deal to remove osprofiler from pypi08:53
boris-42jhesketh and remove tag08:53
jheskeththat shouldn't be necessary08:53
boris-42jhesketh btw what about osprofiler on pypi08:54
boris-42jhesketh should I remove it, to be able to publish tags?08:54
*** unicell has quit IRC08:54
boris-42jhesketh and make new versions via openstack ci08:54
*** Alexei_987 has quit IRC08:54
jheskethboris-42: to be honest I'm not sure. I don't think it matters. I'd have to double check what the tarball jobs do, but I'm late for meeting some people so I don't have time at the moment sorry08:57
jheskethPerhaps somebody else in here knows?08:57
jheskethotherwise I'll take a look tomorrow08:57
boris-42jhesketh I'll just get rid of tag & lib08:57
boris-42jhesketh it will work 100%  lol=)08:57
boris-42jhesketh and it will take just a couple of minute08:58
*** achuprin has quit IRC08:58
*** oomichi has quit IRC09:00
*** jooools has joined #openstack-infra09:01
*** marcoemorais1 has quit IRC09:05
*** achuprin has joined #openstack-infra09:13
openstackgerritBoris Pavlovic proposed a change to openstack-infra/config: Finally add osprofiler to stackforge
*** hashar_ is now known as hashar09:16
*** zhiyan is now known as zhiyan_09:18
*** flaper87|afk is now known as flaper8709:19
*** e0ne has joined #openstack-infra09:21
*** dizquierdo is now known as dizquierdo_out09:21
*** _nadya_ has quit IRC09:29
SergeyLukjanovI'm here now09:31
SergeyLukjanovlifeless, mordred_phone09:31
SergeyLukjanovI presume that 97757 is still needed09:32
SergeyLukjanovso, promoting it to the top of queue09:32
*** oomichi has joined #openstack-infra09:32
SergeyLukjanovlifeless, oh, you're still not sleeping ;)09:32
lifelessI should be soon:)09:33
*** jgallard has joined #openstack-infra09:33
SergeyLukjanovlifeless, done09:34
SergeyLukjanovsdague, do you have a world-fixing CR to devstack? :)09:35
*** derekh_ has joined #openstack-infra09:37
*** flaper87 is now known as flaper87|afk09:38
*** srenatus has quit IRC09:41
*** denis_makogon has joined #openstack-infra09:42
*** srenatus has joined #openstack-infra09:42
KiallHey guys - This commit to add a few new jobs landed recently, but the gate-designate-requirements jobs that were enabled, and which exist in jenkins, are not being triggered. Thoughts?
KiallThe other jobs it added (devstack and -docs) have both been enabled.09:46
sdagueSergeyLukjanov: just getting coffee ... is there a specific bug to fix, or all the bugs?09:50
SergeyLukjanovsdague, there were some msgs re you is the last one devstack core and will self approve some fix today's morning :)09:51
SergeyLukjanovlifeless, 97757,1 just failed the py26 test09:52
sdagueSergeyLukjanov: so there is this one -
sdaguewhich actually has some +2s on it09:52
KiallIgnore me - There is a path filter on the requirements jobs - I just spotted it :)09:52
*** mkoderer has joined #openstack-infra09:53
SergeyLukjanovsdague, nice, +1 from me too09:54
SergeyLukjanovsdague, sounds like it should be promoted too09:54
sdagueSergeyLukjanov: I'll start listing stuff here -
SergeyLukjanovsdague, could you please take a look at nova unit tests failure?09:55
SergeyLukjanov97757,1 has been promoted to unblock ironic/tripleo09:56
SergeyLukjanovbut it fails the py26 btest09:56
sdagueSergeyLukjanov: sure thing09:57
*** basha has left #openstack-infra09:59
sdagueSergeyLukjanov: it appears to be a longish standing unit test issue -
*** dkehn has quit IRC10:00
*** basha has joined #openstack-infra10:01
sdagueSergeyLukjanov: it's
uvirtbotLaunchpad bug 1311778 in nova "Unit tests fail with MessagingTimeout errors" [Critical,New]10:02
*** dkehn has joined #openstack-infra10:02
*** rlandy has joined #openstack-infra10:04
*** jgallard has quit IRC10:06
*** ildikov has quit IRC10:07
*** ildikov has joined #openstack-infra10:07
SergeyLukjanovsdague, so, we could just re-promote10:07
SergeyLukjanovsdague, am I right?10:07
sdaguewe could10:09
sdagueso, honestly, I'm back to my theory that we should dump the gate queue10:10
sdaguebecause I realized something else with the grenade fix10:10
sdaguewe've now upped concurrency on all the jobs10:10
sdagueas we no longer have 4 core nodes10:10
*** yamahata has quit IRC10:10
sdagueand we have a set of things in the gate that passed on a 4 core node10:11
sdaguebut have a race10:11
sdagueonce there are tests from this - we'll want to promote10:11
*** ihrachyshka has quit IRC10:12
sdagueSergeyLukjanov: so realistically ironic is failing it's own unit tests right now10:12
sdaguewhich means I don't think we should bother with a promote on 97757 until that's resolved10:13
sdagueunless it's the same thing10:13
sdagueis there a bug or any context that explains this?10:13
sdaguelifeless: do you know about the ironic unit tests and if they are linked to this?10:14
SergeyLukjanovI don't see anything related10:14
SergeyLukjanovto it10:14
*** hashar has quit IRC10:14
srenatusis there a way to increase the timeout for a job?  with chef cookbooks and extensive chefspec runs, the 30m limit is hit quite often10:14
sdaguethose ironic unit test breaks are at least a day old, as I was manually pulling all their changes out of the queue10:15
SergeyLukjanovso, we have an option to make a zuul reset, promote all important changes, send email with excuse10:15
sdagueSergeyLukjanov: yeh, if you start etherpad drafting things I'll help with the email10:15
sdaguesrenatus: yes, it's configured in the job definition in the openstack-infra/config repo10:16
sdagueSergeyLukjanov: because we kind of need an all hands on deck here as well10:16
*** denis_makogon has quit IRC10:17
SergeyLukjanovsdague, " if you start etherpad drafting things" which things?10:17
SergeyLukjanovyou mean CRs that should be promoted?10:17
sdagueSergeyLukjanov: no, the email explaining to people why we've dumped the gate10:19
SergeyLukjanovsdague, oh, got it10:19
srenatussdague: thanks, found that devstack-gate.yaml serves as an example how to override it10:23
sdaguesrenatus: that's probably not quite the right thing10:23
sdaguelet me get you a more specific link10:23
*** renlt has quit IRC10:24
srenatussdague: oh ok... thanks10:25
sdaguesrenatus: actually, maybe you did find it10:25
sdagueI didn't realize that was only defined in the wrappers part10:26
sdagueyeh, I'd suggest proposing a patch with the fix you think based on reading devstack-gate.yaml, and SergeyLukjanov will correct it if need be :)10:27
SergeyLukjanovsdague, I'm started writing email draft in your etherpad10:30
SpamapSsdague: so is the gate just so far gone that we need to dump it?10:31
sdagueSpamapS: yeh, I think so10:31
*** matsuhashi has quit IRC10:31
sdagueespecially with the concurency change10:32
sdaguea bunch of stuff made it over a 4 ft bar, and ran into an 8 ft brick wall10:32
sdagueit's a side effect of hp cloud 1.0 => 1.1 that we didn't quite notice10:33
*** matsuhashi has joined #openstack-infra10:34
sdaguethough in 12 minutes all  the heat changes will dump out anyway, because the 17 patch series has a merge conflict10:34
sdagueactually that's a 21 patch series?10:35
*** Guest22618 has quit IRC10:36
sdagueSergeyLukjanov: once the trove test lands, and the heat series falls out, we should do the promotes. We can then work on the email while those are churning.10:36
SpamapSsdague: oh, what if I were to push a rebase of that?10:36
sdagueSpamapS: it would take it out of the gate early10:37
SpamapSI knew that series would be trouble. :-P10:37
sdaguewhich would also be fine10:37
SergeyLukjanovsdague, okay10:37
sdaguehonestly, that series will consume all the gate resources we've got for a couple of hours10:37
sdaguelong series like that are a mild dos attack10:38
sdaguethough we have the nodes now10:38
SpamapSsdague: I don't think zaneb though we'd approve the whole thing in 2 days10:38
sdagueyeh, that's fair. It's just funny how different different teams work some times :)10:39
sdaguethe heat team loves everything in long series10:39
sdaguewhen reviewing their changes to tempest I usually have to ask them to collapse a few things10:39
SergeyLukjanovsdague, 97966 already failed it's grenade job :(10:42
sdagueSergeyLukjanov: ok, let me figure out what I did wrong this time10:42
*** praneshp has quit IRC10:42
*** praneshp has joined #openstack-infra10:44
*** praneshp has quit IRC10:45
SpamapSoh ugh10:45
SpamapSthat merge conflict is _nasty10:45
SpamapSI'm going to let ZaneB work that one out10:45
SergeyLukjanovsdague, trove tests just merged10:46
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: set concurrency for grenade
*** jooools has quit IRC10:47
sdagueSergeyLukjanov: turns out we need that one too10:47
sdaguecan you look at that?10:48
SergeyLukjanovsdague, yup, sure, but it was lgtm last time for me10:48
sdaguethe grenade one failed because it was running 8 way again. Basically it's not passing the control var10:48
sdaguethat's the d-g change to pass it10:48
SergeyLukjanovsdague, oh, yeah, see it10:48
*** jooools has joined #openstack-infra10:49
sdagueso the grenade fail was unrelated to the grenade change10:49
sdagueit's just that 8 way testing doesn't work very much10:49
*** basha has quit IRC10:50
*** jooools has quit IRC10:50
SergeyLukjanovsdague, it's lgtm10:50
sdagueso, honestly, I'd say promote the top 4 in the list10:51
sdagueeven though these 2 haven't gotten through check yet10:51
SergeyLukjanov"To Promote" section to promote ;)10:51
SergeyLukjanovsdague, ^^10:51
sdaguewhich means moving these 2 to gate queue as well10:51
*** basha has joined #openstack-infra10:51
*** om has joined #openstack-infra10:52
SergeyLukjanovsdague, yup, working on promote commands10:54
*** e0ne_ has joined #openstack-infra10:54
sdagueawesome, thanks10:55
*** talluri has quit IRC10:55
sdaguealso... it's incredibly awesome to have someone that can do all these things when I'm up in the morning :)10:55
sdague\o/ for global infra team10:56
*** talluri has joined #openstack-infra10:56
*** talluri has quit IRC10:56
sdagueas I'm looking at the things which are bouncing of random bugs, these 4 should address a bunch of them10:56
*** talluri has joined #openstack-infra10:56
sdagueso we might start picking up speed in the gate after them10:56
sdagueso I'll reserve my gate dump idea for now10:57
sdaguewe might just have to pound these 4 through though10:57
SpamapSsdague: what was the hpcloud 1.1 change that we're not dealing with btw?10:57
* SpamapS should be asleep.. stupid indian jet lag won't let go 5 days later.10:58
*** e0ne has quit IRC10:58
SergeyLukjanovsdague, I'm glad to help ;)10:58
SergeyLukjanovsdague, so, all changes are presented in gate now, promoting them in the order like in etherpad10:58
SergeyLukjanovyou have ~90 secs to stop me10:59
sdagueSpamapS: mon 1.0 shut off10:59
sdagueSergeyLukjanov: let me look10:59
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Name fields checked with regex
SergeyLukjanovsdague, sudo zuul promote --pipeline gate --changes 97912,2 97845,2 98080,1 97966,211:00
sdagueyeh, that seems right11:00
SpamapSsdague: OH11:00
sdagueSpamapS: so we're on 1.111:00
SergeyLukjanovsdague, probably we could move grenade fix to the top to avoid grenade failures11:00
SpamapSsdague: so just "scramble oops firedrill new things go"11:00
sdagueSpamapS: well that was mon/tues11:00
SergeyLukjanovsdague, or it's not so bad to fail every time?11:00
sdagueSergeyLukjanov: honestly, lets try it as is11:00
SpamapSsdague: right, and Wed was "find and extinguish embers"11:00
sdagueSpamapS: well, it led to an interesting new edge case11:01
sdaguewe were running at half capacity for 2 days11:01
SpamapSsdague: more interesting than "the spamaps hole"?11:01
sdagueso the check queue got to 500 changes11:01
sdagueand people just kept approving things11:01
sdagueso when nodes showed up, in combination with the "the spamaps hole", we got this giant tidal wave into the gate11:02
SergeyLukjanovsdague, promoted11:02
sdagueSergeyLukjanov: thanks11:02
SergeyLukjanovsdague, np11:03
SergeyLukjanovI'd like to see gate alive as well as you :)11:03
SergeyLukjanovapevec, do we have a lot of pending CRs in gate to release 2014.1.1?11:03
SpamapSsdague: I'm going to go out on a limb and say tidal waves are bad.11:04
SergeyLukjanovapevec, we have 6 in sahara btw11:04
*** dprince has joined #openstack-infra11:04
SpamapSok.. time to attempt sleep again11:05
*** matsuhashi has quit IRC11:06
*** palar has joined #openstack-infra11:06
*** matsuhashi has joined #openstack-infra11:06
*** matsuhashi has quit IRC11:10
SergeyLukjanovsdague, time to have a short meeting and big coffee11:12
*** yamahata has joined #openstack-infra11:15
*** om has quit IRC11:15
*** ihrachyshka has joined #openstack-infra11:19
*** lcostantino has joined #openstack-infra11:19
*** dizquierdo_out is now known as dizquierdo11:19
*** nosnos has quit IRC11:20
*** talluri has quit IRC11:22
apevecSergeyLukjanov, there are only 2 Neutron 2014.1.1 exceptions in the gate: 96919 and its child 9692011:24
*** matsuhashi has joined #openstack-infra11:24
*** matsuhashi has quit IRC11:24
apevec96919 is still all queued only11:24
apevecfor >19h11:24
*** matsuhashi has joined #openstack-infra11:24
*** mmaglana has joined #openstack-infra11:27
*** ihrachyshka has quit IRC11:27
*** ihrachyshka has joined #openstack-infra11:27
SergeyLukjanovapevec, heh, I'm not sure that it'll be merged today :(11:28
*** matsuhashi has quit IRC11:29
apevecSergeyLukjanov, fix the gate :)11:30
*** HenryG has joined #openstack-infra11:32
*** om has joined #openstack-infra11:34
*** talluri has joined #openstack-infra11:35
sdagueSpamapS: enjoy the coffee11:37
sdagueapevec: people need to fix openstack :)11:37
*** lbragstad has quit IRC11:38
*** dprince has quit IRC11:39
sdagueSergeyLukjanov: 98080,1 ended up failing because the other grenade patch wasn't there11:39
sdaguewe might just want to ninja merge those 211:39
sdaguebut that can wait until after coffee11:40
SergeyLukjanovI'm here now11:40
*** amotoki has quit IRC11:41
SergeyLukjanovsdague, I can ninja merge ;)11:41
sdaguewell, lets let the stuff above it try to get through first11:42
SergeyLukjanovsdague, yup, agree11:43
SergeyLukjanovsdague, so, you're proposing to ninja merge 98080 and 97966 after first two merged11:43
*** rfolco has joined #openstack-infra11:44
* SergeyLukjanov just like to save context :) 11:44
sdaguefrom my reading of the failure 98080 is basically just failing on the 8 way concurency11:44
sdaguelet me double check that assumption11:44
openstackgerritStephan Renatus proposed a change to openstack-infra/config: Increase timeout for chef-related jobs, 30min is not enough time
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326754
uvirtbotLaunchpad bug 1326754 in tripleo "Ubuntu CI failures  : Unable to locate package ubuntu-cloud-keyring" [High,Triaged]
sdagueoh, ffs, the top of that failed on something else11:47
sdaguewhich is actually this other thing I was just fixing11:48
*** afazekas has joined #openstack-infra11:48
sdagueSergeyLukjanov: so I'd say ninja merge 97912,2 regardless11:50
sdagueit's top of gate, but failed on what I think will help with11:51
SergeyLukjanovsdague, yup11:52
SergeyLukjanovI'm going to ninja merge it right now11:52
*** julim has joined #openstack-infra11:52
sdagueso I'm going to send a different email which is just appealing to people to work on fixing gate reseting bugs11:53
SergeyLukjanovsdague, ack11:56
SergeyLukjanovsdague, submited11:56
*** PaulMurray has joined #openstack-infra11:57
SergeyLukjanovfungi, clarkb, mordred, fyi I've submitted
*** jgallard has joined #openstack-infra11:59
*** dprince has joined #openstack-infra12:01
*** flaper87|afk is now known as flaper8712:02
*** hashar has joined #openstack-infra12:06
*** pdmars has joined #openstack-infra12:06
*** pdmars has quit IRC12:08
*** Longgeek has quit IRC12:12
*** Longgeek has joined #openstack-infra12:12
*** mriedem has joined #openstack-infra12:15
boris-42sdague around?12:16
boris-42sdague or better in qa12:17
Alex_GaynorHey all, OpenSSL security release on this fine morning just dropped, packages should be out shortly.12:17
sdagueboris-42: if you aren't fixing race conditions in the gate, I don't have time for it12:18
boris-42sdague I would like to speak about12:18
boris-42sdague one bug12:18
sdagueboris-42: ok, shoot12:18
boris-42sdague timeout one12:18
sdaguebug url?12:19
boris-42sdague one sec12:19
boris-42Bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures12:19
uvirtbotLaunchpad bug 1254890 in nova ""Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures" [High,Confirmed]
boris-42sdague ^12:19
boris-42sdague it may be cause by default setup of keystone12:19
boris-42sdague lemme show what is happening with keystone performance under load12:19
boris-42sdague after 2k iterations operation get tokens start working for a lot of time12:20
boris-42sdague and let me show how it affects heat12:20
boris-42sdague so as tempest is running a lot of stuff during 1hrs12:21
boris-42sdague there is a lot of interaction to keystone12:21
boris-42sdague at some point keystone may just suck12:21
boris-42sdague and it produce super long request to get tokens -> which produce timeout in client12:22
sdaguehave you looked at the logs in those failures12:22
sdaguebecause I've never seen that be an issue12:22
boris-42sdague I am not 100% that it is issue, but it can be12:23
boris-42sdague so probably just running under apache keystone in dsvm gate will reduce amount of timeouts12:23
sdagueboris-42: please actually look into the gate logs to find evidence for that12:23
openstackgerritA change was merged to openstack-infra/zuul: Remove trailing spaces in debug log
sdagueI really doubt it12:23
*** om has quit IRC12:23
boris-42sdague so how can I found that operation get/validate token inside any of project started working slow?12:23
boris-42sdague cause it won't fail, it will just work for a really long amount of time12:25
boris-42sdague e.g. couple of minutes..12:25
sdaguewe log all our calls here -
sdaguethe token calls are taking about 0.1s12:25
sdagueeven at the end of the run12:25
sdaguethat's a log for something that failed on this issue12:26
*** trinaths has quit IRC12:26
*** trinaths has joined #openstack-infra12:27
boris-42sdague btw what and where is specified timeout?12:27
boris-42sdague it's time-out in tempest as I understand?12:28
sdagueboris-42: you don't understand12:28
sdaguethe issue is the resource never makes the transition12:28
*** PaulMurray has left #openstack-infra12:29
*** _nadya_ has joined #openstack-infra12:30
*** palar has quit IRC12:30
sdagueboris-42: I'm happy if you'd like to help on gate things. But that means realizing that rally isn't really relevant here, and instead digging in and reading the failure logs for these issues to get to the bottom of the actual fails.12:30
boris-42sdague i just thought that that stuff can be related to this one12:30
*** unicell has joined #openstack-infra12:30
*** HenryG has quit IRC12:31
sdaguebut you did that before reading any of the logs of the failures, which just makes it a distraction. There is a ton of data that we capture on these fails.12:31
boris-42sdague so I'll investigate those logs12:31
sdagueawesome, that would be great12:31
*** pdmars has joined #openstack-infra12:31
boris-42sdague but to safe my time a bit12:31
*** hdd_ has joined #openstack-infra12:31
boris-42sdague timeout is happening inside projects?12:31
boris-42sdague or there is timeout in tempest12:32
sdaguehere is the test that fails -
sdagueand the streams we capture for it12:32
sdagueit did an operation on that resource to make to go to a different state12:32
sdagueand after polling once a second, for 196s, the resource didn't get there12:33
*** Hal has joined #openstack-infra12:33
sdagueyou can see the stream of GET calls polling for status12:33
*** Hal is now known as Guest9324112:33
sdagueyou can cross reference what the test was actually doing by jumping back to the test itself, to figure out which state transitions it was trying to make12:34
*** _nadya_ has quit IRC12:34
sdaguealso corrolate the server logs during those time windows, especially because we've got uuids for resources12:34
boris-42Server 64ee89ec-f9cc-474d-9c43-5013bb93e1ca failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: ACTIVE.12:36
boris-42sdague ^ OHH?12:36
boris-42what that means12:36
*** talluri has quit IRC12:36
boris-42ah got it12:36
*** talluri has joined #openstack-infra12:36
*** weshay has joined #openstack-infra12:37
*** talluri has quit IRC12:41
*** mbacchi has joined #openstack-infra12:41
*** dkranz has joined #openstack-infra12:42
SergeyLukjanovsdague, yay! is merged12:43
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Refresh token support
mtreinishsdague: is there not already an e-r query for the pip thing?12:43
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Refine queries for tripleo jobs
mriedemmtreinish: there isn't a good fingerprint for the pip failure12:44
mriedemmtreinish: i have a query in the bug report though12:44
*** HenryG has joined #openstack-infra12:44
mtreinishmriedem: oh yeah I can see that now, it just kind of disappears in the log without an error message12:44
openstackgerritA change was merged to openstack-infra/devstack-gate: set concurrency for grenade
mriedemmtreinish: there is err_trap at the bottom12:44
mriedemso that's what i based a logstash query on12:45
mriedembut it's probably too generic12:45
*** adalbas has joined #openstack-infra12:45
sdaguemtreinish: it's not really easy to fingerprint12:46
mtreinishwouldn't that just catch any failure in devstack though? (unless I'm misunderstanding what it is)12:46
sdaguemtreinish: it would12:46
mriedemmtreinish: that's why i didn't post that for an e-r query12:46
mtreinishok, makes sense12:46
mriedemi'll have a new unrelated one coming up soon though... :)12:46
sdagueI actually have a change to help debug it in the future12:46
mriedemor two12:46
sdaguewhich I was holding so I didn't accidently rebase the change itself12:47
SergeyLukjanovbtw get-pip change is already ninja-landed12:47
sdagueSergeyLukjanov: right, great12:47
sdague will make it possible to fingerprint in future12:47
*** lbragstad has joined #openstack-infra12:47
SergeyLukjanovsdague, all 4 changes are now landed12:48
sdagueSergeyLukjanov: awesome12:49
*** BAKfr has joined #openstack-infra12:50
*** _nadya_ has joined #openstack-infra12:51
*** aburaschi has joined #openstack-infra12:52
*** radez_g0n3 is now known as radez12:53
*** BobBallAway is now known as BobBall12:55
*** miqui has joined #openstack-infra12:57
*** flaper87 is now known as flaper87|afk13:02
*** liam__ has joined #openstack-infra13:02
*** liam__ has quit IRC13:02
*** jamespage_ has joined #openstack-infra13:03
*** jamespage_ has quit IRC13:04
*** zz_gondoi is now known as gondoi13:07
*** jergerber has joined #openstack-infra13:07
*** HenryG has quit IRC13:08
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova resize/migrate bug 1326778
*** om has joined #openstack-infra13:09
uvirtbotLaunchpad bug 1326778 in nova "test_list_migrations_in_flavor_resize_situation fails with 400 after VERIFY_RESIZE state" [Medium,Confirmed]
*** smarcet has joined #openstack-infra13:09
*** dizquierdo is now known as dizquierdo_afk13:14
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for nova resize/migrate bug 1326778
uvirtbotLaunchpad bug 1326778 in nova "test_list_migrations_in_flavor_resize_situation fails with 400 after VERIFY_RESIZE state" [Medium,Confirmed]
*** rpodolyaka has joined #openstack-infra13:14
*** om has quit IRC13:18
openstackgerritA change was merged to openstack-infra/elastic-recheck: Suppress the graph for bug 1260654
uvirtbotLaunchpad bug 1260654 in openstack-ci "Could not initialize class jenkins.model.Jenkins$MasterComputer" [Critical,Fix released]
*** IvanBerezovskiy has joined #openstack-infra13:19
*** niska has quit IRC13:21
*** jergerber has quit IRC13:22
*** samuelmz has quit IRC13:24
*** luqas has joined #openstack-infra13:24
*** niska has joined #openstack-infra13:26
*** mfer has joined #openstack-infra13:26
*** bknudson has joined #openstack-infra13:27
*** dkliban_afk is now known as dkliban13:28
*** doug-fish has joined #openstack-infra13:29
anteayaI'm going to look at this list gate-ceilometer-python27 : 17 Uncategorized Fails. 15.0% Classification Rate (20 Total Fails)13:29
*** HenryG has joined #openstack-infra13:29
sdagueSergeyLukjanov: what's the log server look like?13:31
*** yfried has quit IRC13:31
sdagueanteaya: thanks13:31
sdagueSergeyLukjanov: is suspiciously truncated13:31
sdagueas are other logs on that run13:31
SergeyLukjanovsdague, me looking13:31
*** wenlock has joined #openstack-infra13:32
SergeyLukjanovsdague, at least, we have enough disk space13:33
SergeyLukjanov>> /dev/mapper/main-logs          12T  9.7T  2.2T  83% /srv/static/logs13:34
*** om has joined #openstack-infra13:34
sdagueI wonder if we are running out of space on these new hp nodes during the gzip13:35
sdaguebecause it's only the big ones that are truncated13:35
SergeyLukjanovsdague, no, enough space and the file itself isn't truncated on server13:35
sdaguethough / is 30 G13:35
*** homeless has joined #openstack-infra13:36
fungiwow, that's a fair amount of scrollback i lack time to read. is there anything urgent which needs my attention?13:36
SergeyLukjanovfungi, I've ninja-submitted get-pip fix13:37
SergeyLukjanovand 4 more changes has been promoted to make gate a bit healthy13:37
sdaguefungi: yeh, using as our message bus13:37
sdagueso the things that have been done should be in there13:38
SergeyLukjanovsdague, oh, I'm wrong, it's truncated on server too - last line is "2014-06-04 13:11:24.245 AUDIT [req-2fd8e9ec-f180-465e-829d-b105433c07a1 ServersTestXML"13:38
KiallIs it normal for a +2 to "stick around" after a new patchset is uploaded?
russellbfungi: no, i missed the freeze13:38
fungisdague: SergeyLukjanov: thanks!13:39
russellbapevec: sounds like we merged without an exception13:39
BAKfrI've a question for the gerrit developers13:40
anteayasdague: so the 5 errors on the bottom of the list are all multiple db errors, the other 15 are events related to 3 failed tests, I'll work on fingerprints of the 3 failed tests first13:40
sdaguefungi: if you have any thoughts on why - is truncated13:40
sdagueanteaya: thanks13:40
BAKfrIs it normal that some commits from "openstack" branches are not used ?13:40
fungiBAKfr: if by "the gerrit developers" you mean the ones at google who write gerrit, you can find them in #gerrit on freenode13:40
luqashi, I would like to change the e-mail contact for our 3rd party gerrit account but I can't register a new e-mail, any idea?13:41
fungiBAKfr: oh, a question for openstack's gerrit admins. can you elaborate? i'm not sure what you mean13:41
fungisdague: looking13:41
*** signed8bit has joined #openstack-infra13:41
*** otherwiseguy has joined #openstack-infra13:42
sdague3 of the larger log files in that run are truncated13:42
BAKfrSomes commits done for old versions are not used13:42
BAKfrlike this:
sdaguebefore I write it off to sun spots, I want to make sure it's not systemic13:43
fungisdague: all screen logs? maybe screen sessions weren't terminated properly before log copying began and buffered writes were lost? (completely off-the-wall theory)13:43
anteayaluqas: hold on that thought, in a bit of time we will have a system where you will have a wiki page and can edit that contact email until your heart's content13:43
BAKfrWe've a few commits in an unused "openstack" branch .13:43
sdaguefungi: yeh, I was only looking at screen logs13:43
*** om has quit IRC13:43
*** dangers_away is now known as dangers13:43
luqasanteaya: thanks :-)13:43
anteayaluqas: I will find you if there is an issue with midokura13:44
fungiBAKfr: ahh, you're talking about custom patches to gerrit. a couple months ago upstream gerrit finally got to the point where it was mostly usable for us. there are a few things like that ugly hack you pointed out which we couldn't really find a good upstream solution for. point is we're trying not to actually modify the gerrit source code for the instance we're running now13:45
fungiluqas: yeah, we're about to blank out the e-mail addresses in gerrit for all those service accounts and move them to a wiki13:45
fungiluqas: partly to work around a limitation in gerrit where two accounts can't share the same e-mail address, and it would be nice for service account administrators to be able to list their normal e-mail address as a point of contact but still use the same address for their normal gerrit user too13:46
BAKfrfungi, it's answers my interrogations. thanks13:47
sdaguefungi / SergeyLukjanov: after the cinder change lands, I'd say promote 9810813:47
sdaguethat's what killed nova change #213:47
sdagueand the tempest change13:47
sdaguethe cinder change just landed13:47
sdagueso it's good promoting time13:47
*** habib has joined #openstack-infra13:47
fungiBAKfr: probably the best way for us to get gerrit 2.9 or later to solve that problem would be a patch that could (with some configuration) pass through the response from the remote api and embed it into the on-screen error message13:48
*** sandywalsh_ has joined #openstack-infra13:48
luqasanteaya: fungi: ok, I'll wait then it wasn't something critical, thanks for your clarification13:48
SergeyLukjanovsdague, okay, I'll do it13:48
sdagueSergeyLukjanov: it needs to be moved to the gate queue13:48
fungiBAKfr: but "soon" we anticipate having our own separate openid provider, at which point we can provide a much tighter and lighter-weight integration between foundation system accounts and gerrit accounts, so we can drop the hackish way we're using the gerrit contactstore feature to do that today13:49
*** dmakogon_ has joined #openstack-infra13:49
openstackgerritA change was merged to openstack-infra/config: Add dib-utils project
*** otherwiseguy has quit IRC13:49
SergeyLukjanovsdague, is cinder change already landed?13:50
sdagueSergeyLukjanov: yeh13:50
sdagueit was in front of the failing nova change13:50
SergeyLukjanovsdague, okay, promoting 9810813:50
SergeyLukjanovsdague, I was too slow to see it ;)13:50
*** prad has joined #openstack-infra13:50
sdaguefungi: is hosted at rax?13:51
fungisdague: yeah, it's just an apache vhost on static.openstack.org13:51
fungiit's in rax dfw13:51
fungisdague: looking into connectivity issues for it?13:52
SergeyLukjanovsdague, 98108 promoted13:52
sdagueit's just a couple, but hp region b3 => pypi.openstack.org13:52
sdaguefailures to pip install13:52
*** otherwiseguy has joined #openstack-infra13:52
sdaguefungi: is there already an appropriate bug for that?13:53
fungisdague: ianw reported a bug yesterday about stray tcp/rst on connections in hpcloud 1.1 and was able to get the service tech who picked it up to reproduce it consistently13:53
fungi(in hp cloud services)13:53
fungithough that says socket timeout, not connection reset by peer13:54
fungimight be a separate issue13:54
sdagueyeh, this is the best way to fingerprint pip just choking13:54
*** pcrews has joined #openstack-infra13:54
fungisdague: i think we have blanket bugs for "the internet sucks" which are perpetually open or replaced by new ones when lp auto-closes them for inactivity. don't know the bug numbers off the top of my head13:54
sdaguewell, I'd like to track this one. It's a little different than just internet sucks13:55
sdaguebecause it's pip to our servers from our servers13:55
sdagueI'll just open a new one13:55
*** yaguang has quit IRC13:55
*** sballe has joined #openstack-infra13:55
fungiactually, it seems far less terrible than hpcloud 1.0 az2 was, from a network stability standpoint13:56
fungiconnections from there used to crap out very frequently13:56
dmakogon_hello, guys, i caught a problem with launching keystone,, timestamp 13:20:4513:56
*** arnaud__ has joined #openstack-infra13:57
*** d0ugal has joined #openstack-infra13:57
fungidmakogon_: ERROR: Exception raised: six>=1.6.013:58
fungithat or is it breaking down earlier?13:58
dmakogon_fungi, but before that keystone wasn't able to start13:58
dmakogon_fungi, key failed to start13:58
dmakogon_fungi, ah, get it, seems that it's true, six and client problem13:59
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: tracking bug for connection issues between HP and RAX
fungidmakogon_: i only see that string in the xtrace output from the screen command13:59
*** otherwiseguy has quit IRC13:59
fungidmakogon_: it doesn't actually get echoed, so keystone appears to have started right up14:00
dmakogon_fungi, than it seems that the problem is in openstackclient14:00
*** crc32 has joined #openstack-infra14:00
fungidmakogon_: yeah, looks like it14:01
BobBallAny idea my nodepool doc change fails unit tests? Doesn't seem like something that could be a race, but it passes in my local env:
*** gokrokve has joined #openstack-infra14:01
*** markwash has joined #openstack-infra14:03
dmakogon_fungi, could you please tell who's on charge of openstackclient ?14:03
fungidmakogon_: dtroyer14:03
BobBallPerhaps I don't understand the test, but it seems to be doing the same read-only(!) thing twice, with the second time it gets a different answer... :)14:03
fungiBobBall: huh... No handlers could be found for logger "nodepool.NodePool"14:03
*** _nadya_ has quit IRC14:03
BobBalloh - I saw that one before... let me look for something to do with that.  I was just focused on the final error14:03
BobBallthe mismatch14:04
BobBall2014-04-01T16:53:01  <fungi> jeblair: are the 'No handlers could be found for logger "nodepool.NodePool"' errors during test_subnodes on your radar?14:04
*** _nadya_ has joined #openstack-infra14:04
BobBall:) I was _just_ looking at that...14:04
fungiBobBall: so the MismatchError: 3 != 214:04
fungiBobBall: oh, that seems to have cropped up after the timeout14:04
dmakogon_dtroyer, ping14:05
anteayadmakogon_: he has been away since last week, I haven't seen him around yet14:05
fungiBobBall: so it looks like it threw an exception in test_subnodes, hung, then got killed via timeout about 35 minutes later14:06
anteayathough I am told he is expected back by week's end, which could be today, tomorrow or Monday14:06
dmakogon_anteaya, thanks14:06
*** pblaho has joined #openstack-infra14:07
*** Mithrandir has quit IRC14:08
*** yfried has joined #openstack-infra14:10
fungiBobBall: also that traceback looks slightly unpossible... i think it's actually truncated and then interrupted by the "no loggers" error14:10
fungimaybe subunit crossed the streams14:10
BobBallnever cross the streams!14:10
fungii'm going to recheck no bug that change just to see if this is repeatable or some stray weirdness14:11
BobBallok, thanks.  I don't like bugs that don't give any details but couldn't figure out what to put in a bug report!14:11
*** wenlock has quit IRC14:11
fungiran in rax dfw, which aside from boot errors has been pretty solid lately, so seems marginally unlikely that the disk fell out from under it or something14:11
anteayasdague: I'm doing something wrong - I should have 5 hits for the logstash query: message:"FAIL: tests.alarm.test_rpc"14:12
*** Mithrandir has joined #openstack-infra14:12
fungianteaya: did you change the query timeframe to something other than the default "last 15 minutes"?14:14
anteayasdague: the oldest is May 30th14:14
anteayafungi: yes, All Time14:14
*** e0ne_ has quit IRC14:14
anteayawhich is the last 2 weeks, I do believe14:14
*** e0ne has joined #openstack-infra14:14
sdagueanteaya: log ?14:14
fungi"all time" also beats up the cluster pretty badly, i recall clarkb saying14:14
sdagueyeh, 7 days is probably the right thing to do14:15
anteayaI can switch to 7 days14:15
*** andreaf has quit IRC14:16
anteayalast 7 days, that query, no results returned14:16
sdagueanteaya: I think '.' is not tokenized14:16
sdagueso you have to search for the whole test14:17
sdagueotherwise it's not matching it as a word14:17
sdaguemessage:"FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence"14:17
anteayasorry june 3 for that one14:17
anteayathat failed on me as well14:17
anteayaI will try again though14:17
sdaguetests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence is treated as a single word14:17
sdaguethat one just returned for me14:17
sdaguefungi: oh, that was the thing Alex_Gaynor pinged about. openssl patch14:18
fungisdague: yeah, we'll have it when ubuntu auto updates. i'll double-check but if it's not ready the alternative is that we all take a holiday14:18
anteaya28 hits, I wonder if there is a bug open for it yet14:19
*** viktors is now known as viktors|afk14:19
sdaguefungi: :)14:19
sdaguewell could just turn off +A rights in gerrit, and tell everyone to go away for 2 days14:19
fungispot checking systems, but i'll also see if centos has patches out yet14:20
fungion precise it looks like we want to be using libssl1.0.0 1.0.1-4ubuntu5.1414:20
openstackgerritKen'ichi Ohmichi proposed a change to openstack/requirements: Add rfc3987 library
*** aysyd has joined #openstack-infra14:22
*** unicell has quit IRC14:22
apevecrussellb, fixes for gate issues have free ride14:22
apevecrussellb, I'd just appreciate setting the target in LP14:23
fungiapevec: good point. broken stable gate means no stable point releases ;)14:23
russellbapevec: k14:23
*** primemin1sterp is now known as primeministerp14:23
*** primeministerp has quit IRC14:24
apevecfungi, indeed14:24
*** primeministerp has joined #openstack-infra14:24
fungifixed package version for rhel 6 seems to be openssl-1.0.1e-16.el6_5.14.i686.rpm so will probably be the same on centos once they get it ported in14:25
fungibut i don't see an announcement for it yet14:25
fungior probably openssl-1.0.1e-16.el6_5.14.x86_64.rpm in our case14:26
fungiugh. i wish rackspace wouldn't configure their ubuntu base images for security updates from their mirror. it hasn't updated yet and so lacks fixed packages14:28
openstackgerritThierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326820
uvirtbotLaunchpad bug 1326820 in ceilometer "FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence" [Undecided,New]
anteayasdague: my line length is probably wrong in that one, I'm not sure how to fix it14:29
dstufftoh yea14:29
dstufftI need to update some OpenSSL14:29
*** basha has quit IRC14:29
fungidstufft: and talk to your base image people about not configuring customers to use an outdated security repository14:30
*** gokrokve has quit IRC14:30
dstufftI'm not sure who those people are14:30
*** basha has joined #openstack-infra14:30
dstufftI could probably figure it out14:30
fungiyeah, me neither :/14:30
openstackgerritThierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication
fungiworth noting, this is why debian (even well before ubuntu) configured a separate repository for security updates... so that users could get release packages from a very laggy mirror but still get the latest security updates straight from the distro14:31
*** Longgeek has quit IRC14:33
fungirestarting apache on review.o.o, so there could be a blip from the webui14:33
fungiseems to be still working14:34
sdaguefungi: I know DinaBelova was looking into it, but there is a way to get back reseting votes on commit message changes right?14:34
fungisdague: i *think* it's a config option. looking real fast14:34
sdaguebecause the lack of that behavior is killing me. About 1/3 of feedback follow up I have to do manually because people's votes are sticky14:34
DinaBelovafungi, oh, thank you!14:34
sdagueand their feedback was about adding a bug14:34
*** jgrimm has joined #openstack-infra14:35
sdagueor they get mad at me that I have a -1 on their patch after they updated a thing in the message, and I never knew it14:35
*** Lingo_ has joined #openstack-infra14:36
*** Lingo_ is now known as TravelingBear14:37
*** maxbit has joined #openstack-infra14:37
*** skolekonov has quit IRC14:37
*** gokrokve has joined #openstack-infra14:39
*** TravelingBear is now known as BadCub0114:39
*** wenlock has joined #openstack-infra14:40
*** pblaho has quit IRC14:40
*** mburned is now known as mburned_out14:42
fungisdague: DinaBelova:
*** dims_ has joined #openstack-infra14:44
fungistrangely, that doesn't seem editable in the All-Projects acl within the gerrit webui14:44
fungiwe may need to edit and push an update to it via git14:44
*** jaypipes has joined #openstack-infra14:45
fungisdague: DinaBelova:;a=blob;f=project.config;h=ca6b31f;hb=HEAD#l10014:46
DinaBelovafungi, oh, really strange!14:46
*** timrc-afk is now known as timrc14:46
*** thedodd has joined #openstack-infra14:47
*** mburned_out is now known as mburned14:47
fungibut yeah, docs say it defaults to false, so we could just remove that one line. i'll wait for confirmation from clarkb and zaro that it's sane/desirable, but seems like a reasonable change to me14:48
fungigets us closer to the earlier behavior we had in 2.414:48
*** doddstack has joined #openstack-infra14:49
DinaBelovafungi, well - it's not only close behaviour14:49
DinaBelovathat's more logical really14:49
*** trinaths has quit IRC14:49
fungiBobBall: it's doing it again!
*** rdopiera has quit IRC14:49
DinaBelovaas in the open projects keeping code/commit messages clean is also important14:50
DinaBelovaand sometimes that's good reason for -114:50
fungiBobBall: succeeded in check this time, but now hanging in the gate instead14:50
BobBallThat sucks! :(14:50
fungiBobBall: so it looks like we probably have a race condition in the test14:50
BobBallquite a common one if it's hitting me 2 out of 3 times...14:51
*** thedodd has quit IRC14:51
fungiDinaBelova: i completely agree. that's why we didn't carry scores over on commit message changes before, just pointing out that this gets us closer to that behavior again14:51
fungiBobBall: yes, probably introduced in a recent patch. we've had a lot of fixes go in which required new tests or changes to existing ones for nodepool14:52
BobBallit's a parallel tox thing fungi14:52
DinaBelovafungi, yeah, that's true :)14:52
BobBallI bet you anything :D we're polling the DB for nodes to see how many are ready14:53
BobBallbut at some point another test will be reserving one of those nodes surely14:53
fungiBobBall: oho, probably right!14:53
BobBallwhich means a 3rd node is built14:53
fungiBobBall: well, or holding a lock or something14:53
*** jgrimm has quit IRC14:53
fungiBobBall: though i thought we were using per-runner databases for these tests to avoid that sort of thing14:54
* fungi looks at the fixtures14:54
BobBalldburi: '{dburi}'14:54
*** BAKfr has left #openstack-infra14:54
BobBallwe're always using openstack_citest aren't we?14:55
fungibut regardless, if you're not running this tox env with 4-way parallelization then that could explain why we hit it in ci but you're unable to reproduce locally14:55
BobBallI am - but even then, it's a race condition rather than guaranteed14:55
BobBalland I've got a shiny new machine... so I hope it's faster than the gate ;)14:55
BobBalloh hang on - I see it now14:57
BobBallwe do create a custom DB for each test14:57
BobBallpoo - bang goes my theory14:57
fungiyeah, overriding TestNodepool.dburi14:57
*** achuprin has quit IRC14:58
fungialso, this time it broke in test_node_az, so entirely different test14:58
*** jistr has quit IRC14:58
fungiand a very recently added one (came in with nova os-ext-az support for dealing with hpcloud 1.1 this week)14:59
openstackgerritA change was merged to openstack-infra/storyboard: Added sort parameters to API
*** jistr has joined #openstack-infra15:00
openstackgerritA change was merged to openstack-infra/storyboard: Refresh token support
fungii think i probably want clarkb's eyes on this first thing once he's around, because we were planning to restart nodepool but this could imply a new bug, in which case we're better off identifying it in some way other than running it in production15:00
*** hdd_ has quit IRC15:01
*** otherwiseguy has joined #openstack-infra15:02
*** timrc is now known as timrc-afk15:03
*** timrc-afk is now known as timrc15:06
*** atiwari has joined #openstack-infra15:08
*** eharney has joined #openstack-infra15:09
*** achuprin has joined #openstack-infra15:11
sdaguefungi / SergeyLukjanov - we need a repromote of - - it got hit by another thing15:12
*** radez is now known as radez_g0n315:13
*** dprince has quit IRC15:13
fungiDetails: Service cinder-backup could not be found.15:13
SergeyLukjanovsdague, fungi, I'm able to promote something15:14
SergeyLukjanovis it still needed?15:14
fungiSergeyLukjanov: sure15:14
fungii haven't really started yet15:15
fungigo for it15:15
sdaguefungi: yeh, we need the failure logs that mtreinish has most of a patch for15:15
sdagueI'm also going to try to add some better sanity checks on grenade15:15
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for bug 1323524
SergeyLukjanovfungi, sdague, done15:16
uvirtbotLaunchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed]
sdagueSergeyLukjanov: thanks15:16
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Refresh token support
SergeyLukjanovheh, today is the promoting day15:17
*** ihrachyshka has quit IRC15:18
mtreinishfungi, sdague: I think that patch already merged:
devanandamorning, folks15:19
anteayamorning devananda15:19
sdaguemtreinish: ok, that would indicate that it didn't create that file15:20
devanandafungi: any chance we can bump 97757 to the top of the gate queue? ironic and tripleo are both broken/blocked until this lands -- we tried to land it yesterday, i'm not sure what happened (I had an appt to run to and lifeless / clarkb were working on it when I left)15:20
devanandait looks like it got rechecked and is now at the tail of the queue15:20
mtreinishsdague: or that I screwed something up in the patch15:20
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1321826
uvirtbotLaunchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress]
*** patelna has joined #openstack-infra15:23
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1321826
uvirtbotLaunchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress]
devanandamaybe i'm misinterpreting the zuul status page, but why does it look like non-integrated projects are gating in series with integrated projects?15:26
*** virmitio has joined #openstack-infra15:26
fungidevananda: possibly because they share a job they shouldn't15:26
anteayasdague: in this set of 8 failures, 5 patches failed with the same 8 failures, should I pick one failure and create a fingerprint from that?
sdagueanteaya: yep15:28
fungidevananda: or maybe that's intentional15:28
fungidevananda: which ones?15:28
devanandafungi: eg, if "gate-tempest-dsvm-full" is in an incubated project's gate check, would taht serialize it with, say, nova?15:29
*** gargola has quit IRC15:29
fungidevananda: yes. which incubated project are you seeing that for?15:29
*** gargola has joined #openstack-infra15:30
devanandafungi: i observed that for ironic yesterday15:30
anteayasdague: there have been no hits since May 30th on that fingerprint, categorize it anyway?15:30
devanandafungi: also some projects in related programs, es taskflow (in oslo) and tripleo-image-elements (in tripleo) are also serialized15:31
anteayamessage:"FAIL: tests.api.v2.test_acl_scenarios.TestAPIACL.test_authenticated(sqlite)"15:31
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Treat tripleo builds as openstack projects
fungidevananda: ahh, i don't immediately see jobs for ironic which would have that effect15:32
devanandafungi: none are running at the moment -- we can't land anything until 97757 merges in nova15:32
devanandafungi: so we aren't even approving patches right now15:32
fungidevananda: yeah, more that i was looking at the jobs defined for it in zuul's layout.yaml file15:33
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova pep8 gate fail bug 1326848
uvirtbotLaunchpad bug 1326848 in nova "pep8 F481 checks got caught in the gate" [Undecided,New]
sdagueanteaya: yeh, if it pulls stuff off the uncategorized list15:34
*** jgallard has quit IRC15:34
anteayasdague: k15:35
phschwartzjeblair: fungi: mordred: Sorry I was incapacitated the past few days due to my back. I will take a look at the iad region and also have a member of ops take a look. Are you still seeing a large number of ERROR state instances?15:35
fungiphschwartz: not sure about error state... yesterday's interesting issue was that you seem to have at least one compute node in iad starting instances with time offset by 4 hours (probably configured to edt rather than utc or something)15:36
phschwartzfungi: hmm, weird. Do you have a uuid for an instance that came up that way that I can look at? They should all be utc so that would need to be fixed.15:37
sdaguedevananda: the relationships are transitive15:37
fungiphschwartz: i can get you a uuid, though the instance associated with it is long burned and buried15:37
fungijust a sec15:37
sdagueso if there is any path, anywhere, that all these changes can connect15:37
phschwartzWe still have logs that we can see what hv it was on15:37
sdaguethen they are serialized15:38
*** arnaud__ has quit IRC15:38
*** weshay has quit IRC15:39
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326754
uvirtbotLaunchpad bug 1326754 in tripleo "Ubuntu CI failures  : Unable to locate package ubuntu-cloud-keyring" [High,Triaged]
mordred_phonephschwartz: do you have a copy of our sekrit graphs?15:39
phschwartzmordred_phone: No, I do not.15:39
devanandasdague: "can connect" as defined by the matrix of tests that each job runs?15:39
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1324885
uvirtbotLaunchpad bug 1324885 in ceilometer/icehouse "py26, py27 gate jobs are failing due to inaccurate mocks of keystone tokens" [Critical,Fix committed]
sdaguedevananda: correct15:40
devanandasdague: ack, thanks15:40
*** mrodden has joined #openstack-infra15:41
devanandaso, back to my first question :) any way we can bump 97757 to the top of the queue, so that ironic and tripleo aren't totally blocked for another >24hrs?15:41
anteayaI'm moving on to: gate-ceilometer-python26 : 21 Uncategorized Fails. 19.2% Classification Rate (26 Total Fails)15:41
sdaguedevananda: there is an open question on that15:42
sdagueare the ironic unit tests failed related or not?15:42
sdaguebecause ironic is also failing unit tests15:42
sdague -15:42
devanandaoh? hrm15:42
sdaguethat's our message bus for gate failures15:42
sdagueso I nixed the reenqueue this morning because of that until there is an answer15:43
devanandasdague: where is the failed unit test?15:44
sdagueon any ironic change in the last 24 hrs15:44
fungiwow... novaclient 95426,2 is a serious mess (failing all integration jobs). must be the victim of another api change15:45
sdaguedevananda: I pulled 5 changes manually out of the gate yesterday because of it15:45
*** unicell has joined #openstack-infra15:45
sdagueso please get the ironic house in order first, then we can talk about promotes from other projects15:45
devanandasdague: ack, looking into it. i'm not aware of any *unit* test failures in Ironic -- only the brokenness caused by the nova change that 97757 is reverting15:46
fungioh, or maybe the victim of more rax mirror issues... "Failed to fetch  404  Not Found"15:46
*** jistr has quit IRC15:46
sdaguedevananda: then you haven't been looking at jobs in the last 24 hrs :)15:46
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for nova pep8 gate fail bug 1326848
uvirtbotLaunchpad bug 1326848 in nova "pep8 F481 checks got caught in the gate" [Undecided,New]
fungi(that's the security update related to today's openssl usn)15:46
sdaguethere is one in check run now that's failed15:46
*** afazekas has quit IRC15:47
devanandasdague: last 12 - sure. but I was looking during the 12 before that15:47
*** jistr has joined #openstack-infra15:47
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer test_ordination_assign timeout bug 1326855
uvirtbotLaunchpad bug 1326855 in ceilometer ""tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_assign" race in the gate" [Undecided,New]
openstackgerritMatthew Treinish proposed a change to openstack-infra/devstack-gate: Capture tempest failure files
fungiphschwartz: do you have any involvement with the package mirrors in rax?15:48
dstufftHey, can someone approve ? :D15:48
phschwartzfungi: I do not, but I could find someone that deals with them15:48
*** gokrokve has quit IRC15:49
fungiphschwartz: i think i have a note for someone's irc nick or e-mail address, though it could be cold now15:49
mordredsdague: the thing I find fascinating is that the gate wasn't in bad shape before setuptools broke the world15:49
*** jcoufal has quit IRC15:49
fungimordred: i think that was coincidental timing15:49
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Treat tripleo builds as openstack projects
sdaguemordred: I have a new theory15:49
sdaguemost of what's in the gate queue has never seen an hp 1.1 node15:50
sdaguebecause it passed on rax15:50
fungithis is entirely probable15:50
sdaguewhen we were down a cloud15:50
sdagueso the timing differences might be shaking things out15:50
devanandasdague: that unit test failure in the check queue IS because ofthe nova change - =
devananda2014-06-05 15:36:28.836 | TypeError: __init__() takes at most 4 arguments (5 given)15:50
sdaguedevananda: ok, that's why I asked the question15:50
sdagueand... wat15:51
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Refine queries for tripleo jobs
mordredsdague: ah. fascinating15:51
sdaguewhy are you importing nova in unit tests?15:51
devanandasdague: heh. cause we want to run unit tests on the noav.virt.ironic driver15:51
fungiphschwartz: my notes say carl thompson (all i have for him is an e-mail address)15:51
*** che-arne has joined #openstack-infra15:51
phschwartzLet me see if that is still correct15:51
phschwartzmight take a few15:51
*** radez_g0n3 is now known as radez15:51
devanandasdague: there was a long conversation during I3 about how we could test that out of nova's tree. this is what we're doing to maintain an out of tree driver15:52
sdaguedevananda: ok, can you update the etherpad with that piece of information15:52
devanandasdague: it sucks gating on a nova driver not in nova ... really15:52
*** ominakov has quit IRC15:52
fungiphschwartz: looks like the mirror index updated before the package to which it referred was present, so it's possible your mirroring algorithm doesn't grok sane update ordering for apt repositories15:52
sdaguedevananda: you could always make that part a non voting job15:52
sdaguedevananda: also, I think the transitive dependency is oslotest15:53
sdaguetripleo and ironic cross test with it15:53
sdagueand it cross tests with oslo.messaging15:53
*** rcarrillocruz has quit IRC15:53
*** yolanda has joined #openstack-infra15:53
sdaguewhich gets you back to a tempest-dsvm-full universe15:53
*** ayoung has joined #openstack-infra15:53
ayoungmordred, Zuul merge queue is at 34 hr 34 min.  I assume this is not news?15:54
fungiayoung: depends on the size of the rock you live under15:54
sdagueayoung: apparently you don't read email15:54
yolandahi, mordred, taking a look at your patches about dib and nodepool. But i don't see in nodepool where images-dir and elements-dir are used, do we need some specific version of nodepool?15:54
* NobodyCam needs a bigger rock15:54
ayoungsdague, I went to be with an empty mailbox.  I woke up with 250 message in it...still clearing15:55
* sdague also wants a bigger rock15:55
mordredyolanda: I have a half-finished patch to nodepool15:55
fungiNobodyCam: let me know when you find one, and if you have room for me too15:55
mordredyolanda: that does not work15:55
yolandamordred, what's missing in nodepool? is there so much work pending?15:56
mordredyolanda: that patch ^^ needs to be finished15:56
mordredwhich is mainly about doing sane things with glanceclient15:56
*** ihrachyshka has joined #openstack-infra15:58
*** rcarrillocruz has joined #openstack-infra15:58
*** doude has quit IRC16:00
*** ihrachyshka has quit IRC16:00
*** ihrachyshka has joined #openstack-infra16:00
openstackgerritA change was merged to openstack-infra/config: Adding barbican-specs to infra configs
*** dizquierdo_afk is now known as dizquierdo16:01
*** praneshp has joined #openstack-infra16:02
fungisdague: interestingly, we got a burp of nodes in many places (mostly hpcloud) with time which was waaay off between 20:48 and 20:52 yesterday, right as the ntpdate fix went in... "Job timeout set to: -23365133 minutes" message:"timeout: invalid option -- '2'" AND tags:console16:02
*** marcoemorais has joined #openstack-infra16:02
*** marun_afk is now known as marun16:02
*** IvanBerezovskiy has left #openstack-infra16:02
*** praneshp has quit IRC16:02
sdaguefungi: that was me screwing up that patch16:03
sdaguethat's the fail log for that patch16:03
sdaguein the bad version16:03
fungisdague: oh! right, those were the check tests16:03
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer keystone mock bug 1324885
uvirtbotLaunchpad bug 1324885 in ceilometer/icehouse "py26, py27 gate jobs are failing due to inaccurate mocks of keystone tokens" [Critical,Fix committed]
fungiNobodyCam: heh16:05
*** starmer has joined #openstack-infra16:05
*** weshay has joined #openstack-infra16:06
NobodyCamthat may show my real age.. but I had to :-p16:06
devanandasdague: are there any other concerns prior to promoting 97757 ?16:06
fungiNobodyCam: nothing to be embarrassed about there... we were lucky and got to watch snl in the couple of years before it began to suck16:06
sdaguedevananda: not if you are sure it's all the same issue16:07
sdaguedevananda: but before we do that, how will ironic and nova get over this hump?16:07
*** _afazekas is now known as afazekas16:08
devanandasdague: a) the patch which 97757 reverts broke ironic's gate, therefor b) it's not possible any other broken code landed in ironic after that, and c) we know reverting 97757 fixes ironic, because we've all been doing that for the last ~24 hours locally16:08
*** lcheng_ has joined #openstack-infra16:08
*** lcheng_ has quit IRC16:08
sdaguedevananda: right I get that16:09
devanandasdague: well, some of us have been applying 97447, which is a local fix to ironic, too, but it actually conflicts with the 97757 revert, so we've since blocked it ...16:09
sdaguebut is there a coming change to nova that's going to do this again?16:09
sdagueor is there a non disruptive way to get the behavior that's desired16:09
devanandasdague: the first patch that broke things actually is doing mostly the right thing16:10
devanandasdague: but it did so by changing the function signature of HostManager __init__16:10
devanandasdague: which neither nova reviewers nor I caught16:10
*** starmer has quit IRC16:10
dstufftAnyone able to review/approve ? :D It's for adding barbican-specs to the infra configs stuff16:10
*** mmaglana has quit IRC16:10
devanandasdague: so AFAIK, the plan is to revert it (unbreaks nova_bm and ironic), then redo the original patch in a way that doesn't break things16:11
*** sandywalsh_ has quit IRC16:11
sdaguegotcha. So there some unit test proposed to prevent that change on the nova side with a scary "this is a contract, don't change it"?16:11
*** basha has quit IRC16:11
devanandajogo: around? want to chime in?16:11
*** mmaglana has joined #openstack-infra16:11
*** p5ntangle has joined #openstack-infra16:12
devanandasdague: not afaik, but others (jogo and lifeless) were working on this stuff when I had to run yesterday, and i haven't been able to sync with them yet today16:13
*** e0ne has quit IRC16:13
sdaguefungi: so top of gate, which is the nova issue that's killing lots of stuff, is bouncing off one of the generic neutron network failures16:13
sdaguecan we just ninja that in, I've got another alleviating fix which can't seem to pass until that one is in16:14
*** p5ntangle has quit IRC16:14
fungisdague: as long as nova cores are confident the only failures are due to bugs entirely unrelated to that change, then sure16:15
SergeyLukjanovI have highlighting for word "ninja" :)16:15
sdaguefungi: it only fails on the neutron job16:15
sdagueI'm confident, and still a nova core :)16:15
fungisdague: 98108,1?16:15
sdagueit's the #2 bug on -
*** mmaglana has quit IRC16:16
mordredsdague: that's a good question above - we should figure out a way to test some of these interface contracts16:16
mordredsdague: it's code, afterall, we should be able to figure that out16:16
sdaguemordred: yeh, if ironic team is depending on certain behavior in nova not to change, the thing to do is build unit tests for that16:16
sdaguein nova16:16
sdaguewith big "don't touch" on it16:16
fungisdague: once gate-tempest-dsvm-postgres-full completes on it, i'll elevate my privs and submit it to merge16:17
sdaguefungi: sounds good16:17
*** gyee has joined #openstack-infra16:17
mordredsdague: ++16:17
devanandaright. fwiw, ironic is depending on several /internal/ interfaces in Nova16:18
mordredsdague: someone could always change the unittest in concert - but that would be more likely to be caught in review16:18
sdaguemordred: sure, but put them in test_ironic_contract.py16:18
mordredsdague: ++16:18
*** timrc is now known as timrc-afk16:18
sdagueand then have the ironic team put gerrit watch on it16:18
mordredif someone really needs to change it - then at least everyone is aware and can plan for it16:19
*** openstackgerrit has quit IRC16:20
*** starmer has joined #openstack-infra16:20
*** jgrimm has joined #openstack-infra16:21
*** openstackgerrit has joined #openstack-infra16:21
sdagueright, and you also get to have a very specific conversation about which interfaces everyone agrees on in review16:21
sdaguethat should reduce the surprises16:21
*** Fabio has joined #openstack-infra16:22
*** weshay has quit IRC16:22
*** tkelsey has quit IRC16:23
*** Guest93241 has quit IRC16:23
*** luqas has quit IRC16:25
*** starmer has quit IRC16:26
*** hashar has quit IRC16:26
*** ramashri has joined #openstack-infra16:26
*** sballe has quit IRC16:26
*** ayoung is now known as ayoung-lunch16:28
*** om has joined #openstack-infra16:28
*** che-arne has quit IRC16:29
sdaguefungi: ninja merge on ?16:31
sdagueit's done and reported16:31
*** jistr has quit IRC16:31
Shrewssdague: i'll work on the "scary contract change" patch16:31
*** doddstack has quit IRC16:32
fungisdague: yeah, was just doing that--it's merged now16:32
sdagueShrews: awesome16:32
sdaguefungi: sweet16:32
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for ceilometer test_ordination_allocate bug 1323524
uvirtbotLaunchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed]
*** thedodd has joined #openstack-infra16:33
sdaguefungi: so I think we should see if the sahara stable/icehouse stuff lands after the current nova patch16:33
sdaguethen once that resolves ~ 1hr, do the next promote set16:34
*** mmaglana has joined #openstack-infra16:34
* fungi also needs to grab lunch once done putting this list of uuids together for phschwartz16:34
sdaguefungi: sure, so an hour respit would be useful :)16:34
sdagueok, I'm going to do the lunch thing as well, will check in ~ 1:30 EST on any more promotes16:35
*** starmer has joined #openstack-infra16:35
SergeyLukjanovI'll be happy to see sahara stable/icehouse stuff landed for today's 2014.1.116:35
*** lcheng_ has joined #openstack-infra16:36
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer periodic notifier ut bug 1321826
uvirtbotLaunchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress]
*** rcarrillocruz has quit IRC16:37
openstackgerritDerek Higgins proposed a change to openstack-infra/elastic-recheck: Refine queries for tripleo jobs
anteayamorning zaro16:38
*** asselin_ has quit IRC16:38
*** starmer has quit IRC16:40
*** zzelle has joined #openstack-infra16:40
*** praneshp has joined #openstack-infra16:40
*** signed8bit is now known as signed8bit_ZZZzz16:41
fungiphschwartz: here are the uuids of the 226 instances where we saw boot with time off by 4 hours, and the times when we deleted them...
*** weshay has joined #openstack-infra16:41
*** Ryan_Lane has joined #openstack-infra16:41
phschwartzfungi: ty, I will pull up the logs.16:41
fungiphschwartz: note that we have no idea whether this is persisting now, given that we merged a workaround to devstack-gate to run ntpdate before storing start times for timeout delta calculations16:42
fungiwhich is why there are none after 20:00 utc yesterday16:42
*** rcarrillocruz has joined #openstack-infra16:42
* fungi disappears off to lunch16:42
*** asselin has joined #openstack-infra16:43
*** andreaf has joined #openstack-infra16:44
*** sballe has joined #openstack-infra16:46
*** cyli has quit IRC16:48
*** praneshp has quit IRC16:48
anteayahey clarkb16:49
clarkbfungi: ready to do nodepool whenever you are (assuming it hasn't been done yet)16:49
clarkbmordred: you are the resident python package weirdness expert. want to poke around on nodepool.o.o to see if you can figure out why the nodepool install is so funny? (it is on an older version despite having installed a current version according to pip freeze)16:50
*** harlowja_away is now known as harlowja16:50
mordredclarkb: yeah16:50
*** jaypipes has quit IRC16:50
clarkbmordred: fungi and I were just going to pip uninstall and rm -rf dist-packages/nodepool* and pip install to work around it16:50
openstackgerritDevananda van der Veen proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326312
uvirtbotLaunchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress]
*** starmer has joined #openstack-infra16:50
mordredclarkb: that's probably fine16:50
*** cyli has joined #openstack-infra16:51
*** dprince has joined #openstack-infra16:54
clarkbmordred: thanks16:54
*** starmer has quit IRC16:55
*** ihrachyshka has quit IRC16:56
*** ayoung-lunch is now known as ayoung16:56
*** habib has quit IRC16:56
clarkbfungi: sdague DinaBelova yes I think we want to disable copyallscoresifnocodechange and only copy on trivial rebase16:57
*** aconrad has joined #openstack-infra16:57
*** radez is now known as radez_g0n316:57
clarkbBobBall: fungi nodepool uses per test databases16:58
clarkbit is the first project to properly do so16:58
clarkbwhich is a bit annoying since everyone was screaming they wanted that functionality and I don't think anyone really uses it other than ourselves16:58
*** sarob has joined #openstack-infra16:58
*** jlibosva has quit IRC16:59
*** praneshp has joined #openstack-infra17:02
*** isviridov is now known as isviridov|away17:03
*** sandywalsh_ has joined #openstack-infra17:03
*** smarcet has quit IRC17:05
clarkbfungi: BobBall it looks like the nodepool deficit calculation decided that it needed to add a third node17:05
clarkbwhich could be a race between test assert and nodepools deficit loop calculation17:05
*** markwash_ has joined #openstack-infra17:05
*** smarcet has joined #openstack-infra17:06
*** derekh_ has quit IRC17:06
*** unicell has quit IRC17:07
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Add an option to mark uploaded patchsets as WIP
YorikSarzzelle, fungi: ^17:08
zzelleYorikSar, hello17:08
YorikSarzzelle, fungi: I finally got what was wrong with "git show" - Git is too old in Ubuntu.17:09
*** markwash has quit IRC17:09
*** markwash_ is now known as markwash17:09
YorikSarAnd there finally are some docs.17:09
zzelleYorikSar, git show ?17:09
clarkbif you think it is old on precise don't use it on centos617:09
YorikSarSo please take a look - probably we'll be able to land it this week.17:09
YorikSarzzelle: git show doesn't support --no-patch for that version.17:10
YorikSarzzelle: So I changed it to "git log -n1" for every commit...17:10
zzelleclarkb, in a previous life, i was developping in python 2.4 on centos5 (sniff)17:10
morganfainbergclarkb, there are per-test dbs?17:10
morganfainbergactually... i think keystone has some changes pending to use that.17:10
YorikSarclarkb: Oh... And what's Git version there?17:10
clarkbYorikSar: 1.7.117:11
YorikSarclarkb: I'm sitting on hot versions with my Gentoo, so it's very sad to see such denial in the gate17:11
sdagueclarkb: can we get that defaulted globally?17:11
YorikSarclarkb: Ok, 12.04 has 1.7.917:12
clarkbsdague: get what defaulted?17:12
YorikSarclarkb: I guess, difference is not that big :)17:12
sdaguedisable copyallscoresifnocodechange and only copy on trivial rebase17:12
clarkbsdague: yeah I think one of us will have to push a change to All-Projects directly according to fungis report on it not being guiable17:13
YorikSarsdague: By the way, about gate... May b it would make sense to throw changes that are too old off "gate" queue to the "check" one and then take them back if they pass?17:14
clarkbYorikSar: we do that17:15
YorikSarsdague: So those changes that are there for 30h - it's 30h in Zuul not in the "gate" queue in particular?17:15
clarkbfungi: BobBall that may be related to the soft locking around listing nodes17:15
clarkbcould have introduced a race in the test17:15
YorikSarclarkb: ^17:16
clarkbYorikSar: no its 30 hours in the gate queue17:17
clarkbbut before they got into the gate queue if they had tests more than 72 hours old they had to be retested in check first17:17
*** MarkAtwood has joined #openstack-infra17:17
clarkbsince the gate is self checking it is in theory not necessary to bump those out17:18
YorikSarclarkb: Oh, really.17:18
*** reed has joined #openstack-infra17:18
clarkbyes its an ordered list nothing jumps the queue (except for when we manually do that to get fixes to the head of the queue)17:18
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for cinder out-of-date conf.sample bug 1326890
uvirtbotLaunchpad bug 1326890 in cinder "cinder.conf.sample is not up to date" [Undecided,New]
YorikSarclarkb: Thanks for setting my mind straight :)17:19
*** hashar has joined #openstack-infra17:19
*** starmer has joined #openstack-infra17:20
clarkbfungi: BobBall: ya I think this is a relationship between watermark_sleep in the test and the sleep() that is performed to wait for nodes17:21
*** p5ntangle has joined #openstack-infra17:21
*** hashar has quit IRC17:22
clarkb is where the race happens I think. Basically node 2 comes on line right after nodepool decides it has a deficit of 117:23
clarkbI am going to try bumping watermark sleep to 1, which doubles the amount of time nodepool has to mark that node ready before it checks again17:24
*** starmer has quit IRC17:25
*** sballe has quit IRC17:25
*** BadCub01 has quit IRC17:26
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Increase watermark sleep in tests for reliability.
clarkbthats a bit of a hack and we should get away from sleeping if possible but meh17:26
*** gokrokve has joined #openstack-infra17:27
apevecttx, adam_g - so given gate situation, is pushing 2014.1.1 today realistic - or do tag pushing jobs go to a separate queue from gate?17:29
apevecwe also have approved Neutron exceptions 96919 and 96920 still queued in gate17:29
clarkbapevec: the tag pushing jobs are separate17:29
clarkbbut if you need to merge things before tagging that is in the gate17:29
apevecyeah, we have those two, I'll ask again on stable-maint if we should wait for them17:30
*** marcoemorais has quit IRC17:32
clarkbapevec: we can promote them if they are critical though we try to reserve that for gate fixes17:32
*** marcoemorais has joined #openstack-infra17:32
clarkb(promotion bumps them to the head of the queue)17:32
*** marcoemorais has quit IRC17:33
*** Ryan_Lane1 has joined #openstack-infra17:33
*** marcoemorais has joined #openstack-infra17:33
apevecagreement was they're critical enough as freeze exception, but not sure if17:33
openstackgerritDevananda van der Veen proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326312
uvirtbotLaunchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress]
*** Ryan_Lane has quit IRC17:34
apevecpromotion is justified17:34
*** sarob has quit IRC17:34
apevecI guess, only if there are not current gate fixes in the gate?17:34
SlickNikHey guys17:35
clarkbapevec: ya and we can always put the gate fixes at the very front :) fwiw I am comfortable promoting those changes if you ened them but would be good to get fungi and sdague's input too17:35
clarkb(and SergeyLukjanov + jhesketh feedback if they are awake)17:35
*** starmer has joined #openstack-infra17:36
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for setuptools py26 unicode bug 1326878
uvirtbotLaunchpad bug 1326878 in pbr "Race "TypeError: expecting a bytes object, got unicode" on py26 setup" [Undecided,New]
clarkbI am going to reboot for local updates17:36
SlickNikI'm seeing devstack installs failing to install keystone; during create_keystone_accounts it bails out with  ERROR: Exception raised: six>=1.6.017:36
clarkbback in a bit (assuming that new kernel works)17:36
SlickNikIs this a known issue?17:36
apevecok, let's wait for +1 from more infra folks17:36
devanandajogo: won't prevent E-R from being useful for ironic?17:37
clarkbSlickNik: I am not aware of it but still trying to catch up on scrollback17:37
devanandajogo: "Non-voting jobs are non-voting because the are unstable" -- except for when they're not. Like tempest-dsvm-virtual-ironic17:37
clarkbdevananda: aren't your jobs voting?17:37
clarkbor will be voting17:37
devanandaclarkb: not on nova. they can't be. remember?17:37
devananda"will be"17:37
clarkbdevananda: no the whole point of jhesketh's change is to make them voting..17:37
SlickNikclarkb: Okay, thanks! Will try and dig in some more.17:38
*** dosaboy has quit IRC17:38
devanandaclarkb: gotcha. so that change to E-R won't exclude voting-but-not-gating-because-its-third-party jobs17:39
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for glance _authorized_upload_ bug 1326899
uvirtbotLaunchpad bug 1326899 in glance "FAIL: glance.tests.unit.v1.test_api.TestGlanceAPI.test_add_copy_from_image_authorized_upload_image_authorized" [Undecided,New]
*** e0ne has joined #openstack-infra17:39
*** Sukhdev has joined #openstack-infra17:39
*** starmer has quit IRC17:40
openstackgerritFathi Boudra proposed a change to openstack-infra/jenkins-job-builder: ssh publisher: add always-publish-from-master support
clarkbdevananda: correct17:40
clarkbI wouldn't expect it to at least17:40
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for cinder out-of-date conf.sample bug 1326890
uvirtbotLaunchpad bug 1326890 in cinder "cinder.conf.sample is not up to date" [Undecided,New]
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: tracking bug for connection issues between HP and RAX
clarkbreboot complete. xfce has decided I need a giant mouse pointer17:41
*** dosaboy has joined #openstack-infra17:41
clarkbwe are trying to make those jobs voting for a variety of reasons and if this isn't one of them then we should fix e-r :)17:42
*** srenatus has quit IRC17:42
*** thedodd has quit IRC17:42
*** ramashri has quit IRC17:42
*** ildikov has quit IRC17:42
*** srenatus has joined #openstack-infra17:43
fungiclarkb: apevec: i think not delaying stable point releases is probably worth the relatively minor hit from promoting the changes needed for them17:43
fungiYorikSar: we briefly discussed git review -W in the infra meeting this week and want to make sure we don't make it too openstack-configuration-specific... i'm starting to wonder if we shouldn't instead make the behavior of git review -D be configurable in .gitreview files, defaulting to draft behavior but overridable to set label votes (and in projects for we just patch all ~30017:43
fungi.gitreview files to set workflow -1 for -D)17:43
*** Ajaeger has joined #openstack-infra17:43
fungiYorikSar: i wouldn't expect that to change the mechanics of the patch too substantially, but we probably ought to go ahead and tag a 1.24 and then we can do 1.25 soon after once the work in progress vs draft idea is sorted17:44
clarkbfungi: ++17:44
*** SlickNik has quit IRC17:45
*** spiffxp_ has quit IRC17:45
fungiclarkb: i like when someone just generally ++'s me. i assume that's transitive to all my past (and future) ideas17:45
*** masayukig has quit IRC17:46
clarkbseems reasonable17:46
* fungi is nothing if not reasonable17:46
*** SlickNik has joined #openstack-infra17:46
clarkbfungi: so I think nodepool's test issues are just test issues17:46
*** basha has joined #openstack-infra17:46
clarkbfungi: if the patch I pushed comes back clean I am happy to continue restarting nodepool17:46
*** pelix has quit IRC17:47
fungiclarkb: i am willing to accept that. we can stop it, blow away the pip install, manually re-install it from tip in /opt and do the other cleanup, then start it again17:47
clarkbbut we seem to be hurting for nodes again...17:47
* fungi reviews said patch17:47
*** spiffxp has joined #openstack-infra17:48
apevecclarkb, fungi - ok, so we have +1 from you both - one more from sdague for promoting 96919 and 96920 (latter depends on the former)17:48
apevecmestery, ^ your patches17:48
fungibtw, we need to get the non-profit question settled for the foundation and then get it added to so that i can appreciate the irony of amazon donating to a project which is targeted at replacing their cloud services business17:48
clarkband we are ~24 minutes away from a gate reset17:48
apevecmestery, be ready for sdague's questions :)17:49
clarkbfungi: :)17:49
clarkbsdague: question from me, does that nova backport to icehouse need to be promoted to get the neutron patches to merge?17:49
SlickNikfungi: that's awesome :)17:50
*** ramashri has joined #openstack-infra17:50
*** starmer has joined #openstack-infra17:50
apevecclarkb, which nova backport?17:50
*** ctracey has quit IRC17:51
*** masayukig has joined #openstack-infra17:51
clarkbapevec: let me find it in scrollback17:51
*** ctracey has joined #openstack-infra17:51
*** markmcclain has joined #openstack-infra17:52
clarkbapevec: 9784517:52
apevec97845 was merged today17:52
clarkboh hey it merged17:52
*** basha has quit IRC17:53
*** _nadya_ has quit IRC17:53
*** Sam-I-Am has quit IRC17:53
*** p5ntangle has quit IRC17:54
*** starmer has quit IRC17:55
*** marun has quit IRC17:56
*** marun has joined #openstack-infra17:56
* fungi is catching up on lunchtime scrollback a bit...17:57
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for ceilometer test_ordination_presence bug 1326820
uvirtbotLaunchpad bug 1326820 in ceilometer "FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence (dup-of: 1323524)" [Undecided,New]
uvirtbotLaunchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed]
fungiSlickNik: dmakogon_ mentioned that six-related failure a few hours ago. looks like a problem in openstackclient?17:58
fungii think maybe it's only impacting trove, as i don't think anybody else is seriously using openstackclient in the gate yet18:00
*** ramashri has quit IRC18:00
SlickNikfungi: Yes. Seems like it.18:00
*** arnaud__ has joined #openstack-infra18:00
openstackgerritDevananda van der Veen proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326312
uvirtbotLaunchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress]
*** ramashri has joined #openstack-infra18:01
clarkbso we have had similar problems when people pin six18:01
clarkbis openstackclient or trove pinning six?18:01
SlickNikfungi: Looks like devstack's using it in HEAD.18:01
clarkbSlickNik: do you have a link to the failure18:01
fungiright on. maybe that means it'll be production grade and web scale in no time ;)18:01
*** Sam-I-Am has joined #openstack-infra18:01
fungithat was dmakogon_'s example anyway18:02
*** signed8bit_ZZZzz is now known as signed8bit18:02
SlickNikfungi: yes, thanks for pulling that out.18:02
*** signed8bit is now known as signed8bit_ZZZzz18:02
* fungi uses openstackclient for his own personal accounts/servers, but has workarounds to deal with things like rackspace's glance endpoint not being correct in their keystone service catalog18:03
clarkbopenstackclient requires >= 1.6.018:03
fungi...and the lack of account password protection18:03
clarkbI am guessing something else pins it18:03
*** amcrn has joined #openstack-infra18:03
clarkbdo you have this failure somewhere with a pip freeze capture?18:03
clarkbhard to debug without logs18:03
*** denis_makogon has joined #openstack-infra18:04
fungii should push up my "really ugly shell script" which will decrypt your account credentials for you on the fly from a blowfish blob using a separate passphrase prompted for at invocation time18:04
SlickNikclarkb: working on that.18:05
*** timrc-afk is now known as timrc18:05
*** Sukhdev has quit IRC18:05
*** starmer has joined #openstack-infra18:05
clarkbfungi: nice18:05
fungiit was a reaction to the incomplete/broken keychain implementation in openstackclient18:06
clarkbfungi: bam +118:06
fungii spent more time trying to get the keychain integration working than i did just putting a wrapper in place to do basically the same thing18:06
fungi2m 21s18:07
clarkbfor personal things I just use the web ui18:07
clarkbI only have two servers18:07
clarkbclearly I am using the cloud wrong18:07
fungiwe ought to recheck this several times... we saw it happening at random18:07
clarkbfungi: can do18:07
*** vhoward has left #openstack-infra18:08
*** nati_ueno has joined #openstack-infra18:08
fungivery rough estimate was about 50% occurrence18:08
*** signed8bit_ZZZzz is now known as signed8bit18:09
fungi97835 is the current example18:09
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: removed job configuration duplication in docs
clarkbah yup ~ 50%18:09
*** starmer has quit IRC18:10
fungigiven that change only touches documentation, it made for an obvious canary18:10
*** signed8bit has quit IRC18:11
fungii was working to get unit tests running on my dev box, but they seem to want a working mysql configuration so i stopped18:11
sdaguefungi: so I think two promotes should go in18:11
*** Sukhdev has joined #openstack-infra18:12
clarkbsdague: did you see the questions about promoting icehouse neutron changes for release?18:12
fungisdague: 97842,2 and 97757,1 in that order?18:12
clarkbfungi: yes, 97757 only affects ironic so should go behind the other change18:12
clarkbfungi: and the icehouse changes can go behind those t218:12
fungisdague: and yes, i think the stable branch release managers would appreciate if i promoted remaining bits they need for today's tagging along with these18:13
fungii can add them to the etherpad as a note18:13
sdagueclarkb: I didn't18:14
sdagueat this point if they are not in that etherpad, I don't believe in their existance :)18:14
clarkbsdague: see comments from apevec in recent sb18:14
clarkbsdague: well they aren't gate issues18:14
clarkbso separate18:14
fungisdague: in the etherpad now under "possible"18:15
sdagueyeh, if they aren't gate issues I was ignoring them18:15
*** reed has quit IRC18:15
*** reed has joined #openstack-infra18:16
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: removed job configuration duplication in docs
sdagueare they breaking stable/icehouse gate?18:16
sdagueor nice to have18:16
fungialready part of the drafted release notes i think18:16
sdaguebecause I sort of feel mixed jumping the queue for a release18:16
clarkbsdague: I think they need to merge before release can happen18:16
clarkbsdague: right that is why we are asking you :)18:16
fungitechnically, 96919 is security-related18:17
*** dkliban is now known as dkliban_bbiab18:18
sdagueso consider me a soft -1 on philosophical grounds but feel free to override me on that18:18
apevecI've listed options on stable-maint
apevecno replies from stable-maint audience yet18:19
*** maxbit_ has joined #openstack-infra18:19
apevecmestery, ^ what do you say?18:19
*** signed8bit has joined #openstack-infra18:19
*** starmer has joined #openstack-infra18:20
*** sarob has joined #openstack-infra18:21
fungiwith vmt hat on i'm inclined to push 96919 up to the front anyway18:22
sdaguefungi: I think that's fine18:22
sdagueso, honestly, if your are promoting, just go for it.18:22
*** maxbit has quit IRC18:23
fungione of the two remaining patches for neutron 2014.1.1 addresses a possible security hole, so yeah bringing the other along for the ride is fairly non-impacting18:23
*** markmcclain has quit IRC18:24
fungiso i'm proposing we promote (in sequence) 97842,2 97757,1 96919,1 96920,118:24
*** starmer has quit IRC18:25
clarkbsounds good18:25
SergeyLukjanovfungi, all sahara changes failed in gate too18:25
SergeyLukjanovbecause of neutron18:25
SergeyLukjanovjob failed on the first one18:25
SergeyLukjanovcould we include them too?18:26
fungiSergeyLukjanov: for 2014.1.1?18:26
SergeyLukjanovfungi, yup18:26
*** maxbit has joined #openstack-infra18:26
openstackgerritMarc Abramowitz proposed a change to stackforge/python-jenkins: Fix remaining PEP8 errors in docstrings
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Configure Timeline from Story details
*** sarob has quit IRC18:27
fungiSergeyLukjanov: that's a tougher call. sahara wasn't integrated in icehouse so it technically doesn't have to release point releases in tandem with the integrated projects from icehouse18:28
*** maxbit_ has quit IRC18:28
apevecSergeyLukjanov, yeah, your not about by stable-maint schedule18:29
*** andreykurilin_ has joined #openstack-infra18:29
apevecfor stable/icehouse, you'll be when integrated in stable/juno18:29
SergeyLukjanovfungi, I'm ok if you'd like to not doing it18:29
e0nehi all! i'm tring to setup local CI environment using manifests are hard-coded to use HTTPS for jenkins, but it isn't necessary for all cases. will be it useful for anybody from a community? i'm ready to implement and propose fix for it18:29
SergeyLukjanovapevec, yup, but we're following the process since the icehouse cycle18:29
fungiand since sahara's icehouse branch isn't covered by the vmt, i can't really advocate for an exception for any security fixes there either18:29
*** sarob_ has joined #openstack-infra18:29
apevecthat was bad Enlish. s/your not about/you're not bound/18:29
*** andreaf has quit IRC18:30
apevecSergeyLukjanov, that's great for training your team so you're ready for stable/juno !18:30
SergeyLukjanovfungi, apevec, okay, let's keep sahara stable changes as is18:30
clarkbe0ne: is there a problem with using https?18:30
clarkbe0ne: I would argue it should just be used (our implementation uses snake oil certs so not the greatest but better than nothing)18:30
fungiSergeyLukjanov: thanks for understanding! i know it's tough with gate delays impacting release timelines18:30
*** sarob_ has quit IRC18:31
e0neclarkb: no, but i don't need to use ssl in my local private infra. it's an overhead for me18:31
SergeyLukjanovfungi, np, it's not so critical for now18:31
clarkbe0ne: but it isn't overhead...18:32
*** dizquierdo has quit IRC18:32
clarkbit should just work18:32
*** otherwiseguy has quit IRC18:32
fungithe 1990s called and they want their extra couple of ssl round-trips worth of bandwidth consumption back18:32
*** sarob_ has joined #openstack-infra18:32
fungi(okay, that joke kinda fell flat)18:32
e0neok, maybe something went wrong and it doesn't works out-of-box for me18:33
fungibut yes, if ssl/tls processing overhead is straining your jenkins master or clients, you're in the wrong decade18:33
SergeyLukjanovclarkb, "(and SergeyLukjanov + jhesketh feedback if they are awake)" sorry, but I'm not sure that I understand the target for feedback :)18:33
clarkbe0ne: you may need to specify the snakeoil certs as we do (but I thought they were by default)18:33
e0neclarkb, thanks!18:34
jerryzclarkb, fungi: i met with the same problem pabelanger had a couple of months ago.
*** Alexei_987 has joined #openstack-infra18:35
jerryzturned out that system install of libzmq overrides pip installed pyzmq18:36
jerryzbut it was because i followed the instruction in readme of nodepool18:36
e0neand what about hard-coded git_repos and hostnames? e.g. i want ti use custom jenkins-job-builder repo and/or revision. - a very very draft example of changes. shuold i create bug/blueprint for it?18:36
openstackgerritA change was merged to openstack-infra/elastic-recheck: Added query for cinder out-of-date conf.sample bug 1326890
uvirtbotLaunchpad bug 1326890 in cinder "cinder.conf.sample is not up to date" [Undecided,New]
mesteryapevec: Sorry, was afk, here now, what can I help with regarding my patches for stable?18:37
fungijerryz: would BobBall's documentation patch at have helped you avoid that?18:37
e0nepuppetmaster_host is also should be changed for a local env. e.g.
e0nemy changes is a very draft for now18:37
fungie0ne: i think asselin may have some changes proposed already to make git repository urls and default branches configurable18:38
apevecmestery, justify their importance, but we have +1 now18:38
jerryzfungi: yeah, it would. but i would also be great to update readme18:38
mesteryapevec: OK, thanks! Much appreciated for the help here!18:38
fungie0ne: but i don't know if you need a bug for it... you're welcome to just propose a patch into our gerrit without filing a bug too18:38
apevecSergeyLukjanov, btw at least one sahara patch didn't fail in neutron but in devstack-gate setup?
fungijerryz: do you mind proposing a patch to the readme?18:39
sdagueapevec: that was yesterday18:39
apevecah yes, didn't look at timestamp18:40
*** nati_ueno has quit IRC18:40
SergeyLukjanovapevec, yup, yesterday's one18:40
jerryzfungi: remove libzmq-dev from the installation of dependency part or have the same explanation as in BobBall's patch18:40
apevecjust for my education where do you look further for such devstack-gate-setup failure?18:40
fungimestery: apevec: i've got the remaining neutron icehouse patches ready to push to the front along with a couple gate fixes as soon as we see another tip reset18:40
*** om has quit IRC18:40
jerryzfungi: i think not installing libzmq-dev would do18:41
apevecSergeyLukjanov, nothing in devstack-gate-setup-workspace-new.txt.gz afaict?18:41
SlickNikclarkb / fungi: So I looked at the pip freeze output and it's surprising but "six==1.6.1"18:41
e0nefungi: i saw only from asselin. it doesn't cover all cases18:41
mesteryfungi: Thanks!18:41
clarkbSlickNik: huh18:41
fungie0ne: sounds like additional patches may be called for in that case18:41
rkukuraAnyone have any experience (good, bad, impossible) with gerrit patches that depend on more than one other patch?18:42
clarkbpleia2: fungi mtreinish just booked trip. staying at maritim konferenz hotel in darmstadt18:42
fungirkukura: tons18:42
e0nefungi, you're right, thanks!18:42
clarkbrkukura: it is not possible to have multiple parents without a merge commit18:42
clarkbrkukura: you can serialize them though18:42
rkukurafungi: Good, bad?18:42
*** gyee has quit IRC18:42
fungiclarkb: i was leaning toward the intercityhotel since it looked cheaper, but will go with the flow ;)18:43
clarkbfungi: well maritim is HP preferred18:43
clarkband I could book without a deposit :)18:43
rkukuraclarkb: We’ve basically got a matrix of patches, split into layers (API, DB, plugin, …) and columns (sets of resources)18:43
fungiclarkb: given the number of hp'ers coming, that's a good reason18:43
rkukuraclarkb: So a typical patch will depend on the layer below it in the same column, and the same layer of the previous column18:44
*** om has joined #openstack-infra18:44
fungirkukura: well, good experience mostly, but the dependent patches need to be for the same git repository and branch, and need to be sequenced18:44
jogodevananda: only if ironic jobs have " (non-voting)" next to them in the jenkins comment. and if they do we can make a special exception18:45
pleia2clarkb: cool, thanks18:45
clarkbjogo: we won't have to make an exception shortly we are going to make them voting18:45
clarkbpleia2: and the lufthansa/united flight appears to be lufthansa operated so I took a chance on that18:45
jogoclarkb: even better18:45
pleia2clarkb: hooray :)18:46
rkukurafungi: Same git repo definitely18:46
clarkbkeep in mind you lose a day flying that direction so will have to leave saturday18:46
clarkb(or I had to anyways)18:46
pleia2clarkb: direct to frankfurt?18:46
pleia2I've done that trip a few times, but never left the airport, so exciting!18:47
fungiclarkb: pleia2: arriving in frankfurt via united at noon local on sunday18:47
rkukurafungi, clarkb: If I have a merge commit as part of my patch, when I do “git review”, will gerrit treat that merge commit as a separate dependency?18:47
clarkbrkukura: yes the merge commit will have its own change. keep in mind we don't allow merge commits upstream18:48
fungirkukura: gerrit should treat merge commits just like any other commit, but your account would need to have the necessary access permissions allowing it to push a merge commit to refs/for18:48
*** smarcet has quit IRC18:48
rkukuraclarkb, fungi: Is it reasonable to push initial patch sets with the merge commit for review, and rebase after the dependency has been merged?18:49
* fungi has to fly norfolk->washington dc->frankfurt... bleagh18:49
clarkbfungi: our hp and rax error rates look really similar18:50
clarkbkind of scary18:50
*** nati_ueno has joined #openstack-infra18:50
clarkbrkukura: I think you can just push them serially18:50
clarkbrkukura: I think merge commits and rebases just complicate18:50
fungirkukura: i don't push merge commits for review. i use git rebase to order all the commits and then push that series18:50
pleia2fungi: heh, I have austin > dc > munich > zagreb next weekend18:50
*** thedodd has joined #openstack-infra18:51
clarkbfungi: I arrive ~9am frankfurt local time on sunday18:51
rkukuraclarkb: I was concerned about how to rebase on an update of the branch that I’ve merged18:51
clarkbfungi: depending on how tired I am and light I pack it may be tempting to spend that day in frankfurt before heading to darmstadt18:52
fungirkukura: if you're developing for code review in gerrit, generally you want to keep your topic branch rebased on the tip of the remote branch it's targeting18:52
rkukuraclarkb, fungi: So are you suggesting that I stick with a single chain of dependencies, where the API layer of one column depends on the top (plugin) layer of the previous column, even though it really only depends on the API layer of that column?18:52
*** e0ne has quit IRC18:53
devanandajogo: one does now, pending the change from jhesketh to enable a separate pipeline for it18:53
clarkbrkukura: you can have them branch but a serial list is simple18:53
clarkbrkukura: you just can't have more than one parent18:53
fungirkukura: yes, basically18:53
*** ildikov has joined #openstack-infra18:53
*** e0ne has joined #openstack-infra18:53
rkukuraclarkb, fungi: OK, I’m convinced!18:54
fungigerrit deals well with changes with more than one child commit, but merges (a change with multiple parent commits) are not handled well... the code review interface is fairly messy18:54
fungiand also needs special permission to push those anyway18:54
jogodevananda: when that lands ping me, we can add a non-voting whitelist to elastic-recheck18:55
rkukurafungi, clarkb: Thanks!18:55
*** andreykurilin_ has quit IRC18:56
*** andreykurilin_ has joined #openstack-infra18:56
sdaguefungi: did you do the promote?18:57
fungisdague: was waiting for a nasty gate reset18:58
sdagueah ok18:58
fungithere are a few changes in there which are ~10 minutes from clearing out of the gate18:58
*** e0ne has quit IRC18:58
*** sarob_ has quit IRC18:58
fungiso i don't want to reset their timers unnecessarily18:58
devanandajogo: we should add the non-voting whitelist for [check,gate]-tempest-dsvm-ironic-nv now, and remove it once we're reporting as a third-party CI to nova18:59
*** arnaud__ has quit IRC19:00
jogodevananda: yes, although that won't happen until the gate issues are sorted out19:00
devanandajogo: to recap: there's an ironic*nv job now that we'd like to have included in recheck. several patches are in flight, and some more to be written, which will make it voting (as a third party) sometime this cycle19:00
devanandajraim: absolutely :) gate block comes first19:00
devanandajogo: ^19:00
*** sarob has joined #openstack-infra19:01
clarkbok I am going to go find food19:01
*** smarcet has joined #openstack-infra19:02
*** ramashri_ has joined #openstack-infra19:03
*** ramashri has quit IRC19:03
*** adalbas has quit IRC19:04
jogosdague: wondering if you think that would help at all or not19:04
openstackgerritJerry Zhao proposed a change to openstack-infra/nodepool: Remove libzmq-dev from dependency list
anteayaSergeyLukjanov: if you are still around, did you find anytime to answer any new third party ci requests?19:05
sdaguejogo: shouldn't rwe already be bailing there?19:05
sdague|| /bin/true19:05
* fungi has a bucket in hand for bailing19:05
*** starmer has joined #openstack-infra19:05
openstackgerritA change was merged to openstack-infra/elastic-recheck: tracking bug for connection issues between HP and RAX
sdaguefungi: or a tractor, if they are made of hay19:06
*** ramashri has joined #openstack-infra19:06
fungibailing wire. that's always had a great connotation19:06
*** nati_ueno has quit IRC19:07
sdaguefungi: also, I'm going to backport the get-pip fixes to icehouse and havana19:07
sdagueor more importantly, I did, and pushed those19:07
fungiokay, awesome19:08
*** marcoemorais1 has joined #openstack-infra19:08
sdaguethose might be sensible promotes19:08
fungialready approved?19:08
*** james_li has joined #openstack-infra19:08
*** ramashri_ has quit IRC19:08
fungiif so, i can tack them on19:08
*** marcoemorais has quit IRC19:08
sdaguedtroyer's not back yet. If you want to review them, I'll self approve if there are a couple of sane +1s on them19:09
*** jaypipes has joined #openstack-infra19:09
sdagueI realized I put this one under it -, let me restack that19:10
jogosdague: I think we used to continue since we had 3rd part apt resources19:10
*** starmer has quit IRC19:10
sdaguejogo: no, I'm saying, we do now as well19:10
sdaguefungi: ok, I restacked so the icehouse get-pip doesn't rely on anything19:11
*** sarob has quit IRC19:11
james_liHi clarkb , fungi we (Solum team) wants to enable docker on our gate, we are aware that F20 supports docker fine. We also like to know when Trusty will be an option for gating?19:11
jogosdague: ahh but do you think a retry would help?19:11
*** sarob has joined #openstack-infra19:11
sdaguejogo: why?19:11
jogosdague: I was thinking the apt mirror failures could be very transient19:12
jogoand a retry may just fix it. (but this is just a shot in the dark)19:12
sdaguesure, but why is retrying the package index grab going to help?19:12
sdaguebecause it's not fatal19:12
dtroyersdague: fwiw, I'm kinda back now…just getting started looking at backlog stuff.  anything specific I should start with?19:13
jogosdague: ahh, because apt-get install fails in those cases19:13
jogosdague: so the failure happens in the next step19:13
sdaguedtroyer: well the gate is kind of backed up, but the 3 patches above would be good19:13
sdaguejogo: log link?19:14
devanandasdague: think its worth adding one unit test for each public method of nova.virt.driver.virt_driver:ComputeDriver that we're using in ironic.nova.virt.ironic.driver.IronicDriver?19:14
anteayahey dtroyer, welcome back19:14
sdaguedevananda: sounds fair to me19:14
fungisdague: 98209,2 is an icehouse backport of a gate stability fix as well?19:15
devanandagot the path wrong (nova.virt.driver:ComputeDriver)19:15
devanandasdague: ack, tanks19:15
sdaguefungi: no, it's a make grenade work better for me backport19:15
sdaguewhich is why it's on top19:15
*** stevebaker has quit IRC19:15
*** stevebaker has joined #openstack-infra19:15
fungiright, got it19:15
sdagueif we consider the bottom to be closest to master19:15
sdagueit can land the normal way19:16
*** sarob has quit IRC19:16
*** homeless has quit IRC19:17
zzellehi everyone19:17
*** W00die has quit IRC19:17
*** gargola has quit IRC19:17
*** gargola has joined #openstack-infra19:17
*** w00dy has joined #openstack-infra19:17
zzelleis it normal that no jobs are run in Zuul post pipeline ?19:17
*** w00dy is now known as Guest4380119:18
jogosdague: is it safe to clse no e-r hits?19:18
uvirtbotLaunchpad bug 1298929 in openstack-ci "certificate errors downloading pip or installing pip packages" [Undecided,New]19:18
*** homeless has joined #openstack-infra19:18
sdaguejogo: yeh, I think so19:19
SergeyLukjanovanteaya, I can do it now I think19:19
anteayaSergeyLukjanov: awesome, thank you19:20
SergeyLukjanovanteaya, do we just asking them to wait for some more time?19:20
anteayaare we asking who to wait?19:20
SergeyLukjanov(I'm afraid I could skip something about it in logs)19:20
*** tkelsey has joined #openstack-infra19:20
SergeyLukjanovanteaya, 4rd party testing acc requesters19:20
mtreinishclarkb: cool19:20
SergeyLukjanovheh, 3rd*19:20
*** maxbit_ has joined #openstack-infra19:21
sdagueI want 5th party testing19:21
SergeyLukjanovsdague, it could help19:21
*** sarob has joined #openstack-infra19:21
jogosdague: log link
anteayaSergeyLukjanov: no, as long as you and I and fungi are happy with the naming scheme I outlined yesterday, we have no cause to wait for account creation19:21
anteayasdague: buckaroo bonzai and the 8th dimension19:21
anteayaI want testing in 8 dimensions19:22
SergeyLukjanovanteaya, okay, I'll check request and return back with names19:22
anteayaSergeyLukjanov: thanks19:22
anteayayou can add your suggestions to the bottom of that etherpad if you like19:22
fungizzelle: the post pipeline is backed up about an hour because of an inrush in the check pipeline. once zuul finishes dealing with check changes older than the post commits, it will assign workers to those jobs19:22
anteayaI wish I could get that pretty colour of pink back, but I can't seem to find it19:23
sdaguejogo: we have a deb expert around?19:23
jogofungi: ^ ?19:23
sdaguebecause I don't feel like doing the update would help there19:23
fungisdague: depends on what kind of expertise you're expecting19:23
SergeyLukjanovanteaya, heh, the first requester I see ask for "jenkins-cicd" account ;)19:23
anteayayeah, so that's no19:24
sdaguejogo: I think this is the issue -
sdaguewhich triggers the error19:24
anteayaSergeyLukjanov: did they give a company name?19:24
zzellefungi,fine, just checking19:24
sdaguewe could always --force-yes19:24
sdagueand see what happens19:24
*** maxbit has quit IRC19:24
SergeyLukjanovanteaya, nope, /me writing an email19:24
anteayahopefully you can also get them to tell you what their thing does19:25
anteayathat might help with the naming too19:25
fungizzelle: yeah, there's a heat patch series 24 changes deep starting at which absorbed all our quota a little over an hour ago19:25
*** andreykurilin_ has quit IRC19:25
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Remove some queries with no hits
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added field restrictions and error messages to project forms
jogosdague: right, so the question is will a working pass at apt-get update help with that or not.19:26
*** gokrokve_ has joined #openstack-infra19:26
fungijogo: sdague: so that looks like a broken mirror19:27
zzellefungi, fine19:27
*** otherwiseguy has joined #openstack-infra19:27
sdaguefungi: which means nothing will help, right?19:28
jogofungi: well the apt-get update  command fails half way through19:28
sdaguejogo: no, it fails all the way through19:28
sdagueI think the error messages a just confusing in apt19:29
sdaguebut I don't think it did anything successfully19:29
*** e0ne has joined #openstack-infra19:29
*** _nadya_ has joined #openstack-infra19:30
fungisdague: jogo: the last thing it did successfully was get precise-security Release.gpg19:30
*** gokrokve has quit IRC19:30
fungithen it ignores a bunch of stuff which is already new enough to stay cached19:30
fungithen fails to get precise/main Sources19:30
fungiand everything it wants after that also fails19:30
SergeyLukjanovanteaya, hp-cinder-ci vs hp-storage-ci?19:31
sdagueisn't there like an apt bt client19:31
*** tkelsey has quit IRC19:31
fungiso the issue there is apt-get update didn't really work and ran into connectivity issues partway through, then we proceeded to try and apt-get upgrade anyway which predictably fails due to signature/checksum mismatches19:32
anteayaSergeyLukjanov: they fit the format, as long as hp is only going to have one 3rd party ci account testing cinder19:32
jogosdague: that would be pretty neat19:33
openstackgerritDan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstack_extras
jogofungi: so perhaps a simple retry may help? but I wonder if using bt would be better19:33
mtreinishsdague, jogo:
anteayaSergeyLukjanov: we can create that as requested and if hp asks for a second account to test cinder, they have to name it something else19:33
* jogo looks at the clock and realizes he hasn't eaten today and wonders off to lunch19:33
sdaguemtreinish: yeh, it looks pretty unfinished19:33
SergeyLukjanovanteaya, they are asking for hp-cinder-ci "HP Storage CI", I'll ask them to confirm one of the names19:34
fungijogo: well, i _suspect_ that retrying apt-get update there probably wouldn't have helped. there was probably a somewhat prolonged period of time where that slave was failing to reach the mirror (the mirror might have even been offline)19:34
sdaguefungi: would it make sense to put multiple mirrors in the definitions?19:35
anteayaSergeyLukjanov: kk, I wouldn't do HP Storage CI since that team probably doens't know about swift or trove or any of the other storage services19:35
sdagueor will that just lead to more explody19:35
*** starmer has joined #openstack-infra19:35
fungisdague: it would need some testing. i believe the issue which would arise is if the mirror which was unreachable was the one the slave thought had the most recent version of some package19:36
anteayaSergeyLukjanov: what happens if we give them hp-cinder-ci HP Cinder CI and tell them that is what it is?19:36
anteayathe username has to match the Full Name anyway19:36
sdaguefungi: it looks like we just stared over again19:36
sdagueif you were looking for a rebase window19:36
fungisdague: yup19:36
fungipromoting now19:36
SergeyLukjanovanteaya, I think we should do it19:37
fungidid those two devstack backports get a thumbs-up from dtroyer?19:37
fungilooks like it19:37
*** afazekas is now known as _afazekas_slp19:37
fungienqueuing those and then promoting all together19:37
dtroyerfungi: the ones?  yes19:37
fungimoved 97842,2 97757,1 98210,2 96919,1 96920,1 98211,1 to the front19:38
*** gokrokve_ has quit IRC19:38
anteayaSergeyLukjanov: let'd do it then19:38
sdaguefungi: thanks19:39
*** gokrokve_ has joined #openstack-infra19:40
*** starmer has quit IRC19:40
dhellmannsdague: I don't understand this requirements check failure error message, and would appreciate your interpretation if you have a minute:
sdaguedhellmann: so this was a thing that someone was looking at yesterday19:41
sdaguethat enforcement is supposed to only happen on openstack/requirements19:41
Ajaegerdhellmann: I think that was reverted yesterday19:41
sdaguenot on the projects19:41
dhellmannah, ok, so maybe a recheck will let it pass?19:41
sdagueso we need the script to change19:41
sdagueAjaeger: I don't think it did19:41
sdaguebecause the proposal was to delete everything19:41
dhellmannor, oh, I need to wait for a job change19:42
*** om has quit IRC19:42
sdaguefungi: who was looking at that?19:42
* Ajaeger tries to figure out what he saw flying by ;)19:42
*** krotscheck has left #openstack-infra19:43
*** maxbit has joined #openstack-infra19:44
Ajaegerdhellmann: - not reverted yet19:45
*** arnaud__ has joined #openstack-infra19:45
*** gyee has joined #openstack-infra19:46
*** maxbit_ has quit IRC19:47
*** krotscheck has joined #openstack-infra19:50
* anteaya clicks19:50
SergeyLukjanovanteaya, what we think about personal name CI?19:50
SergeyLukjanovanteaya, IMO it's bad19:50
*** starmer has joined #openstack-infra19:50
*** nati_ueno has joined #openstack-infra19:51
anteayaokay great, email the person who requested the account and ask for additional details19:51
*** maxbit has quit IRC19:52
SergeyLukjanovanteaya, acl19:52
*** maxbit has joined #openstack-infra19:52
*** apevec has quit IRC19:52
fungisdague: i replied to your -1 saying that the "fix" isn't just a matter of changing that script. the change was made to the wrong project19:53
SergeyLukjanovfungi, anteaya, re 3rd party accs: one account created (hp-cinder-ci) and three name clarification requests sent19:53
*** ramashri has quit IRC19:53
dhellmannAjaeger: thanks19:53
SergeyLukjanovfungi, anteaya, no more pending requests19:54
fungisdague: regarding the revert proposed in
*** starmer has quit IRC19:55
*** _nadya_ has quit IRC19:56
*** gokrokve_ has quit IRC19:57
fungisdague: and then i reopened the original bug 1274731 which hasn't actually been fixed19:57
uvirtbotLaunchpad bug 1274731 in openstack-ci "non overalapping change to requirements repo broke gate" [Medium,Confirmed]
sdaguefungi: ok, but that's all the right logic to do it on gr19:57
*** dkranz has quit IRC19:57
anteayaSergeyLukjanov: great, thank you19:57
sdagueso shouldn't we instead make that script take a flag about which mode to run in19:57
sdaguebasically I actually think we should consider fixing the base bug atomic with this19:58
sdaguebecause otherwise the base bug is never going to be fixed19:58
*** otherwiseguy has quit IRC19:58
fungii'm not sure why, since the base bug was never fixed at all and this was merely a new bug introduced19:58
fungisdague: well... the slave script is one which gets run projects consuming global requirements. it isn't used for checking the openstack/requirements project changes19:59
*** otherwiseguy has joined #openstack-infra19:59
*** masayukig has quit IRC19:59
fungiand the functions which were added to it really don't have much else to do with that script19:59
sdaguefungi: ok19:59
sdaguelooks like yolanda handled that patch the first time, maybe we can convince her to pick up the base bug for this :)20:00
fungithe functions can probably be mostly extracted into a script committed to openstack/requirements and run from a tox env as a new job20:00
fungisince they're only relevant to that one project, openstack-infra/config is the wrong place for that logic to reside20:01
fungias ir can't be self-testing there20:01
*** esker has joined #openstack-infra20:02
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for ceilometer keystone mock bug 1324885
uvirtbotLaunchpad bug 1324885 in ceilometer/icehouse "py26, py27 gate jobs are failing due to inaccurate mocks of keystone tokens" [Critical,Fix committed]
sdaguefungi: ok, I cede20:03
sdaguelooks like the nova revert is failing again20:03
fungisdague: i've updated bug 1274731 with additional guidance on addressing it20:03
uvirtbotLaunchpad bug 1274731 in openstack-ci "non overalapping change to requirements repo broke gate" [Medium,Confirmed]
*** masayukig has joined #openstack-infra20:04
*** markwash has quit IRC20:04
sdagueok, I'm stepping away from 'puter for a bit. Need to take a break otherwise will brain mush :)20:04
*** weshay has quit IRC20:04
*** starmer has joined #openstack-infra20:05
*** ramashri has joined #openstack-infra20:06
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1326312
uvirtbotLaunchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress]
clarkbok food coma acquired20:06
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer periodic notifier ut bug 1321826
uvirtbotLaunchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress]
openstackgerritAdrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge
anteayaclarkb: well done20:08
clarkbjames_li: we are hoping to have trusty real soon now20:08
fungiit's out old friend, linux ephemeral ports! "error: [Errno 98] Address already in use"
clarkbjames_li: but we keep fighting old fires20:08
clarkbs/old/old and new/20:08
*** nati_ueno has quit IRC20:08
fungiclarkb: what's old is new again20:08
fungiclarkb: new is the new old20:08
anteayawasn't that a kernel bug last time?20:08
clarkbanteaya: no20:09
james_liclarkb: cool.20:09
anteayaor am I mis-remembering20:09
clarkbanteaya: very much a keystone and IANA sillyness20:09
anteayakk, I'm wrong20:09
clarkbanteaya: well not completely20:09
fungianteaya: s/bug/intentional design choice/20:09
clarkbanteaya: certain people tried to frame it that way. I think linux is actually sane here20:09
*** starmer has quit IRC20:10
james_liclarkb: fungi when you are done, I think it will be announced via ML?20:10
james_limeaning trusty20:10
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: WIP: Add various translation jobs
fungijames_li: i'm pretty sure there will be much discussion on the ml as soon as people are ready to start porting jobs to 14.04, yes20:10
fungijames_li: there are lots of improvements waiting on us being able to run things on newer ubuntu lts20:11
james_liok, thanks!20:11
sdaguejames_li: helpers would probably be appreciated20:11
*** adalbas has joined #openstack-infra20:11
*** Sukhdev has quit IRC20:11
fungijames_li: certainly an easy way to help is to try to run devstack, tempest, unit tests for various projects, et cetera on trusty yourself and report bugs/propose fixes20:12
sdaguefungi: it mostly works, I'm on trusty for all my normal development20:12
fungithe more things "just work" when we fire them up on trusty, the easter that will go20:12
*** weshay has joined #openstack-infra20:12
james_lifungi: yeah, I can definitly help on that,20:12
*** dkliban_bbiab is now known as dkliban20:13
fungisdague: not sure if you saw, but the bug which bounced the nova revert out this time was the keystone vs. ephemeral ports cage match20:14
sdagueoh, nice20:14
*** lcheng_ has quit IRC20:14
sdaguewell morganfainberg is going to fix that by getting apache working :)20:14
fungivery much still a problem. very much still no good solution20:14
fungioh, apache front-end. yes20:14
clarkbsdague: I think he mentioned that the test would have to go bcak to experimental though20:14
clarkbbecause lolnoonefixesanything20:14
* fungi fixes nothing20:15
openstackgerritAdrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge
fungisdague: oh, and your havana backport for get-pip is being stomped on by "BuildErrorException: Server %(server_id)s failed to build and is in ERROR status"20:17
*** adrian_otto has joined #openstack-infra20:17
sdaguefungi: nice20:17
*** rlandy has quit IRC20:17
sdagueoh havana20:17
sdagueok, walking away from computer for real now20:17
* fungi is thrilled that we agreed at the summit to support havana for even longer... sigh20:17
*** hemna_ has joined #openstack-infra20:17
sdagueI thought it was icehouse?20:18
fungioh, right, icehouse20:18
sdaguebut regardless, I want that conversation back on the ML20:18
fungiyeah, that's right. we agreed that the keeping havana working ship had already sailed20:18
sdaguebecause the additional load on infra and qa is to the point where I'm not agreed on anything unless more helpers show up20:18
openstackgerritAdrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge
adrian_ottook, done revising.20:19
*** devananda has quit IRC20:19
mgagnedhellmann: are you working on #1326811 ?20:19
*** HenryG has quit IRC20:19
mgagnebug 132681120:19
uvirtbotLaunchpad bug 1326811 in trove "Client failing with six =>1.6 error" [Undecided,New]
*** godp1301 has joined #openstack-infra20:20
*** slamont has joined #openstack-infra20:20
*** mat128 has joined #openstack-infra20:20
*** starmer has joined #openstack-infra20:20
mgagnedhellmann: we found a very unfortunate timing. installs 1.5.6 by default now instead of what I suspect was 1.4.1 before:
Ajaegerfungi: for how long do we want to support icehouse? I missed that one20:21
dhellmannmgagne: yeah, I was looking at that20:22
mgagnedhellmann: and devstack used to be pinned to 1.4.1 before this change got merged:
*** Samolo has joined #openstack-infra20:22
james_lifungi: clarkb : just curious about if any openstack projects already use Trusty for their gates ?20:23
fungiAjaeger: "ACTION: Agreed on 15 months"20:23
dhellmannmgagne: what does pip 1.5.6 do differently?20:23
fungimgagne: dhellmann: right, we're back to consuming latest pip there20:24
james_lifungi: clarkb basically if I test everthing runs fine for Solum, can I switch to Trusty for our gate?20:24
fungimgagne: dhellmann: there are still a few places where we're stuck on 1.4.1 due to needing to wait for a tox 1.8 release20:24
*** starmer has quit IRC20:24
Ajaegerthanks, fungi20:25
dhellmannhow does the newer pip trigger the problem with six being downgraded?20:25
fungijames_li: the current blocker is that we haven't tried actually adding ubuntu 14.04 lts workers to our nodepool yet, and until very recently none of our providers had base images for it nor glance services we could use to upload20:25
mgagnedhellmann: people are reporting that installing pip 1.4.1 instead works (by reverting 7b63c5ec9e9aea54024105a6521a754efc9f9983)20:25
dhellmannmgagne: I understand that. I don't understand why. :-)20:25
dhellmannmgagne: is there a review for devstack to revert that change?20:26
fungimgagne: dhellmann: right, it's almost certainly either something unorthodox in the six packaging, or a dependency ordering issue which is being exposed through happenstance in later pip20:26
mgagnedhellmann: pip 1.5.6 does not require setuptools anymore. dependencies look to be handled differently20:26
* dhellmann sigh20:27
mgagnefungi: one of my coworker has similar issues with dependency resolutions when upgrading to pip 1.5.620:27
james_lifungi: sorry “our providers” mean what?20:27
mgagnefungi: I'll check with him20:27
clarkbmordred: I have reviewed the nodepool dib change. Please see cover comments and inline comments20:27
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for oslo.messaging unit test race bug 1283926
uvirtbotLaunchpad bug 1283926 in oslo.messaging "test_rabbit.TestRacyWaitForReply failed with a not related change" [Low,Fix committed]
*** andreaf has joined #openstack-infra20:27
fungidhellmann: mgagne: i don't expect that commit to be reverted in its entirety, as it was a fix to a pretty severe gate-blocking bug (manifested as workers trying to connect to to get pip 1.4.1 so they could downgrade the one already preinstalled by get-pip on the server image)20:28
clarkbjames_li: our cloud providers that we run all of our tests on20:28
fungijames_li: currently primary rackspace and hpcloud20:28
fungier, primarily20:28
dhellmannfungi: that's fine; is there another fix?20:28
clarkbwe should ask dstufft20:29
james_liclarkb: fungi ic, thanks!20:29
*** gokrokve has joined #openstack-infra20:29
dstufftclarkb: what's up?20:30
fungidhellmann: we could probably cache a copy of pip 1.4.1 on the images and then have devstack install that, but certainly trying to work out why six is breaking on upgrade with modern pip would be good (and so far i've only seen reports of it affecting trove's third-party ci)20:30
clarkbdhellmann: mgagne I really expect that something is pinning six though20:30
clarkbdstufft: we are using pip 1.5.6 in the devstack gate now and it breaks around six apparently20:30
Shrewssdague: WIP for now until the fix is merged:
dstufftcan you define breaks around six20:30
clarkbdstufft: sorry its a very nebulous error because ENOLOGS20:30
clarkbdstufft: but SlickNik can probably help that20:30
fungidstufft: it looks like
fungi"ERROR: Exception raised: six>=1.6.0"20:31
mgagnedhellmann: ok, I was wrong, pip still requires setuptools20:31
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for heat assert isinstance bug 1326946
uvirtbotLaunchpad bug 1326946 in heat "assert isinstance(definition, rsrc_defn.ResourceDefinition) failing gate due to merge conflict" [Low,In progress]
*** lcheng_ has joined #openstack-infra20:32
clarkb* correction devstack gate is fine. Third parties having trouble20:32
fungimgagne: it vendors in a copy on its own now though, as opposed to expecting it to be there or retrieving one from somewhere20:32
dhellmanndstufft: there may be some useful logs on and
uvirtbotLaunchpad bug 1326811 in trove "Client failing with six =>1.6 error" [Undecided,New]20:32
clarkbI love that 97757 has failed again20:32
anteayaclarkb: sorry am I missing context?20:32
anteayaI am switching windows a lot today20:32
clarkbanteaya: context for which subject?20:33
anteayaThird parties having trouble20:33
clarkbanteaya: trove third party tests are broken20:33
fungianteaya: trove's ci is experiencing some issues around python-openstackclient and six when installing with current versions of pip, but not when using pip 1.4.120:33
anteayakk, thanks20:34
*** maxbit_ has joined #openstack-infra20:34
*** julim has quit IRC20:34
*** HenryG has joined #openstack-infra20:34
anteayathat knits together the bits nicely20:34
clarkbfungi: I think the hypothesis of dep resolution is a good one20:34
* fungi tries to reproduce with a virtualenv20:34
*** gokrokve has quit IRC20:34
openstackgerritA change was merged to openstack-infra/elastic-recheck: Added query for ceilometer test_ordination_allocate bug 1323524
uvirtbotLaunchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed]
dhellmannmgagne: when I just ran devstack, with the change to use pip 1.5.6, I got six==1.6.120:35
clarkbfungi: we are still going to restart nodepool yes? do you want my test change to merge before we do that? It has no affect on running nodepool20:35
mgagnedhellmann: In bug 1326925, it is reported that manually installing setuptools 3.6 fixes the issue20:35
fungiclarkb: yeah, i think it's safe. i'm a little uneasy about restarting it while we're resource-starved again, but i suppose it'll recover soon enough20:35
uvirtbotLaunchpad bug 1326925 in devstack " fails when creating openstack admin project" [Undecided,Confirmed]
*** starmer has joined #openstack-infra20:36
clarkbfungi: hrm good point20:36
godp1301dhellmann: also have six=1.6.1 using pip 1.4.120:36
clarkbthe cloud have been hurting today20:36
AjaegerI see now jobs failing with " sudo: /usr/sbin/apache2ctl: command not found"20:36
*** gokrokve has joined #openstack-infra20:36
dstufftdep resolution hasn't really changed20:36
fungiAjaeger: that's an unrelated error which seems to crop up when devstack dies abruptly in other ways20:36
dstufftIssuing the command `sudo pip install setuptools==3.6` fixes the issue.20:37
dstufftI bet20:37
fungiAjaeger: pretend you never saw that error message and go looking in the devstack log for the actual failure20:37
godp1301dstufft is does i confirm20:37
dstufftclarkb: fungi I bet you're getting six from a wheel in pip 1.520:37
dstufftbut from a sdist in 1.420:37
Ajaegerfungi: Argh ;( It hit one of your promoted patches - 97757 - ok, will scroll further up...20:37
mat128“Issuing the command `sudo pip install setuptools==3.6` fixes the issue.”20:37
*** Ryan_Lane1 has quit IRC20:37
*** otherwiseguy has quit IRC20:37
dstufftand an old setuptools is being used20:38
*** maxbit has quit IRC20:38
Ajaegerfungi: please double check - there's nothing else going on in
dstufftold setuptools that doesn't recognize the dist-info directories20:38
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove some queries with no hits
dstufftwhich wheel uses20:38
AjaegerWhat do I miss?20:38
fungiAjaeger: yeah, that one was clobbered by keystone trying to listen on its iana-assigned port which happens to be within the linux default ephemeral ports range and had something already bound on it20:38
fungiAjaeger: the nova revert i mean20:38
clarkbdstufft: I thought pip bundles setuptools though20:38
AjaegerAh. Hit this again here
clarkbdstufft: is that setuptools not installed globally?20:39
Ajaegerfungi: yeah, one was the nova revert20:39
*** Samolo has quit IRC20:39
dstufftclarkb: pip bundles pkg_resources froms etuptools for our own internal uses, however that doesn't control the pkg_resources that is installed globally20:39
dstufftclarkb: will also install setuptools *if* setuptools isn't already installed20:39
fungiAjaeger: on the nova revert, follow that link at the top of the console log to the location of the detailed logs on logs.o.o, then look in the devstack log and you'll see keystone failed to start20:40
clarkbdstufft: is there a way to force it to install setuptools regardless?20:40
fungiAjaeger: then if you look in the screen-key log you'll see keystone complaining that its port is already in use20:40
dstufftclarkb: No, but you could just do pip install --upgrade pip in
clarkbdstufft: if not I suppose we can just do pip install --upgrade setuptools20:40
clarkbdstufft: or that20:40
clarkbdhellmann: mgagne ^20:40
*** starmer has quit IRC20:40
dstufftpip install --upgrade setuptools I mean20:40
dstufftnot pip20:40
clarkbdstufft: thanks20:40
dhellmannclarkb: I haven't been able to reproduce the issue20:40
dhellmanna simple devstack run with nothing special in localrc doesn't exhibit the problem, so I wonder if it depends on a particular service being enabled20:41
*** lcheng_ has quit IRC20:41
mgagnedhellmann: our use case includes ironic20:42
dstufftdhellmann: assuming you're talking about the six thing, it probably depends on what Os you're using20:42
clarkbso I think we can just have devstack do pip install --upgrade setuptools after installing pip20:42
fungidhellmann: do we run openstackclient for anything in normal devstack/tempest jobs? because it's that i saw complaining in trove's logs20:42
*** Ryan_Lane has joined #openstack-infra20:42
clarkbin the gate that should hit our mirror20:42
dstufftclarkb: Looking at the changelog, you need at least setuptools 0.6.2820:42
dstufftto get dist-info support20:43
dhellmannfungi: I think dtroyer has been updating devstack to use openstackclient20:43
mat128fungi: creating projects, roles and stuff20:43
Ajaegerthanks, fungi!20:43
fungidhellmann: mat128: okay, so that theory's out20:43
dstufftclarkb: precise has 0.6.2420:43
clarkbmgagne: SlickNik can you guys test with an explicit pip install --upgrade setuptools after get pip is installed20:44
fungithe example where it's failing claims to be ubuntu, but no idea what release20:44
dhellmannmat128: thanks, what OS are you on?20:44
mgagnefungi: Ubuntu 12.04.3 is used20:44
mat128Ubuntu 12.04.320:44
fungioh, looks like ubuntu precise actually20:44
mat128I was able to reproduce it on 12.04.420:45
mat128using an Ubuntu-provided vagrant box20:45
fungisince its apt-get update is hitting a precise repository20:45
mriedemhow is the gate looking now? better?20:45
mat128stack@devstack:/usr/local/lib/python2.7/dist-packages/six-1.6.1.dist-info$ cat WHEEL20:46
*** UtahDave has joined #openstack-infra20:46
mat128looks like six is a wheel20:46
openstackgerritAnita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for heat assert isinstance bug 1326946
uvirtbotLaunchpad bug 1326946 in heat "assert isinstance(definition, rsrc_defn.ResourceDefinition) failing gate due to merge conflict" [Low,In progress]
clarkbmriedem: you are funny20:46
mriedemjust coming in late20:46
fungioh! i wonder if we're shielded from this in the upstream gate by not mirroring wheels?20:46
dhellmannmat128: six 1.6.1 should be fine, though, right?20:46
dhellmannmat128: is there a second copy of six that is *not* a wheel?20:47
*** gokrokve has quit IRC20:47
mat128there's a file directly in there20:47
*** marcoemorais1 has quit IRC20:47
clarkbfungi: could be20:47
dstufftmat128: you have setuptools 0.6.24 don't you20:47
*** yjiang5 has quit IRC20:47
clarkbfungi: hrm we shouldn't have wheels there unless hrm20:47
clarkbfungi: is run-mirror smart enough to put non platform specific wheels in the regular mirror?20:48
*** otherwiseguy has joined #openstack-infra20:48
*** marcoemorais has joined #openstack-infra20:48
dstufftPyPI doens't allow you to upload linux wheels anyways20:48
dstufftthat are platform specific to linux that is20:48
fungiclarkb: i think run-mirror generates wheels20:48
clarkbfungi: it does but it should only put them in the wheel mirror20:48
clarkbthat wheel above shouldn't be there20:48
*** yjiang5 has joined #openstack-infra20:49
mat128re-running stack with the old code created a six-1.6.1.egg-info20:49
mat128I have to leave but slamont has the same setup, he will be answering questions20:49
mat128he's right next to me :P20:49
* dhellmann is having network trouble and can't reach his remote server20:50
*** andreykurilin_ has joined #openstack-infra20:50
*** starmer has joined #openstack-infra20:51
*** weshay has quit IRC20:51
slamontdstufft, mat128 : 0.6.24-1ubuntu120:51
dstufftthat's the problem then20:51
*** lcheng_ has joined #openstack-infra20:52
dstufftsetuptools prior to 0.6.28 didn't understand .dist-info, so it doesn't see six as installed at all (becuase it's installed with a .dist-info directory) and the pkg_resources check is failing.20:52
fungiahh, i *believe* we kill distro packages of pip, setuptools, et cetera with fire before running get-pip20:52
fungion our nodepool images20:52
dstufftthe reason the 1.4 -> 1.5 upgrade triggered this, is 1.5 installs wheels by default20:52
fungiso when devstack shows up at the party, old setuptools is already out the door20:52
clarkbfungi: we do20:53
godp1301python-pip is uninstalled by install_pip: uninstall_package python-pip20:53
godp1301nothing else20:53
dstufftyou often times can't unintall python-setuptools becuase things depend on it20:54
godp1301mgagne and I are trying the pip install —update setuptools fix now20:54
fungigodp1301: so it sounds like devstack's install_pip should also uninstall_package python-setuptools20:54
dstufftfungi: that might not work20:54
dhellmannfungi: are we seeing this problem in our gate, or just on developer systems?20:54
dstufftsee: dependencies20:54
fungidhellmann: we haven't seen it in our gate20:54
fungidstufft: yep, good point20:54
dhellmannok, that's good at least20:54
*** Samolo has joined #openstack-infra20:54
dstufftfungi: but luckily pip gives no fucks and will happily stomp all over the apt-get installed setuptools :V20:54
clarkbfungi: it just needs ot upgrade setuptools20:55
*** starmer has quit IRC20:55
dstufftclarkb: fungi I'm going to poke some contacts in cannonical to see if maybe I can convince them to SRU setuptools to patch in dist-info20:56
dstufftpatch in dist-info support*20:56
fungiheh... in devstack lib/infra there's a function unfubar_setuptools20:56
fungidstufft: that sounds helpful20:57
Alex_Gaynordstufft: do you recognize ?20:57
mgagnefungi: looks to never be called =)20:58
fungiit looks like nothing ever calls it, yeah20:58
godp1301fungi: you mean removing python-setuptools from /files/* ? If so, I'll have a look at what needs to be done20:58
dstufftAlex_Gaynor: nope20:58
Alex_Gaynordstufft: oh boy20:58
godp1301fungi: slightly similar to what if in /lib/infra20:59
godp1301fungi:   uninstall_package python-setuptools20:59
godp1301    pip_install -U setuptools20:59
godp1301    pip_install -U pip20:59
clarkbyou won't need to upgrade pip as get pip should do that for you20:59
dstufftmaybe should put a lower bounds on the setuptools it accepts21:00
*** resker has joined #openstack-infra21:00
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Create zuul template translation-jobs
*** stevebaker has quit IRC21:00
*** stevebaker has joined #openstack-infra21:00
*** esker has quit IRC21:01
*** aysyd has quit IRC21:02
fungimgagne: looks like dtroyer removed the last call to it in when he brought in get-pip21:02
fungimgagne: prior to that it was being called from stack.sh21:02
*** prad has quit IRC21:03
fungiAlex_Gaynor: looks like get_script_header() returns a bytes type when run under 3.321:04
Alex_Gaynorfungi: why did it just start showing up though21:04
dstufftbecause computer21:04
fungiAlex_Gaynor: my next step would be to git blame around that function21:05
fungigit blamethrower21:05
dstuffthave you tried turning it off and on again21:05
*** starmer has joined #openstack-infra21:05
fungiAlex_Gaynor: was gate-python-swiftclient-python33 nonvoting recently?21:06
Alex_Gaynorfungi: it's been voting for a week or two21:07
*** rfolco has quit IRC21:07
fungiAlex_Gaynor: in that case, i'd suggest you find the most recent successful run you can and diff the pip freeze from them21:07
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for oslo.messaging unit test race bug 1283926
uvirtbotLaunchpad bug 1283926 in oslo.messaging "test_rabbit.TestRacyWaitForReply failed with a not related change" [Low,Fix committed]
*** smarcet has quit IRC21:08
fungiAlex_Gaynor: when things spontaneously start to break, more often than not it's a new release of an outside dependency doing it21:08
dstufftaren't dependencies awesome21:08
fungithey are when they work and keep you from needing to write as much code21:09
Alex_Gaynorfungi: pytz and keystoneclient are the only things that upgraded21:09
zarojhesketh: ping21:10
*** starmer has quit IRC21:10
* dhellmann wonders if his chef-solo config is "fixing" this before he even runs devstack21:10
dtroyertools/ used to also manually install a reasonable setuptools, maybe it's time to re-introduce that?21:11
fungiAlex_Gaynor: oh... this is breaking down inside i on;y just spotted that21:12
dstuffthere's a question21:12
*** e0ne has quit IRC21:12
dstuffthow come devstack doesn't just use a virtualenv21:12
Alex_Gaynorfungi: yeah, is a real bad place :/21:12
Alex_GaynorI've reproduced this locally, I wonder if it effects other packages21:13
Alex_Gaynorl;et me get a bug filed21:13
dstufftwoe to thee who gazes upon easy_install.py21:13
fungiAlex_Gaynor: so python-swiftclient is uninstallable in a python 3.3 venv i guess, looking closer at the traceback21:13
*** andreykurilin_ has quit IRC21:14
*** pdmars has quit IRC21:14
*** aburaschi has quit IRC21:15
fungiAlex_Gaynor: actually, installing tip of master python-swiftclient in a python3.3 venv seems to work for me... checking what pip freeze claims there21:15
*** Ryan_Lane has quit IRC21:16
fungiAlex_Gaynor: what are your minimal steps to reproduce?21:16
*** slamont has quit IRC21:16
fungilooks like i'm getting python-swiftclient== requests==2.3.0 simplejson==3.5.2 six==1.6.121:16
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for heat assert isinstance bug 1326946
Alex_Gaynorfungi: rm -rf .tox/py33; tox -e py3321:17
uvirtbotLaunchpad bug 1326946 in heat "assert isinstance(definition, rsrc_defn.ResourceDefinition) failing gate due to merge conflict" [Low,In progress]
*** sarob has quit IRC21:17
*** andreaf has quit IRC21:17
zarojhesketh: got myself onto a ubuntu dev env.  ran turbo-hipster tests from master (tox -epy27), got the following error:
*** devananda has joined #openstack-infra21:17
*** sarob has joined #openstack-infra21:18
*** Ryan_Lane has joined #openstack-infra21:18
fungiAlex_Gaynor: yep, reproduced on debian/testing... seeing if i can whittle down the reqs21:19
Alex_Gaynorfungi: is an issue I filed for this21:20
uvirtbotLaunchpad bug 1326972 in python-swiftclient "swiftclient fails to install under tox with Python3" [Undecided,New]21:20
*** andreykurilin_ has joined #openstack-infra21:20
*** starmer has joined #openstack-infra21:20
*** nati_ueno has joined #openstack-infra21:22
*** sarob has quit IRC21:22
*** Ajaeger has quit IRC21:23
fungiAlex_Gaynor: unable to reproduce without tox in a python3.3 virtualenv using './foo/bin/pip install -U -r requirements.txt -r test-requirements.txt .'21:24
fungiso looks fairly tox-specific. hopefully i can narrow it down further21:24
*** starmer has quit IRC21:25
mgagnedhellmann: looks like updating setuptools after installing pip fixes the issue21:25
*** apevec has joined #openstack-infra21:25
Alex_Gaynorfungi: it's weird though, my tox version didn't change, this si going to be subtle21:25
dhellmannmgagne: yeah, and my chef setup installs setuptools and pip for me, so I never saw this at all21:26
fungimgagne: dhellmann: so probably just adding a pip install -U setuptools immediately after running get-pip in devstack is the best way forward21:26
apevecttx, fungi - looks like I lost priv to push tags:  ! [remote rejected] 2014.1.1 -> 2014.1.1 (prohibited by Gerrit)21:26
apevecerror: failed to push some refs to 'ssh://'21:26
fungiapevec: checking into it now21:26
mgagnefungi, dhellmann: what I tested with:
dhellmannfungi: that seems reasonable21:27
apevecfungi, or could be b/c I changed my key to ?21:27
apevecI had before21:27
fungiapevec: you've got membership in a group with access to push signed tags to refs/* for all projects21:27
*** slamont has joined #openstack-infra21:28
apevecfungi, is that checking "tagger" field?21:29
fungiapevec: is your address listed as an additional address in your gerrit settings?21:29
apevecit is not21:29
* apevec fixes21:29
*** mfer has quit IRC21:29
fungiyeah, that might be needed first21:29
fungiit'll want to send you a test e-mail message to activate that additional address too21:29
*** andreykurilin_ has quit IRC21:30
dtroyermgagne: can you try that as 'pip_install —upgrade setuptools'21:30
dtroyerthat will let it work from behind a proxy and other nice things21:30
fungiAlex_Gaynor: aha! it's develop which triggers this, not installing21:30
mgagnedtroyer: is pip_install -U setuptools ok?21:31
SlickNikclarkb: Sorry was away tending to something else. Catching up on scrollback now…21:31
mgagnedtroyer: looks to be the syntax used elsewhere21:31
dtroyersure, I'm just used to using the long opts21:31
*** Guest43801 is now known as Woodie21:32
apevecfungi, yep, that was it, thanks!21:32
fungiapevec: you're welcome21:32
*** prad has joined #openstack-infra21:33
openstackgerritFerrante proposed a change to openstack-infra/jenkins-job-builder: Adds support for system-groovy
*** lbragstad has quit IRC21:34
mgagnedtroyer: alright, gimme a couple of minutes to test it21:34
fungiAlex_Gaynor: further whittling, if you only install requirements.txt into the virtualenv, then develop works fine. the breakage is when you also have test-requirements.txt installed, so it's some interaction there21:34
fungii'll try to bisect the culprit out of test-requirements.txt21:34
*** mrda-away is now known as mrda21:35
*** arnaud__ has quit IRC21:35
*** starmer has joined #openstack-infra21:36
*** nati_uen_ has joined #openstack-infra21:39
*** nati_ueno has quit IRC21:39
Alex_Gaynorfungi: bizzare21:39
jogofungi clarkb: so AFAIK zuul know dynamically calculates the integrated gate? can we get that data into ER?21:39
clarkbjogo: no21:40
jogoclarkb: no to which part?21:40
*** starmer has quit IRC21:40
*** sarob has joined #openstack-infra21:40
jogo has the word integrated21:40
clarkbits not dynamic :)21:40
*** masayukig has quit IRC21:40
clarkbits a zuul layout entry21:40
clarkband getting that to ER is tricky because ER only knows what jenkins tells it21:41
clarkband jenkins has no concept of gates and stuff21:41
clarkbwe could in theory pass through that as a variable but it wouldn't be used for anything else21:41
jogozuul doesn't talk to ES directly?21:41
*** Ryan_Lane has quit IRC21:41
clarkbjogo: no21:41
*** Ryan_Lane has joined #openstack-infra21:41
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for setuptools py26 unicode bug 1326878
jogoclarkb: so the issue I want to address here is:
uvirtbotLaunchpad bug 1326878 in pbr "Race "TypeError: expecting a bytes object, got unicode" on py26 setup (dup-of: 1325514)" [Undecided,New]
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,Fix released]
fungiAlex_Gaynor: hacking!21:42
jogohas no concept of what is integrated and what isn't21:42
SlickNikclarkb / mgagne: I can confirm that an explicit upgrade of setuptools fixes the issue.21:42
clarkbjogo: it could ask zuul maybe21:42
jogoclarkb: ohh, hmm that may work21:43
clarkband cache that for a day or so before asking again21:43
jogogive it a job name and get a yes or no back?21:43
fungiAlex_Gaynor: working commands... ./foo/bin/pip install -U -r requirements.txt ; . foo/bin/activate ; python develop21:43
mgagneSlickNik: yep, testing with suggested approach by dtroyer21:43
jogowell the uncategorized page is just a cron job21:43
jogoas you know21:43
Alex_Gaynorfungi: yup, new hacking release yesterday21:43
jogoAlex_Gaynor: have another version to release today21:43
clarkbjogo: I was thinking grabbing the list of projects for the integrated gate via zuul status.json21:43
jogowaiting for a review21:43
fungiAlex_Gaynor: but this breaks as seen in the job... ./foo/bin/pip install -U -r requirements.txt 'hacking>=0.8.0,<0.9' ; . foo/bin/activate ; python develop21:44
*** gokrokve has joined #openstack-infra21:44
clarkbjogo: all of the projects should be listed in the json file21:44
mtreinishjogo: why not just pull infra/config and parse the zuul yaml...21:44
*** dkliban is now known as dkliban_afk21:44
mtreinishsince it's infrequent21:44
dhellmannwoo, I finally reproduced the six failure21:44
Alex_Gaynorjogo: have you been folloing this disussion between fungi and myself?21:44
*** denis_makogon has quit IRC21:44
openstackgerritAdrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge
jogoAlex_Gaynor: no I just ssaw the words hacking and I cut a new one yesterday21:44
fungijogo: installing hacking into a py33 venv causes subsequent develop to fail for swiftclient21:45
Alex_Gaynorjogo: might want to read back, it looks like the newest hacking on py3k causes some weird breakages21:45
jogomtreinish: hmm that may work want to work on that ?21:45
mgagnedhellmann: you cloned devstack from master and ? nothing else?21:45
fungijogo: my gut says it's one of the newer deps in the latest hacking release21:45
slamontdhellmann: I put a pip freeze output in LP 132681121:45
uvirtbotLaunchpad bug 1326811 in trove "Client failing with six =>1.6 error" [Undecided,New]
*** sarob has quit IRC21:45
fungijogo: but i haven't started bisecting those yet to figure out which21:46
mtreinishjogo: actually I think clarkb's idea makes more sense. But I can do the yaml pull idea too (it shouldn't be hard)21:46
*** maxbit has joined #openstack-infra21:46
*** masayukig has joined #openstack-infra21:46
jogomtreinish: cool I nominate you to make it happen ;)21:46
jogofungi: let me read backlog. not sure what I broke21:46
jogoAlex_Gaynor: so I am a little lost. nothing should be using hacking 0.9.0 yet21:48
*** maxbit_ has quit IRC21:48
fungijogo: i suspect it may be a new release of something hacking depends on21:48
fungijogo: i'm just working my way down the dependency chain21:48
jogofungi: we pin all of those21:48
fungihacking was the first-order dep introducing this behavior21:49
jogoso all uses of hackign should pin to <0.9.021:49
jogoas 0.9.0 adds new rules21:49
jogofungi: have a link to the failed job?21:49
jogofungi: also we gate py33 on new hacking21:49
fungijogo: (though i doubt it will help)21:50
*** starmer has joined #openstack-infra21:50
*** dims__ has joined #openstack-infra21:51
*** Samolo has quit IRC21:51
SlickNikmgagne: Thanks! Let me know if there's something I can help with (testing or otherwise) to speed things up!21:51
*** g_306396236 has joined #openstack-infra21:51
*** marcoemorais has quit IRC21:52
jogofungi: so why do you think its hacking?21:52
fungijogo: Alex_Gaynor: yeah, i've ruled out hacking by only installing its deps, and still reproduce21:52
*** marcoemorais has joined #openstack-infra21:52
jogofungi: phew21:52
mgagneSlickNik, dhellmann: I don't know what to do from now. I know how to fix it but don't know the technical reason as to WHY updating setuptools fixes the issue.21:52
Alex_Gaynorjogo: sorry for the noice21:52
jogoAlex_Gaynor: haha no worries glad I didn't break things  (this time) :)21:52
fungijogo: i don't think it's hacking, but i bisected test-requires.txt down to only the hacking entry and could still reproduce. now i'm moving down into hacking's deps to see which is responsible21:53
dhellmannslamont: thanks21:53
dhellmannmgagne: yeah, I turned off the bits of my chef-solo setup that were pre-installing setuptools and pip, which hid the problem21:53
SlickNikmgagne: I believe the reason is that six is installed using wheels, and the older setuptools doesn't understand the .dist-info format, so it doesn't think six is installed.21:53
*** dims_ has quit IRC21:53
clarkbSlickNik: according to dstufft that is correct21:54
jogofungi: I am rerunning a check on a hacking patch21:54
jogoto see what happens21:54
*** g_306396236 has quit IRC21:54
clarkbin other news I have dib running on precise to build a nodepool dib image21:54
clarkbtesting mordred's dib stuff21:55
clarkbif I get this image built next step is trusty21:55
*** starmer has quit IRC21:55
mgagnedhellmann: still have your "broken" devstack around?21:55
dhellmannmgagne: yeah21:55
mgagnedhellmann: what's the version of setuptools installed?21:55
dhellmannmgagne: oh, wait, no, I fixed it :-/21:56
*** g_306396236 has joined #openstack-infra21:56
jogofungi: so the latest hacking worked with py3321:56
*** g_306396236 has quit IRC21:56
dstufftmgagne: someone with a broken devstack said they had setuptools 0.6.2421:56
dhellmannwhen I updated setuptools it was 0.6.something21:56
dstufftyou need 0.6.2821:56
dhellmannyeah, a dev version of 0.6.2421:56
dstufftfor wheels to not be broken21:56
SlickNikmgagne / dhellmann: I have results of a pip freeze from a borked instance21:56
dhellmannnow I have 4.0.121:56
jogoFWIW we didn't gate hacking 0.8.1 on py3321:56
jogoAlex_Gaynor: ^21:56
dhellmannis that right?21:56
SlickNikI can double check21:56
dstufftdhellmann: that's the latest version yea21:57
dhellmannhow did we get from 0.6 to 4.1?21:57
dhellmannwow, ok, I had no idea21:57
mgagnedhellmann: 0.6.24 <<<<<< 4.0.121:57
dstufftsetuptools is doing the semver thing21:57
dhellmannmakes sense21:57
dstufftso any little backwards cinompat gets a mjor bump21:57
fungiAlex_Gaynor: jogo: narrowed it down to... ./foo/bin/pip install -U -r requirements.txt 'flake8==2.0' ; . foo/bin/activate ; python develop21:57
mgagne0.6.24 comes from the system (Ubuntu)21:57
Alex_Gaynordstufft: s/semver/chrome/firefox/21:57
dstuffti'm not entirely sure it's a good idea, because you can't really reasonably pin setuptools21:57
*** jerryz has quit IRC21:57
dhellmannok, I'm not sure I have anything else useful to add to the setuptools conversation, so I'm going to see about delicious dinner.21:57
Alex_Gaynorfungi: no new flake8 realses since last october though?21:57
fungiAlex_Gaynor: yeah, which is hella strange21:58
jogofungi: ahh that makes sense, latest hacking uses a newer one21:58
jogoAlex_Gaynor: we pin that21:58
jogoso things don't berak21:58
fungiAlex_Gaynor: so installing requirements.txt and flake8 into the virtualenv and then trying to develop goes sideways21:58
jogofungi: what happens if you bump hackign to 0.9.0?21:59
jogodoes that work?21:59
*** g_306396236 has joined #openstack-infra21:59
*** g_306396236 has quit IRC21:59
fungijogo: Alex_Gaynor: so the issue is exhibited with flake8=2.0 installed but not with flake8==2.1.022:00
jogofungi: thats what I expect as hacking 0.9 works22:02
openstackgerritDan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstack_extras
*** markwash has joined #openstack-infra22:02
*** stevebaker has quit IRC22:03
*** stevebaker has joined #openstack-infra22:03
fungijogo: Alex_Gaynor: what's strange is that the only difference in pip freeze output between them is flake8==2.0 vs flake8==2.1.022:03
*** resker has quit IRC22:04
jogohmm is this a  newer pip or something?22:04
fungismoking gun!22:05
mgagneworking on patch22:05
fungipip freeze is incomplete22:05
*** starmer has joined #openstack-infra22:05
*** devananda has quit IRC22:05
fungipip list exposes a difference in setuptools versions22:05
*** sarob has joined #openstack-infra22:06
fungiworking venv has flake8 (2.1.0) and setuptools (3.4.4)22:06
*** devananda has joined #openstack-infra22:06
fungibroken venv has flake8 (2.0) and setuptools (4.0.1)22:06
fungijogo: Alex_Gaynor: ^22:06
*** markwash_ has joined #openstack-infra22:06
*** signed8bit has quit IRC22:06
*** markwash has quit IRC22:06
*** markwash_ is now known as markwash22:06
jogofungi: odd22:07
*** devananda has quit IRC22:07
SlickNikmgagne: Thanks!22:07
openstackgerritBrant Knudson proposed a change to openstack/requirements: Add a tox env to check requirements overlap
mesteryinfra folks, I replied to the gate failures thread with what I hope is a workable plan around the "ssh timeouts" issues with neutron:
mesteryIt involves a revert of a patch from armax which we think is making this worse, which will help lessen the load while we continue working the root cause.22:08
jogomestery: woot22:09
fungiAlex_Gaynor: so the end of this story (and the beginning of the next) is "setuptools 4.0.1 release broke develop for python-swiftclient under python 3.3"22:09
*** slamont has quit IRC22:09
mesteryjogo: Hopefully this can relieve pressure while we continue root causing.22:09
mesteryjogo: Here is the revert:
anteayamestery: I'm for reverting what caused the spike22:09
anteayamestery: what more do you need to keep getting to the root issue?22:10
jogosdague: ^22:10
*** starmer has quit IRC22:10
mesteryanteaya: Well, we think it made it worse by uncovering something, but covering it up for now will move things back to where they were. :)22:10
Alex_Gaynorfungi: wahoo, can I convince you to file a bug with setuptools?22:10
mesteryanteaya: salv-orlando has a patch to dump console logs into the tempest logs, that would help a bit I think.22:10
*** sarob has quit IRC22:11
mesteryanteaya:, but I think mtreinish indicated it was superseded by a few patches from mriedem.22:11
anteayamestery: awesome22:11
mesteryanteaya: See the comments at the bottom.22:11
SlickNikmestery: Nice! :)22:11
*** MarkAtwood has quit IRC22:12
*** gondoi is now known as zz_gondoi22:12
anteayamestery: can you scare up some more neutron reviews for mriedem's patches please?22:12
jogomtreinish: ^ thoughts on the revert22:12
fungiAlex_Gaynor: you bet, but first i need to cook dinner22:12
anteayamestery: another 2 neutron dev reviews would be great22:13
mgagneSlickNik, dhellmann, clarkb:
Alex_Gaynorfungi: thanks /cc dstufft22:13
mesteryanteaya: I will do it.22:13
*** signed8bit has joined #openstack-infra22:13
anteayamestery: thanks22:13
mesteryanteaya: Thanks!22:13
*** godp1301 has quit IRC22:13
jogomestery: can you update teh commit msg on
jogomake it a little more verbose22:13
mesteryjogo: Looking ...22:14
mesteryjogo: Yes, let me do that now quickly.22:14
jogomestery: thanks.22:14
mtreinishmestery: I'm fine with reverting it if the commit msg explains why22:15
mesterymtreinish: Cool! Working on that now.22:15
jogomestery: faster ;)22:15
mtreinishbut will you be able to reproduce it outside of the gate?22:15
*** sarob has joined #openstack-infra22:15
mesteryjogo: :P22:15
mesterymtreinish: We haven't had much luck, we plan to resubmit the patch which broke it and debug it that way.22:15
mgagneI have to go now, I'll address comments on my change (if any) later tonight22:15
*** bknudson has quit IRC22:16
mtreinishmestery: ok makes sense22:16
*** patelna has quit IRC22:16
*** jgrimm has quit IRC22:16
SlickNikmgagne: Can I co-author the patchset with you? I can take over any comments.22:17
anteayaso yeah, I think mriedem's logging changes look a bit more specific and I'm for reverting the thing that created the spike22:17
SlickNikmgagne: Trying to get the fix out as soon as I can, so that the trove tests can run green again. :)22:17
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for setuptools py26 unicode bug 1326878
uvirtbotLaunchpad bug 1326878 in pbr "Race "TypeError: expecting a bytes object, got unicode" on py26 setup (dup-of: 1325514)" [Undecided,New]
mesteryjogo: resubmitted22:18
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,Fix released]
mesteryjogo mtreinish: I have to take off for a bit now, I'll be back later tonight though, PM if you need anthing else for when I return.22:19
fungiAlex_Gaynor: btw i narrowed it down to being introduced between setuptools 3.8 and 4.0 and have added a simpler reproduction in a comment to bug 132697222:19
uvirtbotLaunchpad bug 1326972 in python-swiftclient "swiftclient fails to install under tox with Python3" [Undecided,New]
* mestery goes afk.22:19
mtreinishmestery, jogo: ok +A on the revert22:20
*** starmer has joined #openstack-infra22:21
jogomestery: nice commit msg22:21
*** Woodie is now known as W00die22:21
jogofungi: clarkb ^ can we get that patch bumped?22:21
jogohmm I think fungi is eating22:22
clarkbjogo: which one?22:22
clarkbjogo: once it makes it into the gate sure22:22
*** marcoemorais has quit IRC22:22
anteayajenkins is unhappy on one of mriedem's add logging patches:
jogoclarkb: thanks WFM22:22
*** marcoemorais has joined #openstack-infra22:23
*** starmer has quit IRC22:25
anteayaERROR: the main setup script run by this job failed - exit code: 122:28
clarkbwell that will ruin your day22:28
anteayabut devstacklog finished with an ip22:28
*** stevebaker has quit IRC22:28
*** stevebaker has joined #openstack-infra22:28
anteayamy day?22:28
clarkbwhoever got the error :)22:28
*** maxbit_ has joined #openstack-infra22:28
clarkblooks like the tempest revert failed22:29
anteayathe failure was here:
anteayathe one I'm looking at22:29
*** maxbit has quit IRC22:30
anteayablast it did so (the tempest revert failed)22:30
jogoclarkb: 2014-06-05 22:27:54.866 | 2014-06-05 22:27:54.848 | python: can't open file '/opt/stack/old/devstack/files/': [Errno 2] No such file or directory22:30
clarkbjogo: it will be in new/22:31
clarkbjogo: is that a grenade job?22:31
jogoclarkb: yes22:31
clarkbjogo: did grenade fail because of that or continue and download get pip itself?22:31
clarkbbecause its supposed to fall back on the internet22:31
jogoclarkb: hmm it failed hard22:32
jogosee console22:32
clarkbso probably a bug in grenade and d-g22:33
jogoclarkb: whee22:33
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326754
uvirtbotLaunchpad bug 1326754 in tripleo "Ubuntu CI failures  : Unable to locate package ubuntu-cloud-keyring" [High,Triaged]
jogodidn't we just land a pip change to stable branches of devstack?22:34
jogoclarkb: or are those still in flight22:34
clarkbjogo: maybe I haven't been following22:34
*** slamont has joined #openstack-infra22:34
clarkbwe landed the master one22:34
sdaguejogo: looks solid22:34
clarkbso the problem is the file is curled without being written to disk22:35
clarkbif you look at he log the zip file is just blasted out to stdout22:35
*** starmer has joined #openstack-infra22:36
clarkbso that is grenade/devstack bug22:36
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module
clarkbthen we have to figure out why the file was not copied into the old/ dir22:36
clarkband I know why22:36
zaroclarkb: do you happen to know how to debug a jjb test case?22:36
anteayaI'm going to get something to eat and will continue groking logs when I return22:36
clarkbzaro: what do you mean by 'debug'. Use the debugger?22:36
sdagueclarkb: what's up with the get-pip thing now?22:37
clarkbsdague: jogo d-g does a mv from the cache-files location instead of a cp (because we have disk constraints) and it copies new/ first so new/ has all the cached files but old/ doesn't22:37
zaroclarkb: i'm trying to run a jjb test in eclipse.  i think it's using nose and it's not working.22:37
clarkbsdague: two things. d-g isn't putting the cached files in the old/ dir for grenade for the reason above ^22:37
sdagueclarkb: ah...22:37
clarkbsdague: second thing the fallback of grabbing the file directly fails because the file is downloaded with curl and spit to stdout instead of disk22:37
sdagueclarkb: hmmm22:38
clarkbI can work a fix for d-g22:38
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1326754
uvirtbotLaunchpad bug 1326754 in tripleo "Ubuntu CI failures  : Unable to locate package ubuntu-cloud-keyring" [High,Triaged]
*** apevec has quit IRC22:39
sdagueclarkb: oh, yeh, havana22:40
*** starmer has quit IRC22:40
sdagueyep, havana is completely wrong22:40
sdague... so that's why I merge conflicted22:40
fungiclarkb: sdague: jogo: when did this first manifest (why didn't we see it on the devstack change which started relying on get-pip)?22:40
fungioh, only havana22:40
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Add a real check for the integrated gate jobs
sdaguefungi: yeh, it's the icehouse grenade job22:41
*** eharney has quit IRC22:41
sdagueso old is havana22:41
jogosdague: heh22:41
fungidinnering, then will pitch in if it's still full of fail22:41
*** thedodd has quit IRC22:41
sdagueyep, my havana change is still wrong22:41
sdagueok, let's me get to fixing22:42
sdagueyou know, I have another cleanup I'm going to merge with this22:42
*** doug-fish has left #openstack-infra22:43
*** prad has quit IRC22:43
Alex_Gaynorfungi: if you haven't already, I'll file a bug against setuptools with your reproduction steps?22:44
*** zzelle has quit IRC22:44
fungiAlex_Gaynor: feel free! i was going to look first to see whether there was already one open for this22:44
*** devananda has joined #openstack-infra22:45
*** devananda has quit IRC22:45
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Put cached files in old/ dirs too
clarkbjogo: sdague ^22:45
*** james_li has quit IRC22:46
*** lcostantino has quit IRC22:46
*** jhesketh has quit IRC22:47
fungiAlex_Gaynor: looking over their tracker, nothing jumps out at me22:47
Alex_Gaynorfungi: filed
*** devananda has joined #openstack-infra22:48
fungiAlex_Gaynor: coolio22:48
*** starmer has joined #openstack-infra22:50
*** dangers is now known as dangers_away22:52
sdagueclarkb: - that should fix havana22:52
sdagueclarkb: you want to hardlink them? or are we jumping filesystems22:53
clarkbsdague: jumping filesystemms22:54
sdagueok +2 from me22:54
clarkbsdague: /opt is the big fs mounted on the ephemeral drive /home/jenkins/cache/files is the root fs22:54
*** jamielennox|away is now known as jamielennox22:54
clarkbwe can probably symlink22:54
clarkbbut we are trying to remove disk usage on /22:54
clarkbbecause its only 10G22:55
sdagueyep, no worries22:55
*** starmer has quit IRC22:56
clarkbmordred: I have questions about dib22:56
*** lcheng_ has quit IRC22:56
clarkbmordred: I am now at the point of wondering whether you tested these elements >_>22:56
*** maxbit_ has quit IRC22:56
*** maxbit has joined #openstack-infra22:57
pleia2my favorite part about jboss is how it makes me feel like I don't know how to use computers22:58
pleia2go forth and do magic things (you better work)22:58
pleia2((it doesn't work))22:59
clarkbpleia2: that sounds like dib22:59
clarkbit isn't running puppet22:59
clarkbit should run puppet22:59
clarkbI have no idea what is going on22:59
morganfainbergclarkb, so what you're saying is they successfully ported the jboss feature pleia2 just described to dib... or vice versa22:59
krotscheckAnyone know if there’s a dependency-and/or-other reason that would prevent me from adding/using puppetlabs/rabbitmq to /config?23:00
morganfainbergi dislike "go do magic things" instructions.23:00
clarkbkrotscheck: not that I know of23:01
*** zns has joined #openstack-infra23:01
clarkbmorganfainberg: in this case its deceptive and I think I understand it because its just bash23:01
pleia2morganfainberg: heh, it's just complicated and I've never gotten to deep into it so the screens of launching stuff is greek (well, java) to me :)23:01
clarkbbut in reality I have no idea what is going on23:01
morganfainbergclarkb, :( well if i can help (why do i say these things) let me know. happy to add a spare pair of eyes if it'd help you at all23:02
pleia2mattoliverau: good morning23:02
krotscheckclarkb: Awesome, thanks :)23:02
clarkbmorganfainberg: thanks but so far I have been able to hit it with a hammer until things go23:02
clarkbmost recent hammer was commenting things out23:03
morganfainbergclarkb, hehe23:03
*** amcrn has quit IRC23:03
morganfainbergpleia2, i think that describes most of my recent interactions with Java based apps.23:03
pleia2I'm glad it's not just me23:04
mattoliverauHammers: the only tool your ever really need :P23:04
*** starmer has joined #openstack-infra23:05
openstackgerritJeremy Stanley proposed a change to openstack-dev/pbr: Check for git before querying it for a version
jogoclarkb: I would review but I don't have enough context to about that part o fthe system
clarkbjogo: when fungi gets back he should be able to review23:09
clarkbin fairness to mordred his dib elements have just drifted away from some of the refactoring around image builds we have done23:10
*** starmer has quit IRC23:10
jogogood news is it looks like things aren't getting worse right now23:11
clarkbbut that coupled with my lack of working dib knowledge is confusing. Trial by fire is good for me23:11
anteayamorning mattoliverau23:14
fungiclarkb: i'm a little weirded out that $copy_cache is treated as a bool switch based on whether you put arbitrary string contents in the parameter, but it looks like it should work23:14
clarkbfungi: ya its clunky. Have a better idea?23:14
clarkbfungi: could pass in mv or cp23:14
clarkband get rid of the if23:14
*** jhesketh has joined #openstack-infra23:14
clarkbwould allow you to use rsync and whatever else too23:15
fungimeh. i don't feel strongly enough to suggest anything else23:15
mattoliverauMorning anteaya, how you this fine day?23:16
sdagueyeh, I'd say lets just go with it. It's not the squireliest thing we've got23:16
anteayamattoliverau: tired but still typing23:16
anteayamattoliverau: how are you?23:16
anteayamorning jhesketh23:16
jheskethzaro: Interesting.. that was just running tox?23:16
jheskethheya anteaya23:16
mattoliverauanteaya: lol, don't stay up too late, we will all need you refreshed for tomorrow :) I'm really good. It's Friday before a long weekend! And the wife and I are going to visit Tasmania this weekend, so rather excited!23:17
fungiclarkb: sdague: out of curiosity why wasn't this bothering us before? is the first file grenade's really expected in there (by way of havana devstack)?23:18
fungidoes this go away after havana is no more?23:18
clarkbfungi: because devstack never used get pip before23:18
openstackgerritA change was merged to openstack-infra/elastic-recheck: Added query for glance _authorized_upload_ bug 1326899
uvirtbotLaunchpad bug 1326899 in glance "FAIL: glance.tests.unit.v1.test_api.TestGlanceAPI.test_add_copy_from_image_authorized_upload_image_authorized" [Undecided,New]
clarkbfungi: it was using the pip 1.4.1 sdist23:18
anteayamattoliverau: awesome!!23:18
clarkbfungi: that and the fallbacks usually work23:18
anteayapleia2: I'm lost what is the point of zanata again?23:19
sdaguefungi: right23:19
pleia2anteaya: one of the options to replace transifex23:19
anteayamattoliverau: and thanks for your kind words :D23:19
sdagueor more clarkb right23:19
anteayapleia2: what other the other options23:19
pleia2anteaya: pootle is the other leader23:19
*** maxbit_ has joined #openstack-infra23:19
fungisdague: clarkb: but this doesn't affect icehouse->juno/master grenade?23:19
anteayapleia2: zanata sounds fraught with obstacles23:19
pleia2anteaya: the devs offered to help us, so I'm giving them a few chances :)23:20
anteayapleia2: are you also working on the pootle?23:20
sdaguefungi: no, because we -O in icehouse and juno devstack23:20
anteayapleia2: you generous soul, you23:20
pleia2anteaya: yeah, we've had a pootle dev server up since october23:20
sdaguethe fix did a clean cherry to icehouse23:20
anteayapleia2: ha ha ha, my decision would already be made23:20
sdagueand a merge conflict to havana23:20
*** maxbit has quit IRC23:20
*** praneshp has quit IRC23:20
sdaguebut I didn't pick up on why23:20
sdaguethis was why23:20
*** starmer has joined #openstack-infra23:21
*** maxbit has joined #openstack-infra23:21
sdaguefungi: and this was the road not take23:21
sdaguenot taken23:21
sdaguewe always defaulted through the other path23:21
sdaguepart of the issue with the fact that this code had 2 parallel paths23:21
sdagueand the one we wanted no one was running until today23:21
sdagueso ... it was wrong23:21
fungisdague: aha! got it. so is there some way we can decorate this to be removable come havana eol? otherwise there isn't a lot of indication to future generations why it's doing that23:22
sdaguefungi: ??23:22
sdaguedevstack has branches23:22
sdagueit just dies at some point23:22
*** praneshp has joined #openstack-infra23:22
fungisdague: yeah, but devstack-gate has no branches, so it's accommodating behavior in devstack stable/havana23:23
sdaguefungi: no it isn't23:23
* fungi is confused again then23:23
sdagueit's accomodating grenade generally23:23
sdagueand providing cached files in old and new23:23
sdagueinstead of just new23:23
fungiso were not testing grenade icehouse->master?23:23
sdagueand making old go to the network23:23
sdagueno, we went to the network23:23
sdaguebut icehouse was correct23:24
fungioh, or this is actually broken for icehouse->havana grenade too23:24
fungiright, now i get it23:24
*** radez_g0n3 is now known as radez23:24
fungithe master change merged because the network didn't hiccup23:24
sdagueand the icehouse change merged because the havana one hadn't yet23:25
sdagueso we went to the network the old way23:25
sdaguethen the havana change merged23:25
*** maxbit_ has quit IRC23:25
*** starmer has quit IRC23:25
fungiso we keep this mechanism around indefinitely. wfm23:25
sdaguebut we don't forward upgrade23:25
sdagueso we didn't see the break23:25
sdaguethis is actually a place where the forward grenade job would have potentially expoded it23:25
fungiseparately, havana devstack needs to curl -O i guess23:26
sdagueactually, no it would have 100% expoded it23:26
devanandaclarkb, sdague: looks like 97757 hit a neutron failure and got rechecked, and is now at the *bottom* of the queue again23:26
sdaguefungi: see -
clarkbdevananda: welcome to the jungle23:26
devanandaclarkb: \o/23:26
sdaguedevananda: right, this was part of the "all hands on deck" call. Because until we can get enough races out of openstack as a whole, this is going to keep happening.23:27
*** mrodden has quit IRC23:28
*** spiffxp has quit IRC23:28
*** SlickNik has quit IRC23:28
fungiianw: can i assume red hat ci failures for f20 and rhel7 on havana changes are expected?23:28
*** SlickNik has joined #openstack-infra23:29
*** miqui has quit IRC23:30
sdagueok, I'm going to get some dinner. My brain is pretty toast anyway. Will probably just work on some debuggability fixes for grenade in the morning so those fails will be a little easier to sort out.23:30
*** SlickNik has quit IRC23:30
ianwfungi: yeah, don't think i've fixed them up for that branch23:30
ianwwe don't see a lot of havana changes23:30
ianwredhatci is is a pretty sorry state all over until a few changes get merged ... i'm keeping an eye out here if i can help, let me know ...23:31
clarkbour devstack precise images are 2.6GB qcow223:31
*** ociuhandu has joined #openstack-infra23:32
*** marcoemorais has quit IRC23:33
*** marcoemorais has joined #openstack-infra23:33
fungiclarkb: sounds about right. bare-precise and bare-centos6 were larger according to glanceclient, if memory serves23:34
*** virmitio has quit IRC23:36
*** lcheng_ has joined #openstack-infra23:37
*** hemna_ has quit IRC23:37
*** aconrad has quit IRC23:38
clarkbfungi: yeah its all the cached data23:38
clarkbmordred so I mostly have it working now. I am trying to tighten up the errors cases and use set -e everywhere23:39
clarkbmordred: puppet poses a problem though because puppet will always fail on restarting iptables persistent and rsyslog23:39
clarkbbut in general we want a puppet failure to not result in a new image23:39
clarkbnot sure how to tackle that yet23:39
fungiclarkb: in what way does puppet fail restarting iptables? is this because it's calling the initscript with restart instead of start (since all that initscript does is reapply the rules files)?23:42
*** UtahDave has quit IRC23:42
*** mrodden has joined #openstack-infra23:43
anteayawould 2014-06-05 21:10:52.334 24090 ERROR oslo.messaging.notify.dispatcher [-] Exception during message handling cause check-tempest-dsvm-neutron to fail?23:46
clarkbfungi: it is because it runs in a chroot23:47
clarkbfungi: and iirc dib disables toggling of services23:47
clarkbI am sure there is a way around this and I will hop over to #tripleo to ask23:47
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module
*** markwash has quit IRC23:48
fungiclarkb: oh! this probably needs policy.d gymnastics then23:48
fungier, policy-rc.d23:49
* clarkb reads23:49
fungii can't recall if that made it into ubuntu, or if there's a newer more recommended solution to handling initscripts in chroots nowadays23:49
*** dprince has quit IRC23:49
*** starmer has joined #openstack-infra23:50
anteayawell I found a bug for it and have sent 95831,4 back via recheck, going for a walk now23:52
*** jaypipes has quit IRC23:52
pleia2anteaya: enjoy23:52
fungiclarkb: oh, though policy-rc.d will only end up being relevant for deb maintscripts and other things which call invoke-rc.d, but puppet very well may be doing less distro-specific things to call initscripts23:53
clarkbfungi: probabl23:54
fungiclarkb: an alternative... temporarily symlink all initscripts to bin/true during the build and then restore them after ;)23:54
*** mmaglana has quit IRC23:54
clarkbfungi: that is terrible but I might just do it23:55
* fungi has more terrible ideas if you run out23:55
*** zns has quit IRC23:55
*** starmer has quit IRC23:55
clarkbfungi: well I am tempted to have a global CHROOT flag23:55
clarkband puppet can be smart based on that23:55
fungii think the trick is going to be if puppet installs a package and then tries to manipulate the initscript for it directly or something23:56
clarkbits a bit clunky but at least contained to the source of the problem23:56
fungiwell, it won't apply inside third-party modules we're using though23:56
fungiso no idea if that will present a problem23:56
*** atiwari has quit IRC23:57
clarkboh good point23:57
clarkbeg mysql23:57
clarkbon the on devstack nodes23:57
fungiclarkb: dib really needs kvm instead of just a chroot, i'm betting, for this23:57
clarkbfungi: yes that is what I told lifeless and he said no23:57
clarkbbecause this is a specific instance of a generic problem23:57
clarkbunfortunately it is the only instance that bites us23:58
clarkband the rest of the generic bits don't matter23:58
lifelessfungi: so, debootstrap, same situation - don't run services when preparing operating systems23:58
lifelessfungi: run them when running operating systems23:58
fungilifeless: agreed. downside is we have to avoid puppet then because... not designed to actually care23:58
lifelessthe puppet scripts in question need to be taught the difference between prepare and operate23:59
clarkbfungi: we could use packer23:59
clarkbjust don't tell lifeless23:59
*** SlickNik has joined #openstack-infra23:59
clarkblifeless: so we can do that for our stuff, but not for the third party things23:59
clarkbmysql and postgres being the complicated ones23:59

Generated by 2.14.0 by Marius Gedminas - find it at!