Wednesday, 2014-12-17

mordredclarkb: if it just installs, then it's not vendored/tied, which means that because of how pip works, they can update independently00:00
mordredanteaya: then, also, can you do ".tox/py27/bin/pip --version"00:00
dstufftit looks like tox does specific a min version at least in it's default branch00:00
* anteaya does that00:00
*** sigmavirus24 has left #openstack-infra00:01
fungidstufft: agreed, that seems to be the case00:01
*** melwitt has quit IRC00:02
mordredanteaya: were you doing tox -epep8 ?00:02
fungii was going to look at the tox release notes online, but i get a _very_ odd error from
anteayano, just tox00:02
*** melwitt has joined #openstack-infra00:02
anteaya$ tox00:02
fungi"The OCSP server has no status for the certificate."00:02
anteayano arguments00:02
mordredanteaya: neat. try:00:02
mordredanteaya: then, also, can you do ".tox/pep8/bin/pip --version"00:02
* anteaya notes paste is holding up00:03
* anteaya checks dinner00:04
fungiaha, seems like they're missing a sufficiently long chain cert for that site00:04
mordredk. only thing different so far is pip version in the virtualenv00:04
*** andreaf has quit IRC00:05
anteayaso far00:05
* mordred trying to reproduce with that00:05
mordredoh -also, host system virtualenv is different version00:06
mordredwhich I believe is what's leading to different pip version00:06
dstufftyea virtualenv version controls pip version00:06
*** marun has quit IRC00:06
dstufftunless you have an old virtualenv, which downloaded stuff from the internet00:06
* mordred is testing to verify00:06
*** sputnik13 has quit IRC00:07
*** sputnik13 has joined #openstack-infra00:09
dstufftmordred: clarkb fungi so is pbr omiting the shas now?00:09
anteayapleia2: I was trying to test dhellmann's opml patch, what is a .tmpl file extention?00:09
* dstufft tries to catch up ons tatus00:09
anteayapleia2: the internets point to proprietary things or gaming uses00:09
*** erlon has quit IRC00:09
mordreddstufft: patch is going through the gate00:09
clarkbdstufft: it will as soon as thinbgs merge00:09
mordredas soon as that's done, we'll tag00:10
clarkbthere was a gate reset so we are about an hour away00:10
clarkbnodepool doesn't look happy00:10
*** sputnik13 has quit IRC00:10
clarkbwatch deleting the dns records will have done something silly to iptables00:10
mordredin case anyone is wondering, a great way to notice how many requirements something has is to install a new venv while tethered to a phone00:10
clarkbmordred: I just did that00:11
anteayaI was just wondering that00:11
heyonglianteaya, 3rd comment now should be collection to left colon automaticlly, am riht? or there is a special way to do this?00:11
anteayacan we set up a jenkins job for that00:11
dstufftmordred: pip 6 caches that shit by default00:11
anteayaheyongli: hello00:12
mordreddstufft: neat!00:12
dstufftpip 6 supremacy is what I'm trying to say00:12
mordredwell, one thingat a time00:12
anteayaheyongli: sorry I have no context for your comment00:12
anteayaheyongli: is this in relation to a patch?00:12
pleia2anteaya: I read it as "template" - not a vendor-specific thing00:12
clarkbhrm the deficit numbers don't look right in nodepool00:12
heyonglianteaya, no , i might not express my self claar, let me paste a result00:13
clarkbwell maybe things are ok there is a distinct spike on the graph00:13
anteayapleia2: perhaps I should check with doug00:13
anteayaheyongli: good idea00:13
clarkbit does think it knows about 635 nodes00:13
pleia2anteaya: if you search our repos, the uses .tmpl files too as templates00:13
mordredit's virtualenv00:13
clarkband only 33 nodes are ready so maybe we are ok00:13
pleia2anteaya: (so my brain didn't just make it up, yay!)00:13
anteayapleia2: ah okay, that is probably it then00:13
mordredjeblair, anteaya: upgrade virtualenv on your system and this will work00:13
anteayapleia2: yay for your brain00:13
clarkbI wonder if it just takes that long for zuul to catch up on the merge events00:13
fungianteaya: oh, btw jyuso got up with me this morning. the certs/keys you were concerned about are the ones generated for keystone in that devstack run. we publish them for all our job logs too, for example
mordredof course, upgrading tox should also cause virtualenv to get upgraded00:14
mordredso we _could_ put minversion=1.8.0 in the tox.ini00:14
mordredbut I kinda think that since it's not actually tox that we need 1.8 of we should not do that00:15
anteayafungi: thanks, glad I was the problem here with me not understanding and not them00:15
fungianteaya: no worries00:15
anteayapip install -U virtualenv ?00:15
mordredclarkb: I'm going to comment on that bug just so that next time we encounter this we know how to fix it00:15
mordredanteaya: yes00:15
clarkbmordred: thank you00:15
anteayamordred: thanks00:15
*** ZZelle_ has quit IRC00:16
heyonglianteaya, like the IBM powrKVM CI check, anything appear there is vote or  comments?00:16
anteayaI don't understand what I am looking at00:17
jeblair5591 set_description:jenkins01.openstack.org00:17
anteayawhere did this output come from heyongli00:17
jeblairclarkb: number of outstanding set_description jobs for jenkins0100:17
anteayais this jenkins output you copied from a comment on a patch?00:17
mordredjeblair: ok. I've updated the bug about this with the information about upgrading virtualenv00:17
jeblairmordred: what bug?00:17
uvirtbotLaunchpad bug 1393006 in devstack "testtools 1.2.0 with old unittest2 breaks discover --list" [Critical,Fix released]00:18
anteayaokay so I'm looking at this, what is the question heyongli00:18
*** sputnik13 has joined #openstack-infra00:18
anteayamordred: tox is running00:18
jeblairmordred: neat.  i don't see anything from you on that bug00:18
mordredjeblair: "Timeout error, please try again in a few minutes."00:18
heyonglianteaya, IBM PowerKVM CI checkDec 3, 2014 6:19 PM, this line is vote?00:18
mordredjeblair: ok. the comment saved this time00:19
jeblairmordred: why did tox 161 work for you?00:19
heyonglianteaya, i'm looking for what is look like if a CI just comments.00:19
mordredjeblair: because I had newer virtualenv00:19
mordredjeblair: downgrading tox did not downgrade virtualenv00:19
mordredclarkb: the thing I was trying to say before, is that there is a now-fixed behavior in old pip where if something was in setup_requires, it would download an egg to the current directory00:20
clarkbmordred: rather than install it properly?00:20
mordredclarkb: then, when install_requires processed it, it would think it had it, because it resolved it in the current dir00:20
mordredclarkb: BUT - that's not actually installed00:20
*** signed8bit is now known as signed8bit_ZZZzz00:20
mordredso when you go to run, it would be not found00:20
mordredthis is fixed in all the places now00:21
*** carl_baldwin has quit IRC00:21
mordreddstufft: I may know WAY too much about all o this00:21
clarkbjeblair: so that is what causes the queue graph to spike00:21
anteayaheyongli: okay now I understand00:21
mordreddstufft: I have no idea how you feel knowing much more :)00:21
anteayaat the bottom of that page, do you see a button called "Toggle CI"?00:21
jeblairclarkb: yeah, is it correlated with nodepool weirdness at all?  nodepool timeouts?  zuul timeouts?00:21
anteayaheyongli: in the lower left hand corner00:21
dstufftmordred: tecinhally that was a fix to setuptools not pip00:22
*** andreaf has joined #openstack-infra00:22
dstufftsetuptools 7 fixed that00:22
jeblairclarkb: i suspect that the _drops_ might be zuul timeouts and reconnects (so the queue gets restarted)00:22
mordreddstufft: ah - thanks good point00:22
clarkbjeblair: I think it is, if you look at the graphs on zuul status its a steady climb then it falls to zero when ya the reconnect happens00:22
mordreddstufft: but the virtualenv upgrade gets both, yeah?00:22
dstufftmordred: yea, we upgrade everything00:22
heyonglianteaya, lower left hand corner is voted CI, right?00:22
dstufftvirtualenv 12 will have the latest pip and setuptools when we release00:22
anteayaheyongli: Toggle CI00:22
jeblairclarkb: i'm going to see if i can figure out what's happening with zuul-jenkins; if you can investigate the potential relation to nodepool weirdness, that would be cool00:23
anteayathere is a button in your browser, or should be00:23
clarkbjeblair: ok00:23
dstufftmordred: and just gonna say, I am a poster child for making bad life decisions00:23
*** dpaterson has quit IRC00:23
heyonglianteaya, i see the Toggle CI, button00:24
clarkbjeblair: what was string to grep for zuul gearman reconnections? looking to narrow to timestamps00:24
anteayaheyongli: great00:24
anteayaheyongli: click it00:24
jeblairclarkb: lost due to disconnect00:24
*** amitgandhinz has joined #openstack-infra00:25
anteayaheyongli: that will show all the ci comments in the comments section00:25
anteayado you see the column called Verified in the top left corner?00:25
anteayain a table00:25
heyonglianteaya, i see them00:25
heyonglianteaya, i see ther is a Verified+100:26
*** dpaterson has joined #openstack-infra00:26
anteayaJenkins is -1 and three CI systems are +100:26
anteayaso find a comment from a CI system that doesn't have an entry in the verified column00:26
anteayathat CI system is just commenting00:26
*** dims has quit IRC00:27
heyonglianteaya, i found one, so any get vote right CI could post a Verified+1 column, if you don't have dont' do that. right?00:28
*** dims has joined #openstack-infra00:28
anteayaif you send a comment with a verified portion00:28
anteayaand you don't have voting rights00:28
jeblairclarkb: despite the large queue, job submission seems to still be quite fast.  so i don't think the fact a worker is stuck is causing delays and timeouts00:28
anteayayour comment will not be accepted00:28
anteayait won't be posted00:28
heyonglianteaya, now i know the answer, very helpful.00:29
anteayaglad you are happy00:29
*** amitgandhinz has quit IRC00:29
heyonglianteaya, does vote right need landing test cases to tempest or somewhere?00:29
clarkbjeblair: I am seeing the last reset happening at 1700 yesterday utc00:30
anteayavoting permissions (it isn't actually a right) are decided by the project you want to verify00:30
anteayaheyongli: so many things need to be in place00:30
clarkbjeblair: but the delay in building a ton of nodes is recent00:30
jeblairclarkb: probably before the zuul restart to pick up the newer timeout00:30
anteayaheyongli: and one of them is a stable history of commenting00:30
clarkbjeblair: yup00:30
*** bdpayne has quit IRC00:31
anteayaheyongli: are you testing your system againist the ci-sandbox repository?00:31
heyonglianteaya, i try to understanding the proccess, anything is help.00:31
*** bdpayne has joined #openstack-infra00:31
anteayaheyongli: you are doing a good job00:31
anteayastart by testing your system agains this repository00:32
heyonglianteaya, yeah, we had run our CI in real repo several weeks, but disable posting any comments to testing our CI.00:32
anteayaoh okay00:32
*** sputnik13 has quit IRC00:32
clarkbjeblair: however there are recent timeout errors in nodepool (not many likle before)00:32
*** dims has quit IRC00:32
clarkbjeblair: maybe geard doesn't list quickly when the numbers are large like that?00:32
heyonglianteaya, sandbox stage is not own by me, it should be long time ago00:32
anteayaheyongli: you can always use the sandbox00:32
anteayaif you think using it marks you as a beginner, think again00:33
clarkbjeblair: I can confirm manually running status against zuul from nodepool is slow00:33
anteayayou should disable your system and test against the sandbox repo anytime you make changes to your system00:33
clarkbjeblair: and it killed my connection00:33
anteayawe have had to disable people's systems because they were making changes and didn't disable their systems while they were doing so00:33
anteayaconsider it a good first step for anything new with your system00:34
heyonglianteaya, i get it.00:34
anteayagreat thank you00:34
*** teran has joined #openstack-infra00:34
heyonglianteaya, when the CI come back from sandbox to real repo?00:34
jeblairclarkb: so it seems like geard is stuck right now; so either it will get unstuck within 5 minutes, or zuul will reconnect (which may unstick it)00:34
anteayaheyongli: that can be a step00:34
anteayaheyongli: you can ask to comment on a repo and show your sandbox history as proof you are reliable00:35
*** sputnik13 has joined #openstack-infra00:35
anteayaheyongli: you are working with jogo for commenting on nova right?00:35
anteayaso you need to show jogo that he should allow you to comment on nova00:36
jeblairi'm restarting zuul + geard00:36
anteayathe best way is to show your comment history00:36
anteayafrom the sandbox00:36
*** emagana has quit IRC00:36
heyonglianteaya, i begin to get this.00:36
anteayaI'm so glad00:36
*** teran_ has quit IRC00:37
heyonglianteaya, even just comments also need approve, do you mean? by showing a stable comments log in the sandbox, right?00:37
anteayathat is the preferred way, yes00:37
anteayait isn't always what happens00:37
anteayabut it is what we do like to see00:37
anteayait shows you are paying attention00:37
heyongliand everytime you changing the CI, use sandbox again00:38
anteayaand want to be useful to the community00:38
anteayaheyongli: yes00:38
clarkbjeblair: ok00:38
anteayait means if something is funny with your system and you post 100 comments in 2 minutes00:38
heyonglidoes this process captured in anywhere?00:38
anteayayou just spam the sandbox00:38
anteayawell we do our best to capture it with
anteayabut we have various levels of implementation from the operators00:39
fungilooks like changes are getting reenqueued now00:39
*** zz_sabari is now known as sabari00:40
heyongli anteaya very thanks i'm reading this wiki, if anything i don't understand, i come back to boher you again -:)00:40
anteayaheyongli: you are asking good questions and you and jyuso are doing good work00:40
anteayaheyongli: it is a pleasure to work with you00:41
heyonglianteaya, thanks, i'm very glad to have you mentor us, really.00:41
anteayait is my pleasure00:41
anteayathere is a hope we might have something running for nfv by the end of this release00:42
anteayawhich is good news00:42
heyonglianteaya, we are tying to set the our CI for real HW, maybe fro NVF too.00:42
jeblairclarkb, fungi: i have started tcpdumps on zuul and jenkins0* to see if that yields more info00:43
*** lttrl has quit IRC00:43
*** sabari is now known as zz_sabari00:43
*** sputnik13 has quit IRC00:43
clarkbjeblair: ok, I can report nodepool is now trying to use all our quota so it is doing stuff00:43
heyonglianteaya, i'm going to office, then, see you.00:44
*** sputnik13 has joined #openstack-infra00:44
jeblairclarkb: well, hrm.  jenkins01 is already queuing set_description jobs00:44
anteayaheyongli: see you00:44
jeblairclarkb: i had surmised that there was another bug in geard, but perhaps the jenkins01 manager is really just stuck.00:44
clarkbjeblair: perhaps the worker thread in the gearman plugin there is in trouble? I will try finding it in melody00:44
*** Masahiro has joined #openstack-infra00:45
clarkbjeblair: ya00:45
fungijeblair: this sounds like a distinct possibility00:46
*** carl_baldwin has joined #openstack-infra00:46
fungiperhaps next we should try quiescing jenkins01 and restarting it?00:46
fungii'm happy to give that a roll00:46
clarkbjeblair: the manager thread is number two in cpu time on jenkins0100:47
clarkbit says it is runnable though00:48
*** unicell has joined #openstack-infra00:48
anteayaokay well tox finished that time, for anyone who cared00:49
*** bswartz has joined #openstack-infra00:51
*** ddieterly has joined #openstack-infra00:51
*** kumartin has joined #openstack-infra00:51
asselinhi, I'm still having issues with ip tables. I found the last two lines in rules.v4 are an issue. anything look wrong here?
jeblairclarkb, fungi: my math says that there shoud have been 73 jobs that it processed before getting stuck, but i don't see any packet captures on jenkins01 with that job. however, i may not have finished setting the tcpdump processing up by then00:52
fungiasselin: perhaps the loader assumes comments start with # and a space?00:53
fungiasselin: oh, you're saying when you uncomment them they cause a problem00:54
asselinfungi, yes00:54
*** Longgeek has joined #openstack-infra00:54
*** _sweston is now known as spw00:54
jeblairfungi: let's hold off on a jenkins restart just yet...00:55
fungiasselin: well for starters, the second one isn't -A00:55
jeblairfungi: i might want to restart zuul one more time and see if we see the same00:55
fungijeblair: sure00:55
asselinfungi, line 48?00:55
fungiasselin: 49, and it's using an undefined "stack-INPUT" chain00:56
*** salv-orlando has quit IRC00:56
asselinfungi, yes....weird. I'm seeing it correct in system-config. OK...I will trace it. thanks00:56
*** dannywilson has quit IRC00:56
jogoany further ideas about why is failing00:57
*** timcline has joined #openstack-infra00:57
fungiasselin: i would suggest you try feeding those entries to iptables and seeing what errors they give00:57
jeblairfungi, clarkb: yeah, i'm going to do another zuul restart to confirm this00:58
clarkbjeblair: ok00:58
*** rmcall has joined #openstack-infra00:58
*** Longgeek has quit IRC00:58
*** sabeen2 has quit IRC00:59
*** dims has joined #openstack-infra00:59
*** rmcall has quit IRC00:59
fungijogo: is it just failing on that 134052 change?01:00
*** sputnik13 has quit IRC01:00
jogofungi: no its failing on master01:00
jogoI can reproduce locally01:00
asselinfungi, ok I reapplied the puppet. I probably inadventently changed the file. Here's the new one. Line 24 is the offending line:
clarkbjeblair: though it isn't clear to me what you are trying to confirm01:01
clarkbjogo: what version of setuptools?01:01
jogoclarkb: setuptools.__version__01:01
jogoI am very confused about that01:01
jogoworks fine01:02
*** mtanino has quit IRC01:02
fungiasselin: i'm going to guess it's something to do with your rate limiting or logging. i'd need to compare to some working examples01:02
jeblairclarkb: whether the manager thread on jenkins01 is independently stuck and not picking up jobs, or whether the problem might lie on the geard side01:02
dstufft3.6 is what version of setuptools ships with the current version of virtualenv01:02
*** rmcall has joined #openstack-infra01:03
jogodstufft: ahh how do I get around that?01:03
jogothe nova tox env seems to not have that issue01:03
*** asettle is now known as asettle-lunch01:03
dstufftI'm not sure what your issue, if you just want a newer setuptools you just upgrade setuptools01:03
fungiasselin: though it seems to match what we have in openstack-infra/system-config:modules/openstack_project/manifests/single_use_slave.pp01:03
dstufftpip install --upgrade setuptools inside the virtualenv01:03
asselinfungi, yes
jogodstufft: I want that to happen automatically01:04
jogodstufft: it seems to happen like that in nova01:04
dstufftprobably they are doing pip install --upgrade <something> where <something> has setuptools in it's dependency tree somewhere01:04
jeblairclarkb, fungi: well, jenkins01 is now processing set_describe jobs.  so i think i'm back at "wait for it to happen again and see if the tcpdump catches it"01:05
fungiasselin: introduced by
*** mjturek has quit IRC01:05
fungijeblair: :/01:05
asselinfungi,  :(01:05
fungiasselin: that change merged to system-config ~5 days ago01:06
asselinfungi, yes, my images haven't been updating01:06
fungiasselin: what distro/release are you seeing this problem on?01:06
jeblairfungi, clarkb: geard status is being slow again01:06
asselinfungi, ubuntu 14.0401:06
jogodstufft: ahh makes sense. So should I add setuptools as a dep somewhere?01:07
asselinfungi, | 177 | p221     | dpc    |    | 1418324445 | e1c31014-d4ce-4edc-96f7-07b48269b240 | d0af8410-cec9-4961-b875-4661318bb489 | ready | 12401:07
jogodstufft: wonder why this just broke recently01:07
asselin124 hous is about 5 days01:07
dstufftjogo: what's broken01:07
*** rmcall has quit IRC01:07
jogodstufft: that fails on master too01:07
*** bdpayne has quit IRC01:07
fungiasselin: confirmed, we're broken on the same01:09
fungiasselin: checking now to see what the syntax error is there01:09
asselinfungi, thanks01:09
fungiasselin: if nothing obvious jumps out at me, i'll propose a revert in a few minutes01:09
asselinfungi, sounds reasonable01:09
jogodstufft: any ideas on what could have caused that breakage ?01:10
dstufftjogo: I am confused as to how this is related to setuptools version actually01:10
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Fix the log path returned from zuul
*** otter768 has joined #openstack-infra01:11
*** carl_baldwin has quit IRC01:11
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Ensure we set attrs in test_ids for sql2subunit
dstufftjogo: you say you can repro it with setuptools 3.6 but not 8.0?01:11
jogodstufft: correct01:12
jogodstufft: well not exactly01:12
dstufftjogo: can you repro it with setuptools 7.0?01:12
jogodstufft: let me double check01:12
*** markmcclain has quit IRC01:12
mordredjogo: can you reproduce locally?01:12
jogomordred: yes via the hacking pep8 venv01:12
jogodstufft: it fails for setuptools 8.0.4 as well01:13
mordredjogo: in that venv, can you "import nova.hacking.checks.factory" in python ?01:13
jogomordred: eventlet01:13
mordredjogo: WOOT!01:13
mordredjogo: I love it when it's not me01:14
mordredjogo: what did eventlet do? new release?01:14
jogomordred: no we import it in now01:14
* jogo wonders off to nova land to sort it out01:14
jogomordred dstufft thanks01:14
mordredjogo: have I ever mentioned that importing things in causes evil?01:16
*** otter768 has quit IRC01:16
jogomordred: I see why01:16
jogoit was sdague
greghaynesmordred: Quite an understatement ;)01:16
mordredof course, with eventlet, it's ACTUALLY probably the right thign to do01:17
mordredexcept when it's not01:17
jeblairmordred: why is importing something in evil?01:17
mordredjeblair: unexpected sideeffects01:18
jeblairmordred: that sounds backwards -- importing modules should not have side-effects.  but that's on the modules, not on the side of the importer01:18
greghaynesthey shouldnt, but they often do :(01:19
mordredyeah - sorry - what I meant was "running code that has sideeffects in is evil"01:19
clarkbjeblair: we are just watching it again?01:19
jeblairclarkb: yep01:19
jeblairclarkb: it's behaving :(01:19
jogomordred: do you think it makes more sense to leave nova/hacking/ there and revert01:19
jogoor move nova/hacking/ out of nova/*01:20
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Correct iptables log option order
fungiasselin: jeblair: mordred: clarkb: ^ critical fix for nodepool image building01:21
mordredjogo: neither01:21
fungithat was... not exactly easy to track down01:21
asselinfungi, will test this quickly now01:21
mordredjogo: honestly, you'll unwind so much to try to figure all that out01:21
clarkbfungi: looking01:21
*** camunoz has quit IRC01:22
asselinfungi, +101:22
jogomordred: what do you propose?01:22
clarkbfungi: is that new?01:22
fungiwe're basically 5 days behind on nodepool images (ever since the change adding that line merged)01:23
clarkb(the rule not iptables being funny)01:23
mordredjogo: Id' just figure out how to bypass the import when we're running things - like, maybe inject something somewhere in hacking so taht when it goes to import things nova can detect it and not do eventlet01:23
mordredjogo: OR - don't listen to me - I may be wrong01:23
dstufftyou could probably mock eventlet in hacking01:23
jogomordred: that will break local checks in ova01:23
jogodstufft: this is a nova problem not a universal one01:24
jogomordred: nova.hacking.checks.factory is what registers local checks01:24
mordredjogo: so what about eventlet is breaking hacking - do we just not have hacking in the venv?01:24
dstufftnova_hacking :V01:24
jogomordred: as part of the hacking tests, we build  tox venv on master01:25
*** bdpayne has joined #openstack-infra01:25
jogoand run flake8 on other repos inside of that01:25
jogoto see what trunk hacking will do01:25
jeblairdstufft: i think the _second_ time we split out nova's hacking checks from the repo, the cycle is complete.  :)01:25
mordredjogo: OH01:25
jogothat venv doesn't have eventlet so goes kapow01:25
mordredjogo: right. I think you're going to need a venv with eventlet in it01:25
jogojeblair: ^_^01:25
jogomordred: I guess we can make that test a bit more robust and install the target repos requirements01:26
mordredwell, that'll get expensive quickly01:26
mordredfor now, I'd just add eventlet01:26
mordredit's pervasive across openstack and it does weird things on import01:26
jogomordred: with devpi-server it is quick for me01:27
jogomordred: that is reasonable01:27
jeblairclarkb, fungi: i'm going to leave this for a bit; i'll check in a couple of hours and see if i can stop the packet captures01:28
clarkbjeblair: ok sounds good01:29
clarkbI need to check out now and start thinking about heading to dinner01:29
clarkbdinner with family all the way from micronesia01:30
anteayaclarkb: enjoy01:30
clarkbmordred: in theory pbr changes will merge in ~45 minutes, if we do manage to get them tagged this evening we should push a revert of setuptools pins to devstack to test if it all works01:30
clarkbfungi: ^01:30
fungiclarkb: _after_ tagging pbr 0.10.3 on the feature/0.10 branch, yeah?01:31
dstufftthen I get to break it all over with pip 601:32
fungidstufft: i for one welcome our new meme overlords01:33
clarkbfungi correct and bandersnatch syncing01:33
mordredclarkb: ++01:33
fungiclarkb: oh, yeah, i'll make sure to double-check the bandersnatch mirror contents for latest pbr at that point01:33
dstuffthas that been working better?01:33
fungimordred: will you be around in 45-60 minutes to tag new pbr?01:33
*** nelsnelson has quit IRC01:34
mordredfungi, clarkb: I'm going to push the tag-merge as a real merge btw - so I'll get that taken care of01:34
fungidstufft: i think so. it's only been a couple days since upgrading and the pypi-side race fix, so time will tell01:34
mordredfungi: yes01:34
openstackgerritAnita Kuno proposed openstack/requirements: Revert "Add a tox env to check requirements overlap"
fungimordred: excellent--i'll leave the release end in your capable hands--thanks!01:34
mordreddstufft: btw: next time you get bored ...01:35
openstackgerritAnita Kuno proposed openstack/requirements: Revert "Add a tox env to check requirements overlap"
fungimordred: if you use a real merge into master, then you'll presumably be dropping the git sha/egg-info change you have proposed for master at that point?01:35
mordredfungi: yes01:35
dstufftmordred: I'm not core on bandersnatch01:36
fungidstufft: oh, yeah, that pr. it would be awesome so that we can have pbr write to afs01:36
*** emagana has joined #openstack-infra01:36
fungidstufft: you are our chief lobbyist ;)01:36
mordreddstufft: I know - but you know everyone and they like you01:36
openstackgerritJoe Gordon proposed openstack-dev/hacking: Add eventlet to test-requirements
jogomordred: there ^01:36
dstufftok :)01:37
dstufftI can review it at least01:37
dstufftand add my thoughts01:37
fungithoughts like... wtf is afs?01:37
mordredjogo: that just makes me sad01:38
dstufftfungi: I (sadly?) do know what AFS is01:38
jogomordred: yeah, but like you said that is probably the easiest fix01:38
fungidstufft: be glad and rejoice, for afs is with us always01:39
mordredjogo: totes. I'm +2. Im just a sad +201:39
*** kumartin has quit IRC01:39
jogomordred: haha01:39
*** timcline has quit IRC01:40
*** carl_baldwin has joined #openstack-infra01:42
fungimordred: speaking of afs, any insights on what else i'll need in ? (i know i'll need to give the afsdb servers an acl allowing gerrit2 on review-dev to write into that subtree, but what else?)01:42
*** melwitt has quit IRC01:44
fungimordred: the idea is that once that works, i'll do the same to review.o.o and then mount the corresponding read-only volume on the git servers01:44
*** sarob has quit IRC01:45
*** bdpayne has quit IRC01:46
*** wenlock has joined #openstack-infra01:48
*** emagana has quit IRC01:48
anteayaoh I forgot, off to an appointment tomorrow morning, should be back in the afternoon01:50
*** jaypipes has quit IRC01:52
*** yaguang has joined #openstack-infra01:53
mordredfungi: I mean, it LOOKS about right, but it seems not right enough because it's not super complicated01:53
fungimordred: i just realized i probably need to use the read-write path to the mirror there, for starters01:53
mordredoh. right01:54
*** carl_baldwin has quit IRC01:54
jeblairfungi: will we be vos releasing this mirror?01:54
fungijeblair: i don't think that's probably necessary01:55
*** dims has quit IRC01:55
fungijeblair: it's getting written to basically continuously by gerrit as changes happen to git repos01:55
*** dims has joined #openstack-infra01:56
jeblairfungi: then if we're not going to have a read-only version, then the only path available will be the rw one, so you should be set01:56
fungijeblair: good point. the git servers don't need read-write access to that volume but that's just a matter of acls01:56
*** salv-orlando has joined #openstack-infra01:57
jeblairdstufft: was generated with that patch (and running out of afs behind an apache with those rewrite rules).  it's not running continuously though (yet), so hasn't been updated since.01:57
jeblairfungi: yep01:57
*** gyee has quit IRC01:57
*** yaguang has quit IRC01:59
jeblairfungi: git-sandbox should probably be its own volume though.  so i think your manual steps are create the volume, create a non-interactive pts entry, grant it acls02:00
*** dims has quit IRC02:00
jeblairfungi: probably the big todo that's missing from puppet is setting gerrit up to start inside of k5start once you have the kerberos principal for it02:00
fungijeblair: ooh, great point. we need something to kinit for it02:01
dstufftoh hey02:01
dstufftpbr thing merged02:01
fungidstufft: one to go, ~9 minutes eta02:01
dstufftI got some email02:01
jeblairfungi: oh, yeah, so normally i'd create a principal and set out a keytab for k5start.  i think that's a binary file.  so i'm not sure what we'll end up putting in hiera.02:01
*** salv-orlando has quit IRC02:01
dstufftI don't actually know how gerrit works :D02:01
dstufftI just pretend02:02
fungidstufft: the first change adds the egg-info entry, the second stops sticking it in the version string02:02
dstufftfungi: ah02:02
fungidstufft: separated so as to make them easier to discuss and revert if needed02:02
jogomordred: why is pbr in the integrated gate?02:02
fungijeblair: sounds like we'll need some sort of keytab manager in puppet?02:03
jogoisn't that a little odd02:03
mordredjogo: because if we break pbr we break EVERYTHING02:03
mordredjogo: we don't do that as often anymore - but back in the day it was very important02:03
fungijogo: because it curently runs gate-requirements-integration-dsvm02:03
jogotrue, but shouldn't check jobs cover that?02:04
jeblairfungi: maybe there's a way we could encode it into hiera02:04
fungijeblair: raw hex entries?02:04
fungii see some ideas in there02:05
fungijeblair: looks like they're using base6402:05
*** wenlock has quit IRC02:06
*** shashankhegde has quit IRC02:07
heyonglijogo,  does the pci test cases landing on nova need further wok?:
jogofungi: what does that job do?02:08
fungijogo: it iterates over global requirements and tries to install everything using proposed pbr changes02:08
*** yjiang5 is now known as yjiang5_away02:09
fungijogo: runs
jogofungi: ahh and that is used to gate the requirements job02:09
jogoerr repo02:09
fungijogo: but also used to gate that pbr patches don't render things uninstallable02:09
jogowhich has swift-dsvm-functional  so it is in the integrated gate?02:10
*** dpaterson has quit IRC02:10
jogoand that is run in places with standard tempest-full gate jobs02:10
*** otter768 has joined #openstack-infra02:11
*** yaguang has joined #openstack-infra02:12
*** camunoz has joined #openstack-infra02:12
jogoheyongli: I will have to do a more thorough review in the near future02:12
jogoheyongli: hopefully this isn't blocking you though right?02:13
jogoyou can still work on the tests and start setting up the CI system right?02:13
heyonglijogo, no , it won't, land to nova i good, but i think the third party CI i worked on could be continue without that.02:13
jogoheyongli: agreed02:14
heyonglijogo,  landing to nova is good for sharing and qulity control, so i hope it will happen02:14
fungidstufft: mordred: the second pbr change merged, so i think we're set for a 0.10.3 tag to get pushed02:14
dstufftthen setuptools gets unpinned?02:14
jogoheyongli: if there is lack of consensus about landing them in nova, posting the tests on stackforge isn't a bad option as well02:14
fungidstufft: yep, sounds like clarkb wanted to try that after dinner maybe02:14
*** sweston has quit IRC02:15
*** spw is now known as sweston02:15
heyonglijogo,  ok, that's also make sense,  let's try at least, and tanks you always kind to make it here today.02:15
jogoheyongli: no problem, thanks for working on these tests02:16
heyonglijogo, that's my job, i like it.02:16
clarkbanyone can propose the revert to unpin02:18
clarkbI will try to do it after dinner if not already done02:18
*** ChuckC has quit IRC02:18
openstackgerritAdrian Turjak proposed openstack-infra/project-config: Add new Openstack-Pydashie project
fungii can propose the revert as soon as i see a pbr 0.10.3 on all our mirrors02:20
*** bdpayne has joined #openstack-infra02:21
heyonglijogo, you might intresting to some pci  design,  do you have time to look this sriov interface attach bp? :
jogoheyongli: sorry not tonight02:22
heyonglijogo,  not that rush,  but thanks, that's will be helpfull02:22
*** adalbas has quit IRC02:22
*** MarkAtwood has joined #openstack-infra02:24
*** gokrokve has joined #openstack-infra02:24
*** sarob has joined #openstack-infra02:26
*** sarob has quit IRC02:31
*** wenlock has joined #openstack-infra02:35
*** bdpayne has quit IRC02:35
mordredfungi: ok - pushing tag02:39
*** carl_baldwin has joined #openstack-infra02:40
mordredfungi: done02:40
*** emagana has joined #openstack-infra02:43
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: WIP: Add a db api method to extract run_times for a single test
*** emagana has quit IRC02:48
openstackgerritJoe Gordon proposed openstack-infra/elastic-recheck: Add fingerprint fur bug 1403284
uvirtbotLaunchpad bug 1403284 in devstack "mysql (1040, 'Too many connections')" [Undecided,New]
openstackgerritMerged openstack-infra/system-config: Correct iptables log option order
*** zz_dimtruck is now known as dimtruck02:50
*** carl_baldwin has quit IRC02:51
openstackgerritMerged openstack-dev/hacking: Add eventlet to test-requirements
fungiit's on now02:54
fungioddly, it triggered both the release _and_ pre-release pipelines?!?
clarkboh crap02:55
clarkbI know why02:55
clarkbmy regex for prerelease puts ? on the abc group02:56
*** hdd has quit IRC02:56
clarkbshould just remove the outer ()?02:56
fungiyeah, we need to make that non-optional02:56
fungisorry i missed that in review :/02:56
clarkbI wrote it. I fyou push fix I can approve02:56
fungino harm done i think. looks like stuff released successfully anyway02:56
fungiwill do--just a moment02:57
clarkboh good. maybe check sdist and whl install? as they may have overwritteb each other?02:57
clarkbI am suddenly paranoid I derped hard02:58
*** carl_baldwin has joined #openstack-infra03:00
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Pre-release tags must include non-digits
fungiclarkb: ^03:01
fungichecking now03:01
clarkbI am approving 142313 now03:03
*** ChuckC has joined #openstack-infra03:03
*** timcline has joined #openstack-infra03:04
*** timcline has quit IRC03:04
fungithe wheel on tarballs.o.o is fine03:04
fungias is the tarball03:04
*** timcline has joined #openstack-infra03:04
fungii'm testing the ones on pypi as well03:05
funginice that pbr is self-built too03:06
fungiso pbr freeze in my venv is reporting "pbr==0.10.3  # git sha e73e67a"03:07
*** timcline has quit IRC03:08
fungicrappity. we only got a wheel upload to pypi.python.org03:08
fungii'll retrigger the upload03:08
*** sandywalsh has quit IRC03:09
*** bdpayne has joined #openstack-infra03:09
*** timcline has joined #openstack-infra03:10
*** sandywalsh has joined #openstack-infra03:11
fungiyeah, it raced with the wheel upload from the pre-release pipeline...
fungiHTTPError: 400 Client Error: A file named "pbr-0.10.3-py2.py3-none-any.whl" already exists for  pbr-0.10.3. To fix problems with that file you should create a new release.03:11
clarkbgood that pypi throws an appropriate error03:12
dstuffteventually I plan on making it so you can't delete a file off PyPI and re-upload a new one too03:12
dstufftright now you can03:12
*** dannywilson has joined #openstack-infra03:12
fungiyeah, in this case i think i'll leave it and manually upload the missing tarball03:12
*** Krast has quit IRC03:13
fungirather than try retriggering that job03:13
fungisince that would involve deleting the existing wheel which is not a great habit to get into03:13
openstackgerritMerged openstack-infra/project-config: Pre-release tags must include non-digits
*** timcline has quit IRC03:14
*** timcline has joined #openstack-infra03:15
*** timcline has quit IRC03:17
*** timcline has joined #openstack-infra03:17
*** changbl has joined #openstack-infra03:17
fungiokay, is updated now to include the tarball, and the md5sum matches the one computed by the tarball job03:22
fungiand i've tested it with 'pip install --no-use-wheel pbr' in a venv, all good03:23
fungipolling our mirrors03:23
*** emagana has joined #openstack-infra03:24
*** dannywilson has quit IRC03:24
fungithey're all up to date with wheels and tarballs of the new version03:25
fungisetuptools pin revert on the way in moments03:25
*** dannywilson has joined #openstack-infra03:25
*** harlowja is now known as harlowja_away03:25
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Pin version of setuptools"
fungiclarkb: dstufft: ^03:28
*** timcline has quit IRC03:28
*** dannywilson has quit IRC03:29
clarkbfungi is there a devstack change?03:31
fungiclarkb: i was looking to see if it got pinned anywhere besides our nodepool images03:31
*** sarob has joined #openstack-infra03:32
fungidstufft: ^03:33
*** baoli has quit IRC03:35
fungianyway, that will need to run tests for a while. i may turn in for the night03:35
*** asettle-lunch is now known as asettle03:35
openstackgerritMonty Taylor proposed openstack-dev/pbr: Merge git sha removal
*** Alexandra_ has joined #openstack-infra03:36
*** timcline has joined #openstack-infra03:37
*** sarob has quit IRC03:37
mordredclarkb, fungi: ok, I _think_ that's right - but we should probably hammer on that merge a little better03:38
*** timcline has quit IRC03:38
*** ryanpetrello has quit IRC03:39
*** asettle has quit IRC03:39
*** ivar-lazzaro has quit IRC03:39
*** Alexandra_ is now known as asettle03:40
*** ddieterly has quit IRC03:42
*** jerryz has joined #openstack-infra03:42
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for neutron bug 1403291
uvirtbotLaunchpad bug 1403291 in neutron "test_server_connectivity_pause_unpause fails with "AssertionError: False is not true : Timed out waiting for to become reachable"" [Undecided,New]
openstackgerritMerged openstack/requirements: Add tooz to project list to update
*** carl_baldwin has quit IRC03:47
*** doug-fish1 has left #openstack-infra03:48
*** carl_baldwin has joined #openstack-infra03:48
*** unicell1 has joined #openstack-infra03:48
*** MarkAtwood has quit IRC03:48
*** Ryan_Lane has quit IRC03:49
*** unicell has quit IRC03:50
*** gokrokve has quit IRC03:50
openstackgerritMonty Taylor proposed openstack-infra/project-config: Move to pbr freeze from pip freeze
*** sarob has joined #openstack-infra03:54
*** dkranz has quit IRC03:56
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Install python-magic on devstack slaves
*** zz_sabari is now known as sabari03:59
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Ensure python-magic is installed on all slaves
jheskethIf any cores are still around ^ should fix the devstack gate from failing... however I'm not sure where the best place is to put dependencies for devstack04:02
jhesketh^ being 14232704:02
*** dimtruck is now known as zz_dimtruck04:03
openstackgerritTony Breeds proposed openstack-infra/reviewstats: Sync with current core group
*** wenlock has quit IRC04:07
*** patrickeast has quit IRC04:09
*** salv-orlando has joined #openstack-infra04:09
*** unicell1 is now known as unicell04:10
*** unicell has joined #openstack-infra04:10
*** esker has joined #openstack-infra04:11
openstackgerritTony Breeds proposed openstack-infra/reviewstats: Sync with current core group
dstufftfungi: does the fact that is passing tests mean it worked?04:13
mordreddstufft: not yet - that drives the beginning of image creation04:14
fungidstufft: mordred: but 142322 does04:15
mordredjhesketh: looking04:15
fungi(the devstack revert)04:15
dstufftfungi: ok, that ones not done yet I guess04:15
dstufftwill watch :D04:15
mordredfungi: do we want to change pip freeze to pbr freeze in devstack? did I push one of those up already?04:15
fungimordred: i don't recall, and i've reached the point where i'll have to pick it back up tomorrow04:16
mordredI'm actually wondering if we use the freeze stuff in devstack itself04:17
mordredthere it is. devstack-gate04:17
dstufftnon voting failures on 142322 looking at status.o.o04:18
*** ChuckC_ has joined #openstack-infra04:19
*** ChuckC has quit IRC04:20
dstufftswift :(04:20
mordreddstufft: hrm. why is that 2.2.1c1.6 I wonder ... oh, swift uses post versioning04:21
mordredand that may be post versioning after a pre-release tag04:22
mordredthat's not great04:22
dstufftso that would work if they had used -6 or .post604:22
mordredthey didn't use anything04:22
mordredthey tagged something as 2.2.1rc104:23
mordredand since then, there have been 6 commits04:23
dstufftoh, and pbr jsut appends a .6 to it?04:23
mordredit _used_ to make it 2.2.1c1.6.g$sha04:24
mordredalso, I _think_ the lifeless patches make this bit better04:24
dstufftthe swift thing wasn't an error, though it's going to trigger the < 0 because we can't parse it behavior04:25
dstufftthe error that made them fail was pytz04:25
dstufftpytz 2012d was installed04:25
mordredour favorite04:25
dstufftand babel has a >=0a04:25
mordredyah. that's our way to get it to install something04:26
mordredsince they have bad versions04:26
dstufftthey don't anymore!04:26
dstufftbut the old ones are bad04:26
mordredwe should fix that then - to my knowledge we don't need the old versions for any reason04:26
*** carl_baldwin has quit IRC04:26
dstufftbut they switched from YYYYletter to YYYYY.N04:26
dstuffta = 1, b = 2, etc04:26
mordredbut we should be able to do >=2012.404:27
dstufft2013.6  is the first version they used the YYYY.N versioning on04:27
*** zz_dimtruck is now known as dimtruck04:28
dstufftbut >=2012.4 will match that04:28
mordredok. so the new pbr master fixes the swift version04:28
dstufftanother problem -> some of the tests shell out and run some python code and then assert against the output04:29
dstufftand since ubuntu has some wierdo versioned stuff installed, setuptools warns about it04:29
dstufftmessing up the output04:29
dstufftalso the swift version04:29
mordredwell, those versions are going to be aroudn until the end of time04:30
*** stevemar has quit IRC04:30
dstufftit's a python warning though04:30
mordredI'm just saying04:30
dstufftso you can suppress the warnings using an envvar04:30
mordredwhat's the var?04:30
mordred(also, this is going to make python on ubuntu/debian very ugly04:30
mordredbecause those aren't going to be going away04:30
*** sabari is now known as zz_sabari04:30
dstufftso this is the warning we added to warn people about things being parsed as non PEP 44004:31
dstufftwe could remove that warning04:31
dstufftnot sure what's better04:31
mordredat some point are they going to be errors?04:31
mordredI suppose if you're managing your python with dpkg, you don't care about the setuptools version math, so it's not a big deal to suppress there04:32
mordredin that case04:32
dstufftno they'll never be errors04:32
mordredthe mixed some-system/some-pip is gonna be ugly04:32
mordredbecause suppressing could suppress knowledge of a possible issue04:32
dstufftor well, in the foreseable future it won't be errors04:32
dstufftmaybe someday we'll feel good enough about PEP 440 adoption to do that, but I don't forsee it04:33
mordredI would recommend not making it errors unless you want python to stop being usable on ubuntu04:33
mordredubuntu and debian _cannot_ adopt pep44004:33
*** hdd has joined #openstack-infra04:34
mordredbut ... that's certainly not my problem right now04:34
dstufftmordred: they can certainly put PEP440 versions inside the python metadata04:34
mordredbut then they're lying04:34
mordredoh - unless you mean translating04:34
dstufftwhy is it lying to do a mechanical transformation04:34
mordredsorry, I mean trying to figure out how to get 1.2.0~0ubuntu1 into PEP44004:35
dstufftwouldn't that be 1.2.0-0ubuntu104:36
mordredoh, yeah04:36
mordredyes, what you said04:36
*** ryanpetrello has joined #openstack-infra04:36
mordredk. then maybe it'll all be happy04:36
mordredhow do they get that into the metadata?04:36
mordredyou know what - nevermind - I have other fish04:37
mordreddstufft: 2.2.1.dev3 ...04:37
dstufftright now by patching the, but a future effort is going to be talking to downstream and asking them how they would prefer to inject it04:37
mordreddstufft: that's a dev version that's before 2.2.1, right?04:37
dstufftmordred: yes04:37
mordreddstufft: cool. thanks04:37
dstufftI think04:38
dstufftmaybe we should subclass RuntimeWArning and make a PEP440Warning04:38
mordredcool. I have learned something04:38
dstufftthat describes the -W command line option, anything you pass tot hat can be passed via PYTHONWARNINGS (comma seperated for multiple -W values)04:39
*** Masahiro has quit IRC04:39
*** Ryan_Lane has joined #openstack-infra04:39
*** Masahiro has joined #openstack-infra04:40
*** emagana has quit IRC04:40
dstufft(0.1.21-nmu2ubuntu2 is the wierdo ubuntu version that is causing output, so that's directly translatable to PEP 440 at least, and they must be injecting that somehow)04:40
mordredwhat's that a package of again?04:40
dstufftpython-debian, so maybe it's something they maintain themselves04:41
*** carl_baldwin has joined #openstack-infra04:42
dstufftmordred: looks like all the current errors will be fixed by fixing the swift version thing and turning off the warnings04:42
mordredwow. it's awesoe to see how they're doing that04:42
*** hdd_ has joined #openstack-infra04:42
dstufftoh and pytz04:43
*** hdd has quit IRC04:43
dstufftthose 3 times04:43
*** dimtruck is now known as zz_dimtruck04:43
mordreddstufft: where were you seeing the pytz problem04:44
*** andreaf has quit IRC04:45
mordredyup. I can't fix those04:46
mordredthey're fedora04:46
mordredthey're installed from OS pacakges04:46
mordredpytz and Babel04:46
*** andreaf has joined #openstack-infra04:46
mordredpytz.noarch 0:2012d-5.fc2004:46
dstufftof course04:48
dstufftmordred: what's the point of that test, can it just pip install --upgrade pytz? :V04:48
mordreddstufft: that's installing openstack04:48
mordredthere are a set of things that devstack installs from distro packages04:49
dstufftwhy does it install some things from a distro package04:50
mordredhistory. always has. there was a point in time where we tried to install everything we could from distro packages and only install from pip if we had no other choice04:50
mordredthat's still largely what happens there04:50
mordredbut the situation is unavoidable to a degree - there are python packages that are required by very basic os things - cloud-init for instance, requires python-requests04:51
mordredwhich means it's going to get pulled in from distro04:51
dstufftwell, regardless, I should probably go bother fedora04:52
mordredyup. although that's fedora 2004:52
mordredwhich isn't gonna get fixed04:52
dstufftoh, they don't fix fedora 20?04:52
mordrednot like this they don't04:53
mordreddistros only patch already released things with things liek security updates04:53
mordredcertainly not behavioral changes04:53
mordredand not changes in how they might package things in the future04:53
dstufftI've had distros patch bugs04:54
mordredsdague, dtroyer: ^^ we may need to think MUCH harder about a global venv for devstack04:54
mordredwe may be getting to a spot where using newer setuptools with _any_ python from distros may get very very broken04:54
*** rediskin has quit IRC04:55
*** dmitryme has quit IRC04:55
*** otter768 has quit IRC04:55
*** timrc has quit IRC04:55
*** salv-orlando has quit IRC04:55
mordredsdague: also,
mordredsdague: I know you abandonded it04:56
*** dmitryme has joined #openstack-infra04:56
mordredbut seriously ...04:56
mordreddstufft: that patch ^^ was my last attept to get devstack to stop installing python libraries from distro packages04:57
*** zz_sabari is now known as sabari04:57
*** esker has quit IRC04:58
*** Shrews has quit IRC04:59
*** timrc has joined #openstack-infra05:00
*** Ryan_Lane has quit IRC05:01
*** sarob has quit IRC05:01
*** koolhead17 has joined #openstack-infra05:02
*** koolhead17 has joined #openstack-infra05:02
dstufftmordred: so to move things forward with pytz, do we need to get something like 134732 in, or can we switch debstack to install pytz with pip05:03
*** weshay has quit IRC05:03
mordredwell, it gets pulled in from Babel05:04
dstuffts/Babel/pytz then05:04
mordredwe can probably land that change, yes - although that's going to be a longer term battle to deal with05:04
mordredand I think we'll need to come up with a strategy for dealing with it05:04
*** sarob has joined #openstack-infra05:05
*** camunoz has quit IRC05:05
*** sarob has quit IRC05:05
*** camunoz has joined #openstack-infra05:05
*** talluri has joined #openstack-infra05:06
*** Ryan_Lane has joined #openstack-infra05:06
*** rediskin has joined #openstack-infra05:07
*** carl_baldwin has quit IRC05:09
*** sarob has joined #openstack-infra05:09
*** Ryan_Lane has quit IRC05:10
*** melwitt has joined #openstack-infra05:11
dstufftmordred: so if I udnerstand correctly, horizon already installs babel via pip05:12
*** carl_baldwin has joined #openstack-infra05:14
*** carl_baldwin has quit IRC05:15
mordredBUT - if babel is already installed via rpm05:16
mordredand -U isn't given05:16
dstufftah right05:16
dstufftassuming it matches the version spec05:17
dstufftnot sure waht the version spec is05:17
mordredI believe it's unbounded05:17
mordredoh, really?05:17
mordredoh - for babel05:17
mordredyeah, >=1.305:17
*** hdd_ has quit IRC05:19
*** ryanpetrello has quit IRC05:22
dstufftmordred: oh, couldn't global-requirements just get pytz>=2013.605:24
dstufft(that's the first version that was PEP 440 compat)05:24
mordredyah, I dont think that would be a problem05:25
*** Sukhdev has joined #openstack-infra05:28
*** yaguang has quit IRC05:29
openstackgerritDonald Stufft proposed openstack/requirements: Ensure we have a PEP 440 compatible version of pytz
dstufftmordred: made a pr ^05:33
clarkbhow do we fix the swift thibg?05:35
clarkbwill devN work?05:35
clarkbso 1.2.3c1.dev605:35
dstufftthat is an acceptable PEP 440 version, but it it sorts before 1.2.3c105:36
dstufftas older05:36
mordredclarkb: it's fixed in the lifeless branch05:37
mordredwe could also add "post" before the auto revno, right?05:37
mordredlike 1.2.3c1.post6 is valid?05:37
clarkbbut 1.2.3c1.6 is not valid...05:38
dstufft.post6 is valid yes05:38
dstufftand sorts after05:38
clarkbok so further improvement needed but doable05:38
dstufftI think lifeless branch does 1.2.3.c2.dev605:39
dstuffter, minus one ot05:40
clarkbfor when your visit mars as atourist05:41
mordredlifeless branch actually does 1.2.2.dev305:41
mordredor 1.2.3.dev3 I mean05:41
mordredbecause it gets much fancier with calculation05:41
dstufftmordred: oh05:42
mordredclarkb: I think doing the postX might be a good short term05:42
clarkbmordred ya I think the non lifeless change to do that is easy05:42
clarkbsince you already switch on post vs pre05:43
*** sarob has quit IRC05:44
*** yaguang has joined #openstack-infra05:46
dstufftI gave my +1 for what little it's worth ;)05:47
*** emagana has joined #openstack-infra05:51
*** emagana has quit IRC05:55
*** salv-orlando has joined #openstack-infra05:56
*** achanda has joined #openstack-infra06:01
jeblairclarkb, fungi: is queueing again; i stopped the tcpdumps, will examine tomorrow06:01
*** salv-orlando has quit IRC06:01
*** hdd has joined #openstack-infra06:03
*** harlowja_at_home has joined #openstack-infra06:06
*** sabari is now known as zz_sabari06:12
harlowja_at_homehave u guys seen anything like 'NameError: name 'StandardError' is not defined' before; or recently?06:15
harlowja_at_homei'm seeing that from
harlowja_at_homewhich seems to be coming from sphinx :-/06:16
*** koolhead17 has quit IRC06:16
harlowja_at_homebunch of other examples @
harlowja_at_homeall py33/py34...06:17
*** vigneshvar has joined #openstack-infra06:17
*** asettle has quit IRC06:17
harlowja_at_homeseems to be happening to most anyone (from a sample of 2 reviews) with py33/py3406:18
*** zz_sabari is now known as sabari06:19
*** yfried has joined #openstack-infra06:19
*** dannywilson has joined #openstack-infra06:26
*** harlowja_at_home has quit IRC06:26
*** dannywilson has quit IRC06:30
*** avozza is now known as zz_avozza06:32
*** dims has joined #openstack-infra06:34
*** dims has quit IRC06:39
*** achanda has quit IRC06:40
*** achanda has joined #openstack-infra06:41
*** hdd has quit IRC06:41
*** rlandy has joined #openstack-infra06:42
*** reed has quit IRC06:44
*** achanda_ has joined #openstack-infra06:45
*** achanda has quit IRC06:45
*** garyk has joined #openstack-infra06:47
*** achanda_ has quit IRC06:48
*** yamamoto has quit IRC06:51
*** jgallard_ has joined #openstack-infra06:52
*** otter768 has joined #openstack-infra06:56
*** Shrews has joined #openstack-infra06:59
*** sputnik13 has joined #openstack-infra07:00
*** otter768 has quit IRC07:01
*** liam_ has joined #openstack-infra07:06
*** liam_ is now known as Guest7249407:06
*** ryanpetrello has joined #openstack-infra07:08
*** ryanpetrello has quit IRC07:13
*** pcaruana has joined #openstack-infra07:16
*** yamamoto has joined #openstack-infra07:16
*** AJaeger has joined #openstack-infra07:17
*** dannywilson has joined #openstack-infra07:19
*** dannywilson has quit IRC07:20
*** belmoreira has joined #openstack-infra07:21
*** sputnik13 has quit IRC07:23
*** sputnik13 has joined #openstack-infra07:23
*** armax has quit IRC07:26
*** Sukhdev has quit IRC07:28
*** mrunge has joined #openstack-infra07:32
*** Ala has joined #openstack-infra07:33
*** achanda has joined #openstack-infra07:35
*** berendt has joined #openstack-infra07:39
*** yamamoto has quit IRC07:43
*** naggappan has joined #openstack-infra07:46
*** _nadya_ has joined #openstack-infra07:46
*** mrmartin has joined #openstack-infra07:47
*** melwitt has quit IRC07:50
*** _nadya_ has quit IRC07:52
*** talluri has quit IRC07:54
*** achanda has quit IRC07:55
*** achanda has joined #openstack-infra07:56
*** pblaho has joined #openstack-infra07:57
*** achanda has quit IRC08:01
*** achanda has joined #openstack-infra08:02
*** pcrews has quit IRC08:04
*** e0ne has joined #openstack-infra08:05
*** e0ne has quit IRC08:07
*** pcrews has joined #openstack-infra08:07
*** bdpayne has quit IRC08:07
*** bdpayne has joined #openstack-infra08:09
*** Guest51388 has joined #openstack-infra08:12
*** pcrews has quit IRC08:13
*** arxcruz has joined #openstack-infra08:18
*** yamamoto has joined #openstack-infra08:19
*** mpaolino has joined #openstack-infra08:19
*** yfried is now known as yfried|afk08:20
*** yfried|afk is now known as yfried08:24
*** matel has joined #openstack-infra08:24
*** jedimike has joined #openstack-infra08:25
*** HeOS has quit IRC08:26
*** jcoufal has joined #openstack-infra08:30
*** yamamoto has quit IRC08:30
*** bhunter71 has quit IRC08:31
*** Longgeek has joined #openstack-infra08:33
*** amuller has joined #openstack-infra08:34
*** wuhg has joined #openstack-infra08:37
garykAJaeger: you have a few secs for some question. I am in the process of creating the git history. Any idea how long this should take? The script (similar to the oslo incubabator one) is running for nearly over an hour08:41
*** _nadya_ has joined #openstack-infra08:42
AJaegergaryk: it takes some time to run.08:43
AJaegerneutron is a large repo but an hour looks rather long.08:44
AJaegerIs it really doing something?08:44
AJaegerI haven't used that script, so can't help with it08:44
garykAJaeger: from the trace I am getting: 2014-12-17 00:44:28 /usr/lib/git-core/git-filter-branch: 2: [: 4deaa53bafae2210e0789d6617c5e6e8b8079fa1: unexpected operator08:44
garykRewrite e75ae7f6b636e11c6a78309d1dcba8753053d063 (2019/8964).bzrignore08:44
garykI guess that it is running over all of the commits ever and exclusing the ones that are not related to the files that I am keeping.08:45
AJaegergaryk: yeah, exactly08:46
AJaegerthat trace looks bad08:46
AJaegerthe rewrite is a fourth done and as long as it continues, it's fine - that takes very long08:47
AJaegerhow did you invoke tools/ ?08:48
*** tnovacik has joined #openstack-infra08:49
garykAJaeger: i am running a tweaked script -
*** ajo has joined #openstack-infra08:49
*** talluri has joined #openstack-infra08:50
AJaegergaryk: discuss the trace with dougwig - and once your repo is done, publish it somewhere and check that it looks sane...08:51
garykAJaeger: ok, will do.08:52
*** ChuckC_ is now known as ChuckC08:52
ajogaryk, is the gate ok? I found one of my jobs gone and strange spikes in the zuul job queue08:53
ajomorning btw ;)08:54
*** Guest24100 has quit IRC08:56
*** Guest24100 has joined #openstack-infra08:56
*** otter768 has joined #openstack-infra08:56
*** ryanpetrello has joined #openstack-infra08:57
*** jpich has joined #openstack-infra08:58
garykajo: i am not sure - i have been trying to get a +1 for about 48 hours to no avail08:59
*** sabari is now known as zz_sabari08:59
*** tnovacik has quit IRC08:59
*** Masahiro has quit IRC09:00
*** andreykurilin_ has joined #openstack-infra09:00
*** Masahiro has joined #openstack-infra09:00
*** _nadya_ has quit IRC09:01
*** ryanpetrello has quit IRC09:01
*** otter768 has quit IRC09:02
*** _nadya_ has joined #openstack-infra09:05
*** Masahiro has quit IRC09:05
*** bdpayne has quit IRC09:06
*** tnovacik has joined #openstack-infra09:06
*** hashar has joined #openstack-infra09:08
garykAJaeger: the command that is taking forever is git filter-branch. I guess i will just try and be patient....09:08
AJaegergaryk: yeah, as long as it counts up...09:08
garykajo: gate seems fine now - all those rechecked from this morning pass.09:09
ajogaryk, yes, I just discovered than my last recheck is at least running09:09
garykAJaeger: most of the code that we are pushing to the branch will be refactored - i am not really sure what value the history will have - that is currently in the neutron branch. but i am trying09:09
*** jgallard_ has quit IRC09:10
*** MaxV has joined #openstack-infra09:10
*** jgallard_ has joined #openstack-infra09:10
*** Masahiro has joined #openstack-infra09:12
openstackgerrityolanda.robla proposed openstack-infra/storyboard: Return task count by status for a story dynamically
*** tnovacik has quit IRC09:13
*** BobBall_AWOL is now known as BobBall09:13
*** derekh has joined #openstack-infra09:16
*** _nadya_ has quit IRC09:16
*** mrunge has quit IRC09:18
openstackgerrityolanda.robla proposed openstack-infra/infra-specs: Add nodepool image testing spec.
*** zz_avozza is now known as avozza09:23
garykAJaeger: problem was logging in the scrip - there are 8K neutron commits - i removed the logging and script runs a lot faster - it is 90% done… oh well.09:24
*** mrunge has joined #openstack-infra09:25
*** mrunge has joined #openstack-infra09:25
*** achanda has quit IRC09:28
*** praveenkumar has joined #openstack-infra09:30
*** yamamoto has joined #openstack-infra09:31
*** tnovacik has joined #openstack-infra09:33
praveenkumarCan someone please tell me what's wrong with JJB template?09:33
praveenkumarI am getting error
*** achanda has joined #openstack-infra09:34
hasharpraveenkumar: the string in  dsl: |   is passed to python  str.format()09:35
hasharpraveenkumar: so  {someparam}   is expanded based on a dict like { 'someparam': 'myvalue' }09:35
hasharI think that is it09:35
*** yamamoto has quit IRC09:36
praveenkumarhashar: so {name} value I already passed in project section, but it causing issue in param[] section.09:37
*** teran_ has joined #openstack-infra09:39
praveenkumarhashar: is it also evalute dsl because I think it should just convert dsl section to string after key values and put under dsl section.09:39
hasharI am not sure how to prevent params evaluation09:41
hasharanyway you will need it to evaluate {name}09:41
*** teran has quit IRC09:41
AJaegerpraveenkumar: you might need to change the "{" to "{{" for quoting09:41
hasharso the best thing would be to escape the curly braces by doubling them:  { -> {{09:41
hasharthat is how curly braces are escaped with python str.format()09:41
hasharAJaeger: +1 :]09:42
praveenkumarAJaeger: hashar alright, let me try that also.09:42
*** e0ne has joined #openstack-infra09:42
hasharthat pass on my setup09:42
hasharthough I have no idea whether the resulting XML is valid / doing what is expected :]09:42
*** HeOS has joined #openstack-infra09:44
*** salv-orlando has joined #openstack-infra09:44
praveenkumarAJaeger: hashar: made changes as suggested but still failing to eval "GERRIT_HOST parameter missing to format parallel".09:44
AJaegerpraveenkumar: write:09:45
AJaeger{{ build( "{name}-pep8",09:45
*** acruz_ has joined #openstack-infra09:45
praveenkumarAJaeger: oh alright and end with '}}'09:45
*** acruz_ has quit IRC09:45
AJaegeryou quoted the wrong string09:45
AJaegersure ;909:45
*** acruz_ has joined #openstack-infra09:46
hasharyeah sorry for the misleading info :D09:46
hashartried it, it works09:46
matelHi guys, anyone experienced issues with iptables - applying puppet config fails, with error 6:
matelsomething like Could not start Service[iptables]: Execution of '/etc/init.d/iptables-persistent start' returned 1: * Loading iptables rules...09:46
hasharpraveenkumar: that yields some XML
*** teran_ has quit IRC09:47
*** arxcruz has quit IRC09:47
*** acruz_ is now known as arxcruz09:47
hasharthe example at  probably need to mention it when using a job-template09:47
praveenkumarhashar: I thing generated xml look file09:47
*** acruz_ has joined #openstack-infra09:47
praveenkumarhashar: yes, exactly my point09:47
praveenkumarhashar: here also they used parallel build without any '{{ }}' so may be this need an update.09:48
AJaegerpraveenkumar: Quoting for a template and quoting for a job is different!09:48
AJaegerA template needs variable expansion, a job not.09:49
*** arxcruz has quit IRC09:49
AJaegerSo, the example is a job and thus does not need quoting.09:49
hasharpraveenkumar: you can  git clone ssh://   and edit jenkins_jobs/modules/  :-]09:49
*** acruz_ is now known as arxcruz09:49
*** salv-orlando has quit IRC09:49
AJaegermatel: no idea. Which job is this?09:49
hasharthe double curly braces are definitely documented somewhere09:49
praveenkumarAJaeger: ah alright, so may be an example needed for job_template for flow project type.09:50
hasharwould not hurt to have it repeated on the Flow Project module description as well09:50
hasharpraveenkumar: +1 :]09:50
openstackgerrityolanda.robla proposed openstack-infra/storyboard: Return task count by status for a story dynamically
matelAJaeger: it's our own nodepool.09:50
hasharmatel: no idea sorry might be a transient issue with one of the cloud providers09:50
praveenkumarhashar: I will try to send a example patch set for this.09:51
matelAJaeger: but we use upstream wherever we can.09:51
matelAJaeger: I will try to re-build it again.09:51
*** ssam2 has joined #openstack-infra09:53
*** mrunge has quit IRC09:53
AJaegermatel: I can't help with nodepool, hope others can.09:54
matelAJaeger: i think it's mainly a puppet related issue, the module that does the iptables magic.09:54
matelhashar: thanks, I am re-building it again, but I don't think it has anything to do with the cloud provider.09:55
hasharmatel: I have no clue really. Just suspecting :]  Maybe the image is lacking IPv6 support09:56
hasharmatel: I am afraid  you will have to wait for East coasters to wake up09:56
hashar(US east coasters)09:56
*** ihrachyshka has joined #openstack-infra09:56
matelhashar: thanks, will look into it, maybe I find the issue.09:56
openstackgerritNikita Konovalov proposed openstack-infra/python-storyboardclient: Added logging and i18n dependencies
*** teran has joined #openstack-infra09:59
*** avozza is now known as zz_avozza09:59
*** yamamoto has joined #openstack-infra10:00
*** mrunge has joined #openstack-infra10:00
*** achanda has quit IRC10:01
*** achanda has joined #openstack-infra10:02
*** tnovacik has quit IRC10:03
*** teran has quit IRC10:03
*** lifeless_ is now known as lifeless10:04
*** jamielennox is now known as jamielennox|away10:04
*** skolekonov has joined #openstack-infra10:06
*** adalbas has joined #openstack-infra10:06
*** achanda has quit IRC10:07
*** adalbas has quit IRC10:08
*** ociuhandu has joined #openstack-infra10:08
*** zz_avozza is now known as avozza10:08
*** garyk has left #openstack-infra10:10
*** AJaeger has quit IRC10:10
*** zz_johnthetubagu is now known as johnthetubaguy10:13
*** akuznetsova has left #openstack-infra10:14
*** liusheng has quit IRC10:21
*** jamielennox|away is now known as jamielennox10:21
*** liusheng has joined #openstack-infra10:21
*** e0ne is now known as e0ne_10:21
*** jp_at_hp has joined #openstack-infra10:22
*** mrmartin has quit IRC10:22
*** e0ne_ is now known as e0ne10:23
openstackgerritNikita Konovalov proposed openstack-infra/python-storyboardclient: Setting up base classes
*** ociuhandu has quit IRC10:26
openstackgerritNikita Konovalov proposed openstack-infra/python-storyboardclient: Added required dependencies
openstackgerritNikita Konovalov proposed openstack-infra/python-storyboardclient: Setting up base classes
*** Masahiro has quit IRC10:32
*** talluri has quit IRC10:33
*** e0ne is now known as e0ne_10:33
*** yamamoto has quit IRC10:39
*** alexpilotti has joined #openstack-infra10:41
*** e0ne_ is now known as e0ne10:42
*** yamamoto has joined #openstack-infra10:42
*** unicell has quit IRC10:45
*** unicell has joined #openstack-infra10:46
*** ryanpetrello has joined #openstack-infra10:46
*** talluri has joined #openstack-infra10:49
*** jamielennox is now known as jamielennox|away10:51
*** ryanpetrello has quit IRC10:51
*** otter768 has joined #openstack-infra10:57
*** jgallard_ has quit IRC10:58
*** ZZelle has quit IRC10:58
*** ZZelle has joined #openstack-infra10:58
*** otter768 has quit IRC11:02
*** hashar has quit IRC11:03
*** yamamoto has quit IRC11:03
openstackgerritgaryk proposed openstack-infra/project-config: Add VMware-NSX to StackForge
*** garyk has joined #openstack-infra11:14
*** andreykurilin_ has quit IRC11:18
openstackgerritMauro Rodrigues proposed openstack-infra/elastic-recheck: Update query for Bug #1365046
uvirtbotLaunchpad bug 1365046 in openstack-gate "Job failed due to no devstack directory" [Undecided,Confirmed]
*** garyk has quit IRC11:27
*** teran has joined #openstack-infra11:28
*** teran has quit IRC11:32
*** Masahiro has joined #openstack-infra11:33
*** praveenkumar has quit IRC11:35
*** teran has joined #openstack-infra11:35
*** mpaolino has quit IRC11:38
*** Masahiro has quit IRC11:38
*** ajo has quit IRC11:39
*** ajo has joined #openstack-infra11:39
*** hashar has joined #openstack-infra11:43
*** pc_m has joined #openstack-infra11:45
*** hashar has quit IRC11:46
mikahm, how do you manage to get those "Jenkins check", "IBM PowerKVM CI check", etc subsections on gerrit, like on ? is this some special configuration of gerrit? is zuul reporting that?11:46
*** ociuhandu has joined #openstack-infra11:47
*** aysyd has joined #openstack-infra11:49
*** e0ne is now known as e0ne_11:49
*** ociuhandu has quit IRC11:52
*** dims has joined #openstack-infra11:55
*** adalbas has joined #openstack-infra11:55
*** Guest24100 is now known as Bob11:56
*** Bob is now known as Bob211:56
*** Bob2 is now known as Bob111:56
*** ociuhandu has joined #openstack-infra11:57
*** e0ne_ has quit IRC12:00
*** yaguang has quit IRC12:01
*** salv-orlando has joined #openstack-infra12:01
*** erlon has joined #openstack-infra12:02
*** matel is now known as matel_eating12:02
*** yamamoto has joined #openstack-infra12:04
*** garyk has joined #openstack-infra12:04
*** salv-orlando has quit IRC12:05
*** salv-orlando has joined #openstack-infra12:05
*** dmsimard_away is now known as dmsimard12:06
*** salv-orlando has quit IRC12:07
*** salv-orlando has joined #openstack-infra12:08
*** ddieterly has joined #openstack-infra12:08
*** yamamoto has quit IRC12:09
openstackgerritChristian Berendt proposed openstack/requirements: Bump version of openstack-doc-tools >= 0.21
*** salv-orlando has quit IRC12:10
*** talluri has quit IRC12:12
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: move oslo.version to attic
*** amitgandhinz has joined #openstack-infra12:17
*** afazekas has joined #openstack-infra12:21
*** AJaeger has joined #openstack-infra12:22
*** cnesa has joined #openstack-infra12:27
*** doug-fish has joined #openstack-infra12:28
*** MaxV_ has joined #openstack-infra12:29
*** ajo_ has joined #openstack-infra12:29
*** ajo has quit IRC12:30
*** [HeOS] has joined #openstack-infra12:31
*** cnesa2 has joined #openstack-infra12:31
*** cnesa has quit IRC12:31
*** HeOS has quit IRC12:31
*** [HeOS] is now known as HeOS12:32
*** MaxV has quit IRC12:32
*** praveenkumar has joined #openstack-infra12:33
*** baoli has joined #openstack-infra12:33
*** baoli has quit IRC12:33
*** baoli has joined #openstack-infra12:34
*** igorbelikov has joined #openstack-infra12:34
*** ryanpetrello has joined #openstack-infra12:35
*** BobBall has quit IRC12:36
*** matel_eating is now known as matel12:36
*** yamamoto has joined #openstack-infra12:38
*** ryanpetrello has quit IRC12:39
*** lttrl has joined #openstack-infra12:40
*** yamamoto has quit IRC12:41
*** yamamoto has joined #openstack-infra12:41
*** yamamoto has quit IRC12:42
*** Masahiro has joined #openstack-infra12:49
gilliardmika: they are added by a javascript overlay. Check the file hideci.js in your browser.12:49
*** bobba has joined #openstack-infra12:51
*** dprince has joined #openstack-infra12:52
*** hashar has joined #openstack-infra12:52
*** mase_x200 has joined #openstack-infra12:53
*** Masahiro has quit IRC12:54
bookwarhi everyone, can you please check what's happening with merge-check pipeline: we have stackforge/fuel-main pending there for a long time12:56
AJaegerbookwar: for 13min, correct?12:57
AJaegerbookwar: upper left corner shows "Queue lengths: 77 events, 136 results."12:57
AJaegerAnd all nodes are busy, so will take a bit longer12:58
AJaegermerge-check is nothing that should stop you, no need to wait for it, just continue coding ;)12:58
*** otter768 has joined #openstack-infra12:58
bookwarAJaeger: yes, sorry, i misunderstood, i thought this is the pipeline which prevents merging12:59
bookwarbut it is only the checker for unmerged commits12:59
*** nfedotov has joined #openstack-infra13:00
*** ildikov has quit IRC13:01
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/infra-specs: Create Nodepool REST API spec
*** otter768 has quit IRC13:03
AJaegerbookwar: correct13:04
*** imcsk8 has quit IRC13:05
*** imcsk8 has joined #openstack-infra13:05
*** vipul has quit IRC13:05
*** jgallard_ has joined #openstack-infra13:05
*** jesusaurus has quit IRC13:05
*** amotoki has joined #openstack-infra13:08
*** weshay has joined #openstack-infra13:08
*** lucasagomes has joined #openstack-infra13:09
Shrewsfungi: hi!13:09
*** e0ne has joined #openstack-infra13:09
*** fandi has joined #openstack-infra13:09
garykAJaeger: i have addressed your comments - please see - this has a git repo with the history13:10
AJaegergreat, garyk! Will review later13:10
lucasagomesfungi, hi there. Once you have a time mind taking a look at ? I understand it was mostly a find&replace for all projects13:10
*** vipul has joined #openstack-infra13:10
garykAJaeger: thanks!13:11
*** talluri has joined #openstack-infra13:12
*** belmoreira has quit IRC13:13
*** ildikov has joined #openstack-infra13:13
*** _buzz has quit IRC13:15
*** talluri_ has joined #openstack-infra13:16
*** talluri has quit IRC13:17
*** ssam2 has quit IRC13:17
*** jesusaurus has joined #openstack-infra13:18
sdaguebookwar: it checks for merge conflicts in unmerged patches13:19
*** afazekas has quit IRC13:20
*** kashyap has quit IRC13:20
*** kgiusti has joined #openstack-infra13:22
*** kevinbenton has quit IRC13:23
*** kashyap has joined #openstack-infra13:25
*** kevinbenton has joined #openstack-infra13:25
*** mbacchi has joined #openstack-infra13:26
*** afazekas has joined #openstack-infra13:26
*** ssam2 has joined #openstack-infra13:31
*** tkelsey has joined #openstack-infra13:36
*** mjturek has joined #openstack-infra13:37
*** yamamoto has joined #openstack-infra13:42
*** vigneshvar has quit IRC13:43
*** dkliban_afk is now known as dkliban13:43
*** mpaolino has joined #openstack-infra13:45
mikagilliard: thanks for the hint, nice trick :)13:47
*** jcoufal_ has joined #openstack-infra13:47
*** yamamoto has quit IRC13:47
*** cdent has joined #openstack-infra13:49
openstackgerritThierry Carrez proposed openstack-infra/project-config: openstack/requirements stable branch ACL update
*** che-arne has joined #openstack-infra13:49
*** jcoufal has quit IRC13:50
*** hdd has joined #openstack-infra13:50
*** mrunge has quit IRC13:51
*** vdo has joined #openstack-infra13:52
*** jaypipes has joined #openstack-infra13:52
*** vdo has quit IRC13:52
*** vdo has joined #openstack-infra13:52
*** berendt has quit IRC13:52
*** vdo is now known as vdo_13:53
*** vdo_ is now known as vdo13:54
*** vdo has quit IRC13:54
*** vdo has joined #openstack-infra13:54
gilliardwoah did gerrit just update?13:56
*** talluri_ has quit IRC14:00
*** talluri has joined #openstack-infra14:01
garykAJaeger: i blew away the 'nsx' branch. that was my bad14:02
*** mpaolino has quit IRC14:03
*** esker has joined #openstack-infra14:03
*** dmsimard is now known as dmsimard_away14:04
*** dpaterson has joined #openstack-infra14:06
annegentleis it 1500 UTC now?14:06
*** miqui has joined #openstack-infra14:07
annegentlethanks gilliard :)14:07
gilliardI live in UTC so it's an easy one :)14:07
*** julim has joined #openstack-infra14:07
*** signed8bit_ZZZzz is now known as signed8bit14:09
*** emagana has joined #openstack-infra14:09
*** radez_g0n3 is now known as radez14:11
mesteryanteaya: Regarding, I have given my +1 now. I think garyk is on the right track there.14:11
*** mfink has quit IRC14:12
*** mpaolino has joined #openstack-infra14:13
*** doug-fish has quit IRC14:15
*** dustins has joined #openstack-infra14:16
*** ala_ has joined #openstack-infra14:17
openstackgerritMerged openstack-infra/elastic-recheck: Update query for Bug #1365046
uvirtbotLaunchpad bug 1365046 in openstack-gate "Job failed due to no devstack directory" [Undecided,Confirmed]
*** esker has quit IRC14:18
*** ddieterly has quit IRC14:18
*** mfink has joined #openstack-infra14:19
*** esker has joined #openstack-infra14:19
*** ddieterly has joined #openstack-infra14:19
*** Ala has quit IRC14:19
*** mattfarina has joined #openstack-infra14:20
*** ala_ has quit IRC14:20
*** Ala has joined #openstack-infra14:20
AJaegergaryk: cool!14:21
AJaegermestery: Yeah, 140409 looks fine now. I can't check all the content of the upstream repo14:22
mesteryAJaeger: I trust garyk has this all working smoothly now. :)14:23
*** ddieterly has quit IRC14:23
*** dkranz has joined #openstack-infra14:29
*** e0ne is now known as e0ne_14:29
*** mpaolino has quit IRC14:29
*** resker has joined #openstack-infra14:30
*** esker has quit IRC14:30
*** e0ne_ is now known as e0ne14:31
*** erikmwilson has joined #openstack-infra14:31
sdagueso something remains very odd with zuul's event queue14:32
*** resker has quit IRC14:33
*** Sincler has quit IRC14:34
*** dizquierdo has joined #openstack-infra14:35
*** mase_x200 has quit IRC14:35
*** hdd has quit IRC14:36
*** mtanino has joined #openstack-infra14:37
fungiShrews: what up?14:37
AJaegersdague: yeah, looking at it yesterday it seem driven by a cron job running every 15 mins14:37
AJaegerbut I didn't look close enough or long enough, just looked odd14:37
Shrewsfungi: was just going to query you about 1393714:37
AJaegersorry, just anecdotal reference14:38
*** AJaeger has quit IRC14:38
*** Masahiro has joined #openstack-infra14:38
Shrewsfungi: mostly if you'd mind if i made a change for you since that seemed like a global search/replace through all projects14:38
*** nelsnelson has joined #openstack-infra14:39
Shrewsfungi: sorry, 13937414:40
* Shrews needs more coffee before cut & paste will work correctly14:41
*** ssam2 has quit IRC14:41
*** ssam2 has joined #openstack-infra14:41
*** luqas has joined #openstack-infra14:42
*** Masahiro has quit IRC14:43
*** dpaterson has quit IRC14:44
*** e0ne is now known as e0ne_14:46
fungiShrews: lucasagomes: ahh, yeah, hack it up all you like14:48
*** talluri has quit IRC14:48
Shrewsfungi: already did  :-P14:49
lucasagomesfungi, cool :) Shrews just did it, patch is approved now14:49
fungiShrews: lucasagomes: i have >200 patches and while i spot-checked a bunch of them looking for anything i might have missed, there were still some rare patterns i failed to catch or which led to misleading prose/links14:49
Shrewsfungi: yeah, i figured as much14:49
fungiso those were to a great extent a starting point/placeholder for others to comment on or take over as they saw fit14:49
*** doude has quit IRC14:50
*** zz_gondoi is now known as gondoi14:50
*** mriedem has joined #openstack-infra14:50
fungimainly just trying to drive down traffic to the old wiki documents which have been supplanted and get more traffic flowing directly to the infra manual when appropriate14:50
*** armax has joined #openstack-infra14:50
*** pcaruana has quit IRC14:51
*** ddieterly has joined #openstack-infra14:52
*** Sincler has joined #openstack-infra14:52
*** e0ne_ is now known as e0ne14:55
*** bobba is now known as BobBall14:56
*** mpaolino has joined #openstack-infra14:56
*** esker has joined #openstack-infra14:57
*** erikmwilson has quit IRC14:57
*** sigmavirus24 has joined #openstack-infra14:57
*** resker has joined #openstack-infra14:58
*** doug-fish has joined #openstack-infra14:58
*** gnuoy has quit IRC14:59
*** otter768 has joined #openstack-infra14:59
*** gnuoy has joined #openstack-infra14:59
* annegentle waves15:00
annegentlettx mtreinish jeblair up for a chat?15:00
*** stevemar has joined #openstack-infra15:00
*** signed8bit has quit IRC15:00
mtreinishannegentle: yep15:00
ttxannegentle: sure15:00
annegentlettx: dumb question, how do I invite the four of us to a channel/room?15:01
*** talluri has joined #openstack-infra15:01
ttxannegentle: create channel, then /invite15:01
*** talluri has quit IRC15:01
*** esker has quit IRC15:01
ttxlike /invite ttx while on desired channel15:01
*** doude has joined #openstack-infra15:02
*** carl_baldwin has joined #openstack-infra15:02
*** erikwilson is now known as erikmwilson15:04
jeblairsdague: there's a stuck jenkins administrative job, we're still debugging15:04
*** otter768 has quit IRC15:04
KiallIs it just me, or has the gate been acting up over the last few days? Jobs complete for a patchset, yet still on the statis page 45 mins later etc15:04
fungijeblair: oh, btw it looks like we have the same backup you were tracking last night--hopefully caught a fish in your net this time15:05
fungiahh, nevermind that's what you just commented onm15:05
fungiKiall: the queue lengths listed in the top left corner are things zuul has outstanding it needs to process, which could be acting on job completions among other tasks15:06
fungiKiall: and yes, we recently switched the gearman implementation it relies on to do async i/o, still possibly some bugs in there jeblair is trying to pin down15:08
KiallAh, change of lib makese sense.. I've just noticed this large delay between jobs completing and being reported back to gerrit over the last few days and figured something was uop15:09
*** timcline has joined #openstack-infra15:09
*** ayoung has joined #openstack-infra15:11
*** garyk has quit IRC15:12
Kiallfungi: (graphite.o.o link) - looks like the lib change made a big difference ;)15:13
openstackgerritSean Dague proposed openstack/requirements: remove docs projects from g-r constraints
*** sputnik13 has quit IRC15:14
*** garyk has joined #openstack-infra15:14
openstackgerritPaul Karikh proposed openstack/requirements: Bump version of django_openstack_auth to >=1.1.8
*** lttrl has quit IRC15:16
*** lttrl has joined #openstack-infra15:16
*** yfried has quit IRC15:18
*** yfried_ has joined #openstack-infra15:18
fungiKiall: that doesn't really indicate a direct correlation. that library was last updated on 9 days ago15:19
dhellmannanteaya: ".tmpl" is "template"15:20
dhellmannanteaya: I just made up the extension, we can change it if you want15:20
dhellmannfungi: thanks for the heads-up on the merge; I'll send another email encouraging reviews of the stable requirements updates15:21
*** nuritv has quit IRC15:23
*** lucasagomes is now known as lucas-hungry15:24
*** ihrachyshka has quit IRC15:25
*** signed8bit has joined #openstack-infra15:25
*** lttrl has quit IRC15:28
*** marun has joined #openstack-infra15:31
*** hdd has joined #openstack-infra15:32
*** wenlock has joined #openstack-infra15:35
*** Ala has quit IRC15:36
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for trove unit test bug 1379729
uvirtbotLaunchpad bug 1379729 in trove "test_tranformer unit test fails with ModelNotFoundError: InstanceServiceStatus Not Found" [Medium,Triaged]
*** belmoreira has joined #openstack-infra15:41
*** dkranz has quit IRC15:43
*** achanda has joined #openstack-infra15:44
fungimordred: ttx: just a head up, post-setuptools-8 discussion on the distutils-sig ml indicates there's a good likelihood that pep 440 will soon be revised to switch from "c" to "rc"15:44
fungiclarkb: ^15:44
odyssey4mehaha fungi :)15:45
mordredfungi: cool15:45
mordredfungi: also, re: mailing list around putting global requirements entries into requirements proposals ...15:46
mordredfungi: we do have some amount of similar code in the pbr changelog writing15:47
mordredfungi: so if someone decides to work on such a thing, it may be possible to learn from or even directly leverage that15:47
* fungi nods15:47
fungii think the trick in this case is that you need to filter git blame output to the specific lines you're proposing, and then use the hashes listed by that result to grab commit messages from the commit history and concatenate them, or something15:48
*** miqui_ has joined #openstack-infra15:48
*** carl_baldwin has quit IRC15:50
mordredyah. blech15:50
*** dkranz has joined #openstack-infra15:50
mordredfungi: I know it's a tiring subject, but I'd like to repeat my statement that perhaps our efforts to ensure a single common requirements list may be not worth it15:51
*** amotoki has quit IRC15:51
mordredfungi: also, if you haven't already, definitely read the scrollback between dstufft and I from after you dropped off last night15:52
fungioh, i have15:52
mordredjust making sure15:52
fungii get that requirements files in general are an ugly hack and guarantee almost nothing, so trying to sync them is actually of less benefit than people would expect15:52
*** harlowja_at_home has joined #openstack-infra15:52
dstufftmordred: probably you still need that until pip gets a real dep solver15:53
ttxfungi: sigh15:53
mordreddstufft: unless we gave up on the idea of a single machine coordinated install without any namespacing15:53
ttxfungi: well, we have both tags for swift 2.2.1. Will that be fixed before 2015.1 RC period ?15:54
dstufftmordred: well, yea that's a thing, though I was under the impression that was a hard requirement15:54
mordreddstufft: it is, or at least, we treat it as if it is15:54
mordreddstufft: but there are times when I think that the amount of effort it takes to still not quite get it right may be misspent15:54
dstufftttx: the c -> rc thing? the code to fix it is already done, I'll probably be saying something like "okay unless someone has an objection i'm gonna do the needful" sometime today15:55
mordredwhen in _reality_ - people who are deploying openstack are deploying it across many different machines - and the one "production" thing I know of that co-locates services does it with virtualenvs15:55
fungittx: unknown... it would need to appear in an updated version of the spec and then get implemented by setuptools et al15:55
*** belmoreira has quit IRC15:55
harlowja_at_homemordred, i can vouch for yahoo co-locating at least some of the api services together; not all but some...15:56
harlowja_at_homeand not in virtualenvs15:56
fungimordred: right, again our requirements syncing is almost certainly a reaction to/symptom of devstack all-in-one installs15:56
*** mrunge has joined #openstack-infra15:56
mordredharlowja_at_home: cool. good to know15:56
mordredharlowja_at_home: does yahoo locally create OS distro packages of them first?15:57
* mordred asks leading question, knowing software that harlowja_at_home has written ...15:57
clarkbfungi: sort of, its a problem you have anytime you try to install >1 python package from source on a single machine15:58
*** hashar is now known as hasharCall15:58
*** skolekonov has quit IRC15:58
*** mpaolino has quit IRC15:58
mordredthat's the thing - if we're going to stick with pip, then I kinda think that venvs or containers make more sense, and if we're going to do all-in-one, we should figure out local distro packaging which has better dep solving15:58
clarkbfungi: in fact I expect to run into it with nodepool/zuul/gear/jeepyb more often as the all in third party stuff gets more steam15:58
mordredbecause it's not that installing the software is hard15:58
mordredit's that figuring out the dep solution is15:58
*** tonytan4ever has joined #openstack-infra15:58
clarkbI am a fan of venvs15:58
harlowja_at_homemordred, +2 to that; one recommended way would be cool, and likely its not the all-in-one install way...15:59
* dstufft is also a fan of venvs15:59
clarkblocally I have venvs all over the place for all different things. The only globally installed python I have is pip/virtualenv/tox15:59
* dstufft might be biased15:59
*** reed has joined #openstack-infra15:59
fungimordred: yes, i do feel like system-wide installation via pip is grotesque. on my servers and personal machines i leave the system-wide installation managed solely by the distro packaging system and use virtualenvs for anything pip-obtained15:59
dstufftmordred: there's another thing too, that is kind of interesting16:00
mordredclarkb: especially since upstream python, in the form of dstufft, has been telling us for ages that installing globally is a Very Bad Idea16:00
dstufftmordred: not sure ifit's useful at all for openstack16:00
morganfainbergmordred, +1 to venvs (smaller +1 to containers but really thats just a personal bias)16:00
mordredfungi: ++16:00
dstufftsec let me get link16:00
fungiclarkb: ha, i have you beat then. i install tox, virtualenv, et cetera into a virtualenv created by the distro python-virtualenv package16:00
*** pcrews has joined #openstack-infra16:00
mordredfungi: hah. nice16:00
fungibut i actually directly call ~/venvs/tox/bin/tox whatever16:00
harlowja_at_homei install tox into a a mini-tox; that goes into a micro-tox that has virtualenv; so there! :-P16:01
mordreddstufft: zomg16:01
clarkbfungi: I think I decided that virtualenv was too old and liked ease of not virtualenving to virtualenv :)16:01
dstufftmordred: it lets you pip install command line things into their own virtualenv16:01
mordredI was JUST going to say that a thing like that would be super helpful16:01
mordredbecause that way you don't have to know that $service is in venv $foo16:01
*** mpaolino has joined #openstack-infra16:01
fungiclarkb: yeah, i have a virtualenv virtualenv so i can ~/venvs/virtualenv/bin/virtualenv whatever to create other virtualenvs16:01
mordredclarkb, fungi: ^^ go check that out16:01
dstufftmordred: at some point I want to figure out how to make pip do this, but there's the problem where you'd sometimes want it and sometimes not want it, so I haven't touched it yet16:01
SergeyLukjanovoh, I see "Queue lengths: 694 events, 303 results." in zuul16:02
SergeyLukjanovmorning folks16:02
dstufftmordred: but yea, pipsi is sort of like static linking for python16:02
rcarrillocruzyou're making my brain bleed with such nesting :-)16:03
mordreddstufft: of course, it would be nice if pipsi was pip installable ...16:03
*** marcusvrn has quit IRC16:03
*** asselin_ has joined #openstack-infra16:04
*** spzala has joined #openstack-infra16:04
dstufftmordred: I know armin, I could probably go bother him to ask him if he could16:04
mordreddstufft: I'm mostly carping16:05
*** spzala has quit IRC16:05
dstufftI think the reason it probably isn't is so pipsi can itself be installed into a per command virtualenv16:05
mordreddstufft: but one could also imagine wanting to have an option to not write to ~/.local/bin but instead to /usr/local/bin16:06
*** rmcall has joined #openstack-infra16:06
*** Sukhdev has joined #openstack-infra16:06
clarkbmordred: can you summarize current state of setuptools world for us? do we just need the postN thing to get released?16:06
clarkbmordred: and new version for pytz?16:06
dstufftmordred: it has --home and --bin-dir to control where the venvs go and the bins go16:06
fungidstufft: i already do a lot of that. for example i have a ~/bin directory with symlinks into separate virtualenv bindirs for git-review, gertty, et cetera16:07
fungiworks surprisingly well16:07
mordredclarkb: yes to both16:07
*** prad has joined #openstack-infra16:07
clarkbmordred: anything else that is important?16:08
dstufftmordred: clarkb also need to turn off warnings16:08
fungitweak manpath a little and ~/man can do similarly for symlinks to manpages in virtualenvs16:08
mordreddstufft: yah - but that's a lot of work to achieve what woudl be a common case for me16:08
mordreddstufft: but maybe I'll just make a PR16:08
mordredclarkb: not that I know of16:09
dstufftmordred: clarkb the warning we added for telling people their non PEP 440 version is gonna sort < 0 is making some CLI tests fail b/c ubuntu has an unrelated non PEP 440 verisoned package installed16:09
mordredclarkb: although we _Do_ need to have a longer conversation with sdague (not today, I'm slammed) about the distro-packages vs. python packages in devstack16:09
Kiallfungi: you sure there's no issue? I only see 1 job running out of all the queued jobs, and the results from jobs 1hr 45min ago still havent reported :/16:09
mordredclarkb: oh- right - pep440 warning suppression16:10
mordredthere's an env var we can set16:10
dstufftPYTHONWARNINGS=ignore::RuntimeWarning:pkg_resources should shut up all the runtimewarning's from pkg_resources16:10
clarkbdstufft: so tests are parsing that output and saying lol fail?16:10
dstufftclarkb: yea16:10
clarkb:( ok16:10
dstufft(I didn't verify that syntax for PYTHONWARNINGS, but it's something like that)16:11
*** rcarrill` has joined #openstack-infra16:11
* morganfainberg exits.16:11
*** jerryz has quit IRC16:12
dstufftclarkb: mordred also if you're concerned at all about silencing all runtime warnings in pkg_resources I can make a patch to subclass, but if you're not I won't bother16:12
*** dangers_away is now known as dangers16:13
openstackgerritMauro Rodrigues proposed openstack-infra/elastic-recheck: Query for bug #1253896
uvirtbotLaunchpad bug 1253896 in tempest/havana "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,Confirmed]
clarkbdstufft: I think having the warnings is good even if it makes tests fail because it means we have stuff to fix16:13
clarkbdstufft: may not be directly related to a particular test but now we have things saying "Fix me"16:13
*** mpaolino has quit IRC16:13
*** amuller has quit IRC16:13
dstufftclarkb: in this case, the warning is completely unrelated to openstack, because when you import pkg_resources it parses all the versions of things installed16:14
*** rcarrillocruz has quit IRC16:14
dstufftand ubuntu has some completely unrelated thing installed16:14
dstufftwhich triggers the warning16:14
clarkbI see16:14
fungiKiall: did i say no issue? i said no obvious correlation to the most recent changes we made (indicating that while they're a place to start looking, they're not necessarily the cause)16:14
*** flip214 has joined #openstack-infra16:15
Kiallfungi: ah, sorry, I thought you meant it's just backed up a little ;)16:15
fungiKiall: the graph you linked shows a change in behavior two days ago, we changed the code 9 days ago16:15
dstufftclarkb: there's also a warning for swift's post release version in pbr, but that's getting (or is?) fixed I think16:15
flip214is it possible to change the "Identity" OpenID link for a user account?16:15
*** marcusvrn has joined #openstack-infra16:15
fungiflip214: where?16:15
clarkbSergeyLukjanov: ya zuul looks super unhappy processing those events16:16
clarkbSergeyLukjanov: jeblair was looking into it last night and should be around soon so I don't want to go poking anything just yet16:16
flip214I'd like to switch identity for my user16:16
fungiflip214: those get autogenerated when you log into gerrit with a new identity and it provides no facility to move/combine them16:16
fungiflip214: basically gerrit has open authentication, so the first time you auth with a new identity it creates a whole new account tied to it16:17
flip214fungi: yeah, but there'll be someone who can do an UPDATE statement in the right place, no?16:17
flip214It's just so that the work is attributed to the correct self.16:19
morganfainbergflip214, is it always possible to do crazy things like direct DB manipulation, it's a question of risk vs value. -16:19
*** sabeen has joined #openstack-infra16:19
morganfainbergflip214, you can also put a patch into stackalytics and gitdm to map the details correctly (for the email)16:19
flip214yeah, of course. I don't know whether there's a way to get that field writable, that's why I ask here.16:19
morganfainbergflip214, but the OpenID is not responsible for the email used on a commit message16:19
morganfainbergflip214, changing OpenID connect field only changes who can login, not the committer/author of patches16:20
flip214it's interesting that has a "delete" button, but not an "add"16:21
*** ihrachyshka has joined #openstack-infra16:21
flip214morganfainberg: yeah, I'm trying to get the "right" identity associated.16:22
morganfainbergflip214, the worst case scenario is make a new gerrit id, there is no reason it should have much (if any) impact except the username itself.16:22
*** lucas-hungry is now known as lucasagomes16:23
*** lucasagomes has left #openstack-infra16:23
morganfainbergflip214, that is if the nice folks here in -infra aren't doing the update for you. (i don't claim they will/wont etc, just offering details/workarounds)16:23
*** mpaolino has joined #openstack-infra16:23
*** harlowja_at_home has quit IRC16:23
flip214yeah, right. thank you, perhaps it's simpler to just switch.16:23
*** dizquierdo has quit IRC16:24
bookwari see that is up and running now, can we add publishing of our stackforge/fuel-pecs repository there?16:25
*** pblaho has quit IRC16:26
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova unit test race bug 1403564
uvirtbotLaunchpad bug 1403564 in nova "nova.tests.unit.conductor.test_conductor._BaseTaskTestCase.test_build_instances_scheduler_failure is racy" [Medium,Confirmed]
*** vigneshvar has joined #openstack-infra16:27
*** Masahiro has joined #openstack-infra16:27
tchaypoIt looks like there’s something affecting py33 jobs on many projects16:29
tchaypoat least and have been raised16:29
uvirtbotLaunchpad bug 1403557 in python-openstackclient "unable to build py33 environment (dup-of: 1403510)" [Undecided,New]16:29
uvirtbotLaunchpad bug 1403510 in os-apply-config "gate tests failed with "NameError: name 'StandardError' is not defined"" [Low,Triaged]16:29
*** rlandy has quit IRC16:29
*** ociuhandu has quit IRC16:30
*** jcoufal_ is now known as jcoufal16:31
nfedotovHello All! These ir a feature in the grenade project. The target tempest uses images of a base tempest (they were downloaded by base devstack). And it makes several tests to fail in a patchset Can somebody comment it?16:31
*** Masahiro has quit IRC16:31
*** jgallard_ has quit IRC16:33
ruhetchaypo: yeah. Steve Martinelli reported this issue in openstack-dev ML16:33
openstackgerritMerged openstack-infra/subunit2sql: Add new db api methods for getting test data from runs
openstackgerritMerged openstack-infra/subunit2sql: Refactor sql2subunit to use get_tests_run_dicts_from_run_id
openstackgerritMerged openstack-infra/subunit2sql: Correct writing of timestampes in write_test() for sql2subunit
*** salv-orlando has joined #openstack-infra16:33
ruhetchaypo: i have the same issue on other projects too: openstack/python-saharaclient and stackforge/yaql . i guess all the py34 jobs are affected16:34
tchaypoYou're seeing it on py34? I've only seen py3316:34
* clarkb reminds everyone setuptools released...16:35
ruhetchaypo: i see it on py3416:36
tchaypothanks clarkb16:36
*** patrickeast has joined #openstack-infra16:36
tchayposo i was about to say that one change i noticed between a patchset that passed tests, and the same patchset later, was a different order of requirement installation16:37
tchaypothe one that worked installed docutils as a dep of sphinx, then jinja2; the one that failed installed docutils as a dep of jinja2 as a dep of sphinx. sounds like maybe the new version of setuptools changed the ordering, amongst other things?16:37
tchaypobut both working/failing logs say they’re using setuptools 3.616:38
dstufftmordred: fedora is going to update Babel so it doesn't break with their version of pytz with setuptools 8, they said they'll push to f20 and f21 too16:38
flip214in case somebody else needs that: launchpad allows to change the openid link, via<username>/+edit16:38
*** flip214 has left #openstack-infra16:38
dstufftwill take awhile to make it into the release archive tho :(16:39
ruheclarkb: but why would it fail (tox -e py34) if there is still a cap on setuptools?16:39
clarkbruhe: the cap is artificial, it could be installing things itself16:39
*** ZZelle_ has joined #openstack-infra16:39
dstufftI don't believe anything in setuptools 8 should have affected the functionality of py316:40
dstufftthe only change in 8 is the PEP 440 stuff16:40
*** achanda has quit IRC16:41
clarkbdstufft: I think it may be how docutils is using setuptools? In any case it is a failure to install16:41
clarkband to not first rule out setuptools 8.0 would be silly imo16:41
mordredclarkb: should we set a global status on that? :)16:41
tchaypodocutils handles py3 compatibility by install-time conversion of its files16:41
*** achanda has joined #openstack-infra16:41
mordreddstufft: AWESOME16:42
dstufft2to3 ftl16:42
tchaypo is the failing log from my change16:43
tchaypo is the earlier run of the same patchset that passed16:43
dstufftmordred: they said 24-48 hours for it to get into the updates-testing, and then 2 weeks for it to make it out of there and into updates.. so it's a bit long :/16:43
jeblairclarkb, fungi: there's a performance hit in geard on job submission with reaally high queues.  with, honestly, a pretty trivial fix.  though i believe it would mean a behavioral change from C gearmand16:43
mordreddstufft: yah - but the only problem is a warning, right?16:43
tchaypoas far as i can tell both of them use setuptools 3.616:44
morganfainberghogepodge, ping.16:44
dstufftmordred: no, that's a hard failure when pkg_resources loads16:44
morganfainberghogepodge, re interop stuff.16:44
dstufftbecause Babel has a dependency on >0a16:44
mordredoh. awesome16:44
jeblairclarkb, fungi: but basically, every job submission right now (including the thousands of set_description jobs) probably takes > 1 second16:44
dstufftand 2012d < 0a16:44
clarkbjeblair: and then we run status on it and it gets really sad?16:44
mordredwell, that one we can take care of by removing the babel requirement from devstack rpms/debs16:44
morganfainberghogepodge, and you know you're not in openstack-dev right? ;)16:44
jeblairclarkb: no, i think that's okay, though it's one of the reasons status is slow to respond sometimes16:45
jeblairclarkb: so to track back...16:45
*** yamamoto has joined #openstack-infra16:45
*** achanda has quit IRC16:46
jeblairclarkb: i believe that because set_dsecription:jenkins01 is stuck, the queues get larger, and large queues cause slow job submission, and since zuul submits jobs at a very high rate, zuul's main loop is slowing16:46
dstufftoh wow16:46
dstufftthat py33 issue16:46
clarkbjeblair: gotcha16:46
jeblairclarkb: i do not know why set_description:jenkins01 is stuck, i'm still downloading pcaps :)16:46
clarkbdstufft: do you understand it?16:46
clarkbjeblair: ok, anything I can do to assist? should I put 01 in shutdown mode now that you have pcps?16:47
dstufftclarkb: I udnerstand what's happening, I don't understand why it just started happening16:47
clarkbdstufft: thats a start at least and way more than I understand16:47
mordredwow. we install sphinx in devstack16:47
jeblairclarkb: yeah, i think it's okay to shut down 0116:47
dstufftoh wait16:48
mtreinishmordred: heh, what pulls it in?16:48
dstufftI do know what made it fail recently16:48
dstufftit's pbr's fault16:48
mordredmtreinish: nothing16:48
mordredmtreinish: it's directly requested16:48
tchaypodstufft: comments on plz16:48
uvirtbotLaunchpad bug 1403510 in os-apply-config "gate tests failed with "NameError: name 'StandardError' is not defined"" [Low,Triaged]16:48
mordredmtreinish: and it's what causes distro babel to be installed16:48
mordredwhich is what's breaking that bit16:48
mtreinishoh wow, ok.16:48
clarkbjeblair: ok I am starting shutdown mode there16:48
*** marcusvrn has quit IRC16:49
dstufftdocutil's install is being executed16:49
openstackgerritJames E. Blair proposed openstack-infra/gear: Only wake relevant connections
dstufftin ./ there is a py2 version of docutils, which is being translated into py3 in ./build16:49
jeblairclarkb, fungi: ^ should make job submission fast even on large queues16:49
*** marcusvrn has joined #openstack-infra16:49
*** marun has quit IRC16:49
*** yamamoto has quit IRC16:50
*** sigmavirus24 is now known as sigmavirus24_awa16:50
dstufftas part of installing docutils, setuptools interates of the egg_writer entry point and imports things, which causes pbr to get imported16:50
dstufftpbr imports sphinx16:50
mordreddstufft, clarkb, fungi, mtreinish:
dstufftsphionx imports docutils16:50
mordredthat removes sphinx from devstack16:50
dstufftand ./ is on sys.path16:50
dstufftso it imports the py2 docutils from ./16:50
*** nfedotov has left #openstack-infra16:50
*** achanda has joined #openstack-infra16:51
mordreddstufft: that's so fantastic I want to spit16:51
tchaypoand then we’re running non-translated docutils in py3, so it fails.16:51
dstufftI am both impressed with the round about way that error happened, and that I manaaged to figure it out so quickly16:51
tchaypodstufft: so why did it just start breaking?16:52
*** sigmavirus24_awa is now known as sigmavirus2416:52
dstuffttchaypo: because pbr just recently released a version that was the first version to have an egg_writer entry point16:52
tchaypodstufft: we love you and owe you many $DRINK_OF_CHOICE :)16:52
mordreddstufft: oh for the love of16:52
tchaypodstufft: it’s not saturday, why would there be a pbr release?16:52
*** kgiusti has quit IRC16:52
mordreddstufft: so, I can protect the sphinx import from the egg_write code path16:52
openstackgerritMauro Rodrigues proposed openstack-infra/elastic-recheck: Query for bug #1253896
uvirtbotLaunchpad bug 1253896 in tempest/havana "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,Confirmed]
mordredit's really only needed if someone is actually trying to build sphinx docs16:53
tchayposo for this to fail, we need to have a python2 version of docutils in the system path, but be running python3 checks?16:53
dstufftin sys.path yea16:53
dstufftand the only reason you do16:53
dstufftis because you're installing docutils16:53
dstufftand sys.path[0] is "."16:53
dstufftthis error would only ever happen while installing docutils16:54
dstufftthis particular one16:54
mordredoh - wow, no I cannot16:54
* mordred digs deeper16:54
dstufftbecause if you wern't installing docutils into a py3 environment, you wouldn't haev the untranslated py2 copy of the code in "."16:54
openstackgerritChristian Berendt proposed openstack/requirements: Bump version of openstack-doc-tools >= 0.21.1
mtreinishmordred: +2 on 14249716:55
tchaypomordred: I’ve added pbr to
uvirtbotLaunchpad bug 1403510 in os-apply-config "gate tests failed with "NameError: name 'StandardError' is not defined"" [Low,Triaged]16:55
mordreddstufft: ok. I think I can see an out in pbr16:56
*** packet has joined #openstack-infra16:56
*** melwitt has joined #openstack-infra16:56
*** mpaolino has quit IRC16:56
*** bhunter71 has joined #openstack-infra16:57
*** marun has joined #openstack-infra16:57
*** cnesa2 has quit IRC16:58
openstackgerritAleksey proposed openstack-infra/storyboard: Fix i18n
*** marcusvrn has quit IRC16:58
tchaypooh, I just realised - it’s not that we’re importing the system docutils, it’s that we’re importing the version of docutils that we’re in the middle of translating, before the file has been translated16:58
dstuffttchaypo: yes16:59
tchaypoit sounds like this is going to break all py3x jobs - worth putting in status?17:00
garykancan you please take a look at
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Update ironic ACLs for ironic-stable-maint
*** otter768 has joined #openstack-infra17:00
*** zz_sabari is now known as sabari_17:01
dstufftmordred: y'all made my new favorite setuptools bug ever17:01
dstufftmordred: so congrats :D17:01
*** harlowja_away is now known as harlowja17:01
dstufftwell setuptools involved bug17:01
openstackgerritMerged openstack-infra/reviewstats: Sync with current core group
clarkbI am not sure I understand why distutils is being imported during setup17:01
clarkboh I know17:01
clarkbbecause sphinx is dumb and pbr has to monkey patch it to be usefulk17:01
*** doug-fish has quit IRC17:01
*** asselin_ has quit IRC17:02
*** doug-fish has joined #openstack-infra17:02
*** sarob has joined #openstack-infra17:02
*** dkranz has quit IRC17:02
clarkbok going to focus on gear now17:03
mordredclarkb, dstufft:
mordredtchaypo: ^^17:04
*** otter768 has quit IRC17:05
*** changbl has quit IRC17:05
mordredI believe that will prevent pbr from importing sphinx when it's running the egg_info writer hook17:05
tchaypomordred: closes-bug #140351017:05
uvirtbotLaunchpad bug 1403510 in os-apply-config "gate tests failed with "NameError: name 'StandardError' is not defined"" [Low,Triaged]
clarkbjeblair: so I am reading this geard change and is it divergent from C gearman in a meaningful way? the protocol isn't changed its merely an change on which connections are woken17:06
dstufftmordred: that depends on another patch17:06
dstufftmordred: that other patch can't pass py33 gates I don't think17:06
dstufftwithout this patch17:07
clarkbya flip the order and it should self test and all that17:07
tchaypoGood, because I was about to say that I haven’t been able to reproduce on my system so i don’t know how to test this17:08
clarkbmordred: so that fixes because it will fail and catch exception returning false then everything continues?17:08
mordredclarkb: nope - it fixes because the function to create the classes will not get run at import17:09
*** mpaolino has joined #openstack-infra17:09
mordredso when pbr/packaging is imported by setuptools to get the write_pbr_json hook, it will not run the import code at all17:10
mordredonly when you're actually operating pbr will that happen17:10
*** spzala has joined #openstack-infra17:10
mordredclarkb: I'm runnign out the door/not really here today - so if I missed something stupid, I can't fix it until tomorrow - however, if those pass tests, feel free to release without me17:12
*** hdd has quit IRC17:13
*** MaxV_ has quit IRC17:13
tchayposhould I be able to see checks in the queue on ?17:13
clarkbtchaypo: yes? gearman is currently unhappy though so things may be funny17:13
*** ssam2 has quit IRC17:14
*** emagana has quit IRC17:14
dhellmannmordred: there's a pbr refactoring patch that may help with some of the extraneous imports:
mordredoh god17:16
anteayadhellmann: ah thank you, that would explain why I couldn't find it17:17
mordreddhellmann: I'd love to take that - but there's so much in flight that there's no way that's landing17:17
mordredit's gonna have to be redone I think17:17
mordredbut I'll look at it tomorrow17:17
anteayadhellmann: no need to change I just didn't know where it came from17:17
dhellmannmordred: yeah17:17
anteayadhellmann: apparently your brain is not yet indexed :D17:17
dhellmannmordred: +2a on
dhellmannanteaya: I'm sure google is working on that17:17
openstackgerritJoshua Harlow proposed openstack-infra/elastic-recheck: Add query for bug 1403510 affecting docutils/py33/py34
uvirtbotLaunchpad bug 1403510 in os-apply-config "gate tests failed with "NameError: name 'StandardError' is not defined"" [Low,Triaged]
*** dkranz has joined #openstack-infra17:18
clarkbdhellmann: what about the child of 14250917:18
clarkbdhellmann: ideally we would release both of those to cover the current known issues17:18
anteayadhellmann: no doubt, wear your tinfoil hat17:18
dhellmannclarkb: ?17:19
dstufftdhellmann: yea17:19
clarkbdhellmann: yup17:19
clarkbdhellmann: it turns out doing 1.2.3a1.6 is not valid under pep440 but 1.2.3a1.post6 is17:19
*** sarob has quit IRC17:20
*** achanda has quit IRC17:20
lifelessclarkb: we should use devN not postN, for untagged builds17:20
dhellmann+2a, in spite of lack of tests17:20
clarkblifeless: we actually use both depending17:21
lifelessclarkb: postN is a release17:21
*** sarob has joined #openstack-infra17:21
lifelessclarkb: depending on what? There was no ention of postN in the discussion around versioning rules that the semver work provoked17:21
*** sabari_ is now known as zz_sabari_17:21
clarkblifeless: its based on sort order. so for servers you get 2014.1.0a1.devN and for clients you get 1.2.3a1.postN17:22
clarkblifeless: depending on how things should sort17:22
lifelessclarkb: wat17:22
clarkbbeacuse devN says you are working up to that release and postN says this is after that release17:22
clarkbdstufft: ^ did I capture that properly?17:22
lifelessclarkb: yes but we've consolidated all that17:22
lifelessclarkb: after a1 we go to a2.devN17:22
lifelessclarkb: for servers and clients17:22
jeblairclarkb: i believe c gearmand needlessly wakes connections.  i like the more efficient approach myself17:23
*** gyee has joined #openstack-infra17:23
clarkbjeblair: +117:23
*** Sukhdev_ has joined #openstack-infra17:23
jeblairclarkb: but no, afaik, it's not violating any protocol promises17:23
*** Ryan_Lane has joined #openstack-infra17:23
clarkblifeless: right but we are not affecting that stack, this is all pre semver in pbr17:23
clarkblifeless: with semver you would bump the version and always devN17:24
lifelessclarkb: yah, ok well I'll go back into lurk mode, still doing house cleanup17:24
dstufftclarkb: yea more or less, generally .postN is recommend not to include code changes, more like, doc fixes and stuff, but it's not a hard requirement17:24
*** gyee has quit IRC17:24
lifelessclarkb: someone is merging these changes into tip right?17:25
*** patrickeast has quit IRC17:25
*** gyee has joined #openstack-infra17:25
clarkblifeless: mordred has been shepharding it all17:25
clarkbhe has proposals up for the things that make sense against master17:25
lifelessok, I shall not stress17:25
mordredlook it's lifeless17:25
*** melwitt1 has joined #openstack-infra17:25
*** jcoufal has quit IRC17:25
mordredlifeless: yeah, don't stress - I've been merging on to your changes too - but I will want to have a non-rushed conversation with you about the reader class17:25
dhellmannclarkb, lifeless : so is correct or do we need another patch?17:26
mordredas there may be some new cases it needs to handle now17:26
*** e0ne has quit IRC17:26
clarkbdhellmann: it is correct for the pre semver pbr world which is why that change is on the "feature" branch17:26
*** melwitt has quit IRC17:26
dhellmannclarkb: k17:26
mordredwe can't do the full devX plan yet17:26
*** melwitt1 is now known as melwitt17:26
Sukhdev_anteaya: Ping17:26
mordredbecause that requires lifeless logic17:26
*** Sukhdev has quit IRC17:26
mordredso for a bit, we'll do post-release logic on top of pre-release releases which is valid pep44017:27
*** jpich has quit IRC17:27
lifelessmordred: ok17:27
lifelessmordred: you're at LCA right? Maybe we can grab 20m side by side17:27
anteayaSukhdev_: how can I help?17:28
mordredlifeless: yah17:29
*** hasharCall has quit IRC17:29
anteayaSukhdev_: I am right here and getting tired of pms17:29
anteayaplease talk to me in public17:29
mordredlifeless: although I _really_ want to get your changes into a release version of pbr sooner than that - that might be a pipe dream though17:29
anteayaso hopefully other people can read and learn from your question17:29
mordredlifeless: I do not like needing to make fake feature branches to release new features on top of old code base17:30
clarkbjeblair: is your plan to restart geard/zuul on zuul.o.o with that patch soon?17:30
lifelessmordred: well, can always mail me :)17:30
lifelessmordred: or make a time17:30
Sukhdev_anteaya: OK - did not want to clutter and bore others :-)17:30
mordredlifeless: bah. I expect you to telepathically know what I'm thinking!!!17:30
anteayaask away17:30
lifelessok, -> breakfast then house evisceration17:30
clarkbdhellmann: mordred we should port the distutils fix to master though17:30
clarkbif that hasn't already been pushed17:30
dstufftclarkb: docutils*17:30
clarkbdstufft: sorry type driver is failing me17:31
*** doug-fish has left #openstack-infra17:31
Sukhdev_anteaya: Want to set up a new meeting on one of the IRC channels17:31
dhellmannclarkb: that's right?17:31
clarkbdhellmann: yup17:31
dstufftdhellmann: yes17:31
dhellmannugh, cherry pick failed17:32
dhellmannI need lunch before I try that by hand.17:32
*** Kat has joined #openstack-infra17:32
Sukhdev_anteaya: you are very knowleageable about this stuff - was wondering if there is any guideline to follow17:32
*** Ryan_Lane has quit IRC17:32
*** achanda has joined #openstack-infra17:32
Sukhdev_anteaya: I see the list of meetings here
*** Kat is now known as Guest2572717:32
anteayajeblair: any objection to me adding a section to the infra manual about steps to follow to create a new meeting17:32
anteayaSukhdev_: right now I don't see a section in our infra manual17:33
anteayaSukhdev_: but you are demonstrating there is a need17:33
mordredclarkb: we should - but I do not have time to do that righ tnow17:33
mordredso I will next time I'm on laptop17:33
clarkbmordred: sounds like dhellmann might get it done after lunch17:33
*** wuhg has quit IRC17:33
anteayaSukhdev_: I'll create a patch, you can review the patch and give your feedback based on whether or not I was able to answer all your questions17:34
*** Guest25727 has quit IRC17:34
*** sarob has quit IRC17:34
clarkbI need food too and have a new laptop that needs to have new OS installed17:34
clarkbjeblair: so ping if you need/want help with gear things17:34
mordredclarkb, dhellmann: cool17:34
Sukhdev_anteaya: sure - but, for get me going for now - can you give me a hint?17:34
*** arxcruz has quit IRC17:35
Sukhdev_anteaya: I am having hard time trying to figure out which time slot is available on which channel and how do I reserve the slot so that there is no conflicts17:35
*** _nadya_ has joined #openstack-infra17:36
*** emagana has joined #openstack-infra17:36
*** shashankhegde has joined #openstack-infra17:37
*** Katherine has joined #openstack-infra17:38
anteayaSukhdev_: give me 10 minutes17:38
Sukhdev_anteaya: no worries…thanks17:39
*** avozza is now known as zz_avozza17:42
*** sputnik13 has joined #openstack-infra17:42
*** tkelsey has quit IRC17:44
*** sputnik13 has quit IRC17:44
*** AJaeger has joined #openstack-infra17:48
*** armax has quit IRC17:48
Katherinehi folks.  i'm hoping someone may be able to help me with a zuul issue i'm seeing.  i'm working with openstack anymore but am an hp employee trying to set up zuul in our ci environment17:48
*** gilliard is now known as gilliard_afk17:49
*** _nadya_ has quit IRC17:50
fungiKatherine: what's the issue?17:50
Katherinei have my environment hooked up well, it seems, and the zuul server and merger are running and getting stream events from gerrit17:51
*** jp_at_hp has quit IRC17:51
Katherinehowever, in my zuul debug log, i'm seeing:  2014-12-17 10:55:05,704 WARNING zuul.Scheduler: Project fusion not found17:51
Katherinei googled this and found some information but did not see a final resolution17:52
Katherinesince the scheduler doesn't recognize my project, gate jobs are not being triggered17:52
*** andreykurilin_ has joined #openstack-infra17:53
*** dannywilson has joined #openstack-infra17:53
*** thedodd has joined #openstack-infra17:54
Katherinesorry, realized i had a type in my intro.  "i'm NOT working with openstack anymore..."17:54
Katherinei'd originally worked with it on a former project here at hp17:55
*** ryanpetrello has joined #openstack-infra17:56
fungiKatherine: i believe that error message means zuul saw an event from the gerrit stream for a project named "fusion" but doesn't have an entry for it in the layout.yaml17:57
Katherineyes, that's the gist i got from the stream i found.  however, i do have an entry in my layout.yaml:17:57
*** Sukhdev has joined #openstack-infra17:57
Katherineprojects:   - name: fusion     check:       - zuul-gate-unittest17:57
*** sarob has joined #openstack-infra17:58
fungifor reference, it's coming from this method:
*** marcusvrn has joined #openstack-infra17:59
fungiso this means that self.layout.projects.get('fusion') is failing17:59
pleia2anteaya: if you haven't already, I'd ask ttx since he seems to be the meeting guardian17:59
pleia2not really sure it belongs in an infra manual17:59
*** shashankhegde has quit IRC17:59
fungipleia2: it may belong in infra-manual once the meeting schedule management tool is further along18:00
pleia2fungi: yeah, once our tool is running it might18:00
Katherinei found the code in the but couldn't glean anything else from it18:00
fungipleia2: at which point it will all be determined/changed via code review of configuration in an infra project18:00
*** derekh has quit IRC18:00
*** kgiusti has joined #openstack-infra18:01
Katherinehow can i figure out why the projects.get is failing?18:01
pleia2yeah, I've been following the progress of it18:01
pleia2hard nut to crack :)18:01
fungiKatherine: so is that the only error you're getting? it could be a log of things... is the layout.yaml readable by the user zuul's running under?18:01
*** bdpayne has joined #openstack-infra18:01
*** Sukhdev_ has quit IRC18:01
*** thedodd has quit IRC18:01
openstackgerritAnita Kuno proposed openstack-infra/infra-manual: Add a instructions for meetings
Katherinelol!  no, it's not.  good catch18:02
fungiKatherine: or maybe the yaml in that file isn't being parsed correctly due to a typo?18:02
fungiaha! file permissions18:02
fungithere you go18:02
fungiKatherine: chances are there were some more severe errors in your logs in that case18:02
jogowow bug 403510 just triggered a massive gate reset18:03
uvirtbotLaunchpad bug 403510 in launchpad-buildd "package maintainers in PPA uploads are what ever was uploaded, results in Debian developers receiving spam" [High,Triaged]
*** tonytan4ever has quit IRC18:03
fungijogo: missing a digit?18:03
jogobug 140351018:03
uvirtbotLaunchpad bug 1403510 in os-apply-config "gate tests failed with "NameError: name 'StandardError' is not defined"" [High,Confirmed]
anteayaSukhdev: see if you can wade through
Katherineno errors in the log.  just that one warning.  i change perms.  let me restart things18:03
anteayaSukhdev: if you have problems or questions, add them as comments to the patch so I can improve it18:03
Sukhdevanteaya: cool - let me review it18:04
jogoI don't see in zuul anywhere either18:05
anteayapleia2: thanks18:05
*** vdo has quit IRC18:05
anteayapleia2: yeah I don't know either but at least is is a jumping off point18:05
anteayasince i keep getting asked about it18:05
jeblairclarkb: i'm going to work on debugging the other problem18:05
jeblairclarkb: then make a decision about restarting gear18:05
*** dkranz has quit IRC18:05
jeblairanteaya: yeah, i don't think infra-manual is a good place for meetings right now18:07
*** dannywilson has quit IRC18:07
anteayaand I didn't mention the new tool yet, since I don't know what to say about it18:07
anteayajeblair: okay18:07
anteayajeblair: any redirect then?18:07
anteayaI can abandon and then just keep it to point people at who keep asking me18:07
*** dannywilson has joined #openstack-infra18:07
jeblairanteaya: the wiki?18:08
*** BobBall is now known as BobBall_AWOL18:08
*** resker has quit IRC18:08
Sukhdevanteaya: I did a quick read - it gives good steps, but, one thing which is tripping me is to have some kind of a view where I can see what slots are available on which channel18:08
anteayalet's see what ttx thinks18:08
anteayaSukhdev: yeah, we don't have that yet18:09
*** markmcclain has joined #openstack-infra18:09
anteayaSukhdev: it doesn't exist18:09
jeblairthere's the ical feed18:09
Sukhdevanteaya: If there was a tool (or a way) to see that, that will be really cool18:09
*** HeOS has quit IRC18:09
anteayaSukhdev: great18:09
*** gondoi is now known as zz_gondoi18:09
anteayathat tool is under development18:09
anteayaSukhdev: what to help develop it?18:09
*** shayneburgess has joined #openstack-infra18:09
anteayajeblair: there is, which I have never gotten to work for me, so I can tell people how to use it since I don't know myself18:10
Sukhdevanteaya: I used that iCal app and it messed up my calander on the mac - having hard time cleaning it18:10
*** markmcclain has quit IRC18:10
*** Guest51388 has quit IRC18:11
*** dannywilson has quit IRC18:12
Sukhdevanteaya: So, for now, I have to pick a time and work iteratively and see if it is taken, if yes, pick another and keep going!!!! Yikes :-(18:12
*** Ryan_Lane has joined #openstack-infra18:12
anteayathat is what I have to do18:13
Sukhdevanteaya: Thanks for such a quick response and putting the patch out. I will re-read and add the comments that I posted here on this channel.18:13
anteayaI honestly don't know what else to tell you in terms of booking a meeting18:13
Sukhdevanteaya: BTW, one question for clarification - say I figure out the time, and update the meeting wiki to add my meeting there18:14
anteayattx so I have composed some steps for creating and chairing a meeting and they need a home, how do you feel about me adding them to the meetings wikipage?
Sukhdevanteaya: how does it tie into the back-end? For example, when #startmeeting <meeting-name> is executed, how does the back-end know if it is a valid meeting or not?18:15
*** thedodd has joined #openstack-infra18:15
*** garyk has quit IRC18:15
anteayaany meeting that uses the #startmeeting command is a valid meeting18:15
anteayathere is no tie in between the bot and the wikipage18:16
*** Masahiro has joined #openstack-infra18:16
anteayathe meetings are booked on the honour system18:16
anteayaif you run a meeting and you haven't booked the space18:16
anteayayou will get the look of disapproval from others18:16
Sukhdevanteaya: Oh - so, if I misspell the name of the meeting, it will think it is a different meeting :-):-)18:16
anteayaand a new log category will be created18:17
Sukhdevanteaya: As usual, you have been very helpful - I knew you will provide some insight to this….and, you have done so….appreciate it, and big thank you….18:18
*** sarob has quit IRC18:18
anteayagood luck with your meeting18:20
anteayaand you are welcome18:20
*** marun has quit IRC18:20
*** Masahiro has quit IRC18:20
nelsnelsonGreetings, channel.  I'm trying to come up with an easy way to modify the tempest.conf file prior to the running of the tempest tests during a devstack-gate invocation run.  Is there currently a best way to accomplish this at present?18:21
tchaypojogo: i can’t see it either. I’m assuming that’s because of the gearman issues that jeblair/clarkb are (i think) working on18:23
tchaypoSukhdev: fwiw, the infra team can manually go into the servers and rename files for you if the meeting name is wrong18:24
tchaypoSukhdev: bug obviously that’s manual work and it’s going to make them sad, so try to avoid it if you can ;)18:24
*** shashankhegde has joined #openstack-infra18:24
Sukhdevtchaypo: Understood - thanks18:25
*** bdpayne has quit IRC18:25
*** ivar-lazzaro has joined #openstack-infra18:26
Katherinefungi, i must have also had a typo in my layout.yaml.  i simplified it and that fixed that issue18:26
*** dkranz has joined #openstack-infra18:26
Katherinethank you18:26
fungiKatherine: you're welcome--glad you managed to work it all out18:26
*** mpaolino has quit IRC18:27
jogotchaypo: ahhh makes sense18:28
funginelsnelson: devstack's lib/tempest does a lot of tempest configuration tweaking18:29
*** baoli has quit IRC18:29
nelsnelsonfungi, taking a look...18:29
funginelsnelson: looks like it starts from the sample tempest config, so you might be able to substitute your own...18:30
*** patrickeast has joined #openstack-infra18:30
funginelsnelson: ot18:30
*** luqas has quit IRC18:30
funginelsnelson: or it's possible you could make iniset calls in devstack through a custom plugin18:31
*** ryanpetrello has quit IRC18:31
nelsnelsonfungi, I am not familiar with iniset calls18:31
funginelsnelson: the calls you see in lib/tempest and lib/heat to change entries in $TEMPEST_CONFIG18:32
*** armax has joined #openstack-infra18:32
nelsnelsonoh i see now18:32
nelsnelsonfungi, yeah this looks good.18:33
jeblairfungi, clarkb: there is a network error involved in the set_description problem18:33
nelsnelsonfungi, I'm not sure how to create a custom plugin, tho18:33
funginelsnelson: it's possible we have some job examples where tempest configuration is tweaked18:33
fungijeblair: that makes a lot of sense actually18:33
*** zz_sabari_ is now known as zz_zz_sabari_18:33
*** SumitNaiksatam has joined #openstack-infra18:34
fungijeblair: so maybe we have some network connectivity problems with jenkins01 which are periodically triggering this behavior18:34
anteayaare there any guidelines for when someone can add instructions to install a new package on a devstack node?18:34
jeblairfungi: seems likely.  i'm still piecing the tcp events together18:34
anteayaI am sure jhesketh is doing the right thing, but I don't know what guidelines to follow in these cases18:35
anteayajust let people add packages if the commit message makes sense?18:35
*** marun has joined #openstack-infra18:35
fungianteaya: can you link the review you're talking about?18:36
anteayait isn't a question about this patch18:36
anteayait is about this kind of patch in general18:36
anteayaif there are no guidelines that is fine18:37
anteayaI was just wondering if there were18:37
fungianteaya: as with all patches of this kind, the trick is to find out why and if it can't be done another way, of course18:37
anteayaI will look into that18:38
fungifor example, in this specific case, i'm curious whether the upload can't run from a virtualenv with that python library installed18:38
fungievery python library package we preinstall on a worker has the chance to drag in others and potentially conflict with things we try to run from virtualenvs with system site packages enabled, et cetera18:39
anteayaI guess I don't understand why a virtualenv would be different from a node in this case18:39
jeblairfungi: i think we put zuul-cloner in a venv, right?18:39
anteayaright, hence my question about just tossing in another dependency18:39
*** salv-orlando has quit IRC18:39
fungianteaya: for a simple example, what if tempest also uses python-magic but isn't compatible with the version in debian18:40
anteayagood point18:40
fungijeblair: i believe so, yes18:40
anteayaso create a venv on the devstack node and install that lib within the venv18:40
anteayais that what you are saying?18:40
fungianteaya: if the uploader can be run from within a venv, then yes18:41
fungii can't immediately imagine why it wouldn't be able to18:41
anteayaI'll put that in a comment, thank you18:41
fungiand that has the benefit of isolating it from causing trouble for other jobs run on those workers18:41
jeblairanteaya: (to very specifically answer your question -- even though we don't have good guidelines for this kind of change, they can be tricky, so definitely don't just merge if the commit msg makes sense; asking for help is the way to go.  thanks for doing so!)18:42
fungiyep, when in doubt, ask. that's why there are multiple people collaborating on reviews ;)18:42
anteayajeblair: okay thanks18:42
anteayaI'd rather be told to be quiet and go away rather than make a mistake with a merge decision18:43
anteayafungi: thanks18:43
fungii'd rather not tell you to be quiet and go away. make more noise and ask questions18:44
jeblairand stick around18:44
fungithough also, if you do merge something which causes trouble, it's not the end of the world (we've all done it). there's always git revert18:45
*** Hefeweizen has quit IRC18:45
anteayaright I will be sure to let you know once I realize I have made a mistake when I do, so it can get fixed as fast as possible18:46
harlowjaqq, doesn't anyone have a sec to look into where went (it doesn't appear to be in zuul, wondering if it got lost somewhere)18:47
*** teran has quit IRC18:48
*** matel has quit IRC18:49
anteayafungi: can you -2 this oslo.version -> openstack-attic patch please?18:49
anteayafungi: no sense me doing it in case I am not around when the rename happens18:50
*** ryanpetrello has joined #openstack-infra18:50
anteayajust to prevent it getting merged prematurely18:50
*** tonytan4ever has joined #openstack-infra18:50
fungiharlowja: i think that's stuck in the zuul event queue backlog jeblair is debugging18:51
*** Sukhdev has quit IRC18:51
harlowjafungi thx18:51
fungianteaya: sure thing18:51
anteayafungi: thanks18:52
AJaegeranteaya: thanks for asking!18:52
anteayaAJaeger: :D18:52
anteayaAJaeger: we all get to learn new things18:52
dhellmannclarkb, mordred : I'm looking at that backport now and getting a weird error where LocalBuildDoc is being used before it is defined18:52
AJaegeranteaya: a lot - and it's difficult to keep up to date...18:52
anteayaAJaeger: true18:53
dhellmannclarkb, mordred : that's in the pbr unit tests, possibly because I have sphinx installed?18:53
fungiAJaeger: i've been core/root on the team for more than two years now and still continually find things i didn't know about18:53
fungithat's part of the fun! ;)18:54
dimsanteaya: fungi: thanks for the oslo.version attic review...we were not sure about the process when we filed the review :)18:54
*** esker has joined #openstack-infra18:54
*** achanda has quit IRC18:54
AJaegerfungi: if even you can't keep up, I'm giving up to try ;)18:54
fungidims: yeah, looks good. it's a low-priority task we'll batch up with other rename requests in a maintenance window since there's gerrit down-time required to do it18:54
anteayadims: it looks like you have everything right on the patch, we just have to make sure it merges during a rename window, which means a scheduled gerrit downtime18:54
anteayaAJaeger: dude, no way I can keep up with you18:55
dimsthanks a bunch, yes, absolutely +1 low priority18:55
anteayaAJaeger: but you are inspiring :D18:55
dimsanteaya: i told AJaeger the same :)18:55
*** achanda has joined #openstack-infra18:55
dimstoo fast!18:56
anteayadims: :D he is18:56
fungihe rocks the stackalytics and reviewstats data constantly18:56
anteayaso efficient18:56
anteayasomeone has to18:56
anteayaglad it is someone I get to work with18:56
*** markmcclain has joined #openstack-infra18:56
AJaegerstill, I miss a lot since I can't be the whole day on IRC like you are...18:56
fungihe and dims are apparently tied on right now18:57
dimsah, never saw that view :)18:57
dimsfungi: i see you tied as well!18:58
AJaegerfungi, you're there as well!18:58
fungidims: yes, proof we all spend too much time on computers i guess18:58
AJaegerhow is that metric calculated?18:58
*** koolhead17 has joined #openstack-infra18:59
fungino clue. i assume it's tarot cards and tea leaves18:59
*** gyee has quit IRC19:00
AJaegerthat would explain it - I expect that fungi spend more "real" person days on the project than I did. Just some of his work is not visible ;(19:00
clarkbdhellmann: I think that was the issue we were attempting to fix. maybe the code changed enough since 0.10.0 that the same fix does not apply?19:01
*** otter768 has joined #openstack-infra19:01
dhellmannclarkb, mordred : is approved in that feature branch but is still pending merge. When I "git review -d" it the tests fail with the same error.19:01
clarkbdhellmann: hrm maybe it is not a complete test /me tries to reproduce19:01
dhellmannclarkb: see my comment on
fungihuh, yeah it hasn't merited a check pipeline run yet19:02
*** afazekas has quit IRC19:02
dhellmannno tests have run in CI for that patch -- there are no comments by jenkins19:02
*** bdpayne has joined #openstack-infra19:03
clarkbdhellmann: oh so it still has an import time error19:03
dhellmannclarkb: yeah19:03
fungii get the same19:03
fungiin py27 even19:03
clarkbya reproduced19:03
fungiplaying with the patch real quick to see if i can come up with a relevant solution19:04
dhellmannfixing that error by changing the base class to add the "Sphinx" prefix gives me a different error where we are trying to call a None somewhere19:04
clarkbawesome I am going to go back to hiding behind "make new machine work"19:04
fungiyeah, saw your comment on the change19:04 is giving me internal errors19:04
clarkbwe should just untrove that instance19:05
dhellmannthe new error from pbr:
asselinany apache experts have tips to help out with mod_cache_disk?
*** tonytan4ever has quit IRC19:05
*** gyee has joined #openstack-infra19:06
*** otter768 has quit IRC19:06
anteayaopenstack-infra has a stackalytics page:
*** dannywilson has joined #openstack-infra19:08
anteayaAJaeger: to follow up on and to include mestery19:09
dhellmannclarkb, fungi, mordred : is the master branch of pbr broken?19:09
anteayaAJaeger: so yes you are correct in general we discourage tags and branches when creating new repos19:09
anteayaAJaeger: but this is not a general case19:09
anteayaAJaeger: this is a new repo created as a split from neutron19:09
fungidhellmann: broken with setuptools 8 still i think19:10
dhellmannah, ok19:10
fungidhellmann: but also has not-yet-completed semver work on it19:10
*** hdd has joined #openstack-infra19:10
fungidhellmann: basically we have inverted master and feature branches here19:10
anteayaAJaeger: and we are following the pattern we created when we did the neutron services split (lbaas, fwaas, vpnaas)19:10
dhellmannfungi: ok, I think I get it19:10
fungidhellmann: think of feature/0.10 as the actual master branch of pbr, and master would at the moment be more correctly a feature/semver branch19:11
anteayaAJaeger: and in those cases, we want the seed repo to have tags and branches if there were tags and branches from the code in neutron19:11
*** dannywil_ has joined #openstack-infra19:11
AJaegeranteaya: I wasn't aware of that. So far I worked on the assumption that split up repos were never released with these tags/branches, so there's no sense to import them.19:11
anteayaAJaeger: so both you and mestery are correct, but your comments AJaeger apply to the general case, which this patch is not19:11
dhellmannfungi: do you have any idea why zuul isn't running the tests for the feature/0.10 branch?19:12
anteayaAJaeger: yes sorry, this happened when I was at the neutron code sprint last week and clark and fungi were helping from infra19:12
AJaegeranteaya: and indeed the neutron-lbaas repo has the tags and branches19:12
anteayaAJaeger: monday and tuesday19:12
*** dannywilson has quit IRC19:12
dhellmannfungi: or at least that one patch, I guess, I haven't checked all of them19:12
AJaegeranteaya: you're ahead of me on this one ;)19:12
AJaegeranteaya: thanks for telling me.19:13
fungidhellmann: zuul has an event backlog, jeblair is debugging19:13
anteayaAJaeger: on this one yes19:13
dhellmannfungi: ok19:13
anteayaAJaeger: sorry I didn't think to tell you before19:13
* AJaeger removes his +2 for now and let you comment19:13
openstackgerritStefano Maffulli 'reed' proposed openstack-infra/system-config: Remove from the list of server
anteayaAJaeger: the patch itself looks fine and I'm fine with the history if mestery is, right now I am waiting on clean up19:13
AJaegeranteaya: no problem, I don't have a problem with learning and revisiting my comments when I get new knowledge. Thanks for telling me!19:13
anteayaAJaeger: they don't have the patches ready to merge to delete the split out code once the patch is merged, and they don't yet see the importance of ensuring the two patches (repo creation and code deletion) happen in quick successtion19:14
anteayabut we will get there19:14
anteayaAJaeger: thank you for understanding, and being so patient with me19:15
pleia2anteaya: hopefully a quick question, are there still plans to flesh out -manual with a new/useful version of jaypipes' blog posts? I'd like to help with review+testing of this work if so (plus, it'll help with my understanding of building one from the ground up, where now I'm only familiar with maintenance+management)19:16
anteayawell I defer to jeblair here19:17
*** btran has joined #openstack-infra19:17
pleia2I think there is a need to put it *somewhere* not-on-a-blog, since we get so many questions that I have trouble answering, and wish I could19:17
anteayasince the latest is that third party documentation gets directed to system-config/.. third-party.rst19:17
anteayame too19:17
anteayaI agree that somewhere not-on-a-blog would be good19:18
anteayaat this very moment, I have no confidence where that place is19:18
*** koolhead17 has quit IRC19:18
anteayabut I encourage you to champion finding out19:18
anteayaand to put the information in that yet to be discovered place19:18
anteayaand I will happliy point to it when asked19:19
pleia2anteaya: ok, I assume the third party folks don't  care, as long as it exists in a link we can point them to?19:19
anteayawell less that they don't care and more that it isn't their decision19:19
jaypipespleia2: ++. I will help with that effort, and I can try and get someone else at Mirantis to contribute to the effort as well.19:19
*** erlon has quit IRC19:19
pleia2great, so it's just up to us to find it a home19:19
anteayaand if we point them to it they can read it19:20
pleia2jaypipes: wonderful, I know you put some effort in previously so I really don't want to see it wasted again19:20
anteayajaypipes: thanks jay :D19:20
pleia2I'll stick it on the meeting agenda for now, will remove if we get an answer earlier than that19:20
jaypipespleia2: I just feel bad that you all have gotten questions on it and I have not been able to answer them :(19:20
anteayajaypipes: we all get so busy19:21
pleia2jaypipes: yeah, they come from all over the place so it's tricky to keep track unless you work on it full time19:21
anteayaglad to have your help whenever you can19:21
pleia2it == infra19:21
*** AJaeger has quit IRC19:21
fungidhellmann: trying to interpret the intent of mordred's patch, i think the LocalBuildDoc definition was meant to come between the SphinxLocalBuildDoc and SphinxLocalBuildLatex class definitions, but i'm still getting another error when i fix that19:21
anteayapleia2: yes putting it on the agenda can't hurt, then it won't get lost19:21
jaypipespleia2: you want me to give an initial stab at it in the docs repo?19:21
anteayajaypipes: well that is what we are just discussing19:22
anteayawhere is the best place19:22
dhellmannfungi: I'm trying another approach to solve the problem with some proper refactoring19:22
pleia2jaypipes: we're trying to determine whether to put it in ci. or under docs/infra-manual, so I'd hold off until we know which19:22
pleia2jaypipes: that saidy, they're both .rst, so you could do some prep and then move to land it whereever we select19:22
fungidhellmann: okay, cool. i've so far reached "ImportError: pbr.packaging.LocalBuildDoc"19:22
dhellmannfungi: oh, try adding global statements for the symbols that function modifies and see if that helps19:22
fungidhellmann: i did, but not after reordering. testing now19:23
jaypipespleia2: k, in the holiday week I plan to rework a number of things and do the ansible stuff I planned to do earlier, so I can adapt the blog posts into RST files at the same time.19:23
jaypipespleia2: shall we aim for early-to-mid January to make this a reality?19:24
pleia2jaypipes: perfect19:24
anteayahopefully we should know where it should go by then19:24
fungidhellmann: nah, but i'll see how far this rabbit hole goes19:25
*** bitblt has joined #openstack-infra19:25
jaypipeskrotscheck: Is the Login button on storyboard broken? I click on it and it just goes to!/auth/authorize and doesn't do anything.19:26
jaypipeskrotscheck: I'd log a bug about it... but ... can't log in :)19:27
krotscheckjaypipes: WFM - can you take a quick look at your browser console and see what kind of an error it’s throwing?19:27
jaypipessure, sec19:27
krotscheckAlso, what browser are you using?19:27
jaypipeskrotscheck: chromium:
krotscheckjaypipes: Ok, lemme bounce out to chromium and see if I can repro.19:29
pleia2jaypipes: does it work if you hit it again? (I think Chrome's caching does something weird so it works for me the 2nd time sometimes)19:29
jaypipeskrotscheck: here's all of it:
jaypipeslemme check a shift f519:29
jaypipeshmm, yep, that did it. a shift F5 worked.19:29
krotscheckjaypipes: When was the last time you used StoryBoard?19:30
jaypipeshmm, I think maybe a couple weeks ago?19:30
*** fifieldt_ has quit IRC19:30
krotscheckjaypipes: That might be it. I’m pondering there might be an edge case with expired refresh tokens.19:30
jaypipeswell, at least I know now what to do :)19:30
jaypipesthx guys19:31
krotscheckjaypipes: We did fix a couple of bugs in refresh tokens a few days ago, but that oculd have introduced somethign else. Either way, I’m glad we got you working again :)19:31
*** andreykurilin_ has quit IRC19:32
jaypipeskrotscheck: cheers!19:32
jaypipesand now I can go work on!/story/2000084 :)19:32
dhellmannfungi, clarkb, mordred : do we have a simple way to test a fix for this issue with sphinx and pbr?19:32
fungidhellmann: as in not running pbr unit tests?19:33
dhellmannfungi: yeah, because the unit tests have sphinx installed and I thought this was an issue with not having sphinx installed?19:33
fungioh, recreating the bug19:33
clarkbya its install a thing that installs docutils under python319:33
fungiseeing what i can come up with now19:33
morganfainbergfungi, clarkb, i'm going hold off on tagging new keystoneclient / middleware etc until i hear from you guys we're all clear from the stuff going on today (whatever it is) - i don't want to have a ksc release or something go out and have a huge delay to fix it if there is something wrong with it and breaking lots of people.19:33
clarkbbecause docutils relies on 2to319:33
dhellmannclarkb: so if I can run pbr's unit tests on py33 we should be ok?19:34
clarkbdhellmann: yes I think that should cover it19:34
clarkbor py3419:34
dhellmannok, cool, new patch incoming19:34
dhellmannwell, let me check my setuptools version before I declare victory19:35
morganfainbergnot that i expect something to be broken with client or middleware :P but you know... same reason i wont release on a friday if avoidable19:35
*** dims has quit IRC19:35
*** adalbas has quit IRC19:35
*** dims has joined #openstack-infra19:36
*** alexpilotti has quit IRC19:36
*** flwang has quit IRC19:37
*** hashar has joined #openstack-infra19:37
fungiclarkb: "NameError: name 'StandardError' is not defined" bubbling up from docutils/ is the hallmark of this bug, right?19:38
*** johnthetubaguy is now known as zz_johnthetubagu19:39
*** ChuckC has quit IRC19:39
dstufftfungi: yes19:40
*** dims has quit IRC19:40
*** salv-orlando has joined #openstack-infra19:40
fungimost interestingly, with 142509 checked out i still get that19:41
dhellmannwhat do I need to put in my .gitreview file to make it submit the patch against the feature branch?19:41
dhellmannis that option called "defaultbranch"?19:41
fungidhellmann: yes, or just type the branchname at the end of the command19:42
fungigit review feature/0.1019:42
dhellmannfungi: and
dhellmannclarkb, mordred : ^^19:42
dhellmannthe second one is the interesting one, with the fix19:43
dougwigdhellmann: defaultbranch=feature/lbaasv219:43
dhellmannthe new options module was needed to avoid circular import issues, but the rest of the changes should be pretty straightforward code shuffling19:43
clarkbcool I will look in a bit. I just about have new machine working the way I want it19:44
dhellmannthe solution was to put the sphinx command classes in a place where they can always be imported, and have commands refer to that only if sphinx is available19:44
dhellmannthat way commands never refers to a thing in a module that might be None19:44
*** dpyzhov has joined #openstack-infra19:45
fungidhellmann: makes sense. that was quick for such a significant refactoring19:45
*** salv-orlando has quit IRC19:45
fungitrying it out now19:45
dpyzhovHi guys. We have a really critical fix in merge queue. It was approved an hour ago. But it still not merged. What can be done in order to push it?19:46
dhellmannfungi: oslo has given me lots of practice at this :-)19:47
dpyzhov and
*** fifieldt_ has joined #openstack-infra19:47
*** e0ne has joined #openstack-infra19:47
*** SumitNaiksatam has quit IRC19:47
*** yamamoto has joined #openstack-infra19:47
fungidhellmann: strangely, i'm still reproducing the docutils exception with tox -epy34 (on debian, though shouldn't matter i think?)19:47
*** eharney has quit IRC19:48
dhellmannfungi: hmm, I upgraded setuptools in my venv and ran the tests but I didn't rebuild the venv at that point19:49
dhellmannwhere do you see the failure?19:49
fungidhellmann: docutils/", line 6819:49
dhellmannI mean what do you do to trigger it?19:49
fungidhellmann: tox -r -e py3419:49
dhellmannok, yeah, that's what I didn't do19:50
dhellmannlet me set up a cloud vm with latest setuptools and try that19:50
nibalizerpleia2: there you go!19:51
pleia2nibalizer: I don't know if this turns it into an action item, but it's something :)19:51
*** changbl has joined #openstack-infra19:51
*** yamamoto has quit IRC19:52
anteayaso armax and I were talking about neutron code split process19:53
anteayaand we were just trying to get our steps straight to ensure both ends were doing the agreed upon thing19:53
anteayaand I appreciate him putting in the effort here so we can get a process19:53
*** _nadya_ has joined #openstack-infra19:53
anteayanow the question of when the code needs to be deleted in neutron has come up19:53
*** HeOS has joined #openstack-infra19:54
anteayaI feel that part of my responsiblity to +2 or +3 the new repo creation patch with a seed repo of code split from neutron19:54
armaxanteaya: thanks, my assumption was that we’d create the project on stackforge, seeded with an existing code + history19:54
fungipleia2: nibalizer: the amusing part is not only does their tool come in about google search result #25 for "git-review" (the first 20+ of which are our tool) but even gets beat out by
armaxand then we’d go about taking on the steps of cleaning up the neutron repo and getting in sync with the external module19:55
anteayais to ensure there are clean up patches against neutron that have been reviewed, are ready to merge, and have an approver standing by to +A as soon as the new repo is successfully created19:55
marunanteaya: I'm not sure I agree with that approach.19:55
anteayaarmax is concerned that this will prevent the work from happening as he had thought that code clean up could happen when it was convienent for neutron19:55
marunanteaya: It's more likely that we'd need to iterate on the cleanup patch(es).19:55
anteayaincluding experiementing when the code was in both repos19:55
pleia2fungi: hah, yeah, I thought about making a crack about using a tool called "google" to search for existing project names ;)19:56
armaxanteaya: my understing was that we’d only need to ensure history was preserved and then that we could iterate on it19:56
marunanteaya: We're all adults here, I think we can and should be trusted to do the removal in a timely fashion rather than requiring it be immediate.19:56
dpyzhovfungi: can you help me or route to someone to help? we have hanged merge requests19:56
anteayaso we brought the conversation to the public channel to get some input19:56
pleia2fungi: but I wasn't grumpy enough19:56
armaxthe other question I had, is how much history should be preserved19:56
fungidpyzhov: zuul is backed up at the moment, currently being debugged19:56
armaxjust the one of master, or the one of master + stable + tags19:56
dpyzhovfungi: do you have any estimates?19:56
fungidpyzhov: no, i do not19:57
*** sarob has joined #openstack-infra19:57
ttxanteaya: ok... content sounds good. Maybe create a "how to run a meeting" page and link it from the top of the Meetings page ?19:57
dpyzhovfungi: ok, thanks. we’ll wait19:57
anteayamarun: it has nothing to do with maturity, and everything to do with infra responsiblity for repo accuracy and cleanup19:57
ttxanteaya: since that page should be autogenerated from openstack-infra/irc-meetings soon19:57
anteayattx I can do that, thanks19:57
armaxso in a nutshell, what to expect when one specifies an  ‘upstream’ repo on the project creation?19:58
ttxthat will allow to link to same page once that is done19:58
anteayattx great19:58
anteayaarmax marun I appreciate you coming here to talk about it as I think it needs to be discussed19:58
anteayaapparently right now I'm alone on this one19:58
marunanteaya: I'm not clear on what infra's responsibility is, exactly. Can you please explain and/or point me to documentation?19:59
marun(in this instance, of course)19:59
*** jcoufal has joined #openstack-infra19:59
anteayabut I do hope others will read backscroll and offer their thoughts19:59
anteayamarun: I don't know as we have documentation, more like axioms19:59
anteayalike "don't delete history"19:59
anteayaand I am assuming we have one that goes along the lines of "code exists in one place"20:00
marunanteaya: So is the concern that the cleanup patch has to match up with what is retained in the other repo?20:00
*** armax has quit IRC20:00
anteayathough I haven't heard that stated outright20:00
marunanteaya: to avoid the process being lossy?20:00
anteayamarun: yes, to fit the "code exists in one place" axiom20:00
*** armax has joined #openstack-infra20:00
anteayamarun: or to avoid divergence20:00
marunanteaya: I'm not entirely clear that this is necessary, though.20:01
anteayaand then a forced cleanup20:01
marunanteaya: The code we're talking about is going to be vendor-specific.20:01
anteayawhich is why we are discussing it here20:01
marunanteaya: It's not generally applicable or useful to the project as a whole - that's why we want to externalize it.20:01
*** markmcclain has quit IRC20:01
marunanteaya: So I'm not sure the same rules as community code need to be followed.20:01
marunanteaya: the services split was different in this regard.20:02
anteayasince part of the code split workflow I am familiar with it freeze the repo, split, seed new repo, delete old code, unfreeze20:02
anteayathen we need to come up with new rules20:02
anteayaso we know what we are doing and everyone is aware20:02
anteayabecause right now it sounds like we aren't20:02
anteayaand if the rest of infra is fine with code in two places20:02
anteayaI can go along with that20:03
anteayabut I don't want it to just be me20:03
*** ryanpetrello has quit IRC20:03
marunanteaya: understood.20:03
dhellmannfungi: what version of tox and virtualenv do I need to get setuptools 8?20:03
marunanteaya: I look forward to hearing input from other infra stakeholders.20:04
anteayame too20:04
*** reed has quit IRC20:04
anteayasince we don't have anyone chiming in at the moment, could I convince you to add an item to next weeks infra meeting agenda20:04
dstufftdhellmann: fungi note the py33 thing shouldn't be related to the setuptools version20:04
fungianteaya: marun: armax: to be blunt, i mostly just care that the desired end state is reached and that the process for getting there doesn't require any more direct assistance from the infra team than is absolutely necessary (and of course that community values aren't compromised to that end)20:04
*** Masahiro has joined #openstack-infra20:04
dhellmanndstufft: oh? I thought it was for some reason. Is this just a pbr thing?20:05
fungidhellmann: o20:05
fungidhellmann: i'm just running with tox 1.8.120:05
dhellmannfungi: ok, I'm not able to reproduce this issue.20:05
dstufftdhellmann: pbr added some code in response to setuptools 8, and that code breaks when trying to install docutils on py3, on pretty much any verison of setuptools20:06
dhellmannoh, wait, I wonder if that's because I have a bunch of wheels cached already20:06
fungidhellmann: in this case my tox's calling environment has pip 1.4.1 and setuptools 0.9.820:06
anteayafungi: are you fine with a new repo being created and old code staying in master for some indeterminate time?20:06
anteayaneutron master20:06
dhellmanndstufft, fungi : let me disable my wheel cache and see if that lets me reproduce it20:06
dstufftdhellmann: it won't be triggered if you have a py3 wheel for docutils20:06
fungianteaya: it's not my code. if that's what's convenient for the neutron developers (and if it doesn't bite them later requiring infra help to bail them out of some mess) then i don't think it's my place to care20:06
dstufftbecause there's no executing of in that case20:07
anteayaah it is the undefined "requiring infra help to bail them out of some mess" that I am trying to avoid20:07
anteayasince deleting code right away would most definitely avoid that20:07
tchaypopleia2: don’t forget that there was already a language called Go, and they were aware of that, but decided to go public with their language called Go anyway20:08
marunanteaya: moving vendor code out-of-tree isn't very risky.20:08
fungianteaya: yep, as long as they have contingency plans for dealing with people proposing driver changes to the wrong repo (and possibly getting merged, et cetera) then they're probably covered. just a matter of gettign all the neutron cores on board with an indefinite hard freeze on those files probably20:08
marunanteaya: it's not nearly as large a task a moving something like services out.20:09
marunanteaya: and I'm not sure I can imagine a scenario that would require infra intervention.20:09
marunanteaya: we have the power of git on our side, after all.20:09
pleia2tchaypo: heh, yeah, greghaynes posted the infamous bug #9 yesterday20:09
uvirtbotLaunchpad bug 9 in launchpad "Rosetta's po parser is too strict" [Medium,Fix released]
*** Masahiro has quit IRC20:09
bhunter71sorry to interrupt - a quick/dumb question; when a CI/jenkins server runs 'jenkins-jobs update' - a service account could be used, as set up in jenkins>credentials>global credentials?  My Jenkins is, otherwise, configured with OAuth/matrix authentication20:09
*** zz_gondoi is now known as gondoi20:10
pleia2tchaypo: hoping that they would be willing to change the name of a supplimental tool that just launched 2 months ago, even if they wouldn't rename their whole language20:10
*** rlandy has joined #openstack-infra20:10
fungibhunter71: yes, that's what we do20:10
bhunter71fungi: thanks much!20:10
armaxanteaya, fungi: so my question is: how much information we retain in the migration process to avoid any chance of bail out?20:10
fungibhunter71: we have openid authentication on our jenkins masters, but explicitly create a service account for jenkins-job-builder to interact with20:11
tchaypotheir “it was originally called review, git forces us to add the git- prefix” logic20:11
armaxanteaya, fungi: just master, or master+stable+tags?20:11
armaxanteaya: I think that’s the crux of the matter that I am trying to undersatnd20:11
tchaypoI can’t tell if they’re talking about our git-review, or their git-review20:11
armaxand how much of the history preservation process is automatic vs manual20:11
armaxI was working under the assumption that when the stackforge project was being seeded, *everything* was being kept20:12
*** aysyd has quit IRC20:12
armaxit does not seem to be clear on the creators’ manual...20:13
clarkbarmax: everything from the source repo is kept but you provide the source repo20:13
armaxI am not sure that’s intentional20:13
armaxclarkb: right20:13
armaxso everything is being slurped, it’s my understanding.20:13
clarkbeverything with a named ref yes20:13
clarkbso brnaches and tags20:14
armaxclarkb: thanks for clearing this up20:14
anteayaarmax: so you have what you need?20:15
armaxso, anteaya, fungi if we agree that we can keep the split process loosely a two stages effort20:15
dhellmannfungi: ok, now I'm getting the StandardError is not defined error20:15
fungidhellmann: wheel cache was mitigating it i guess?20:16
armaxfungi, anteaya: and we make sure we warn projects owners of the implications of their history preservation exercise20:16
armaxthen we should be good20:16
dhellmanndo you know how the pbr/hooks/ module is being triggered? I wonder if we can say "if you are already running the install command, disable sphinx"20:16
armaxI’ll stress this out on the docs I am putting together20:16
anteayaarmax: fungi said that you need a contigency plan for what to do if someone proposes a patch against code taht has been split out20:16
armaxanteaya: in Neutron you mean?20:16
anteayaarmax: yes20:16
fungii don't think that's something that needs to be written down anywhere, just make sure you keep it in mind20:17
armaxanteaya: I just have to behave like a bloodhound and -2 like crazy :)20:17
anteayaarmax: if you are willing to take that resposiblity, so be it20:17
fungipeople will accidentally propose patches to those files in the wrong repo for as long as they're still there, so yeah just got to be on top of them20:17
anteayawhat fungi said20:17
anteayathat is my concern20:17
fungiand if your core review team is large enough, some of them probably will get approved20:17
anteayaso remove code or -2 proposed patches20:17
*** flwang1 has joined #openstack-infra20:18
anteayawhatever it takes20:18
armaxfungi: agreed20:18
anteayaand remove code is my preference20:18
armaxit makes perfect sense to be very diligent during the process20:18
armaxanteaya: there’s been ample warning of a soft freeze on vendor code20:18
fungibut it's a balancing act. i'm definitely not going to try to tell teh neutron devs what workflow is best for them, just pointing out potential gotchas20:18
anteayaarmax: okay20:19
tchaypodhellmann: fungi: do we know which change in pbr caused this? is it something we could fix by reverting a change?20:19
anteayabut I think you are able to hear my concerns now20:19
dhellmanntchaypo: I have no idea20:19
fungitchaypo: it's all tripleo's fault ;)20:19
anteayawhen I wasnt' sure that message was getting across before20:19
armaxso once the split effort has been initiated it’s up to the maintaner to retain consistency, and I would imagine the same person who takes care of the split and possible bug fix to understand what’s going on20:19
anteayaarmax: warnings don't protect you at all20:19
fungitchaypo: cd people wanting to keep track of git shas in packages. setuptools 8/pep 440 makes that hard, so mordred came up with embedding teh data in egg-info20:20
armaxanteaya: agreed20:20
anteayaarmax: it just gives you something to point at when people question why you -2'd something20:20
fungitchaypo: we can revert that, but then we're back to not being able to run our test infrastructure with setuptools 820:20
anteayaarmax: ah, I would take responsiblity yourself, at least to ensure the split is clean20:20
anteayaI am learning to not depend on anyone else20:20
anteayaeven nice people20:21
dhellmannfungi: I don't think I realized how much time pre-building wheels was saving me running unit tests.20:21
*** reed has joined #openstack-infra20:21
armaxanteaya: fair enough…I’ll capture these notes20:21
clarkbdhellmann: wheels are pretty awesome20:21
anteayaarmax: great thanks for your persistence20:21
anteayaarmax: I appreciate it20:21
armaxas for gary’s patch, I think it’s fair to signal him the fact that is currently seeding the history just for master20:21
fungitchaypo: dhellmann: as dstufft pointed out, it's pbr tweaking egg-info which causes us to have non-translated docutils loaded into the execution space20:21
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Use threads to update jobs in jenkins
*** kevinbenton has quit IRC20:22
anteayaarmax: if that it is what it is doing20:22
anteayaarmax: to be honest I don't know20:22
armaxand he’s not retaining any tags, if he wants to be conservative, he should revise that, otherwise he gets only one shot at doing it20:22
dstufftfungi: to be specific, it's the fact that pbr added a specific entry point, and setuptools imports pbr as part of loading that entry point20:22
anteayaarmax: conservative is my favourite way20:22
dhellmannfungi: is that then?20:22
anteayaarmax: and I want to take some time for the conversation to steep20:23
anteayaarmax: we can talk again tomorrow20:23
armaxas clarkb pointed out the seeding captures everything, so I’d be temped to suggest him to exploit this opportunity20:23
fungidhellmann: yep, that one20:23
anteayaarmax: I encourage your suggestion20:23
armaxanteaya, clarkb, fungi: thanks for the input, much appreciated20:23
anteayathanks armax and marun20:23
clarkbdstufft: dhellmann fungi so to fix this can we split the import paths for egg info and sphinx? so that when setuptools imports nothing bad happens?20:23
dhellmannfungi: and it's just the fact that we *have* an egg_info.writer plugin?20:24
tchaypofungi: my head hurts even more.20:24
dstufftdhellmann: the fact you have that causes you to be imported20:24
dhellmannclarkb: that's sort of what my patch does, except it tries to figure out if sphinx is present20:24
dstufftthe fact pbr has that causes pbrt o be imported by*20:24
dstufftimporting pbr imports sphinx which imports docutils20:24
*** salv-orlando has joined #openstack-infra20:24
dhellmannoh, hmm, we were only catching ImportError, but that's not the error being thrown20:25
fungiso maybe we can factor out just the bits which need to write egg-info into a separate module and only import that for the entryoint?20:25
clarkbfungi: ya I think that is what I was trying to sy. you said it much better20:26
dstufftyes that's a good way to fix it20:27
fungijeblair: clarkb: jenkins01 is long-idle at this point. can i restart it, or are we waiting on further diagnostic data to be collected?20:27
jeblairfungi: i think we should just keep it offline for now20:27
dhellmannfungi: it's not that code that's causing the problem, it's the entry point scan itself20:27
dhellmannfungi: I think i see how to fix it, give me a few minutes to test20:27
jeblairfungi: (i suspect brining it online may allow the problem to recur)20:28
dstufftyou can probably also paper over the bug just by expanding the scope of exceptions which you considder for marking sphinx support as no20:30
dstufftdhellmann: speaking of wheels saving time, it won't be in 6, but hopefully pip 7 will have an on by default automatic wheel cache, so that if you install something that doens't have a wheel, it builds a wheel the first time, caches it, and then uses that in the future20:32
dhellmanndstufft: nice!20:32
dstufftthat'll probably break a few things too20:32
dhellmanndstufft: yeah, I'm going to convert any exceptions importing sphinx into ImportError20:32
tchaypoon a related note - it sounds like mordred’s patch doesn’t do what we need, so20:34
clarkbtchaypo: hrm?20:34
dstufftwhich patch20:34
dstufftthe one that broke py33? it does something that you need, it just introduced another bug that only effects installing docutils20:35
tchaypoperhaps we should mark it as such20:36
tchayposorry, got halfway through typing, switched windows to check must have hit enter as i was switching windows20:36
tchaypoI was talking about change 14250920:36
anteayaianw: if you are about can you help me understand why we need a whole new job here rather than making use of the -20 job: can we make the -{version} an option on that job and use it for both?20:37
tchaypowhich seems to have been superseded by and will shortly be super-duper-seded by something else20:37
*** rmcall has quit IRC20:37
*** zz_zz_sabari_ is now known as zz_zz_zz_sabari_20:37
*** _nadya_ has quit IRC20:39
*** rmcall has joined #openstack-infra20:39
*** adrian_otto has joined #openstack-infra20:40
*** baoli has joined #openstack-infra20:41
adrian_ottohey, I have a question… I just tagged a release 2014.2.2 of solum and 1.2.2 of python-solumclient, but I'm not seeing any jobs show up under "release" at and no tarballs showed up in and respectively. Did I miss something?20:42
dhellmanndstufft, clarkb, fungi : something in pbr's test requirements list is installing a released version of pbr, which still has the problem, and which is preventing the fix in the version of pbr I'm *testing* from being used instead20:44
fungiadrian_otto: zuul is working through events very slowly at the moment. there's an event queue ~500 deep right now, so it's probably in there somewhere20:44
dstufftdhellmann: that would probably explain why it wasn't caught20:44
adrian_ottofungi: thanks! Is there anywhere I can look to see the queue?20:44
dhellmanndstufft: I'm fiddling with the tox.ini to see if I can force the local copy of pbr to be installed before anything else20:44
fungiadrian_otto: counts at the top left of
clarkbdhellmann: can you at least confirm the fix works by manually installing that version of pbr in the venv then rerunning tests?20:45
clarkbdhellmann: you may b eable to do that via the install command20:45
fungidhellmann: that explains why i was suddenly reproducing it on older pbr versions--i was beating my head on teh keyboard trying to figure out what i was doing wrong there20:45
clarkbor evn listing . in the deps list20:45
*** unicell has quit IRC20:45
dhellmannclarkb: the failure is triggered when installing docutils into the test environment. I can try redoing the steps by hand after this run completes20:45
dhellmannfungi: yeah, it took me a while, too, until I looked very closely at the traceback20:46
clarkbif putting . in the deps list works that would be amazing20:46
dhellmannclarkb: that's what I'm trying now20:46
*** markmcclain has joined #openstack-infra20:46
*** gondoi is now known as zz_gondoi20:46
dhellmannrunning 2to3 on docutils takes forever20:47
dhellmanndstufft: while we're talking about how nice wheels are, I'd like your input on
*** zz_gondoi is now known as gondoi20:48
dstufftdhellmann: you know you can just do pip wheel -r global-requirements.txt ?20:48
*** achanda has quit IRC20:49
*** r-daneel has joined #openstack-infra20:49
*** achanda has joined #openstack-infra20:49
dhellmanndstufft: some of our requirements can't build wheels properly, and when pip hits one of those it stops. This installs everything that works right, and ignores errors for packages that can't build wheels.20:49
dstufftdhellmann: ah20:50
dhellmannclarkb, fungi, dstufft : new version of ready for review20:50
*** jcoufal has quit IRC20:50
dhellmannadding "." to the deps list seems to work20:51
*** kevinbenton has joined #openstack-infra20:51
fungidhellmann: clarkb: ingenious20:52
fungii'm confirmign and checking for regressions now20:52
*** marun has quit IRC20:52
clarkbI am running tests locally too also I love that . works20:53
*** jcoufal has joined #openstack-infra20:53
*** achanda has quit IRC20:54
clarkblooks like pep8 may fail?20:54
*** dustins has quit IRC20:54
fungii hadn't tried that yet20:55
fungipy33 is happy though20:55
*** spzala has quit IRC20:55
*** Katherine has quit IRC20:55
*** dustins has joined #openstack-infra20:56
dhellmannlet me see about pep820:56
fungipy34 is working for me too20:57
fungiand py2720:57
fungii don't have py26 handy on my workstation any longer20:57
clarkbya 34 and 27 work for me20:58
dhellmannI forgot zuul was backlogged, I'll run all of the tests locally here20:58
clarkbbut pep8 is cranky20:58
*** bdpayne has quit IRC20:58
*** dims has joined #openstack-infra20:58
clarkbso now I just need to review it20:58
*** thedodd has quit IRC20:58
*** ecosta has joined #openstack-infra20:59
dhellmannhmm, I'm seeing errors in py26 because it doesn't like the testrepository database format21:00
dhellmannI wish testr would use a version-specific directory for that database21:00
clarkbdhellmann: the fix for that is to always run py33 first iirc21:01
clarkbbut still very annoying21:01
fungigit clean -dfx ; tox -e py2621:01
fungiworks for me anyway21:01
dhellmannclarkb: yeah, I usually set up projects that way21:01
*** mancdaz has quit IRC21:02
anteayaclarkb: you did a gerrit group cleanup last time we had an oops, this isn't really an oops but armax found a non compliant acl file whist trying to implement a job to test acl format:
*** otter768 has joined #openstack-infra21:02
anteayaclarkb: do you think this should be merged and cleaned up manually to avoid an orphaned group?21:02
*** mancdaz has joined #openstack-infra21:03
jeblairclarkb, fungi: so jenkins kernel retransmitted a dropped packet, and the second time it made it.  the sequence was "grab_job_uniq, job_assign_uniq, work_complete, grab_job_uniq" with the work_complete being the dropped packet.21:03
fungianteaya: "vmt" is the vulnerability management team21:03
fungianteaya: it has no subsets21:03
anteayaah so that would be no21:03
*** thedodd has joined #openstack-infra21:03
jeblairclarkb, fungi: i'm assuming that just means tha geard received "work_complete, grab_job_uniq" all at once21:03
anteayaso we have to allow for an exception for this file in the test21:03
anteayafungi: thanks21:03
clarkbdhellmann: is that options code really (C) AURA? (just double checking as it jumped out at me as odd)21:04
fungijeblair: could it have received those in an unexpected order?21:04
jeblairclarkb, fungi: i netcatted those two into geard, and it processed it fine...21:04
*** dprince has quit IRC21:04
jeblairfungi: i think tcp will take care of that, right?21:04
fungijeblair: yep, my brain was on datagrams for a moment, but there would be no retransmit21:04
fungijeblair: definitely would be received in the order sent, anyway21:05
dhellmannclarkb: I don't know; I copied the copyright settings from the file where it used to live. Isn't that our policy? I can remove them entirely, if that's better.21:05
jeblairfungi: yeah, so i'm assuming it was in the correct order and all at once (as soon as the missing piece was supplied)21:05
clarkbdhellmann: oh thats from over there? good neough for me21:05
jeblairbut i can't find anything about that which would cause geard to stop processing that connection21:05
fungiclarkb: yeah, it was a refactor, that code just moved21:05
jeblairclarkb, fungi: i've got one more data point i can try to triangulate with.  waiting for wireshark to load it.21:06
anteayathat might be my first -2 that is meant as a minus 2 rather than a don't merge this by accident21:06
*** __TheDodd__ has joined #openstack-infra21:07
*** otter768 has quit IRC21:07
*** ecosta has quit IRC21:08
clarkbanteaya: ya that will need coordination21:08
clarkbanteaya: because wherever possible we should avoid orphan groups21:08
clarkbjeblair: rgr standing by for whatever next course of action is21:09
anteayaclarkb: actually fungi kyboshed that patch, since vmt (vulnerability mangagement team) will not become a vmt-core gerrit group21:09
*** thedodd has quit IRC21:09
clarkboh even better21:09
anteayaclarkb: so I -2'd it21:09
clarkbanteaya: perfect21:09
dhellmannclarkb, fungi : I'm seeing "db type is dbm.gnu, but the module is not available" under py33 now. Do you know what package I need off the top of your head? This is 14.04.121:09
dhellmannI would have thought libgdbm3, but that's installed21:10
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Add nodepool script to preseed testrepository from subunit2sql
jeblairmtreinish: ooh how exciting! ^21:10
fungidhellmann: git clean -dfx and rerunning doesn't make it go away?21:10
clarkbdhellmann: I don't know sorry21:10
dhellmannyeah, I'm doing that now, i was trying to just fix up my box so running this in the right order would work21:10
mtreinishjeblair: yeah it'll be cool to get that running. (although I doubt it'll make a noticeable performance improvement)21:11
clarkbok so I grok this change. basically make options a thing to avoid importing when not necessary. and put the docutils things off to the side21:11
fungidhellmann: i thought that was just one of the usual symptoms of running with an existing testrepository db created by a previous run under a different interpreter21:11
anteayaI'll stop shoving stuff at zuul for a bit and go for a walk21:11
dhellmannfungi: it is, but depending on what you have installed and what order you run, you get a different dbm format21:12
clarkbjeblair: by the way gertty sync queue is over 1k on ne wmachine :)21:12
dhellmannthat's a *terrible* way to create a database :-/21:12
jeblairclarkb: we should review some patches :)21:12
openstackgerritAdrian Turjak proposed openstack-infra/project-config: Add new Openstack-Pydashie project
clarkbjeblair: indeed21:13
jeblairclarkb: and some of that (~half) will end up being quick no-ops.21:13
clarkbif only these fires would go away21:13
mtreinishjeblair: acutally I had a qq on that patch, I'm not sure of the best way to inject that script into the image creation process. Any suggestions?21:14
fungidhellmann: `tox -evenv -- python build_sphinx` is not looking so good...21:14
clarkbmtreinish: if you make the script a nodepool script it iwll happen automagically21:14
*** mwhudson_ is now known as mwhudson21:14
mtreinishclarkb: oh cool, I put it project-config/nodepool/scripts21:15
dhellmannfungi: ok, I'll test that one, too21:15
fungidhellmann: i think i see the issue21:15
clarkbfungi: oh good idea21:15
*** teran has joined #openstack-infra21:15
fungidhellmann: inline comment fixes it21:16
*** mwhudson has quit IRC21:16
*** sweston is now known as spw21:16
dhellmannfungi: ok, pep8 caught that but I'm still re-running the other tests21:16
*** amitgandhinz has quit IRC21:16
dhellmannit's a little disturbing that running the tests didn't catch missing imports in that file21:17
fungidhellmann: oh, i hadn't tried pep8. and yes that's definitely pointing to a coverage issue21:17
clarkbdhellmann: well you ignore import errors :)21:17
*** sweston has joined #openstack-infra21:17
fungihrm, there's that :/21:17
dhellmannclarkb: it means nothing is trying to use that code in the tests, though21:17
clarkbdhellmann: ya21:18
clarkbdhellmann: however we should test it now via building docs and your .21:18
clarkbso not unittested but will be covered once this change goes in21:18
*** mwhudson has joined #openstack-infra21:18
dhellmannclarkb: the tests take ages because each env has to wait for docutils to run through 2to321:18
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers.
*** mancdaz has quit IRC21:19
openstackgerritSpencer Krum proposed openstack-infra/system-config: Bump rabbitmq module to 5.0.0
*** mancdaz has joined #openstack-infra21:19
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers.
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added manifest for
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added manifest for
dhellmannfungi, clarkb, dstufft : new version of
dhellmann"tox -r" and "tox -e venv -- python build_sphinx" are all green21:25
openstackgerritRamy Asselin proposed openstack-infra/project-config: Prepare project-config for puppet module split #1
jaypipesjeblair: so I spent a couple hours this afternoon working on making gertty a bit zippier. :) should have a patch for you later tonight. basically, I've reworked a few of the sync tasks to not take the DB thread lock for as long, and modified the list changes view to not need to do a full refresh() when a change is toggled reviewed/hidden. sync events at startup and during use seem to be noticeably snappier.21:26
fungidhellmann: aha, yeah i can see where stringio would have gone unspotted21:26
clarkbdhellmann: woot, I am going to read through it again. I do think you will have to flip the order of your two commits though21:26
clarkbdhellmann: since the gitreview update will fail tests21:26
fungiclarkb: i think we probably just need to ditch the .gitreview addition. it will only be one more potential gotcha come time to do the merge commit21:27
clarkbfungi: that works too21:27
clarkbalso we should rebase mordred's postN change atop dhellmann's change21:27
dhellmannclarkb: ok, I can flip them21:28
jeblairjaypipes: awesome!  thanks!21:28
jaypipesjeblair: hey, thank YOU for an awesome tool :)21:28
dhellmannfungi: oh, or I can drop it21:28
clarkbdhellmann: maybe just drop it like fungi suggestions? you can edit the file and just not commit it21:28
fungii honestly hadn't expected this "feature" branch to stick around so long21:28
clarkbdhellmann: git review doesn't need the edits to be committed for you to make use of them21:28
clarkbfungi: ya21:28
clarkbwow it is time for lunch already21:29
clarkblet me do that before I try to review code21:29
fungiit was really just supposed to be a quick hack to backport a patch while the semver bits were being polished up21:29
dhellmannfungi, clarkb : dropped21:29
asselinnibalizer, any idea where these modules are? puppet-openssl puppet-puppetboot21:29
dhellmannfungi: maybe we need to find a good place to create a "feature/semver" branch, then drop that stuff from master?21:30
*** yjiang5_away is now known as yjiang521:30
fungidhellmann: that's what i would have proposed had i realized we'd need to do so much more of this. but to do that in a fast-forward-friendly manner would involve a lot of reverts on master21:30
* dhellmann nods21:31
fungi(which mordred had started down the path to initially)21:31
fungii think his though was just an inverted diff between master and the last tag21:32
*** Rockyg has joined #openstack-infra21:32
*** andreykurilin_ has joined #openstack-infra21:33
dhellmannI suppose renaming the existing branches would just lead to all sorts of pitchforks and brimstone21:34
fungidhellmann: yep, it would cause some real messes on people's existign clones21:34
fungiso at this point i'm sort of just hoping that this is the last of the backport issues, and then the semver stuff on master gets sorted to some conclusion quickly21:36
fungiand then we learn from the experience21:36
*** Shrews has quit IRC21:37
*** Hefeweizen has joined #openstack-infra21:38
*** julim has quit IRC21:38
*** rmcall_ has joined #openstack-infra21:38
*** __TheDodd__ has quit IRC21:39
*** rmcall has quit IRC21:39
*** rmcall_ is now known as rmcall21:39
dhellmannfungi: I've learned we need to use feature branches more, and that we need someone to actually own this library and manage its development instead of having several of us working on it part time.21:41
fungidhellmann: yeah, at least the semver implementation was enough individual behavior changes that they probably shouldn't have been just dripping into master bit by bit21:42
openstackgerritBen Nemec proposed openstack-dev/hacking: Remove Git commit message checks
openstackgerritBen Nemec proposed openstack-dev/hacking: Remove complex import rules
*** flwang1 has quit IRC21:42
*** mrunge has quit IRC21:42
jeblairclarkb, fungi: i'd like to restart geard with extra logging but without stopping zuul.  i believe i can do that by killing the process and then restarting by hand.  i will save queues beforehand just in case though.21:44
jeblairclarkb: i don't want to apply my change yet21:44
nibalizerasselin: hrm?21:44
nibalizeri might have moved the puppet-openssl module into openstack_project21:44
asselinnibalizer, those two modules are in the puppet split spec, but not in system-config21:44
*** marun has joined #openstack-infra21:44
asselinnibalizer, ok, and puppet-puppetboot?21:45
*** chuckC_ has joined #openstack-infra21:45
fungijeblair: okay, that sounds like a winner if it will work21:45
asselinnibalizer, link/ref to puppet-openssl move?21:46
dstufftzuul's queue is massive21:46
openstackgerritDoug Hellmann proposed openstack/requirements: Add developer script to build wheels
*** zz_zz_zz_sabari_ is now known as sabari21:48
fungidstufft: it's been worse, but not in a while21:48
openstackgerritJoe Gordon proposed openstack-dev/hacking: Add docstring to is_import_exception
clarkbjeblair: ok21:49
*** amitgandhinz has joined #openstack-infra21:49
nibalizerasselin: looks like i refactored openssl in ff007f29a2d8fe834cb1caa638ee0c848241e65a21:49
asselinnibalizer, noted21:50
clarkbshould I go ahead and rebase mordred's postN change?21:50
clarkbfungi: dhellmann ^?21:50
dhellmannclarkb: yeah, good idea21:51
fungiclarkb: yep21:51
*** amitgandhinz has quit IRC21:52
*** amitgandhinz has joined #openstack-infra21:52
*** mbacchi has quit IRC21:52
dstufftso with the postN change and the py3 fix21:53
dstufftpytz and warnings are what's left that I know of21:53
*** Masahiro has joined #openstack-infra21:53
david-lyleI pushed tag 1.1.8 to openstack/django_openstack_auth a week ago, it's visible on, seems like the publish job to pypi failed, is there an easy way to re-trigger the publish?21:53
nibalizerasselin: it looks like mordred killed puppetboot in c8a29bea6dde8bf54a82c47bd8f014e142a0d58e21:53
jeblairclarkb, fungi: i believe this is the first time we've swapped out a running geard without disrupting zuul21:54
dstufftdhellmann: for setuptools 821:54
asselinnibalizer, noted. thank you!21:54
dhellmanndstufft: ok, so not in pbr?21:54
fungidhellmann: non-pep-440-compliant versions of things preinstalled from distro system packages21:54
dstufftdhellmann: nah, pbr should be fine after this, as far as I know21:54
nibalizergit bisect ftw21:54
fungijeblair: it worked?!?21:54
clarkb142351 has been reparented21:55
dstufftpytz is 2012d on fedora, and Babel has a dep on >=0a which makes it not accept 2012d21:55
dstufftso setputools 8 pkg_resourced blows up21:55
fungijeblair: so you just killed the process for geard and invoked it directlt?21:55
jeblairfungi: so far... i expect it to have to restart some jobs, but i waited until _most_ jobs were finished, so hopefully that's a minimum21:55
clarkbdstufft: ya so to fix pytz I think we can just upgrade our version on our slvae images21:55
dstufftand ubuntu has some random stuff installed with non pep 440 versions, which cuases a warning to be output when pkg_resources is imported, and some tests execute a subprocess and assert against the stdout of that process21:56
clarkbdstufft: or do I not fully understand the issue?21:56
dstufftclarkb: that should do it yea21:56
dstufftclarkb: it might effect devstack in not the CI infra though21:56
clarkbdstufft: ya21:56
nibalizerclarkb: gah the git ui21:56
jeblairclarkb, fungi: i'm going to put jenkins01 back online in the hopes that it malfunctions again21:57
nibalizerits git bisect start then git bisect reset21:57
fungijeblair: okay21:57
dstufft 1.3-7 has fedora fixed for the pytz issue21:57
nibalizerstop would have been just wayy to obvious21:57
dstufftbut it's only in testing right now21:57
clarkbjeblair: ok, but you haven't applied any changes at this point right?21:57
clarkbjeblair: just new geard21:57
fungimore log detail?21:57
*** dkliban is now known as dkliban_afk21:57
jeblairclarkb: correct.  still running same gear version as before, just more logs21:57
*** Masahiro has quit IRC21:58
*** sarob has quit IRC21:58
*** dannywilson has joined #openstack-infra21:59
clarkbdhellmann: so I cna whip up a change to fix pytz on fedora as soon as I properly review your change which I am doing now21:59
*** changbl has quit IRC21:59
*** dizquierdo has joined #openstack-infra21:59
*** rmcall_ has joined #openstack-infra22:00
dstufftclarkb: for the record, 2013.6 is the first PEP 440 version of pytz22:01
*** rmcall has quit IRC22:01
*** sarob has joined #openstack-infra22:01
*** rmcall_ is now known as rmcall22:01
fungiclarkb: switch it from distro package to pip?22:01
dhellmannfungi, clarkb, dstufft, mordred : is there a list of the setuptools 8 fallout that we're fixing?22:01
*** dannywil_ has quit IRC22:02
*** kgiusti has left #openstack-infra22:02
fungidhellmann: i think the most recent "list" if you can call it that is my summary to the ml from monday22:02
dhellmannfungi: ok, dstufft had me worried there was another surprise coming :-)22:02
clarkbfungi: ya was just going to do pip install pytz with the version string in requirements22:03
dhellmannclarkb: do we have control over pytz releases?22:03
clarkbdhellmann: not directy, but we can use pip to install a working version into the base os22:03
fungioh, right, we have "pytz" in global-requirements.txt22:03
*** mattfarina has quit IRC22:03
dstufftI made a global requirements thing to upgrade that to pytz>2013.6 last night, dunno if that's the right solution22:04
*** dims has quit IRC22:04
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Extend ACL checks to OpenStack namespace.
pleia2nibalizer: working on a new puppet module and I noticed we get a pile of "Setting templatedir is deprecated." warnings, know if we're tracking this, or should be fixing? (I can have a look if we should and aren't)22:04
dstufftit failed but I don't understand the failure22:04
dhellmanndstufft: link?22:04
clarkbI have approved the pbr stack22:05
nibalizerpleia2: hopefully we've found all the places that are setting templatedir in puppet.conf22:05
nibalizerit could be just that your local puppet install has it set and shouldn't22:05
clarkbmtreinish: is the failure on 142348 familiar to you?22:05
dhellmanndstufft: that looks like an unrelated error at first glance, unless one of the servers is actually breaking on a pytz call somewhere22:06
*** pcrews has quit IRC22:06
pleia2nibalizer: aha, I'm getting them from /usr/lib/ruby/vendor_ruby/puppet/settings.rb which maybe we can't do anything about?22:07
*** amitgandhinz has quit IRC22:07
dstufftdhellmann: that's what I thought, but I wasn't sure22:07
dhellmanndstufft: I see a timeout error in the n-api screen log22:07
nibalizerpleia2: no its just beause you have templatedir set in your /etc/puppet/puppet.conf22:07
nibalizerat least, i think thats the issue22:07
*** mriedem has quit IRC22:08
*** rlandy has quit IRC22:08
pleia2nibalizer: ah, so I do, where did we fix this? I'm running fresh trusty install + instructions at
mgagneI'm having issues with latest pbr when git is not installed.22:09
clarkbmgagne: can you be more specific?22:09
dstufftclarkb: I wonder if ^ is the problem from openstack-dev22:09
mgagneI found this commit which would address my issue: but latest version includes a commit which reverted this change:
*** marun has quit IRC22:09
mgagneclarkb: getting File not found when installing pbr.22:10
*** jcoufal has quit IRC22:10
*** marun has joined #openstack-infra22:10
mgagneclarkb: the issue was identified by one of our devs so info is second hand.22:10
clarkbmgagne: gah I see22:10
fungimgagne: we haven't released from master for a while. we've been releasing minor backports22:10
clarkbI will write a patch for that22:10
mgagneclarkb: maybe issue is fixed but not released22:10
clarkbfungi: no mordred derped22:10
clarkband me too becuase  reviewed22:10
mgagne0.10.3: 20 hours ago22:10
fungimgagne: that release was based off 0.10 not off master22:11
mgagneOur tests started failing less than 24 hours ago22:11
openstackgerritJay Pipes proposed stackforge/gertty: Release DB session thread lock earlier in syncs
mgagnefungi: right22:11
fungibut yeah, sounds like it's a separate error22:11
fungiso we probably want to tack that on top of the other stack of fixes we have in the feature/0.10 branch i guess22:12
mgagneso let me summarize when I understood: bug #1326682 is fixed in master, not 0.10 branch right?22:12
uvirtbotLaunchpad bug 1326682 in pbr "pbr doesn't check if git is not installed and/or not in the PATH" [Undecided,Fix committed]
*** dkranz has quit IRC22:12
fungiand release them together in 0.10.422:12
mtreinishclarkb: looking22:13
mgagnefungi: that would be great22:13
*** esker has quit IRC22:14
nibalizerpleia2: yea you might get that on the first run22:14
nibalizerwe made the decision not to munge puppet.conf in install_puppet.sh22:14
openstackgerritJay Pipes proposed stackforge/gertty: Speed up the toggling of reviewed/hidden changes
nibalizerwell we made the decision not to copy in a puppet.conf, we could add a sed -i i suppose22:15
pleia2nibalizer: ok, thanks22:15
fungimgagne: looks like it was actually released in 0.9.0 so the issue in 0.10.3 is a regression22:15
dstufftdhellmann: there's a way to re-run the tests on the pytz change right? I'm not remembering what it is22:15
jaypipesjeblair, krotscheck: is there a commit message tag I should use to link a storyboard story to a Gerrit review?22:15
nibalizerso hopefully after you run puppet apply it should have put a templatedirless puppet.conf in place for you22:15
dhellmanndstufft: leave a comment with "recheck"22:15
*** SumitNaiksatam has joined #openstack-infra22:16
dstufftdhellmann: ah right22:16
krotscheckjaypipes: “story: ###” But I believe it’s not quite hooked up yet.22:16
jaypipeskrotscheck: gotcha.22:16
jaypipeskrotscheck: yeah, must not be... see the two commits above from me. they have Storyboard story: XXX in them.22:16
krotscheckjaypipes: That’s on zaro’s plate right now.22:17
jaypipesya, no worries and not a big deal :)22:17
mtreinishclarkb: no I don't think I've seen that before, it's weird looking. Nova complains about a connection timeout to neutron but the neutron service log shows everything is normal for the requests that failed22:18
*** ayoung is now known as ayoung_dreidl22:18
mtreinishclarkb: I'll open a bug and file a query for it22:18
*** dustins has quit IRC22:20
*** mfink has quit IRC22:22
clarkbdstufft: dhellmann fungi mgagne
*** MarkAtwood has joined #openstack-infra22:23
mgagneclarkb: ty22:24
clarkbmgagne: any chance you can confirm that fixes for you?22:24
dhellmannclarkb: that looks good; is it based on another commit that should be referenced in the message?22:24
*** rmcall has quit IRC22:25
clarkbdhellmann: no its fixing the change that mordred made22:25
mgagneclarkb: it's kind of hard because the issue is within an other project of ours which installs openstack clients which in turn installs pbr22:25
dhellmannclarkb: ok22:25
clarkbdhellmann: mordred added the _run_git_functions function which didn't check for the git dir first22:25
clarkbalso I think I may need to reorder the function defs so that everything is happy importing?22:25
*** prad has quit IRC22:26
*** jaypipes has quit IRC22:26
dhellmannonly if one of those functions is run at import time, I think22:26
*** rmcall has joined #openstack-infra22:27
*** markmcclain has quit IRC22:27
*** vigneshvar has quit IRC22:27
fungiclarkb: lgtm. i'm going to run out to grab something to eat, but will return22:27
*** armax has quit IRC22:27
clarkboh right22:27
*** spzala has joined #openstack-infra22:27
*** thedodd has joined #openstack-infra22:27
pleia2latest wildfly running with and an off the shelf puppet module (with some config adjustments), applying gold stars to self22:27
dhellmannI'm going to step out, too. I probably won't be back online before tomorrow morning.22:27
*** zz_avozza is now known as avozza22:28
clarkbdhellmann: ok thanks for all the help22:28
*** ddieterly has quit IRC22:28
fungiif we get these merged i or someone can cut 0.10.4 and see where else we're broken22:28
*** melwitt has quit IRC22:28
*** bitblt has quit IRC22:28
*** melwitt has joined #openstack-infra22:28
*** ihrachyshka has quit IRC22:28
*** dims__ has joined #openstack-infra22:29
*** ddieterly has joined #openstack-infra22:29
dhellmannclarkb: ping me if you need any other reviews22:29
dhellmannfungi: cool, thanks22:29
fungiwill do, and thanks22:29
* fungi will bbiaw22:29
mtreinishclarkb: oh, would explain why neutron says it sent a response and nova timed out...22:30
jeblairclarkb: your 'read more' gear change seems to work well in stress testing22:30
clarkbjeblair: thats good, I haven't been able to swing back around and clean that up22:31
mtreinishconnection errors in the kernel log aren't ever a good sign22:31
jeblairclarkb: i'm thinking it's probably what we want to do after the selective wake22:31
*** bitblt has joined #openstack-infra22:32
clarkbjeblair: in particular I wanted to address your comment on the change22:32
clarkband I think the admin detection is wrong as I have pushed it let me check locally22:32
anteayaQueue lengths: 760 events, 1524 results.22:33
*** ddieterly has quit IRC22:33
anteayais there a point we gently request folks to maybe change focus in their workflow?22:33
jeblairanteaya: people are not the problem22:34
anteayavery good22:34
*** dannywilson has quit IRC22:36
*** armax has joined #openstack-infra22:36
*** dannywilson has joined #openstack-infra22:37
*** Kat has joined #openstack-infra22:37
*** Kat is now known as Guest6920822:37
*** Guest69208 has quit IRC22:38
*** Katherine has joined #openstack-infra22:38
*** jedimike has quit IRC22:38
*** cdent has quit IRC22:44
*** dizquierdo has quit IRC22:46
openstackgerritClark Boylan proposed openstack-infra/gear: Read more bytes per readPacket iteration
clarkbjeblair: ^ that should be in much better shape. addresses your comment and fixes admin status22:47
*** bhunter71 has quit IRC22:47
*** sarob has quit IRC22:51
jeblairclarkb: echo "status"|nc localhost 4730 doesn't work for me with your change22:52
*** e0ne has quit IRC22:52
clarkbjeblair: the most recent patch doesn't work?22:52
clarkbjeblair: I know the old one wouldn't have22:52
*** jamielennox|away is now known as jamielennox22:52
jeblairclarkb: oh sorry, yeah that was ps2, retesting with ps322:52
clarkbjeblair: cool22:52
jeblairyeah, better.22:53
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: move neutron-api job to use gate hooks
*** ryanpetrello has joined #openstack-infra22:54
*** spzala has quit IRC22:54
*** stevemar has quit IRC22:55
openstackgerritJames E. Blair proposed openstack-infra/gear: Fix accounting bug when peeking for job
jeblairclarkb: that's not an issue, just happened to notice it ^22:56
Katherineis anyone still online who may be able to help with another zuul question?22:56
*** stevemar has joined #openstack-infra22:56
jeblairKatherine: probably; it's usually best just to go ahead and ask :)22:57
*** armax has quit IRC22:57
*** sarob has joined #openstack-infra22:57
Katherineworks for me!22:57
Katherineok, i'm cloning my project on my jenkins instance but when i try to fetch from my zuul server, for example, git fetch refs/zuul/master/Zc75be890f63d4d938aedb4d99f73a503, i get fatal: not found: did you run git update-server-info on the server?22:59
clarkbKatherine: is apache (or some other web server) running on that host and serving out the zuul ref repos?22:59
Katherinelet me double check on the "serving out the ref repos".  apache is running23:00
Katherineah, you know, apache is not serving out the repos.  i forgot to add that part to the apache config.  i'm not sure what apache file that should go in, though23:02
*** Rockyg has quit IRC23:03
*** ryanpetrello has quit IRC23:03
*** otter768 has joined #openstack-infra23:03
clarkbusually a vhost definition in /etc/apache2/sites-enabled23:03
*** Rockyg has joined #openstack-infra23:03
clarkbI can dig up our exampl23:03
*** timcline has quit IRC23:03
*** sarob has quit IRC23:04
*** shashankhegde has quit IRC23:06
Katherinethanks, clarkb!23:06
clarkbno problem23:07
*** otter768 has quit IRC23:08
openstackgerritMerged openstack-infra/project-config: openstack/requirements stable branch ACL update
openstackgerritMerged openstack-infra/project-config: Fix the log path returned from zuul
openstackgerritMerged openstack-infra/project-config: python-surveilclient: Added gate checks
*** timcline has joined #openstack-infra23:11
*** shashankhegde has joined #openstack-infra23:12
*** wenlock has quit IRC23:12
*** zz_dimtruck is now known as dimtruck23:12
*** marun has quit IRC23:13
*** Sukhdev has joined #openstack-infra23:14
*** andreykurilin_ has quit IRC23:14
*** MaxV has joined #openstack-infra23:16
david-lyleI pushed tag 1.1.8 to openstack/django_openstack_auth a week ago, it's visible on, seems like the publish job to pypi failed, is there an easy way to re-trigger the publish?23:20
*** signed8bit has quit IRC23:21
*** ayoung_dreidl is now known as ayoung23:23
*** sigmavirus24 is now known as sigmavirus24_awa23:26
*** gondoi is now known as zz_gondoi23:26
*** nelsnelson has quit IRC23:29
*** armax has joined #openstack-infra23:32
dstufftor do /me wonders if openstack CI shouldn't fail fast if queue length gets over a certain number23:37
dstufftleft overs :(23:37
jeblairdstufft: i don't think that's a solution to the underlying network/protocol problem we are trying to debug23:38
dstufftjeblair: No, just a way to make the queue go faster when it's very long :)23:38
*** avozza is now known as zz_avozza23:39
jeblairdstufft: i think it would just be a way to make people angry.  right now it's backlogged, but making forward progress and not losing any data23:39
openstackgerritMerged openstack-infra/elastic-recheck: Add query for trove unit test bug 1379729
uvirtbotLaunchpad bug 1379729 in trove "test_tranformer unit test fails with ModelNotFoundError: InstanceServiceStatus Not Found" [Medium,Triaged]
anteayadstufft: it would just cause people to recheck and stuff things back in the queue23:39
dstufftwhy would they recheck a job that's not going to pass anyways23:40
*** bitblt has quit IRC23:40
anteayadavid-lyle: not ignoring you, yes there is a way to re-trigger the publish, I don't know what it is, I think that jeblair clarkb or fungi would have to do it23:40
dstufftfail fast == one of the things have already failed, you know this particular queue item isn't going to pass successfully, so don't run the rest of the tests23:40
jeblairdstufft: iow, the problem is not that there is too much going on, there is an underlying fault that we haven't isolated yet.23:40
anteayadavid-lyle: and they are all busy trying to get zuul and geard into a happy state23:40
jeblairdstufft: oh, but why tie that to queue length?23:40
dstufftjeblair: becasue if the queue isn't busy the extra information from getting _all_ the errors that would happen can be useful23:41
jeblairdstufft: fail-fast is reasonable in some circumstances, but it's not always what we want; it's often useful to get all the data.23:41
jeblairdstufft: the current backlog is not due to insufficient resources23:42
david-lyleanteaya: no worries, I saw I tried in the midst of turmoil, then things looked quieter when I popped back in. Thanks23:42
jeblairrunning fewer jobs would not appreciably help23:42
anteayadavid-lyle: yup, good thoughts23:42
*** Masahiro has joined #openstack-infra23:42
*** sabeen has quit IRC23:42
anteayadavid-lyle: if this fire gets contained someone will read backscroll and try to address23:42
anteayadavid-lyle: if you dont' see any action, try us tomorrow, fair?23:43
david-lyleanteaya: absolutely23:43
anteayadavid-lyle: thanks23:43
clarkbanteaya: thank you23:43
david-lylehappy firefighting :)23:43
clarkbdavid-lyle: and sorry we aren't super responsive right now23:43
anteayadoing my best to do what I can to help23:43
dstufftjeblair: eh, well I've thought about it previously too, not just tied to tonight, tonight is just particuarly back logged and reminded me about it23:43
* anteaya mans the phone23:43
david-lyleclarkb: I completely understand23:44
*** kumartin has joined #openstack-infra23:44
*** fandi has quit IRC23:44
clarkbjeblair: are you currently still waiting for gear to fall into the description update trap again? jenkins01 is running jobs so it should be getting told to update descriptions23:44
jeblairclarkb: yep.  i think it's only recently gotten into the state where it might happen again.23:45
dstufftjeblair: the other thing that might be useful is to have a new patch revision abort existing jobs (I wish travis did this too, I almost always manually abort jobs when I can :/)23:46
jeblairdstufft: it already does23:46
dstufftjeblair: it does? why is there two queued items for the same gerrit CR, is that a side effect of the current issues?23:46
*** Masahiro has quit IRC23:46
jeblairdstufft: likely side effect of current issue23:47
jeblairwhich change?23:47
dstufftjeblair: It jsut fell off I think, becuase I can't find it now. it was one of the pbr changes23:49
*** wenlock has joined #openstack-infra23:49
*** sabari is now known as zz_sabari23:49
anteayaI see four pbr changes all unique23:49
jeblairyeah, that may have just been the lag in processing that event23:49
*** andreaf has quit IRC23:50
*** yamamoto has joined #openstack-infra23:50
*** MaxV has quit IRC23:51
*** MaxV has joined #openstack-infra23:52
*** armax has quit IRC23:53
*** timcline has quit IRC23:53
*** yamamoto has quit IRC23:55
*** thedodd has quit IRC23:56
hasharmight be totally unrelated, but I had an issue with changes being stuck in Zuul queue for quite a long time23:56
hasharthe reason is the gearman server dieing somehow. All merge:merge jobs are then marked as pending although they haven't been actually submitted23:57
hasharso they are stuck in the queue23:57
hasharit is a logic error fixed by (have been running it for a couple months)23:57
hasharjeblair: ^^ ::D23:57
hashari can't remember whether it caused the result queue to not be processed though :(23:58
*** hdd has quit IRC23:58
*** wenlock has quit IRC23:58
jeblairhashar: that doesn't seem to be related to the problems we're seeing right now, but i'll keep it in mind, thanks23:58
anteayahashar: thanks for trying to assist23:59

Generated by 2.14.0 by Marius Gedminas - find it at!