Saturday, 2013-07-27

jeblairmordred: 900 test minutes per hour peak for all jobs00:05
fungithat's crazytown00:06
jeblairif that's correct that's about equivalent to 15 machines running full time00:06
jeblair(that needs double checking)00:07
*** johnthetubaguy has joined #openstack-infra00:07
*** johnthetubaguy has quit IRC00:09
*** Ryan_Lane has quit IRC00:13
jeblairi think it does have a bug. there is a mean that needs to be multiplied by a count.  but getting close.00:16
jog0anyone have any insight into the babel 1.0 issue?00:17
jeblairsorry im about to board a plane and am on bad wifi.00:19
jog0clarkb: ?00:19
*** zaro0508 has quit IRC00:19
jog0jeblair: thanks anyway00:19
*** Ryan_Lane has joined #openstack-infra00:19
*** Ryan_Lane has quit IRC00:20
*** Ryan_Lane has joined #openstack-infra00:20
jog0when I run locally it wall works00:24
reedrockstar, any news from LOSA?00:26
rockstarreed, not yet. I'll let you know.00:27
rockstarI'll have some time blocked out with him, so I can iterate with him.00:27
*** harlowja has joined #openstack-infra00:27
reedrockstar, do you need me for anything?00:27
rockstarreed, nope. I think I've got this.00:28
jog0fungi: ahhhh
uvirtbotLaunchpad bug 1205546 in openstack-ci "babel 1.0 dependency pytz isn't found" [Undecided,New]00:28
jog0fungi: so we can blame pip haha00:28
jog0that explains why it worked for me00:28
jog0I have pip 1.3.1 somehow00:28
reedrockstar, thanks. I'll hold on to publishing the newsletter until you tell me either abort or go00:28
jog0fungi: now what?00:29
fungijog0: further up in the scrollback it was jocularly remarked by bknudson that "pytz is the pits"00:30
jog0fungi: haha00:30
fungii suppose we need to temporarily cap babel until pytz versioning is fixed or pip 1.4.1 comes?00:30
jog0fungi: that only fixes infra not people using pip 1.4 without our mirrors00:31
jog0unless we cap it everywhere ...00:31
fungior temporarily add pytz==2013b00:31
jog0fungi: once again do we do that everywhere?00:31
jog0maybe just put that in reqs (one of those options) and email people warning them if they use pip 1.4?00:32
fungianywhere tickling that transitive dependency00:32
fungiunless anyone can think of something more elegant00:32
jog0fungi: we should have an emergency stub package that allows us to pin things transitively00:33
jog0something like openstack-ohshitpackagesbroke00:33
fungiheh. python-openstack-zomg00:33
jog0oh wait that may not work, because we do pip install -U00:33
fungiahh, jog0:
jog0unless pip is smart enough to not fall apart over that (which I think it may be now that I think about it)00:34
fungiseems requirements has pytz>=2010h for a few weeks now00:34
jog0that didn't work because its a babel dep00:35
jog0ohhh so we have to pin babel00:35
*** ant-tree-ya is now known as anteaya00:35
jog0so babel is bad boy,00:35
*** anteaya has quit IRC00:36
fungior pin pytz in each project of ours which depends on babel before the babel dependency is declared?00:36
jog0fungi: I am not sure if that works00:36
jog0do you have pip 1.4 somewhere?00:36
jog0if so we can test that out00:36
jog0either way i think the easiest thing going forward is pin something in our mirrors and email ppl about 1.400:37
fungijog0: i can have pip 1.4 somewhere momentarily, or i can pull a slave out of service and try it00:37
jog0fungi: cool, so try pip install babel00:37
jog0that should fail00:37
*** bclifford has left #openstack-infra00:37
jog0then pip install pyz=2013b00:37
jog0then pip install babel again00:37
jog0 and see00:37
* fungi tries that into a venv on precise-dev.slave00:37
* jog0 is glad he hasn't moved to the new hotness, pip 1.4 yet on his local box00:39
fungiyou know you want this ;)00:41
jog0I am trying the same thing on a VM btw00:45
jog0fungi: it worked00:46
jog0pip install pytz=2013b ; pip install babel00:46
jog0err pytz==2013b00:46
fungijust about finished testing in a virtualenv with an upgraded pip inside it00:46
fungiSuccessfully installed babel00:48
fungiso virtualenv on our slaves should be similarly happy00:48
jog0I treid pip install babel=0.9.600:48
*** vipul-away is now known as vipul00:48
jog0and that works too00:48
jog0I think that is easier00:48
fungigonna try pytz>=2010h00:48
fungilike is currently in openstack/requirements00:49
jog0I think that should work too, but the question is, do we think this will be around long enough that we *need* to fix it for everyone or jsut the gate00:49
*** DinaBelova has joined #openstack-infra00:50
jog0as in can we just say people running pip 1.4 use our mirrors or do this step first00:50
jog0or do we have to put this in every project00:50
fungiseems to work as well00:51
jog0fungi: have to jet, BBIAB but I am hoping we can at least get openstack-ci back up before monday00:52
jog0I may backonline in abit00:52
jog0fungi: thanks you rock00:52
fungii get the impression the reason for adding it to openstack/requirements a few weeks ago was so that we could add it to any projects depending on it. new babel simply introduced it as a transient dependency to things not already listing it00:52
fungijog0: any time00:52
*** DinaBelova has quit IRC00:54
*** blamar has joined #openstack-infra00:59
*** nati_ueno has joined #openstack-infra01:02
*** prad has joined #openstack-infra01:06
fungijog0: added you to seeing if tests succeed01:07
*** michchap has joined #openstack-infra01:08
mordredjog0: do you grok the pytz issue now?01:19
mordredand/or do you need the info? (sorry, my plane landed)01:19
mordredjog0, fungi: you're going to need to do this everywhere - and probably file a bug on Babel too01:19
mordredthe underlying thing is that pip 1.4 does not install pre-release software by default01:20
mordredbut will if you tell it to01:20
*** michchap has quit IRC01:20
mordredfoo>=1.2a flips on the install-pre-release bit01:20
mordredso will work01:20
mordredletters in versions are all automatically pre-release01:20
fungimakes sense01:20
mordredwhich tags all of pytz's releases as pre-release01:20
*** michchap has joined #openstack-infra01:20
mordredso pip >= 1.4 will never install just plain pytz01:20
*** nati_uen_ has quit IRC01:21
*** mgagne has joined #openstack-infra01:21
fungii'll have to see if i remember my sourceforge login01:22
*** mriedem has joined #openstack-infra01:24
fungiopening a bug for them01:27
*** ^demon has quit IRC01:29
*** beanemec is now known as bnemec_away01:29
jog0fungi: you are handling the babel bug?01:32
jog0mordred: I was thinking if there are probablly not many people using pip 1.4 yet so we could pin babel in requirments instead, and that would fix everthing on infra then we can fix things properly depending on how fast babel gets it fixed01:33
jog0as babel 0.9.6 doesn't ahve a pytz dep01:34
dstufftisn't pytz on launchpad01:35
*** mgagne1 has joined #openstack-infra01:35
*** mgagne1 has joined #openstack-infra01:35
*** dkliban has quit IRC01:35
fungijog0: i am, but inadvertently opened a bug against something else called babel on sourceforge. opening one against the right babel on github01:35
fungiassuming someone else hasn't already01:36
uvirtbotLaunchpad bug 1204837 in pytz "pytz version number (2013b) is considered a pre-release by pip 1.4" [Undecided,New]01:36
dstufftfungi: What's the Babel issue?01:36
jog0dstufft: that babel just requires pytz and not a versioned copy01:37
uvirtbotLaunchpad bug 1205546 in nova "babel 1.0 dependency pytz isn't found" [Critical,In progress]01:37
*** mgagne has quit IRC01:37
jog0I opened that to track this01:37
dstufftcount down till armin yells at me for breaking his install_requires :|01:38
fungipytz could solve this by releasing 2013.0.1 or something01:39
jog0fungi: so in your opinion what is the best approach to unblock all of openstack?01:40
mordredI think we should pin babel01:40
mordredand/or add pytz to nova's list before Babel01:41
fungimordred: i already uploaded the latter as a review for nova on that lp bug01:41
mordredfungi: great.01:42
jog0well then we have to fix every other project ...01:42
fungiat least until pytz and/or babel fix themselves01:42
jog0fungi: right, I doubt they will be faster then us01:42
mordredwe could in Babel in the requirements file01:43
fungibecause unlike us, they take their weekends off ;)01:43
mordred(will have to to land the nova pin patch anyway)01:43
mordredand then remove Babel 1.0 from our mirror01:43
jog0mordred: not if we assume no one uses pip 1.4 yet01:43
mordredthe gate uses 1.401:43
dstufft here's relevant information for linking to people01:43
mordredwait- what thing were you responding to?01:43
jog0and gate uses our mirrors which we can pin01:43
jog0as a quick and dirty get the code flowing01:44
fungithe temporary versioned pytz dependency has the benefit of not having to muck with the mirror contents01:44
mordredjog0: I meant we have to land the change to openstack/requirements01:44
jog0mordred: ahh yeah01:44
mordredadding pytz is 'easy' - but we would have to land the change on every project we have that uses Babel01:44
jog0mordred: I will propose that patch so we have the option01:44
fungidstufft: thanks, i'll add that to the upstream babel bug i opened01:44
mordredand then we'd have temporary depends in all the projects01:44
mordredthat we'd want to remember to go remove01:44
mordredhonestly, in the world of yuck- I think deleting Babel 1.0.0 from our mirror is a better short-term choice01:45
mordredbecause the roll-it-back-out step is really low cost01:45
fungimordred: but that leaves our projects uninstallable with current pip for anyone not using our mirror01:45
mordredthis is true01:45
fungiunless we go and add that babel cap to all the projects where we'd instead add the temporary pytz minimum version entry01:46
jog0mordred: true but a simple if you use pip 1.4 do this .tox/bin/pip install pytz=...01:46
fungihow many of our projects currently use babel without a version cap? lots?01:46
mordredfungi: I'm going to say all01:46
dstufftFWIW pip 1.4 has 26k downloads from PyPI and Virtualenv 1.10 (which has pip 1.4 bundled) has 11k01:47
jog0dstufft: how land has pip 1.4  been out for01:48
Alex_GaynorAnd users with them I suspect are more likely to try to install OpenStack reps on a weekend than average01:48
jog0oh 3 days01:48
dstufftjog0: 3 days?01:48
dstufftgive or take some hours01:48
*** melwitt has quit IRC01:49
mordredok. SO ...01:49
mordredI think we have to do both things01:49
mordredwe need to remove Babel from our mirror and pin requirements so that we can _land_ the other patches01:49
mordredthen I think we need to land patches to the projects either pinning Babel or pinning pytz01:50
mordredto unbreak people who are not in our gate01:50
*** michchap has quit IRC01:50
dstuffthistorically pip has near immediate uptake as far as which versions people download (e.g. once a near version is released downloads for old versions basically fall off to near zero). Unknown how many people just don't bother upgrading01:50
*** michchap has joined #openstack-infra01:50
jog0hmm not many things use Babel somehow01:51
jog0ceilo, cinder, heat neutron, nova and keystoneclient01:51
jog0ignore me01:52
jog0that wasn't a full list01:52
mordredironic and nova are the only problem children01:52
fungimordred: are you saying the pytz entry in each package won't help by itself?01:52
mordredfor people consuming not on our mirror01:52
mordredbecause the other uses are in test-requirements01:52
mordredwe have a nova core here, so we can land a fix to nova01:53
mordredno offense, but I think if ironic is broken due to a req for a weekend, it will not impact anyway01:53
mordredso I'm back to - let's fix our mirror, then fix nova01:53
dstufftSorry for breaking this for you guys. In my defense the pytz guy knew about this 2 months ago :[01:53
mordreddstufft: hey man, I put the pytz pin in our file in prep for this a while ago01:54
mordredBabel is the one who broke us01:54
fungimordred: in that order for expediency? because i think nova will be able to land 38896 unless i've mischaracterized the issue01:54
Alex_Gaynormordred: If it'd be helpful I can blast the rackspace dev-list asking for any openstack core reviewers to come help out? I imagine dstufft can do the same for nebula if he's comfortable and you think it'd be useful01:55
mordredjog0: any clue which version we should pin to? >=0.9.6,<1.0.0 ?01:55
*** erfanian has joined #openstack-infra01:55
mordredAlex_Gaynor: nah, I thnk we can land the nova change most likely01:55
mordredand that's the only one we really need help with01:55
Alex_Gaynorokey doke01:55
openstackgerritJoe Gordon proposed a change to openstack/requirements: Pin Babel to <1.0 since it doesn't play well with pip 1.4
dstufftYea I can if folks need it but looks like you got it covered01:55
jog0mordred: yeah I checked pytz isn't a dep in 0.9.601:55
fungijog0: trying that as an alternative to 38896 or should we do both?01:56
fungioh, right, that's to requirements01:56
jog0fungi: I agree with mordred both01:56
fungiyeah, me too. this will get things moving again01:56
jog0so the full list: cinder, ceilo, heat, ironic keystone, neutron nova keystoneclient nova01:56
funginot as many as i expected01:57
jog0fungi: yeah01:57
dstufftmordred: I expect similar fallout to 1.5 as a lot of people probably won't start hosting on PyPI until pip is failing by default01:57
jog0I guess we don't translate clients01:57
mordredok. mirror cleaned01:58
mordredand the mirror builders are cleaned from their caches01:58
jog0so should I go ahead and A+ the nova patch01:58
jog0as well01:58
jog0can you +1 it too01:58
mordredwill it pass though?01:58
jog0mordred: fungi and I tried that approach out01:59
mordredI think requirements has to land before the nova change ..01:59
mordredoh - are you doing the pytz pin in nova?01:59
mordredI'd think that you'd want the Babel pin so that we know what's going on, no?01:59
mordredor - whichever, honestly01:59
jog0fungi: haha01:59
jog0mordred: good point02:00
*** ^demon has joined #openstack-infra02:00
mordredjog0: I think you wrote that hacking check too02:00
jog0lets pin babel everywhere to make it easier to debug02:00
mordredI've aprv'd the Babel change in requirements02:00
fungiokay, i'll abandon my change02:00
mordredI'm kinda tempted to super-user approve it bypassing the gate - but that's just because I'm impatient02:01
fungior repropose it with the babel pin instead i guess02:01
jog0fungi: repropose with babel02:01
jog0so when I A+ it, its not all coming from me02:02
mordredfungi: btw - COMPLETELY unreleated - - I did plane hacking02:02
jog0and do forget the period (btw that test wasn't me)02:02
openstackgerritA change was merged to openstack/requirements: Pin Babel to <1.0 since it doesn't play well with pip 1.4
mordredjog0: you might find that patch interesting too02:02
jog0mordred: puppet?02:03
mordredjog0: that's step one in making out devstack-gate nodes using diskimage-builder02:03
mordredwhich, once done, will mean there will be downloadable copies of the image that runs d-g02:04
mordredso that debugging locally in a kvm or something will be easy :)02:04
jog0d-g is that god spelled backwards without the o?02:04
jog0mordred:  ohh nice02:04
jog0(to the download images and kvm)02:05
mordredalso, it's devstack-gate without as many letters02:05
jog0mordred: ahh02:05
mordredbut now I'm thinking we should re-name d-g to god02:05
jog0fungi: let me know when your patch is up and I will A+02:05
mordredbut spell it g-d - or gate-devstack for short02:05
fungijog0: already there02:05
jog0fungi: title is wrong02:06
jog0pin babel02:06
fungiyep, just spotted that02:06
fungithere now02:07
*** markmcclain has quit IRC02:08
jog0thanks, mordred can you +1 that02:08
*** ^demon has quit IRC02:08
mordredyes. link?02:08
jog0or any other infra, so its at least not just me
mordredit's too bad I let my nova core status lapse02:09
openstackfungi: Error: "!!1!!1eleven" is not a valid command.02:09
fungiopenstack: it should be02:10
jog0dansmith: is around02:11
fungii'll get started proposing the same to the other listed affected projects if someone isn't already02:11
jog0and getting him to do a proper second review02:12
dansmithit's done02:12
mordredoh wow. nice02:12
mordredit's almost like none of us take time off! :)02:12
Alex_GaynorSo now we have to wait fo rit work it's way up the review stack02:13
jog0I need to run out and buy some whisky soon actaully, but this took priority02:13
jog0Alex_Gaynor: the openstack mirror is fixed02:13
jog0so recheck away02:13
fungijog0: but now you need twice as much whiskey02:13
*** koolhead17 has quit IRC02:13
Alex_Gaynorjog0: no joke, my patch didn't make its way up to the top yet because stuff was so slow02:13
Alex_Gaynorjog0: is it recheck for gate jobs? I thougth there was some other command02:14
Alex_Gaynorreverify, that's it02:14
jog0woot things are going green
jog0ok time to get that whisky02:17
mordredjog0: enjoy!02:18
clarkbjeblair: ++ to that graph02:18
clarkbwhy is there so much sb on friday evening. I grab a couple ebers and this happens02:18
*** hartsocks has quit IRC02:19
Alex_Gaynorteh gating system totally needs an "Your estimated wait time is N minutes" display02:20
clarkbAlex_Gaynor: tl;dr maybe next week02:21
Alex_Gaynorclarkb: awesome.02:21
fungiclarkb: beat me to the link02:21
Alex_Gaynorclarkb: You know this is just encouraging me to propose hair brained ideas right?02:21
fungimordred: jog0: can one of you confirm i didn't miss adding an effected project to
uvirtbotLaunchpad bug 1205546 in nova "babel 1.0 dependency pytz isn't found" [Critical,In progress]02:22
fungier, affected02:22
fungigetting started uploading changes for each of those now02:23
*** vipul is now known as vipul-away02:25
clarkbfungi: nova, glance, swift, ?02:25
fungiclarkb: nova's there. glance and swift don't seem to be the ones jog0 mentioned above02:26
fungibut let me know if you see babel dependencies specified in them02:27
fungior i'l check once i get these knocked out02:27
clarkbfungi: I think they both use babel02:27
clarkbalso wow babel made a new release?02:27
jog0fungi: that looks right, I only checked repos under
*** sdake_ has joined #openstack-infra02:29
fungijog0: thanks02:30
mgagne1clarkb: mitsuhiko took over the maintenance of Babel02:31
*** mgagne1 is now known as mgagne02:31
mgagneclarkb: Armin Ronacher in fact02:32
jog0fungi: still seeing failures
*** changbl has joined #openstack-infra02:33
jog0does it take time for the Babel change in mirror to propogate out?02:33
mordredit shouldn't02:34
jog0because dansmith is still angry02:34
fungijog0: are those new or did those jobs run before we fixed the mirror?02:34
mordredI don't like it when dansmith is angry02:34
dansmithum, my *patches* are still angry :)02:34
dansmithit's friday, so I'm happy :)02:34
fungidansmith: that's a healthy outlook02:35
mordredoh. crappit. it's back in the mirror02:35
jog0fungi: not sure but the mirror change is in and these are still running02:35
jog0mordred: haha02:35
* mordred cries02:35
fungimordred: was there perhaps a mirror job running when you cleaned the mirror copy?02:35
mordredmaybe I missed a cached file someone02:35
mordredfungi: yeah. or that02:35
jog0btw my browser things is danish02:36
mordredok. removed from the mirror again02:36
jog0will it stay  removed this time?02:37
mordredand removed from teh mirror builder caches again02:37
mordredsorry bout that02:37
dstufftbabel refuses to be constrainted by your petty attempts02:40
jog0now if only we had a way to use logstash to find out which jobs failed because of this and autoretry all of em02:40
Alex_GaynorI can imagine a thing to scape all reviews in the last 8 hours and do things.02:40
jog0I think you just volunteered02:41
dstufftI saw it02:41
mordredjog0: I believe if clarkb was here, you could TOTALLY do that with logstash02:41
Alex_GaynorIt only looked like that, but I didn't.02:41
Alex_GaynorI think clarkb was deifnitely just volunteered though02:42
mordredalso, I feel like fungi has scripts to retry things ...02:42
Alex_Gaynormordred: do the test runners have use-wheels = true yet?02:42
mordredAlex_Gaynor: not yet02:42
mordredthat's coming02:42
dansmiththere is a zuul tool that will recheck everything currently in a pipeline02:42
mordredI wanted to get one cycle of madness done before we started down that road02:42
Alex_Gaynordstufft: have you done a timing of use-wheels = true vs. false for openstack/requirements?02:43
jog0dansmith:  nice, we also want things out of the pipeline02:43
jog0aka all of your patches02:43
dansmithall of mine are in the pipeline right now02:43
fungimordred: i do not, in fact. but it would just be a matter of multiple gerrit review api calls02:43
dstufftAlex_Gaynor: nope02:43
fungiiterated from a list of change numbers02:43
Alex_Gaynordstufft: how do I disable use-wheel without commenting it out?02:44
mordredAlex_Gaynor: I expect it will be several minutes quicker02:44
Alex_Gaynormordred: sounds about right, that should be nice02:44
dstufftAlex_Gaynor: edit the file and set it to false?02:44
Alex_Gaynordstufft: ...02:44
dstufftAlex_Gaynor: hey that's without commenting it out ;P02:44
Alex_Gaynorwhat do normal people do at 8PM on a friday?02:44
mordredI don't know!02:44
*** zehicle_at_dell has joined #openstack-infra02:44
mordredI mean, it's 10:45pm here - which is WAY too early to go out drinking02:44
dstufftI dunno but probably the same thing they are doing at 1102:45
mordredI think the first step for wheel is that I want to change our mirror builder to build wheels of everything and upload those02:45
mordredbecause I believe mirror builder is working on 1.4 already02:45
mordredI also want to modify out publish-to-pypi jobs to publish wheels02:46
dstufftmordred: universal = 102:46
Alex_GaynorOooh, yes please02:46
mordreddstufft: aroo?02:46
dstufftmordred: wheels are specific to the python you create them with, unless you set universal in the setup.cfg02:47
mordredyou're kidding02:47
dstufftmordred: no. They are a binary file format02:47
mordredeven if they are pure-python?02:47
dstufftmordred: yes02:47
mordredis there a way to set that at build time?02:47
Alex_Gaynorany reason not to just put it in setup.cfg, we already use that format?02:48
mordredso - I can add universal = 1 to setup.cfg for all of our stuff, so that when we upload universal stuff02:48
mordredfor stuff from pypi of which we are building wheels for our mirror consumption02:48
mordredwe actually do a py26 and a py27 run already02:48
mordredso those being non-universal is probably fine02:48
dstufft^ example use02:49
uvirtbotdstufft: Error: "example" is not a valid command.02:49
mordredI will slightly apologize to anyone on osx - I can't help you02:49
dstufftI think it's smart enough not to make a platform specific wheel02:49
*** Ryan_Lane has quit IRC02:49
dstufftfor pure python02:49
mordreddstufft: what's consuming that metadata section in there. anything yet?02:50
dstufftI think the universal flag is because it doesn't know if you have any sort of transformation like 2to302:50
dstufftmordred: I think bdist_wheel does.02:50
mordredneat. I should make sure our metadata is compat02:50
*** DinaBelova has joined #openstack-infra02:51
dstufftit allows specififying conditional dependencies as well02:51
mordredyah. does pip grok those yet?02:51
fungiokay, all changes uploaded now...,n,z02:52
clarkblogstash is too far behind and you really want to query gerrit for that info anyways02:52
*** changbl has quit IRC02:52
fungii didn't bother with stable branches of anything yet02:52
jog0clarkb: you can check if pep8 py26 and py27 failed02:53
jog0if so its prob this bug02:53
fungimaybe one of the upstreams will obviate the issue before we need another stable patch to something02:53
clarkbjog0: not until tomorrow :( logstash is about 20 hours behind and the monmeth02:53
dstufftmordred: Pip does not use a setup.cfg and the Wheel format does not include the setup.cfg in the metadata02:53
mordreddstufft: ok. so, what processes the conditional requirements?02:53
dstufftmordred: wheel will read the setup.cfg and use it to feed the Wheel format02:53
jog0clarkb: you can use the gerrit api02:53
jog0anyway i'm out for the weekend02:54
dstufftthe wheel format supports conditional requirements afaik02:54
dstufftit's not stored as setup.cfg inside the wheel though02:54
jog0have a good one everybody02:54
dstufftsetup.cfg is just an input is what I'm trying to say02:54
*** jog0 is now known as jog0-up-up-and-a02:55
dstufftI'm looking to see if pip understands conditional reqs from a wheel02:55
*** jog0-up-up-and-a is now known as jog0-away02:55
*** DinaBelova has quit IRC02:55
mordredso in _theory_ if I only published wheels, I could start getting conditional reqs02:56
clarkbwere we giving pytz a version or pinning babel?02:57
* mordred no have current use for this - just grokking02:57
mordredclarkb: pinning babel02:57
clarkbI mean I think I would be ok with the same version of babel that has been around for the last 5 years02:57
Alex_Gaynorfor at least a few days02:57
mordredclarkb: yeah. we'll unpin it when it fixes its pytz depend02:57
clarkbalso why do people do releases on friday?02:58
mordredchicken butt02:58
dstufftmordred: you can emulate conditional reqs in a too02:59
dstufftjust do the same conditions in code02:59
mordredwas just wondering what the current state of the art is02:59
dstufftmake them match and you have codnitional requirements in all systems, and in wheel it's all without executing code02:59
mordredI mean, we already process those files ... we could totally do markerlib matching same as wheel03:00
mordredbut for now - extra complex - not needed - too many changes already - aaaaaa03:01
mordredooh! I see green things in zuul03:02
fungislimy green things?03:03
* mordred wants jeblair's new status bar patch03:03
*** mriedem has quit IRC03:05
*** erfanian has quit IRC03:09
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add branch tarball job for tempest.
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Clean local jeepyb repo when installing/upgrading
*** afazekas_zz is now known as afazekas03:34
mikalI take it from scroll back that the gate, she is broken?03:34
fungimikal: the gate she should be fixed now03:39
fungibabel decided to release a new version with an unversioned dependency on pytz, which breaks under latest pip03:40
*** zehicle_at_dell has quit IRC03:44
Alex_Gaynorgrumble, tests that fail non-deterministically really screw up the efficiency of gating03:48
*** ladquin has quit IRC03:48
*** DinaBelova has joined #openstack-infra03:51
fungiAlex_Gaynor: yes, no argument there. bad tests should either be fixed or removed03:51
Alex_Gaynorfungi: I guess any failer really, it costs you an hour per failure basically. We should hvae a test performance hackday or something.03:52
Alex_GaynorI wonder if there's some optimization that could be done so that you don't lose everything after the failer as well. Hmmm03:53
fungimordred: no luck on with removing the cached eggs. so far the only solution i've found is manually upgrading pbr first. even pip install -U . is broken03:54
fungiAlex_Gaynor: well, you don't want to continue testing subsequent changes on top of one you know is being ejected from the series03:55
*** DinaBelova has quit IRC03:55
Alex_Gaynorfungi: well, you don't know it's being ejected until it's at the top of stack03:56
fungiAlex_Gaynor: yeah, but the one failing furthest up will either be ejected because those in front of it succeed, or will be restarted because one ahead of it failed03:57
Alex_Gaynorright, but if something down the stack fails, as happens with non-deterministic ones, everything after it gets cancaelled03:58
Alex_Gaynorand then rerun after the failer becomes ToS03:58
fungiwhich is inevitable anyway, so no point in continuing to waste computing resources running a test which will have to be discarded and rerun03:58
fungiif at least one change ahead of you is failing, your tests will be rerun because either it will fail to merge *or* it will succeed on a retry because one ahead of it fails to merge. either way the list of patches ahead of you is guaranteed to change before you merge so you need to rerun03:59
fungiand with the current single-branch-prediction model, you have to wait to rerun until you know which patch ahead of you is going to get removed from the series04:01
fungiwe've talked about multiple-branch prediction, but that would speculatively ultimately multiply the combinations of changes being tested and use even more resources04:02
Alex_GaynorRight, you need more cloud then04:02
Alex_Gaynorbetter wall clock, more CPU time04:03
fungiit's a trade-off, plus more code04:03
Alex_Gaynormore code is the most compelling argument :)04:03
funginot necessarily out of the cards though, just not something we have today and needs further discussion04:03
*** leifmadsen has quit IRC04:06
*** pcrews has joined #openstack-infra04:22
*** SergeyLukjanov has joined #openstack-infra04:26
*** mgagne has quit IRC04:46
*** DinaBelova has joined #openstack-infra04:52
*** DinaBelova has quit IRC04:57
SpamapS2013-07-27 01:18:33.592 | Downloading/unpacking pytz (from babel)04:57
SpamapS2013-07-27 01:18:33.592 |   Could not find a version that satisfies the requirement pytz (from babel) (from versions: 2012c, 2012c, 2012c, 2012d, 2012d, 2012d, 2012f, 2012f, 2012f, 2012g, 2012g, 2012g, 2012h, 2012h, 2012j, 2012j, 2012j, 2013b, 2013b, 2013b, 2009r, 2008b,04:57
SpamapSKnown bug?04:57
uvirtbotLaunchpad bug 1205546 in python-keystoneclient "babel 1.0 dependency pytz isn't found" [Critical,In progress]04:57
SpamapSahh n/m04:57
* fungi nods04:59
fungiwe all dogpiled onto it fairly quickly, but someone will need to round up a few core reviewers for the various projects affected05:01
SpamapSdi-b is affected too.. making patches now05:01
*** reed has quit IRC05:04
fungiyeah, we scanned for affected stuff in the openstack org heirarchy but didn't look for other projects. this is likely a very short-term situation until pytz and/or babel address it properly05:07
fungiin the meantime, enjoy the round of ugly hacks05:07
*** sarob has joined #openstack-infra05:09
*** sarob_ has joined #openstack-infra05:11
*** sarob has quit IRC05:11
*** nati_ueno has quit IRC05:12
*** vogxn has joined #openstack-infra05:16
*** pcrews has quit IRC05:21
SpamapShrm, hacking seems to be confused05:28
*** yaguang has joined #openstack-infra05:28
SpamapS./build/lib.linux-x86_64-2.7/os_apply_config/ H302  import only modules.'from os_apply_config import collect_config' does not import a module05:28
SpamapSexcept.. it does05:28
*** yaguang has quit IRC05:37
SpamapSah picking up a build dir.. weird05:37
*** Ryan_Lane has joined #openstack-infra05:39
*** yaguang has joined #openstack-infra05:49
*** yaguang has quit IRC05:51
*** yaguang has joined #openstack-infra05:51
*** DinaBelova has joined #openstack-infra05:52
*** DinaBelova has quit IRC05:57
*** shhu has joined #openstack-infra05:57
*** westmau5 is now known as westmaas06:10
*** dkehn has quit IRC06:34
*** dkehn has joined #openstack-infra06:35
*** dkehn has joined #openstack-infra06:36
openstackgerritSergey Lukjanov proposed a change to openstack-dev/pbr: Improve AUTHORS file generation
*** pcrews has joined #openstack-infra06:49
*** sarob_ has quit IRC06:50
*** sarob has joined #openstack-infra06:51
*** DinaBelova has joined #openstack-infra06:53
*** sarob has quit IRC06:56
*** DinaBelova has quit IRC06:57
*** jjmb has joined #openstack-infra06:59
openstackgerritSergey Lukjanov proposed a change to openstack-dev/pbr: Fix .mailmap file search location
*** SergeyLukjanov has quit IRC07:01
*** DinaBelova has joined #openstack-infra07:02
*** pcrews has quit IRC07:03
*** plomakin has quit IRC07:03
*** mirrorbox has quit IRC07:03
*** ogelbukh has quit IRC07:03
*** sshturm__ has quit IRC07:03
*** ogelbukh has joined #openstack-infra07:04
*** sshturm__ has joined #openstack-infra07:04
*** mirrorbox has joined #openstack-infra07:05
*** plomakin has joined #openstack-infra07:05
*** mirrorbox has quit IRC07:06
*** mirrorbox has joined #openstack-infra07:06
*** jjmb has quit IRC07:31
*** afazekas has quit IRC07:34
*** ogelbukh has quit IRC07:36
*** mirrorbox has quit IRC07:36
*** plomakin has quit IRC07:36
*** sshturm__ has quit IRC07:36
*** sshturm__ has joined #openstack-infra07:36
*** plomakin has joined #openstack-infra07:36
*** ogelbukh has joined #openstack-infra07:37
*** mirrorbox has joined #openstack-infra07:37
*** afazekas has joined #openstack-infra07:38
*** dkliban has joined #openstack-infra07:40
*** Ryan_Lane has quit IRC07:45
*** dkliban has quit IRC07:46
*** dkliban has joined #openstack-infra07:52
*** vogxn has quit IRC07:55
*** SergeyLukjanov has joined #openstack-infra08:04
*** DinaBelova has quit IRC08:07
*** dkliban has quit IRC08:12
*** michchap has quit IRC08:14
*** michchap has joined #openstack-infra08:15
*** UtahDave has joined #openstack-infra08:15
*** afazekas has quit IRC08:19
*** afazekas has joined #openstack-infra08:23
*** dkliban has joined #openstack-infra08:41
*** dkliban has quit IRC08:50
*** ogelbukh has quit IRC08:51
*** ogelbukh has joined #openstack-infra08:51
*** fbo_away is now known as fbo08:57
*** fbo is now known as fbo_away08:58
*** DinaBelova has joined #openstack-infra09:07
*** DinaBelova has quit IRC09:12
*** DinaBelova has joined #openstack-infra09:18
*** SergeyLukjanov has quit IRC09:19
*** DinaBelova has quit IRC09:23
*** afazekas has quit IRC09:26
*** afazekas has joined #openstack-infra09:31
*** yaguang has quit IRC09:32
*** fbo_away is now known as fbo10:08
*** fbo is now known as fbo_away10:08
*** DinaBelova has joined #openstack-infra10:19
*** DinaBelova has quit IRC10:24
*** DinaBelova has joined #openstack-infra11:19
*** DinaBelova has quit IRC11:24
*** jjmb has joined #openstack-infra11:43
*** UtahDave has quit IRC11:49
*** w_ has joined #openstack-infra11:51
*** olaph has quit IRC11:53
*** DinaBelova has joined #openstack-infra11:57
*** DinaBelova has quit IRC12:02
*** DinaBelova has joined #openstack-infra12:15
*** hashar has joined #openstack-infra12:15
*** jjmb has quit IRC12:16
*** hughsaunders_ has joined #openstack-infra12:20
*** hughsaunders has quit IRC12:21
*** hughsaunders_ is now known as hughsaunders12:21
*** DinaBelova has quit IRC12:27
*** DinaBelova has joined #openstack-infra12:39
*** SergeyLukjanov has joined #openstack-infra12:39
*** hashar has quit IRC12:53
*** SergeyLukjanov has quit IRC12:56
*** jjmb has joined #openstack-infra13:08
*** jjmb has quit IRC13:10
*** afazekas has quit IRC13:23
*** afazekas has joined #openstack-infra13:25
*** DinaBelova has quit IRC13:27
*** DinaBelova has joined #openstack-infra13:30
*** DinaBelova has quit IRC13:31
*** krtaylor has quit IRC13:42
*** DinaBelova has joined #openstack-infra13:44
*** vogxn has joined #openstack-infra13:59
*** DinaBelova has quit IRC14:24
*** afazekas has quit IRC14:33
*** yaguang has joined #openstack-infra14:35
*** afazekas has joined #openstack-infra14:38
*** nati_ueno has joined #openstack-infra14:41
openstackgerritA change was merged to openstack-dev/pbr: Also patch easy_install script creation
*** dkliban has joined #openstack-infra14:51
*** DinaBelova has joined #openstack-infra14:55
openstackgerritA change was merged to openstack-dev/pbr: Revert include_package_data change
openstackgerritA change was merged to openstack-dev/pbr: Add support for globbing in data files
openstackgerritA change was merged to openstack-dev/pbr: Loop over test output for better readability
*** sarob has joined #openstack-infra14:59
*** sarob has quit IRC15:04
*** dkliban has quit IRC15:07
*** vogxn has quit IRC15:07
*** mriedem has joined #openstack-infra15:27
*** DinaBelova has quit IRC15:29
Alex_Gaynormordred, clarkb, dstufft: New version of babel has been released15:30
*** SergeyLukjanov has joined #openstack-infra15:40
*** vogxn has joined #openstack-infra15:42
*** DinaBelova has joined #openstack-infra15:43
*** SergeyLukjanov has quit IRC15:52
*** DinaBelova has quit IRC15:58
*** sarob has joined #openstack-infra16:00
*** sarob has quit IRC16:04
*** SergeyLukjanov has joined #openstack-infra16:09
mordredAlex_Gaynor: w00t!16:15
mordredalso, woot my pbr patches landed16:15
rockstarmordred, did you just get an email from me about the mailing list upgrade being complete?16:16
mordredrockstar: yup16:16
rockstarmordred, \o/16:16
*** mriedem1 has joined #openstack-infra16:17
*** mriedem has quit IRC16:18
*** yaguang has quit IRC16:23
*** changbl has joined #openstack-infra16:29
*** vogxn has left #openstack-infra16:32
*** DinaBelova has joined #openstack-infra16:32
*** sarob has joined #openstack-infra16:36
*** sarob has quit IRC16:41
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Build a devstack-gate image with diskimage-builder
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Proof of concept of kexec/takeovernode
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Build a devstack-gate image with diskimage-builder
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Proof of concept of kexec/takeovernode
*** DinaBelova has quit IRC16:44
*** CaptTofu has joined #openstack-infra16:46
*** dkliban has joined #openstack-infra16:49
*** SergeyLukjanov has quit IRC16:54
*** nati_ueno has quit IRC16:56
*** michchap has quit IRC16:57
*** DinaBelova has joined #openstack-infra16:59
*** reed has joined #openstack-infra17:05
reedhi all17:05
reedrockstar, thanks for persisting :)17:07
*** DinaBelova has quit IRC17:09
*** UtahDave has joined #openstack-infra17:15
openstackgerritJeremy Stanley proposed a change to openstack/requirements: Revert "Pin Babel to <1.0 since it doesn't play well with pip 1.4"
*** locke1051 has joined #openstack-infra17:21
*** mrodden1 has joined #openstack-infra17:21
*** locke105 has quit IRC17:22
*** mrodden has quit IRC17:22
fungimordred: SpamapS: you should be able to revert safely now17:23
fungii've confirmed that the latest babel installs in a virtualenv using the latest pip and no longer trips over pytz17:24
*** hartsocks has joined #openstack-infra17:25
*** CaptTofu has quit IRC17:25
*** michchap has joined #openstack-infra17:27
*** yaguang has joined #openstack-infra17:30
*** SergeyLukjanov has joined #openstack-infra17:31
*** dkliban has quit IRC17:34
*** michchap has quit IRC17:35
*** sarob has joined #openstack-infra17:37
*** dkliban has joined #openstack-infra17:40
*** sarob has quit IRC17:42
*** yaguang has quit IRC17:43
*** CaptTofu has joined #openstack-infra17:45
*** dkliban has quit IRC17:52
*** michchap has joined #openstack-infra18:10
*** michchap_ has joined #openstack-infra18:11
*** michcha__ has joined #openstack-infra18:14
*** michchap has quit IRC18:14
*** michchap has joined #openstack-infra18:15
*** michchap_ has quit IRC18:16
*** prad has quit IRC18:16
*** michchap_ has joined #openstack-infra18:16
*** michch___ has joined #openstack-infra18:17
*** michcha__ has quit IRC18:18
*** DinaBelova has joined #openstack-infra18:19
*** michchap has quit IRC18:19
*** UtahDave has quit IRC18:20
*** michchap_ has quit IRC18:21
*** michch___ has quit IRC18:21
*** jesusaurus has quit IRC18:22
*** DinaBelova has quit IRC18:24
*** jesusaurus has joined #openstack-infra18:24
mordredfungi: awesome. thanks18:26
fungiabove is a revert for the requirements repo. none of the per-project changes besides dib's merged (due to a combination of too few core reviewers and constant failures because of bug 1205344) so i've abandoned them all18:29
uvirtbotLaunchpad bug 1205344 in tempest "mkfs error in test_stamp_pattern" [Critical,In progress]
openstackgerritA change was merged to openstack/requirements: Revert "Pin Babel to <1.0 since it doesn't play well with pip 1.4"
*** sarob has joined #openstack-infra18:37
*** CaptTofu has quit IRC18:40
*** sarob has quit IRC18:42
*** prad has joined #openstack-infra18:55
*** UtahDave has joined #openstack-infra19:08
*** DinaBelova has joined #openstack-infra19:24
*** DinaBelova has quit IRC19:28
fungiheh, dstufft called our ci infrastructure "massive" on distutils-sig a couple days ago (i'm still catching up on e-mail)19:28
*** jpeeler has quit IRC19:29
dstufftfungi: I did?19:30
clarkbfungi: if you need numbers to back it up *may take a while to load19:30
fungidstufft: in the deprecating official pypi mirrors thread19:30
dstufftfungi: aah19:31
*** pcrews has joined #openstack-infra19:31
* dstufft forgets what things he's saying a lot 19:32
fungii just found it amusing, that's all19:32
fungii don't often expect to see openstack infra plugged on that list19:33
mordredfungi, clarkb: update-image hasn't worked since July 2319:33
fungioh fun19:34
* fungi looks at the log19:34
clarkbis that when the setuptools upgrade stuff went in/out?19:34
clarkbhmm no 23rd was tuesday19:34
mordredit's because it's trying to download an 11.04 openvz image19:34
mordredwhich does not exist19:34
fungithis airplane doesn't make log browsing easy19:34
mordredI cannot find the reference19:34
mordredI mean - I cannot find where we're asking for that image19:35
mordredthere it is19:35
fungiin az2 it was still working into the 24th19:35
clarkbis it a devstack image url?19:36
clarkbmordred: yup. its in devstack stackrc19:36
fungipresumably we need instead anyway19:37
fungibut yeah, the 11.10 one has been yanked19:37
clarkbI will propose the devstack fix as I don't have +2 there19:37
clarkband I think some of you do19:37
mordredI do not either19:37
clarkbmordred: wins19:37
mordredbtw - building devstack-gate image locally is awesome19:37
fungii only have +2 on devstack when i get the okay to temporarily add myself (like for the folsom fix)19:38
*** sarob has joined #openstack-infra19:38
mordredlike, it made me notice that we're installing puppet3 on our devstack-gate nodes19:38
clarkbmordred: noice19:39
mordredbecause none of our install puppet logic has been ported in there19:39
*** DinaBelova has joined #openstack-infra19:39
clarkbI guess only jeblair has the +2 power19:39
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Build a devstack-gate image with diskimage-builder
mordredbtw - that almost works19:39
clarkbI don't think this is super critical to merge this weekend though.19:39
mordredI've got one more thing to track down19:39
clarkbProbably before we merge my setuptools puppet change though19:39
mordredbut it works, and is honestly really tight19:40
mordredthe dib guys have done some excellent work on local caching of thigns19:40
mordredin a general manner19:40
fungimordred: not sure if you saw in last night's scrollback, but the chicken-and-egg issue with the jeepyb puppet module needing to pull in newer non-broken pbr was not as we thought19:41
mordredfungi: did not see that19:41
fungiclearing the local eggs in the repo had no effect19:41
mordredoh. what?19:41
clarkbfungi: :( I didn't see that either19:41
fungiand we also talked about switching to calling pip install -U . instead, but that also breaks in the same way as python install19:42
*** w_ is now known as olaph19:42
mordredfungi: what about upping the lower requirement?19:42
fungii'll give that a shot19:42
fungiforgot to test that option19:43
*** sarob has quit IRC19:43
*** DinaBelova has quit IRC19:43
mordredI'm getting this, btw:19:44
mordredFATAL: Could not load /lib/modules/3.8.0-25-generic/modules.dep: No such file or directory19:44
mordredin tryuing to install iptables-persistent19:44
mordredanybody ever seen that or know what I might want to investigate?19:44
clarkbmordred: its complaining about your kernel modules and that isn't the precise kernel19:44
Mithrandirprobably an upgraded kernel without a reboot19:44
Mithrandirwith the old kernel image removed in the meantime19:45
mordredah. interesting. cool19:45
fungimordred: can you put version matches on setup_requires list items? pkg_resources.VersionConflict: (pbr 0.5.17 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('pbr>=0.5.19'))19:46
mordredyou can - but I would expect that to do the right thing and upgrade19:46
mordrednot just conflict19:46
fungieasy to recreate this... cd into a current clone of the jeepyb repo, pip install pbr 0.5.17 and then try to install jeepyb from the local source tree19:47
fungipip 1.0 from /usr/lib/python2.7/dist-packages (python 2.7)19:48
mordredk. I will work on it19:49
fungimanually upgrading pbr first allows jeepyb to install just fine, but so far that's the *only* solution i've gotten to work19:51
dstufftouch pip 1.019:52
fungidstufft: welcome to ubuntu lts19:52
*** dkliban has joined #openstack-infra19:53
mordredit might just be that we have to upgrade pbr first - the intereactions here don't seem great - but I'm going to see if I can figure something out19:54
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Build a devstack-gate image with diskimage-builder
mordredclarkb, fungi: woot! that ^^ creates an d-g image!19:57
fungimordred: worst case, i'll roll back to patchset #1, which did actually work, just didn't meet with aesthetic standards and sanity19:58
mordredfungi: I thnk that's probably the best thing for moving forward19:59
mordredthe other thing is likely to take some pbr dev work on my part19:59
*** hartsocks has quit IRC20:00
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Work around catch-22 in jeepyb with broken pbr
dansmithsometimes you guys pull out a stat about "test foo is failing at 40% now" -- is that actually calculated anywhere automatically?20:14
mordreddansmith: not that I'm aware of - I believe clarkb can do some magic20:15
dansmithokay.. was wondering how bad the grenade fail is, because it seems to be hitting lots of stuff,20:16
dansmithbut was then thinking,20:16
dansmitha graph of fails-per-day for each test for the last 30 days might be a good "oh, that's breaking now" indicator20:16
fungimight be more accurate if only limited to gate pipeline and not reported for check pipeline in that case20:17
fungijenkins will give you some of that fromits web interface, but it's quite neutered since we can't have it preserve results for any reasonable timeframe20:19
clarkbI use logstash to get those numbers. I currently only have 2 weeks of semi broken logs :/20:19
clarkbI can tell you that for yesterday UTC time grenade failed in the gate ~10% of the time20:20
clarkb83 successes to 9 failures20:20
dansmithhmm, okay, I guess it seems much higher than that, so I was thinking the departure from the norm would be more visually striking20:24
clarkbdansmith: I think you are right that the recent rate is much higher than that20:29
clarkbbut logstash is too far behind to check current numbers :(20:29
mordredwe could probably report job name + success or fail to statsd20:31
mordredto make those sorts of charts20:31
mordredd-g adds 500M on top of the base ubuntu vm20:34
pleia2yeah, that's why we've talked about caching the packages20:37
*** sarob has joined #openstack-infra20:39
fungimordred: is that after an apt-get clean?20:39
pleia2(whether we run a full ubuntu mirror as openstack, or a partial one)20:39
fungia partial caching proxy mirror is very easy to accomplish for apt, luckily... i run one at home since i have so many machines20:40
*** jaypipes has quit IRC20:41
*** Ryan_Lane has joined #openstack-infra20:41
mordredfungi: no - because we pre-cache all the packages we're going to install - this is doing the d-g dib elements I've been poking at20:41
*** sarob has quit IRC20:43
fungiso /var/cache/apt/archives/ is empty in the resultant image? or has the packages cached but not installed there?20:44
*** dkliban has quit IRC20:44
mordredhas the packages cached there but not installed20:48
mordredit has _all_ of the packages cached that _any_ devstack run _might_ want to install20:48
fungirockstar: the new openstack@ list seems not to add a list-id header?20:48
mordredso that it can install them during devstack under the control of devstack20:48
mordredbut withouth talking to the internets20:48
fungimordred: okay, makes sense. so that's an additional 500mib of compressed packages, cached in the image, not yet installed20:49
mordredand also all the git repos20:51
mordredfungi: I have bad news20:52
mordredfungi:  I cannot reproduce your problem20:52
mordredwith jeepyb20:52
*** prad has quit IRC20:52
fungimordred: well, at a minimum review and review-dev are currently still in that state20:52
fungibut i will see if i can recreate it with a fresh precise vm20:52
* mordred goes to look at review-dev20:53
*** DinaBelova has joined #openstack-infra20:54
*** DinaBelova has quit IRC20:58
*** hartsocks has joined #openstack-infra21:08
fungirockstar: nevermind my previous comment. i finally got far enough through my inbox to see that reed mentioned it to you already21:08
*** afazekas has quit IRC21:13
*** mindjiver has joined #openstack-infra21:17
openstackgerritBob Ball proposed a change to openstack-infra/reviewday: Added time-based score for patches
*** SergeyLukjanov has quit IRC21:21
*** koobs has quit IRC21:28
*** koobs has joined #openstack-infra21:31
openstackgerritBob Ball proposed a change to openstack-infra/reviewday: Added time-based score for patches
*** sarob has joined #openstack-infra21:35
*** sarob has quit IRC21:37
*** sarob_ has joined #openstack-infra21:37
*** dkliban has joined #openstack-infra21:40
mordredclarkb: w00t. I have takovernoded and kexec'd and it worked21:46
*** johnthetubaguy has joined #openstack-infra21:49
*** johnthetubaguy has left #openstack-infra21:50
fungimordred: btw i was able to reproduce... vanilla precise vm with git, pip and python-mysql packages added. clone jeepyb and cd into it, pip install setuptools pbr=0.5.1722:13
fungiat that point python install will work the first time, but will fail when run a second time22:13
fungiclearing the cached eggs in the local source tree doesn't fix it22:14
mordredah! I got it22:15
fungialso, i always forget that there's nothing quite like a summer afternoon layover in chicago when it comes to turbulence22:17
mordredfungi: yes. I believe we are just going to have to upgrade by hand this time22:18
fungiat the moment, it looks like jeepyb is the only infra project which uses pbr and is being deployed in our environment with vcsrepo, so i think that's the only puppet module which needs the workaround thankfully22:31
*** zaro0508 has joined #openstack-infra22:48
*** zaro0508 has quit IRC23:06
*** UtahDave has quit IRC23:29
*** sarob_ has quit IRC23:38
*** sarob has joined #openstack-infra23:39
*** sarob has quit IRC23:39
*** michchap has joined #openstack-infra23:39
*** sarob has joined #openstack-infra23:40
*** koolhead17 has joined #openstack-infra23:49
*** sarob has quit IRC23:50
*** sarob has joined #openstack-infra23:51
clarkbmordred: nice23:54
clarkbfungi: fwiw our cgroups should prevent OOMing :)23:54
clarkbfungi: or rather the tests can OOM all they want but the slave should escape unharmed23:54
fungiclarkb: good point. just worth noting that tmpfs contents will occupy virtual memory23:55
*** sarob has quit IRC23:55
fungiand will most likely fall outside the enforcement of cgroups for that user's processes23:55
clarkbI am actually not sure how tmpfs contents will get accounted against the cgroups23:56
fungibut we do cap their normal memory usage artificially low for the amount of ram we get on those slaves, so maybe not a concern unless we start going 4g+ on the tmpfs sizes23:56
clarkbI am slightly worried about the dirty /tmp situation we have23:58
clarkbit might be worth mounting the tmpfs under a different path and letting specific things use that rather than /tmp23:58

Generated by 2.14.0 by Marius Gedminas - find it at!