| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Make unhandled exceptions not fatal https://review.openstack.org/194146 | 00:00 |
|---|---|---|
| *** dannywilson has quit IRC | 00:00 | |
| *** DericHorn-HP has quit IRC | 00:01 | |
| *** dannywil_ has quit IRC | 00:01 | |
| *** ashleighfarnham has quit IRC | 00:02 | |
| *** docaedo has quit IRC | 00:02 | |
| *** Longgeek has quit IRC | 00:03 | |
| *** amitgandhinz has joined #openstack-infra | 00:03 | |
| jhesketh | lifeless, fungi: looking | 00:04 |
| *** docaedo has joined #openstack-infra | 00:04 | |
| *** Rockyg has quit IRC | 00:05 | |
| *** sdake has quit IRC | 00:07 | |
| *** tqtran-afk is now known as tqtran_ | 00:07 | |
| *** xarses has joined #openstack-infra | 00:07 | |
| *** fallenpegasus has joined #openstack-infra | 00:07 | |
| *** amitgandhinz has quit IRC | 00:07 | |
| jhesketh | fungi: reviewed, but I think I might be missing something | 00:08 |
| pleia2 | my turn for the dentist tomorrow, so I'll be strolling in late (will be back by meeting time) | 00:09 |
| *** dboik has joined #openstack-infra | 00:11 | |
| *** dims has joined #openstack-infra | 00:12 | |
| *** sputnik13 has quit IRC | 00:12 | |
| *** austin81 has joined #openstack-infra | 00:14 | |
| *** dboik has quit IRC | 00:15 | |
| wshao | on moving project from stackforget to openstack, do I submit change to openstack/governance first? the project creator page indicates that I submit changes to infra/project-config first? but I have seen others submit to governance first. | 00:15 |
| *** rm_work is now known as rm_work|away | 00:15 | |
| jhesketh | pleia2: blargh, dentists, don't remind me :-( | 00:16 |
| *** mtanino has quit IRC | 00:16 | |
| pleia2 | just a checkup, hopefully all is well :) | 00:18 |
| *** marun has joined #openstack-infra | 00:18 | |
| fungi | wshao: if you're moving a repo from stackforge to openstack and its adopted by an official project-team, you can submit the change to project-config first | 00:18 |
| *** rlandy has quit IRC | 00:18 | |
| fungi | wshao: if you're applying to the tc to create a new official project-team, you need to submit a change to governance first so the application for forming a new openstack project-team can be discussed by the tc | 00:19 |
| wshao | fungi: project-team already exsits from stackforge projects. | 00:20 |
| *** claudiub has joined #openstack-infra | 00:21 | |
| *** esmute__ has quit IRC | 00:23 | |
| openstackgerrit | Weidong Shao proposed openstack-infra/project-config: Add compas to OpenStack projects. https://review.openstack.org/196916 | 00:24 |
| *** wshao has quit IRC | 00:24 | |
| fungi | wshao: then submit the rename change to project-config (and link it in the renames section of the infra meeting agenda so we can schedule it). also submit a change to update the governance repo so it stays current | 00:24 |
| *** amotoki has joined #openstack-infra | 00:27 | |
| *** ildikov_ has quit IRC | 00:28 | |
| *** austin81 has quit IRC | 00:29 | |
| *** boris-42 has joined #openstack-infra | 00:30 | |
| *** yamamoto has joined #openstack-infra | 00:31 | |
| *** chlong has joined #openstack-infra | 00:32 | |
| *** jsavak has quit IRC | 00:32 | |
| *** geoffarnold has quit IRC | 00:32 | |
| *** ddieterly has joined #openstack-infra | 00:33 | |
| *** marun_ has joined #openstack-infra | 00:40 | |
| davideagnello | hello, this review is pending one more +2: https://review.openstack.org/#/c/195991/ | 00:40 |
| *** ildikov_ has joined #openstack-infra | 00:41 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch constraints proposal to use zuul-git-prep https://review.openstack.org/196907 | 00:41 |
| *** marun has quit IRC | 00:41 | |
| fungi | jhesketh: i think that ^ covers the missing bit | 00:41 |
| jhesketh | fungi: oh, I see, it's a jjb replacement! | 00:42 |
| fungi | yeah, waiting to see if the xml diff job does what i think it should this time | 00:43 |
| jhesketh | cool | 00:43 |
| *** fallenpegasus has quit IRC | 00:45 | |
| fungi | though after this, i realize that we have a branch-git-prep macro which basically does this with the old prep script. still, i want a zuul-git-prep macro anyway | 00:48 |
| fungi | so that maybe we can eventually get rid of the old script | 00:48 |
| *** gmann has quit IRC | 00:48 | |
| *** thorst has joined #openstack-infra | 00:48 | |
| *** notnownikki has quit IRC | 00:50 | |
| tchaypo | fungi: lifeless: actually, both versions of the code in setuptools were okay - the problem was that one version of the code in setuptools is pkg_resources, which is vendored into pip, but hadn’t been updated in Quite Some Time. | 00:53 |
| *** patrickeast has quit IRC | 00:53 | |
| *** abitha has quit IRC | 00:54 | |
| *** larainema has joined #openstack-infra | 00:54 | |
| tchaypo | Nakato’s PR https://github.com/pypa/pip/pull/2894 updates pkg_resources in pip to the version from setuptools 17.1 | 00:54 |
| tchaypo | so as soon as pip gets a 7.1…. | 00:55 |
| Nakato | 18.0 got pulled in instead | 00:56 |
| tchaypo | which sounds like an equally acceptable fix | 00:56 |
| fungi | ahh, yep we can probably update our requirements lists to use inequality matches at that point, sounds like | 00:57 |
| Nakato | Yep | 00:58 |
| *** ayoung has joined #openstack-infra | 00:58 | |
| *** yamamoto has quit IRC | 00:59 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/infra-specs: Zuul github-integration spec. https://review.openstack.org/163637 | 00:59 |
| waynr | jhesketh: ^^ addressed comments in patchset 6 for the zull_github-integration spec | 01:00 |
| *** gmann has joined #openstack-infra | 01:00 | |
| *** marun_ has quit IRC | 01:01 | |
| *** annegentle has joined #openstack-infra | 01:02 | |
| *** otter768 has joined #openstack-infra | 01:03 | |
| *** ashleighfarnham has joined #openstack-infra | 01:03 | |
| waynr | I don't understand the difference between "Rollcall-Vote", "Council-Vote", and probably others I have seen | 01:05 |
| waynr | Is there a good place to read up on these? | 01:06 |
| dkehn | lifeless: interesting, I'm actually thinking its the setup of the venv in the devstack | 01:06 |
| *** annegentle has quit IRC | 01:06 | |
| lifeless | dkehn: can has log or pastebin or something ? | 01:06 |
| *** otter768 has quit IRC | 01:07 | |
| *** ashleighfarnham has quit IRC | 01:07 | |
| dkehn | lifeless: sure but its an aboration of devstack. kind of working on something different, but leveraging the oslo and devestack | 01:08 |
| lifeless | dkehn: ah ok | 01:09 |
| lifeless | dkehn: so in which case you need to copy the code added to devstack recently to make a venv and pip install requirements | 01:09 |
| lifeless | dkehn: without consulting requirements to do so (circular logic :)) | 01:09 |
| dkehn | lifeless: http://paste.openstack.org/show/326551/ | 01:10 |
| dkehn | lifeless: yep, figured as much. just didn't realize that they were installing into venv env | 01:10 |
| *** ildikov_ has quit IRC | 01:11 | |
| lifeless | dkehn: they weren't... :) | 01:11 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch constraints proposal to use zuul-git-prep https://review.openstack.org/196907 | 01:13 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a zuul-git-prep builder macro https://review.openstack.org/196906 | 01:13 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:13 | |
| dkehn | lifeless: ? devstack Install Python packages into a virtualenv so that we can track them | 01:13 |
| lifeless | dkehn: nope | 01:13 |
| lifeless | dkehn: gimme a sec, looking up ref for you | 01:14 |
| dkehn | lifeless: ok, I'm not following that the call to python tht call update.py is premissed of the venv | 01:14 |
| lifeless | look at 7ebe8e0751dab545091e0b114589087009cc4e22 and 40f3e33f734330c32f27882898ab1bb4ab8f3217 in devstack | 01:14 |
| lifeless | 40f3 was first, and slightly buggy | 01:15 |
| dkehn | lifeless: ok, https://github.com/openstack-dev/devstack/blob/master/inc/python#L221 | 01:15 |
| lifeless | dkehn: so yes, thats calling it using the python from the venv | 01:16 |
| lifeless | dkehn: the commits I linked just before should be self explanatory | 01:16 |
| dkehn | lifeless: ok, the crux of the issue that when running update.py of the pastebin it can path its imports which caused me to start looking at what changed | 01:17 |
| dkehn | lifeless: looking at the commits, yes I see | 01:18 |
| *** markvoelker has quit IRC | 01:19 | |
| *** ivar-laz_ has joined #openstack-infra | 01:19 | |
| *** markvoelker has joined #openstack-infra | 01:19 | |
| *** markvoelker has quit IRC | 01:22 | |
| *** ivar-lazzaro has quit IRC | 01:22 | |
| *** markvoelker has joined #openstack-infra | 01:23 | |
| *** markvoelker has quit IRC | 01:23 | |
| *** ivar-laz_ has quit IRC | 01:23 | |
| *** markvoelker has joined #openstack-infra | 01:24 | |
| *** DericHorn-HP has joined #openstack-infra | 01:24 | |
| *** enikanorov2 has quit IRC | 01:24 | |
| *** otter768 has joined #openstack-infra | 01:27 | |
| *** jasondotstar has quit IRC | 01:30 | |
| *** Sukhdev has quit IRC | 01:34 | |
| *** laughterwym_ has joined #openstack-infra | 01:34 | |
| *** laughterwym has quit IRC | 01:34 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Drop use of 'oslo' namespace package https://review.openstack.org/196872 | 01:36 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Switch constraints proposal to use zuul-git-prep https://review.openstack.org/196907 | 01:37 |
| *** wenlock has joined #openstack-infra | 01:39 | |
| *** enikanorov2 has joined #openstack-infra | 01:42 | |
| *** yamahata has quit IRC | 01:43 | |
| *** RichardRaseley has joined #openstack-infra | 01:44 | |
| openstackgerrit | Merged openstack/requirements: keystoneclient: require at least >= 1.2.0 for stable/kilo https://review.openstack.org/196887 | 01:45 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Drop use of 'oslo' namespace package https://review.openstack.org/196872 | 01:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Set temporary version cap on oslo.db https://review.openstack.org/196924 | 01:46 |
| *** esmute__ has joined #openstack-infra | 01:48 | |
| *** tqtran_ has quit IRC | 01:49 | |
| *** RichardRaseley has quit IRC | 01:50 | |
| *** yamamoto has joined #openstack-infra | 01:52 | |
| *** gyee has quit IRC | 01:53 | |
| *** koolhead17 has joined #openstack-infra | 01:54 | |
| *** otter768 has quit IRC | 01:55 | |
| *** wenlock has quit IRC | 01:59 | |
| *** jasondotstar has joined #openstack-infra | 01:59 | |
| *** EricGonczer_ has joined #openstack-infra | 01:59 | |
| *** jasondotstar has quit IRC | 02:00 | |
| *** jasondotstar has joined #openstack-infra | 02:00 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 02:05 | |
| *** jasondotstar has quit IRC | 02:06 | |
| *** jasondotstar has joined #openstack-infra | 02:06 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Set temporary version cap on oslo.db https://review.openstack.org/196924 | 02:06 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Drop use of 'oslo' namespace package https://review.openstack.org/196872 | 02:07 |
| *** ayoung has quit IRC | 02:07 | |
| fungi | jhesketh: looks like i've finally worked out how to get the replacement passed through to the builder macro http://logs.openstack.org/07/196907/4/check/project-config-compare-xml/91e8561/console.html#_2015-06-30_01_46_41_434 | 02:08 |
| *** chlong_ has joined #openstack-infra | 02:08 | |
| *** ayoung has joined #openstack-infra | 02:09 | |
| mtreinish | jeblair, greghaynes: ^^^ I'll push the release now to include those fixes for the latest oslo releases | 02:09 |
| *** ianw has quit IRC | 02:09 | |
| *** chlong__ has joined #openstack-infra | 02:10 | |
| *** chlong has quit IRC | 02:10 | |
| *** camunoz has quit IRC | 02:10 | |
| *** camunoz has joined #openstack-infra | 02:10 | |
| *** ianw has joined #openstack-infra | 02:10 | |
| fungi | lifeless: "Requirement xxx present in multiple files" http://logs.openstack.org/76/193976/9/check/gate-ceilometer-requirements/cfb368d/console.html | 02:11 |
| *** chlong_ has quit IRC | 02:13 | |
| lifeless | sec | 02:13 |
| *** mkj_wk has quit IRC | 02:14 | |
| *** esmute__ has quit IRC | 02:15 | |
| *** mriedem_away has quit IRC | 02:16 | |
| *** mkj_wk has joined #openstack-infra | 02:17 | |
| mtreinish | jeblair, greghaynes: https://pypi.python.org/pypi/subunit2sql/0.7.0 should fix this week's subunit2sql x oslo issue | 02:17 |
| *** weshay has quit IRC | 02:17 | |
| *** unicell has quit IRC | 02:19 | |
| *** bhunter71 has joined #openstack-infra | 02:26 | |
| *** bhunter71 has quit IRC | 02:26 | |
| *** esmute__ has joined #openstack-infra | 02:27 | |
| *** tlian2 has quit IRC | 02:27 | |
| *** EricGonczer_ has quit IRC | 02:27 | |
| *** ashleighfarnham has joined #openstack-infra | 02:27 | |
| *** camunoz_ has joined #openstack-infra | 02:28 | |
| *** dangers_away has quit IRC | 02:28 | |
| *** asselin has quit IRC | 02:28 | |
| *** chlong_ has joined #openstack-infra | 02:28 | |
| *** ianw has quit IRC | 02:29 | |
| *** asselin has joined #openstack-infra | 02:29 | |
| *** dangers_away has joined #openstack-infra | 02:29 | |
| *** camunoz__ has joined #openstack-infra | 02:30 | |
| *** homeless has quit IRC | 02:30 | |
| *** homeless has joined #openstack-infra | 02:31 | |
| *** chlong__ has quit IRC | 02:31 | |
| *** chlong has joined #openstack-infra | 02:31 | |
| *** camunoz has quit IRC | 02:31 | |
| *** baoli has joined #openstack-infra | 02:31 | |
| *** puranamr has quit IRC | 02:33 | |
| openstackgerrit | lifeless proposed openstack/requirements: Require fixtures 1.3.1. https://review.openstack.org/196863 | 02:34 |
| *** camunoz_ has quit IRC | 02:34 | |
| *** chlong_ has quit IRC | 02:34 | |
| lifeless | fungi: looking | 02:36 |
| *** rossella_s has quit IRC | 02:36 | |
| *** rossella_s has joined #openstack-infra | 02:37 | |
| lifeless | fungi: its correct | 02:38 |
| lifeless | test-requirements-py3.txt:29:gabbi>=0.12.0 # Apache-2.0 | 02:38 |
| lifeless | test-requirements.txt:32:gabbi>=0.12.0 # Apache-2.0 | 02:38 |
| lifeless | fungi: this isn't actually what I intended to happen, and I'll fix it, because of stable branches. | 02:38 |
| lifeless | fungi: but hey, its a LOVELY demonstration of the DRY violation those files were :) | 02:38 |
| fungi | lifeless: oh, great point. i didn't spot that was being triggered by having the version-specific reqs lists | 02:39 |
| lifeless | in fact | 02:39 |
| lifeless | I wonder | 02:39 |
| lifeless | yeah | 02:40 |
| lifeless | the old infra code wasn't parsing those files AT ALL | 02:40 |
| lifeless | hur hur hur | 02:40 |
| SlickNik | sdague: When you get a chance can I get some eyeballs on https://review.openstack.org/#/c/192939/, please? Thanks! | 02:41 |
| fungi | lifeless: that doesn't surprise me. we wrote it before projects started working around version-specific reqs by adding those extra lists on their own accord | 02:42 |
| lifeless | fungi: well pbr supported those lists | 02:42 |
| lifeless | fungi: its just a reflection of having separated codebases doing the same thing :) | 02:43 |
| fungi | i think that came later and the requirements check was never updated with similar logic | 02:43 |
| fungi | yes, dry fails again | 02:43 |
| lifeless | but | 02:43 |
| lifeless | I'm going to have to think somewhat carefully about how to address this | 02:43 |
| fungi | looks like nova has the same issue | 02:43 |
| lifeless | in the meantime, unified files FTW | 02:43 |
| *** tlian has joined #openstack-infra | 02:44 | |
| *** mriedem has joined #openstack-infra | 02:45 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Fix misspelled words in some docs https://review.openstack.org/196295 | 02:46 |
| lifeless | fungi: https://review.openstack.org/#/c/196930/ - putting that up the flagpole | 02:48 |
| *** fbo_ has quit IRC | 02:48 | |
| *** mmedvede has quit IRC | 02:50 | |
| fungi | ugh. it seems like while i wasn't looking, basically all the projects grew requirements-py3.txt files | 02:50 |
| fungi | that was supposed to be an ugly hack in one or two libraries | 02:50 |
| *** dimtruck is now known as zz_dimtruck | 02:50 | |
| *** fbo has joined #openstack-infra | 02:51 | |
| mordred | fungi, Shrews, SpamapS, dguerri`:https://review.openstack.org/#/c/168792 turns out jamielennox already had written the ksc patch we need | 02:54 |
| fungi | mordred: he's a stealthy ninja | 02:54 |
| mordred | fungi: I made the mistake of handing him a bunny rabbit | 02:54 |
| fungi | he writes the code you need before you even know you want it | 02:54 |
| fungi | anyway, i'm going to sleep | 02:57 |
| fungi | more hacking on the dawn | 02:57 |
| * fungi zzzzzz | 02:57 | |
| greghaynes | At dawn, we hack. | 02:58 |
| mordred | greghaynes: ++ | 02:58 |
| *** koolhead17 has quit IRC | 02:59 | |
| mordred | greghaynes: sorry - I was AFK mostly since last we spoke - any luck on the swiftclient hack? | 02:59 |
| greghaynes | oh, no, I got a test case set up then got distracted by HP NOC stuff | 02:59 |
| *** Longgeek has joined #openstack-infra | 03:00 | |
| *** laughterwym has joined #openstack-infra | 03:02 | |
| *** laughterwym_ has quit IRC | 03:02 | |
| *** Longgeek has quit IRC | 03:05 | |
| greghaynes | mordred: Is there a pattern in testtools or testr for skipping a test by default but allowing it to be run individually? | 03:05 |
| greghaynes | I want to add my rax + swift testcase with that setup | 03:06 |
| lifeless | fungi: yeah | 03:06 |
| *** homeless has quit IRC | 03:06 | |
| lifeless | fungi: I've had to push a little hard on halting the rot, it was going to get even worse | 03:06 |
| SpamapS | gah | 03:06 |
| greghaynes | lifeless: ^ you would know | 03:06 |
| SpamapS | drives me nuts when I use --debug, and it feeds me a curl command that will never ever work | 03:07 |
| SpamapS | sha1 of the token.... very.. useful? | 03:07 |
| lifeless | greghaynes: skipIf(os.environ.get('OS_SOME_FLAG')=='1', 'Not enabled by OS_SOME_FLAG') | 03:07 |
| greghaynes | lifeless: ah! ty | 03:07 |
| lifeless | erm | 03:07 |
| lifeless | skipUnless | 03:07 |
| lifeless | or != | 03:07 |
| lifeless | or whateves | 03:07 |
| greghaynes | yea, I got the pattern ;) | 03:08 |
| *** kevinbenton has joined #openstack-infra | 03:08 | |
| lifeless | fungi: in nova | 03:09 |
| lifeless | fungi: do this | 03:09 |
| lifeless | diff -u test-requirements{,-py3}.txt | 03:09 |
| lifeless | fungi: whats the PyMySQL status there ? | 03:09 |
| lifeless | fungi: e.g. do I union the MySQL-python thing | 03:09 |
| lifeless | or ... | 03:09 |
| *** woodster_ has quit IRC | 03:11 | |
| *** Guest26658 has quit IRC | 03:12 | |
| *** flashgordon has quit IRC | 03:13 | |
| lifeless | fungi: or nvm, since I had an old tree apparently :) | 03:14 |
| lifeless | I'm starting over | 03:14 |
| *** claudiub has quit IRC | 03:15 | |
| *** tlian has quit IRC | 03:15 | |
| greghaynes | mordred: hrm, my test case doesnt seem to fail in the same way - im doing a clouds.yaml with profile: rackspace and specifying project_id, username, pass, region | 03:16 |
| greghaynes | and AFAICT it 'just works' | 03:16 |
| *** baoli has quit IRC | 03:18 | |
| lifeless | fungi: https://review.openstack.org/196934 | 03:19 |
| lifeless | now, to release unittest2. muahahahahahahahaha | 03:19 |
| openstackgerrit | greghaynes proposed openstack-infra/shade: WIP test case for rax image uploads https://review.openstack.org/196936 | 03:26 |
| greghaynes | fungi: mordred yea, im somewhat confused ^ with a clouds.yaml that looks just like ours in system-config doesnt fail for me | 03:26 |
| openstackgerrit | lifeless proposed openstack/requirements: Require fixtures 1.3.1. https://review.openstack.org/196863 | 03:28 |
| *** masayukig has quit IRC | 03:31 | |
| *** markvoelker has quit IRC | 03:31 | |
| *** camunoz has joined #openstack-infra | 03:31 | |
| *** camunoz__ has quit IRC | 03:33 | |
| openstackgerrit | lifeless proposed openstack-infra/project-config: Don't fail on dupes in version specific files https://review.openstack.org/196940 | 03:33 |
| *** xingchao has quit IRC | 03:33 | |
| lifeless | fungi: ^this might be zomg urgent to unwedge nova | 03:34 |
| lifeless | mordred: ^ | 03:34 |
| lifeless | jhesketh: ^ | 03:34 |
| lifeless | mriedem: ^ FYI | 03:34 |
| lifeless | [btw untested] | 03:34 |
| *** yamada-h has joined #openstack-infra | 03:35 | |
| *** masayukig has joined #openstack-infra | 03:36 | |
| mriedem | lifeless: i've added the depends-on so we'll see | 03:36 |
| mriedem | that's like 5 patches now to unwedge nova :) | 03:36 |
| *** thorst has quit IRC | 03:36 | |
| lifeless | mriedem: I don't think depends-on can help you with that | 03:36 |
| mriedem | oh right b/c it's project-config | 03:37 |
| mriedem | well, at least it'll be an indication that we need it for when people are trying to figure out why the requirements job in the nova mega change is failing | 03:37 |
| *** Longgeek has joined #openstack-infra | 03:37 | |
| *** yamada-h has quit IRC | 03:40 | |
| *** yamahata has joined #openstack-infra | 03:41 | |
| *** fallenpegasus has joined #openstack-infra | 03:41 | |
| *** ddieterly has quit IRC | 03:43 | |
| *** laughterwym_ has joined #openstack-infra | 03:44 | |
| *** laughterwym has quit IRC | 03:44 | |
| *** laughterwym__ has joined #openstack-infra | 03:44 | |
| jhesketh | lifeless: looking | 03:45 |
| lifeless | its needed because the trunk version of nova etc has the defect | 03:46 |
| *** jasondotstar has quit IRC | 03:47 | |
| lifeless | AJaeger: if you're awake ^ | 03:47 |
| lifeless | I don't know if a new release is needed .. | 03:47 |
| lifeless | sory | 03:47 |
| lifeless | new image | 03:47 |
| lifeless | fungi: ^? | 03:47 |
| *** dims has quit IRC | 03:48 | |
| jhesketh | lifeless: I'm trying to understand the problem as I check your change.. could you help explain to me what checking for -py is doing? | 03:52 |
| jhesketh | lifeless: are we excluding test-requirements-py3.txt from being checked? | 03:53 |
| *** mriedem has quit IRC | 03:53 | |
| lifeless | we're excluding it from triggering an error | 03:53 |
| *** sarob has quit IRC | 03:54 | |
| lifeless | jhesketh: we want to tell folk about the problem no matter what | 03:54 |
| lifeless | thats the move of the print | 03:54 |
| lifeless | and we want to not fail if the problem occurs in a py2 or py3 file | 03:54 |
| *** ianw has joined #openstack-infra | 03:54 | |
| lifeless | because the old infra code didn't check that at all | 03:54 |
| *** otter768 has joined #openstack-infra | 03:56 | |
| jhesketh | lifeless: hmm okay, just checking on -py is very loose. Should we make that tighter or leave a comment do you think? | 03:57 |
| lifeless | jhesketh: all the version specific requirements files are named -pyN.txt | 03:57 |
| lifeless | jhesketh: and none of the other requirements files have -py in their name | 03:57 |
| lifeless | jhesketh: so its actually quite specific | 03:58 |
| lifeless | jhesketh: but I can add a comment if you like | 03:58 |
| jhesketh | I'm mostly satisfied, just wondering if somebody would ever innocently add requirements-pyproject.txt or something and have it excluded | 03:58 |
| jhesketh | that'd be pretty odd though | 03:59 |
| lifeless | they'd have to get it through pbr review first | 03:59 |
| lifeless | and then global-requirements review | 03:59 |
| lifeless | and our goal is to remove all these requirements files | 03:59 |
| lifeless | so I'm thinking we're fairly safe | 03:59 |
| jhesketh | right :-) | 03:59 |
| jhesketh | lifeless: so, excuse my ignorance, would the correct fix be to make sure the requirement is only listed once in one of the version specific reqs files? | 04:00 |
| jhesketh | ie maybe we should determine the version and just check those strictly | 04:00 |
| lifeless | jhesketh: hysterical raisins appplies here | 04:00 |
| lifeless | we can't without considerable effort use markers on kilo and juno | 04:00 |
| lifeless | the -pyN files are not additive, they replace the other file. (Yes this is silly. No, it wasn't my fault). | 04:01 |
| *** otter768 has quit IRC | 04:01 | |
| lifeless | so when you have a -pyN file, its going to overlap 99% with the other file | 04:01 |
| lifeless | whatever that may be | 04:01 |
| jhesketh | hmm okay | 04:01 |
| jhesketh | so how come this is blocking nova all of a sudden, what changed? | 04:01 |
| lifeless | the prior infra parser didnt' check the pyN files at all. | 04:02 |
| lifeless | jhesketh: look at the diff in b7a7641677250c8316bddecd5c93681433736209 | 04:02 |
| jhesketh | lifeless: I can't find that sha, which project? | 04:03 |
| lifeless | project-config | 04:03 |
| jhesketh | ah, sorry, was looking in reqs and nova | 04:04 |
| jhesketh | lifeless: okay, I'm satisifed, thanks for helping explain it :-) | 04:05 |
| jhesketh | lifeless: if no other infra pops up soon I can push that change through if it's causing pain for nova | 04:05 |
| lifeless | its blocking them unwedging their gate | 04:05 |
| *** Longgeek has quit IRC | 04:05 | |
| lifeless | https://jenkins07.openstack.org/job/gate-nova-requirements/263/console | 04:05 |
| lifeless | assuming my fix is solid | 04:06 |
| lifeless | I'm going to cross check that in a sec | 04:06 |
| *** Longgeek has joined #openstack-infra | 04:06 | |
| jhesketh | lifeless: I've +A'd it, I'll be around if we need to undo it or do further fixes | 04:06 |
| *** coolsvap|away is now known as coolsvap | 04:08 | |
| lifeless | jhesketh: ack | 04:09 |
| *** sdake has joined #openstack-infra | 04:09 | |
| *** sdake_ has joined #openstack-infra | 04:10 | |
| lifeless | jhesketh: one thing I don't know | 04:10 |
| lifeless | do we run that from the image | 04:10 |
| lifeless | or from a clone | 04:10 |
| lifeless | e.g. I suspect we need a new image with it on it too | 04:10 |
| jhesketh | lifeless: yeah the scripts are seeded to the images (afaik) | 04:11 |
| jhesketh | so we might need to rebuild | 04:11 |
| jhesketh | I can kick that off once it merges | 04:11 |
| *** unicell has joined #openstack-infra | 04:12 | |
| lifeless | cool | 04:12 |
| *** david-lyle has quit IRC | 04:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't fail on dupes in version specific files https://review.openstack.org/196940 | 04:13 |
| *** achanda has joined #openstack-infra | 04:13 | |
| *** sdake has quit IRC | 04:13 | |
| *** david-lyle has joined #openstack-infra | 04:14 | |
| *** zz_ja has quit IRC | 04:14 | |
| *** zz_jgrimm has quit IRC | 04:15 | |
| *** maurosr has quit IRC | 04:15 | |
| *** asselin_ has joined #openstack-infra | 04:17 | |
| jhesketh | lifeless: okay, rebuilding bare-trusty nodes | 04:19 |
| *** kmartin_ has joined #openstack-infra | 04:19 | |
| *** kmartin has quit IRC | 04:19 | |
| *** unicell1 has joined #openstack-infra | 04:22 | |
| *** unicell has quit IRC | 04:23 | |
| *** zz_ja has joined #openstack-infra | 04:24 | |
| *** ihrachyshka has joined #openstack-infra | 04:24 | |
| *** ihrachyshka has quit IRC | 04:25 | |
| *** ihrachyshka_ has joined #openstack-infra | 04:25 | |
| *** sdake has joined #openstack-infra | 04:25 | |
| *** zz_jgrimm has joined #openstack-infra | 04:26 | |
| *** maurosr has joined #openstack-infra | 04:27 | |
| ihrachyshka_ | any project-config cores? I have real need for someone to check https://review.openstack.org/#/c/196718/ to get testing up for qos neutron sprint today... clarkb, mordred, fungi sdague, anteaya, jhesketh maybe? | 04:27 |
| jhesketh | ihrachyshka_: looking | 04:27 |
| ihrachyshka_ | thanks! the idea is to modify neutron.conf a bit for specific branch to get a service plugin that is in developement to be enabled. | 04:28 |
| *** DericHorn-HP has quit IRC | 04:28 | |
| ihrachyshka_ | the stub of the plugin is already in the tree | 04:28 |
| *** sdake_ has quit IRC | 04:29 | |
| *** achanda has quit IRC | 04:30 | |
| *** hogepodge has quit IRC | 04:32 | |
| *** markvoelker has joined #openstack-infra | 04:32 | |
| jhesketh | ihrachyshka_: lgtm | 04:32 |
| *** hogepodge has joined #openstack-infra | 04:32 | |
| ihrachyshka_ | jhesketh, thanks a lot! those branch envs are a bit of mystery for me, I copied them from some other place. Hopefully it will work as I expect. :) | 04:33 |
| jlvillal | tonyb, Any look getting some love for the eavesdrop 'calendars' patch? | 04:34 |
| jlvillal | s/look/luck/ | 04:34 |
| jlvillal | https://review.openstack.org/194488 | 04:34 |
| *** DericHorn-HP has joined #openstack-infra | 04:35 | |
| *** maishsk has quit IRC | 04:35 | |
| *** yamada-h has joined #openstack-infra | 04:36 | |
| *** markvoelker has quit IRC | 04:36 | |
| *** maishsk has joined #openstack-infra | 04:37 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 04:39 | |
| *** zz_jgrimm has quit IRC | 04:39 | |
| *** laughterwym__ has quit IRC | 04:40 | |
| *** zz_ja has quit IRC | 04:40 | |
| *** maurosr has quit IRC | 04:40 | |
| *** yamada-h has quit IRC | 04:41 | |
| *** wshao has joined #openstack-infra | 04:41 | |
| *** zz_jgrimm has joined #openstack-infra | 04:42 | |
| *** maurosr has joined #openstack-infra | 04:42 | |
| *** zz_ja has joined #openstack-infra | 04:43 | |
| *** ddieterly has joined #openstack-infra | 04:44 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Run diskimage-builder-install macro for akanda image build https://review.openstack.org/196946 | 04:45 |
| *** maishsk has quit IRC | 04:47 | |
| *** esmute__ has quit IRC | 04:47 | |
| *** yfried|afk has quit IRC | 04:48 | |
| *** ddieterly has quit IRC | 04:49 | |
| *** amotoki_ has joined #openstack-infra | 04:49 | |
| *** laughterwym_ has quit IRC | 04:51 | |
| *** kiran-r has joined #openstack-infra | 04:52 | |
| *** rm_work|away is now known as rm_work | 04:52 | |
| tonyb | jlvillal: no not yet. I/we need to work out if there is a delete part of the publish job. That'll become important over time. | 04:53 |
| *** DericHorn-HP has quit IRC | 04:58 | |
| *** sdake_ has joined #openstack-infra | 04:59 | |
| *** kiran-r has quit IRC | 05:00 | |
| *** erlon has quit IRC | 05:00 | |
| *** kiran-r has joined #openstack-infra | 05:03 | |
| *** sdake has quit IRC | 05:03 | |
| *** fallenpegasus has quit IRC | 05:03 | |
| *** sputnik13 has joined #openstack-infra | 05:04 | |
| *** tjones1 has joined #openstack-infra | 05:04 | |
| *** fallenpegasus has joined #openstack-infra | 05:07 | |
| jlvillal | tonyb, Thanks for the info :) | 05:09 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Update the run_time graph image in docs https://review.openstack.org/196952 | 05:10 |
| *** ihrachyshka_ has quit IRC | 05:13 | |
| jhesketh | lifeless: images are rebuilt, just need to wait for some nodes to launch with the new images | 05:14 |
| *** kiran-r has quit IRC | 05:17 | |
| *** fawadkhaliq has joined #openstack-infra | 05:18 | |
| *** kiran-r has joined #openstack-infra | 05:18 | |
| *** ildikov has joined #openstack-infra | 05:22 | |
| *** _kiran_ has joined #openstack-infra | 05:23 | |
| *** fallenpegasus has quit IRC | 05:23 | |
| *** amotoki__ has joined #openstack-infra | 05:24 | |
| *** kiran-r has quit IRC | 05:24 | |
| *** amotoki_ has quit IRC | 05:24 | |
| *** fallenpegasus has joined #openstack-infra | 05:27 | |
| *** _kiran_ is now known as kiran-r | 05:27 | |
| *** Sukhdev has joined #openstack-infra | 05:28 | |
| *** abregman has joined #openstack-infra | 05:37 | |
| *** BharatK has joined #openstack-infra | 05:39 | |
| openstackgerrit | Weidong Shao proposed openstack-infra/project-config: Add compass to OpenStack projects https://review.openstack.org/196916 | 05:42 |
| *** asettle has joined #openstack-infra | 05:42 | |
| *** hogepodge has quit IRC | 05:42 | |
| *** hogepodge has joined #openstack-infra | 05:42 | |
| *** tjones1 has quit IRC | 05:44 | |
| *** sputnik1_ has joined #openstack-infra | 05:45 | |
| *** ddieterly has joined #openstack-infra | 05:45 | |
| *** abregman has quit IRC | 05:46 | |
| *** kiran-r has quit IRC | 05:46 | |
| *** sputnik13 has quit IRC | 05:48 | |
| *** achanda has joined #openstack-infra | 05:48 | |
| lifeless | jhesketh: cool | 05:50 |
| jhesketh | lifeless: hmm, looks like a lot of jobs are failing though | 05:50 |
| *** ddieterly has quit IRC | 05:50 | |
| jhesketh | just trying to figure out if it's the new nodes or something else | 05:50 |
| *** deepakcs has joined #openstack-infra | 05:51 | |
| lifeless | jhesketh: ack | 05:52 |
| jhesketh | looks like it's failing on more than just bare-trusty | 05:53 |
| lifeless | jhesketh: got a log or something? | 05:54 |
| *** yamamoto has quit IRC | 05:55 | |
| jhesketh | lifeless: there's a bunch of unstable jobs in the queue https://jenkins02.openstack.org/job/gate-nova-python27/835/console | 05:55 |
| *** fallenpegasus has quit IRC | 05:55 | |
| jhesketh | they all appear to have trouble uploading logs | 05:55 |
| *** markvoelker has joined #openstack-infra | 05:56 | |
| *** otter768 has joined #openstack-infra | 05:57 | |
| lifeless | hmmm | 05:57 |
| lifeless | I can't tell if the job ran with the new code or not | 05:57 |
| lifeless | for https://jenkins01.openstack.org/job/gate-nova-requirements/37/console | 05:58 |
| *** yamamoto has joined #openstack-infra | 05:58 | |
| jhesketh | lifeless: me either, but I don't think that problem is related to the others which is more urgent at the moment | 05:58 |
| lifeless | the /usr/zuul-swift-logs-env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true | 05:59 |
| lifeless | lines are noise | 05:59 |
| *** markvoelker has quit IRC | 06:01 | |
| lifeless | jhesketh: ah higher up | 06:01 |
| *** wshao has quit IRC | 06:01 | |
| *** otter768 has quit IRC | 06:02 | |
| jhesketh | lifeless: sorry? what are you looking at? | 06:02 |
| *** abregman has joined #openstack-infra | 06:02 | |
| *** wshao has joined #openstack-infra | 06:02 | |
| lifeless | 2015-06-30 05:44:40.802 | 2: No such file | 06:03 |
| lifeless | 2015-06-30 05:44:40.802 | at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2833) | 06:03 |
| lifeless | 2015-06-30 05:44:40.803 | at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2185) | 06:03 |
| lifeless | 2015-06-30 05:44:40.803 | at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2202) | 06:03 |
| lifeless | 2015-06-30 05:44:40.803 | at com.jcraft.jsch.ChannelSftp.stat(ChannelSftp.java:2159) | 06:03 |
| lifeless | 2015-06-30 05:44:40.803 | at be.certipost.hudson.plugin.SCPSite.mkdirs(SCPSite.java:314) | 06:03 |
| lifeless | jhesketh: from https://jenkins02.openstack.org/job/gate-nova-python27/835/console | 06:03 |
| jhesketh | right, we're seeing a lot of that | 06:03 |
| *** achanda has quit IRC | 06:04 | |
| wshao | on openstack infra access to irc: do I need to set up permission somewhere? I got irc-access check error. openstackinfra does not have permissions on #openstack-compass: | 06:05 |
| jhesketh | hmm, I don't have the passwords to jenkins to check the creds :-( | 06:05 |
| *** fallenpegasus has joined #openstack-infra | 06:05 | |
| jhesketh | the disk certainly has space and other logs exist | 06:06 |
| *** ihrachyshka_ has joined #openstack-infra | 06:07 | |
| *** Sukhdev_ has joined #openstack-infra | 06:08 | |
| lifeless | jhesketh: they'll be in hiera? | 06:09 |
| *** Sukhdev has quit IRC | 06:11 | |
| *** k4n0 has joined #openstack-infra | 06:12 | |
| *** ParsectiX has joined #openstack-infra | 06:16 | |
| *** larainema has quit IRC | 06:17 | |
| *** maishsk_afk has joined #openstack-infra | 06:17 | |
| *** maishsk_afk is now known as maishsk | 06:17 | |
| *** wshao has quit IRC | 06:18 | |
| *** _nadya_ has joined #openstack-infra | 06:23 | |
| *** yfried|afk has joined #openstack-infra | 06:24 | |
| *** achanda has joined #openstack-infra | 06:26 | |
| *** Qiming has joined #openstack-infra | 06:28 | |
| *** yfried has joined #openstack-infra | 06:29 | |
| *** yfried|afk has quit IRC | 06:29 | |
| *** sdake_ has quit IRC | 06:29 | |
| *** mpaolino has joined #openstack-infra | 06:30 | |
| openstackgerrit | Romil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under openstack namespace https://review.openstack.org/196958 | 06:33 |
| *** scheuran has joined #openstack-infra | 06:33 | |
| AJaeger | fungi, clarkb, pleia2 : All the translation jobs fail in the "git review" step. Error is "fatal: ambiguous argument '(detached': unknown revision or path not in the working tree." | 06:35 |
| AJaeger | fungi, is this a problem due to changing to the new proposal slave? | 06:35 |
| AJaeger | fungi, clarkb, pleia2: log file at https://jenkins.openstack.org/job/nova-propose-translation-update/55/console | 06:35 |
| openstackgerrit | Wayne Warren proposed openstack-infra/infra-specs: Zuul github-integration spec. https://review.openstack.org/163637 | 06:36 |
| *** AzherKhan has joined #openstack-infra | 06:36 | |
| *** rossella_s has quit IRC | 06:36 | |
| AJaeger | fungi, and a different error on https://jenkins.openstack.org/job/horizon-propose-translation-update/55/console | 06:37 |
| *** rossella_s has joined #openstack-infra | 06:37 | |
| AJaeger | fungi: upstream-translation-update looks fine, indeed. | 06:38 |
| *** asettle has quit IRC | 06:39 | |
| jhesketh | lifeless: do you know where in Jenkins scp dests are set? | 06:43 |
| openstackgerrit | Romil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under OpenStack namespace https://review.openstack.org/196958 | 06:44 |
| *** ddieterly has joined #openstack-infra | 06:46 | |
| *** Sukhdev_ has quit IRC | 06:47 | |
| *** mrunge has joined #openstack-infra | 06:48 | |
| *** mrunge has quit IRC | 06:48 | |
| *** mrunge has joined #openstack-infra | 06:48 | |
| lifeless | jhesketh: I don't sorry | 06:49 |
| lifeless | jhesketh: I can help you poke around for that once C is in bed | 06:49 |
| jhesketh | lifeless: all good | 06:49 |
| jhesketh | thanks :-) | 06:49 |
| *** BharatK has quit IRC | 06:49 | |
| *** ddieterly has quit IRC | 06:51 | |
| StevenK | GAH | 06:52 |
| StevenK | Zanata, you terrible thing | 06:52 |
| AJaeger | StevenK: What has Zanata done? | 06:53 |
| *** devvesa has joined #openstack-infra | 06:53 | |
| *** doude has joined #openstack-infra | 06:53 | |
| StevenK | AJaeger: It looks like both django_openstack_auth and horizon pull and push from the same horizon project with different resources. Telling zanata to push the django_openstack_auth translations to horizon caused it to remove horizon's files | 06:55 |
| *** KanagarajM has joined #openstack-infra | 06:56 | |
| *** sshnaidm has joined #openstack-infra | 06:57 | |
| KanagarajM | Hi, could some cores kindly review https://review.openstack.org/#/c/195531/ . thanks | 06:57 |
| openstackgerrit | Romil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under OpenStack namespace https://review.openstack.org/196958 | 06:58 |
| AJaeger | StevenK: yes, the push from the same project. | 06:58 |
| AJaeger | StevenK: We can split this up if needed for Zanata so that "Zanata project" == "repository" | 06:58 |
| AJaeger | You have the same challenge with Manuals - openstack-manuals, security-doc, ha-guide and api-site repositories each have a resource in the Transifex "Manuals" project | 06:59 |
| StevenK | AJaeger: Yup, I have a thinko in the populate script that I'm about the push up, then I'll do that. | 06:59 |
| StevenK | AJaeger: Ah ha, manuals are already split out into 4 projects | 06:59 |
| StevenK | Which I did by accident, rather than design | 06:59 |
| AJaeger | StevenK: Please document this change (The 1:1 relationship) of behaviour somewhere. | 06:59 |
| openstackgerrit | Kiall Mac Innes proposed openstack-infra/project-config: Add openstack/designate-dashboard to designate DSVM gates https://review.openstack.org/196961 | 07:00 |
| StevenK | AJaeger: Where would you recommend? | 07:00 |
| openstackgerrit | Steve Kowalik proposed openstack-infra/project-config: Correct project name for django_openstack_auth https://review.openstack.org/196962 | 07:00 |
| *** dtantsur has joined #openstack-infra | 07:00 | |
| *** pblaho_ has joined #openstack-infra | 07:00 | |
| AJaeger | StevenK: WE currently have https://wiki.openstack.org/wiki/Translations#Adding_a_new_project_to_transifex and https://wiki.openstack.org/wiki/Translations/Infrastructure. I think adding something to the later patch might be best | 07:01 |
| *** dtantsur|afk has quit IRC | 07:01 | |
| *** fallenpegasus has quit IRC | 07:01 | |
| *** boris-42 has quit IRC | 07:02 | |
| StevenK | AJaeger: I was planning on taking an axe to those scripts after we remove Transifex support, since they can stop being special snowflakes | 07:02 |
| *** dguerri` is now known as dguerri | 07:02 | |
| *** achanda has quit IRC | 07:03 | |
| *** BharatK has joined #openstack-infra | 07:03 | |
| AJaeger | StevenK: Cool! | 07:04 |
| *** pblaho has quit IRC | 07:04 | |
| *** sputnik1_ has quit IRC | 07:05 | |
| *** dtantsur has quit IRC | 07:05 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/project-config: Change Zanata project for django_openstack_auth https://review.openstack.org/196964 | 07:05 |
| openstackgerrit | Weidong Shao proposed openstack-infra/irc-meetings: Add Compass weekly meeting https://review.openstack.org/196965 | 07:06 |
| StevenK | AJaeger: WRT to adding a project to Zanata, https://review.openstack.org/187867 does that -- we might be able to store a translation-projects.txt file somewhere, and just run that script when we want to add a new one. | 07:06 |
| *** pblaho_ has quit IRC | 07:06 | |
| *** pblaho has joined #openstack-infra | 07:06 | |
| *** dguerri is now known as dguerri` | 07:07 | |
| *** mpavlase has joined #openstack-infra | 07:07 | |
| *** mpavlase has quit IRC | 07:09 | |
| *** mpavlase has joined #openstack-infra | 07:09 | |
| *** kiran-r has joined #openstack-infra | 07:09 | |
| AJaeger | StevenK: WE could do this automatically. Add translation-projects.txt to project config and whenever we change it, we have a post job that runs and creates the project. | 07:09 |
| AJaeger | For that we would need to move your script to project-config/jenkins/scripts | 07:09 |
| *** Hal has joined #openstack-infra | 07:10 | |
| AJaeger | StevenK: That would make translation setup much nicer ;). pleia2, what do you think of the above idea? ^ | 07:10 |
| *** Hal is now known as Guest16240 | 07:10 | |
| *** dtantsur has joined #openstack-infra | 07:10 | |
| *** hdd has quit IRC | 07:11 | |
| StevenK | AJaeger: That idea could certainly work. I do like the idea of commit a file, magic happens. | 07:11 |
| AJaeger | StevenK: so do I - see how we manage project creation for git.openstack.org. A change to project-config/gerrit/projects.yaml creates new repos... | 07:12 |
| lifeless | jhesketh: ok, backish | 07:13 |
| lifeless | 'sup | 07:13 |
| StevenK | AJaeger: Maybe we leverage projects.yaml and add translation: yes or something? | 07:13 |
| *** Guest16240 has quit IRC | 07:14 | |
| AJaeger | StevenK: please discuss with the rest of the team what's the best way to do this. I'm not sure about the best architecture here. | 07:14 |
| AJaeger | Leveraging projects.yaml looks like a nice idea | 07:14 |
| StevenK | AJaeger: I was just about to ask if you'll be at the infra meeting, because it's 6am for me | 07:14 |
| AJaeger | It'S 9pm for me ;) | 07:15 |
| AJaeger | StevenK: I'll try to join | 07:15 |
| StevenK | AJaeger: It's all good, let me just drop pleia2 an e-mail and she can bring it up | 07:15 |
| jhesketh | lifeless: trying to figure out why jenkins can't upload its logs still | 07:15 |
| jhesketh | looks like the jenkins nodes have the right sshkeys etc | 07:16 |
| jhesketh | currently looking for some more info in logs than just "failed to upload" | 07:16 |
| *** hichihara has quit IRC | 07:17 | |
| AJaeger | Thanks, StevenK ! | 07:17 |
| *** markus_z has joined #openstack-infra | 07:19 | |
| *** fhubik has joined #openstack-infra | 07:19 | |
| *** kaisers1 has joined #openstack-infra | 07:19 | |
| openstackgerrit | Romil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under OpenStack namespace https://review.openstack.org/196958 | 07:19 |
| *** samuelBartel has joined #openstack-infra | 07:20 | |
| *** HeOS has quit IRC | 07:22 | |
| *** matbu has joined #openstack-infra | 07:23 | |
| ihrachyshka_ | SergeyLukjanov, hi! any chance you check out the following patch to enable testing for a neutron qos branch that we currently work on a sprint: https://review.openstack.org/196718 ? | 07:23 |
| openstackgerrit | Jaume Devesa proposed openstack-infra/puppet-openstackci: Add Jenkins Job Builder manifest https://review.openstack.org/184919 | 07:23 |
| *** matbu has left #openstack-infra | 07:23 | |
| AJaeger | ihrachyshka_, SergeyLukjanov: I'll check... | 07:24 |
| ihrachyshka_ | AJaeger, thanks! I hope this branch magic is valid; I stole it from some other place. | 07:25 |
| AJaeger | ihrachyshka_: Looks fine, let's try it ;) | 07:27 |
| AJaeger | ihrachyshka_: the if could be much simpler, you don't need pattern matching with =~, a simple "=" would work as well - but it's not wrong, so I +2A | 07:27 |
| ihrachyshka_ | AJaeger, thanks! it's a temporary thing, we'll kill it once merged back later this cycle | 07:28 |
| *** kaisers1 has quit IRC | 07:29 | |
| *** kaisers1 has joined #openstack-infra | 07:29 | |
| *** gema has left #openstack-infra | 07:30 | |
| openstackgerrit | Merged openstack-infra/project-config: neutron: enable qos service plugin for feature/qos branch https://review.openstack.org/196718 | 07:32 |
| *** e0ne has joined #openstack-infra | 07:33 | |
| *** camunoz has quit IRC | 07:34 | |
| *** soren has joined #openstack-infra | 07:34 | |
| *** chlong has quit IRC | 07:35 | |
| jhesketh | lifeless: I'm going to delete the new images see if it is them | 07:37 |
| *** jlanoux has joined #openstack-infra | 07:37 | |
| *** e0ne has quit IRC | 07:37 | |
| *** hashar has joined #openstack-infra | 07:41 | |
| *** jistr has joined #openstack-infra | 07:43 | |
| *** fhubik is now known as fhubik_afk | 07:44 | |
| *** markvoelker has joined #openstack-infra | 07:45 | |
| *** camunoz has joined #openstack-infra | 07:45 | |
| *** jtomasek has joined #openstack-infra | 07:46 | |
| jhesketh | SergeyLukjanov: are you around per chance? | 07:47 |
| *** ddieterly has joined #openstack-infra | 07:47 | |
| *** mrunge has quit IRC | 07:48 | |
| *** armax has quit IRC | 07:48 | |
| *** mpavone has joined #openstack-infra | 07:49 | |
| *** markvoelker has quit IRC | 07:50 | |
| *** enikanorov2_ has joined #openstack-infra | 07:50 | |
| *** amotoki__ has quit IRC | 07:50 | |
| *** mrmartin has joined #openstack-infra | 07:50 | |
| jhesketh | lifeless: are you still around? | 07:51 |
| *** ddieterly has quit IRC | 07:51 | |
| *** yamahata has quit IRC | 07:52 | |
| *** enikanorov2 has quit IRC | 07:52 | |
| lifeless | ish | 07:52 |
| *** rm_work is now known as rm_work|away | 07:53 | |
| *** rm_work|away is now known as rm_work | 07:54 | |
| *** dizquierdo has joined #openstack-infra | 07:55 | |
| jhesketh | lifeless: so I thought [SCP] No file(s) found: **/*nose_results.html was a common problem and that it doesn't matter if they don't exist.. now I'm wondering if it's failing because they don't. So my question is do you know of anything that might have changed in the test suite to stop nose_results.html from being generated? | 07:55 |
| *** sdake_ has joined #openstack-infra | 07:55 | |
| lifeless | jhesketh: its a common thing, we don't use nose for most projects | 07:56 |
| lifeless | jhesketh: the backtrace in jenkins is coming from mkdirs | 07:56 |
| jhesketh | yeah, I'm trying to figure out why some jobs are still passing | 07:56 |
| lifeless | common server? | 07:57 |
| lifeless | statics a single box right? | 07:57 |
| jhesketh | yes | 07:57 |
| lifeless | what version of jcsh is this ? | 07:57 |
| * lifeless wants to look up the lines of code | 07:58 | |
| *** shardy_ has joined #openstack-infra | 07:58 | |
| *** otter768 has joined #openstack-infra | 07:58 | |
| jhesketh | lifeless: jcsh sorry? | 07:59 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Add periodic Kilo jobs https://review.openstack.org/196709 | 07:59 |
| *** shardy has quit IRC | 07:59 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Compass weekly meeting https://review.openstack.org/196965 | 08:00 |
| lifeless | the java sftp client | 08:00 |
| lifeless | com.jcraft.jsch.ChannelSftp.throwStatusErro | 08:00 |
| lifeless | error 2 is indeed: | 08:01 |
| lifeless | 08:01 | |
| lifeless | 2No such fileA reference was made to a file which does not exist. | 08:02 |
| *** pblaho_ has joined #openstack-infra | 08:02 | |
| *** sergmelikyan has joined #openstack-infra | 08:02 | |
| *** dtantsur_ has joined #openstack-infra | 08:02 | |
| *** claudiub has joined #openstack-infra | 08:02 | |
| jhesketh | lifeless: here is the scp plugin details, not sure where to find that particular module http://paste.openstack.org/show/327729/ | 08:02 |
| *** fhubik_lunch has joined #openstack-infra | 08:02 | |
| *** otter768 has quit IRC | 08:02 | |
| jhesketh | hmm, I wonder if something changed in csftp | 08:03 |
| *** yfried is now known as yfried|afk | 08:03 | |
| anteaya | lifeless: in response to http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-06-29.log.html#t2015-06-29T22:10:49 | 08:03 |
| *** jistr has quit IRC | 08:03 | |
| anteaya | lifeless: not to the best of my knowledge, no | 08:03 |
| hashar | iirc your scp plugin is a fork from upstream | 08:03 |
| hashar | with the ability to copy the console | 08:03 |
| hashar | (hello /// just saying) | 08:03 |
| *** shardy_ has quit IRC | 08:03 | |
| *** dtantsur has quit IRC | 08:03 | |
| jhesketh | hashar: ah thanks, that's helpful :-) | 08:03 |
| hashar | and some publishers can be made to not fail on error or when some material/log is not found | 08:04 |
| lifeless | hashar: this is erroring on the sftp side, not the source side | 08:04 |
| *** shardy has joined #openstack-infra | 08:04 | |
| anteaya | lifeless: would you like to complain about them? if so might I have the url of a patch? | 08:05 |
| lifeless | hashar: (have a look at the error in https://jenkins02.openstack.org/job/gate-nova-python27/835/console) | 08:05 |
| *** pblaho has quit IRC | 08:05 | |
| hashar | untrusted cert!!!!!!!!!!!!!! | 08:05 |
| *** Hal has joined #openstack-infra | 08:05 | |
| *** fhubik_afk has quit IRC | 08:05 | |
| *** Hal is now known as Guest16792 | 08:06 | |
| hashar | seems it fails while doing the mkdirs for /srv/static/logs/91/195191/6/check/gate-nova-python27/4a1fdae | 08:06 |
| hashar | might not be recursive | 08:06 |
| hashar | or one of the parent path lack appropriate access rights | 08:06 |
| *** fhubik_lunch has quit IRC | 08:07 | |
| *** fhubik_lunch has joined #openstack-infra | 08:07 | |
| *** fhubik_lunch is now known as fhubik_afk | 08:07 | |
| *** pblaho_ has quit IRC | 08:07 | |
| *** dtantsur_ has quit IRC | 08:07 | |
| *** pblaho has joined #openstack-infra | 08:08 | |
| jhesketh | lifeless: http://paste.openstack.org/show/327739/ (the version(s)) | 08:08 |
| jhesketh | hashar: I believe the scp plugin will do it recusviely | 08:08 |
| hashar | yeah seems so | 08:09 |
| jhesketh | hashar: either way some of the jobs that have failed have the parent dir and the right permissions | 08:09 |
| jhesketh | as far as I can tell anyway | 08:09 |
| hashar | at be.certipost.hudson.plugin.SCPSite.mkdirs(SCPSite.java:314) | 08:09 |
| hashar | but that might be a perm issue on static.openstack.org ? | 08:09 |
| hashar | just speculating | 08:09 |
| *** jistr has joined #openstack-infra | 08:10 | |
| jhesketh | hashar: possibly, but jenkins owns all the dirs | 08:11 |
| jd__ | AJaeger: could you add me into aodh-core 'cause the group is empty for now :-) | 08:11 |
| jhesketh | in /srv/static/logs | 08:11 |
| openstackgerrit | Sirushti Murugesan proposed stackforge/gertty: Specify color for unstable test job https://review.openstack.org/196974 | 08:11 |
| * anteaya goes back to bed | 08:11 | |
| *** mpavlase has quit IRC | 08:11 | |
| AJaeger | jd__: I don't have permissions, you need one of the other infra roots. jhesketh can you help, jd__, please? | 08:11 |
| jhesketh | the scp module(s) aren't very new so I don't think anything has changed there | 08:11 |
| jd__ | AJaeger: you looked so smart and useful I thought you were root by now ;) | 08:12 |
| *** dtantsur has joined #openstack-infra | 08:12 | |
| AJaeger | jd__: Nope, I have enough on my plate already and would reject that extra work ;) | 08:12 |
| AJaeger | jd__: thanks! | 08:12 |
| jd__ | hehe | 08:12 |
| *** acruz has joined #openstack-infra | 08:12 | |
| jhesketh | jd__: what's your gerrit username and where is the aodh repo/group created (ie the change to show me you are meant to be core) | 08:13 |
| jd__ | jhesketh: jdanjou and https://review.openstack.org/#/c/192688/ | 08:14 |
| *** sergmelikyan has quit IRC | 08:14 | |
| eantyshev | asselin, patrickeast, lennyb: thanks to all for yesterday's duscussion, I've summarized my findings there: https://storyboard.openstack.org/#!/story/2000309 | 08:14 |
| *** sergmelikyan has joined #openstack-infra | 08:15 | |
| jhesketh | jd__: done, let me know if that doesn't work | 08:15 |
| jd__ | jhesketh: looks good, thanks a lot! | 08:15 |
| hashar | so looking at the java code | 08:16 |
| hashar | it attempts to create the dir remotely | 08:16 |
| hashar | catch SftpException | 08:16 |
| hashar | since mkdir raise an exception if the dir already exists | 08:16 |
| *** pelix has joined #openstack-infra | 08:16 | |
| hashar | then it does dirstat = channel.stat(curdir + "/" + pathnames[i]); | 08:16 |
| hashar | to ensure the dir is actually there | 08:16 |
| hashar | so in short: seems it can't create the dir :D | 08:17 |
| hashar | but there is no logging for the error :/ | 08:17 |
| *** mase_x200 has joined #openstack-infra | 08:19 | |
| lifeless | jhesketh: anything in syslog ? | 08:20 |
| *** haypo has joined #openstack-infra | 08:20 | |
| lifeless | jhesketh: auditlog? etc | 08:20 |
| hashar | a previous passing build is https://jenkins02.openstack.org/job/gate-nova-python27/828/console | 08:20 |
| hashar | [SCP] Trying to create /srv/static/logs/91/195191/3/gate/gate-nova-python27/c03c360/tox | 08:20 |
| hashar | [SCP] uploading file: '/srv/static/logs/91/195191/3/gate/gate-nova-python27/c03c360//tox/py27-0.log' | 08:21 |
| jhesketh | lifeless: good idea, I'll look in a second, but I'm beginning to think scp has had a release | 08:21 |
| haypo | hi. i see that oslo.db 2.0 was released whereas my fix for nova was not merged (https://review.openstack.org/#/c/195191/). i also see that most nova check jobs are "UNSTABLE" :-/ | 08:21 |
| hashar | don't you deploy the Jenkins plugin from a local mirror ? | 08:21 |
| AJaeger | jd__: thanks for approving the translation setup for aodh. It might take several months until translators work on it, so don't worry. I'll check that the scripts work... | 08:21 |
| jhesketh | lifeless: well http://paste.openstack.org/show/327729/ shows we're running 1.9-SNAPSHOT, while the latest release is 1.8 | 08:21 |
| jhesketh | so I'm trying to figure out if we need to pin down that version | 08:21 |
| *** yfried|afk is now known as yfried | 08:22 | |
| jd__ | AJaeger: thanks for the help! | 08:22 |
| jhesketh | hashar: I don't think so... it looks like it's downloading from http://updates.jenkins-ci.org/download, but I'm trying to figure that out | 08:22 |
| lifeless | haypo: yes, the world has assploded | 08:22 |
| lifeless | haypo: we have a munged up thing ready to go | 08:22 |
| *** derekh has joined #openstack-infra | 08:22 | |
| lifeless | haypo: blocked on fixing the requirements check | 08:22 |
| lifeless | haypo: which is counfounded by the new images that fix it being exploding | 08:23 |
| hashar | jhesketh: https://jenkins02.openstack.org/pluginManager/api/json?pretty=true&depth=1 yields the list of plugins | 08:24 |
| dtantsur | so rechecks won't work, right? | 08:24 |
| hashar | and scp is definitely at "1.9-SNAPSHOT (private-01/21/2014 12:53-khaido)" | 08:24 |
| jhesketh | dtantsur: no, I doubt it | 08:24 |
| dtantsur | ack | 08:24 |
| jhesketh | hashar: and there is this http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/jenkins_master.pp#n55 | 08:25 |
| jhesketh | but I can't find where scp is actually being installed :-( | 08:25 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: heat temprevert: Add port_security_enabled for Neutron https://review.openstack.org/196659 | 08:25 |
| *** sergmelikyan has quit IRC | 08:25 | |
| *** john-davidge has joined #openstack-infra | 08:26 | |
| *** sergmelikyan has joined #openstack-infra | 08:26 | |
| lifeless | jhesketh: oh, its remoted by the plugin from the server | 08:26 |
| lifeless | jhesketh: (I'm 99% sure, hashar can confirm) | 08:27 |
| jhesketh | lifeless: as in the jenkins master does the upload? yes, I believe that's correct | 08:27 |
| hashar | ah yeah most probably the master is doing it | 08:27 |
| hashar | instead of the slave | 08:27 |
| *** yamamoto has quit IRC | 08:27 | |
| hashar | I would suspect the plugin on the master to retrieve the files from the slaves over the master/slave connection | 08:28 |
| hashar | then the master to ssh to static and scp stuff there | 08:28 |
| hashar | but that is really speculation | 08:28 |
| lifeless | so the sftp code isn't the issue | 08:29 |
| *** isviridov_away is now known as isviridov | 08:29 | |
| hashar | on static, you might want to look at /var/log/auth.log to figure out what is scping | 08:32 |
| hashar | to confirm whether it is the master or slave | 08:32 |
| hashar | might want to also reproduce the mkdir manually, I have no clue how jsch mkdir() handles it | 08:32 |
| *** fhubik_afk is now known as fhubik_lunch | 08:33 | |
| openstackgerrit | Weidong Shao proposed openstack-infra/project-config: Add compass to OpenStack projects https://review.openstack.org/196916 | 08:33 |
| jhesketh | hashar: jenkins is logging in okay (Accepted publickey for jenkins from 162.242.149.68 port 35600 ssh2) | 08:35 |
| openstackgerrit | Serg Melikyan proposed openstack/requirements: Exclude yaql 0.3.0 from requirements https://review.openstack.org/196977 | 08:35 |
| *** xarses has quit IRC | 08:36 | |
| *** xarses has joined #openstack-infra | 08:36 | |
| *** xarses has quit IRC | 08:37 | |
| hashar | jhesketh: time to try a mkdir ? :D | 08:37 |
| *** sergmelikyan has quit IRC | 08:37 | |
| *** xarses has joined #openstack-infra | 08:37 | |
| *** Kennan has quit IRC | 08:38 | |
| *** xarses has quit IRC | 08:38 | |
| *** Kennan has joined #openstack-infra | 08:38 | |
| jhesketh | hashar: great idea, and it fails | 08:38 |
| jhesketh | the filessytem is read-only | 08:38 |
| *** alex_xu_ is now known as alex_xu | 08:38 | |
| jhesketh | and I had just discovered http://paste.openstack.org/show/327759/ | 08:38 |
| *** xarses has joined #openstack-infra | 08:39 | |
| *** xarses has quit IRC | 08:39 | |
| *** bauzas has joined #openstack-infra | 08:39 | |
| *** yamada-h has joined #openstack-infra | 08:39 | |
| *** HeOS has joined #openstack-infra | 08:40 | |
| hashar | ohhh | 08:40 |
| *** xarses has joined #openstack-infra | 08:40 | |
| lifeless | jhesketh: yay | 08:40 |
| *** xarses has quit IRC | 08:40 | |
| jhesketh | so it's got errors=remount-ro | 08:41 |
| hashar | well caught! | 08:41 |
| jhesketh | so I could do a fsck, but I'm wondering if I should wait for other roots to help | 08:41 |
| *** xarses has joined #openstack-infra | 08:41 | |
| jhesketh | in case things get worse | 08:41 |
| jhesketh | hashar: can't beleive I didn't check dmesg sooner :-( | 08:41 |
| lifeless | :) | 08:41 |
| *** xarses has quit IRC | 08:41 | |
| *** dguitarbite has joined #openstack-infra | 08:42 | |
| hashar | when I worked for a network operating center, the first checks we would do were something like: | 08:42 |
| hashar | 1) is the power cord plugged in | 08:42 |
| hashar | 2) are there any lights on the equipment | 08:42 |
| *** xarses has joined #openstack-infra | 08:42 | |
| hashar | 3) are the network cables / optical fiber properly plugged in :D | 08:42 |
| *** mase_x200 has quit IRC | 08:42 | |
| hashar | but often I would forgot about those basic checks and waste time debugging something else | 08:43 |
| lifeless | checklists | 08:43 |
| lifeless | checklists are good | 08:43 |
| hashar | yeah | 08:43 |
| hashar | and once you have checklists, you can automatize / monitor them :-D | 08:43 |
| AJaeger | ttx, lifeless, jhesketh: What about sending a IRC notification about the problems to all channels? | 08:43 |
| bauzas | AJaeger: +100 | 08:43 |
| lifeless | I think thats a great idea | 08:43 |
| jhesketh | AJaeger: heh, I was just about to, I'm trying to find the command | 08:43 |
| ttx | AJaeger: +1 want me to do it ? | 08:43 |
| bauzas | is this something blocker for merging ? | 08:43 |
| AJaeger | ttx, teach jhesketh how to do it, plesae | 08:43 |
| *** mrunge has joined #openstack-infra | 08:43 | |
| *** yamada-h has quit IRC | 08:44 | |
| ttx | jhesketh: http://docs.openstack.org/infra/system-config/irc.html#statusbot | 08:44 |
| jhesketh | oh it's the openstackstatus user right? | 08:44 |
| ttx | but you need to be on the whitelist, not sur if you are | 08:44 |
| hashar | so in the ideal world you end up with a notification stating: "Disk dead on static.openstack.org || Automatically filled bug #1234" | 08:44 |
| openstack | bug 1234 in Launchpad itself "Gina is an unmaintainable mess of command line options, environment variables and shell scripts" [Medium,Fix released] https://launchpad.net/bugs/1234 - Assigned to Daniel Henrique Debonzi (debonzi) | 08:44 |
| jhesketh | ttx: yep, was trying to find the user | 08:44 |
| jhesketh | ttx: lets find out | 08:44 |
| ttx | worst case I'll send it | 08:44 |
| AJaeger | jhesketh: you're not - see project-config/accessbot/channels.yaml | 08:45 |
| jhesketh | AJaeger: hmm, boo, will fix that later | 08:45 |
| *** sdake_ has quit IRC | 08:45 | |
| ttx | jhesketh: let me know what the message is and I'll parrot it | 08:45 |
| jhesketh | ttx: please send something along the lines of #status alert OpenStack CI is down due to hard drive failures. | 08:45 |
| lifeless | we need a checklist for 'adding root access steps' | 08:45 |
| AJaeger | ttx, I think we can remove maffulli from the list, correct? | 08:45 |
| ttx | #status alert OpenStack CI is down due to hard drive failures | 08:45 |
| openstackstatus | ttx: sending alert | 08:46 |
| ttx | AJaeger: probably yes | 08:46 |
| AJaeger | jhesketh: I'll create a patch to add you | 08:46 |
| jhesketh | ttx: thanks | 08:46 |
| jhesketh | AJaeger: thanks :-) | 08:46 |
| AJaeger | Ah, lifeless has also the powers to do it ;) | 08:46 |
| ttx | to be fair, the only reason I'm on that list was so that we can send alerts in European mornings. If we have good coverage of infra-core people there now, I can probably also be removed from list | 08:47 |
| jhesketh | ttx: it's still helpful.. lifeless and I aren't always around for example | 08:47 |
| AJaeger | ttx: please stay on it, I'm not sure we have that great coverage yet | 08:47 |
| ttx | I'm fine with MoarPower[tm] | 08:47 |
| *** yamamoto has joined #openstack-infra | 08:47 | |
| hashar | it is probably better to have many people able to status log :D | 08:47 |
| *** ddieterly has joined #openstack-infra | 08:47 | |
| -openstackstatus- NOTICE: OpenStack CI is down due to hard drive failures | 08:48 | |
| *** ChanServ changes topic to "OpenStack CI is down due to hard drive failures" | 08:48 | |
| hashar | and tweet! https://twitter.com/wikimediatech :-D | 08:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add jhesketh to accessbot, remove maffulli. https://review.openstack.org/196983 | 08:49 |
| AJaeger | Should we send out an email as well? | 08:49 |
| openstackstatus | ttx: finished sending alert | 08:50 |
| lifeless | AJaeger: oh, I didn't realise. I shall next time. | 08:50 |
| jhesketh | lifeless, ttx, AJaeger: do you think I should fire off a fsck shall we wait for more infra to wake up? | 08:52 |
| *** ddieterly has quit IRC | 08:52 | |
| bauzas | AJaeger: you could at least provide a status on http://lists.openstack.org/pipermail/openstack-dev/2015-June/068308.html | 08:52 |
| bauzas | ? | 08:52 |
| bauzas | (sorry, missed the question mark) | 08:52 |
| lifeless | jhesketh: you're the root | 08:52 |
| lifeless | jhesketh: I'd start by checking for rackspace status allerts etc | 08:53 |
| openstackgerrit | Steve Kowalik proposed openstack-infra/system-config: Pass through zanata client parameters https://review.openstack.org/176440 | 08:53 |
| hashar | assuming it is a VM, that might be a disk failure on the host system :/ | 08:53 |
| ttx | aren't we supposed to shoot the cattle in the head ? | 08:53 |
| AJaeger | jhesketh: your call, I have too few experience with infra root to guide you in any direction | 08:53 |
| * ttx doesn't know enough of the specific situation to provide good advice | 08:54 | |
| *** dkehn has quit IRC | 08:55 | |
| lifeless | jhesketh: also check the cinder? status of the drive | 08:55 |
| *** bradjones has quit IRC | 08:55 | |
| AJaeger | jhesketh: let me write a short email. Which server has the harddisk failure? | 08:55 |
| jhesketh | hashar: they are cinder mounts afaik | 08:55 |
| jhesketh | AJaeger: static.openstack.org (which is the log server etc) | 08:56 |
| lifeless | AJaeger: the static volume where all the job logs go | 08:56 |
| AJaeger | ok, will write a short email... | 08:56 |
| *** dkehn has joined #openstack-infra | 08:56 | |
| *** kushal has joined #openstack-infra | 08:57 | |
| *** bradjones has joined #openstack-infra | 08:57 | |
| *** bradjones has quit IRC | 08:57 | |
| *** bradjones has joined #openstack-infra | 08:57 | |
| lifeless | jhesketh: OOI did you rollback the images? | 08:59 |
| *** _nadya_ has quit IRC | 09:00 | |
| *** sergmelikyan has joined #openstack-infra | 09:00 | |
| *** _nadya_ has joined #openstack-infra | 09:00 | |
| jhesketh | lifeless: yes I did | 09:01 |
| *** koolhead17 has joined #openstack-infra | 09:02 | |
| *** e0ne has joined #openstack-infra | 09:02 | |
| *** _nadya_ has quit IRC | 09:04 | |
| jhesketh | lifeless: hmm, I don't think I have access to the cloud creds yet to check the cinder status :-( | 09:05 |
| jhesketh | I might just hold off for more roots to turn up | 09:05 |
| *** mpavlase has joined #openstack-infra | 09:05 | |
| bauzas | AJaeger: thanks :) | 09:06 |
| AJaeger | jhesketh: fungi is normally around in 3 to 4 hours from now on... | 09:06 |
| SergeyLukjanov | jhesketh, hey, just arrived to office | 09:06 |
| jhesketh | SergeyLukjanov: Hey | 09:07 |
| SergeyLukjanov | ihrachyshka_, AJaeger sorry folks, just arrived to office | 09:07 |
| jhesketh | SergeyLukjanov: static.o.o has failing drives | 09:07 |
| SergeyLukjanov | jhesketh, oh... | 09:07 |
| jhesketh | need to decide what to do | 09:07 |
| jhesketh | most likely a fsck | 09:07 |
| jhesketh | but I don't have access to the cloud to see what cinder or anything says | 09:07 |
| AJaeger | SergeyLukjanov: ihrachyshka_ is happy now, now worries. | 09:08 |
| SergeyLukjanov | jhesketh, I have, will check | 09:08 |
| *** HeOS has quit IRC | 09:09 | |
| *** HeOS has joined #openstack-infra | 09:09 | |
| SergeyLukjanov | jhesketh, all drives are green in rackspace cloud ui | 09:10 |
| SergeyLukjanov | jhesketh, which drive are failing | 09:11 |
| SergeyLukjanov | ? | 09:11 |
| hashar | xvdm / dm-0 | 09:11 |
| hashar | seems the kernel can't ensure the writes / journal / cache is properly written to disk. | 09:11 |
| jhesketh | dm-1 is also failing | 09:11 |
| hashar | might be an issue with the underlying host disk | 09:11 |
| hashar | (but don't quote me :D ) | 09:11 |
| *** _nadya__ has joined #openstack-infra | 09:12 | |
| *** kushal has quit IRC | 09:13 | |
| *** dguerri` is now known as dguerri | 09:14 | |
| *** e0ne is now known as e0ne_ | 09:15 | |
| *** Ala has joined #openstack-infra | 09:16 | |
| lifeless | jhesketh: thats another checklist item I guess, but I thought the api keys were also in hiera | 09:18 |
| openstackgerrit | Qiming Teng proposed openstack-infra/system-config: Add #senlin to the list of eavesdrop channels https://review.openstack.org/196993 | 09:19 |
| SergeyLukjanov | jhesketh, yup, most likely fsck | 09:21 |
| ihrachyshka_ | AJaeger, SergeyLukjanov, I am indeed, thanks! (except that CI is broken) | 09:21 |
| *** soren has quit IRC | 09:22 | |
| *** evrardjp has joined #openstack-infra | 09:22 | |
| jhesketh | SergeyLukjanov: righto, shall we do that now? | 09:24 |
| SergeyLukjanov | jhesketh, I have no experience with it... have you? | 09:24 |
| rakhmerov | hi, what does UNSTABLE mean for a gate? | 09:25 |
| rakhmerov | e.g. https://jenkins04.openstack.org/job/gate-mistral-python27/58/console | 09:25 |
| jhesketh | SergeyLukjanov: not on an LVM | 09:25 |
| lifeless | rakhmerov: infra has a major outage | 09:25 |
| lifeless | rakhmerov: see topic | 09:25 |
| rakhmerov | ok | 09:25 |
| lifeless | rakhmerov: see also the dev list | 09:25 |
| rakhmerov | sure | 09:25 |
| *** e0ne_ has quit IRC | 09:26 | |
| *** Ala has quit IRC | 09:28 | |
| *** fhubik_lunch is now known as fhubik_afk | 09:28 | |
| *** e0ne has joined #openstack-infra | 09:29 | |
| *** kushal has joined #openstack-infra | 09:30 | |
| *** mattt has joined #openstack-infra | 09:33 | |
| *** markvoelker has joined #openstack-infra | 09:34 | |
| *** fhubik_afk is now known as fhubik_lunch | 09:36 | |
| *** john-davidge has quit IRC | 09:36 | |
| jhesketh | SergeyLukjanov: there is no raid setup is there... what backups (if any) do we have? | 09:39 |
| *** john-davidge has joined #openstack-infra | 09:39 | |
| *** markvoelker has quit IRC | 09:39 | |
| SergeyLukjanov | jhesketh, probably we have backups for some parts of its data in bup | 09:40 |
| jhesketh | SergeyLukjanov: bup doesn't appear to be on static.o.o | 09:41 |
| Kennan | hi lifeless: | 09:43 |
| Kennan | do you know why jenkins jobs failured with "Unstable"? | 09:43 |
| SergeyLukjanov | jhesketh, so, it sounds like we have no backups at all for it | 09:43 |
| Kennan | I found gerrit marked with Unstable | 09:43 |
| Kennan | gate-magnum-python27 UNSTABLE in 2m 46s | 09:43 |
| *** soren has joined #openstack-infra | 09:43 | |
| Kennan | what does that mean? | 09:43 |
| AJaeger | Kennan: see mailing list and topic. | 09:44 |
| ekarlso | I thought the cloud couldn't get taken down by harddrive failures :D | 09:44 |
| jhesketh | SergeyLukjanov: do you want to take a snapshot of the instance then? (If you think that's a good idea) | 09:45 |
| ekarlso | or stuff running it too :D | 09:45 |
| AJaeger | Kennan: http://lists.openstack.org/pipermail/openstack-dev/2015-June/068311.html | 09:45 |
| jhesketh | and then the next question is, do you think it's worth snapshotting the cinder devices, or will that take too long | 09:45 |
| SergeyLukjanov | jhesketh, of the whole instance or of the volumes? | 09:45 |
| SergeyLukjanov | jhesketh, no idea - I've never used rackspace cloud | 09:46 |
| Kennan | Thanks Ajaeger: | 09:46 |
| Kennan | AJaeger: | 09:46 |
| jhesketh | SergeyLukjanov: snapshotting the instance should only capture the vm driver surely? | 09:46 |
| SergeyLukjanov | jhesketh, yup, it'll not touch the attached drives | 09:46 |
| jhesketh | we should start with that then, just in case | 09:46 |
| jhesketh | SergeyLukjanov: then we should consider if we want to snapshot the attached drives | 09:47 |
| *** ddieterly has joined #openstack-infra | 09:48 | |
| SergeyLukjanov | jhesketh, there is only an option to create image from server - I think it's the VM snapshot | 09:50 |
| SergeyLukjanov | jhesketh, now trying to do it | 09:50 |
| *** ociuhandu has quit IRC | 09:52 | |
| jhesketh | SergeyLukjanov: yeah that's the one | 09:52 |
| *** yfried is now known as yfried|afk | 09:52 | |
| SergeyLukjanov | jhesketh, in progress | 09:52 |
| jhesketh | SergeyLukjanov: what do you think about snapshoting the volumes, or should we just run the fsck next? | 09:52 |
| *** ddieterly has quit IRC | 09:53 | |
| SergeyLukjanov | jhesketh, AFAIK it could occasionally cause data corruption | 09:55 |
| jhesketh | SergeyLukjanov: snapshotting? or fsck? | 09:55 |
| SergeyLukjanov | jhesketh, I mean fsck | 09:55 |
| jhesketh | SergeyLukjanov: should have checked this sooner, but https://status.rackspace.com/index/viewincidents?group=11 | 09:55 |
| spredzy | To core could I get a review on this patchset please : https://review.openstack.org/#/c/175849/ | 09:56 |
| SergeyLukjanov | jhesketh, shit, we missed this important notice | 09:56 |
| jhesketh | SergeyLukjanov: it would explain the corruption | 09:57 |
| *** otter768 has joined #openstack-infra | 09:59 | |
| jhesketh | SergeyLukjanov: the incident also hasn't been fully resolved, so anything we do now may be futile | 09:59 |
| SergeyLukjanov | jhesketh, agreed, so, we should wait while it'll be marked as solved | 10:00 |
| jhesketh | okay, roger that | 10:00 |
| *** yfried|afk is now known as yfried | 10:00 | |
| AJaeger | spredzy: Let'S first fix our CI infrastructure, see the topic... | 10:01 |
| spredzy | AJaeger, sorry I haven't read it first. My bad. | 10:02 |
| AJaeger | SergeyLukjanov: to me it reads: Just remount the volume and we should be fine - but it might hit us again ;/ | 10:03 |
| *** otter768 has quit IRC | 10:04 | |
| openstackgerrit | Serg Melikyan proposed openstack/requirements: Exclude yaql 0.3.0 from requirements https://review.openstack.org/196977 | 10:05 |
| *** Hal has joined #openstack-infra | 10:07 | |
| *** Guest16792 has quit IRC | 10:08 | |
| *** Hal is now known as Guest21021 | 10:08 | |
| *** soren_ has joined #openstack-infra | 10:08 | |
| *** soren has quit IRC | 10:08 | |
| jhesketh | AJaeger: there's also a good chance we got some corruption during the incident that'll need repairing | 10:09 |
| *** asahlin_away has quit IRC | 10:09 | |
| AJaeger | jhesketh: ;( | 10:10 |
| *** sergmelikyan has quit IRC | 10:10 | |
| jhesketh | AJaeger: given that we write a lot of logs, it's likely just a job or two, which really isn't too bad | 10:10 |
| jhesketh | a recheck etc | 10:10 |
| *** asahlin has joined #openstack-infra | 10:11 | |
| AJaeger | jhesketh: indeed | 10:11 |
| *** sergmelikyan has joined #openstack-infra | 10:11 | |
| *** fhubik_lunch is now known as fhubik_afk | 10:13 | |
| *** coolsvap is now known as coolsvap|away | 10:14 | |
| *** Qiming has quit IRC | 10:15 | |
| *** fhubik_afk is now known as fhubik_lunch | 10:17 | |
| *** sergmelikyan has quit IRC | 10:17 | |
| *** koolhead17 has quit IRC | 10:22 | |
| *** e0ne is now known as e0ne_ | 10:23 | |
| *** e0ne_ is now known as e0ne | 10:25 | |
| *** mpaolino_ has joined #openstack-infra | 10:27 | |
| *** soren_ has quit IRC | 10:27 | |
| *** jlanoux_ has joined #openstack-infra | 10:29 | |
| *** fhubik_lunch is now known as fhubik_afk | 10:29 | |
| *** jlanoux has quit IRC | 10:31 | |
| *** fhubik_afk is now known as fhubik_lunch | 10:33 | |
| *** soren has joined #openstack-infra | 10:33 | |
| *** soren has quit IRC | 10:33 | |
| *** soren has joined #openstack-infra | 10:33 | |
| *** rossella_s has quit IRC | 10:36 | |
| *** rossella_s has joined #openstack-infra | 10:37 | |
| *** sergmelikyan has joined #openstack-infra | 10:37 | |
| *** soren has quit IRC | 10:37 | |
| *** romcheg has joined #openstack-infra | 10:38 | |
| romcheg | Hi folks | 10:38 |
| romcheg | I've just made a release 6.1.1 of stackforge/python-fuelclient but looks like it didn't get uploaded to PyPi | 10:39 |
| romcheg | AJaeger, could you please take a look? | 10:39 |
| romcheg | I'm not sure who else is here at this time | 10:39 |
| jhesketh | romcheg: it won't be uploaded until our static server disks are fixed | 10:40 |
| jhesketh | which is currently due to an outage | 10:40 |
| AJaeger | romcheg: didn't you see the topic of our channels? | 10:40 |
| *** claudiub has quit IRC | 10:40 | |
| romcheg | jhesketh: thanks for the update! My fault, I haven't looked at the topic | 10:41 |
| romcheg | jhesketh: Do I have to do anything after hard drives are fixed in order to get it to pypi? | 10:41 |
| AJaeger | romcheg: ask one of the infra roots to push it again through our tag queue... | 10:42 |
| AJaeger | romcheg: once you get the all clear, bring some beer and ask ;) | 10:43 |
| jhesketh | yeah we'll have to retrigger the post job | 10:43 |
| *** shardy_ has joined #openstack-infra | 10:43 | |
| *** shardy has quit IRC | 10:45 | |
| romcheg | Thanks folks! | 10:45 |
| *** sergmelikyan has quit IRC | 10:47 | |
| *** shardy_ has quit IRC | 10:48 | |
| *** ddieterly has joined #openstack-infra | 10:49 | |
| *** shardy has joined #openstack-infra | 10:49 | |
| *** chlong has joined #openstack-infra | 10:49 | |
| *** ociuhandu has joined #openstack-infra | 10:50 | |
| *** yamamoto has quit IRC | 10:50 | |
| *** pc_m has joined #openstack-infra | 10:50 | |
| *** BharatK has quit IRC | 10:51 | |
| *** jlanoux has joined #openstack-infra | 10:51 | |
| *** fhubik_lunch is now known as fhubik_afk | 10:52 | |
| *** soren has joined #openstack-infra | 10:53 | |
| *** ddieterly has quit IRC | 10:54 | |
| *** jlanoux_ has quit IRC | 10:55 | |
| *** hichihara has joined #openstack-infra | 10:58 | |
| *** kushal has quit IRC | 10:59 | |
| *** BharatK has joined #openstack-infra | 11:02 | |
| *** dims has joined #openstack-infra | 11:07 | |
| openstackgerrit | Szymon Wróblewski proposed openstack/requirements: tooz 0.16.0 or better required https://review.openstack.org/197021 | 11:07 |
| *** EricGonczer_ has joined #openstack-infra | 11:10 | |
| *** larainema has joined #openstack-infra | 11:13 | |
| *** sergmelikyan has joined #openstack-infra | 11:14 | |
| *** sshnaidm has quit IRC | 11:14 | |
| *** Qiming has joined #openstack-infra | 11:15 | |
| *** hashar has quit IRC | 11:16 | |
| *** ParsectiX has quit IRC | 11:18 | |
| *** ParsectiX has joined #openstack-infra | 11:19 | |
| *** weshay has joined #openstack-infra | 11:21 | |
| *** mpaolino_ has quit IRC | 11:23 | |
| *** mpaolino has quit IRC | 11:23 | |
| *** EricGonczer_ has quit IRC | 11:23 | |
| *** sshnaidm has joined #openstack-infra | 11:23 | |
| *** markvoelker has joined #openstack-infra | 11:24 | |
| *** thorst has joined #openstack-infra | 11:27 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Allow for git base override https://review.openstack.org/151034 | 11:29 |
| *** markvoelker has quit IRC | 11:29 | |
| *** KanagarajM has quit IRC | 11:29 | |
| *** sshnaidm has quit IRC | 11:29 | |
| *** sshnaidm has joined #openstack-infra | 11:30 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: post_test_hook is not executed when the gate test returns an error https://review.openstack.org/148863 | 11:31 |
| markus_z | Hi, I'm developing a script which should help me automatize cleanups in Launchpad. I'm wondering to which project I could push a patch set to get some feedback? | 11:32 |
| *** EricGonczer_ has joined #openstack-infra | 11:34 | |
| *** Ala has joined #openstack-infra | 11:35 | |
| Kiall | markus_z: probably https://github.com/openstack-infra/release-tools | 11:36 |
| *** radez is now known as radez_g0n3 | 11:36 | |
| Kiall | (there's a pile of other LP related scripts used for the release process in there) | 11:36 |
| *** ildikov has quit IRC | 11:38 | |
| markus_z | Kiall: Yes, that looks good, I'll try it there, thank you! | 11:38 |
| *** EricGonczer_ has quit IRC | 11:39 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: [smarcet] https://review.openstack.org/196898 | 11:39 |
| *** dprince has joined #openstack-infra | 11:40 | |
| *** baoli has joined #openstack-infra | 11:40 | |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| *** hichihara has quit IRC | 11:47 | |
| *** jschlueter has joined #openstack-infra | 11:47 | |
| jschlueter | status | 11:48 |
| *** michchap has quit IRC | 11:48 | |
| *** michchap has joined #openstack-infra | 11:49 | |
| thorst | Hi. Can someone take a look at this change: https://review.openstack.org/#/c/196641/ | 11:49 |
| thorst | its blocking some change sets in other components as we add support for that hypervisor | 11:49 |
| *** ddieterly has joined #openstack-infra | 11:50 | |
| *** fawadkhaliq has quit IRC | 11:51 | |
| *** ildikov has joined #openstack-infra | 11:51 | |
| *** _nadya__ has quit IRC | 11:52 | |
| *** EricGonczer_ has joined #openstack-infra | 11:53 | |
| *** _nadya_ has joined #openstack-infra | 11:53 | |
| *** ddieterly has quit IRC | 11:54 | |
| *** mwagner_lap has joined #openstack-infra | 11:55 | |
| *** EricGonc_ has joined #openstack-infra | 11:57 | |
| *** _nadya_ has quit IRC | 11:57 | |
| *** radez_g0n3 is now known as radez | 11:58 | |
| *** _nadya_ has joined #openstack-infra | 11:58 | |
| *** EricGonczer_ has quit IRC | 11:58 | |
| *** fhubik_afk is now known as fhubik_lunch | 11:58 | |
| *** jistr is now known as jistr|class | 11:59 | |
| *** fhubik_lunch is now known as fhubik_afk | 11:59 | |
| *** yfried is now known as yfried|afk | 11:59 | |
| *** EricGonczer_ has joined #openstack-infra | 12:00 | |
| *** otter768 has joined #openstack-infra | 12:00 | |
| *** hichihara has joined #openstack-infra | 12:00 | |
| *** EricGonc_ has quit IRC | 12:01 | |
| *** prad has joined #openstack-infra | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:02 | |
| *** AzherKhan has quit IRC | 12:03 | |
| *** sergmelikyan has quit IRC | 12:04 | |
| *** marcusvrn has joined #openstack-infra | 12:05 | |
| *** otter768 has quit IRC | 12:05 | |
| *** jistr|class is now known as jistr | 12:07 | |
| *** jyuso1 has quit IRC | 12:07 | |
| *** maishsk has quit IRC | 12:09 | |
| *** claudiub has joined #openstack-infra | 12:10 | |
| *** bcourt has joined #openstack-infra | 12:10 | |
| *** markvoelker_ has joined #openstack-infra | 12:11 | |
| *** markvoel_ has joined #openstack-infra | 12:13 | |
| *** markvoelker has quit IRC | 12:13 | |
| *** sslypushenko has joined #openstack-infra | 12:13 | |
| *** maishsk_afk has joined #openstack-infra | 12:14 | |
| *** maishsk_afk is now known as maishsk | 12:14 | |
| *** michchap has quit IRC | 12:15 | |
| *** hashar has joined #openstack-infra | 12:15 | |
| *** michchap has joined #openstack-infra | 12:17 | |
| *** markvoelker_ has quit IRC | 12:17 | |
| *** yfried|afk is now known as yfried | 12:19 | |
| *** rkukura has quit IRC | 12:26 | |
| *** zz_dimtruck is now known as dimtruck | 12:27 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/releasestatus: Loop to retrieve all reviews https://review.openstack.org/197042 | 12:29 |
| fungi | wow--fun! hard drive failures? | 12:30 |
| haypo | fungi: yes. http://lists.openstack.org/pipermail/openstack-dev/2015-June/068311.html | 12:30 |
| *** aysyd has joined #openstack-infra | 12:32 | |
| fungi | thanks for the details. i'll start digging into it now | 12:32 |
| fungi | looks like /dev/xvdm is unhappy | 12:33 |
| *** _nadya_ has quit IRC | 12:34 | |
| *** maishsk has quit IRC | 12:36 | |
| fungi | okay, i've got all processes with open filehandles in that mountpoint killed and have umounted | 12:37 |
| *** maishsk_afk has joined #openstack-infra | 12:37 | |
| *** maishsk_afk is now known as maishsk | 12:37 | |
| *** hichihara has quit IRC | 12:38 | |
| *** mmedvede has joined #openstack-infra | 12:38 | |
| *** _nadya_ has joined #openstack-infra | 12:38 | |
| *** michchap_ has joined #openstack-infra | 12:38 | |
| *** gordc_afk is now known as gordc | 12:38 | |
| fungi | getting the rest of the volumes from that vg unmounted now so i can deactivate it | 12:38 |
| *** sergmelikyan has joined #openstack-infra | 12:39 | |
| *** yamamoto has joined #openstack-infra | 12:39 | |
| *** hichihara has joined #openstack-infra | 12:40 | |
| *** michchap has quit IRC | 12:40 | |
| fungi | the main vg is now deactivated | 12:40 |
| *** _nadya_ has quit IRC | 12:42 | |
| *** _nadya_ has joined #openstack-infra | 12:42 | |
| *** kgiusti has joined #openstack-infra | 12:42 | |
| *** shardy has quit IRC | 12:43 | |
| *** _nadya_ has joined #openstack-infra | 12:43 | |
| *** shardy has joined #openstack-infra | 12:43 | |
| fungi | rebooting the server now, and will detach/reattach all cinder volumes momentarily | 12:43 |
| *** nagyz has joined #openstack-infra | 12:44 | |
| *** mpaolino_ has joined #openstack-infra | 12:45 | |
| *** EricGonczer_ has quit IRC | 12:45 | |
| AJaeger | thanks, fungi! | 12:45 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add propose-backports jobs to openstack/puppet-* https://review.openstack.org/175850 | 12:45 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag https://review.openstack.org/175849 | 12:45 |
| *** mpaolino has joined #openstack-infra | 12:45 | |
| *** mpavone has quit IRC | 12:46 | |
| AJaeger | fungi, once everything is ok again: Please give a status update on the openstack-dev mailing list where I wrote this one: http://lists.openstack.org/pipermail/openstack-dev/2015-June/068311.html and also update the IRC status. I'll be offline for the next few hours therefore writing this now... | 12:46 |
| fungi | will do | 12:46 |
| *** bknudson has joined #openstack-infra | 12:47 | |
| *** ayoung has quit IRC | 12:48 | |
| *** maishsk has quit IRC | 12:48 | |
| fungi | oh, actually i found the cinder volume corresponding to /dev/xvdm so i'm just detaching/attaching that one for now | 12:48 |
| jhesketh | fungi: oh hey, thanks for that. Let me know if I can help | 12:48 |
| *** _nadya_ has quit IRC | 12:48 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add propose-backports jobs to openstack/puppet-* https://review.openstack.org/175850 | 12:48 |
| *** _nadya_ has joined #openstack-infra | 12:49 | |
| *** _nadya_ has quit IRC | 12:49 | |
| *** _nadya_ has joined #openstack-infra | 12:49 | |
| *** scheuran has quit IRC | 12:49 | |
| *** rfolco has joined #openstack-infra | 12:50 | |
| *** ddieterly has joined #openstack-infra | 12:51 | |
| *** markvoel_ has quit IRC | 12:51 | |
| *** _diana_ has joined #openstack-infra | 12:51 | |
| *** asadoughi has joined #openstack-infra | 12:51 | |
| fungi | running filesystem checks on the volumes in that vg now | 12:51 |
| *** Guest27041 has joined #openstack-infra | 12:51 | |
| *** markvoelker_ has joined #openstack-infra | 12:52 | |
| *** _nadya_ has quit IRC | 12:52 | |
| *** jsavak has joined #openstack-infra | 12:52 | |
| fungi | luckily all were clean except the logs volume | 12:52 |
| fungi | though as that one's >12tb it may take a few minutes to complete its fsck :/ | 12:53 |
| *** yamamoto has quit IRC | 12:53 | |
| jhesketh | Cool :-) | 12:53 |
| *** maishsk_afk has joined #openstack-infra | 12:53 | |
| jhesketh | fungi: the incident hasn't been resolved fully yet so we may get more problems | 12:53 |
| *** maishsk_afk is now known as maishsk | 12:53 | |
| fungi | was there an acknowledged incident in dfw? | 12:53 |
| *** bknudson has quit IRC | 12:54 | |
| jhesketh | https://status.rackspace.com | 12:54 |
| *** _nadya_ has joined #openstack-infra | 12:54 | |
| fungi | i just assumed this was the same "your compute node lost contact with the storage backend where that volume resided" problem we'd seen in the past | 12:54 |
| jhesketh | fungi: yeah the cinder control nodes are playing up afaik | 12:54 |
| AJaeger | fungi, https://status.rackspace.com/index/viewincidents?group=11 | 12:55 |
| fungi | oh, i see the cloud block storage ! triangles now | 12:55 |
| fungi | i skimmed earlier just looking at the alerts and maintenance block at the top | 12:55 |
| *** ddieterly has quit IRC | 12:55 | |
| *** rlandy has joined #openstack-infra | 12:56 | |
| fungi | also anybody know about the trove maintenance they've got planned tomorrow for dfw? they're supposedly rebuilding all our database instances, but i'm worried that means they'll end up with new dns names since they seem to be randomly-generated hex values to start with | 12:58 |
| jhesketh | No idea sorry | 12:59 |
| fungi | i'll ask fanatically in the support ticket they opened for me | 13:00 |
| mrmartin | fungi: are we using that for groups and openstackid? | 13:00 |
| *** puranamr has joined #openstack-infra | 13:00 | |
| *** sebrandon1 has joined #openstack-infra | 13:00 | |
| fungi | mrmartin: yes. according to the support ticket it'll be a ~5 minute outage | 13:01 |
| mrmartin | ok. | 13:01 |
| mrmartin | do we have proper backups ? | 13:01 |
| fungi | presumably impacting the www site too | 13:01 |
| *** markvoelker_ has quit IRC | 13:02 | |
| *** sergmelikyan has quit IRC | 13:02 | |
| *** tlian has joined #openstack-infra | 13:02 | |
| *** markvoelker has joined #openstack-infra | 13:02 | |
| *** sergmelikyan has joined #openstack-infra | 13:03 | |
| *** markvoelker_ has joined #openstack-infra | 13:03 | |
| *** ddieterly has joined #openstack-infra | 13:03 | |
| fungi | worth checking. i see a 0-byte /var/backups/mysql_backups/groups.sql.gz on groups.o.o from 06:36 utc today | 13:04 |
| fungi | need to look into that once i've got static.o.o back on track | 13:04 |
| *** mase_x200 has joined #openstack-infra | 13:04 | |
| mrmartin | fungi: it is not looking nice | 13:05 |
| *** deepakcs has quit IRC | 13:05 | |
| *** markvoelker_ has quit IRC | 13:05 | |
| mrmartin | something should be wrong with those backup scripts | 13:05 |
| fungi | openstackid.org doesn't seem to be configured to do database backups, but i'll get that in and working today | 13:05 |
| fungi | mrmartin: yeah, i expect it's misconfigured | 13:06 |
| *** markvoelker_ has joined #openstack-infra | 13:06 | |
| fungi | i'll find out | 13:06 |
| *** ddieterly has quit IRC | 13:06 | |
| *** changbl has quit IRC | 13:06 | |
| fungi | i've got a few minutes while this fsck continues anyway | 13:06 |
| *** markvoelker has quit IRC | 13:07 | |
| fungi | looks like groups is relying on /root/.groups_db.cnf for the backups and its got the wrong details. i'll fix that | 13:08 |
| *** sergmelikyan has quit IRC | 13:08 | |
| *** miqui has joined #openstack-infra | 13:09 | |
| *** mrunge has quit IRC | 13:10 | |
| *** bknudson has joined #openstack-infra | 13:10 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies https://review.openstack.org/197060 | 13:10 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies https://review.openstack.org/197060 | 13:12 |
| *** peristeri has joined #openstack-infra | 13:12 | |
| *** _nadya_ has quit IRC | 13:12 | |
| *** jsavak has quit IRC | 13:13 | |
| *** jsavak has joined #openstack-infra | 13:13 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies https://review.openstack.org/197060 | 13:14 |
| markus_z | stupid spaces ... | 13:14 |
| *** sergmelikyan has joined #openstack-infra | 13:14 | |
| *** puranamr has quit IRC | 13:14 | |
| *** sergmelikyan has quit IRC | 13:14 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Add a non-voting py34 check job to os-brick https://review.openstack.org/197064 | 13:18 |
| openstackgerrit | Wayne Warren proposed openstack-infra/infra-specs: Zuul github-integration spec. https://review.openstack.org/163637 | 13:18 |
| *** julim has joined #openstack-infra | 13:19 | |
| *** kushal has joined #openstack-infra | 13:20 | |
| *** sergmelikyan has joined #openstack-infra | 13:20 | |
| *** EricGonczer_ has joined #openstack-infra | 13:21 | |
| *** Guest27041 is now known as agentle | 13:21 | |
| *** EricGonc_ has joined #openstack-infra | 13:22 | |
| dims | fungi: looks like some reviews are in gate queue and green | 13:22 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies https://review.openstack.org/197060 | 13:23 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag https://review.openstack.org/175849 | 13:24 |
| fungi | dims: yeah, log uploads are spooling to the parent volume. i'm not expecting to try and find a way to preserve them | 13:24 |
| fungi | the fsck has made it to phase 2 now though, so shouldn't be much longer. then i'll blow away what's accumulated out from under the mountpoint before i bring it back online | 13:25 |
| *** _nadya_ has joined #openstack-infra | 13:25 | |
| fungi | and then one more reboot of the server to make sure everything comes back up normally | 13:25 |
| *** EricGonczer_ has quit IRC | 13:25 | |
| *** blaedd has joined #openstack-infra | 13:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Make Magnum functional tests voting https://review.openstack.org/191921 | 13:26 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add propose-backports jobs to openstack/puppet-* https://review.openstack.org/175850 | 13:27 |
| *** nelsnelson has joined #openstack-infra | 13:27 | |
| openstackgerrit | Merged openstack-infra/project-config: oslo.messaging: remove gate_hook https://review.openstack.org/191773 | 13:28 |
| dims | thanks fungi | 13:28 |
| *** e0ne is now known as e0ne_ | 13:29 | |
| *** alivigni has joined #openstack-infra | 13:29 | |
| *** homeless has joined #openstack-infra | 13:30 | |
| *** boris-42 has joined #openstack-infra | 13:31 | |
| fungi | mrmartin: actually, groups db backups are fine. i was looking in the wrong spot. it has 30 days of full mysqldumps of all the remote databases | 13:31 |
| fungi | mrmartin: i also just ran a fresh one to make sure it's working correctly | 13:31 |
| mrmartin | ok. | 13:32 |
| *** cody-somerville has joined #openstack-infra | 13:32 | |
| *** cody-somerville has quit IRC | 13:32 | |
| *** cody-somerville has joined #openstack-infra | 13:32 | |
| mrmartin | so we can survive the trove maintenance tomorrow | 13:32 |
| fungi | i'll get the mysql-backups module added to openstackid.org now for good measure | 13:32 |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: Fix Jenkins nodes methods to use POST https://review.openstack.org/193600 | 13:32 |
| openstackgerrit | Merged openstack-infra/project-config: Add pre/post test hooks for Heat grenade job https://review.openstack.org/188147 | 13:32 |
| *** doug-fish has quit IRC | 13:33 | |
| *** blaedd has quit IRC | 13:34 | |
| *** mattymo has joined #openstack-infra | 13:35 | |
| mattymo | fungi, are you around? | 13:36 |
| *** soren has quit IRC | 13:36 | |
| kashyap | Hi folks, I see that "check-grenade-dsvm" job failed for one of the changes I submitted. However, clicking on the said failed job throws "File Not Found" - http://logs.openstack.org/97/193497/2/check/check-grenade-dsvm/500d054/ | 13:36 |
| kashyap | Anyone know why? | 13:36 |
| jroll | kashyap: see topic :) | 13:36 |
| fungi | mattymo: yeah, i'm here | 13:36 |
| kashyap | jroll: Darn, sorry. | 13:36 |
| jroll | np | 13:36 |
| kashyap | jroll: I _did_ notice it this morning in one of the channels, but missed to remember it. | 13:37 |
| jroll | hah | 13:37 |
| mattymo | fungi, I have a really curious gerrit dilemma. I would like to attribute a patchset to an email address from another company | 13:37 |
| mattymo | for example, I am developing a patch for company XYZ and john.smith@xyzcompany.com is a registered openstack foundation email address | 13:37 |
| mattymo | how can I make such an attribution? | 13:37 |
| fungi | mattymo: that's more of a git dilemma | 13:37 |
| mattymo | I can tell git config anything... but I believe gerrit will reject fake author that doesn't match my ssh key | 13:38 |
| fungi | mattymo: gerrit only cares about the committer address matching your account, and doesn't really care at all what the author name/address on the patch is | 13:38 |
| *** mriedem has joined #openstack-infra | 13:39 | |
| *** e0ne_ has quit IRC | 13:39 | |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Add non-voting Sahara grenade job for upgrades testing https://review.openstack.org/197076 | 13:39 |
| fungi | mattymo: so all you need to do is reset the author on the commit (if it's incorrect, though if you brought the patch into git via git am it should be preserved), then commit --amend to update the committer to be you | 13:39 |
| fungi | in fact, git am may already do that part too | 13:39 |
| *** blaedd has joined #openstack-infra | 13:39 | |
| fungi | assuming you received it in git format-patch output format or similar enough for git am to import it correctly | 13:40 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:40 | |
| mattymo | fungi, and let's say I have the code already on review and I want to modify it. Just commit --amend --author=john.smith@xyzcompany.com is good enough? | 13:40 |
| fungi | mattymo: should be. then look at git log --pretty=fuller to see if the author and committer are the way you want (them, you) | 13:41 |
| *** tvanderwiel has joined #openstack-infra | 13:41 | |
| *** ZZelle has quit IRC | 13:41 | |
| mattymo | fungi, thanks for your help! | 13:42 |
| *** zz_jgrimm is now known as jgrimm | 13:42 | |
| *** ZZelle has joined #openstack-infra | 13:42 | |
| fungi | mattymo: it's worth noting that the gerrit change "owner" will still be the committer of the first patchset for a change | 13:42 |
| *** BharatK has quit IRC | 13:42 | |
| fungi | since committer is what gerrit actually cares about (to support projects which may receive outside contributions but only allow certain committers to upload them for review) | 13:43 |
| *** agentle has quit IRC | 13:43 | |
| *** ayoung has joined #openstack-infra | 13:43 | |
| mattymo | fungi, which value matters to stackalytics then? | 13:43 |
| mattymo | commiter or author? | 13:43 |
| fungi | mattymo: i don't know. you'd have to ask the people who wrote stackalytics or look at its source code | 13:44 |
| *** sebrandon1 has quit IRC | 13:44 | |
| fungi | mattymo: if your reason for attributing that patchset to the other person is to get them "active technical contributor status" then the only effective solution is to abandon your change and get them to upload it themselves | 13:45 |
| *** VivCheri has joined #openstack-infra | 13:45 | |
| fungi | mattymo: or else submit a patch to the governance repo adding them to the "extra-atcs" file and convince the tc to approve it | 13:45 |
| mattymo | I don't think it's for ATC (it's a stackforge commit anyway), but instead for some sort of copyright attribution of the commit | 13:46 |
| *** e0ne has joined #openstack-infra | 13:46 | |
| fungi | mattymo: if it's just for copyright attribution, simply making sure it's accompanied by a line in the file(s) being changed with the additional copyright holder information should be sufficient, but updating the author field correctly is still useful for reasons of provenance | 13:47 |
| jeblair | fungi: can you join #openstack-infra-incident ? | 13:47 |
| fungi | mattymo: and in that case you don't need to care about gerrit change owner | 13:47 |
| fungi | jeblair: oh, yep! | 13:47 |
| *** doug-fish has joined #openstack-infra | 13:49 | |
| *** soren has joined #openstack-infra | 13:51 | |
| *** sshnaidm has quit IRC | 13:52 | |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Add non-voting Sahara grenade job for upgrades testing https://review.openstack.org/197076 | 13:53 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add database backups for openstackid.org https://review.openstack.org/197083 | 13:53 |
| fungi | mrmartin: ^ | 13:53 |
| fungi | i'm going to test that out with a puppet dev env in just a sec | 13:53 |
| *** jsavak has quit IRC | 13:53 | |
| mrmartin | awesome, don't we have backups for openstackid? | 13:53 |
| *** sshnaidm has joined #openstack-infra | 13:53 | |
| *** pblaho has quit IRC | 13:53 | |
| fungi | does it have local files which aren't in configuration management? | 13:54 |
| *** fhubik_afk is now known as fhubik_lunch | 13:54 | |
| mrmartin | under app/storage, | 13:54 |
| *** BharatK has joined #openstack-infra | 13:55 | |
| *** wenlock has joined #openstack-infra | 13:55 | |
| mrmartin | but it is just for cache, so db backup is enough there | 13:55 |
| *** eharney has joined #openstack-infra | 13:55 | |
| fungi | yeah, generally we only add filesystem backups if there's something on the local filesystem which is irreplaceable. database dumps are a grey area since they theoretically are a copy of data which is in the database server instance too, but backing them up to a separate server in a different region/provider via bup is still potentially useful | 13:56 |
| *** jsavak has joined #openstack-infra | 13:57 | |
| mrmartin | fungi: just a little update required for https://review.openstack.org/197083 | 13:57 |
| *** fawadkhaliq has joined #openstack-infra | 13:57 | |
| fungi | mrmartin: oh, that's a good idea | 13:58 |
| *** yamahata has joined #openstack-infra | 13:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Add beaker-localhost and beaker-nodepool hypervisors https://review.openstack.org/185754 | 13:58 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add database backups for openstackid.org https://review.openstack.org/197083 | 13:59 |
| *** sergmelikyan has quit IRC | 13:59 | |
| *** baoli has quit IRC | 13:59 | |
| yfried | where do I submit a bug against requirements? | 13:59 |
| *** sergmelikyan has joined #openstack-infra | 13:59 | |
| *** baoli has joined #openstack-infra | 14:00 | |
| *** dims has quit IRC | 14:01 | |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| *** otter768 has joined #openstack-infra | 14:01 | |
| *** dims has joined #openstack-infra | 14:01 | |
| *** marun has joined #openstack-infra | 14:01 | |
| *** kiran-r has quit IRC | 14:02 | |
| *** dboik has joined #openstack-infra | 14:02 | |
| fungi | i've got the server rebooted again and the logs volume back in place following a filesystem check. if nobody spots any new problems with it i'll stand down the alert and follow up to the ml shortly | 14:05 |
| *** otter768 has quit IRC | 14:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Make tempest-dsvm-cells job voting https://review.openstack.org/190894 | 14:06 |
| mattymo | fungi, last q.. how can I check a given email address's company affiliation in the openstack foundation? | 14:07 |
| *** jsavak has quit IRC | 14:07 | |
| openstackgerrit | Merged openstack-infra/project-config: Making integration tests gate voting https://review.openstack.org/194324 | 14:07 |
| *** _nadya_ has quit IRC | 14:07 | |
| fungi | mattymo: http://www.openstack.org/community/members/ | 14:08 |
| fungi | yfried: i don't think there is a bug tracker for the requirements repo | 14:09 |
| *** scheuran has joined #openstack-infra | 14:09 | |
| fungi | yfried: it didn't really contain software of note until the past few weeks | 14:09 |
| yfried | fungi: https://bugs.launchpad.net/openstack-gate/+bug/1462421 | 14:09 |
| openstack | Launchpad bug 1462421 in OpenStack-Gate "jsonschema 2.5.0 pulling in functools32 which breaks py26 and py34 jobs" [Undecided,Confirmed] | 14:09 |
| fungi | yfried: that's exciting | 14:09 |
| yfried | fungi: https://bugs.launchpad.net/rally/+bug/1466055 | 14:10 |
| openstack | Launchpad bug 1466055 in Rally "functools32 should be added to requirements" [Undecided,New] | 14:10 |
| *** fhubik_lunch is now known as fhubik | 14:10 | |
| *** jsavak has joined #openstack-infra | 14:10 | |
| yfried | fungi: seems like a bug in jsonschema latest | 14:10 |
| yfried | fungi: I wonder if other project are hitting this | 14:10 |
| mriedem | yfried: i think that was a known thing a couple of weeks ago | 14:10 |
| *** sshnaidm has quit IRC | 14:11 | |
| fungi | yfried: in that case i'd suggest filing a bug against jsonschema not openstack requirements | 14:11 |
| mriedem | yfried: it was already fixed upstream from what i remember | 14:11 |
| mriedem | was some issue with wheels | 14:11 |
| yfried | mriedem: not in rally | 14:11 |
| *** mtanino has joined #openstack-infra | 14:11 | |
| yfried | fungi: I agree, but until it's fixed, our code breaks... | 14:11 |
| *** pblaho has joined #openstack-infra | 14:11 | |
| mriedem | it should be fixed already | 14:11 |
| mriedem | i'm looking for the upstream issue | 14:12 |
| *** sshnaidm has joined #openstack-infra | 14:12 | |
| mriedem | yfried: https://github.com/Julian/jsonschema/issues/228 | 14:12 |
| *** ihrachyshka_ has quit IRC | 14:13 | |
| *** ParsectiX has quit IRC | 14:13 | |
| *** ihrachyshka has joined #openstack-infra | 14:14 | |
| fungi | mrmartin: i've used a test of 197083 to get current backup dumps of both databases on openstackid.org just now. looks right (5mib compressed for the id db, 50mib compressed for the ss db). took only a few seconds to complete | 14:14 |
| yfried | mriedem: looks like it solved the 1st bug (py26 and py34) but introduced the 2nd | 14:14 |
| *** annegentle has joined #openstack-infra | 14:14 | |
| mrmartin | fungi: thank you, so we are prepared now | 14:14 |
| mriedem | yfried: open an issue against jsonschema upstream | 14:15 |
| *** puranamr has joined #openstack-infra | 14:15 | |
| fungi | mrmartin: yep! and thanks for the reminder to check those servers | 14:15 |
| yfried | mriedem: ack | 14:15 |
| yfried | mriedem: but how to proceed until it's fixed? | 14:15 |
| fungi | yfried: you could introduce blacklist entries for the affected releases in global requirements | 14:16 |
| fungi | assuming they truly are broken | 14:16 |
| mriedem | yfried: https://github.com/Julian/jsonschema/issues/234 | 14:16 |
| mriedem | it's already reported | 14:16 |
| *** nihilifer has joined #openstack-infra | 14:17 | |
| *** _nadya_ has joined #openstack-infra | 14:17 | |
| mriedem | yfried: use newer pip | 14:17 |
| mriedem | is the workaround | 14:17 |
| mattymo | fungi, it doesn't take email addresses | 14:18 |
| fungi | mattymo: you may need to figure out the person's name | 14:18 |
| mattymo | I know the name | 14:18 |
| *** baoli has quit IRC | 14:18 | |
| yfried | mriedem: ok. so I must update pip for this, so - not a bug | 14:19 |
| yfried | mriedem: tnx | 14:19 |
| *** dbite has joined #openstack-infra | 14:19 | |
| fungi | mattymo: i've got a wishlist feature request for a more robust member querying api. it would be extremely useful for things like generating electoral rolls too | 14:19 |
| *** puranamr has quit IRC | 14:19 | |
| yfried | mriedem: I guess the 2 bugs can be closed? | 14:20 |
| *** r-daneel has joined #openstack-infra | 14:20 | |
| mriedem | yfried: i already did :) | 14:20 |
| *** unicell has joined #openstack-infra | 14:21 | |
| yfried | mriedem: I wanted to do that :( | 14:22 |
| *** baoli has joined #openstack-infra | 14:22 | |
| yfried | mriedem: [17:21] <stpierre> that's odd, global requirements specifies pip>=6.0 | 14:22 |
| *** unicell1 has quit IRC | 14:22 | |
| *** cindyo has joined #openstack-infra | 14:23 | |
| *** _diana_ has quit IRC | 14:23 | |
| *** rkukura has joined #openstack-infra | 14:23 | |
| *** tonytan4ever has joined #openstack-infra | 14:23 | |
| *** shardy_ has joined #openstack-infra | 14:24 | |
| AJaeger | fungi: Please remember to scroll back later to around 6:10 UTC, the translation proposal jobs all failed, I gave some comments on them. | 14:24 |
| fungi | AJaeger: thanks, i'll have a look after my next meeting | 14:24 |
| thorst | sdague: Would you mind taking a peak at https://review.openstack.org/#/c/196641/ ? | 14:24 |
| *** shardy has quit IRC | 14:24 | |
| thorst | It has an updated requirement for the PowerVM API which enables the hypervisor driver support we're working on. | 14:24 |
| *** fhubik is now known as fhubik_afk | 14:25 | |
| openstackgerrit | Merged openstack/os-testr: Disable printing percent change on run time by default https://review.openstack.org/194865 | 14:29 |
| *** shardy_ has quit IRC | 14:29 | |
| *** amitgandhinz has joined #openstack-infra | 14:30 | |
| *** mpavlase has quit IRC | 14:30 | |
| *** shardy has joined #openstack-infra | 14:30 | |
| *** baoli has quit IRC | 14:30 | |
| *** sshnaidm has quit IRC | 14:31 | |
| *** baoli has joined #openstack-infra | 14:31 | |
| *** mase_x200 has quit IRC | 14:31 | |
| *** sshnaidm has joined #openstack-infra | 14:31 | |
| *** koolhead17 has joined #openstack-infra | 14:31 | |
| haypo | mriedem: thanks for https://review.openstack.org/#/c/195191/ | 14:32 |
| mriedem | haypo: np | 14:33 |
| *** baoli has quit IRC | 14:33 | |
| *** baoli has joined #openstack-infra | 14:33 | |
| *** baoli has quit IRC | 14:33 | |
| mriedem | thorst: is that py34 compliant? | 14:33 |
| thorst | mriedem: I believe so, but I'm going to verify right now...just a sec | 14:34 |
| *** baoli has joined #openstack-infra | 14:34 | |
| mriedem | -1 | 14:34 |
| mriedem | it's not listed https://github.com/pypowervm/pypowervm/blob/master/setup.cfg#L18 | 14:34 |
| *** bcourt has quit IRC | 14:34 | |
| thorst | good pt. We'll get that updated now | 14:34 |
| haypo | mriedem: python 3.3 & 3.4 are very close | 14:34 |
| haypo | mriedem: it's pure python, i don't expect major issue with 3.4 | 14:35 |
| mriedem | thems the rules | 14:35 |
| *** nikil89_ has joined #openstack-infra | 14:35 | |
| thorst | I'll get it updated and complaint. Should have another revision on it soon. | 14:36 |
| *** hdd has joined #openstack-infra | 14:36 | |
| thorst | haypo: could I add you as a reviewer once I get that updated? | 14:36 |
| haypo | thorst: yes | 14:36 |
| *** BharatK has quit IRC | 14:36 | |
| *** rossella_s has quit IRC | 14:36 | |
| thorst | thx! Appreciate the feedback | 14:37 |
| *** rossella_s has joined #openstack-infra | 14:37 | |
| *** pblaho has quit IRC | 14:39 | |
| *** nikil89_ has quit IRC | 14:39 | |
| *** pblaho has joined #openstack-infra | 14:39 | |
| *** sshnaidm has quit IRC | 14:41 | |
| *** jsavak has quit IRC | 14:41 | |
| *** sshnaidm has joined #openstack-infra | 14:41 | |
| mriedem | thorst: i see there is a 0.0.1 version on pypi but no tags or releases in github? https://github.com/pypowervm/pypowervm/releases | 14:42 |
| mriedem | https://github.com/pypowervm/pypowervm/tags | 14:42 |
| thorst | mreidem: yeah, we'll branch and do a 0.0.2 version before the next proposal, get it 3.4 doc'd, etc... | 14:43 |
| *** jsavak has joined #openstack-infra | 14:44 | |
| *** fallenpegasus has joined #openstack-infra | 14:46 | |
| *** sergmelikyan has quit IRC | 14:47 | |
| *** jsavak has quit IRC | 14:47 | |
| *** sergmelikyan has joined #openstack-infra | 14:49 | |
| *** sergmelikyan has quit IRC | 14:49 | |
| asselin_ | fungi, good morning. Shall we try to merge those common-ci zuul patches today? | 14:50 |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Add a non-voting py34 check job to os-brick https://review.openstack.org/197064 | 14:51 |
| mriedem | thorst: s/branch/tag/ | 14:51 |
| fungi | asselin_: today is shaping up to be a bad day for our systems (well, one of our service providers) breaking. it's also the day most of us have all our meetings | 14:52 |
| *** doude has quit IRC | 14:52 | |
| *** laughterwym has joined #openstack-infra | 14:52 | |
| *** jsavak has joined #openstack-infra | 14:53 | |
| thorst | mriedem: yep, tag. | 14:53 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 14:53 | |
| -openstackstatus- NOTICE: The log volume was repaired and brought back online at 14:00 UTC. Log links today from before that time may be missing, and changes should be rechecked if fresh job logs are desired for them. | 14:53 | |
| yfried | mriedem: looks like latest pip doesn't fix the issue | 14:54 |
| mriedem | yfried: i'd comment on the upstream jsonschema issue then with the details of the issue you're seeing | 14:55 |
| *** e0ne is now known as e0ne_ | 14:55 | |
| *** EricGonc_ has quit IRC | 14:55 | |
| *** shardy has quit IRC | 14:56 | |
| *** shardy has joined #openstack-infra | 14:56 | |
| *** k4n0 has quit IRC | 14:57 | |
| *** dangers_away is now known as dangers | 14:58 | |
| *** jsavak has quit IRC | 14:58 | |
| *** fawadkhaliq has quit IRC | 14:58 | |
| *** jaypipes has quit IRC | 14:58 | |
| *** e0ne_ is now known as e0ne | 14:58 | |
| *** sergmelikyan has joined #openstack-infra | 14:58 | |
| *** jsavak has joined #openstack-infra | 14:58 | |
| yfried | mriedem: could it that pip version comes after jsonschema version? | 14:59 |
| *** e0ne is now known as e0ne_ | 14:59 | |
| *** doude has joined #openstack-infra | 14:59 | |
| *** e0ne_ is now known as e0ne | 14:59 | |
| *** maishsk has quit IRC | 15:00 | |
| yfried | mriedem: if so, this means that pip should be the first global-req | 15:00 |
| frickler | is graphite.openstack.org also affected by the rackspace thingy? | 15:01 |
| frickler | also image builds seem to have mostly failed today because of bad timing for creating openstack-infra/beaker-localhost | 15:01 |
| AJaeger | frickler: yes, fungi is looking at graphite right now | 15:02 |
| *** jsavak has quit IRC | 15:03 | |
| *** sambetts has quit IRC | 15:03 | |
| *** cindyo has quit IRC | 15:04 | |
| sc68cal | so uh, I was downloading some logs from logs.openstack.org via wget early this morning | 15:04 |
| sc68cal | hopefully I didn't break anything | 15:04 |
| *** fawadkhaliq has joined #openstack-infra | 15:04 | |
| fungi | frickler: our graphite server was hung completely. when i force restarted it, it came back up without working ipv4 through its default gateway. i have a ticket open with rackspace support to get that taken care of | 15:05 |
| fungi | sc68cal: nope, the breakage was completely internal to rackspace. some major issue with their block storage backend | 15:05 |
| asselin_ | fungi, understood | 15:05 |
| fungi | sc68cal: your downloads definitely did not cause it | 15:05 |
| *** _diana_ has joined #openstack-infra | 15:05 | |
| sc68cal | fungi: ok good :) | 15:05 |
| *** sergmelikyan has quit IRC | 15:06 | |
| frickler | sc68cal: except you used this hacker tool called IPv6 maybe ... :D | 15:06 |
| sc68cal | haha :) | 15:07 |
| *** sambetts has joined #openstack-infra | 15:07 | |
| jhesketh | okay I'm off | 15:07 |
| jhesketh | see you at the meeting, night! | 15:07 |
| *** asselin__ has joined #openstack-infra | 15:08 | |
| *** _nadya_ has quit IRC | 15:09 | |
| *** _nadya_ has joined #openstack-infra | 15:10 | |
| *** scheuran has quit IRC | 15:11 | |
| *** asselin_ has quit IRC | 15:12 | |
| *** shardy has quit IRC | 15:12 | |
| *** ihrachyshka has quit IRC | 15:12 | |
| *** hashar has quit IRC | 15:13 | |
| *** xyang1 has joined #openstack-infra | 15:13 | |
| *** _nadya_ has quit IRC | 15:14 | |
| *** jasondotstar has joined #openstack-infra | 15:15 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: [smarcet] https://review.openstack.org/196898 | 15:15 |
| yfried | mriedem: http://paste.openstack.org/show/328808/ | 15:15 |
| yfried | mriedem: search for "@@" | 15:15 |
| *** kiran-r has joined #openstack-infra | 15:16 | |
| yfried | mriedem: looks like pip is updated before jsonschema is installed, and yet - jsonschema is installed without functools32 | 15:16 |
| yfried | mriedem: however, when I install jsonschema again, it works as expected. | 15:16 |
| *** bcourt has joined #openstack-infra | 15:17 | |
| *** Qiming has quit IRC | 15:17 | |
| yfried | mriedem: seems like an issue with pip were it's still using the old version in the same operations | 15:17 |
| *** wenlock_ has joined #openstack-infra | 15:17 | |
| *** shardy has joined #openstack-infra | 15:18 | |
| fungi | yfried: chances are this is related to virtualenvs... since virtualenv vendors a copy of pip you get whatever version of pip was concurrent with the virtualenv version used (not the version of pip installed on your system, much to the surprise of many), unless you explicitly upgrade pip inside that virtualenv afterward | 15:19 |
| yfried | fungi: that's not the issue | 15:19 |
| yfried | fungi: specifying pip version in requirements update pip | 15:20 |
| fungi | yfried: not necessarily soon enough though | 15:20 |
| *** austin81 has joined #openstack-infra | 15:20 | |
| yfried | fungi: but it uses the same old pip to install all other packages in that file | 15:20 |
| fungi | yfried: if you pip install some list which includes a new version constraint on pip, then you get all packages in that list installed by the old version | 15:20 |
| *** mikeit has joined #openstack-infra | 15:20 | |
| yfried | fungi: exactly | 15:21 |
| fungi | yfried: this is why we never bothered to include pip in our requirements lists in the past | 15:21 |
| yfried | fungi: ack | 15:21 |
| fungi | you need a separate bootstrapping step to upgrade pip before you use it to install other things | 15:21 |
| yfried | fungi: so, I'm still at loss for the bug | 15:21 |
| fungi | yfried: i haven't looked at the bug, sorry. plenty of issues with our systems today, so i'll leave issues with tests to people for whom that's more in their sphere of interest | 15:22 |
| *** _diana_ has quit IRC | 15:22 | |
| * fungi is pretty much 100% in systems administrator mode at the moment | 15:22 | |
| yfried | fungi: well, that's not an issue of tests. this will hit everyone who installs using an old pip | 15:23 |
| *** rlandy has quit IRC | 15:23 | |
| fungi | yfried: yep, software | 15:23 |
| fungi | today for me is dealing with broken filesystems and networks, sorry | 15:23 |
| *** rlandy has joined #openstack-infra | 15:23 | |
| *** cindyo has joined #openstack-infra | 15:23 | |
| fungi | oh, also databases and backups | 15:24 |
| greghaynes | wow, looks like there was a fun morning | 15:24 |
| yfried | fungi: tnx anyway | 15:24 |
| *** haypo has left #openstack-infra | 15:24 | |
| *** _diana_ has joined #openstack-infra | 15:25 | |
| *** yamahata has quit IRC | 15:26 | |
| fungi | hrm, and troubleshooting e-mail deliveries | 15:26 |
| fungi | all things sysadmin | 15:26 |
| openstackgerrit | Merged openstack-infra/project-config: Handle missing pot files in translation proposal https://review.openstack.org/193784 | 15:26 |
| *** laughterwym has quit IRC | 15:27 | |
| *** armax has joined #openstack-infra | 15:27 | |
| *** erikmwilson_ has joined #openstack-infra | 15:28 | |
| greghaynes | thats exciting to see huge explosions fixed before west coast wakes :) | 15:28 |
| *** Ala has quit IRC | 15:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix gate_hook in rally-cue experimental job https://review.openstack.org/195991 | 15:28 |
| *** erikmwilson_ is now known as erikwilson_ | 15:28 | |
| fungi | yeah, exciting isn't the word i would have used | 15:29 |
| greghaynes | hah | 15:29 |
| fungi | seems most of the explosions were well before the east coast of north america woke up for that matter | 15:29 |
| pabelanger | fungi, jeblair: Be curious your thoughts about adding the EPEL repos into the centos6 host to get access to new binaries. In this case, git. | 15:29 |
| fungi | pabelanger: is there newer git in epel? i thought we already had epel activated on those servers | 15:30 |
| *** hodos has joined #openstack-infra | 15:30 | |
| *** bradjones has quit IRC | 15:30 | |
| pabelanger | fungi, that is what I am checking out, the version shipped with EPEL | 15:30 |
| *** changbl has joined #openstack-infra | 15:31 | |
| *** alivigni has quit IRC | 15:31 | |
| *** homeless has quit IRC | 15:31 | |
| fungi | pabelanger: 1.7.1-3.el6_4.1 is in base according to yum list. it's not spotting one available in centos 6 epel | 15:31 |
| *** homeless has joined #openstack-infra | 15:32 | |
| *** bradjones has joined #openstack-infra | 15:32 | |
| *** bradjones has quit IRC | 15:32 | |
| *** bradjones has joined #openstack-infra | 15:32 | |
| fungi | though i'll readily admit my yum-fu is limited | 15:32 |
| openstackgerrit | Merged openstack-infra/project-config: Surveil: Enable pypy jobs https://review.openstack.org/196247 | 15:33 |
| pabelanger | fungi, ya, looks like 1.7.1.3 is the latest :( | 15:33 |
| pabelanger | oh well | 15:33 |
| pabelanger | trying to figure out ways to remove the manual compile | 15:33 |
| pabelanger | will have to do that until we get our packages going | 15:33 |
| pabelanger | I did what nibalizer suggested and did a JJB job to upgrade git inline | 15:34 |
| *** puranamr has joined #openstack-infra | 15:34 | |
| *** marun has quit IRC | 15:34 | |
| *** kiran-r has quit IRC | 15:35 | |
| *** _diana_ has quit IRC | 15:35 | |
| fungi | AJaeger: okay, i'm taking a look at the translations issues now that my first meeting has wrapped up | 15:35 |
| *** _diana_ has joined #openstack-infra | 15:35 | |
| *** puranamr_ has joined #openstack-infra | 15:36 | |
| *** puranamr has quit IRC | 15:36 | |
| *** DericHorn-HP has joined #openstack-infra | 15:36 | |
| *** enikanorov2_ has quit IRC | 15:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add coverage to Manila check to aid reviewers https://review.openstack.org/196283 | 15:36 |
| openstackgerrit | Merged openstack-infra/project-config: Fetch all heads and tags when caching source repos https://review.openstack.org/195650 | 15:37 |
| openstackgerrit | Merged openstack-infra/project-config: Set up jobs to test new ubuntu-trusty nodes https://review.openstack.org/196382 | 15:37 |
| *** abregman has quit IRC | 15:37 | |
| *** sshnaidm has quit IRC | 15:37 | |
| openstackgerrit | Merged openstack-infra/nodepool: Make diskimage example config work https://review.openstack.org/190538 | 15:37 |
| *** sshnaidm has joined #openstack-infra | 15:38 | |
| *** sabeen1 has quit IRC | 15:38 | |
| *** enikanorov2 has joined #openstack-infra | 15:38 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: add 3.8 release tests support https://review.openstack.org/189890 | 15:40 |
| pabelanger | fungi, IUS is rolling git 2.4.3: https://iuscommunity.org/pages/About.html | 15:40 |
| openstackgerrit | Merged openstack-infra/project-config: run-docs.sh: execute docs job if present https://review.openstack.org/191828 | 15:40 |
| fungi | AJaeger: clarkb: pleia2: i think there's a good chance that "fatal: ambiguous argument '(detached': unknown revision or path not in the working tree." is a symptom of a behavior change in the version of git in ubuntu trusty, or possibly with a newer git-review | 15:40 |
| pabelanger | fungi, using yum replace it upgraded properly on my host | 15:40 |
| openstackgerrit | Merged openstack-infra/project-config: Make keystone* bandit jobs voting https://review.openstack.org/196277 | 15:40 |
| zigo | What's the name of that git repo which is a template for new projects? | 15:40 |
| *** _nadya_ has joined #openstack-infra | 15:41 | |
| fungi | zigo: http://git.openstack.org/cgit/openstack-dev/cookiecutter/ | 15:41 |
| zigo | fungi: Thanks my friend! | 15:41 |
| pabelanger | fungi, and I believe you need to explicitly call yum replace to pull in packages from IUS | 15:41 |
| fungi | zigo: you're welcome! | 15:41 |
| pleia2 | fungi: aha, I assumed it was related to trusty upgrade but it's not clear to me why that's happening | 15:42 |
| fungi | pabelanger: okay, that's good to know. we should probably also be careful not to do that in such a way that it bleeds over into non-job-worker servers | 15:42 |
| pabelanger | fungi, Right. I should see how many centos6 jobs we have going right now. I don't imagine it being too much | 15:42 |
| fungi | pabelanger: since we know that latest git breaks things like shallow clones so we want to make sure we don't accidentally upgrade our centos 6 based git servers | 15:42 |
| pabelanger | fungi, I know puppet-apply job will be going away once the beaker jobs are all up and running. But another option would to just bring centos7 online and move forward with that | 15:43 |
| zigo | fungi: This is what is used for new Oslo libs too, right? | 15:44 |
| zigo | I don't think that's the one git repo that I'm looking for. | 15:45 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: update syntax-future job to test puppet ~> 4.2.0 https://review.openstack.org/196386 | 15:45 |
| EmilienM | AJaeger, SergeyLukjanov I had to rebase (conflict) can you +A again? | 15:45 |
| SergeyLukjanov | EmilienM, sure | 15:45 |
| fungi | zigo: oslo has their own flavor of it at http://git.openstack.org/cgit/openstack-dev/oslo-cookiecutter/ | 15:46 |
| openstackgerrit | Merged openstack-infra/grafyaml: Move yaml functions into its own class https://review.openstack.org/192431 | 15:46 |
| openstackgerrit | Merged openstack-infra/grafyaml: Re-run tox -egenconfig to fix oslo.config bug https://review.openstack.org/194169 | 15:46 |
| fungi | zigo: there are similar cookiecutter template repos for devstack plugins, puppet modules... | 15:46 |
| *** dtantsur is now known as dtantsur|afk | 15:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable python-jobs on designate-dashboard https://review.openstack.org/196678 | 15:47 |
| *** dizquierdo has quit IRC | 15:47 | |
| openstackgerrit | Merged openstack-infra/zuul: transpose typo'd characters https://review.openstack.org/192459 | 15:47 |
| *** geoffarnold has joined #openstack-infra | 15:48 | |
| openstackgerrit | Merged openstack-infra/zuul: Be more verbose on non-tagged versions https://review.openstack.org/188597 | 15:48 |
| *** kiran-r has joined #openstack-infra | 15:49 | |
| *** esmute__ has joined #openstack-infra | 15:49 | |
| *** _nadya_ has quit IRC | 15:49 | |
| *** abregman has joined #openstack-infra | 15:50 | |
| *** abitha has joined #openstack-infra | 15:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding new repo for security specs https://review.openstack.org/196165 | 15:51 |
| openstackgerrit | Thomas Goirand proposed openstack-dev/cookiecutter: Add insentive to write a long description https://review.openstack.org/197146 | 15:51 |
| *** kiran-r has quit IRC | 15:51 | |
| *** jasondotstar has quit IRC | 15:52 | |
| fungi | AJaeger: so... git-review thinking the local branch name is "(detached" happens when you checkout a specific ref rather than a refname (branch, et cetera). we could probably work around this by making sure we tell checkout to make a named branch with -B | 15:52 |
| *** stevemar has joined #openstack-infra | 15:53 | |
| stevemar | bump, new requirements request https://review.openstack.org/#/c/196339/ | 15:53 |
| *** mpavlase has joined #openstack-infra | 15:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Ensure we have py2.6 jobs enabled for cloud-init https://review.openstack.org/196884 | 15:53 |
| stevemar | but its a new oslo lib... so low impact | 15:54 |
| *** sdake has joined #openstack-infra | 15:54 | |
| fungi | AJaeger: or maybe just use git branch -M to force move the current head renaming it to a branch | 15:54 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/elastic-recheck: Add fingerprint for Linux Bridge hard reboot race https://review.openstack.org/197150 | 15:56 |
| *** achanda has joined #openstack-infra | 15:56 | |
| *** sdake_ has joined #openstack-infra | 15:56 | |
| sc68cal | anteaya: ^ | 15:58 |
| *** unicell has quit IRC | 15:58 | |
| *** asselin__ has quit IRC | 15:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Change Zanata project for django_openstack_auth https://review.openstack.org/196964 | 15:58 |
| pabelanger | fungi, jeblair: Do you think we'll add a bare-centos7 image? to go along side bare-precise and bare-trusty? | 15:59 |
| *** markus_z has quit IRC | 15:59 | |
| openstackgerrit | Merged openstack-infra/project-config: Correct project name for django_openstack_auth https://review.openstack.org/196962 | 16:00 |
| openstackgerrit | Merged openstack-infra/system-config: Change review->change in zuul table https://review.openstack.org/169934 | 16:00 |
| *** achanda has quit IRC | 16:00 | |
| fungi | pabelanger: we'll probably add a centos-7 diskimage to go along with the centos-6 and ubuntu-trusty diskimages | 16:00 |
| timrc | jeblair, Hi. You 'round? | 16:00 |
| *** sdake has quit IRC | 16:00 | |
| fungi | pabelanger: and use bindep to install distro packages in it on demand | 16:00 |
| *** jistr has quit IRC | 16:01 | |
| pabelanger | fungi, okay. Is the plan to retire the bare-precise and bare-trusty images? | 16:01 |
| fungi | pabelanger: yes | 16:01 |
| pabelanger | okay | 16:01 |
| fungi | we're trying to retire all the snapshot-based nodepool images | 16:01 |
| *** sergmelikyan has joined #openstack-infra | 16:01 | |
| *** _diana_ has quit IRC | 16:01 | |
| *** otter768 has joined #openstack-infra | 16:02 | |
| *** yfried has quit IRC | 16:02 | |
| *** kiran-r has joined #openstack-infra | 16:02 | |
| openstackgerrit | Merged openstack-infra/zuul: Status: Don't raise "abort" as error to the user https://review.openstack.org/172241 | 16:02 |
| *** acruz has quit IRC | 16:02 | |
| *** spredzy is now known as spredzy|afk | 16:04 | |
| *** spzala has joined #openstack-infra | 16:05 | |
| *** jlanoux has quit IRC | 16:05 | |
| *** shardy has quit IRC | 16:06 | |
| *** jasondotstar has joined #openstack-infra | 16:06 | |
| *** sarob has joined #openstack-infra | 16:06 | |
| openstackgerrit | Cody Littley proposed openstack-infra/nodepool: Fixed crash in exception handler https://review.openstack.org/195172 | 16:06 |
| *** otter768 has quit IRC | 16:07 | |
| pleia2 | SergeyLukjanov: thanks for the zanata +As, was about to add them to the meeting agenda as things we need to push through :) | 16:07 |
| *** shardy has joined #openstack-infra | 16:07 | |
| *** fhubik_afk is now known as fhubik | 16:07 | |
| *** mpavlase has quit IRC | 16:07 | |
| SergeyLukjanov | pleia2, :) | 16:07 |
| *** mpaolino_ has quit IRC | 16:11 | |
| *** dannywilson has joined #openstack-infra | 16:11 | |
| frickler | SergeyLukjanov: if you want https://review.openstack.org/194751, you will have to +A https://review.openstack.org/189142, too ;) | 16:11 |
| *** pelix has quit IRC | 16:12 | |
| *** fallenpegasus has quit IRC | 16:12 | |
| SergeyLukjanov | frickler, I've not approved https://review.openstack.org/#/c/189142/ due to the open question | 16:12 |
| SergeyLukjanov | frickler, do you think it's not important q. to wait for and it should done separately? | 16:13 |
| openstackgerrit | Merged openstack-infra/system-config: Adjust Zanata admins https://review.openstack.org/193677 | 16:13 |
| frickler | SergeyLukjanov: I think we can look at that later, yes | 16:13 |
| pabelanger | 713750 | 16:13 |
| *** dannywilson has quit IRC | 16:13 | |
| pabelanger | oops | 16:13 |
| frickler | SergeyLukjanov: next step would be https://review.openstack.org/189141, with that we would be ready to build images with vivid | 16:14 |
| *** dannywilson has joined #openstack-infra | 16:14 | |
| fungi | AJaeger: pleia2: were either of you working on the translation proposals fix? if not i'll start testing a possible solution | 16:14 |
| frickler | SergeyLukjanov: and then we can set up some jobs with that to see what happens in the "real world" with it | 16:14 |
| fungi | as soon as i eat my lunch | 16:14 |
| pleia2 | fungi: no, an I'm about to head out the door for dentist | 16:14 |
| openstackgerrit | Merged openstack-infra/system-config: Switch to openstackid.org for Zanata auth https://review.openstack.org/196735 | 16:15 |
| fungi | pleia2: ahh, righth. best of luck. they're like vampires | 16:15 |
| pleia2 | all sparkling teeth in infra this week | 16:15 |
| *** kiran-r has quit IRC | 16:15 | |
| pleia2 | thanks | 16:15 |
| jesusaurus | fungi: have you ever had anyone accidentally propose a stack of changes for a rebased branch? and do you know of a quick way to abandon several thousand changes? | 16:16 |
| *** weshay has quit IRC | 16:16 | |
| fungi | jesusaurus: a dirty script looping calls to the api is probably the easiest way | 16:17 |
| *** sshnaidm has quit IRC | 16:17 | |
| *** shardy_ has joined #openstack-infra | 16:17 | |
| jesusaurus | gotcha | 16:18 |
| *** shardy has quit IRC | 16:18 | |
| *** beekneemech is now known as bnemec | 16:19 | |
| *** weshay has joined #openstack-infra | 16:19 | |
| zigo | dims: Should we upgrade all the oslo and friends libs to the latest release for releasing liberty-1 ? | 16:19 |
| fungi | jesusaurus: oh! or latest gertty has batch actions you can apply to a gerrit query | 16:19 |
| fungi | jesusaurus: not sure whether it does abandon though | 16:20 |
| *** dizquierdo has joined #openstack-infra | 16:20 | |
| cody-somerville | fungi: how do you dequeue them from zuul though? | 16:20 |
| *** _diana_ has joined #openstack-infra | 16:21 | |
| *** sergmelikyan has quit IRC | 16:21 | |
| fungi | cody-somerville: dump the zuul queue to a local file, stop zuul, filter the file contents to remove the changes you don't want to test, then start zuul and only reenqueue the good changes | 16:22 |
| *** devvesa has quit IRC | 16:22 | |
| fungi | cody-somerville: or get https://review.openstack.org/95035 working | 16:22 |
| openstackgerrit | Fawad Khaliq proposed openstack-infra/project-config: Move networking-plumgrid into the openstack namespace https://review.openstack.org/197169 | 16:23 |
| *** EricGonczer_ has joined #openstack-infra | 16:23 | |
| *** shardy_ has quit IRC | 16:23 | |
| *** rlandy has quit IRC | 16:23 | |
| *** e0ne is now known as e0ne_ | 16:23 | |
| *** rlandy has joined #openstack-infra | 16:23 | |
| *** shardy has joined #openstack-infra | 16:24 | |
| *** notnownikki has joined #openstack-infra | 16:24 | |
| *** fhubik has quit IRC | 16:24 | |
| timrc | fungi, FWIW https://review.openstack.org/#/c/95035/ i technically working :) We even have a user! There is just a request for more exhaustive testing. | 16:25 |
| *** smarcet has joined #openstack-infra | 16:25 | |
| greghaynes | mordred: SpamapS I just noticed something wierd on http://paste.openstack.org/show/321629/ (traceback nodepool is giving us uploading to rax) - line 12 - why is a .get on the kwargs dict causing us call create_object? | 16:25 |
| openstackgerrit | Merged openstack-infra/system-config: Make puppet work with Ubuntu Vivid https://review.openstack.org/189142 | 16:25 |
| smarcet | fungi: hello there waiting on approval for https://review.openstack.org/#/c/196898/ | 16:26 |
| smarcet | fungi: should i add you as reviewer? | 16:26 |
| SpamapS | greghaynes: reading | 16:26 |
| *** jaypipes has joined #openstack-infra | 16:26 | |
| cody-somerville | fungi: How do you protect yourself from a DoS? | 16:26 |
| fungi | cody-somerville: we try not to get people mad at us? | 16:27 |
| greghaynes | SpamapS: oh, wait, disreguard me - im being silly (theres an extra ) on that line) | 16:27 |
| * greghaynes chugs more coffee | 16:27 | |
| SpamapS | greghaynes: yeah disregarding you :) | 16:27 |
| SpamapS | greghaynes: but confirming, thats the trace down into the call to create_object, not the get. :) | 16:28 |
| *** unicell has joined #openstack-infra | 16:28 | |
| *** _diana_ has quit IRC | 16:28 | |
| fungi | smarcet: i was just talking to jimmy about that. it looks like we never added anyone to the openstackid-resources-core group in gerrit. in theory it should have the same initial members as the openstackid-core group. jeblair: can you confirm if you're okay with that (or take care of updating it yourself)? | 16:28 |
| SpamapS | greghaynes: the error itself looks like maybe something swift needs isn't making it into the SwiftService | 16:28 |
| *** sergmelikyan has joined #openstack-infra | 16:28 | |
| jeblair | fungi: will do | 16:28 |
| openstackgerrit | Merged openstack-infra/system-config: Trim user specific package a bit https://review.openstack.org/194751 | 16:29 |
| *** dontalton has joined #openstack-infra | 16:29 | |
| *** samuelBartel has quit IRC | 16:29 | |
| smarcet | fungi: ok , and what about the new open id connect changes, this is the first one on that direction https://review.openstack.org/#/c/196898/ | 16:29 |
| greghaynes | SpamapS: Yea, I have a test case against rax but its not causing that traceback and having a hard time figuring out why im not hitting it - for some reason https://review.openstack.org/#/c/196936/ isnt calling swift.client.get_auth( | 16:29 |
| *** baoli has quit IRC | 16:29 | |
| smarcet | fungi: should i autoapprove that one? is more like a refactoring to start the work on oidc | 16:30 |
| jeblair | fungi, smarcet: done | 16:30 |
| *** doude has quit IRC | 16:30 | |
| smarcet | fungi: i am planning to start the whole oidc feature one one single commit ( is that ok?) | 16:30 |
| greghaynes | SpamapS: (with a clouds.yaml of rax cloud defined, obv) | 16:30 |
| smarcet | jeblair: thx :) | 16:30 |
| *** baoli_ has joined #openstack-infra | 16:31 | |
| fungi | smarcet: seems fine to me. it's your project so whatever you and anyone else actively reviewing changes on it (if there are any at present) would like to see | 16:33 |
| fungi | smarcet: and yeah, if you don't have anyone else reviewing changes on those projects yet, then approving the changes yourself is fine | 16:34 |
| smarcet | fungi : ok got it , thx :) | 16:34 |
| *** e0ne_ has quit IRC | 16:34 | |
| jeblair | when other folks take an interest, that's probably a better time to be more formal :) | 16:34 |
| *** kiran-r has joined #openstack-infra | 16:34 | |
| fungi | exactly. and until then, don't let a lack of interested reviewers in that repo hold you up from landing features/fixes it needs | 16:35 |
| smarcet | fungi: jeblair: sure understood | 16:35 |
| *** achanda has joined #openstack-infra | 16:35 | |
| *** jasondotstar has quit IRC | 16:36 | |
| *** e0ne has joined #openstack-infra | 16:36 | |
| *** jasondotstar has joined #openstack-infra | 16:37 | |
| pabelanger | timrc, This just happened in grafana: https://github.com/grafana/grafana/commit/ae0f8c77d16840bf4526cab68da54460b043db9b | 16:38 |
| pabelanger | they now allow username / password access to REST api | 16:38 |
| pabelanger | vs api_key | 16:38 |
| *** smarcet has quit IRC | 16:40 | |
| timrc | pabelanger, That's cool. | 16:41 |
| *** bhunter71 has joined #openstack-infra | 16:42 | |
| zaro | morning | 16:42 |
| pabelanger | timrc, Yup, we'll need to run 2.1.0, but the last issue for doing automated deployments now | 16:43 |
| *** acruz has joined #openstack-infra | 16:46 | |
| *** smarcet has joined #openstack-infra | 16:46 | |
| *** fallenpegasus has joined #openstack-infra | 16:49 | |
| *** openstackgerrit has quit IRC | 16:50 | |
| *** cindyo has quit IRC | 16:50 | |
| *** openstackgerrit has joined #openstack-infra | 16:51 | |
| *** ildikov has quit IRC | 16:53 | |
| * SpamapS struggles with api paste thingies | 16:53 | |
| *** hichihara has quit IRC | 16:54 | |
| openstackgerrit | Merged openstack-infra/system-config: Use vim-enhanced on centos https://review.openstack.org/196461 | 16:55 |
| *** romcheg has quit IRC | 16:56 | |
| *** fawadkhaliq has quit IRC | 16:57 | |
| *** marun has joined #openstack-infra | 16:57 | |
| *** dizquierdo has quit IRC | 16:57 | |
| *** geoffarnold has quit IRC | 16:58 | |
| *** geoffarnold has joined #openstack-infra | 16:58 | |
| *** abregman has quit IRC | 16:59 | |
| *** derekh has quit IRC | 16:59 | |
| *** jasondotstar has quit IRC | 16:59 | |
| *** ociuhandu has quit IRC | 17:00 | |
| *** kiran-r has quit IRC | 17:02 | |
| timrc | pabelanger, Do we get a default Org? | 17:02 |
| krotscheck | Is grenade still being touchy? I've got a build that was passing, but then failed on a recheck. | 17:02 |
| timrc | pabelanger, I didn't see anything 'bout that in the code reviews. | 17:02 |
| krotscheck | Something about failing to build an image. | 17:02 |
| pabelanger | timrc, Ya, I didn't make any changes to the OrgID in my reviews. So, we are defaulting I guess | 17:03 |
| *** jasondotstar has joined #openstack-infra | 17:03 | |
| krotscheck | http://logs.openstack.org/27/192327/21/check/check-grenade-dsvm/855e452/console.html#_2015-06-30_16_24_53_219 | 17:03 |
| pabelanger | timrc, what are you thinking. Creating one for openstack and stackforge? | 17:03 |
| greghaynes | fungi: Hey - is there a log line in the nodepool upload logs about "endpoint not found" around where we get the swift upload traceback? | 17:03 |
| *** kiran-r has joined #openstack-infra | 17:03 | |
| timrc | pabelanger, I dunno... Orgs in grafana are kind of idiotic IMHO. | 17:03 |
| *** Swami has joined #openstack-infra | 17:04 | |
| greghaynes | SpamapS: ^ I suspect get_session_endpoint is returning None in shade... that shouldnt happen right? | 17:04 |
| pabelanger | timrc, Ya, for the most part, I'm just using a single org right now | 17:04 |
| timrc | pabelanger, Unless things have changed only *one* org can be anonymous. | 17:04 |
| pabelanger | Oh, I didn't notice that | 17:04 |
| *** fawadkhaliq has joined #openstack-infra | 17:04 | |
| timrc | pabelanger, The way we have it setup in HP is to tag dashboards and then create dashboard lists they list other dashboards with a certain tag. | 17:04 |
| timrc | No one wants to log in to see pretty graphs. | 17:04 |
| pabelanger | timrc, That seems reasonable | 17:05 |
| pabelanger | timrc, agreed | 17:05 |
| fungi | greghaynes: no grep matches for "endpoint not found" (case insensitive) in our nodepool debug logs | 17:05 |
| *** DericHorn-HP has quit IRC | 17:05 | |
| greghaynes | fungi: do we run with debug logs enabled? | 17:05 |
| timrc | I really don't agree with the thinking on orgs in Grafana... We need an extra layer... like "teams" to separate dashboards and delegate authority. | 17:05 |
| *** e0ne has quit IRC | 17:05 | |
| timrc | Not such an issue upstream, though. | 17:05 |
| *** mrmartin_ has joined #openstack-infra | 17:05 | |
| fungi | greghaynes: yep | 17:05 |
| greghaynes | hrm | 17:05 |
| fungi | greghaynes: no matches on get_session_endpoint either | 17:06 |
| SpamapS | greghaynes: I don't know if that should happen. Not super familiar with it. | 17:06 |
| *** mrmartin has quit IRC | 17:06 | |
| SpamapS | greghaynes: rather deep in infra-cloud debugging ATM | 17:06 |
| greghaynes | SpamapS: ah, fun | 17:07 |
| * greghaynes will still ramble but feel free to ignore | 17:07 | |
| *** cindyo has joined #openstack-infra | 17:07 | |
| *** sabeen has joined #openstack-infra | 17:08 | |
| *** samueldmq has quit IRC | 17:08 | |
| *** tonytan4ever has quit IRC | 17:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Initial unit tests for CLI https://review.openstack.org/189770 | 17:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Fix --version CLI output https://review.openstack.org/189788 | 17:08 |
| *** samueldmq has joined #openstack-infra | 17:08 | |
| greghaynes | Seems like os_storage_url isnt set for our swift.client - it does get set automagically in my test case and looks like that happens via the shade get_session_endpoint method | 17:08 |
| greghaynes | I guess it might be time to try and run nodepool against rax :/ | 17:09 |
| *** sergmelikyan has quit IRC | 17:09 | |
| *** sergmelikyan has joined #openstack-infra | 17:10 | |
| *** baoli_ has quit IRC | 17:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Add datasource functionality https://review.openstack.org/194265 | 17:11 |
| greghaynes | looks like theres a case where we would log that we couldnt find the endpoint, but I wouldnt put it past some of the clients to also return None in wierd cases | 17:11 |
| *** amuller has joined #openstack-infra | 17:11 | |
| amuller | G'day! On this wonderful day neutron pep8 job is broken. Pylint is failing, looks like there was no new pylint release, but there was an astroid and logilab-common release today. | 17:12 |
| amuller | Getting some import error in pylint | 17:12 |
| *** sabeen has quit IRC | 17:12 | |
| amuller | I see that we're pinning pylint now in global reqs, which is good | 17:12 |
| openstackgerrit | Merged openstack-infra/openstackid: [smarcet] https://review.openstack.org/196898 | 17:13 |
| amuller | I wonder about pinning pylint's major dependencies, I don't know how would that look like or if that's a good idea | 17:13 |
| amuller | lifeless: ^ | 17:13 |
| *** _diana_ has joined #openstack-infra | 17:14 | |
| *** sabeen1 has joined #openstack-infra | 17:14 | |
| *** jasondotstar has quit IRC | 17:15 | |
| *** mpaolino has quit IRC | 17:15 | |
| openstackgerrit | Merged openstack-infra/yaml2ical: start_date now supported in schedules https://review.openstack.org/195262 | 17:17 |
| freerunner | dhellmann: Heya! Are u here? | 17:17 |
| *** sdake has joined #openstack-infra | 17:18 | |
| AJaeger | fungi: Haven't looked at the translation problem yet. Looking at your description, I wonder whether this would hit the requirements proposal as well. | 17:18 |
| AJaeger | fungi, thanks for looking into it. | 17:18 |
| dhellmann | freerunner: here | 17:18 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add neutron to single node baremetal0 https://review.openstack.org/196334 | 17:19 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Add nova to baremetal0 https://review.openstack.org/197187 | 17:19 |
| freerunner | dhellmann: Yesterday I've talked with morganfainberg about python-keystoneclient releases for juno and kilo and he forwarded me to you. | 17:19 |
| morganfainberg | freerunner: kilo was just released | 17:20 |
| dhellmann | freerunner: yes, let's talk about that in #openstack-relmgr-office | 17:20 |
| morganfainberg | freerunner: i don't thnk we have anything pending on juno | 17:20 |
| SpamapS | pabelanger: ^^ note that there are updates, in case you're working on splitting that into sane files. :) | 17:20 |
| *** jsavak has joined #openstack-infra | 17:20 | |
| *** mikeit has quit IRC | 17:21 | |
| *** sdake_ has quit IRC | 17:21 | |
| *** kiran-r has quit IRC | 17:22 | |
| *** sabeen1 has quit IRC | 17:23 | |
| sdake | mordred fine sir happen to be around? | 17:24 |
| *** sabeen has joined #openstack-infra | 17:24 | |
| *** dguerri is now known as dguerri` | 17:25 | |
| *** jasondotstar has joined #openstack-infra | 17:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Add a non-voting py34 check job to os-brick https://review.openstack.org/197064 | 17:28 |
| *** annegentle has quit IRC | 17:28 | |
| *** dannywilson has quit IRC | 17:29 | |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Add missing get_repo calls https://review.openstack.org/197189 | 17:29 |
| *** sambetts has quit IRC | 17:29 | |
| *** whoops has joined #openstack-infra | 17:29 | |
| asselin | Hi, I'm looking through puppet-jenkins module, and wondering why the master doesn't need to create the jenkins user? it's created indirectly while installing the plugins | 17:29 |
| *** notnownikki has quit IRC | 17:29 | |
| *** sambetts has joined #openstack-infra | 17:29 | |
| nibalizer | timrc: probably knows asselin | 17:29 |
| jeblair | dstanek: missed a few things with getRepo; fixed in https://review.openstack.org/197189 | 17:30 |
| openstackgerrit | Merged stackforge/gertty: Add missing get_repo calls https://review.openstack.org/197189 | 17:30 |
| *** sergmelikyan has quit IRC | 17:31 | |
| fungi | fanatical support just updated my graphite.o.o ticket and is escalating to operations (presumably to troubleshoot connectivity for the compute node where it's running) | 17:31 |
| greghaynes | fanatical operations? | 17:31 |
| *** jsavak has quit IRC | 17:33 | |
| fungi | i think they like to be known as "extremist operations" | 17:33 |
| *** fawadkhaliq has quit IRC | 17:33 | |
| *** notnownikki has joined #openstack-infra | 17:33 | |
| greghaynes | seems pretty scary | 17:34 |
| *** _diana_ has quit IRC | 17:35 | |
| *** jsavak has joined #openstack-infra | 17:35 | |
| fungi | i would have also accepted "phrenetical" and "heretical" | 17:36 |
| openstackgerrit | Drew Thorstensen proposed openstack/requirements: Add pypowervm to global requirements https://review.openstack.org/196641 | 17:36 |
| EmilienM | pabelanger: I saw your question on #openstack-puppet, (we are in #puppet-openstack now) - no we don't plan to support centos6 at all | 17:36 |
| EmilienM | pabelanger: "beaker jobs for unit testing puppet" > they do *functional* testing | 17:37 |
| *** kiran-r has joined #openstack-infra | 17:37 | |
| *** baoli has joined #openstack-infra | 17:37 | |
| asselin | timrc, do you know what this is for? ssh_authorized_key { 'jenkins-master-2014-04-24': https://github.com/openstack-infra/puppet-jenkins/blob/master/manifests/jenkinsuser.pp#L58 | 17:37 |
| *** jschlueter has left #openstack-infra | 17:38 | |
| *** yfried has joined #openstack-infra | 17:38 | |
| *** amuller is now known as amuller_afk | 17:38 | |
| *** baoli has quit IRC | 17:38 | |
| *** achanda_ has joined #openstack-infra | 17:38 | |
| *** dboik has quit IRC | 17:39 | |
| *** baoli has joined #openstack-infra | 17:39 | |
| *** dboik has joined #openstack-infra | 17:39 | |
| *** patrickeast has joined #openstack-infra | 17:39 | |
| fungi | asselin: the jenkins::jenkinsuser class is what gets applied to all our jenkins slaves, and that authorized key allows a jenkins master to ssh into them so it can push and initialize the slave agent | 17:40 |
| *** spzala has quit IRC | 17:40 | |
| *** abitha has quit IRC | 17:40 | |
| *** achanda has quit IRC | 17:40 | |
| asselin | fungi, to be clear, you mean all jenkins slaves, or just static slaves? | 17:41 |
| fungi | asselin: unless someone has changed things around, the jenkins::jenkinsuser doesn't (or didn't previously) get applied on jenkins masters | 17:41 |
| *** tqtran has joined #openstack-infra | 17:41 | |
| *** vnod has joined #openstack-infra | 17:41 | |
| asselin | fungi, correct, it still doesn't | 17:41 |
| fungi | asselin: i think all, but i'll double-check | 17:41 |
| *** annegentle has joined #openstack-infra | 17:41 | |
| *** achanda has joined #openstack-infra | 17:42 | |
| *** pradk has joined #openstack-infra | 17:44 | |
| fungi | asselin: it's included by the jenkins::slave class which even our openstack_project::single_use_slave nodes instantiate | 17:44 |
| fungi | asselin: unless user=false gets passed in | 17:44 |
| *** achanda_ has quit IRC | 17:44 | |
| fungi | to be clear, openstack_project::single_use_slave includes openstack_project::slave_common which includes jenkins::slave which includes jenkins::jenkinsuser | 17:45 |
| AJaeger | fungi, are you looking into the translation stuff? I have one idea and can propose a change... | 17:45 |
| *** mrmartin_ has quit IRC | 17:45 | |
| *** kiran-r has quit IRC | 17:45 | |
| fungi | AJaeger: i was about to get started on it, but feel free if you want to give it a try and i'll manually test it by rerunning one of those jobs with your modification in place | 17:46 |
| *** sputnik13 has joined #openstack-infra | 17:46 | |
| *** sdake_ has joined #openstack-infra | 17:47 | |
| AJaeger | thanks, fungi! | 17:48 |
| AJaeger | change will come in the next 10 mins - after I done some basic testing myself | 17:48 |
| fungi | i'll be watching | 17:48 |
| *** fawadkhaliq has joined #openstack-infra | 17:50 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:50 | |
| *** sdake has quit IRC | 17:50 | |
| *** sdake_ is now known as sdake | 17:50 | |
| *** xarses has quit IRC | 17:51 | |
| *** ihrachyshka has joined #openstack-infra | 17:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix proposal operations https://review.openstack.org/197200 | 17:52 |
| AJaeger | fungi: here's the patch to use git checkout -B as you suggested | 17:52 |
| *** vnod has quit IRC | 17:52 | |
| AJaeger | fungi: This works fine - but I could not reproduce the failure I got, so I might have tested something else ;( | 17:52 |
| AJaeger | For me, git review always worked, even in detached state ;( Even when I updated to latest git review from pypi | 17:53 |
| fungi | AJaeger: i'll give it a try. should i rerun nova-propose-translation-update with it, or is there a better job to try first? | 17:53 |
| *** HeOS has quit IRC | 17:53 | |
| AJaeger | nrun glance-propose-translation-update - that's a smaller repo and might be faster ;) | 17:53 |
| fungi | thanks, giving it a try now | 17:54 |
| timrc | asselin, Sorry I had to resolve an incident in BigCorpLand. Reading up. | 17:55 |
| *** mrmartin has joined #openstack-infra | 17:56 | |
| timrc | asselin, Ah it looks like fungi already addressed your question? | 17:56 |
| timrc | I wish my phone was as responsive as fungi :) | 17:56 |
| *** chlong has quit IRC | 17:56 | |
| *** alivigni has joined #openstack-infra | 17:57 | |
| *** enikanorov2 has quit IRC | 17:57 | |
| openstackgerrit | James Slagle proposed openstack-infra/irc-meetings: Update the TripleO meeting schedule https://review.openstack.org/197202 | 17:57 |
| openstackgerrit | John Griffith proposed openstack-infra/project-config: Add config for Cinder LVM Thin configuration https://review.openstack.org/182499 | 17:57 |
| *** baoli has quit IRC | 17:58 | |
| *** e0ne has joined #openstack-infra | 17:58 | |
| *** yamahata has joined #openstack-infra | 17:59 | |
| *** baoli has joined #openstack-infra | 17:59 | |
| AJaeger | fungi, if you look at https://jenkins.openstack.org/job/glance-propose-translation-update/55/consoleFull, you see that it still complains about "A new version of git-review is availble on PyPI. Please update your copy...". | 17:59 |
| AJaeger | fungi: Is there a chance to easily upgrade git-review on the proposal node so that we don't see this warning anymore? | 17:59 |
| *** amuller_afk is now known as amuller | 17:59 | |
| fungi | AJaeger: yeah, we probably specify a version in our puppet manifest | 18:00 |
| docaedo | Who else is looking forward to July 9th when Wo gets back in the office? I can't wait! | 18:00 |
| nibalizer | EmilienM: haha your upgrade jobs leaked into infras beaker use too | 18:00 |
| EmilienM | oops | 18:01 |
| *** jsavak has quit IRC | 18:01 | |
| EmilienM | nibalizer: is it a big deal? | 18:01 |
| *** dannywilson has joined #openstack-infra | 18:01 | |
| fungi | docaedo: i've already marked it on the calendar | 18:01 |
| *** tonytan4ever has joined #openstack-infra | 18:01 | |
| EmilienM | nibalizer: I honestly think they can help | 18:01 |
| EmilienM | nibalizer: though it requires a fix to work properly (cc crinkle) | 18:01 |
| nibalizer | what is the purpose of the upgrade test? | 18:02 |
| pabelanger | EmilienM, okay, perfect | 18:02 |
| EmilienM | nibalizer: 1/ run beaker on top of HEAD^1 2/ run beaker again on top of the patchset | 18:02 |
| AJaeger | fungi: thanks to hound, I found: http://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/slave.pp - requesting 1.17 | 18:02 |
| EmilienM | and see if the resources survive | 18:02 |
| fungi | AJaeger: wow, that's crazy ancient | 18:02 |
| EmilienM | it's a grenade like | 18:02 |
| pabelanger | EmilienM, Trying to see what is actually using centos6 within -infra, so far only puppet-apply is what I see | 18:02 |
| *** otter768 has joined #openstack-infra | 18:03 | |
| * AJaeger prepares a change... | 18:03 | |
| pabelanger | which will be removed once beaker is working | 18:03 |
| fungi | pabelanger: you mean jobs or other things too? | 18:03 |
| pabelanger | fungi, mostly puppet jobs for now | 18:03 |
| fungi | pabelanger: we have more than a few python 2.6 unit tests jobs running on centos 6 for example. we also have our asterisk pbx and our git servers and haproxy for them on centos 6 | 18:04 |
| pabelanger | fungi, however, anything using git in centos6 will be affected with performance issues | 18:04 |
| fungi | yep | 18:04 |
| pabelanger | fungi, right, I knew of asterisk and git using centos 6 | 18:04 |
| *** e0ne is now known as e0ne_ | 18:04 | |
| *** cindyo has quit IRC | 18:05 | |
| *** notnown20 has joined #openstack-infra | 18:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-jenkins: Update git-review https://review.openstack.org/197203 | 18:05 |
| *** jlanoux has joined #openstack-infra | 18:05 | |
| *** notnownikki has quit IRC | 18:06 | |
| *** geoffarnold has quit IRC | 18:06 | |
| *** notnownikki has joined #openstack-infra | 18:07 | |
| *** e0ne_ is now known as e0ne | 18:07 | |
| *** xarses has joined #openstack-infra | 18:07 | |
| AJaeger | lifeless: we wondered recently about this error message, here's a patch to fix it ^ | 18:07 |
| *** otter768 has quit IRC | 18:07 | |
| pabelanger | so, am I missing something or can't you abandon a patchset in gertty? | 18:08 |
| AJaeger | fungi, I see a change from the translation bot! | 18:08 |
| pabelanger | CTRL-a it look slike | 18:08 |
| AJaeger | fungi, is the log file available somewhere or did you run this manually? | 18:09 |
| nibalizer | EmilienM: oh okay | 18:09 |
| *** ivar-laz_ has joined #openstack-infra | 18:09 | |
| nibalizer | i thought it was like try kilo then try liberty | 18:09 |
| *** ivar-laz_ has quit IRC | 18:09 | |
| *** ivar-lazzaro has quit IRC | 18:09 | |
| fungi | AJaeger: disabled puppet and reran manually on that worker with your one-liner patch applied. result just posted at https://review.openstack.org/197206 | 18:10 |
| *** ivar-lazzaro has joined #openstack-infra | 18:10 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Fix puppet deprecated warning https://review.openstack.org/195256 | 18:10 |
| *** notnown20 has quit IRC | 18:10 | |
| *** Akshay00 has joined #openstack-infra | 18:10 | |
| EmilienM | nibalizer: it's long term goal | 18:10 |
| nibalizer | ya that is pretty useful i think | 18:10 |
| nibalizer | I'll have to noodle on it | 18:10 |
| EmilienM | nibalizer: but we don't have beaker in stable/juno | 18:10 |
| EmilienM | nibalizer: I'll propose such a change after our release | 18:10 |
| *** spzala has joined #openstack-infra | 18:11 | |
| *** SlickNik_ has quit IRC | 18:11 | |
| *** vnod has joined #openstack-infra | 18:11 | |
| *** SlickNik_ has joined #openstack-infra | 18:12 | |
| jeblair | yeah, the branch->branch upgrade case is handled fairly well in zuul-cloner because of grenade, but we don't have many (or perhaps any current) jobs doing commit->commit differences | 18:12 |
| jeblair | (that would probably be a good enhancement to zuul-cloner) | 18:12 |
| AJaeger | fungi, thanks a lot for testing! | 18:12 |
| EmilienM | jeblair: yeah that's why it's just for experimentation now | 18:12 |
| EmilienM | jeblair: when we cut our kilo branch, I'll change the workflow to do branch2branch | 18:13 |
| jeblair | oh cool | 18:13 |
| AJaeger | jeblair, could you review https://review.openstack.org/197200 as well to fix the proposal bot scripts, please? | 18:13 |
| *** annegentle has quit IRC | 18:13 | |
| nibalizer | EmilienM: does it leverage catalog-diff? | 18:13 |
| Akshay00 | When we try to to run git review we get an error that says we need to update our current contact information. But when we try to enter in our contact info it says it cannot be updated. Would anyone know about this? | 18:13 |
| *** ericksonsantos has joined #openstack-infra | 18:14 | |
| EmilienM | nibalizer: do you mean, test if it's idempotent ? | 18:14 |
| jeblair | AJaeger: er, how did that break? | 18:14 |
| Akshay00 | My colleague and I are currently interning at Intel | 18:14 |
| AJaeger | jeblair: see https://jenkins.openstack.org/job/glance-propose-translation-update/55/consoleFull . I don't fully understand the breakage but it seems related to the fact that we start with a detached state | 18:15 |
| fungi | Akshay00: https://ask.openstack.org/question/56720 | 18:15 |
| *** annegentle has joined #openstack-infra | 18:16 | |
| fungi | Akshay00: though the short answer is to make sure you're following the instructions listed at http://docs.openstack.org/infra/manual/developers.html#account-setup thoroughly and in the sequence described | 18:16 |
| *** Sukhdev_ has joined #openstack-infra | 18:16 | |
| AJaeger | fungi: regarding the git-review update for https://review.openstack.org/197203: I'm fine with using latest as well. Shall I update? | 18:17 |
| fungi | AJaeger: go for it | 18:17 |
| *** olaph has joined #openstack-infra | 18:17 | |
| *** e0ne is now known as e0ne_ | 18:18 | |
| *** ildikov has joined #openstack-infra | 18:18 | |
| jeblair | AJaeger: i feel like i will need to do about an hour of research before i can review that :/ | 18:18 |
| fungi | also i have hopes that https://review.openstack.org/190825 may reduce the number of people who end up here asking about contact information update errors from gerrit, if anyone gets a chance to eventually review and approve it | 18:18 |
| *** puranamr_ has quit IRC | 18:18 | |
| *** jason10258 has joined #openstack-infra | 18:19 | |
| jeblair | fungi: oh, huh, i thought i had. +2. | 18:19 |
| fungi | thanks! | 18:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-jenkins: Always install latest git-review https://review.openstack.org/197203 | 18:20 |
| fungi | AJaeger: i'm going to test latest git-review with the nova translation updates just to make sure it's not hiding any surprises | 18:21 |
| *** achanda has quit IRC | 18:21 | |
| *** e0ne_ is now known as e0ne | 18:23 | |
| *** gyee has joined #openstack-infra | 18:24 | |
| AJaeger | fungi, thanks! | 18:25 |
| fungi | AJaeger: https://review.openstack.org/197213 but seems to have updated a lot of linenumber comments. is that expected? | 18:25 |
| Akshay00 | fungi thank you I will try to work through these links although I have tried the second one already | 18:25 |
| timrc | fungi, Does upstream throttle reviews? We just had a user create hundreds of reviews (on accident) | 18:26 |
| *** shardy has quit IRC | 18:27 | |
| timrc | AFAICT there has never been a legitimate need to do that... would be good for Gerrit to say "Uh, no" when someone tries to do that. | 18:27 |
| fungi | timrc: we do not, no | 18:27 |
| *** shardy has joined #openstack-infra | 18:27 | |
| *** jasondotstar has quit IRC | 18:27 | |
| AJaeger | fungi: yeah, it is ;( We should review at one point whether we really need the pot files (source). We removed the line numbers from the po files because of that noise a few weeks ago | 18:27 |
| fungi | timrc: if a feature like that existed in gerrit, what threshold would you set it at? | 18:28 |
| AJaeger | fungi, strange things are happening: https://jenkins.openstack.org/job/api-site-manuals-upstream-translation-update/100/console | 18:28 |
| timrc | fungi, !insane... dunno... I would need to look at it heuristically... there is probably some reasonable upper bound. | 18:28 |
| fungi | i'm sort of curious. i mean we have devs (who shall remain nameless) intentionally push 50+ change series topic branches somewhat regularly | 18:29 |
| *** wenlock_ has quit IRC | 18:29 | |
| AJaeger | "Exception: A strange error happened." | 18:29 |
| *** jlanoux has quit IRC | 18:29 | |
| timrc | fungi, If there is a legitimate use case then it should be supported, but safe guards to prevents accidents that can bring zuul to a grinding halt would be a Good Thing (tm) in my opinion. | 18:29 |
| openstackgerrit | Merged openstack-infra/infra-manual: Update account setup to specify foundation member https://review.openstack.org/190825 | 18:30 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Make include_path deterministic https://review.openstack.org/187075 | 18:30 |
| *** maishsk_afk has joined #openstack-infra | 18:30 | |
| nibalizer | asselin: i've completed my voting on openstackci so if you want to try to schedule a time for that stuff I'm in | 18:30 |
| nibalizer | i wont be able to infra meeting today | 18:30 |
| *** maishsk_afk is now known as maishsk | 18:30 | |
| AJaeger | fungi, https://jenkins.openstack.org/job/openstack-manuals-manuals-upstream-translation-update/ failed the same way ;( I'll try to reproduce tomorrow if it still fails... | 18:31 |
| fungi | timrc: out of curiosity, was the user running git-review? it provides a warning when you're pushing more than one commit, so it would have given them the number of new changes they were creating | 18:31 |
| timrc | fungi, Yeah this was done through git-review. | 18:31 |
| timrc | fungi, Merged a bunch of commits to a new branch and then git-review'ed and it created a review per commit. | 18:32 |
| fungi | and asked them to confirm | 18:32 |
| *** samuelBartel has joined #openstack-infra | 18:32 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: rename puppet-syntax to puppet-parser-validation https://review.openstack.org/197216 | 18:32 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: rename puppet-syntax-future to puppet-syntax https://review.openstack.org/197217 | 18:32 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job https://review.openstack.org/197218 | 18:32 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches https://review.openstack.org/197219 | 18:32 |
| EmilienM | nibalizer, _ody ^ | 18:32 |
| fungi | timrc: well, also merge commits are often a sign something is being done wrong. were your acls set to allow them to propose merge commits? | 18:32 |
| *** Akshay00 has quit IRC | 18:32 | |
| *** ZZelle_ has joined #openstack-infra | 18:33 | |
| fungi | timrc: we _do_ disallow merge commits by default, except generally to release managers maintaining merges in and out of feature branches | 18:33 |
| timrc | fungi, Looks to be. I'm just now starting to look into root cause of this snafu. Generally speaking we give people enough rope to hang themselves and the rest of us with it seems :) | 18:33 |
| *** enikanorov2 has joined #openstack-infra | 18:34 | |
| EmilienM | crinkle: you might want to see https://review.openstack.org/#/q/topic:puppet-jobs-refacto,n,z | 18:34 |
| *** maishsk_afk has joined #openstack-infra | 18:34 | |
| *** Akshay00 has joined #openstack-infra | 18:35 | |
| *** kushal has quit IRC | 18:35 | |
| *** maishsk has quit IRC | 18:35 | |
| *** rossella_s has quit IRC | 18:37 | |
| *** rossella_s has joined #openstack-infra | 18:37 | |
| *** pradk has quit IRC | 18:37 | |
| *** olaph has quit IRC | 18:38 | |
| *** Akshay00 has quit IRC | 18:38 | |
| fungi | AJaeger: that tx/txclib (transifex client?) exception is indeed strange and useless. i'm not sure what to make of it, but i'm going to guess something in the content of those projects is triggering an error in whatever version of txclib ended up on the new proposal worker | 18:38 |
| *** Akshay00 has joined #openstack-infra | 18:38 | |
| AJaeger | fungi: tx is the transifex client. Let me check whether I can reproduce it on my system... | 18:39 |
| *** davideagnello has quit IRC | 18:39 | |
| *** cindyo has joined #openstack-infra | 18:39 | |
| fungi | i mean, an error message at least _describing_ what was happening in that function call would be far more useful than "Exception: A strange error happened. | 18:40 |
| AJaeger | fungi: indeed | 18:40 |
| * AJaeger can't find that error message in the client | 18:41 | |
| *** ivar-lazzaro has quit IRC | 18:41 | |
| fungi | AJaeger: it's being raised from a call to push self.get_full_path(sfile) -> do_url_request so i'd first suspect some odd characters in file/directory names which aren't getting properly munged into a url? | 18:41 |
| fungi | AJaeger: the error seems to be coming from txclib which tx calls into | 18:42 |
| *** ivar-lazzaro has joined #openstack-infra | 18:42 | |
| timrc | fungi, I'm not sure this was even a proper merge. It looks like the person cherry-picked individual changes into a new branch and then git-review'ed em. | 18:42 |
| fungi | timrc: oh! so probably intentional | 18:43 |
| *** annegentle has quit IRC | 18:43 | |
| *** notnownikki has quit IRC | 18:43 | |
| fungi | timrc: we definitely have devs do that though it's generally in the sub-100 change order of scale. and we do have quota of ~900 job workers to chew through them when it happens | 18:44 |
| timrc | fungi, Yeah... but... but why. | 18:44 |
| *** davideagnello has joined #openstack-infra | 18:44 | |
| fungi | i've personally scripted pushing 200+ new changes at a time on multiple occasions, though i tend to wait until the ci system is mostly idle | 18:45 |
| *** xarses has quit IRC | 18:45 | |
| fungi | and i don't do them a a dependent change series | 18:45 |
| fungi | er, as a | 18:45 |
| *** xarses has joined #openstack-infra | 18:45 | |
| *** sarob_ has joined #openstack-infra | 18:45 | |
| timrc | fungi, What's weird(er) is that 300+ changes hit gerrit but we had something like 79K jobs in zuul (I didn't confirm that myself) | 18:45 |
| *** xarses has quit IRC | 18:45 | |
| fungi | that does sound weird | 18:46 |
| fungi | maybe zuul's event/result queue spiked up to 79k? | 18:46 |
| *** sdake has quit IRC | 18:46 | |
| *** geoffarnold has joined #openstack-infra | 18:46 | |
| *** sdake has joined #openstack-infra | 18:47 | |
| timrc | More appropriately maybe 79k changesets in the queue -- not sure how that happened :/ | 18:47 |
| AJaeger | fungi, reproduced it locally... | 18:47 |
| *** kushal has joined #openstack-infra | 18:48 | |
| AJaeger | fungi, pushing the failing file again, everything is fine. | 18:48 |
| AJaeger | I wonder whether transifex is broken ;( | 18:49 |
| fungi | AJaeger: given it's an opaque proprietary service, it's especially hard to know | 18:49 |
| AJaeger | fungi: yeah ;( | 18:50 |
| fungi | graphite's ipv4 connectivity _just_ came back up | 18:50 |
| fungi | no word from phrenetical support yet on what they did to fix it | 18:51 |
| jhesketh | Morning | 18:51 |
| *** yamamoto has quit IRC | 18:51 | |
| AJaeger | morning, jhesketh ! | 18:51 |
| fungi | jhesketh: enjoyed those 3 hours of sleep i hope? | 18:51 |
| greghaynes | yea, werent you just here jhesketh ? | 18:52 |
| jhesketh | fungi: fantastic ;-) | 18:52 |
| AJaeger | fungi: the post queue is still not moving forward, top job is there since 11hours. Do we have a problem? | 18:52 |
| jhesketh | I'll probably snooze again after the meeting, otherwise lots of coffee today :-) | 18:53 |
| fungi | AJaeger: i'll check into that next--thanks for the reminder | 18:53 |
| AJaeger | fungi: thanks a lot! | 18:53 |
| openstackgerrit | Assaf Muller proposed openstack/requirements: Update pylint pin to 1.4.4 https://review.openstack.org/197226 | 18:53 |
| *** vnod has quit IRC | 18:54 | |
| fungi | AJaeger: my guess is we've somehow accumulated a lot of deadweight workers in nodepool or something bringing all the low-priority jobs to a halt since the few available workers are being snatched up by the higher priority queues, but i'm digging in now | 18:54 |
| AJaeger | fungi, now the strange error happened at a different file. I'll keep an eye on this, might need to raise a bug with transifex | 18:54 |
| greghaynes | hrm, now im especially confused - my nodepool appears to be uploading a dib image to rax just fine | 18:55 |
| *** jasondotstar has joined #openstack-infra | 18:55 | |
| *** sarob_ has quit IRC | 18:56 | |
| openstackgerrit | Merged openstack-dev/cookiecutter: Add insentive to write a long description https://review.openstack.org/197146 | 18:56 |
| fungi | AJaeger: yep, we have ~100 nodes in a ready state for more than an hour at the moment | 18:56 |
| *** Longgeek has quit IRC | 18:57 | |
| AJaeger | fungi, looking at the txlib source code: The Exception comes from an error message from the remote transifex server ;( | 18:58 |
| fungi | i'll keep working on this in the background while we're meeting | 18:58 |
| *** vnod has joined #openstack-infra | 18:58 | |
| AJaeger | fungi: mmh, that looks nice | 18:58 |
| AJaeger | fungi, thanks for the reminder of the meeting ;) | 18:58 |
| *** Longgeek has joined #openstack-infra | 18:58 | |
| *** hamzy has quit IRC | 18:59 | |
| AJaeger | fungi, third time pushing the *same* content to transifex, it did not fail but succeeded... | 19:00 |
| jeblair | meeting time! | 19:00 |
| fungi | AJaeger: yep, i bet transifex is broken | 19:00 |
| * AJaeger won't bet against fungi on this | 19:01 | |
| *** kgiusti has left #openstack-infra | 19:01 | |
| *** hamzy has joined #openstack-infra | 19:03 | |
| *** dboik has quit IRC | 19:04 | |
| *** dboik has joined #openstack-infra | 19:04 | |
| fungi | oh wow, >450 floating ips in hpcloud with no associated instance. cleaning those up now | 19:05 |
| *** fallenpegasus has quit IRC | 19:05 | |
| mordred | fungi: yay! | 19:06 |
| *** vnod has quit IRC | 19:06 | |
| *** Guest80179 has joined #openstack-infra | 19:06 | |
| greghaynes | mordred: so, im having a pretty painful time trying to replicate that traceback locally | 19:08 |
| *** maishsk_afk has quit IRC | 19:08 | |
| greghaynes | er, context | 19:08 |
| greghaynes | the nodepool swift traceback | 19:08 |
| mordred | greghaynes: oh awesome. that's great to hear | 19:08 |
| *** _nadya_ has joined #openstack-infra | 19:09 | |
| *** achanda has joined #openstack-infra | 19:10 | |
| *** maishsk_afk has joined #openstack-infra | 19:10 | |
| fungi | not so great for solving our upload failures though | 19:10 |
| *** geoffarnold has quit IRC | 19:10 | |
| greghaynes | I did notice that the only way it happens is if swift client doesnt get an object_storage_url specified in os_options | 19:10 |
| *** dguerri` is now known as dguerri | 19:11 | |
| *** _nadya__ has joined #openstack-infra | 19:11 | |
| greghaynes | and in my shade test that is being set via the shade get_session_endpoint... so I wonder if theres something about our nodepool config which is causing that to fail | 19:11 |
| *** _nadya_ has quit IRC | 19:11 | |
| greghaynes | not sure if that sends of any lightbulbs... | 19:11 |
| mordred | hrm. lemme go look at something | 19:12 |
| *** shardy has quit IRC | 19:13 | |
| *** e0ne is now known as e0ne_ | 19:14 | |
| *** Guest80179 has quit IRC | 19:14 | |
| *** e0ne_ is now known as e0ne | 19:15 | |
| *** dguerri is now known as dguerri` | 19:15 | |
| greghaynes | mordred: btw - what im referring to re: object_storage_url is line 44 on http://paste.openstack.org/show/321629/ - that only happens if self.url is none which seems like something shade should be passing in | 19:16 |
| *** tonytan4ever has quit IRC | 19:16 | |
| *** puranamr has joined #openstack-infra | 19:17 | |
| *** puranamr has quit IRC | 19:18 | |
| *** puranamr has joined #openstack-infra | 19:18 | |
| mordred | greghaynes: ZOMG | 19:18 |
| *** jason10258 has quit IRC | 19:18 | |
| greghaynes | zomg? | 19:18 |
| mordred | greghaynes: there is no swift endpoint in ORD | 19:18 |
| mordred | in rax | 19:19 |
| greghaynes | :) | 19:19 |
| fungi | wow | 19:19 |
| greghaynes | that would explain why I cant replicate | 19:19 |
| fungi | how do we... | 19:19 |
| fungi | i mean... | 19:19 |
| greghaynes | because I dont have access to ORD | 19:19 |
| mordred | so - | 19:19 |
| fungi | glance... | 19:19 |
| fungi | v2 | 19:19 |
| fungi | images | 19:19 |
| fungi | ord | 19:19 |
| fungi | i don' | 19:19 |
| fungi | t even | 19:19 |
| greghaynes | hahaa | 19:19 |
| mordred | next quesion - can we import an image into glance in ORD from swift in IAD? | 19:19 |
| mordred | nikhil_k-away: ^^ ??? | 19:19 |
| mordred | greghaynes: so - shade is working | 19:19 |
| greghaynes | yes | 19:19 |
| mordred | greghaynes: or - perhaps we should throw an exception there if endpoint is None | 19:20 |
| mordred | so that the error is clearerer | 19:20 |
| greghaynes | Yea, definitely - looks like our get endpoint method tries to but for some reason this case it just returns none | 19:20 |
| greghaynes | anyhow, that is amazing | 19:21 |
| greghaynes | mordred: hrm, we dont specify a region or anything AFAIK when we do the glance task-create, so not sure how we would pull from another region... | 19:22 |
| *** geoffarnold has joined #openstack-infra | 19:23 | |
| *** cindyo has quit IRC | 19:25 | |
| mordred | greghaynes: it's a good question | 19:26 |
| *** jason10258 has joined #openstack-infra | 19:26 | |
| *** jsavak has joined #openstack-infra | 19:26 | |
| greghaynes | ok, im going to make a patch to not do the dib.ubuntu-trusty in ORD for now | 19:26 |
| greghaynes | because thers not a whole lot else we can do | 19:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/infra-specs: Host trystack.o.o site https://review.openstack.org/195098 | 19:27 |
| pc_m | mtreinish: hi | 19:29 |
| *** tonytan4ever has joined #openstack-infra | 19:30 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/nodepool: Clarify meaning of Jenkins master configuration settings in target https://review.openstack.org/167878 | 19:31 |
| *** rm_work is now known as rm_work|away | 19:31 | |
| mtreinish | pc_m: hello | 19:31 |
| pc_m | mtreinish: I'm still battling with getting functional jobs running that can reference local neutron repo with changes. | 19:32 |
| mtreinish | pc_m: what was the link to your patch again? | 19:32 |
| pc_m | mtreinish: Here's what I have... committed code passes Jenkins. | 19:32 |
| pc_m | https://review.openstack.org/#/c/196237/ | 19:32 |
| pc_m | mtreinish: In my local area however, both functional tests fail. | 19:32 |
| *** rm_work|away is now known as rm_work | 19:33 | |
| mtreinish | pc_m: your local env is screwed up then :) | 19:33 |
| pc_m | mtreinish: What I see is that in .tox/dsvm-functional/bin, it is missing many files, especially the neutron-rootwrap and neutron-rootwrap-daemon files that are needed for these tests. | 19:33 |
| *** HeOS has joined #openstack-infra | 19:33 | |
| *** crc32 has joined #openstack-infra | 19:33 | |
| *** ivar-lazzaro has quit IRC | 19:34 | |
| pc_m | mtreinish: Do you know how that bin dir gets populated? | 19:34 |
| *** armax has quit IRC | 19:34 | |
| *** dimtruck is now known as zz_dimtruck | 19:34 | |
| *** aysyd has quit IRC | 19:34 | |
| mtreinish | pc_m: pip will put things in there when it installs a package | 19:34 |
| openstackgerrit | Thanh Ha proposed openstack-infra/nodepool: Add prerequisites for nodepool on RHEL7 https://review.openstack.org/170700 | 19:35 |
| mtreinish | like when it installs neutron it should install all the console_scripts in neutron's setup.cfg in there | 19:35 |
| openstackgerrit | Thanh Ha proposed openstack-infra/nodepool: Add MySQL DB creation example https://review.openstack.org/170699 | 19:35 |
| *** ivar-lazzaro has joined #openstack-infra | 19:35 | |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Remove ubuntu-trusty from rax ORD https://review.openstack.org/197238 | 19:36 |
| pc_m | mtreinish: So, it the working case, where we install neutron via master (https:), it gets placed in .tox area with source is src subdir. Does it also place bin and etc files too? | 19:36 |
| hogepodge | wiki.openstack.org is down | 19:37 |
| hogepodge | (for me at least) | 19:37 |
| pc_m | mtreinish: Wondering how to handle that for this case, where we are not installing neutron in the venv. | 19:38 |
| hogepodge | nevermind, it's back | 19:38 |
| *** xyang1 has quit IRC | 19:38 | |
| hogepodge | weird | 19:38 |
| fungi | hogepodge: i'll check on it anyway just in case | 19:38 |
| *** xyang1 has joined #openstack-infra | 19:38 | |
| pc_m | mtreinish: Thinking out loud. One option may be to copy the needed files over, as part of the rootwrap_deploy.sh script. | 19:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: refactorize puppet-syntax job https://review.openstack.org/197216 | 19:40 |
| pc_m | mtreinish: Another would be to refer to the scripts in the Neutron area via the OS_ROOTWRAP_CMD, but use the venv .conf file (not sure if these files get tweaked though) | 19:40 |
| pc_m | mtreinish: I guess they don't. | 19:41 |
| mtreinish | pc_m: neither of those is anything you want to do. | 19:41 |
| pc_m | mtreinish: Suggestion? | 19:41 |
| pc_m | mtreinish: Currently, the rootwrap_deploy.sh creates the .conf file, and moves the filters into the .tox area. | 19:42 |
| *** jason10258 has quit IRC | 19:43 | |
| mtreinish | pc_m: you'll have to give me a min. I'm in the middle of a couple things | 19:43 |
| pc_m | mtreinish: The tox.ini runs the rootwrap scripts from the .tox/*/bin area and uses the .conf file in etc. | 19:43 |
| mtreinish | I need to take a look at what that script is doing | 19:43 |
| pc_m | mtreinish: sure... | 19:43 |
| mtreinish | but my gut feeling is that something is wrong in your env (no idea what) but since the gate job passes it probably means that neutron is properly packaging what you need and pip is installing it in the venv for the run | 19:44 |
| *** nihilifer has quit IRC | 19:44 | |
| *** maishsk_afk is now known as maishsk | 19:44 | |
| pc_m | mtreinish: That was another question... why it works in gate job. FYI: On one VM, I wiped out /opt/stack and restacked with DevStack to get a fresh environment. Tested it worked, then patched in 196237, removed .tox and tried and it faile=s there too. | 19:47 |
| *** tqtran is now known as tqtran-afk | 19:47 | |
| *** john-davidge has quit IRC | 19:48 | |
| *** baoli has quit IRC | 19:48 | |
| *** jsavak has quit IRC | 19:49 | |
| mordred | jeblair, fungi, greghaynes: RAX ORD doesn't have swift. we also only have 55 nodes of capacity there ... perhaps we should just kill it | 19:49 |
| greghaynes | mordred: kill it with fire | 19:49 |
| jeblair | mordred: ! | 19:49 |
| greghaynes | mordred: does rax support glance import from http url? | 19:50 |
| *** aysyd has joined #openstack-infra | 19:50 | |
| mordred | jeblair: SYD and HKG have swift. we could maybe add one of them if we're worried about the capacity | 19:50 |
| notmyname | mordred: RAX ORD does have swift | 19:50 |
| *** dboik has quit IRC | 19:50 | |
| mordred | notmyname: it does not show up in my keystone catalog | 19:51 |
| *** dboik has joined #openstack-infra | 19:51 | |
| notmyname | mordred: RAX has swift in ORD, DFW, HKG, SYD, IAD, LON | 19:51 |
| mordred | notmyname: I show DFW, IAD, HKG and SYD | 19:51 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job https://review.openstack.org/197218 | 19:51 |
| *** Rockyg has joined #openstack-infra | 19:51 | |
| *** yamamoto has joined #openstack-infra | 19:52 | |
| mordred | notmyname: oh - weird - I DO see it in the openstackci account. hrm | 19:52 |
| dstanek | jeblair: nice. i didn't realize that all the gertty stuff was getting approved. thx! | 19:52 |
| *** pblaho has quit IRC | 19:52 | |
| pleia2 | oops, switching from openstackid-dev to openstackid.org broke zanata login | 19:53 |
| pleia2 | ^[[0m^[[0m2015-06-30T19:44:43,727Z INFO [org.openid4java.discovery.html.HtmlResolver] (default task-16) HTML discovery completed on: https://openstackid.org/ | 19:53 |
| pleia2 | ^[[0m^[[0m2015-06-30T19:44:43,727Z INFO [org.openid4java.discovery.Discovery] (default task-16) Discovered 0 OpenID endpoints. | 19:53 |
| jeblair | dstanek: i merged a couple, but have some comments on the next one | 19:53 |
| openstackgerrit | Merged openstack/requirements: Require fixtures 1.3.1. https://review.openstack.org/196863 | 19:53 |
| mordred | greghaynes: ok - we may have to dig further. we're winding up with endpoint = None I believe | 19:53 |
| pleia2 | did I get the endpoint wrong? | 19:53 |
| greghaynes | mordred: oh? You see the endpoint listed somewhere? | 19:53 |
| dstanek | jeblair: i'll take a look and get them addressed later today | 19:53 |
| mordred | greghaynes: yah | 19:53 |
| greghaynes | mordred: I dont have ORD access at all, is that another just ask nicely to get thing? | 19:53 |
| mtreinish | pc_m: so looking at deploy_rootwrap script itself there is nothing really wrong there | 19:54 |
| *** yamahata has quit IRC | 19:54 | |
| pc_m | mtreinish: Yeah, it seems to be doing the right thing to me. | 19:54 |
| mordred | greghaynes: maybe? | 19:54 |
| *** yamahata has joined #openstack-infra | 19:54 | |
| mordred | greghaynes: http://paste.openstack.org/show/329514/ | 19:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/186829 | 19:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements https://review.openstack.org/197249 | 19:54 |
| mtreinish | pc_m: but the way your calling it your tox.ini has a hardcoded expectation that neutron is cloned in the dir as neutron-vpnaas | 19:54 |
| *** achanda has quit IRC | 19:54 | |
| mtreinish | which is not true in every case | 19:54 |
| mordred | greghaynes: it's also possible that we need to do something with that tenantId from the catalog | 19:54 |
| mtreinish | I expect you're local env is that way while the gate is not | 19:55 |
| greghaynes | mordred: so easy fix for now is that we can pass in that that url manually | 19:55 |
| *** aysyd has quit IRC | 19:55 | |
| pc_m | mtreinish: I din't quite follow that. | 19:55 |
| *** achanda has joined #openstack-infra | 19:55 | |
| mordred | greghaynes: are you sure we're getting None for endpoint there? | 19:56 |
| pc_m | mtreinish: did you mean that it has the expectation that neutron is a peer to neutron-vpnaas? | 19:56 |
| greghaynes | mordred: No, I cant actually replicate, but there is a if url is None right above that line in the traceback which causes this | 19:56 |
| mtreinish | pc_m: I'll leave a comment in the review | 19:56 |
| *** yamamoto has quit IRC | 19:56 | |
| mordred | greghaynes: I don't see if url is None | 19:57 |
| *** cindyo has joined #openstack-infra | 19:57 | |
| mordred | greghaynes: line number? | 19:57 |
| greghaynes | http://git.openstack.org/cgit/openstack/python-swiftclient/tree/swiftclient/client.py#n1444 | 19:57 |
| mordred | oh! gotcha | 19:58 |
| fungi | this is interesting... | 19:58 |
| pc_m | mtreinish: I don't see how that can affect this issue, as, in the local area, neutron is in a peer dir to neutron-vpnaas | 19:58 |
| mordred | greghaynes: OH!!!!! | 19:58 |
| mordred | "type": "object-store", | 19:58 |
| mordred | oh - nevermind | 19:58 |
| greghaynes | fungi: Do you know if the other regions worked for our nodepool uploads/ | 19:59 |
| *** fawadkhaliq has quit IRC | 19:59 | |
| greghaynes | locally they seem to work for me... | 19:59 |
| *** zz_dimtruck is now known as dimtruck | 19:59 | |
| fungi | "I was able to determine that your IP address had been null routed at our AGGRs due to an apparent glean attack. It appears that while your graphite server was frozen, there was a large influx of traffic. As there was no MAC for this IP address in the AGGR's arp cache, and your cloud server was not responding, all traffic was forwarded to the AGGR's CPU for processing causing performance degradation | 19:59 |
| fungi | for other customers." | 19:59 |
| *** achanda has quit IRC | 19:59 | |
| fungi | greghaynes: it did not appear so, but i can check the upload for, say, iad to confirm | 20:00 |
| greghaynes | yesplz | 20:00 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job https://review.openstack.org/197218 | 20:00 |
| pleia2 | mrmartin: perhaps you can help me? having some trouble with openstackid.org | 20:01 |
| *** abregman has joined #openstack-infra | 20:01 | |
| mtreinish | pc_m: you should *not* have the test commands in the tox.ini have a hard coded dependency on the location of the neutron repo | 20:01 |
| mrmartin | pleia2: what's wrong? | 20:01 |
| mtreinish | that's just going to cause issues | 20:01 |
| pleia2 | mrmartin: I switched translate-dev.openstack.org from openstackid-dev.o.o to openstackid.org, now it's not finding endpoints (see my comments about 8 minutes ago) | 20:01 |
| pc_m | mtreinish: OK. How do I determine where neutron is located? | 20:02 |
| mtreinish | you need to make sure things are properly packaged in neutron, or included in the neutron-vpnaas repo | 20:02 |
| fungi | greghaynes: sorry but same traceback in iad and dfw | 20:02 |
| greghaynes | hrm | 20:02 |
| mtreinish | pc_m: you shouldn't need to know that. That's what I'm saying, requiring a specific path outside of the repo is what is causing your issues | 20:02 |
| pc_m | mtreinish: I think you're starting to lose me... | 20:02 |
| mrmartin | pleia2: is everything ok with the SSL certificates? it is using a different X509 | 20:03 |
| greghaynes | that does help a lot though - so I should be able to replicate | 20:03 |
| *** agentle has joined #openstack-infra | 20:03 | |
| *** patrickeast_ has joined #openstack-infra | 20:03 | |
| mrmartin | pleia2: maybe you should check you can open a simple SSL connection to openstackid.org:443 | 20:03 |
| pleia2 | mrmartin: it's a self signed cert :\ but it was with openstackid-dev too | 20:03 |
| *** matbu_ has joined #openstack-infra | 20:03 | |
| mrmartin | pleia2: openstackid.org using a real cert I guess, and I'm not sure it is included in java's keystore | 20:03 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/196504 | 20:03 |
| *** otter768 has joined #openstack-infra | 20:03 | |
| *** patrickeast has quit IRC | 20:04 | |
| *** patrickeast_ is now known as patrickeast | 20:04 | |
| pc_m | mtreinish: Sure, and our goal is to separate the two, making neutron a library, but we are far from being able to do that currently. How do we deal with it in the interim. | 20:04 |
| mrmartin | pleia2: but it is just a quick tip based on my previous java / ssl experience | 20:04 |
| pleia2 | mrmartin: hm, it worked ok with openstackid-dev | 20:04 |
| AJaeger | pleia2: could you review https://review.openstack.org/#/c/197200/ , please? | 20:04 |
| mtreinish | pc_m: that's not what I'm talking about | 20:04 |
| mtreinish | pc_m: this is about where the configuration files for running commands in the test runs live | 20:04 |
| mrmartin | pleia2: I see, dev is using the similar Comodo CA as an issuer | 20:05 |
| fungi | pleia2: yeah, not snakeoil/self-signed on the production openstackid.org | 20:05 |
| mtreinish | pc_m: right now in the tox.ini you have a hard coded dependency on the neutron repo living somewhere | 20:05 |
| mrmartin | fungi: but we have a proper cert at openstackid-dev.o.o too | 20:05 |
| fungi | pleia2: i installed a cert bought from the same ca i use for our other https sites that aren't snakeoil | 20:05 |
| *** jasondotstar has quit IRC | 20:05 | |
| pleia2 | fungi: sorry, I meant translate-dev.o.o itself is self-signed | 20:05 |
| pc_m | mtreinish: Soa re you saying to clone all the .conf, rootwrap scripts from neutron to VPN repo? | 20:06 |
| mtreinish | pc_m: that's not something you want to be doing, you need to make sure neutron either installs that into a known location (so after you pip install neutron in the venv you know where the file lives) | 20:06 |
| fungi | pleia2: oh, good point. that may pose a problem since i think openstackid does a callback to the url? | 20:06 |
| mrmartin | pleia2: I see some banned ips, with auth errors on openstackid.org | 20:06 |
| mtreinish | pc_m: or you keep a local copy in the vpnaas repo that you can point to directly (in the alternative you could have a script generate it I guess) | 20:06 |
| mrmartin | pleia2: I did a cleanup now, so please try again to auth | 20:07 |
| *** armax has joined #openstack-infra | 20:07 | |
| mtreinish | pc_m: but saying neutron lives at ../neutron and crossing your fingers is not a good idea | 20:07 |
| pleia2 | mrmartin: still failing | 20:07 |
| pc_m | mtreinish: understood, and I was wondering how to handle that. | 20:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Output some debug information for swift endpoints https://review.openstack.org/197292 | 20:07 |
| pc_m | mtreinish: In this case, however, option #1 is not possible, right? | 20:07 |
| mrmartin | pleia2: I don't have access to that server, but if you have one, what you think about checking the logs at /srv/.../app/storage/ ? | 20:08 |
| SpamapS | \o/ nova talks to me now | 20:08 |
| * SpamapS builds an image and enrolls a server | 20:08 | |
| pleia2 | mrmartin: the translate-dev server? | 20:08 |
| *** otter768 has quit IRC | 20:08 | |
| *** aysyd has joined #openstack-infra | 20:08 | |
| pc_m | mtreinish: We're not installing neutron, because we're trying to be able to use an already installed neutron (so that we get the change set that may be present there), right? | 20:08 |
| mrmartin | pleia2: no, the openstackid.org server | 20:08 |
| *** dimtruck is now known as zz_dimtruck | 20:08 | |
| pleia2 | mrmartin: ah, sure, sec | 20:08 |
| mtreinish | pc_m: no you're installing neutron, that's what tox_install.sh is ensuring | 20:09 |
| mrmartin | pleia2: maybe you can see some exceptions there | 20:09 |
| pleia2 | mrmartin: thanks, looking | 20:09 |
| mtreinish | it either uses a copy already present in the venv, leverages zuul-cloner if it's present to get an up-to-date neutron repo (which is a dirty hack) and installs that in the venv, or locally checks it out from git and installs that in the venv | 20:09 |
| mrmartin | pleia2: the two site have different cert issuer | 20:10 |
| pc_m | mtreinish: OK, well, as it stands right now, in a local environment, when I run the functional test it is not getting installed. | 20:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches https://review.openstack.org/197294 | 20:10 |
| *** gokrokve has joined #openstack-infra | 20:10 | |
| mtreinish | pc_m: which comes back to my original statement, something is weird in your local env :) | 20:11 |
| mrmartin | pleia2: the dev was issued by: PositiveSSL CA 2 and the openstackid.o.o cert was issued by COMODO RSA Domain Validation Secure Server CA | 20:11 |
| *** erikwilson_ has quit IRC | 20:11 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches https://review.openstack.org/197219 | 20:11 |
| mrmartin | pleia2: so I still suggest to check the default java cert keystore | 20:11 |
| pleia2 | http://paste.openstack.org/show/329544/ | 20:11 |
| pc_m | mtreinish: Are you saying that when this runs in the gate... the .tox/dsvm-functional/ area will have neutron source, bin, and etc? | 20:12 |
| pleia2 | java itself isn't complaining directly about ssl | 20:12 |
| *** maishsk has quit IRC | 20:12 | |
| mtreinish | pc_m: neutron is being installed in the venv used for running the jobs. tox_install.sh is calling zuul-cloner to update the locally checked out copy of neutron and then installing that in the venv | 20:13 |
| *** frobware_ has joined #openstack-infra | 20:13 | |
| *** sergmelikyan has joined #openstack-infra | 20:13 | |
| mtreinish | tbh, I'm not sure how it has perms to do that in the gate env, but it doesn't appear to error out | 20:13 |
| mtreinish | so I'm assuming it's all working | 20:14 |
| *** nihilifer has joined #openstack-infra | 20:14 | |
| pc_m | mtreinish: ah, so in the local area, doing "pip install -U $*" is not installing. | 20:14 |
| fungi | mrmartin: pleia2: yep, the change in issuer is because the ca we're using converted from 1024->2048 bit certs between when i bought those | 20:14 |
| mrmartin | pleia2: and do you know what is this openid discovery process exactly? I guess it must be a simple GET request | 20:14 |
| *** maishsk has joined #openstack-infra | 20:14 | |
| mtreinish | pc_m: well that command is used to install any other requirements | 20:14 |
| mordred | greghaynes: https://review.openstack.org/197292 is a stab at looking at what's going on if we don't have a swift endpoint - including dumping the catalog to the logs | 20:14 |
| pc_m | mtreinish: I see that script called twice, once with args of -rtest-requirements.txt -rrequirements.txt (with paths), and again with neutron-vpnaas | 20:14 |
| pleia2 | mrmartin: I can dig into it, cinerama was handling this previously | 20:15 |
| mtreinish | if something fails before that point it'll never install any requirements | 20:15 |
| pc_m | mtreinish: Never called with neutron | 20:15 |
| *** changbl has quit IRC | 20:15 | |
| mrmartin | pleia2: if you can do the same with curl, maybe it should help to identify the differences between the two deployment | 20:15 |
| pleia2 | mrmartin: ok, thanks | 20:15 |
| cinerama | hi mrmartin pleia2 | 20:16 |
| greghaynes | mordred: +1 | 20:16 |
| pleia2 | cinerama: hey! was there a particular reason we were using openstackid-dev rather than openstackid.org? | 20:16 |
| mordred | greghaynes: I'm trying to replicate touching swift | 20:16 |
| mrmartin | hi cinerama | 20:16 |
| cinerama | mrmartin, pleia2: so i submitted a patch to openstackid which got applied to -dev but perhaps not to the main one | 20:16 |
| greghaynes | mordred: hrm? | 20:16 |
| pleia2 | cinerama: aha! | 20:16 |
| mrmartin | cinerama: eh, it can be an issue | 20:16 |
| *** Sukhdev_ has quit IRC | 20:16 | |
| mrmartin | cinerama: what was the patch exactly? | 20:16 |
| mtreinish | fungi, jeblair, mordred: if you get a sec tempest needs a quick reqs bump on tempest-lib to land a patch: https://review.openstack.org/#/c/194951/ | 20:16 |
| openstackgerrit | greghaynes proposed openstack-infra/shade: WIP test case for rax image uploads https://review.openstack.org/196936 | 20:17 |
| pleia2 | cinerama: thanks, we're finally getting this in the hands of translators to test out so we need to get off the -dev id sever so they can use their regular accounts | 20:17 |
| greghaynes | mordred: I just do ^ with a clouds.yaml that looks like the one in system-config and make an image.vhd file manually using dib | 20:17 |
| *** sergmelikyan has quit IRC | 20:17 | |
| cinerama | mrmartin, pleia2: https://review.openstack.org/#/c/165696/ | 20:18 |
| *** claudiub has quit IRC | 20:18 | |
| greghaynes | but, that seems to work just fine... so something is amiss | 20:18 |
| pc_m | mtreinish: it doesn't fail, and gets to the pip install line. How does that line install neutron, if it is never specified as one of the args coming in? | 20:18 |
| *** sergmelikyan has joined #openstack-infra | 20:18 | |
| mtreinish | pc_m: that L37 or L33 is what installs neutron | 20:18 |
| *** matbu_ has quit IRC | 20:19 | |
| mordred | greghaynes: hrm. yah. it's working for me | 20:19 |
| cinerama | mrmartin, pleia2: so zanata is generating a more complex request header than openstackid was expecting - that change is necessary to get zanata to work with openstackid | 20:19 |
| mtreinish | pc_m: it does it under the covers | 20:19 |
| pleia2 | cinerama: gotcha, thank you for digging that up | 20:19 |
| mtreinish | pc_m: it's kinda the whole point of adding that script | 20:19 |
| *** _nadya__ has quit IRC | 20:19 | |
| mtreinish | pc_m: it's the hack to get around not actually being able to put neutron in your requirements file | 20:19 |
| mtreinish | fungi: thanks | 20:19 |
| pc_m | mtreinish: Maybe you're missing the point I'm trying to make. Let me rephrase... | 20:20 |
| *** samuelBartel has quit IRC | 20:20 | |
| pleia2 | mrmartin: so, do you know how we can get changes applied to openstackid.org? | 20:20 |
| greghaynes | mordred: yea, and so I tried pdb'ing along that traceback and that line in swiftclient is what seems to be divergent from my local setup and that traceback | 20:20 |
| mrmartin | pleia2, cinerama: yeah, I can confirm, that no new release had been rolled out to openstackid.org since march of 13, so cinerama's patch was not applied there | 20:20 |
| *** matbu has joined #openstack-infra | 20:20 | |
| * pleia2 nods | 20:20 | |
| pleia2 | just missed the window :) | 20:21 |
| mordred | greghaynes: oh. I wonder ... | 20:21 |
| pc_m | mtreinish: If I have a machine with neutron and neutron-vpnaas repos installed, have changes in my neutron repo, and want to run the neutron-vpnaas functional test to verify that the changes do not affect neutorn-vpnaas code. | 20:21 |
| lifeless | fungi: so how is requirements check looking now the fun of static is over? | 20:21 |
| lifeless | fungi: and le periodic constraints job? | 20:22 |
| pleia2 | AJaeger: approved the proposal fix | 20:22 |
| pc_m | mtreinish: In that local host scenario, how do I get neutron-vpnaas to refer to the neutron code? | 20:22 |
| mtreinish | pc_m: then based on the way that script is written you have to manually install neutron in your venv | 20:22 |
| AJaeger | thanks, pleia2 ! | 20:22 |
| mrmartin | pleia2: ok, so what's the plan with that? someone needs to roll out a new release there | 20:22 |
| fungi | lifeless: dunno--i haven't looked again yet. everything decided to blow up on meeting day | 20:22 |
| pleia2 | mrmartin: yeah, I don't know how that's done :) | 20:22 |
| mtreinish | from you local repo with your local changes | 20:22 |
| mordred | greghaynes: looking to see what v of swiftclient is running there | 20:23 |
| greghaynes | mordred: hah | 20:23 |
| mrmartin | just need to tag the git repo | 20:23 |
| mtreinish | pc_m: it's basically what that zuul-cloner if does in that script, but since it's your local env you have to do it manually | 20:23 |
| greghaynes | mordred: looked like there hasnt been a swiftclient release in a while | 20:23 |
| lifeless | fungi: boomala boomala | 20:23 |
| pc_m | mtreinish: Would L40 (already installed) every be reached, if running in the gate? IOW, can we assume we are on a local machine at L40? | 20:23 |
| *** serg_melikyan has joined #openstack-infra | 20:24 | |
| mordred | greghaynes: 2.4.0 is on nodepool | 20:24 |
| mordred | greghaynes: I have 2.3.1 locally on my laptop | 20:24 |
| mtreinish | pc_m: L40 is always run (unless there is an error) | 20:24 |
| greghaynes | python-swiftclient==2.4.0 | 20:24 |
| pleia2 | mrmartin: ok, so you and smarcet are part of openstackid-release, so does one of you tag it? | 20:24 |
| greghaynes | for me | 20:24 |
| pc_m | mtreinish: whoops. Duh | 20:24 |
| *** sergmelikyan has quit IRC | 20:25 | |
| mrmartin | pleia2: technically yes | 20:25 |
| greghaynes | mordred: can you paste the whole pip freexe? | 20:25 |
| mordred | greghaynes: yah. one sec | 20:25 |
| *** zz_dimtruck is now known as dimtruck | 20:25 | |
| pc_m | mtreinish: Trying to see if there is a way to tell if local and then install neutron using reference to existing area. | 20:25 |
| mordred | greghaynes: http://paste.openstack.org/show/329574/ | 20:25 |
| *** weshay has quit IRC | 20:26 | |
| pleia2 | mrmartin: well I don't want to force a release into production if it's a problem/not ready/whatever, and I'm not in a good position to judge that, do you want to consult with smarcet and see if it's ok? | 20:26 |
| *** weshay has joined #openstack-infra | 20:26 | |
| *** erikmwilson_ has joined #openstack-infra | 20:27 | |
| lifeless | fungi: ok so can I provoke an eyeball ;) | 20:27 |
| *** erikmwilson_ has quit IRC | 20:27 | |
| pleia2 | mrmartin: we definitely need this fix soon, but I didn't realize it until today so that's kind of on me | 20:27 |
| mrmartin | yeah, my only issues that he enrolled a large patchset today: https://github.com/openstack-infra/openstackid/commit/50bbfa7a8f86f1b32af355fb0a4f913b5cf656a3 | 20:27 |
| mtreinish | pc_m: you have to stop trying to manually automagically using local clones of repos in your scripts. If you have a local repo living somewhere that you want to use in the test env you have to install in the venv yourself. | 20:27 |
| pleia2 | mrmartin: ok | 20:27 |
| mrmartin | pleia2: I can talk with smarcet, and can roll out the new release tomorrow if everything works well, I can do some tests with groups.o.o | 20:27 |
| mtreinish | pc_m: the script you're adding allows for that, and it's the only way to sanely work with a local checkout in this framework you've built | 20:27 |
| pleia2 | mrmartin: that would be wonderful, thank you | 20:28 |
| mrmartin | pleia2: try to catch me tomorrow | 20:28 |
| pleia2 | mrmartin: will do :) | 20:28 |
| mrmartin | thnx | 20:28 |
| *** hodos|2 has joined #openstack-infra | 20:28 | |
| *** nihilifer has quit IRC | 20:28 | |
| asselin | pleia2, nibalizer any reason not to merge this? https://review.openstack.org/#/c/175970/ | 20:29 |
| pc_m | mtreinish: So, if working locally, and want to handle the usecase mentioned, I need to manually install neutron into venv? | 20:29 |
| mtreinish | pc_m: yes | 20:29 |
| mtreinish | pc_m: you could modify the script to take in a repo path as an option env var and just run pip install -e env_var or something I guess | 20:30 |
| greghaynes | mordred: nothing interesting in the diff between me and that freeze :( | 20:30 |
| mtreinish | pc_m: but that's more or less the same thing as doing it manually | 20:30 |
| fungi | lifeless: i hope so. we'll see how this afternoon goes. i'd like to pay attention in the tc and x-proj meetings but so far too many distractions from people asking for help with things | 20:30 |
| mordred | greghaynes: me either | 20:30 |
| *** hodos has quit IRC | 20:31 | |
| pc_m | mtreinish: Is there a way to build the venv w/o running the test? Otherwise, I have to run the test, fail, install neutron, rerun the test. | 20:31 |
| *** dprince has quit IRC | 20:31 | |
| mtreinish | pc_m: tox has a --notest arg | 20:32 |
| mordred | greghaynes: what about keystoneclient? | 20:32 |
| *** serg_melikyan has quit IRC | 20:32 | |
| greghaynes | mordred: 1.6.0 for both | 20:33 |
| pc_m | mtreinish: thanks | 20:33 |
| openstackgerrit | Merged openstack-infra/project-config: Fix proposal operations https://review.openstack.org/197200 | 20:33 |
| *** alivigni has quit IRC | 20:33 | |
| *** sergmelikyan has joined #openstack-infra | 20:34 | |
| *** _nadya_ has joined #openstack-infra | 20:34 | |
| mordred | greghaynes: I can't figure out how it's getting None | 20:35 |
| *** e0ne has quit IRC | 20:35 | |
| greghaynes | mordred: did you merge the debug patch? | 20:35 |
| mordred | greghaynes: no - running locally | 20:35 |
| mordred | greghaynes: trying something real quick | 20:36 |
| mordred | greghaynes: I can do swift operations as the nodepool user on nodepool.o.o against rax iad | 20:37 |
| *** jtomasek has quit IRC | 20:37 | |
| mordred | greghaynes: and ORD | 20:37 |
| mordred | greghaynes: so I'm VERY confused\ | 20:38 |
| mordred | greghaynes: http://paste.openstack.org/show/329591/ | 20:38 |
| *** EricGonczer_ has quit IRC | 20:39 | |
| *** _nadya_ has quit IRC | 20:39 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Modify secgroup rule processing https://review.openstack.org/197308 | 20:39 |
| *** _nadya_ has joined #openstack-infra | 20:40 | |
| Shrews | mordred: https://review.openstack.org/197308 | 20:40 |
| nibalizer | asselin: i think given the content that a jeblair +2 is in order for that patc | 20:41 |
| nibalizer | but thats just my opinon | 20:41 |
| *** ChuckC has joined #openstack-infra | 20:42 | |
| greghaynes | mordred: get_capabilities is the issue though, not sure get_container will kick that off? | 20:42 |
| pc_m | mtreinish: thanks. I'm trying to test it locally. | 20:42 |
| mordred | oh. hrm | 20:42 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: refactorize puppet-syntax job https://review.openstack.org/197216 | 20:42 |
| *** ChuckC has left #openstack-infra | 20:43 | |
| *** mrmartin has quit IRC | 20:43 | |
| *** amit213 has quit IRC | 20:44 | |
| *** amit213 has joined #openstack-infra | 20:44 | |
| *** _nadya_ has quit IRC | 20:44 | |
| *** amit213 has quit IRC | 20:44 | |
| mordred | greghaynes: I just ran get_object_capabilities() and it ran fine too | 20:44 |
| *** dimtruck is now known as zz_dimtruck | 20:45 | |
| greghaynes | mordred: hah | 20:45 |
| *** zz_dimtruck is now known as dimtruck | 20:46 | |
| *** claudiub has joined #openstack-infra | 20:47 | |
| *** ociuhandu_ has joined #openstack-infra | 20:48 | |
| greghaynes | mordred: is it possible that nodepool got restarted in a bad state? like maybe clouds.yaml was missing or we upgraded deps while it was running so it didnt restart and is running against older libs than are installed? | 20:49 |
| mordred | fungi, jeblair, clarkb, pleia2, jhesketh, SergeyLukjanov: can we install requests[security] on the machines that have requests installed via pip? | 20:49 |
| fungi | lifeless: resetting my brainstate to where i passed out last night... the requirements check job was breaking on most projects because they still had requirements-py3k.txt files. what was out next step there? mass-propose deletion patches for those? thread on the ml? switch careers to longshoremen? | 20:49 |
| mordred | greghaynes: I'm not sure- I'm slightly worried that maybe because it's more complex than my test script, state is getting unhappied somewhere | 20:50 |
| lifeless | fungi: jhesketh landed the patch, and built new images. | 20:50 |
| lifeless | fungi: and then static blew its ass out | 20:50 |
| fungi | mordred: i'm not opposed to that, but it will require making sure we have the additional library headers that needs installed on those systems | 20:50 |
| clarkb | mordred only if you also install all the C dev packages too | 20:50 |
| fungi | what clarkb said | 20:50 |
| clarkb | also good morning | 20:50 |
| lifeless | fungi: I don't know if the new images are still live or if he rolled them back as part of the troubleshooting steps | 20:50 |
| mordred | fungi, clarkb: cool. I'll work on puppet patch | 20:51 |
| greghaynes | mordred: well, AIUI we assert all the nodepool python deps as latest, but we dont restart nodepool when they change, so pip freeze doesnt necessarially say what the running nodepool is using | 20:51 |
| fungi | lifeless: oh, okay. so in theory they should pass now. i'll have a look | 20:51 |
| *** ZZelle_ has quit IRC | 20:51 | |
| lifeless | fungi: https://review.openstack.org/#/c/196930/ might be a good litmus test | 20:51 |
| lifeless | fungi: since the old code blew up on master | 20:51 |
| fungi | lifeless: from the looks of https://review.openstack.org/#/q/status:open+topic:openstack/requirements,n,z i expect we're fine now? | 20:52 |
| lifeless | fungi: I don't think so - https://jenkins03.openstack.org/job/gate-ceilometer-requirements/23/console | 20:52 |
| fungi | oh, poop | 20:53 |
| lifeless | fungi: I can't tell if that ran the old code or the new | 20:53 |
| lifeless | fungi: I might put up a patch to make that clearer | 20:53 |
| fungi | lifeless: 2015-06-30 20:03:37 would have run on images built today. i'll check and see if they actually succeeded | 20:54 |
| fungi | lifeless: i guess not. these images all appear to have been the ones i built about 23 hours ago | 20:55 |
| *** ociuhandu_ has quit IRC | 20:55 | |
| *** dboik has quit IRC | 20:55 | |
| lifeless | fungi: yeah, 23 hours is about when I started writing the patch IIRC | 20:55 |
| lifeless | fungi: so yeah. can has nu ones plox. | 20:55 |
| lifeless | fungi: I know jhesketh built some. | 20:55 |
| lifeless | fungi: maybe they are stashed not deleted? | 20:56 |
| *** dimtruck is now known as zz_dimtruck | 20:56 | |
| *** radez is now known as radez_g0n3 | 20:58 | |
| *** stevemar has quit IRC | 20:58 | |
| jhesketh | lifeless, fungi: nope, sorry, delete during all the debugging. | 20:58 |
| *** zz_dimtruck is now known as dimtruck | 20:59 | |
| fungi | jhesketh: so we're okay to roll out new bare-trusty images? i'll start generating those now | 20:59 |
| jhesketh | Yep, I don't see why jot | 20:59 |
| lifeless | was there a post-mortem sent to the list? I missed all the recovery excitement | 20:59 |
| fungi | yep | 21:00 |
| jhesketh | Deleting them wasn't necessary, I was just being cautious as it was my first time building them | 21:00 |
| fungi | i sent a brief reply | 21:00 |
| fungi | the post-mortem is that block storage backends went toes up in rackspace and yanked a cinder volume from the lvm vg to which we write out logs on static.o.o | 21:01 |
| fungi | so i needed to detach/reattach the right one via nova/cinder, then fsck a >12tib fs | 21:01 |
| *** olaph has joined #openstack-infra | 21:03 | |
| *** armax has quit IRC | 21:03 | |
| mordred | greghaynes: I'm going to beer | 21:03 |
| *** ddieterly has joined #openstack-infra | 21:03 | |
| greghaynes | mordred: that is probably the best idea | 21:03 |
| greghaynes | mordred: I think maybe the thing to try is restart nodepool with the debug patch, will answer two questions at once | 21:04 |
| greghaynes | but... we can do that post beer | 21:05 |
| *** Akshay00 has quit IRC | 21:05 | |
| *** aysyd has quit IRC | 21:06 | |
| *** john-davidge has joined #openstack-infra | 21:07 | |
| *** dizquierdo has joined #openstack-infra | 21:07 | |
| *** abregman has quit IRC | 21:08 | |
| *** vahidh has quit IRC | 21:09 | |
| *** puranamr has quit IRC | 21:10 | |
| *** rlandy has quit IRC | 21:10 | |
| *** xnox has quit IRC | 21:10 | |
| pc_m | mtreinish: Giving up for today. will try again tomorrow. Even with manually installing neutron, the tests are failing. | 21:10 |
| pc_m | mtreinish: Thanks for all the help hashing through this... | 21:10 |
| *** armax has joined #openstack-infra | 21:11 | |
| clarkb | fungi we still need to do the trove stuff? | 21:11 |
| *** rlandy has joined #openstack-infra | 21:11 | |
| clarkb | I can start with that on etherpad-dev and do the encoding switch | 21:12 |
| *** xnox has joined #openstack-infra | 21:12 | |
| *** julim has quit IRC | 21:13 | |
| *** olaph has quit IRC | 21:13 | |
| *** olaph has joined #openstack-infra | 21:13 | |
| fungi | clarkb: i asked support and they indicated that the migration, while it will cause a ~5 minute outage for database access, won't end up renaming the instance in dns. if we don't want to rebuild some instances and can suffer the 5-minute outage at some random time tomorrow then for most of our systems that's probably a lot less work | 21:14 |
| fungi | if we dump, destroy, create and import in to a new mysql version then we're going to also need to update hiera for the new hostname | 21:14 |
| fungi | so for etherpad we do need to do that for the encoding fix, but it's not necessarily critical that we do it before the db instance migrations happen tomorrow | 21:15 |
| clarkb | gotcha | 21:15 |
| clarkb | I am going to work on etherpad-dev as it needs doing but good to know on the others | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| fungi | clarkb: oh, though i did discover we're not backing up the database for openstackid.org yet. i ran a manual backup today and proposed https://review.openstack.org/197083 to solve it longer term | 21:16 |
| *** claudiub has quit IRC | 21:17 | |
| *** sergmelikyan has quit IRC | 21:17 | |
| *** cindyo has quit IRC | 21:17 | |
| mtreinish | pc_m: sure, np | 21:17 |
| openstackgerrit | Merged openstack-infra/shade: Correctly name the functional TestImage class https://review.openstack.org/196891 | 21:20 |
| *** vahidh has joined #openstack-infra | 21:21 | |
| *** tonytan4ever has quit IRC | 21:22 | |
| *** whoops has quit IRC | 21:25 | |
| clarkb | I am tempted to suggest we use on instance dbs again though | 21:25 |
| *** hodos|2 has quit IRC | 21:25 | |
| clarkb | trove seems to just be an additional complicated piece in a thibg that shouldnt be complicated | 21:25 |
| clarkb | only problem with that is performance nodes arent setup for mich peristent disk | 21:27 |
| pc_m | mtreinish: One quick question... | 21:27 |
| *** tvanderwiel_ has joined #openstack-infra | 21:28 | |
| pc_m | mtreinish: IF I source the activate, and do a "pip install -U /opt/stack/neutron" should that install the src dir, in addition to bin and etc? | 21:28 |
| *** Ryan_Lane has joined #openstack-infra | 21:28 | |
| Ryan_Lane | neat: https://github.com/stackforge/bandit | 21:28 |
| *** smarcet has quit IRC | 21:28 | |
| *** rfolco has quit IRC | 21:28 | |
| Ryan_Lane | just now seeing that, but whoever made that: thanks :) | 21:28 |
| fungi | Ryan_Lane: its developers are part of the openstack security project-team | 21:29 |
| *** tvanderwiel has quit IRC | 21:29 | |
| clarkb | would be nice if it didnt remake the flake8 wheel but ok | 21:30 |
| *** maishsk has quit IRC | 21:30 | |
| Ryan_Lane | :D | 21:30 |
| *** maishsk has joined #openstack-infra | 21:31 | |
| *** otter768 has joined #openstack-infra | 21:32 | |
| *** john-davidge has quit IRC | 21:32 | |
| *** tvanderwiel_ has quit IRC | 21:33 | |
| *** maishsk has quit IRC | 21:33 | |
| *** pc_m has quit IRC | 21:33 | |
| *** teran has joined #openstack-infra | 21:36 | |
| *** thorst has quit IRC | 21:38 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman https://review.openstack.org/195175 | 21:38 |
| *** fifieldt has quit IRC | 21:39 | |
| *** ayoung has quit IRC | 21:39 | |
| *** tqtran-afk is now known as tqtran | 21:39 | |
| *** jsavak has joined #openstack-infra | 21:41 | |
| *** peristeri has quit IRC | 21:41 | |
| *** mriedem has quit IRC | 21:41 | |
| fungi | greghaynes: ooh!!! | 21:41 |
| greghaynes | :) | 21:41 |
| * fungi bounces up and down | 21:41 | |
| greghaynes | thats just the start of it - that patch is still sharing db state with nodepool and such but it using gearman to kick off the builds | 21:42 |
| greghaynes | then im going to start pulling out the shared state | 21:42 |
| fungi | i for one welcome our new distributed and heterogeneous image build worker overlords | 21:43 |
| * SpamapS peruses the list of servers for one to enroll in baremetal0 | 21:43 | |
| greghaynes | SpamapS: just not the bastion! | 21:43 |
| greghaynes | hehe | 21:43 |
| SpamapS | right! | 21:43 |
| *** isviridov is now known as isviridov_away | 21:45 | |
| *** puranamr has joined #openstack-infra | 21:45 | |
| *** jsavak has quit IRC | 21:45 | |
| *** ddieterly has quit IRC | 21:45 | |
| *** ZZelle_ has joined #openstack-infra | 21:46 | |
| *** ociuhandu has joined #openstack-infra | 21:46 | |
| *** amitgandhinz has quit IRC | 21:46 | |
| *** koolhead17 has quit IRC | 21:47 | |
| *** puranamr has quit IRC | 21:47 | |
| *** puranamr has joined #openstack-infra | 21:48 | |
| * morganfainberg steals mordred's stash of wet cats. (or is he onto emus now?) | 21:49 | |
| *** skraynev has quit IRC | 21:49 | |
| *** ilyashakhat has quit IRC | 21:49 | |
| *** asahlin is now known as asahlin_away | 21:50 | |
| *** jsavak has joined #openstack-infra | 21:51 | |
| clarkb | fungi: I am going to call the new db etherpad-dev-mysql-5.6 so that it is clear which is which | 21:51 |
| *** fifieldt has joined #openstack-infra | 21:52 | |
| fungi | clarkb: that's fine. or you could just delete and replace it unless you really think you need to switch back and forth rapidly | 21:52 |
| *** tlian has quit IRC | 21:53 | |
| clarkb | more thinking about how the non -dev instance should look | 21:53 |
| clarkb | and for that one less comfortable with an extended outage while new db populates | 21:53 |
| *** VivCheri has quit IRC | 21:53 | |
| SpamapS | doh, default quota for nova is 20 cores | 21:53 |
| SpamapS | one baremetal is 24 .. doh | 21:53 |
| fungi | clarkb: well, we'll want to keep the service offline while we export and import anyway to avoid people adding data that gets undone in the migration | 21:54 |
| clarkb | fungi: true | 21:54 |
| fungi | clarkb: so i'm not sure we can make it much shorter by having two in parallel, but i guess it could shave a couple minutes off | 21:54 |
| fungi | up to you | 21:55 |
| fungi | i think the instance name is entirely cosmetic anyway, not even reflected in any of our configs, so it doesn't make a ton of difference if it changes | 21:55 |
| fungi | and we have to update the hostname for the instance in hiera when we cut over regardless | 21:56 |
| *** fallenpegasus has joined #openstack-infra | 21:56 | |
| clarkb | yup | 21:56 |
| clarkb | I will preserve the username and password of the old host though | 21:56 |
| *** timburke has joined #openstack-infra | 21:56 | |
| clarkb | unless you think its worth updating that too | 21:56 |
| fungi | shouldn't be necessary to change those | 21:57 |
| nibalizer | how does the openstack-infra/gerrit post stuff work? I would expect to see tags from openstackers in the tag list but I do not | 21:57 |
| fungi | nibalizer: i have no idea what that combination of words means. can you rephrase>? | 21:57 |
| *** ddieterly has joined #openstack-infra | 21:58 | |
| *** ihrachyshka has quit IRC | 21:58 | |
| fungi | nibalizer: you mean our backport-enabling local fork of the gerrit source tree? | 21:58 |
| fungi | nibalizer: and git tags? | 21:58 |
| *** matbu has left #openstack-infra | 21:58 | |
| clarkb | fungi: also this way I can build the non -dev db now, and get to using it later :) | 21:58 |
| nibalizer | fungi: yes | 21:59 |
| *** agentle has quit IRC | 21:59 | |
| *** puranamr has quit IRC | 21:59 | |
| nibalizer | fungi: i mean the gerrit-package-buck job | 21:59 |
| fungi | nibalizer: we deploy specific ref ids from backport branches like http://git.openstack.org/cgit/openstack-infra/gerrit/log/?h=openstack/2.10.2 | 21:59 |
| *** frobware_ has quit IRC | 21:59 | |
| *** ddieterly has quit IRC | 21:59 | |
| nibalizer | which seems to run in post, but i thought we needed to push tags into openstack-infra/gerrit to make post jobs fire | 21:59 |
| fungi | nibalizer: zuul fires that job in the post pipeline when we approve a patch on one of those branches, and then uploads the results to http://tarballs.openstack.org/ci/ | 22:00 |
| *** otter768 has quit IRC | 22:00 | |
| fungi | nibalizer: nope, the post pipeline is triggered by ref-updated (similar to change-merged) | 22:01 |
| nibalizer | cool | 22:01 |
| fungi | nibalizer: our release, pre-release and tag pipelines trigger on tags | 22:01 |
| nibalizer | so taron and I are looking at hound which is a go thing and wondering if something like that would be a good model | 22:01 |
| nibalizer | oh release is not post, right | 22:01 |
| clarkb | fungi: for 197083 those backups run at 0000UTC and since we don't use a fancy tool likethe percona backup thing it locks a bunch of the DB, for small DBs this isn't an issue but for a DB like silverstripe which is large could that cause cascading service issues? | 22:01 |
| fungi | nibalizer: right see http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n77 | 22:02 |
| clarkb | fungi: mostly not familiar enough with everything tlaking to that db to know what may be affected | 22:02 |
| *** doug-fish has left #openstack-infra | 22:02 | |
| fungi | clarkb: "large" in this case is 40mib compressed. took something like 3 seconds when i ran it manually | 22:02 |
| fungi | er, 50mib compressed | 22:02 |
| clarkb | ah ok not too bad | 22:02 |
| lifeless | fungi: images building? | 22:02 |
| clarkb | etherpad.o.o is bigger | 22:02 |
| fungi | lifeless: about half have completed, some are still going | 22:03 |
| lifeless | cool | 22:03 |
| fungi | lifeless: the rest seem to be wrapping up, though one needed restarting early on so may be a little bit longer still | 22:03 |
| *** jsavak has quit IRC | 22:03 | |
| *** spzala has quit IRC | 22:04 | |
| *** dizquierdo has quit IRC | 22:04 | |
| *** yamahata has quit IRC | 22:04 | |
| clarkb | fungi: db backups approved | 22:05 |
| *** puranamr has joined #openstack-infra | 22:05 | |
| *** hdd has quit IRC | 22:06 | |
| *** gokrokve has quit IRC | 22:07 | |
| *** gordc is now known as gordc_afk | 22:09 | |
| *** davideag_ has joined #openstack-infra | 22:09 | |
| *** claudiub has joined #openstack-infra | 22:12 | |
| *** Rockyg has quit IRC | 22:12 | |
| openstackgerrit | Merged openstack-infra/system-config: Add database backups for openstackid.org https://review.openstack.org/197083 | 22:12 |
| *** davideagnello has quit IRC | 22:13 | |
| lifeless | fungi: ok so here is something odd | 22:14 |
| lifeless | https://review.openstack.org/#/c/196930/ | 22:14 |
| lifeless | fungi: look at the top | 22:14 |
| lifeless | fungi: then look at the CI messages in detail | 22:14 |
| *** olaph has quit IRC | 22:15 | |
| *** olaph has joined #openstack-infra | 22:15 | |
| clarkb | taking a lock on hieradata | 22:17 |
| fungi | lifeless: what's odd there? | 22:18 |
| *** dims_ has joined #openstack-infra | 22:19 | |
| lifeless | fungi: gate-ceilometer-requirements is missing from the summary at the top | 22:19 |
| lifeless | fungi: but failed and is shown down the bottom | 22:19 |
| lifeless | fungi: so when you look at just the top it says 'jenkins -1' and all the tests are green. | 22:19 |
| fungi | "incompatible requirements" results from the gate-ceilometer-requirements job don't get extracted into the job result summary table because the extended result text doesn't fit our usual success/failure pattern | 22:19 |
| *** sergmelikyan has joined #openstack-infra | 22:19 | |
| lifeless | so, thats the why... its still funny :) | 22:19 |
| lifeless | odd-funny, not haha funny | 22:20 |
| fungi | lifeless: we can sacrifice the descriptive message and just make it "success" or "failure" instead | 22:20 |
| lifeless | fungi: what prevents us having schrodinger cake? | 22:20 |
| clarkb | https://etherpad-dev.openstack.org/p/clarkb-test is running with mysql5.6 and the special utf8 encoding | 22:20 |
| fungi | and make people pull up the log, or know that a failure of that job means their requirements are probably incompatible | 22:20 |
| clarkb | so please test and throw 4 byte utf8 chars at it | 22:20 |
| *** rlandy has quit IRC | 22:20 | |
| *** dims has quit IRC | 22:22 | |
| fungi | lifeless: i can get behind the suggestion that it should just fail consistent with other jobs and then the errors in the job log can instead explain what's wrong and link to appropriate documentation to resolve the issue | 22:22 |
| *** gokrokve has joined #openstack-infra | 22:22 | |
| fungi | i think the long-form error message was instated early on when people had no idea what that job was nor why we were testing such a thing | 22:22 |
| *** pleia2 has quit IRC | 22:23 | |
| asselin | nibalizer, any idea what I'm doing wrong her? http://paste.openstack.org/show/329897/ | 22:24 |
| taron | clarkb: pasted a bunch at the end and nothing seems to have exploded yet | 22:24 |
| clarkb | taron: awesome thanks | 22:25 |
| taron | at least not on my end, anyway | 22:25 |
| asselin | fbo, ?? ^^ | 22:25 |
| *** pleia2 has joined #openstack-infra | 22:25 | |
| *** bknudson has quit IRC | 22:26 | |
| *** jsavak has joined #openstack-infra | 22:28 | |
| *** bhunter71 has quit IRC | 22:28 | |
| nibalizer | huh | 22:28 |
| nibalizer | asselin: uh oh did we pyhton | 22:29 |
| clarkb | for 37k rows with etherpad-dev the utf8 conversion took less than one second | 22:29 |
| lifeless | fungi: well, I guess I'm more curious about the technical undeprinnings | 22:29 |
| *** ayoung has joined #openstack-infra | 22:29 | |
| asselin | nibalizer, trying to validate those lines: $gerrit_ssh_ident = inline_template('<%= (@gerrit_ssh_host_identity).join(",") %> <%= @gerrit_ssh_host_key %>') | 22:29 |
| lifeless | fungi: like, why can't we say [FAILURE] [details...here] | 22:29 |
| nibalizer | ya | 22:29 |
| lifeless | fungi: and have the regex for the summary pull just the first bit out | 22:29 |
| zaro | is it just me or is review.o.o slow for others as well? | 22:30 |
| *** jgrimm is now known as zz_jgrimm | 22:31 | |
| *** mriedem has joined #openstack-infra | 22:32 | |
| fungi | lifeless: also possible. would be an update to http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/gerrit/hideci.js | 22:33 |
| nibalizer | asselin: so it works for me | 22:33 |
| clarkb | ok with etherpad-dev done, when is a good time to do etherpad.o.o? | 22:33 |
| fungi | lifeless: okay, bare-trusty images are fresh in all providers now | 22:33 |
| lifeless | fungi: (plus presumably a change to the reporting thign ?) | 22:33 |
| clarkb | fungi: pleia2 opinions ^ ? | 22:33 |
| nibalizer | http://paste.ubuntu.com/11801775/ | 22:33 |
| pleia2 | clarkb: if you're looking for a quiet window, with the US holiday coming up things will likely be very quiet on thursday | 22:34 |
| fungi | lifeless: yeah, adjustment to the failure message can be done at http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n946 | 22:34 |
| asselin | nibalizer, ok...will check what's different. thanks | 22:34 |
| fungi | clarkb: i'd say any time. rough estimate of how long you expect we'd need to have it offline? | 22:35 |
| openstackgerrit | Merged openstack/requirements: Bump tempest-lib version https://review.openstack.org/194951 | 22:35 |
| clarkb | fungi: let me grab a count of the rows on that db and figure 1 second per 37k rows | 22:35 |
| clarkb | plus overhead to switch | 22:35 |
| clarkb | pleia2: good point | 22:35 |
| *** soren has quit IRC | 22:36 | |
| *** rossella_s has quit IRC | 22:36 | |
| *** dontalton has quit IRC | 22:37 | |
| *** rossella_s has joined #openstack-infra | 22:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Bump etherpad version to version running on -dev https://review.openstack.org/197358 | 22:37 |
| *** ninag has joined #openstack-infra | 22:38 | |
| *** olaph has quit IRC | 22:38 | |
| clarkb | fungi: pleia2 ^ that is the first bit of this to make sure we have the correct version in place before restarting everything. Feel free to poke at -dev to test | 22:38 |
| *** DericHorn-HP has joined #openstack-infra | 22:38 | |
| greghaynes | clarkb: might be nice to put the upstream bug in that commit msg :) | 22:39 |
| greghaynes | (where is that btw?) | 22:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job https://review.openstack.org/197218 | 22:40 |
| clarkb | oh good point let me update | 22:40 |
| *** zz_jgrimm is now known as jgrimm | 22:40 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches https://review.openstack.org/197219 | 22:40 |
| *** ayoung has quit IRC | 22:41 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Bump etherpad version to version running on -dev https://review.openstack.org/197358 | 22:41 |
| clarkb | pleia2: fungi rough math based on a 37k rows per second is 317 seconds to do conversion | 22:42 |
| clarkb | so we should probably figure about half an hour to be generous for total downtime | 22:42 |
| *** jsavak has quit IRC | 22:42 | |
| *** jsavak has joined #openstack-infra | 22:43 | |
| *** julim has joined #openstack-infra | 22:43 | |
| pleia2 | haven't managed to break etherpad-dev | 22:43 |
| *** gokrokve has quit IRC | 22:44 | |
| greghaynes | https://etherpad-dev.openstack.org/p/%E2%98%83 works even :) | 22:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/196504 | 22:46 |
| clarkb | will anyone be around on friday? if so I am happy to do most of the work then just want to make sure I am the only root not outside in the sun celebrating FREEDOM | 22:47 |
| *** amuller has quit IRC | 22:47 | |
| clarkb | or on the other side of the dateline and otherwise weekending | 22:47 |
| pleia2 | I'm out on friday | 22:47 |
| greghaynes | I will be around for moral support | 22:47 |
| *** sabeen has quit IRC | 22:48 | |
| clarkb | I can also do thursday if that is just better | 22:48 |
| fungi | i have no friday plans | 22:48 |
| fungi | or thursday plans for that matter | 22:48 |
| fungi | i'm pretty boring | 22:49 |
| fungi | happy to drive it too if you have the commands worked out | 22:49 |
| greghaynes | fungi: youre not going to go enjoy all the neighbors setting off fireworks? :p | 22:49 |
| greghaynes | s/neighbors/tourists | 22:49 |
| clarkb | fungi: cool, let me write it up on etherpad-dev now (since etherpad itself will be down) | 22:49 |
| fungi | greghaynes: saturday we'll walk out to the beach and watch the municipalities a couple miles north and south of us set fireworks off from their piers | 22:50 |
| *** nelsnels_ has joined #openstack-infra | 22:50 | |
| asselin | nibalizer, do you think it makes sense to put that inline_template inside the if $gerrit_ssh_host_key != '' { ? | 22:50 |
| nibalizer | sure | 22:51 |
| fungi | but yes, every night for thw next couple of weeks will be nonstop tourists launching fireworks | 22:51 |
| greghaynes | heh | 22:51 |
| nibalizer | really i'd prefer to change the api to hard require those things | 22:51 |
| nibalizer | but i know moving to openstackci is going to be rough enough without doing that so I'm keeping my peace | 22:51 |
| asselin | what do you mean by hard-require? | 22:51 |
| asselin | remove the if? | 22:52 |
| *** sergmelikyan has quit IRC | 22:52 | |
| *** eharney has quit IRC | 22:53 | |
| openstackgerrit | Cody Herriges proposed openstack-infra/project-config: puppet: refactorize puppet-syntax job https://review.openstack.org/197216 | 22:53 |
| *** nelsnelson has quit IRC | 22:53 | |
| nibalizer | asselin: set it to $gerrit_ssh_host_key, in the class parameters | 22:53 |
| nibalizer | so that puppet will fail if you don't pass one in | 22:53 |
| openstackgerrit | Catherine Diep proposed openstack-infra/irc-meetings: Add agenda page URL to Refstack team meeting. https://review.openstack.org/197365 | 22:54 |
| *** larainema has quit IRC | 22:54 | |
| *** jsavak has quit IRC | 22:54 | |
| asselin | nibalizer, so get rid of the inline template and pass it in directly? That would be much simpler | 22:55 |
| asselin | nibalizer, and match what I was using before trying to migrate, so easy sell for me :) | 22:56 |
| *** otter768 has joined #openstack-infra | 22:56 | |
| clarkb | fungi: https://etherpad-dev.openstack.org/p/eplite-db-updates | 22:57 |
| asselin | nibalizer, also gerrit_ssh_host_key isn't used anywhereat all | 22:57 |
| fungi | clarkb: awesome--thanks!!! | 22:58 |
| clarkb | fungi: does midday friday (PDT) work for you? mornings have been pretty terrible for me lately but afternoons are much better | 22:59 |
| clarkb | I can send mail to the list about a Noon PDT (1900UTC iirc) etherpad downtime | 22:59 |
| fungi | clarkb: sounds great. and if you can't make it then, it's no big deal. the plan looks clean and dead simple | 23:00 |
| *** dims_ has quit IRC | 23:01 | |
| *** cindyo has joined #openstack-infra | 23:04 | |
| *** markvoelker_ has quit IRC | 23:06 | |
| *** ashleighfarnham has quit IRC | 23:06 | |
| *** ninag has quit IRC | 23:08 | |
| *** sarob_ has joined #openstack-infra | 23:08 | |
| *** ZZelle_ has quit IRC | 23:10 | |
| *** jsavak has joined #openstack-infra | 23:13 | |
| *** sarob has quit IRC | 23:14 | |
| *** mwagner_lap has quit IRC | 23:15 | |
| *** dimtruck is now known as zz_dimtruck | 23:15 | |
| clarkb | fungi: notice sent to dev list | 23:16 |
| *** sarob_ is now known as sarob | 23:16 | |
| *** mase_x200 has joined #openstack-infra | 23:16 | |
| pleia2 | thanks clarkb and fungi \o/ | 23:18 |
| *** hichihara has joined #openstack-infra | 23:20 | |
| *** sarob_ has joined #openstack-infra | 23:21 | |
| asselin | nibalizer, gave up and put in my -1.... | 23:21 |
| *** claudiub_ has joined #openstack-infra | 23:23 | |
| zaro | pabelanger: trystack seems broken, http://x86.trystack.org | 23:23 |
| *** krtaylor has quit IRC | 23:23 | |
| *** jsavak has quit IRC | 23:24 | |
| *** austin81 has left #openstack-infra | 23:25 | |
| *** claudiub has quit IRC | 23:25 | |
| nibalizer | asselin: okay | 23:25 |
| *** sarob_ has quit IRC | 23:27 | |
| asselin | nibalizer, what's your opinion? shall I upload another patch or hold off a bit? Really trying to get this done before the break.... | 23:27 |
| nibalizer | 'the break' ? | 23:27 |
| *** jsavak has joined #openstack-infra | 23:27 | |
| *** yamahata has joined #openstack-infra | 23:28 | |
| asselin | nibalizer, yes...july 4th....guess it doesn't mean much in UTC | 23:28 |
| nibalizer | oh right | 23:28 |
| nibalizer | um im not looking at that exact issue right now | 23:28 |
| nibalizer | ill try to circle around and look at it today/this evening | 23:28 |
| asselin | nibalizer, ok thanks | 23:29 |
| *** DericHorn-HP has quit IRC | 23:31 | |
| tchaypo | http://status.openstack.org/zuul/ gives a lovely overview of things in the queue right now; is there somewhere that shows me the history of jobs that have been run? | 23:34 |
| *** dannywilson has quit IRC | 23:35 | |
| *** jsavak has quit IRC | 23:35 | |
| *** puranamr has quit IRC | 23:36 | |
| *** mase_x200 has quit IRC | 23:37 | |
| *** dangers is now known as dangers_away | 23:37 | |
| *** puranamr has joined #openstack-infra | 23:37 | |
| clarkb | tchaypo: graphite for results over time, logstash if you need it mapped to other data over short term and gerrit for long term | 23:37 |
| tchaypo | http://logs.openstack.org seems to have some data too | 23:38 |
| *** Qiming has joined #openstack-infra | 23:39 | |
| *** cindyo has quit IRC | 23:39 | |
| tchaypo | in graphite, stats.zuul.pipeline.periodic-stable.job.periodic-tempest-dsvm-neutron-juno only shows SUCCESS and I’m trying to figure out why | 23:39 |
| clarkb | yes though the way it is structured and the vast quantity make it hard to consume for trend data | 23:39 |
| clarkb | hrm | 23:40 |
| clarkb | etherpad-dev's clarkb-test pad may have exploded | 23:40 |
| clarkb | the rest of the service continues to run though | 23:40 |
| *** _nadya_ has joined #openstack-infra | 23:40 | |
| clarkb | and we need to upgrade anyways so we should move forward with downtime, but what is going on here | 23:40 |
| clarkb | is https://etherpad-dev.openstack.org/p/clarkb-test working for anyone? I wonder if its my client | 23:41 |
| tchaypo | logs has data for periodic-tempest-dsvm-neutron-full-juno but not periodict-tempest-dsvm-neutron-juno which makes me think the latter is not a job we run any more | 23:41 |
| clarkb | there are no errors in the etherpad logs that I can see | 23:41 |
| *** krtaylor has joined #openstack-infra | 23:41 | |
| Clint | clarkb: i can see the chat | 23:41 |
| tchaypo | i can see the chat | 23:41 |
| clarkb | ya I see the chat but the content is perpetually loading | 23:41 |
| pleia2 | just Loading... | 23:41 |
| greghaynes | yep | 23:41 |
| greghaynes | same | 23:42 |
| clarkb | woo | 23:42 |
| pleia2 | etherpad is the worst :( | 23:42 |
| tchaypo | same | 23:42 |
| clarkb | so anyways we should do the db move on friday but not call the issue fixed fixed until some other time | 23:42 |
| pleia2 | +1 | 23:42 |
| *** cindyo has joined #openstack-infra | 23:42 | |
| clarkb | I bet what happens is the encoding change allows us to write the bytes to mysql, but etherpad lite then fails to pull them out properly | 23:42 |
| Clint | you put a 4-byte char in? | 23:43 |
| *** dannywilson has joined #openstack-infra | 23:43 | |
| clarkb | taron did | 23:44 |
| Clint | and the json in the db is sane? | 23:44 |
| clarkb | I am looking at it with chromium dev tools and the content response looks funky | 23:44 |
| *** _nadya_ has quit IRC | 23:45 | |
| *** dims has joined #openstack-infra | 23:45 | |
| *** dims has quit IRC | 23:46 | |
| greghaynes | clarkb: yea, I think your hypothesis is plausible based on error I see | 23:46 |
| greghaynes | seeing Error: Failed assertion: Invalid changeset (checkRep failed in console | 23:47 |
| *** markvoelker has joined #openstack-infra | 23:47 | |
| greghaynes | which googling leads me to a bug report on an encoding issue | 23:47 |
| taron | interesting that the top bar disappeared as well | 23:47 |
| clarkb | this is an improvement in that the other pads served by the server continue to work | 23:48 |
| clarkb | rather than the whole thing dying in a loop | 23:48 |
| Clint | it sure is | 23:48 |
| *** teran has quit IRC | 23:48 | |
| greghaynes | not that it isnt hilarious to watch etherpad do that | 23:49 |
| clarkb | and is that happening in the js vm? | 23:49 |
| clarkb | socket.io fails due to the encoding issue | 23:50 |
| greghaynes | clarkb: wah? | 23:50 |
| clarkb | oh nevermind I was reading too much into the filename | 23:51 |
| *** marun has quit IRC | 23:51 | |
| clarkb | comments indicate "this file is mostly from the old etherpad" (gread) | 23:51 |
| clarkb | exports.pack takes a string encoding | 23:53 |
| clarkb | so that must not be correct or the source data is being munged | 23:53 |
| *** sputnik1_ has joined #openstack-infra | 23:54 | |
| *** dannywilson has quit IRC | 23:54 | |
| *** yamada-h has joined #openstack-infra | 23:54 | |
| *** dannywilson has joined #openstack-infra | 23:54 | |
| openstackgerrit | Emma Barber proposed openstack-infra/project-config: Add hound mirror https://review.openstack.org/197376 | 23:55 |
| *** dboik has joined #openstack-infra | 23:56 | |
| clarkb | I am going to update the upstream bug now but this is beyond my ability to debug on little sleep | 23:56 |
| clarkb | fungi: ^ but we should consider not changing the encoding until this is fixed and just do the db upgrade on friday | 23:56 |
| *** sputnik13 has quit IRC | 23:57 | |
| *** yamada-h has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!