Tuesday, 2015-06-30

openstackgerritMerged openstack-infra/puppet-subunit2sql: Make unhandled exceptions not fatal  https://review.openstack.org/19414600:00
*** dannywilson has quit IRC00:00
*** DericHorn-HP has quit IRC00:01
*** dannywil_ has quit IRC00:01
*** ashleighfarnham has quit IRC00:02
*** docaedo has quit IRC00:02
*** Longgeek has quit IRC00:03
*** amitgandhinz has joined #openstack-infra00:03
jheskethlifeless, fungi: looking00:04
*** docaedo has joined #openstack-infra00:04
*** Rockyg has quit IRC00:05
*** sdake has quit IRC00:07
*** tqtran-afk is now known as tqtran_00:07
*** xarses has joined #openstack-infra00:07
*** fallenpegasus has joined #openstack-infra00:07
*** amitgandhinz has quit IRC00:07
jheskethfungi: reviewed, but I think I might be missing something00:08
pleia2my turn for the dentist tomorrow, so I'll be strolling in late (will be back by meeting time)00:09
*** dboik has joined #openstack-infra00:11
*** dims has joined #openstack-infra00:12
*** sputnik13 has quit IRC00:12
*** austin81 has joined #openstack-infra00:14
*** dboik has quit IRC00:15
wshaoon moving project from stackforget to openstack, do I submit change to openstack/governance first? the project creator page indicates that I submit changes to infra/project-config first? but I have seen others submit to governance first.00:15
*** rm_work is now known as rm_work|away00:15
jheskethpleia2: blargh, dentists, don't remind me :-(00:16
*** mtanino has quit IRC00:16
pleia2just a checkup, hopefully all is well :)00:18
*** marun has joined #openstack-infra00:18
fungiwshao: if you're moving a repo from stackforge to openstack and its adopted by an official project-team, you can submit the change to project-config first00:18
*** rlandy has quit IRC00:18
fungiwshao: if you're applying to the tc to create a new official project-team, you need to submit a change to governance first so the application for forming a new openstack project-team can be discussed by the tc00:19
wshaofungi: project-team already exsits from stackforge projects.00:20
*** claudiub has joined #openstack-infra00:21
*** esmute__ has quit IRC00:23
openstackgerritWeidong Shao proposed openstack-infra/project-config: Add compas to OpenStack projects.  https://review.openstack.org/19691600:24
*** wshao has quit IRC00:24
fungiwshao: then submit the rename change to project-config (and link it in the renames section of the infra meeting agenda so we can schedule it). also submit a change to update the governance repo so it stays current00:24
*** amotoki has joined #openstack-infra00:27
*** ildikov_ has quit IRC00:28
*** austin81 has quit IRC00:29
*** boris-42 has joined #openstack-infra00:30
*** yamamoto has joined #openstack-infra00:31
*** chlong has joined #openstack-infra00:32
*** jsavak has quit IRC00:32
*** geoffarnold has quit IRC00:32
*** ddieterly has joined #openstack-infra00:33
*** marun_ has joined #openstack-infra00:40
davideagnellohello, this review is pending one more +2:  https://review.openstack.org/#/c/195991/00:40
*** ildikov_ has joined #openstack-infra00:41
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Switch constraints proposal to use zuul-git-prep  https://review.openstack.org/19690700:41
*** marun has quit IRC00:41
fungijhesketh: i think that ^ covers the missing bit00:41
jheskethfungi: oh, I see, it's a jjb replacement!00:42
fungiyeah, waiting to see if the xml diff job does what i think it should this time00:43
*** fallenpegasus has quit IRC00:45
fungithough after this, i realize that we have a branch-git-prep macro which basically does this with the old prep script. still, i want a zuul-git-prep macro anyway00:48
fungiso that maybe we can eventually get rid of the old script00:48
*** gmann has quit IRC00:48
*** thorst has joined #openstack-infra00:48
*** notnownikki has quit IRC00:50
tchaypofungi: lifeless: actually, both versions of the code in setuptools were okay - the problem was that one version of the code in setuptools is pkg_resources, which is vendored into pip, but hadn’t been updated in Quite Some Time.00:53
*** patrickeast has quit IRC00:53
*** abitha has quit IRC00:54
*** larainema has joined #openstack-infra00:54
tchaypoNakato’s PR https://github.com/pypa/pip/pull/2894 updates pkg_resources in pip to the version from setuptools 17.100:54
tchayposo as soon as pip gets a 7.1….00:55
Nakato18.0 got pulled in instead00:56
tchaypowhich sounds like an equally acceptable fix00:56
fungiahh, yep we can probably update our requirements lists to use inequality matches at that point, sounds like00:57
*** ayoung has joined #openstack-infra00:58
*** yamamoto has quit IRC00:59
openstackgerritWayne Warren proposed openstack-infra/infra-specs: Zuul github-integration spec.  https://review.openstack.org/16363700:59
waynrjhesketh: ^^ addressed comments in patchset 6 for the zull_github-integration spec01:00
*** gmann has joined #openstack-infra01:00
*** marun_ has quit IRC01:01
*** annegentle has joined #openstack-infra01:02
*** otter768 has joined #openstack-infra01:03
*** ashleighfarnham has joined #openstack-infra01:03
waynrI don't understand the difference between "Rollcall-Vote", "Council-Vote", and probably others I have seen01:05
waynrIs there a good place to read up on these?01:06
dkehnlifeless: interesting, I'm actually thinking its the setup of the venv in the devstack01:06
*** annegentle has quit IRC01:06
lifelessdkehn: can has log or pastebin or something ?01:06
*** otter768 has quit IRC01:07
*** ashleighfarnham has quit IRC01:07
dkehnlifeless: sure but its an aboration of devstack. kind of working on something different, but leveraging the oslo and devestack01:08
lifelessdkehn: ah ok01:09
lifelessdkehn: so in which case you need to copy the code added to devstack recently to make a venv and pip install requirements01:09
lifelessdkehn: without consulting requirements to do so (circular logic :))01:09
dkehnlifeless: http://paste.openstack.org/show/326551/01:10
dkehnlifeless: yep, figured as much. just didn't realize that they were installing into venv env01:10
*** ildikov_ has quit IRC01:11
lifelessdkehn: they weren't... :)01:11
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Switch constraints proposal to use zuul-git-prep  https://review.openstack.org/19690701:13
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add a zuul-git-prep builder macro  https://review.openstack.org/19690601:13
*** sigmavirus24 is now known as sigmavirus24_awa01:13
dkehnlifeless: ? devstack Install Python packages into a virtualenv so that we can track them01:13
lifelessdkehn: nope01:13
lifelessdkehn: gimme a sec, looking up ref for you01:14
dkehnlifeless: ok, I'm not following that the call to python tht call update.py is premissed of the venv01:14
lifelesslook at 7ebe8e0751dab545091e0b114589087009cc4e22 and 40f3e33f734330c32f27882898ab1bb4ab8f3217 in devstack01:14
lifeless40f3 was first, and slightly buggy01:15
dkehnlifeless: ok, https://github.com/openstack-dev/devstack/blob/master/inc/python#L22101:15
lifelessdkehn: so yes, thats calling it using the python from the venv01:16
lifelessdkehn: the commits I linked just before should be self explanatory01:16
dkehnlifeless: ok, the crux of the issue that when running update.py of the pastebin it can path its imports which caused me to start looking at what changed01:17
dkehnlifeless: looking at the commits, yes I see01:18
*** markvoelker has quit IRC01:19
*** ivar-laz_ has joined #openstack-infra01:19
*** markvoelker has joined #openstack-infra01:19
*** markvoelker has quit IRC01:22
*** ivar-lazzaro has quit IRC01:22
*** markvoelker has joined #openstack-infra01:23
*** markvoelker has quit IRC01:23
*** ivar-laz_ has quit IRC01:23
*** markvoelker has joined #openstack-infra01:24
*** DericHorn-HP has joined #openstack-infra01:24
*** enikanorov2 has quit IRC01:24
*** otter768 has joined #openstack-infra01:27
*** jasondotstar has quit IRC01:30
*** Sukhdev has quit IRC01:34
*** laughterwym_ has joined #openstack-infra01:34
*** laughterwym has quit IRC01:34
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Drop use of 'oslo' namespace package  https://review.openstack.org/19687201:36
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Switch constraints proposal to use zuul-git-prep  https://review.openstack.org/19690701:37
*** wenlock has joined #openstack-infra01:39
*** enikanorov2 has joined #openstack-infra01:42
*** yamahata has quit IRC01:43
*** RichardRaseley has joined #openstack-infra01:44
openstackgerritMerged openstack/requirements: keystoneclient: require at least >= 1.2.0 for stable/kilo  https://review.openstack.org/19688701:45
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Drop use of 'oslo' namespace package  https://review.openstack.org/19687201:46
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Set temporary version cap on oslo.db  https://review.openstack.org/19692401:46
*** esmute__ has joined #openstack-infra01:48
*** tqtran_ has quit IRC01:49
*** RichardRaseley has quit IRC01:50
*** yamamoto has joined #openstack-infra01:52
*** gyee has quit IRC01:53
*** koolhead17 has joined #openstack-infra01:54
*** otter768 has quit IRC01:55
*** wenlock has quit IRC01:59
*** jasondotstar has joined #openstack-infra01:59
*** EricGonczer_ has joined #openstack-infra01:59
*** jasondotstar has quit IRC02:00
*** jasondotstar has joined #openstack-infra02:00
*** sigmavirus24_awa is now known as sigmavirus2402:05
*** jasondotstar has quit IRC02:06
*** jasondotstar has joined #openstack-infra02:06
openstackgerritMerged openstack-infra/subunit2sql: Set temporary version cap on oslo.db  https://review.openstack.org/19692402:06
openstackgerritMerged openstack-infra/subunit2sql: Drop use of 'oslo' namespace package  https://review.openstack.org/19687202:07
*** ayoung has quit IRC02:07
fungijhesketh: looks like i've finally worked out how to get the replacement passed through to the builder macro http://logs.openstack.org/07/196907/4/check/project-config-compare-xml/91e8561/console.html#_2015-06-30_01_46_41_43402:08
*** chlong_ has joined #openstack-infra02:08
*** ayoung has joined #openstack-infra02:09
mtreinishjeblair, greghaynes: ^^^ I'll push the release now to include those fixes for the latest oslo releases02:09
*** ianw has quit IRC02:09
*** chlong__ has joined #openstack-infra02:10
*** chlong has quit IRC02:10
*** camunoz has quit IRC02:10
*** camunoz has joined #openstack-infra02:10
*** ianw has joined #openstack-infra02:10
fungilifeless: "Requirement xxx present in multiple files" http://logs.openstack.org/76/193976/9/check/gate-ceilometer-requirements/cfb368d/console.html02:11
*** chlong_ has quit IRC02:13
*** mkj_wk has quit IRC02:14
*** esmute__ has quit IRC02:15
*** mriedem_away has quit IRC02:16
*** mkj_wk has joined #openstack-infra02:17
mtreinishjeblair, greghaynes: https://pypi.python.org/pypi/subunit2sql/0.7.0 should fix this week's subunit2sql x oslo issue02:17
*** weshay has quit IRC02:17
*** unicell has quit IRC02:19
*** bhunter71 has joined #openstack-infra02:26
*** bhunter71 has quit IRC02:26
*** esmute__ has joined #openstack-infra02:27
*** tlian2 has quit IRC02:27
*** EricGonczer_ has quit IRC02:27
*** ashleighfarnham has joined #openstack-infra02:27
*** camunoz_ has joined #openstack-infra02:28
*** dangers_away has quit IRC02:28
*** asselin has quit IRC02:28
*** chlong_ has joined #openstack-infra02:28
*** ianw has quit IRC02:29
*** asselin has joined #openstack-infra02:29
*** dangers_away has joined #openstack-infra02:29
*** camunoz__ has joined #openstack-infra02:30
*** homeless has quit IRC02:30
*** homeless has joined #openstack-infra02:31
*** chlong__ has quit IRC02:31
*** chlong has joined #openstack-infra02:31
*** camunoz has quit IRC02:31
*** baoli has joined #openstack-infra02:31
*** puranamr has quit IRC02:33
openstackgerritlifeless proposed openstack/requirements: Require fixtures 1.3.1.  https://review.openstack.org/19686302:34
*** camunoz_ has quit IRC02:34
*** chlong_ has quit IRC02:34
lifelessfungi: looking02:36
*** rossella_s has quit IRC02:36
*** rossella_s has joined #openstack-infra02:37
lifelessfungi: its correct02:38
lifelesstest-requirements-py3.txt:29:gabbi>=0.12.0  # Apache-2.002:38
lifelesstest-requirements.txt:32:gabbi>=0.12.0  # Apache-2.002:38
lifelessfungi: this isn't actually what I intended to happen, and I'll fix it, because of stable branches.02:38
lifelessfungi: but hey, its a LOVELY demonstration of the DRY violation those files were :)02:38
fungilifeless: oh, great point. i didn't spot that was being triggered by having the version-specific reqs lists02:39
lifelessin fact02:39
lifelessI wonder02:39
lifelessthe old infra code wasn't parsing those files AT ALL02:40
lifelesshur hur hur02:40
SlickNiksdague: When you get a chance can I get some eyeballs on https://review.openstack.org/#/c/192939/, please? Thanks!02:41
fungilifeless: that doesn't surprise me. we wrote it before projects started working around version-specific reqs by adding those extra lists on their own accord02:42
lifelessfungi: well pbr supported those lists02:42
lifelessfungi: its just a reflection of having separated codebases doing the same thing :)02:43
fungii think that came later and the requirements check was never updated with similar logic02:43
fungiyes, dry fails again02:43
lifelessI'm going to have to think somewhat carefully about how to address this02:43
fungilooks like nova has the same issue02:43
lifelessin the meantime, unified files FTW02:43
*** tlian has joined #openstack-infra02:44
*** mriedem has joined #openstack-infra02:45
openstackgerritMerged openstack-infra/subunit2sql: Fix misspelled words in some docs  https://review.openstack.org/19629502:46
lifelessfungi: https://review.openstack.org/#/c/196930/ - putting that up the flagpole02:48
*** fbo_ has quit IRC02:48
*** mmedvede has quit IRC02:50
fungiugh. it seems like while i wasn't looking, basically all the projects grew requirements-py3.txt files02:50
fungithat was supposed to be an ugly hack in one or two libraries02:50
*** dimtruck is now known as zz_dimtruck02:50
*** fbo has joined #openstack-infra02:51
mordredfungi, Shrews, SpamapS, dguerri`:https://review.openstack.org/#/c/168792 turns out jamielennox already had written the ksc patch we need02:54
fungimordred: he's a stealthy ninja02:54
mordredfungi: I made the mistake of handing him a bunny rabbit02:54
fungihe writes the code you need before you even know you want it02:54
fungianyway, i'm going to sleep02:57
fungimore hacking on the dawn02:57
* fungi zzzzzz02:57
greghaynesAt dawn, we hack.02:58
mordredgreghaynes: ++02:58
*** koolhead17 has quit IRC02:59
mordredgreghaynes: sorry - I was AFK mostly since last we spoke - any luck on the swiftclient hack?02:59
greghaynesoh, no, I got a test case set up then got distracted by HP NOC stuff02:59
*** Longgeek has joined #openstack-infra03:00
*** laughterwym has joined #openstack-infra03:02
*** laughterwym_ has quit IRC03:02
*** Longgeek has quit IRC03:05
greghaynesmordred: Is there a pattern in testtools or testr for skipping a test by default but allowing it to be run individually?03:05
greghaynesI want to add my rax + swift testcase with that setup03:06
lifelessfungi: yeah03:06
*** homeless has quit IRC03:06
lifelessfungi: I've had to push a little hard on halting the rot, it was going to get even worse03:06
greghayneslifeless: ^ you would know03:06
SpamapSdrives me nuts when I use --debug, and it feeds me a curl command that will never ever work03:07
SpamapSsha1 of the token.... very.. useful?03:07
lifelessgreghaynes: skipIf(os.environ.get('OS_SOME_FLAG')=='1', 'Not enabled by OS_SOME_FLAG')03:07
greghayneslifeless: ah! ty03:07
lifelessor !=03:07
lifelessor whateves03:07
greghaynesyea, I got the pattern ;)03:08
*** kevinbenton has joined #openstack-infra03:08
lifelessfungi: in nova03:09
lifelessfungi: do this03:09
lifelessdiff -u test-requirements{,-py3}.txt03:09
lifelessfungi: whats the PyMySQL status there ?03:09
lifelessfungi: e.g. do I union the MySQL-python thing03:09
lifelessor ...03:09
*** woodster_ has quit IRC03:11
*** Guest26658 has quit IRC03:12
*** flashgordon has quit IRC03:13
lifelessfungi: or nvm, since I had an old tree apparently :)03:14
lifelessI'm starting over03:14
*** claudiub has quit IRC03:15
*** tlian has quit IRC03:15
greghaynesmordred: hrm, my test case doesnt seem to fail in the same way - im doing a clouds.yaml with profile: rackspace and specifying project_id, username, pass, region03:16
greghaynesand AFAICT it 'just works'03:16
*** baoli has quit IRC03:18
lifelessfungi: https://review.openstack.org/19693403:19
lifelessnow, to release unittest2. muahahahahahahahaha03:19
openstackgerritgreghaynes proposed openstack-infra/shade: WIP test case for rax image uploads  https://review.openstack.org/19693603:26
greghaynesfungi: mordred yea, im somewhat confused ^ with a clouds.yaml that looks just like ours in system-config doesnt fail for me03:26
openstackgerritlifeless proposed openstack/requirements: Require fixtures 1.3.1.  https://review.openstack.org/19686303:28
*** masayukig has quit IRC03:31
*** markvoelker has quit IRC03:31
*** camunoz has joined #openstack-infra03:31
*** camunoz__ has quit IRC03:33
openstackgerritlifeless proposed openstack-infra/project-config: Don't fail on dupes in version specific files  https://review.openstack.org/19694003:33
*** xingchao has quit IRC03:33
lifelessfungi: ^this might be zomg urgent to unwedge nova03:34
lifelessmordred: ^03:34
lifelessjhesketh: ^03:34
lifelessmriedem: ^ FYI03:34
lifeless[btw untested]03:34
*** yamada-h has joined #openstack-infra03:35
*** masayukig has joined #openstack-infra03:36
mriedemlifeless: i've added the depends-on so we'll see03:36
mriedemthat's like 5 patches now to unwedge nova :)03:36
*** thorst has quit IRC03:36
lifelessmriedem: I don't think depends-on can help you with that03:36
mriedemoh right b/c it's project-config03:37
mriedemwell, at least it'll be an indication that we need it for when people are trying to figure out why the requirements job in the nova mega change is failing03:37
*** Longgeek has joined #openstack-infra03:37
*** yamada-h has quit IRC03:40
*** yamahata has joined #openstack-infra03:41
*** fallenpegasus has joined #openstack-infra03:41
*** ddieterly has quit IRC03:43
*** laughterwym_ has joined #openstack-infra03:44
*** laughterwym has quit IRC03:44
*** laughterwym__ has joined #openstack-infra03:44
jheskethlifeless: looking03:45
lifelessits needed because the trunk version of nova etc has the defect03:46
*** jasondotstar has quit IRC03:47
lifelessAJaeger: if you're awake ^03:47
lifelessI don't know if a new release is needed ..03:47
lifelessnew image03:47
lifelessfungi: ^?03:47
*** dims has quit IRC03:48
jheskethlifeless: I'm trying to understand the problem as I check your change.. could you help explain to me what checking for -py is doing?03:52
jheskethlifeless: are we excluding test-requirements-py3.txt from being checked?03:53
*** mriedem has quit IRC03:53
lifelesswe're excluding it from triggering an error03:53
*** sarob has quit IRC03:54
lifelessjhesketh: we want to tell folk about the problem no matter what03:54
lifelessthats the move of the print03:54
lifelessand we want to not fail if the problem occurs in a py2 or py3 file03:54
*** ianw has joined #openstack-infra03:54
lifelessbecause the old infra code didn't check that at all03:54
*** otter768 has joined #openstack-infra03:56
jheskethlifeless: hmm okay, just checking on -py is very loose. Should we make that tighter or leave a comment do you think?03:57
lifelessjhesketh: all the version specific requirements files are named -pyN.txt03:57
lifelessjhesketh: and none of the other requirements files have -py in their name03:57
lifelessjhesketh: so its actually quite specific03:58
lifelessjhesketh: but I can add a comment if you like03:58
jheskethI'm mostly satisfied, just wondering if somebody would ever innocently add requirements-pyproject.txt or something and have it excluded03:58
jheskeththat'd be pretty odd though03:59
lifelessthey'd have to get it through pbr review first03:59
lifelessand then global-requirements review03:59
lifelessand our goal is to remove all these requirements files03:59
lifelessso I'm thinking we're fairly safe03:59
jheskethright :-)03:59
jheskethlifeless: so, excuse my ignorance, would the correct fix be to make sure the requirement is only listed once in one of the version specific reqs files?04:00
jheskethie maybe we should determine the version and just check those strictly04:00
lifelessjhesketh: hysterical raisins appplies here04:00
lifelesswe can't without considerable effort use markers on kilo and juno04:00
lifelessthe -pyN files are not additive, they replace the other file. (Yes this is silly. No, it wasn't my fault).04:01
*** otter768 has quit IRC04:01
lifelessso when you have a -pyN file, its going to overlap 99% with the other file04:01
lifelesswhatever that may be04:01
jheskethhmm okay04:01
jheskethso how come this is blocking nova all of a sudden, what changed?04:01
lifelessthe prior infra parser didnt' check the pyN files at all.04:02
lifelessjhesketh: look at the diff in b7a7641677250c8316bddecd5c9368143373620904:02
jheskethlifeless: I can't find that sha, which project?04:03
jheskethah, sorry, was looking in reqs and nova04:04
jheskethlifeless: okay, I'm satisifed, thanks for helping explain it :-)04:05
jheskethlifeless: if no other infra pops up soon I can push that change through if it's causing pain for nova04:05
lifelessits blocking them unwedging their gate04:05
*** Longgeek has quit IRC04:05
lifelessassuming my fix is solid04:06
lifelessI'm going to cross check that in a sec04:06
*** Longgeek has joined #openstack-infra04:06
jheskethlifeless: I've +A'd it, I'll be around if we need to undo it or do further fixes04:06
*** coolsvap|away is now known as coolsvap04:08
lifelessjhesketh: ack04:09
*** sdake has joined #openstack-infra04:09
*** sdake_ has joined #openstack-infra04:10
lifelessjhesketh: one thing I don't know04:10
lifelessdo we run that from the image04:10
lifelessor from a clone04:10
lifelesse.g. I suspect we need a new image with it on it too04:10
jheskethlifeless: yeah the scripts are seeded to the images (afaik)04:11
jheskethso we might need to rebuild04:11
jheskethI can kick that off once it merges04:11
*** unicell has joined #openstack-infra04:12
*** david-lyle has quit IRC04:13
openstackgerritMerged openstack-infra/project-config: Don't fail on dupes in version specific files  https://review.openstack.org/19694004:13
*** achanda has joined #openstack-infra04:13
*** sdake has quit IRC04:13
*** david-lyle has joined #openstack-infra04:14
*** zz_ja has quit IRC04:14
*** zz_jgrimm has quit IRC04:15
*** maurosr has quit IRC04:15
*** asselin_ has joined #openstack-infra04:17
jheskethlifeless: okay, rebuilding bare-trusty nodes04:19
*** kmartin_ has joined #openstack-infra04:19
*** kmartin has quit IRC04:19
*** unicell1 has joined #openstack-infra04:22
*** unicell has quit IRC04:23
*** zz_ja has joined #openstack-infra04:24
*** ihrachyshka has joined #openstack-infra04:24
*** ihrachyshka has quit IRC04:25
*** ihrachyshka_ has joined #openstack-infra04:25
*** sdake has joined #openstack-infra04:25
*** zz_jgrimm has joined #openstack-infra04:26
*** maurosr has joined #openstack-infra04:27
ihrachyshka_any project-config cores? I have real need for someone to check https://review.openstack.org/#/c/196718/ to get testing up for qos neutron sprint today... clarkb, mordred, fungi sdague, anteaya, jhesketh maybe?04:27
jheskethihrachyshka_: looking04:27
ihrachyshka_thanks! the idea is to modify neutron.conf a bit for specific branch to get a service plugin that is in developement to be enabled.04:28
*** DericHorn-HP has quit IRC04:28
ihrachyshka_the stub of the plugin is already in the tree04:28
*** sdake_ has quit IRC04:29
*** achanda has quit IRC04:30
*** hogepodge has quit IRC04:32
*** markvoelker has joined #openstack-infra04:32
jheskethihrachyshka_: lgtm04:32
*** hogepodge has joined #openstack-infra04:32
ihrachyshka_jhesketh, thanks a lot! those branch envs are a bit of mystery for me, I copied them from some other place. Hopefully it will work as I expect. :)04:33
jlvillaltonyb, Any look getting some love for the eavesdrop 'calendars' patch?04:34
*** DericHorn-HP has joined #openstack-infra04:35
*** maishsk has quit IRC04:35
*** yamada-h has joined #openstack-infra04:36
*** markvoelker has quit IRC04:36
*** maishsk has joined #openstack-infra04:37
*** sigmavirus24 is now known as sigmavirus24_awa04:39
*** zz_jgrimm has quit IRC04:39
*** laughterwym__ has quit IRC04:40
*** zz_ja has quit IRC04:40
*** maurosr has quit IRC04:40
*** yamada-h has quit IRC04:41
*** wshao has joined #openstack-infra04:41
*** zz_jgrimm has joined #openstack-infra04:42
*** maurosr has joined #openstack-infra04:42
*** zz_ja has joined #openstack-infra04:43
*** ddieterly has joined #openstack-infra04:44
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Run diskimage-builder-install macro for akanda image build  https://review.openstack.org/19694604:45
*** maishsk has quit IRC04:47
*** esmute__ has quit IRC04:47
*** yfried|afk has quit IRC04:48
*** ddieterly has quit IRC04:49
*** amotoki_ has joined #openstack-infra04:49
*** laughterwym_ has quit IRC04:51
*** kiran-r has joined #openstack-infra04:52
*** rm_work|away is now known as rm_work04:52
tonybjlvillal: no not yet.  I/we need to work out if there is a delete part of the publish job.  That'll become important over time.04:53
*** DericHorn-HP has quit IRC04:58
*** sdake_ has joined #openstack-infra04:59
*** kiran-r has quit IRC05:00
*** erlon has quit IRC05:00
*** kiran-r has joined #openstack-infra05:03
*** sdake has quit IRC05:03
*** fallenpegasus has quit IRC05:03
*** sputnik13 has joined #openstack-infra05:04
*** tjones1 has joined #openstack-infra05:04
*** fallenpegasus has joined #openstack-infra05:07
jlvillaltonyb, Thanks for the info :)05:09
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Update the run_time graph image in docs  https://review.openstack.org/19695205:10
*** ihrachyshka_ has quit IRC05:13
jheskethlifeless: images are rebuilt, just need to wait for some nodes to launch with the new images05:14
*** kiran-r has quit IRC05:17
*** fawadkhaliq has joined #openstack-infra05:18
*** kiran-r has joined #openstack-infra05:18
*** ildikov has joined #openstack-infra05:22
*** _kiran_ has joined #openstack-infra05:23
*** fallenpegasus has quit IRC05:23
*** amotoki__ has joined #openstack-infra05:24
*** kiran-r has quit IRC05:24
*** amotoki_ has quit IRC05:24
*** fallenpegasus has joined #openstack-infra05:27
*** _kiran_ is now known as kiran-r05:27
*** Sukhdev has joined #openstack-infra05:28
*** abregman has joined #openstack-infra05:37
*** BharatK has joined #openstack-infra05:39
openstackgerritWeidong Shao proposed openstack-infra/project-config: Add compass to OpenStack projects  https://review.openstack.org/19691605:42
*** asettle has joined #openstack-infra05:42
*** hogepodge has quit IRC05:42
*** hogepodge has joined #openstack-infra05:42
*** tjones1 has quit IRC05:44
*** sputnik1_ has joined #openstack-infra05:45
*** ddieterly has joined #openstack-infra05:45
*** abregman has quit IRC05:46
*** kiran-r has quit IRC05:46
*** sputnik13 has quit IRC05:48
*** achanda has joined #openstack-infra05:48
lifelessjhesketh: cool05:50
jheskethlifeless: hmm, looks like a lot of jobs are failing though05:50
*** ddieterly has quit IRC05:50
jheskethjust trying to figure out if it's the new nodes or something else05:50
*** deepakcs has joined #openstack-infra05:51
lifelessjhesketh: ack05:52
jheskethlooks like it's failing on more than just bare-trusty05:53
lifelessjhesketh: got a log or something?05:54
*** yamamoto has quit IRC05:55
jheskethlifeless: there's a bunch of unstable jobs in the queue https://jenkins02.openstack.org/job/gate-nova-python27/835/console05:55
*** fallenpegasus has quit IRC05:55
jheskeththey all appear to have trouble uploading logs05:55
*** markvoelker has joined #openstack-infra05:56
*** otter768 has joined #openstack-infra05:57
lifelessI can't tell if the job ran with the new code or not05:57
lifelessfor https://jenkins01.openstack.org/job/gate-nova-requirements/37/console05:58
*** yamamoto has joined #openstack-infra05:58
jheskethlifeless: me either, but I don't think that problem is related to the others which is more urgent at the moment05:58
lifelessthe /usr/zuul-swift-logs-env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true05:59
lifelesslines are noise05:59
*** markvoelker has quit IRC06:01
lifelessjhesketh: ah higher up06:01
*** wshao has quit IRC06:01
*** otter768 has quit IRC06:02
jheskethlifeless: sorry? what are you looking at?06:02
*** abregman has joined #openstack-infra06:02
*** wshao has joined #openstack-infra06:02
lifeless2015-06-30 05:44:40.802 | 2: No such file06:03
lifeless2015-06-30 05:44:40.802 | at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2833)06:03
lifeless2015-06-30 05:44:40.803 | at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2185)06:03
lifeless2015-06-30 05:44:40.803 | at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2202)06:03
lifeless2015-06-30 05:44:40.803 | at com.jcraft.jsch.ChannelSftp.stat(ChannelSftp.java:2159)06:03
lifeless2015-06-30 05:44:40.803 | at be.certipost.hudson.plugin.SCPSite.mkdirs(SCPSite.java:314)06:03
lifelessjhesketh: from https://jenkins02.openstack.org/job/gate-nova-python27/835/console06:03
jheskethright, we're seeing a lot of that06:03
*** achanda has quit IRC06:04
wshaoon openstack infra access to irc: do I need to set up permission somewhere? I got irc-access check error. openstackinfra does not have permissions on #openstack-compass:06:05
jheskethhmm, I don't have the passwords to jenkins to check the creds :-(06:05
*** fallenpegasus has joined #openstack-infra06:05
jheskeththe disk certainly has space and other logs exist06:06
*** ihrachyshka_ has joined #openstack-infra06:07
*** Sukhdev_ has joined #openstack-infra06:08
lifelessjhesketh: they'll be in hiera?06:09
*** Sukhdev has quit IRC06:11
*** k4n0 has joined #openstack-infra06:12
*** ParsectiX has joined #openstack-infra06:16
*** larainema has quit IRC06:17
*** maishsk_afk has joined #openstack-infra06:17
*** maishsk_afk is now known as maishsk06:17
*** wshao has quit IRC06:18
*** _nadya_ has joined #openstack-infra06:23
*** yfried|afk has joined #openstack-infra06:24
*** achanda has joined #openstack-infra06:26
*** Qiming has joined #openstack-infra06:28
*** yfried has joined #openstack-infra06:29
*** yfried|afk has quit IRC06:29
*** sdake_ has quit IRC06:29
*** mpaolino has joined #openstack-infra06:30
openstackgerritRomil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under openstack namespace  https://review.openstack.org/19695806:33
*** scheuran has joined #openstack-infra06:33
AJaegerfungi, clarkb, pleia2 : All the translation jobs fail in the "git review" step. Error is "fatal: ambiguous argument '(detached': unknown revision or path not in the working tree."06:35
AJaegerfungi, is this a problem due to changing to the new proposal slave?06:35
AJaegerfungi, clarkb, pleia2: log file at https://jenkins.openstack.org/job/nova-propose-translation-update/55/console06:35
openstackgerritWayne Warren proposed openstack-infra/infra-specs: Zuul github-integration spec.  https://review.openstack.org/16363706:36
*** AzherKhan has joined #openstack-infra06:36
*** rossella_s has quit IRC06:36
AJaegerfungi, and a different error on https://jenkins.openstack.org/job/horizon-propose-translation-update/55/console06:37
*** rossella_s has joined #openstack-infra06:37
AJaegerfungi: upstream-translation-update looks fine, indeed.06:38
*** asettle has quit IRC06:39
jheskethlifeless: do you know where in Jenkins scp dests are set?06:43
openstackgerritRomil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under OpenStack namespace  https://review.openstack.org/19695806:44
*** ddieterly has joined #openstack-infra06:46
*** Sukhdev_ has quit IRC06:47
*** mrunge has joined #openstack-infra06:48
*** mrunge has quit IRC06:48
*** mrunge has joined #openstack-infra06:48
lifelessjhesketh: I don't sorry06:49
lifelessjhesketh: I can help you poke around for that once C is in bed06:49
jheskethlifeless: all good06:49
jheskeththanks :-)06:49
*** BharatK has quit IRC06:49
*** ddieterly has quit IRC06:51
StevenKZanata, you terrible thing06:52
AJaegerStevenK: What has Zanata done?06:53
*** devvesa has joined #openstack-infra06:53
*** doude has joined #openstack-infra06:53
StevenKAJaeger: It looks like both django_openstack_auth and horizon pull and push from the same horizon project with different resources. Telling zanata to push the django_openstack_auth translations to horizon caused it to remove horizon's files06:55
*** KanagarajM has joined #openstack-infra06:56
*** sshnaidm has joined #openstack-infra06:57
KanagarajMHi, could some cores kindly review https://review.openstack.org/#/c/195531/ . thanks06:57
openstackgerritRomil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under OpenStack namespace  https://review.openstack.org/19695806:58
AJaegerStevenK: yes, the push from the same project.06:58
AJaegerStevenK: We can split this up if needed for Zanata so that "Zanata project" == "repository"06:58
AJaegerYou have the same challenge with Manuals - openstack-manuals, security-doc, ha-guide and api-site repositories each have a resource in the Transifex "Manuals" project06:59
StevenKAJaeger: Yup, I have a thinko in the populate script that I'm about the push up, then I'll do that.06:59
StevenKAJaeger: Ah ha, manuals are already split out into 4 projects06:59
StevenKWhich I did by accident, rather than design06:59
AJaegerStevenK: Please document this change (The 1:1 relationship) of behaviour somewhere.06:59
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Add openstack/designate-dashboard to designate DSVM gates  https://review.openstack.org/19696107:00
StevenKAJaeger: Where would you recommend?07:00
openstackgerritSteve Kowalik proposed openstack-infra/project-config: Correct project name for django_openstack_auth  https://review.openstack.org/19696207:00
*** dtantsur has joined #openstack-infra07:00
*** pblaho_ has joined #openstack-infra07:00
AJaegerStevenK: WE currently have https://wiki.openstack.org/wiki/Translations#Adding_a_new_project_to_transifex and https://wiki.openstack.org/wiki/Translations/Infrastructure. I think adding something to the later patch might be best07:01
*** dtantsur|afk has quit IRC07:01
*** fallenpegasus has quit IRC07:01
*** boris-42 has quit IRC07:02
StevenKAJaeger: I was planning on taking an axe to those scripts after we remove Transifex support, since they can stop being special snowflakes07:02
*** dguerri` is now known as dguerri07:02
*** achanda has quit IRC07:03
*** BharatK has joined #openstack-infra07:03
AJaegerStevenK: Cool!07:04
*** pblaho has quit IRC07:04
*** sputnik1_ has quit IRC07:05
*** dtantsur has quit IRC07:05
openstackgerritSteve Kowalik proposed openstack-infra/project-config: Change Zanata project for django_openstack_auth  https://review.openstack.org/19696407:05
openstackgerritWeidong Shao proposed openstack-infra/irc-meetings: Add Compass weekly meeting  https://review.openstack.org/19696507:06
StevenKAJaeger: WRT to adding a project to Zanata, https://review.openstack.org/187867 does that -- we might be able to store a translation-projects.txt file somewhere, and just run that script when we want to add a new one.07:06
*** pblaho_ has quit IRC07:06
*** pblaho has joined #openstack-infra07:06
*** dguerri is now known as dguerri`07:07
*** mpavlase has joined #openstack-infra07:07
*** mpavlase has quit IRC07:09
*** mpavlase has joined #openstack-infra07:09
*** kiran-r has joined #openstack-infra07:09
AJaegerStevenK: WE could do this automatically. Add translation-projects.txt to project config and whenever we change it, we have a post job that runs and creates the project.07:09
AJaegerFor that we would need to move your script to project-config/jenkins/scripts07:09
*** Hal has joined #openstack-infra07:10
AJaegerStevenK: That would make translation setup much nicer ;). pleia2, what do you think of the above idea? ^07:10
*** Hal is now known as Guest1624007:10
*** dtantsur has joined #openstack-infra07:10
*** hdd has quit IRC07:11
StevenKAJaeger: That idea could certainly work. I do like the idea of commit a file, magic happens.07:11
AJaegerStevenK: so do I - see how we manage project creation for git.openstack.org. A change to project-config/gerrit/projects.yaml creates new repos...07:12
lifelessjhesketh: ok, backish07:13
StevenKAJaeger: Maybe we leverage projects.yaml and add translation: yes or something?07:13
*** Guest16240 has quit IRC07:14
AJaegerStevenK: please discuss with the rest of the team what's the best way to do this. I'm not sure about the best architecture here.07:14
AJaegerLeveraging projects.yaml looks like a nice idea07:14
StevenKAJaeger: I was just about to ask if you'll be at the infra meeting, because it's 6am for me07:14
AJaegerIt'S 9pm for me ;)07:15
AJaegerStevenK: I'll try to join07:15
StevenKAJaeger: It's all good, let me just drop pleia2 an e-mail and she can bring it up07:15
jheskethlifeless: trying to figure out why jenkins can't upload its logs still07:15
jheskethlooks like the jenkins nodes have the right sshkeys etc07:16
jheskethcurrently looking for some more info in logs than just "failed to upload"07:16
*** hichihara has quit IRC07:17
AJaegerThanks, StevenK !07:17
*** markus_z has joined #openstack-infra07:19
*** fhubik has joined #openstack-infra07:19
*** kaisers1 has joined #openstack-infra07:19
openstackgerritRomil Gupta proposed openstack-infra/project-config: Move networking-vsphere project under OpenStack namespace  https://review.openstack.org/19695807:19
*** samuelBartel has joined #openstack-infra07:20
*** HeOS has quit IRC07:22
*** matbu has joined #openstack-infra07:23
ihrachyshka_SergeyLukjanov, hi! any chance you check out the following patch to enable testing for a neutron qos branch that we currently work on a sprint: https://review.openstack.org/196718 ?07:23
openstackgerritJaume Devesa proposed openstack-infra/puppet-openstackci: Add Jenkins Job Builder manifest  https://review.openstack.org/18491907:23
*** matbu has left #openstack-infra07:23
AJaegerihrachyshka_, SergeyLukjanov: I'll check...07:24
ihrachyshka_AJaeger, thanks! I hope this branch magic is valid; I stole it from some other place.07:25
AJaegerihrachyshka_: Looks fine, let's try it ;)07:27
AJaegerihrachyshka_: the if could be much simpler, you don't need pattern matching with =~, a simple "=" would work as well - but it's not wrong, so I +2A07:27
ihrachyshka_AJaeger, thanks! it's a temporary thing, we'll kill it once merged back later this cycle07:28
*** kaisers1 has quit IRC07:29
*** kaisers1 has joined #openstack-infra07:29
*** gema has left #openstack-infra07:30
openstackgerritMerged openstack-infra/project-config: neutron: enable qos service plugin for feature/qos branch  https://review.openstack.org/19671807:32
*** e0ne has joined #openstack-infra07:33
*** camunoz has quit IRC07:34
*** soren has joined #openstack-infra07:34
*** chlong has quit IRC07:35
jheskethlifeless: I'm going to delete the new images see if it is them07:37
*** jlanoux has joined #openstack-infra07:37
*** e0ne has quit IRC07:37
*** hashar has joined #openstack-infra07:41
*** jistr has joined #openstack-infra07:43
*** fhubik is now known as fhubik_afk07:44
*** markvoelker has joined #openstack-infra07:45
*** camunoz has joined #openstack-infra07:45
*** jtomasek has joined #openstack-infra07:46
jheskethSergeyLukjanov: are you around per chance?07:47
*** ddieterly has joined #openstack-infra07:47
*** mrunge has quit IRC07:48
*** armax has quit IRC07:48
*** mpavone has joined #openstack-infra07:49
*** markvoelker has quit IRC07:50
*** enikanorov2_ has joined #openstack-infra07:50
*** amotoki__ has quit IRC07:50
*** mrmartin has joined #openstack-infra07:50
jheskethlifeless: are you still around?07:51
*** ddieterly has quit IRC07:51
*** yamahata has quit IRC07:52
*** enikanorov2 has quit IRC07:52
*** rm_work is now known as rm_work|away07:53
*** rm_work|away is now known as rm_work07:54
*** dizquierdo has joined #openstack-infra07:55
jheskethlifeless: so I thought [SCP] No file(s) found: **/*nose_results.html was a common problem and that it doesn't matter if they don't exist.. now I'm wondering if it's failing because they don't. So my question is do you know of anything that might have changed in the test suite to stop nose_results.html from being generated?07:55
*** sdake_ has joined #openstack-infra07:55
lifelessjhesketh: its a common thing, we don't use nose for most projects07:56
lifelessjhesketh: the backtrace in jenkins is coming from mkdirs07:56
jheskethyeah, I'm trying to figure out why some jobs are still passing07:56
lifelesscommon server?07:57
lifelessstatics a single box right?07:57
lifelesswhat version of jcsh is this ?07:57
* lifeless wants to look up the lines of code07:58
*** shardy_ has joined #openstack-infra07:58
*** otter768 has joined #openstack-infra07:58
jheskethlifeless: jcsh sorry?07:59
openstackgerritThierry Carrez proposed openstack-infra/project-config: Add periodic Kilo jobs  https://review.openstack.org/19670907:59
*** shardy has quit IRC07:59
openstackgerritMerged openstack-infra/irc-meetings: Add Compass weekly meeting  https://review.openstack.org/19696508:00
lifelessthe java sftp client08:00
lifelesserror 2 is indeed:08:01
lifeless2No such fileA reference was made to a file which does not exist.08:02
*** pblaho_ has joined #openstack-infra08:02
*** sergmelikyan has joined #openstack-infra08:02
*** dtantsur_ has joined #openstack-infra08:02
*** claudiub has joined #openstack-infra08:02
jheskethlifeless: here is the scp plugin details, not sure where to find that particular module http://paste.openstack.org/show/327729/08:02
*** fhubik_lunch has joined #openstack-infra08:02
*** otter768 has quit IRC08:02
jheskethhmm, I wonder if something changed in csftp08:03
*** yfried is now known as yfried|afk08:03
anteayalifeless: in response to http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-06-29.log.html#t2015-06-29T22:10:4908:03
*** jistr has quit IRC08:03
anteayalifeless: not to the best of my knowledge, no08:03
hashariirc your scp plugin is a fork from upstream08:03
hasharwith the ability to copy the console08:03
hashar(hello /// just saying)08:03
*** shardy_ has quit IRC08:03
*** dtantsur has quit IRC08:03
jheskethhashar: ah thanks, that's helpful :-)08:03
hasharand some publishers can be made to not fail on error or when some material/log is not found08:04
lifelesshashar: this is erroring on the sftp side, not the source side08:04
*** shardy has joined #openstack-infra08:04
anteayalifeless: would you like to complain about them? if so might I have the url of a patch?08:05
lifelesshashar: (have a look at the error in https://jenkins02.openstack.org/job/gate-nova-python27/835/console)08:05
*** pblaho has quit IRC08:05
hasharuntrusted cert!!!!!!!!!!!!!!08:05
*** Hal has joined #openstack-infra08:05
*** fhubik_afk has quit IRC08:05
*** Hal is now known as Guest1679208:06
hasharseems it fails while doing the mkdirs for /srv/static/logs/91/195191/6/check/gate-nova-python27/4a1fdae08:06
hasharmight not be recursive08:06
hasharor one of the parent path lack appropriate access rights08:06
*** fhubik_lunch has quit IRC08:07
*** fhubik_lunch has joined #openstack-infra08:07
*** fhubik_lunch is now known as fhubik_afk08:07
*** pblaho_ has quit IRC08:07
*** dtantsur_ has quit IRC08:07
*** pblaho has joined #openstack-infra08:08
jheskethlifeless: http://paste.openstack.org/show/327739/ (the version(s))08:08
jheskethhashar: I believe the scp plugin will do it recusviely08:08
hasharyeah seems so08:09
jheskethhashar: either way some of the jobs that have failed have the parent dir and the right permissions08:09
jheskethas far as I can tell anyway08:09
hasharat be.certipost.hudson.plugin.SCPSite.mkdirs(SCPSite.java:314)08:09
hasharbut that might be a perm issue on static.openstack.org ?08:09
hasharjust speculating08:09
*** jistr has joined #openstack-infra08:10
jheskethhashar: possibly, but jenkins owns all the dirs08:11
jd__AJaeger: could you add me into aodh-core 'cause the group is empty for now :-)08:11
jheskethin /srv/static/logs08:11
openstackgerritSirushti Murugesan proposed stackforge/gertty: Specify color for unstable test job  https://review.openstack.org/19697408:11
* anteaya goes back to bed08:11
*** mpavlase has quit IRC08:11
AJaegerjd__: I don't have permissions, you need one of the other infra roots. jhesketh can you help, jd__, please?08:11
jheskeththe scp module(s) aren't very new so I don't think anything has changed there08:11
jd__AJaeger: you looked so smart and useful I thought you were root by now ;)08:12
*** dtantsur has joined #openstack-infra08:12
AJaegerjd__: Nope, I have enough on my plate already and would reject that extra work ;)08:12
AJaegerjd__: thanks!08:12
*** acruz has joined #openstack-infra08:12
jheskethjd__: what's your gerrit username and where is the  aodh repo/group created (ie the change to show me you are meant to be core)08:13
jd__jhesketh: jdanjou and https://review.openstack.org/#/c/192688/08:14
*** sergmelikyan has quit IRC08:14
eantyshevasselin, patrickeast, lennyb: thanks to all for yesterday's duscussion, I've summarized my findings there: https://storyboard.openstack.org/#!/story/200030908:14
*** sergmelikyan has joined #openstack-infra08:15
jheskethjd__: done, let me know if that doesn't work08:15
jd__jhesketh: looks good, thanks a lot!08:15
hasharso looking at the java code08:16
hasharit attempts to create the dir remotely08:16
hasharcatch SftpException08:16
hasharsince mkdir raise an exception if the dir already exists08:16
*** pelix has joined #openstack-infra08:16
hasharthen it does dirstat = channel.stat(curdir + "/" + pathnames[i]);08:16
hasharto ensure the dir is actually there08:16
hasharso in short:  seems it can't create the dir :D08:17
hasharbut there is no logging for the error :/08:17
*** mase_x200 has joined #openstack-infra08:19
lifelessjhesketh: anything in syslog ?08:20
*** haypo has joined #openstack-infra08:20
lifelessjhesketh: auditlog? etc08:20
hashara previous passing build is https://jenkins02.openstack.org/job/gate-nova-python27/828/console08:20
hashar[SCP] Trying to create /srv/static/logs/91/195191/3/gate/gate-nova-python27/c03c360/tox08:20
hashar[SCP] uploading file: '/srv/static/logs/91/195191/3/gate/gate-nova-python27/c03c360//tox/py27-0.log'08:21
jheskethlifeless: good idea, I'll look in a second, but I'm beginning to think scp has had a release08:21
haypohi. i see that oslo.db 2.0 was released whereas my fix for nova was not merged (https://review.openstack.org/#/c/195191/). i also see that most nova check jobs are "UNSTABLE" :-/08:21
hashardon't you deploy the Jenkins plugin from a local mirror ?08:21
AJaegerjd__: thanks for approving the translation setup for aodh. It might take several months until translators work on it, so don't worry. I'll check that the scripts work...08:21
jheskethlifeless: well http://paste.openstack.org/show/327729/ shows we're running 1.9-SNAPSHOT, while the latest release is 1.808:21
jheskethso I'm trying to figure out if we need to pin down that version08:21
*** yfried|afk is now known as yfried08:22
jd__AJaeger: thanks for the help!08:22
jheskethhashar: I don't think so... it looks like it's downloading from http://updates.jenkins-ci.org/download, but I'm trying to figure that out08:22
lifelesshaypo: yes, the world has assploded08:22
lifelesshaypo: we have a munged up thing ready to go08:22
*** derekh has joined #openstack-infra08:22
lifelesshaypo: blocked on fixing the requirements check08:22
lifelesshaypo: which is counfounded by the new images that fix it being exploding08:23
hasharjhesketh: https://jenkins02.openstack.org/pluginManager/api/json?pretty=true&depth=1  yields the list of plugins08:24
dtantsurso rechecks won't work, right?08:24
hasharand scp is definitely at "1.9-SNAPSHOT (private-01/21/2014 12:53-khaido)"08:24
jheskethdtantsur: no, I doubt it08:24
jheskethhashar: and there is this http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/jenkins_master.pp#n5508:25
jheskethbut I can't find where scp is actually being installed :-(08:25
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: heat temprevert: Add port_security_enabled for Neutron  https://review.openstack.org/19665908:25
*** sergmelikyan has quit IRC08:25
*** john-davidge has joined #openstack-infra08:26
*** sergmelikyan has joined #openstack-infra08:26
lifelessjhesketh: oh, its remoted by the plugin from the server08:26
lifelessjhesketh: (I'm 99% sure,  hashar can confirm)08:27
jheskethlifeless: as in the jenkins master does the upload? yes, I believe that's correct08:27
hasharah yeah most probably the master is doing it08:27
hasharinstead of the slave08:27
*** yamamoto has quit IRC08:27
hasharI would suspect the plugin on the master to retrieve the files from the slaves over the master/slave connection08:28
hasharthen the master to ssh to static and scp stuff there08:28
hasharbut that is really speculation08:28
lifelessso the sftp code isn't the issue08:29
*** isviridov_away is now known as isviridov08:29
hasharon static, you might want to look at /var/log/auth.log to figure out what is scping08:32
hasharto confirm whether it is the master or slave08:32
hasharmight want to also reproduce the mkdir manually, I have no clue how jsch mkdir()  handles it08:32
*** fhubik_afk is now known as fhubik_lunch08:33
openstackgerritWeidong Shao proposed openstack-infra/project-config: Add compass to OpenStack projects  https://review.openstack.org/19691608:33
jheskethhashar: jenkins is logging in okay (Accepted publickey for jenkins from port 35600 ssh2)08:35
openstackgerritSerg Melikyan proposed openstack/requirements: Exclude yaql 0.3.0 from requirements  https://review.openstack.org/19697708:35
*** xarses has quit IRC08:36
*** xarses has joined #openstack-infra08:36
*** xarses has quit IRC08:37
hasharjhesketh: time to try a mkdir ? :D08:37
*** sergmelikyan has quit IRC08:37
*** xarses has joined #openstack-infra08:37
*** Kennan has quit IRC08:38
*** xarses has quit IRC08:38
*** Kennan has joined #openstack-infra08:38
jheskethhashar: great idea, and it fails08:38
jheskeththe filessytem is read-only08:38
*** alex_xu_ is now known as alex_xu08:38
jheskethand I had just discovered http://paste.openstack.org/show/327759/08:38
*** xarses has joined #openstack-infra08:39
*** xarses has quit IRC08:39
*** bauzas has joined #openstack-infra08:39
*** yamada-h has joined #openstack-infra08:39
*** HeOS has joined #openstack-infra08:40
*** xarses has joined #openstack-infra08:40
lifelessjhesketh: yay08:40
*** xarses has quit IRC08:40
jheskethso it's got errors=remount-ro08:41
hasharwell caught!08:41
jheskethso I could do a fsck, but I'm wondering if I should wait for other roots to help08:41
*** xarses has joined #openstack-infra08:41
jheskethin case things get worse08:41
jheskethhashar: can't beleive I didn't check dmesg sooner :-(08:41
*** xarses has quit IRC08:41
*** dguitarbite has joined #openstack-infra08:42
hasharwhen I worked for a network operating center,  the first checks we would do were something like:08:42
hashar1) is the power cord plugged in08:42
hashar2) are there any lights on the equipment08:42
*** xarses has joined #openstack-infra08:42
hashar3) are the network cables  / optical fiber properly plugged in :D08:42
*** mase_x200 has quit IRC08:42
hasharbut often I would forgot about those basic checks and waste time debugging something else08:43
lifelesschecklists are good08:43
hasharand once you have checklists, you can automatize / monitor them :-D08:43
AJaegerttx, lifeless, jhesketh: What about sending a IRC notification about the problems to all channels?08:43
bauzasAJaeger: +10008:43
lifelessI think thats a great idea08:43
jheskethAJaeger: heh, I was just about to, I'm trying to find the command08:43
ttxAJaeger: +1 want me to do it ?08:43
bauzasis this something blocker for merging ?08:43
AJaegerttx, teach jhesketh how to do it, plesae08:43
*** mrunge has joined #openstack-infra08:43
*** yamada-h has quit IRC08:44
ttxjhesketh: http://docs.openstack.org/infra/system-config/irc.html#statusbot08:44
jheskethoh it's the openstackstatus user right?08:44
ttxbut you need to be on the whitelist, not sur if you are08:44
hasharso in the ideal world you end up with a notification stating: "Disk dead on static.openstack.org || Automatically filled bug #1234"08:44
openstackbug 1234 in Launchpad itself "Gina is an unmaintainable mess of command line options, environment variables and shell scripts" [Medium,Fix released] https://launchpad.net/bugs/1234 - Assigned to Daniel Henrique Debonzi (debonzi)08:44
jheskethttx: yep, was trying to find the user08:44
jheskethttx: lets find out08:44
ttxworst case I'll send it08:44
AJaegerjhesketh: you're not - see project-config/accessbot/channels.yaml08:45
jheskethAJaeger: hmm, boo, will fix that later08:45
*** sdake_ has quit IRC08:45
ttxjhesketh: let me know what the message is and I'll parrot it08:45
jheskethttx: please send something along the lines of #status alert OpenStack CI is down due to hard drive failures.08:45
lifelesswe need a checklist for 'adding root access steps'08:45
AJaegerttx, I think we can remove maffulli from the list, correct?08:45
ttx#status alert OpenStack CI is down due to hard drive failures08:45
openstackstatusttx: sending alert08:46
ttxAJaeger: probably yes08:46
AJaegerjhesketh: I'll create a patch to add you08:46
jheskethttx: thanks08:46
jheskethAJaeger: thanks :-)08:46
AJaegerAh, lifeless has also the powers to do it ;)08:46
ttxto be fair, the only reason I'm on that list was so that we can send alerts in European mornings. If we have good coverage of infra-core people there now, I can probably also be removed from list08:47
jheskethttx: it's still helpful.. lifeless and I aren't always around for example08:47
AJaegerttx: please stay on it, I'm not sure we have that great coverage yet08:47
ttxI'm fine with MoarPower[tm]08:47
*** yamamoto has joined #openstack-infra08:47
hasharit is probably better to have many people able to status log :D08:47
*** ddieterly has joined #openstack-infra08:47
-openstackstatus- NOTICE: OpenStack CI is down due to hard drive failures08:48
*** ChanServ changes topic to "OpenStack CI is down due to hard drive failures"08:48
hasharand tweet! https://twitter.com/wikimediatech  :-D08:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add jhesketh to accessbot, remove maffulli.  https://review.openstack.org/19698308:49
AJaegerShould we send out an email as well?08:49
openstackstatusttx: finished sending alert08:50
lifelessAJaeger: oh, I didn't realise. I shall next time.08:50
jheskethlifeless, ttx, AJaeger: do you think I should fire off a fsck shall we wait for more infra to wake up?08:52
*** ddieterly has quit IRC08:52
bauzasAJaeger: you could at least provide a status on http://lists.openstack.org/pipermail/openstack-dev/2015-June/068308.html08:52
bauzas(sorry, missed the question mark)08:52
lifelessjhesketh: you're the root08:52
lifelessjhesketh: I'd start by checking for rackspace status allerts etc08:53
openstackgerritSteve Kowalik proposed openstack-infra/system-config: Pass through zanata client parameters  https://review.openstack.org/17644008:53
hasharassuming it is a VM, that might be a disk failure on the host system :/08:53
ttxaren't we supposed to shoot the cattle in the head ?08:53
AJaegerjhesketh: your call, I have too few experience with infra root to guide you in any direction08:53
* ttx doesn't know enough of the specific situation to provide good advice08:54
*** dkehn has quit IRC08:55
lifelessjhesketh: also check the cinder? status of the drive08:55
*** bradjones has quit IRC08:55
AJaegerjhesketh: let me write a short email. Which server has the harddisk failure?08:55
jheskethhashar: they are cinder mounts afaik08:55
jheskethAJaeger: static.openstack.org (which is the log server etc)08:56
lifelessAJaeger: the static volume where all the job logs go08:56
AJaegerok, will write a short email...08:56
*** dkehn has joined #openstack-infra08:56
*** kushal has joined #openstack-infra08:57
*** bradjones has joined #openstack-infra08:57
*** bradjones has quit IRC08:57
*** bradjones has joined #openstack-infra08:57
lifelessjhesketh: OOI did you rollback the images?08:59
*** _nadya_ has quit IRC09:00
*** sergmelikyan has joined #openstack-infra09:00
*** _nadya_ has joined #openstack-infra09:00
jheskethlifeless: yes I did09:01
*** koolhead17 has joined #openstack-infra09:02
*** e0ne has joined #openstack-infra09:02
*** _nadya_ has quit IRC09:04
jheskethlifeless: hmm, I don't think I have access to the cloud creds yet to check the cinder status :-(09:05
jheskethI might just hold off for more roots to turn up09:05
*** mpavlase has joined #openstack-infra09:05
bauzasAJaeger: thanks :)09:06
AJaegerjhesketh: fungi is normally around in 3 to 4 hours from now on...09:06
SergeyLukjanovjhesketh, hey, just arrived to office09:06
jheskethSergeyLukjanov: Hey09:07
SergeyLukjanovihrachyshka_, AJaeger sorry folks, just arrived to office09:07
jheskethSergeyLukjanov: static.o.o has failing drives09:07
SergeyLukjanovjhesketh, oh...09:07
jheskethneed to decide what to do09:07
jheskethmost likely a fsck09:07
jheskethbut I don't have access to the cloud to see what cinder or anything says09:07
AJaegerSergeyLukjanov: ihrachyshka_ is happy now, now worries.09:08
SergeyLukjanovjhesketh, I have, will check09:08
*** HeOS has quit IRC09:09
*** HeOS has joined #openstack-infra09:09
SergeyLukjanovjhesketh, all drives are green in rackspace cloud ui09:10
SergeyLukjanovjhesketh, which drive are failing09:11
hasharxvdm  /  dm-009:11
hasharseems the kernel can't ensure the writes  / journal /  cache  is properly written to disk.09:11
jheskethdm-1 is also failing09:11
hasharmight be an issue with the underlying host disk09:11
hashar(but don't quote me :D )09:11
*** _nadya__ has joined #openstack-infra09:12
*** kushal has quit IRC09:13
*** dguerri` is now known as dguerri09:14
*** e0ne is now known as e0ne_09:15
*** Ala has joined #openstack-infra09:16
lifelessjhesketh: thats another checklist item I guess, but I thought the api keys were also in hiera09:18
openstackgerritQiming Teng proposed openstack-infra/system-config: Add #senlin to the list of eavesdrop channels  https://review.openstack.org/19699309:19
SergeyLukjanovjhesketh, yup, most likely fsck09:21
ihrachyshka_AJaeger, SergeyLukjanov, I am indeed, thanks! (except that CI is broken)09:21
*** soren has quit IRC09:22
*** evrardjp has joined #openstack-infra09:22
jheskethSergeyLukjanov: righto, shall we do that now?09:24
SergeyLukjanovjhesketh, I have no experience with it... have you?09:24
rakhmerovhi, what does UNSTABLE mean for a gate?09:25
rakhmerove.g. https://jenkins04.openstack.org/job/gate-mistral-python27/58/console09:25
jheskethSergeyLukjanov: not on an LVM09:25
lifelessrakhmerov: infra has a major outage09:25
lifelessrakhmerov: see topic09:25
lifelessrakhmerov: see also the dev list09:25
*** e0ne_ has quit IRC09:26
*** Ala has quit IRC09:28
*** fhubik_lunch is now known as fhubik_afk09:28
*** e0ne has joined #openstack-infra09:29
*** kushal has joined #openstack-infra09:30
*** mattt has joined #openstack-infra09:33
*** markvoelker has joined #openstack-infra09:34
*** fhubik_afk is now known as fhubik_lunch09:36
*** john-davidge has quit IRC09:36
jheskethSergeyLukjanov: there is no raid setup is there... what backups (if any) do we have?09:39
*** john-davidge has joined #openstack-infra09:39
*** markvoelker has quit IRC09:39
SergeyLukjanovjhesketh, probably we have backups for some parts of its data in bup09:40
jheskethSergeyLukjanov: bup doesn't appear to be on static.o.o09:41
Kennanhi lifeless:09:43
Kennando you know why jenkins jobs failured with "Unstable"?09:43
SergeyLukjanovjhesketh, so, it sounds like we have no backups at all for it09:43
KennanI found gerrit marked with Unstable09:43
Kennangate-magnum-python27    UNSTABLE in 2m 46s09:43
*** soren has joined #openstack-infra09:43
Kennanwhat does that mean?09:43
AJaegerKennan: see mailing list and topic.09:44
ekarlsoI thought the cloud couldn't get taken down by harddrive failures :D09:44
jheskethSergeyLukjanov: do you want to take a snapshot of the instance then? (If you think that's a good idea)09:45
ekarlsoor stuff running it too :D09:45
AJaegerKennan: http://lists.openstack.org/pipermail/openstack-dev/2015-June/068311.html09:45
jheskethand then the next question is, do you think it's worth snapshotting the cinder devices, or will that take too long09:45
SergeyLukjanovjhesketh, of the whole instance or of the volumes?09:45
SergeyLukjanovjhesketh, no idea - I've never used rackspace cloud09:46
KennanThanks Ajaeger:09:46
jheskethSergeyLukjanov: snapshotting the instance should only capture the vm driver surely?09:46
SergeyLukjanovjhesketh, yup, it'll not touch the attached drives09:46
jheskethwe should start with that then, just in case09:46
jheskethSergeyLukjanov: then we should consider if we want to snapshot the attached drives09:47
*** ddieterly has joined #openstack-infra09:48
SergeyLukjanovjhesketh, there is only an option to create image from server - I think it's the VM snapshot09:50
SergeyLukjanovjhesketh, now trying to do it09:50
*** ociuhandu has quit IRC09:52
jheskethSergeyLukjanov: yeah that's the one09:52
*** yfried is now known as yfried|afk09:52
SergeyLukjanovjhesketh, in progress09:52
jheskethSergeyLukjanov: what do you think about snapshoting the volumes, or should we just run the fsck next?09:52
*** ddieterly has quit IRC09:53
SergeyLukjanovjhesketh, AFAIK it could occasionally cause data corruption09:55
jheskethSergeyLukjanov: snapshotting? or fsck?09:55
SergeyLukjanovjhesketh, I mean fsck09:55
jheskethSergeyLukjanov: should have checked this sooner, but https://status.rackspace.com/index/viewincidents?group=1109:55
spredzyTo core could I get a review on this patchset please : https://review.openstack.org/#/c/175849/09:56
SergeyLukjanovjhesketh, shit, we missed this important notice09:56
jheskethSergeyLukjanov: it would explain the corruption09:57
*** otter768 has joined #openstack-infra09:59
jheskethSergeyLukjanov: the incident also hasn't been fully resolved, so anything we do now may be futile09:59
SergeyLukjanovjhesketh, agreed, so, we should wait while it'll be marked as solved10:00
jheskethokay, roger that10:00
*** yfried|afk is now known as yfried10:00
AJaegerspredzy: Let'S first fix our CI infrastructure, see the topic...10:01
spredzyAJaeger, sorry I haven't read it first. My bad.10:02
AJaegerSergeyLukjanov: to me it reads: Just remount the volume and we should be fine - but it might hit us again ;/10:03
*** otter768 has quit IRC10:04
openstackgerritSerg Melikyan proposed openstack/requirements: Exclude yaql 0.3.0 from requirements  https://review.openstack.org/19697710:05
*** Hal has joined #openstack-infra10:07
*** Guest16792 has quit IRC10:08
*** Hal is now known as Guest2102110:08
*** soren_ has joined #openstack-infra10:08
*** soren has quit IRC10:08
jheskethAJaeger: there's also a good chance we got some corruption during the incident that'll need repairing10:09
*** asahlin_away has quit IRC10:09
AJaegerjhesketh: ;(10:10
*** sergmelikyan has quit IRC10:10
jheskethAJaeger: given that we write a lot of logs, it's likely just a job or two, which really isn't too bad10:10
jhesketha recheck etc10:10
*** asahlin has joined #openstack-infra10:11
AJaegerjhesketh: indeed10:11
*** sergmelikyan has joined #openstack-infra10:11
*** fhubik_lunch is now known as fhubik_afk10:13
*** coolsvap is now known as coolsvap|away10:14
*** Qiming has quit IRC10:15
*** fhubik_afk is now known as fhubik_lunch10:17
*** sergmelikyan has quit IRC10:17
*** koolhead17 has quit IRC10:22
*** e0ne is now known as e0ne_10:23
*** e0ne_ is now known as e0ne10:25
*** mpaolino_ has joined #openstack-infra10:27
*** soren_ has quit IRC10:27
*** jlanoux_ has joined #openstack-infra10:29
*** fhubik_lunch is now known as fhubik_afk10:29
*** jlanoux has quit IRC10:31
*** fhubik_afk is now known as fhubik_lunch10:33
*** soren has joined #openstack-infra10:33
*** soren has quit IRC10:33
*** soren has joined #openstack-infra10:33
*** rossella_s has quit IRC10:36
*** rossella_s has joined #openstack-infra10:37
*** sergmelikyan has joined #openstack-infra10:37
*** soren has quit IRC10:37
*** romcheg has joined #openstack-infra10:38
romchegHi folks10:38
romchegI've just made a release 6.1.1 of stackforge/python-fuelclient but looks like it didn't get uploaded to PyPi10:39
romchegAJaeger, could you please take a look?10:39
romchegI'm not sure who else is here at this time10:39
jheskethromcheg: it won't be uploaded until our static server disks are fixed10:40
jheskethwhich is currently due to an outage10:40
AJaegerromcheg: didn't you see the topic of our channels?10:40
*** claudiub has quit IRC10:40
romchegjhesketh: thanks for the update! My fault, I haven't looked at the topic10:41
romchegjhesketh: Do I have to do anything after hard drives are fixed in order to get it to pypi?10:41
AJaegerromcheg: ask one of the infra roots to push it again through our tag queue...10:42
AJaegerromcheg: once you get the all clear, bring some beer and ask ;)10:43
jheskethyeah we'll have to retrigger the post job10:43
*** shardy_ has joined #openstack-infra10:43
*** shardy has quit IRC10:45
romchegThanks folks!10:45
*** sergmelikyan has quit IRC10:47
*** shardy_ has quit IRC10:48
*** ddieterly has joined #openstack-infra10:49
*** shardy has joined #openstack-infra10:49
*** chlong has joined #openstack-infra10:49
*** ociuhandu has joined #openstack-infra10:50
*** yamamoto has quit IRC10:50
*** pc_m has joined #openstack-infra10:50
*** BharatK has quit IRC10:51
*** jlanoux has joined #openstack-infra10:51
*** fhubik_lunch is now known as fhubik_afk10:52
*** soren has joined #openstack-infra10:53
*** ddieterly has quit IRC10:54
*** jlanoux_ has quit IRC10:55
*** hichihara has joined #openstack-infra10:58
*** kushal has quit IRC10:59
*** BharatK has joined #openstack-infra11:02
*** dims has joined #openstack-infra11:07
openstackgerritSzymon Wróblewski proposed openstack/requirements: tooz 0.16.0 or better required  https://review.openstack.org/19702111:07
*** EricGonczer_ has joined #openstack-infra11:10
*** larainema has joined #openstack-infra11:13
*** sergmelikyan has joined #openstack-infra11:14
*** sshnaidm has quit IRC11:14
*** Qiming has joined #openstack-infra11:15
*** hashar has quit IRC11:16
*** ParsectiX has quit IRC11:18
*** ParsectiX has joined #openstack-infra11:19
*** weshay has joined #openstack-infra11:21
*** mpaolino_ has quit IRC11:23
*** mpaolino has quit IRC11:23
*** EricGonczer_ has quit IRC11:23
*** sshnaidm has joined #openstack-infra11:23
*** markvoelker has joined #openstack-infra11:24
*** thorst has joined #openstack-infra11:27
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Allow for git base override  https://review.openstack.org/15103411:29
*** markvoelker has quit IRC11:29
*** KanagarajM has quit IRC11:29
*** sshnaidm has quit IRC11:29
*** sshnaidm has joined #openstack-infra11:30
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: post_test_hook is not executed when the gate test returns an error  https://review.openstack.org/14886311:31
markus_zHi, I'm developing a script which should help me automatize cleanups in Launchpad. I'm wondering to which project I could push a patch set to get some feedback?11:32
*** EricGonczer_ has joined #openstack-infra11:34
*** Ala has joined #openstack-infra11:35
Kiallmarkus_z: probably https://github.com/openstack-infra/release-tools11:36
*** radez is now known as radez_g0n311:36
Kiall(there's a pile of other LP related scripts used for the release process in there)11:36
*** ildikov has quit IRC11:38
markus_zKiall: Yes, that looks good, I'll try it there, thank you!11:38
*** EricGonczer_ has quit IRC11:39
openstackgerritsebastian marcet proposed openstack-infra/openstackid: [smarcet]  https://review.openstack.org/19689811:39
*** dprince has joined #openstack-infra11:40
*** baoli has joined #openstack-infra11:40
*** ldnunes has joined #openstack-infra11:44
*** hichihara has quit IRC11:47
*** jschlueter has joined #openstack-infra11:47
*** michchap has quit IRC11:48
*** michchap has joined #openstack-infra11:49
thorstHi.  Can someone take a look at this change:  https://review.openstack.org/#/c/196641/11:49
thorstits blocking some change sets in other components as we add support for that hypervisor11:49
*** ddieterly has joined #openstack-infra11:50
*** fawadkhaliq has quit IRC11:51
*** ildikov has joined #openstack-infra11:51
*** _nadya__ has quit IRC11:52
*** EricGonczer_ has joined #openstack-infra11:53
*** _nadya_ has joined #openstack-infra11:53
*** ddieterly has quit IRC11:54
*** mwagner_lap has joined #openstack-infra11:55
*** EricGonc_ has joined #openstack-infra11:57
*** _nadya_ has quit IRC11:57
*** radez_g0n3 is now known as radez11:58
*** _nadya_ has joined #openstack-infra11:58
*** EricGonczer_ has quit IRC11:58
*** fhubik_afk is now known as fhubik_lunch11:58
*** jistr is now known as jistr|class11:59
*** fhubik_lunch is now known as fhubik_afk11:59
*** yfried is now known as yfried|afk11:59
*** EricGonczer_ has joined #openstack-infra12:00
*** otter768 has joined #openstack-infra12:00
*** hichihara has joined #openstack-infra12:00
*** EricGonc_ has quit IRC12:01
*** prad has joined #openstack-infra12:02
*** markvoelker has joined #openstack-infra12:02
*** AzherKhan has quit IRC12:03
*** sergmelikyan has quit IRC12:04
*** marcusvrn has joined #openstack-infra12:05
*** otter768 has quit IRC12:05
*** jistr|class is now known as jistr12:07
*** jyuso1 has quit IRC12:07
*** maishsk has quit IRC12:09
*** claudiub has joined #openstack-infra12:10
*** bcourt has joined #openstack-infra12:10
*** markvoelker_ has joined #openstack-infra12:11
*** markvoel_ has joined #openstack-infra12:13
*** markvoelker has quit IRC12:13
*** sslypushenko has joined #openstack-infra12:13
*** maishsk_afk has joined #openstack-infra12:14
*** maishsk_afk is now known as maishsk12:14
*** michchap has quit IRC12:15
*** hashar has joined #openstack-infra12:15
*** michchap has joined #openstack-infra12:17
*** markvoelker_ has quit IRC12:17
*** yfried|afk is now known as yfried12:19
*** rkukura has quit IRC12:26
*** zz_dimtruck is now known as dimtruck12:27
openstackgerritThierry Carrez proposed openstack-infra/releasestatus: Loop to retrieve all reviews  https://review.openstack.org/19704212:29
fungiwow--fun! hard drive failures?12:30
haypofungi: yes. http://lists.openstack.org/pipermail/openstack-dev/2015-June/068311.html12:30
*** aysyd has joined #openstack-infra12:32
fungithanks for the details. i'll start digging into it now12:32
fungilooks like /dev/xvdm is unhappy12:33
*** _nadya_ has quit IRC12:34
*** maishsk has quit IRC12:36
fungiokay, i've got all processes with open filehandles in that mountpoint killed and have umounted12:37
*** maishsk_afk has joined #openstack-infra12:37
*** maishsk_afk is now known as maishsk12:37
*** hichihara has quit IRC12:38
*** mmedvede has joined #openstack-infra12:38
*** _nadya_ has joined #openstack-infra12:38
*** michchap_ has joined #openstack-infra12:38
*** gordc_afk is now known as gordc12:38
fungigetting the rest of the volumes from that vg unmounted now so i can deactivate it12:38
*** sergmelikyan has joined #openstack-infra12:39
*** yamamoto has joined #openstack-infra12:39
*** hichihara has joined #openstack-infra12:40
*** michchap has quit IRC12:40
fungithe main vg is now deactivated12:40
*** _nadya_ has quit IRC12:42
*** _nadya_ has joined #openstack-infra12:42
*** kgiusti has joined #openstack-infra12:42
*** shardy has quit IRC12:43
*** _nadya_ has joined #openstack-infra12:43
*** shardy has joined #openstack-infra12:43
fungirebooting the server now, and will detach/reattach all cinder volumes momentarily12:43
*** nagyz has joined #openstack-infra12:44
*** mpaolino_ has joined #openstack-infra12:45
*** EricGonczer_ has quit IRC12:45
AJaegerthanks, fungi!12:45
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add propose-backports jobs to openstack/puppet-*  https://review.openstack.org/17585012:45
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag  https://review.openstack.org/17584912:45
*** mpaolino has joined #openstack-infra12:45
*** mpavone has quit IRC12:46
AJaegerfungi, once everything is ok again: Please give a status update on the openstack-dev mailing list where I wrote this one: http://lists.openstack.org/pipermail/openstack-dev/2015-June/068311.html and also update the IRC status. I'll be offline for the next few hours therefore writing this now...12:46
fungiwill do12:46
*** bknudson has joined #openstack-infra12:47
*** ayoung has quit IRC12:48
*** maishsk has quit IRC12:48
fungioh, actually i found the cinder volume corresponding to /dev/xvdm so i'm just detaching/attaching that one for now12:48
jheskethfungi: oh hey, thanks for that. Let me know if I can help12:48
*** _nadya_ has quit IRC12:48
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add propose-backports jobs to openstack/puppet-*  https://review.openstack.org/17585012:48
*** _nadya_ has joined #openstack-infra12:49
*** _nadya_ has quit IRC12:49
*** _nadya_ has joined #openstack-infra12:49
*** scheuran has quit IRC12:49
*** rfolco has joined #openstack-infra12:50
*** ddieterly has joined #openstack-infra12:51
*** markvoel_ has quit IRC12:51
*** _diana_ has joined #openstack-infra12:51
*** asadoughi has joined #openstack-infra12:51
fungirunning filesystem checks on the volumes in that vg now12:51
*** Guest27041 has joined #openstack-infra12:51
*** markvoelker_ has joined #openstack-infra12:52
*** _nadya_ has quit IRC12:52
*** jsavak has joined #openstack-infra12:52
fungiluckily all were clean except the logs volume12:52
fungithough as that one's >12tb it may take a few minutes to complete its fsck :/12:53
*** yamamoto has quit IRC12:53
jheskethCool :-)12:53
*** maishsk_afk has joined #openstack-infra12:53
jheskethfungi: the incident hasn't been resolved fully yet so we may get more problems12:53
*** maishsk_afk is now known as maishsk12:53
fungiwas there an acknowledged incident in dfw?12:53
*** bknudson has quit IRC12:54
*** _nadya_ has joined #openstack-infra12:54
fungii just assumed this was the same "your compute node lost contact with the storage backend where that volume resided" problem we'd seen in the past12:54
jheskethfungi: yeah the cinder control nodes are playing up afaik12:54
AJaegerfungi, https://status.rackspace.com/index/viewincidents?group=1112:55
fungioh, i see the cloud block storage ! triangles now12:55
fungii skimmed earlier just looking at the alerts and maintenance block at the top12:55
*** ddieterly has quit IRC12:55
*** rlandy has joined #openstack-infra12:56
fungialso anybody know about the trove maintenance they've got planned tomorrow for dfw? they're supposedly rebuilding all our database instances, but i'm worried that means they'll end up with new dns names since they seem to be randomly-generated hex values to start with12:58
jheskethNo idea sorry12:59
fungii'll ask fanatically in the support ticket they opened for me13:00
mrmartinfungi: are we using that for groups and openstackid?13:00
*** puranamr has joined #openstack-infra13:00
*** sebrandon1 has joined #openstack-infra13:00
fungimrmartin: yes. according to the support ticket it'll be a ~5 minute outage13:01
mrmartindo we have proper backups ?13:01
fungipresumably impacting the www site too13:01
*** markvoelker_ has quit IRC13:02
*** sergmelikyan has quit IRC13:02
*** tlian has joined #openstack-infra13:02
*** markvoelker has joined #openstack-infra13:02
*** sergmelikyan has joined #openstack-infra13:03
*** markvoelker_ has joined #openstack-infra13:03
*** ddieterly has joined #openstack-infra13:03
fungiworth checking. i see a 0-byte /var/backups/mysql_backups/groups.sql.gz on groups.o.o from 06:36 utc today13:04
fungineed to look into that once i've got static.o.o back on track13:04
*** mase_x200 has joined #openstack-infra13:04
mrmartinfungi: it is not looking nice13:05
*** deepakcs has quit IRC13:05
*** markvoelker_ has quit IRC13:05
mrmartinsomething should be wrong with those backup scripts13:05
fungiopenstackid.org doesn't seem to be configured to do database backups, but i'll get that in and working today13:05
fungimrmartin: yeah, i expect it's misconfigured13:06
*** markvoelker_ has joined #openstack-infra13:06
fungii'll find out13:06
*** ddieterly has quit IRC13:06
*** changbl has quit IRC13:06
fungii've got a few minutes while this fsck continues anyway13:06
*** markvoelker has quit IRC13:07
fungilooks like groups is relying on /root/.groups_db.cnf for the backups and its got the wrong details. i'll fix that13:08
*** sergmelikyan has quit IRC13:08
*** miqui has joined #openstack-infra13:09
*** mrunge has quit IRC13:10
*** bknudson has joined #openstack-infra13:10
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies  https://review.openstack.org/19706013:10
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies  https://review.openstack.org/19706013:12
*** peristeri has joined #openstack-infra13:12
*** _nadya_ has quit IRC13:12
*** jsavak has quit IRC13:13
*** jsavak has joined #openstack-infra13:13
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies  https://review.openstack.org/19706013:14
markus_zstupid spaces ...13:14
*** sergmelikyan has joined #openstack-infra13:14
*** puranamr has quit IRC13:14
*** sergmelikyan has quit IRC13:14
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add a non-voting py34 check job to os-brick  https://review.openstack.org/19706413:18
openstackgerritWayne Warren proposed openstack-infra/infra-specs: Zuul github-integration spec.  https://review.openstack.org/16363713:18
*** julim has joined #openstack-infra13:19
*** kushal has joined #openstack-infra13:20
*** sergmelikyan has joined #openstack-infra13:20
*** EricGonczer_ has joined #openstack-infra13:21
*** Guest27041 is now known as agentle13:21
*** EricGonc_ has joined #openstack-infra13:22
dimsfungi: looks like some reviews are in gate queue and green13:22
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/release-tools: launchpad: Add script to clean up bug inconsistencies  https://review.openstack.org/19706013:23
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag  https://review.openstack.org/17584913:24
fungidims: yeah, log uploads are spooling to the parent volume. i'm not expecting to try and find a way to preserve them13:24
fungithe fsck has made it to phase 2 now though, so shouldn't be much longer. then i'll blow away what's accumulated out from under the mountpoint before i bring it back online13:25
*** _nadya_ has joined #openstack-infra13:25
fungiand then one more reboot of the server to make sure everything comes back up normally13:25
*** EricGonczer_ has quit IRC13:25
*** blaedd has joined #openstack-infra13:26
openstackgerritMerged openstack-infra/project-config: Make Magnum functional tests voting  https://review.openstack.org/19192113:26
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add propose-backports jobs to openstack/puppet-*  https://review.openstack.org/17585013:27
*** nelsnelson has joined #openstack-infra13:27
openstackgerritMerged openstack-infra/project-config: oslo.messaging: remove gate_hook  https://review.openstack.org/19177313:28
dimsthanks fungi13:28
*** e0ne is now known as e0ne_13:29
*** alivigni has joined #openstack-infra13:29
*** homeless has joined #openstack-infra13:30
*** boris-42 has joined #openstack-infra13:31
fungimrmartin: actually, groups db backups are fine. i was looking in the wrong spot. it has 30 days of full mysqldumps of all the remote databases13:31
fungimrmartin: i also just ran a fresh one to make sure it's working correctly13:31
*** cody-somerville has joined #openstack-infra13:32
*** cody-somerville has quit IRC13:32
*** cody-somerville has joined #openstack-infra13:32
mrmartinso we can survive the trove maintenance tomorrow13:32
fungii'll get the mysql-backups module added to openstackid.org now for good measure13:32
openstackgerritAntoine Musso proposed openstack-infra/nodepool: Fix Jenkins nodes methods to use POST  https://review.openstack.org/19360013:32
openstackgerritMerged openstack-infra/project-config: Add pre/post test hooks for Heat grenade job  https://review.openstack.org/18814713:32
*** doug-fish has quit IRC13:33
*** blaedd has quit IRC13:34
*** mattymo has joined #openstack-infra13:35
mattymofungi, are you around?13:36
*** soren has quit IRC13:36
kashyapHi folks, I see that "check-grenade-dsvm" job failed for one of the changes I submitted. However, clicking on the said failed job throws "File Not Found" - http://logs.openstack.org/97/193497/2/check/check-grenade-dsvm/500d054/13:36
kashyapAnyone know why?13:36
jrollkashyap: see topic :)13:36
fungimattymo: yeah, i'm here13:36
kashyapjroll: Darn, sorry.13:36
kashyapjroll: I _did_ notice it this morning in one of the channels, but missed to remember it.13:37
mattymofungi, I have a really curious gerrit dilemma. I would like to attribute a patchset to an email address from another company13:37
mattymofor example, I am developing a patch for company XYZ and john.smith@xyzcompany.com is a registered openstack foundation email address13:37
mattymohow can I make such an attribution?13:37
fungimattymo: that's more of a git dilemma13:37
mattymoI can tell git config anything... but I believe gerrit will reject fake author that doesn't match my ssh key13:38
fungimattymo: gerrit only cares about the committer address matching your account, and doesn't really care at all what the author name/address on the patch is13:38
*** mriedem has joined #openstack-infra13:39
*** e0ne_ has quit IRC13:39
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Add non-voting Sahara grenade job for upgrades testing  https://review.openstack.org/19707613:39
fungimattymo: so all you need to do is reset the author on the commit (if it's incorrect, though if you brought the patch into git via git am it should be preserved), then commit --amend to update the committer to be you13:39
fungiin fact, git am may already do that part too13:39
*** blaedd has joined #openstack-infra13:39
fungiassuming you received it in git format-patch output format or similar enough for git am to import it correctly13:40
*** sigmavirus24_awa is now known as sigmavirus2413:40
mattymofungi, and let's say I have the code already on review and I want to modify it. Just commit --amend --author=john.smith@xyzcompany.com is good enough?13:40
fungimattymo: should be. then look at git log --pretty=fuller to see if the author and committer are the way you want (them, you)13:41
*** tvanderwiel has joined #openstack-infra13:41
*** ZZelle has quit IRC13:41
mattymofungi, thanks for your help!13:42
*** zz_jgrimm is now known as jgrimm13:42
*** ZZelle has joined #openstack-infra13:42
fungimattymo: it's worth noting that the gerrit change "owner" will still be the committer of the first patchset for a change13:42
*** BharatK has quit IRC13:42
fungisince committer is what gerrit actually cares about (to support projects which may receive outside contributions but only allow certain committers to upload them for review)13:43
*** agentle has quit IRC13:43
*** ayoung has joined #openstack-infra13:43
mattymofungi, which value matters to stackalytics then?13:43
mattymocommiter or author?13:43
fungimattymo: i don't know. you'd have to ask the people who wrote stackalytics or look at its source code13:44
*** sebrandon1 has quit IRC13:44
fungimattymo: if your reason for attributing that patchset to the other person is to get them "active technical contributor status" then the only effective solution is to abandon your change and get them to upload it themselves13:45
*** VivCheri has joined #openstack-infra13:45
fungimattymo: or else submit a patch to the governance repo adding them to the "extra-atcs" file and convince the tc to approve it13:45
mattymoI don't think it's for ATC (it's a stackforge commit anyway), but instead for some sort of copyright attribution of the commit13:46
*** e0ne has joined #openstack-infra13:46
fungimattymo: if it's just for copyright attribution, simply making sure it's accompanied by a line in the file(s) being changed with the additional copyright holder information should be sufficient, but updating the author field correctly is still useful for reasons of provenance13:47
jeblairfungi: can you join #openstack-infra-incident ?13:47
fungimattymo: and in that case you don't need to care about gerrit change owner13:47
fungijeblair: oh, yep!13:47
*** doug-fish has joined #openstack-infra13:49
*** soren has joined #openstack-infra13:51
*** sshnaidm has quit IRC13:52
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Add non-voting Sahara grenade job for upgrades testing  https://review.openstack.org/19707613:53
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add database backups for openstackid.org  https://review.openstack.org/19708313:53
fungimrmartin: ^13:53
fungii'm going to test that out with a puppet dev env in just a sec13:53
*** jsavak has quit IRC13:53
mrmartinawesome, don't we have backups for openstackid?13:53
*** sshnaidm has joined #openstack-infra13:53
*** pblaho has quit IRC13:53
fungidoes it have local files which aren't in configuration management?13:54
*** fhubik_afk is now known as fhubik_lunch13:54
mrmartinunder app/storage,13:54
*** BharatK has joined #openstack-infra13:55
*** wenlock has joined #openstack-infra13:55
mrmartinbut it is just for cache, so db backup is enough there13:55
*** eharney has joined #openstack-infra13:55
fungiyeah, generally we only add filesystem backups if there's something on the local filesystem which is irreplaceable. database dumps are a grey area since they theoretically are a copy of data which is in the database server instance too, but backing them up to a separate server in a different region/provider via bup is still potentially useful13:56
*** jsavak has joined #openstack-infra13:57
mrmartinfungi: just a little update required for https://review.openstack.org/19708313:57
*** fawadkhaliq has joined #openstack-infra13:57
fungimrmartin: oh, that's a good idea13:58
*** yamahata has joined #openstack-infra13:58
openstackgerritMerged openstack-infra/project-config: Add beaker-localhost and beaker-nodepool hypervisors  https://review.openstack.org/18575413:58
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add database backups for openstackid.org  https://review.openstack.org/19708313:59
*** sergmelikyan has quit IRC13:59
*** baoli has quit IRC13:59
yfriedwhere do I submit a bug against requirements?13:59
*** sergmelikyan has joined #openstack-infra13:59
*** baoli has joined #openstack-infra14:00
*** dims has quit IRC14:01
*** yamamoto has joined #openstack-infra14:01
*** otter768 has joined #openstack-infra14:01
*** dims has joined #openstack-infra14:01
*** marun has joined #openstack-infra14:01
*** kiran-r has quit IRC14:02
*** dboik has joined #openstack-infra14:02
fungii've got the server rebooted again and the logs volume back in place following a filesystem check. if nobody spots any new problems with it i'll stand down the alert and follow up to the ml shortly14:05
*** otter768 has quit IRC14:06
openstackgerritMerged openstack-infra/project-config: Make tempest-dsvm-cells job voting  https://review.openstack.org/19089414:06
mattymofungi, last q.. how can I check a given email address's company affiliation in the openstack foundation?14:07
*** jsavak has quit IRC14:07
openstackgerritMerged openstack-infra/project-config: Making integration tests gate voting  https://review.openstack.org/19432414:07
*** _nadya_ has quit IRC14:07
fungimattymo: http://www.openstack.org/community/members/14:08
fungiyfried: i don't think there is a bug tracker for the requirements repo14:09
*** scheuran has joined #openstack-infra14:09
fungiyfried: it didn't really contain software of note until the past few weeks14:09
yfriedfungi:  https://bugs.launchpad.net/openstack-gate/+bug/146242114:09
openstackLaunchpad bug 1462421 in OpenStack-Gate "jsonschema 2.5.0 pulling in functools32 which breaks py26 and py34 jobs" [Undecided,Confirmed]14:09
fungiyfried: that's exciting14:09
yfriedfungi: https://bugs.launchpad.net/rally/+bug/146605514:10
openstackLaunchpad bug 1466055 in Rally "functools32 should be added to requirements" [Undecided,New]14:10
*** fhubik_lunch is now known as fhubik14:10
*** jsavak has joined #openstack-infra14:10
yfriedfungi: seems like a bug in jsonschema latest14:10
yfriedfungi: I wonder if other project are hitting this14:10
mriedemyfried: i think that was a known thing a couple of weeks ago14:10
*** sshnaidm has quit IRC14:11
fungiyfried: in that case i'd suggest filing a bug against jsonschema not openstack requirements14:11
mriedemyfried: it was already fixed upstream from what i remember14:11
mriedemwas some issue with wheels14:11
yfriedmriedem: not in rally14:11
*** mtanino has joined #openstack-infra14:11
yfriedfungi: I agree, but until it's fixed, our code breaks...14:11
*** pblaho has joined #openstack-infra14:11
mriedemit should be fixed already14:11
mriedemi'm looking for the upstream issue14:12
*** sshnaidm has joined #openstack-infra14:12
mriedemyfried: https://github.com/Julian/jsonschema/issues/22814:12
*** ihrachyshka_ has quit IRC14:13
*** ParsectiX has quit IRC14:13
*** ihrachyshka has joined #openstack-infra14:14
fungimrmartin: i've used a test of 197083 to get current backup dumps of both databases on openstackid.org just now. looks right (5mib compressed for the id db, 50mib compressed for the ss db). took only a few seconds to complete14:14
yfriedmriedem: looks like it solved the 1st bug (py26 and py34) but introduced the 2nd14:14
*** annegentle has joined #openstack-infra14:14
mrmartinfungi: thank you, so we are prepared now14:14
mriedemyfried: open an issue against jsonschema upstream14:15
*** puranamr has joined #openstack-infra14:15
fungimrmartin: yep! and thanks for the reminder to check those servers14:15
yfriedmriedem: ack14:15
yfriedmriedem: but how to proceed until it's fixed?14:15
fungiyfried: you could introduce blacklist entries for the affected releases in global requirements14:16
fungiassuming they truly are broken14:16
mriedemyfried: https://github.com/Julian/jsonschema/issues/23414:16
mriedemit's already reported14:16
*** nihilifer has joined #openstack-infra14:17
*** _nadya_ has joined #openstack-infra14:17
mriedemyfried: use newer pip14:17
mriedemis the workaround14:17
mattymofungi, it doesn't take email addresses14:18
fungimattymo: you may need to figure out the person's name14:18
mattymoI know the name14:18
*** baoli has quit IRC14:18
yfriedmriedem: ok. so I must update pip for this, so - not a bug14:19
yfriedmriedem: tnx14:19
*** dbite has joined #openstack-infra14:19
fungimattymo: i've got a wishlist feature request for a more robust member querying api. it would be extremely useful for things like generating electoral rolls too14:19
*** puranamr has quit IRC14:19
yfriedmriedem: I guess the 2 bugs can be closed?14:20
*** r-daneel has joined #openstack-infra14:20
mriedemyfried: i already did :)14:20
*** unicell has joined #openstack-infra14:21
yfriedmriedem: I wanted to do that :(14:22
*** baoli has joined #openstack-infra14:22
yfriedmriedem: [17:21] <stpierre> that's odd, global requirements specifies pip>=6.014:22
*** unicell1 has quit IRC14:22
*** cindyo has joined #openstack-infra14:23
*** _diana_ has quit IRC14:23
*** rkukura has joined #openstack-infra14:23
*** tonytan4ever has joined #openstack-infra14:23
*** shardy_ has joined #openstack-infra14:24
AJaegerfungi: Please remember to scroll back later to around 6:10 UTC, the translation proposal jobs all failed, I gave some comments on them.14:24
fungiAJaeger: thanks, i'll have a look after my next meeting14:24
thorstsdague: Would you mind taking a peak at https://review.openstack.org/#/c/196641/ ?14:24
*** shardy has quit IRC14:24
thorstIt has an updated requirement for the PowerVM API which enables the hypervisor driver support we're working on.14:24
*** fhubik is now known as fhubik_afk14:25
openstackgerritMerged openstack/os-testr: Disable printing percent change on run time by default  https://review.openstack.org/19486514:29
*** shardy_ has quit IRC14:29
*** amitgandhinz has joined #openstack-infra14:30
*** mpavlase has quit IRC14:30
*** shardy has joined #openstack-infra14:30
*** baoli has quit IRC14:30
*** sshnaidm has quit IRC14:31
*** baoli has joined #openstack-infra14:31
*** mase_x200 has quit IRC14:31
*** sshnaidm has joined #openstack-infra14:31
*** koolhead17 has joined #openstack-infra14:31
haypomriedem: thanks for https://review.openstack.org/#/c/195191/14:32
mriedemhaypo: np14:33
*** baoli has quit IRC14:33
*** baoli has joined #openstack-infra14:33
*** baoli has quit IRC14:33
mriedemthorst: is that py34 compliant?14:33
thorstmriedem: I believe so, but I'm going to verify right now...just a sec14:34
*** baoli has joined #openstack-infra14:34
mriedemit's not listed https://github.com/pypowervm/pypowervm/blob/master/setup.cfg#L1814:34
*** bcourt has quit IRC14:34
thorstgood pt.  We'll get that updated now14:34
haypomriedem: python 3.3 & 3.4 are very close14:34
haypomriedem: it's pure python, i don't expect major issue with 3.414:35
mriedemthems the rules14:35
*** nikil89_ has joined #openstack-infra14:35
thorstI'll get it updated and complaint.  Should have another revision on it soon.14:36
*** hdd has joined #openstack-infra14:36
thorsthaypo: could I add you as a reviewer once I get that updated?14:36
haypothorst: yes14:36
*** BharatK has quit IRC14:36
*** rossella_s has quit IRC14:36
thorstthx!  Appreciate the feedback14:37
*** rossella_s has joined #openstack-infra14:37
*** pblaho has quit IRC14:39
*** nikil89_ has quit IRC14:39
*** pblaho has joined #openstack-infra14:39
*** sshnaidm has quit IRC14:41
*** jsavak has quit IRC14:41
*** sshnaidm has joined #openstack-infra14:41
mriedemthorst: i see there is a 0.0.1 version on pypi but no tags or releases in github? https://github.com/pypowervm/pypowervm/releases14:42
thorstmreidem: yeah, we'll branch and do a 0.0.2 version before the next proposal, get it 3.4 doc'd, etc...14:43
*** jsavak has joined #openstack-infra14:44
*** fallenpegasus has joined #openstack-infra14:46
*** sergmelikyan has quit IRC14:47
*** jsavak has quit IRC14:47
*** sergmelikyan has joined #openstack-infra14:49
*** sergmelikyan has quit IRC14:49
asselin_fungi, good morning. Shall we try to merge those common-ci zuul patches today?14:50
openstackgerritVictor Stinner proposed openstack-infra/project-config: Add a non-voting py34 check job to os-brick  https://review.openstack.org/19706414:51
mriedemthorst: s/branch/tag/14:51
fungiasselin_: today is shaping up to be a bad day for our systems (well, one of our service providers) breaking. it's also the day most of us have all our meetings14:52
*** doude has quit IRC14:52
*** laughterwym has joined #openstack-infra14:52
*** jsavak has joined #openstack-infra14:53
thorstmriedem: yep, tag.14:53
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"14:53
-openstackstatus- NOTICE: The log volume was repaired and brought back online at 14:00 UTC. Log links today from before that time may be missing, and changes should be rechecked if fresh job logs are desired for them.14:53
yfriedmriedem: looks like latest pip doesn't fix the issue14:54
mriedemyfried: i'd comment on the upstream jsonschema issue then with the details of the issue you're seeing14:55
*** e0ne is now known as e0ne_14:55
*** EricGonc_ has quit IRC14:55
*** shardy has quit IRC14:56
*** shardy has joined #openstack-infra14:56
*** k4n0 has quit IRC14:57
*** dangers_away is now known as dangers14:58
*** jsavak has quit IRC14:58
*** fawadkhaliq has quit IRC14:58
*** jaypipes has quit IRC14:58
*** e0ne_ is now known as e0ne14:58
*** sergmelikyan has joined #openstack-infra14:58
*** jsavak has joined #openstack-infra14:58
yfriedmriedem: could it that pip version comes after jsonschema version?14:59
*** e0ne is now known as e0ne_14:59
*** doude has joined #openstack-infra14:59
*** e0ne_ is now known as e0ne14:59
*** maishsk has quit IRC15:00
yfriedmriedem: if so, this means that pip should be the first global-req15:00
frickleris graphite.openstack.org also affected by the rackspace thingy?15:01
frickleralso image builds seem to have mostly failed today because of bad timing for creating openstack-infra/beaker-localhost15:01
AJaegerfrickler: yes, fungi is looking at graphite right now15:02
*** jsavak has quit IRC15:03
*** sambetts has quit IRC15:03
*** cindyo has quit IRC15:04
sc68calso uh, I was downloading some logs from logs.openstack.org via wget early this morning15:04
sc68calhopefully I didn't break anything15:04
*** fawadkhaliq has joined #openstack-infra15:04
fungifrickler: our graphite server was hung completely. when i force restarted it, it came back up without working ipv4 through its default gateway. i have a ticket open with rackspace support to get that taken care of15:05
fungisc68cal: nope, the breakage was completely internal to rackspace. some major issue with their block storage backend15:05
asselin_fungi, understood15:05
fungisc68cal: your downloads definitely did not cause it15:05
*** _diana_ has joined #openstack-infra15:05
sc68calfungi: ok good :)15:05
*** sergmelikyan has quit IRC15:06
fricklersc68cal: except you used this hacker tool called IPv6 maybe ... :D15:06
sc68calhaha :)15:07
*** sambetts has joined #openstack-infra15:07
jheskethokay I'm off15:07
jheskethsee you at the meeting, night!15:07
*** asselin__ has joined #openstack-infra15:08
*** _nadya_ has quit IRC15:09
*** _nadya_ has joined #openstack-infra15:10
*** scheuran has quit IRC15:11
*** asselin_ has quit IRC15:12
*** shardy has quit IRC15:12
*** ihrachyshka has quit IRC15:12
*** hashar has quit IRC15:13
*** xyang1 has joined #openstack-infra15:13
*** _nadya_ has quit IRC15:14
*** jasondotstar has joined #openstack-infra15:15
openstackgerritsebastian marcet proposed openstack-infra/openstackid: [smarcet]  https://review.openstack.org/19689815:15
yfriedmriedem: http://paste.openstack.org/show/328808/15:15
yfriedmriedem: search for "@@"15:15
*** kiran-r has joined #openstack-infra15:16
yfriedmriedem: looks like pip is updated before jsonschema is installed, and yet - jsonschema is installed without functools3215:16
yfriedmriedem: however, when I install jsonschema again, it works as expected.15:16
*** bcourt has joined #openstack-infra15:17
*** Qiming has quit IRC15:17
yfriedmriedem: seems like an issue with pip were it's still using the old version in the same operations15:17
*** wenlock_ has joined #openstack-infra15:17
*** shardy has joined #openstack-infra15:18
fungiyfried: chances are this is related to virtualenvs... since virtualenv vendors a copy of pip you get whatever version of pip was concurrent with the virtualenv version used (not the version of pip installed on your system, much to the surprise of many), unless you explicitly upgrade pip inside that virtualenv afterward15:19
yfriedfungi: that's not the issue15:19
yfriedfungi: specifying pip version in requirements update pip15:20
fungiyfried: not necessarily soon enough though15:20
*** austin81 has joined #openstack-infra15:20
yfriedfungi: but it uses the same old pip to install all other packages in that file15:20
fungiyfried: if you pip install some list which includes a new version constraint on pip, then you get all packages in that list installed by the old version15:20
*** mikeit has joined #openstack-infra15:20
yfriedfungi: exactly15:21
fungiyfried: this is why we never bothered to include pip in our requirements lists in the past15:21
yfriedfungi: ack15:21
fungiyou need a separate bootstrapping step to upgrade pip before you use it to install other things15:21
yfriedfungi: so, I'm still at loss for the bug15:21
fungiyfried: i haven't looked at the bug, sorry. plenty of issues with our systems today, so i'll leave issues with tests to people for whom that's more in their sphere of interest15:22
*** _diana_ has quit IRC15:22
* fungi is pretty much 100% in systems administrator mode at the moment15:22
yfriedfungi: well, that's not an issue of tests. this will hit everyone who installs using an old pip15:23
*** rlandy has quit IRC15:23
fungiyfried: yep, software15:23
fungitoday for me is dealing with broken filesystems and networks, sorry15:23
*** rlandy has joined #openstack-infra15:23
*** cindyo has joined #openstack-infra15:23
fungioh, also databases and backups15:24
greghayneswow, looks like there was a fun morning15:24
yfriedfungi: tnx anyway15:24
*** haypo has left #openstack-infra15:24
*** _diana_ has joined #openstack-infra15:25
*** yamahata has quit IRC15:26
fungihrm, and troubleshooting e-mail deliveries15:26
fungiall things sysadmin15:26
openstackgerritMerged openstack-infra/project-config: Handle missing pot files in translation proposal  https://review.openstack.org/19378415:26
*** laughterwym has quit IRC15:27
*** armax has joined #openstack-infra15:27
*** erikmwilson_ has joined #openstack-infra15:28
greghaynesthats exciting to see huge explosions fixed before west coast wakes :)15:28
*** Ala has quit IRC15:28
openstackgerritMerged openstack-infra/project-config: Fix gate_hook in rally-cue experimental job  https://review.openstack.org/19599115:28
*** erikmwilson_ is now known as erikwilson_15:28
fungiyeah, exciting isn't the word i would have used15:29
fungiseems most of the explosions were well before the east coast of north america woke up for that matter15:29
pabelangerfungi, jeblair: Be curious your thoughts about adding the EPEL repos into the centos6 host to get access to new binaries.  In this case, git.15:29
fungipabelanger: is there newer git in epel? i thought we already had epel activated on those servers15:30
*** hodos has joined #openstack-infra15:30
*** bradjones has quit IRC15:30
pabelangerfungi, that is what I am checking out, the version shipped with EPEL15:30
*** changbl has joined #openstack-infra15:31
*** alivigni has quit IRC15:31
*** homeless has quit IRC15:31
fungipabelanger: 1.7.1-3.el6_4.1 is in base according to yum list. it's not spotting one available in centos 6 epel15:31
*** homeless has joined #openstack-infra15:32
*** bradjones has joined #openstack-infra15:32
*** bradjones has quit IRC15:32
*** bradjones has joined #openstack-infra15:32
fungithough i'll readily admit my yum-fu is limited15:32
openstackgerritMerged openstack-infra/project-config: Surveil: Enable pypy jobs  https://review.openstack.org/19624715:33
pabelangerfungi, ya, looks like is the latest :(15:33
pabelangeroh well15:33
pabelangertrying to figure out ways to remove the manual compile15:33
pabelangerwill have to do that until we get our packages going15:33
pabelangerI did what nibalizer suggested and did a JJB job to upgrade git inline15:34
*** puranamr has joined #openstack-infra15:34
*** marun has quit IRC15:34
*** kiran-r has quit IRC15:35
*** _diana_ has quit IRC15:35
fungiAJaeger: okay, i'm taking a look at the translations issues now that my first meeting has wrapped up15:35
*** _diana_ has joined #openstack-infra15:35
*** puranamr_ has joined #openstack-infra15:36
*** puranamr has quit IRC15:36
*** DericHorn-HP has joined #openstack-infra15:36
*** enikanorov2_ has quit IRC15:36
openstackgerritMerged openstack-infra/project-config: Add coverage to Manila check to aid reviewers  https://review.openstack.org/19628315:36
openstackgerritMerged openstack-infra/project-config: Fetch all heads and tags when caching source repos  https://review.openstack.org/19565015:37
openstackgerritMerged openstack-infra/project-config: Set up jobs to test new ubuntu-trusty nodes  https://review.openstack.org/19638215:37
*** abregman has quit IRC15:37
*** sshnaidm has quit IRC15:37
openstackgerritMerged openstack-infra/nodepool: Make diskimage example config work  https://review.openstack.org/19053815:37
*** sshnaidm has joined #openstack-infra15:38
*** sabeen1 has quit IRC15:38
*** enikanorov2 has joined #openstack-infra15:38
openstackgerritMerged openstack-infra/project-config: puppet: add 3.8 release tests support  https://review.openstack.org/18989015:40
pabelangerfungi, IUS is rolling git 2.4.3: https://iuscommunity.org/pages/About.html15:40
openstackgerritMerged openstack-infra/project-config: run-docs.sh: execute docs job if present  https://review.openstack.org/19182815:40
fungiAJaeger: clarkb: pleia2: i think there's a good chance that "fatal: ambiguous argument '(detached': unknown revision or path not in the working tree." is a symptom of a behavior change in the version of git in ubuntu trusty, or possibly with a newer git-review15:40
pabelangerfungi, using yum replace it upgraded properly on my host15:40
openstackgerritMerged openstack-infra/project-config: Make keystone* bandit jobs voting  https://review.openstack.org/19627715:40
zigoWhat's the name of that git repo which is a template for new projects?15:40
*** _nadya_ has joined #openstack-infra15:41
fungizigo: http://git.openstack.org/cgit/openstack-dev/cookiecutter/15:41
zigofungi: Thanks my friend!15:41
pabelangerfungi, and I believe you need to explicitly call yum replace to pull in packages from IUS15:41
fungizigo: you're welcome!15:41
pleia2fungi: aha, I assumed it was related to trusty upgrade but it's not clear to me why that's happening15:42
fungipabelanger: okay, that's good to know. we should probably also be careful not to do that in such a way that it bleeds over into non-job-worker servers15:42
pabelangerfungi, Right.  I should see how many centos6 jobs we have going right now. I don't imagine it being too much15:42
fungipabelanger: since we know that latest git breaks things like shallow clones so we want to make sure we don't accidentally upgrade our centos 6 based git servers15:42
pabelangerfungi, I know puppet-apply job will be going away once the beaker jobs are all up and running. But another option would to just bring centos7 online and move forward with that15:43
zigofungi: This is what is used for new Oslo libs too, right?15:44
zigoI don't think that's the one git repo that I'm looking for.15:45
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: update syntax-future job to test puppet ~> 4.2.0  https://review.openstack.org/19638615:45
EmilienMAJaeger, SergeyLukjanov I had to rebase (conflict) can you +A again?15:45
SergeyLukjanovEmilienM, sure15:45
fungizigo: oslo has their own flavor of it at http://git.openstack.org/cgit/openstack-dev/oslo-cookiecutter/15:46
openstackgerritMerged openstack-infra/grafyaml: Move yaml functions into its own class  https://review.openstack.org/19243115:46
openstackgerritMerged openstack-infra/grafyaml: Re-run tox -egenconfig to fix oslo.config bug  https://review.openstack.org/19416915:46
fungizigo: there are similar cookiecutter template repos for devstack plugins, puppet modules...15:46
*** dtantsur is now known as dtantsur|afk15:46
openstackgerritMerged openstack-infra/project-config: Enable python-jobs on designate-dashboard  https://review.openstack.org/19667815:47
*** dizquierdo has quit IRC15:47
openstackgerritMerged openstack-infra/zuul: transpose typo'd characters  https://review.openstack.org/19245915:47
*** geoffarnold has joined #openstack-infra15:48
openstackgerritMerged openstack-infra/zuul: Be more verbose on non-tagged versions  https://review.openstack.org/18859715:48
*** kiran-r has joined #openstack-infra15:49
*** esmute__ has joined #openstack-infra15:49
*** _nadya_ has quit IRC15:49
*** abregman has joined #openstack-infra15:50
*** abitha has joined #openstack-infra15:50
openstackgerritMerged openstack-infra/project-config: Adding new repo for security specs  https://review.openstack.org/19616515:51
openstackgerritThomas Goirand proposed openstack-dev/cookiecutter: Add insentive to write a long description  https://review.openstack.org/19714615:51
*** kiran-r has quit IRC15:51
*** jasondotstar has quit IRC15:52
fungiAJaeger: so... git-review thinking the local branch name is "(detached" happens when you checkout a specific ref rather than a refname (branch, et cetera). we could probably work around this by making sure we tell checkout to make a named branch with -B15:52
*** stevemar has joined #openstack-infra15:53
stevemarbump, new requirements request https://review.openstack.org/#/c/196339/15:53
*** mpavlase has joined #openstack-infra15:53
openstackgerritMerged openstack-infra/project-config: Ensure we have py2.6 jobs enabled for cloud-init  https://review.openstack.org/19688415:53
stevemarbut its a new oslo lib... so low impact15:54
*** sdake has joined #openstack-infra15:54
fungiAJaeger: or maybe just use git branch -M to force move the current head renaming it to a branch15:54
openstackgerritSean M. Collins proposed openstack-infra/elastic-recheck: Add fingerprint for Linux Bridge hard reboot race  https://review.openstack.org/19715015:56
*** achanda has joined #openstack-infra15:56
*** sdake_ has joined #openstack-infra15:56
sc68calanteaya: ^15:58
*** unicell has quit IRC15:58
*** asselin__ has quit IRC15:58
openstackgerritMerged openstack-infra/project-config: Change Zanata project for django_openstack_auth  https://review.openstack.org/19696415:58
pabelangerfungi, jeblair: Do you think we'll add a bare-centos7 image? to go along side bare-precise and bare-trusty?15:59
*** markus_z has quit IRC15:59
openstackgerritMerged openstack-infra/project-config: Correct project name for django_openstack_auth  https://review.openstack.org/19696216:00
openstackgerritMerged openstack-infra/system-config: Change review->change in zuul table  https://review.openstack.org/16993416:00
*** achanda has quit IRC16:00
fungipabelanger: we'll probably add a centos-7 diskimage to go along with the centos-6 and ubuntu-trusty diskimages16:00
timrcjeblair, Hi.  You 'round?16:00
*** sdake has quit IRC16:00
fungipabelanger: and use bindep to install distro packages in it on demand16:00
*** jistr has quit IRC16:01
pabelangerfungi, okay. Is the plan to retire the bare-precise and bare-trusty images?16:01
fungipabelanger: yes16:01
fungiwe're trying to retire all the snapshot-based nodepool images16:01
*** sergmelikyan has joined #openstack-infra16:01
*** _diana_ has quit IRC16:01
*** otter768 has joined #openstack-infra16:02
*** yfried has quit IRC16:02
*** kiran-r has joined #openstack-infra16:02
openstackgerritMerged openstack-infra/zuul: Status: Don't raise "abort" as error to the user  https://review.openstack.org/17224116:02
*** acruz has quit IRC16:02
*** spredzy is now known as spredzy|afk16:04
*** spzala has joined #openstack-infra16:05
*** jlanoux has quit IRC16:05
*** shardy has quit IRC16:06
*** jasondotstar has joined #openstack-infra16:06
*** sarob has joined #openstack-infra16:06
openstackgerritCody Littley proposed openstack-infra/nodepool: Fixed crash in exception handler  https://review.openstack.org/19517216:06
*** otter768 has quit IRC16:07
pleia2SergeyLukjanov: thanks for the zanata +As, was about to add them to the meeting agenda as things we need to push through :)16:07
*** shardy has joined #openstack-infra16:07
*** fhubik_afk is now known as fhubik16:07
*** mpavlase has quit IRC16:07
SergeyLukjanovpleia2, :)16:07
*** mpaolino_ has quit IRC16:11
*** dannywilson has joined #openstack-infra16:11
fricklerSergeyLukjanov: if you want https://review.openstack.org/194751, you will have to +A https://review.openstack.org/189142, too ;)16:11
*** pelix has quit IRC16:12
*** fallenpegasus has quit IRC16:12
SergeyLukjanovfrickler, I've not approved https://review.openstack.org/#/c/189142/ due to the open question16:12
SergeyLukjanovfrickler, do you think it's not important q. to wait for and it should done separately?16:13
openstackgerritMerged openstack-infra/system-config: Adjust Zanata admins  https://review.openstack.org/19367716:13
fricklerSergeyLukjanov: I think we can look at that later, yes16:13
*** dannywilson has quit IRC16:13
fricklerSergeyLukjanov: next step would be https://review.openstack.org/189141, with that we would be ready to build images with vivid16:14
*** dannywilson has joined #openstack-infra16:14
fungiAJaeger: pleia2: were either of you working on the translation proposals fix? if not i'll start testing a possible solution16:14
fricklerSergeyLukjanov: and then we can set up some jobs with that to see what happens in the "real world" with it16:14
fungias soon as i eat my lunch16:14
pleia2fungi: no, an I'm about to head out the door for dentist16:14
openstackgerritMerged openstack-infra/system-config: Switch to openstackid.org for Zanata auth  https://review.openstack.org/19673516:15
fungipleia2: ahh, righth. best of luck. they're like vampires16:15
pleia2all sparkling teeth in infra this week16:15
*** kiran-r has quit IRC16:15
jesusaurusfungi: have you ever had anyone accidentally propose a stack of changes for a rebased branch? and do you know of a quick way to abandon several thousand changes?16:16
*** weshay has quit IRC16:16
fungijesusaurus: a dirty script looping calls to the api is probably the easiest way16:17
*** sshnaidm has quit IRC16:17
*** shardy_ has joined #openstack-infra16:17
*** shardy has quit IRC16:18
*** beekneemech is now known as bnemec16:19
*** weshay has joined #openstack-infra16:19
zigodims: Should we upgrade all the oslo and friends libs to the latest release for releasing liberty-1 ?16:19
fungijesusaurus: oh! or latest gertty has batch actions you can apply to a gerrit query16:19
fungijesusaurus: not sure whether it does abandon though16:20
*** dizquierdo has joined #openstack-infra16:20
cody-somervillefungi: how do you dequeue them from zuul though?16:20
*** _diana_ has joined #openstack-infra16:21
*** sergmelikyan has quit IRC16:21
fungicody-somerville: dump the zuul queue to a local file, stop zuul, filter the file contents to remove the changes you don't want to test, then start zuul and only reenqueue the good changes16:22
*** devvesa has quit IRC16:22
fungicody-somerville: or get https://review.openstack.org/95035 working16:22
openstackgerritFawad Khaliq proposed openstack-infra/project-config: Move networking-plumgrid into the openstack namespace  https://review.openstack.org/19716916:23
*** EricGonczer_ has joined #openstack-infra16:23
*** shardy_ has quit IRC16:23
*** rlandy has quit IRC16:23
*** e0ne is now known as e0ne_16:23
*** rlandy has joined #openstack-infra16:23
*** shardy has joined #openstack-infra16:24
*** notnownikki has joined #openstack-infra16:24
*** fhubik has quit IRC16:24
timrcfungi, FWIW https://review.openstack.org/#/c/95035/ i technically working :) We even have a user!  There is just a request for more exhaustive testing.16:25
*** smarcet has joined #openstack-infra16:25
greghaynesmordred: SpamapS I just noticed something wierd on http://paste.openstack.org/show/321629/ (traceback nodepool is giving us uploading to rax) - line 12 - why is a .get on the kwargs dict causing us call create_object?16:25
openstackgerritMerged openstack-infra/system-config: Make puppet work with Ubuntu Vivid  https://review.openstack.org/18914216:25
smarcetfungi: hello there waiting on approval for https://review.openstack.org/#/c/196898/16:26
smarcetfungi: should i add you as reviewer?16:26
SpamapSgreghaynes: reading16:26
*** jaypipes has joined #openstack-infra16:26
cody-somervillefungi: How do you protect yourself from a DoS?16:26
fungicody-somerville: we try not to get people mad at us?16:27
greghaynesSpamapS: oh, wait, disreguard me - im being silly (theres an extra ) on that line)16:27
* greghaynes chugs more coffee16:27
SpamapSgreghaynes: yeah disregarding you :)16:27
SpamapSgreghaynes: but confirming, thats the trace down into the call to create_object, not the get. :)16:28
*** unicell has joined #openstack-infra16:28
*** _diana_ has quit IRC16:28
fungismarcet: i was just talking to jimmy about that. it looks like we never added anyone to the openstackid-resources-core group in gerrit. in theory it should have the same initial members as the openstackid-core group. jeblair: can you confirm if you're okay with that (or take care of updating it yourself)?16:28
SpamapSgreghaynes: the error itself looks like maybe something swift needs isn't making it into the SwiftService16:28
*** sergmelikyan has joined #openstack-infra16:28
jeblairfungi: will do16:28
openstackgerritMerged openstack-infra/system-config: Trim user specific package a bit  https://review.openstack.org/19475116:29
*** dontalton has joined #openstack-infra16:29
*** samuelBartel has quit IRC16:29
smarcetfungi: ok , and what about the new open id connect changes, this is the first one on that direction https://review.openstack.org/#/c/196898/16:29
greghaynesSpamapS: Yea, I have a test case against rax but its not causing that traceback and having a hard time figuring out why im not hitting it - for some reason https://review.openstack.org/#/c/196936/ isnt calling swift.client.get_auth(16:29
*** baoli has quit IRC16:29
smarcetfungi: should i autoapprove that one? is more like a refactoring to start the work on oidc16:30
jeblairfungi, smarcet: done16:30
*** doude has quit IRC16:30
smarcetfungi: i am planning to start the whole oidc feature one one single commit ( is that ok?)16:30
greghaynesSpamapS: (with a clouds.yaml of rax cloud defined, obv)16:30
smarcetjeblair: thx :)16:30
*** baoli_ has joined #openstack-infra16:31
fungismarcet: seems fine to me. it's your project so whatever you and anyone else actively reviewing changes on it (if there are any at present) would like to see16:33
fungismarcet: and yeah, if you don't have anyone else reviewing changes on those projects yet, then approving the changes yourself is fine16:34
smarcetfungi : ok got it , thx :)16:34
*** e0ne_ has quit IRC16:34
jeblairwhen other folks take an interest, that's probably a better time to be more formal :)16:34
*** kiran-r has joined #openstack-infra16:34
fungiexactly. and until then, don't let a lack of interested reviewers in that repo hold you up from landing features/fixes it needs16:35
smarcetfungi: jeblair: sure understood16:35
*** achanda has joined #openstack-infra16:35
*** jasondotstar has quit IRC16:36
*** e0ne has joined #openstack-infra16:36
*** jasondotstar has joined #openstack-infra16:37
pabelangertimrc, This just happened in grafana: https://github.com/grafana/grafana/commit/ae0f8c77d16840bf4526cab68da54460b043db9b16:38
pabelangerthey now allow username / password access to REST api16:38
pabelangervs api_key16:38
*** smarcet has quit IRC16:40
timrcpabelanger, That's cool.16:41
*** bhunter71 has joined #openstack-infra16:42
pabelangertimrc, Yup, we'll need to run 2.1.0, but the last issue for doing automated deployments now16:43
*** acruz has joined #openstack-infra16:46
*** smarcet has joined #openstack-infra16:46
*** fallenpegasus has joined #openstack-infra16:49
*** openstackgerrit has quit IRC16:50
*** cindyo has quit IRC16:50
*** openstackgerrit has joined #openstack-infra16:51
*** ildikov has quit IRC16:53
* SpamapS struggles with api paste thingies16:53
*** hichihara has quit IRC16:54
openstackgerritMerged openstack-infra/system-config: Use vim-enhanced on centos  https://review.openstack.org/19646116:55
*** romcheg has quit IRC16:56
*** fawadkhaliq has quit IRC16:57
*** marun has joined #openstack-infra16:57
*** dizquierdo has quit IRC16:57
*** geoffarnold has quit IRC16:58
*** geoffarnold has joined #openstack-infra16:58
*** abregman has quit IRC16:59
*** derekh has quit IRC16:59
*** jasondotstar has quit IRC16:59
*** ociuhandu has quit IRC17:00
*** kiran-r has quit IRC17:02
timrcpabelanger, Do we get a default Org?17:02
krotscheckIs grenade still being touchy? I've got a build that was passing, but then failed on a recheck.17:02
timrcpabelanger, I didn't see anything 'bout that in the code reviews.17:02
krotscheckSomething about failing to build an image.17:02
pabelangertimrc, Ya, I didn't make any changes to the OrgID in my reviews. So, we are defaulting I guess17:03
*** jasondotstar has joined #openstack-infra17:03
pabelangertimrc, what are you thinking. Creating one for openstack and stackforge?17:03
greghaynesfungi: Hey - is there a log line in the nodepool upload logs about "endpoint not found" around where we get the swift upload traceback?17:03
*** kiran-r has joined #openstack-infra17:03
timrcpabelanger, I dunno... Orgs in grafana are kind of idiotic IMHO.17:03
*** Swami has joined #openstack-infra17:04
greghaynesSpamapS: ^ I suspect get_session_endpoint is returning None in shade... that shouldnt happen right?17:04
pabelangertimrc, Ya, for the most part, I'm just using a single org right now17:04
timrcpabelanger, Unless things have changed only *one* org can be anonymous.17:04
pabelangerOh, I didn't notice that17:04
*** fawadkhaliq has joined #openstack-infra17:04
timrcpabelanger, The way we have it setup in HP is to tag dashboards and then create dashboard lists they list other dashboards with a certain tag.17:04
timrcNo one wants to log in to see pretty graphs.17:04
pabelangertimrc, That seems reasonable17:05
pabelangertimrc, agreed17:05
fungigreghaynes: no grep matches for "endpoint not found" (case insensitive) in our nodepool debug logs17:05
*** DericHorn-HP has quit IRC17:05
greghaynesfungi: do we run with debug logs enabled?17:05
timrcI really don't agree with the thinking on orgs in Grafana... We need an extra layer... like "teams" to separate dashboards and delegate authority.17:05
*** e0ne has quit IRC17:05
timrcNot such an issue upstream, though.17:05
*** mrmartin_ has joined #openstack-infra17:05
fungigreghaynes: yep17:05
fungigreghaynes: no matches on get_session_endpoint either17:06
SpamapSgreghaynes: I don't know if that should happen. Not super familiar with it.17:06
*** mrmartin has quit IRC17:06
SpamapSgreghaynes: rather deep in infra-cloud debugging ATM17:06
greghaynesSpamapS: ah, fun17:07
* greghaynes will still ramble but feel free to ignore17:07
*** cindyo has joined #openstack-infra17:07
*** sabeen has joined #openstack-infra17:08
*** samueldmq has quit IRC17:08
*** tonytan4ever has quit IRC17:08
openstackgerritPaul Belanger proposed openstack-infra/grafyaml: Initial unit tests for CLI  https://review.openstack.org/18977017:08
openstackgerritPaul Belanger proposed openstack-infra/grafyaml: Fix --version CLI output  https://review.openstack.org/18978817:08
*** samueldmq has joined #openstack-infra17:08
greghaynesSeems like os_storage_url isnt set for our swift.client - it does get set automagically in my test case and looks like that happens via the shade get_session_endpoint method17:08
greghaynesI guess it might be time to try and run nodepool against rax :/17:09
*** sergmelikyan has quit IRC17:09
*** sergmelikyan has joined #openstack-infra17:10
*** baoli_ has quit IRC17:10
openstackgerritPaul Belanger proposed openstack-infra/grafyaml: Add datasource functionality  https://review.openstack.org/19426517:11
greghayneslooks like theres a case where we would log that we couldnt find the endpoint, but I wouldnt put it past some of the clients to also return None in wierd cases17:11
*** amuller has joined #openstack-infra17:11
amullerG'day! On this wonderful day neutron pep8 job is broken. Pylint is failing, looks like there was no new pylint release, but there was an astroid and logilab-common release today.17:12
amullerGetting some import error in pylint17:12
*** sabeen has quit IRC17:12
amullerI see that we're pinning pylint now in global reqs, which is good17:12
openstackgerritMerged openstack-infra/openstackid: [smarcet]  https://review.openstack.org/19689817:13
amullerI wonder about pinning pylint's major dependencies, I don't know how would that look like or if that's a good idea17:13
amullerlifeless: ^17:13
*** _diana_ has joined #openstack-infra17:14
*** sabeen1 has joined #openstack-infra17:14
*** jasondotstar has quit IRC17:15
*** mpaolino has quit IRC17:15
openstackgerritMerged openstack-infra/yaml2ical: start_date now supported in schedules  https://review.openstack.org/19526217:17
freerunnerdhellmann: Heya! Are u here?17:17
*** sdake has joined #openstack-infra17:18
AJaegerfungi: Haven't looked at the translation problem  yet. Looking at your description, I wonder whether this would hit the requirements proposal as well.17:18
AJaegerfungi, thanks for looking into it.17:18
dhellmannfreerunner: here17:18
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/system-config: Add neutron to single node baremetal0  https://review.openstack.org/19633417:19
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/system-config: Add nova to baremetal0  https://review.openstack.org/19718717:19
freerunnerdhellmann: Yesterday I've talked with morganfainberg about python-keystoneclient releases for juno and kilo and he forwarded me to you.17:19
morganfainbergfreerunner: kilo was just released17:20
dhellmannfreerunner: yes, let's talk about that in #openstack-relmgr-office17:20
morganfainbergfreerunner: i don't thnk we have anything pending on juno17:20
SpamapSpabelanger: ^^ note that there are updates, in case you're working on splitting that into sane files. :)17:20
*** jsavak has joined #openstack-infra17:20
*** mikeit has quit IRC17:21
*** sdake_ has quit IRC17:21
*** kiran-r has quit IRC17:22
*** sabeen1 has quit IRC17:23
sdakemordred fine sir happen to be around?17:24
*** sabeen has joined #openstack-infra17:24
*** dguerri is now known as dguerri`17:25
*** jasondotstar has joined #openstack-infra17:25
openstackgerritMerged openstack-infra/project-config: Add a non-voting py34 check job to os-brick  https://review.openstack.org/19706417:28
*** annegentle has quit IRC17:28
*** dannywilson has quit IRC17:29
openstackgerritJames E. Blair proposed stackforge/gertty: Add missing get_repo calls  https://review.openstack.org/19718917:29
*** sambetts has quit IRC17:29
*** whoops has joined #openstack-infra17:29
asselinHi, I'm looking through puppet-jenkins module, and wondering why the master doesn't need to create the jenkins user? it's created indirectly while installing the plugins17:29
*** notnownikki has quit IRC17:29
*** sambetts has joined #openstack-infra17:29
nibalizertimrc: probably knows asselin17:29
jeblairdstanek: missed a few things with getRepo; fixed in https://review.openstack.org/19718917:30
openstackgerritMerged stackforge/gertty: Add missing get_repo calls  https://review.openstack.org/19718917:30
*** sergmelikyan has quit IRC17:31
fungifanatical support just updated my graphite.o.o ticket and is escalating to operations (presumably to troubleshoot connectivity for the compute node where it's running)17:31
greghaynesfanatical operations?17:31
*** jsavak has quit IRC17:33
fungii think they like to be known as "extremist operations"17:33
*** fawadkhaliq has quit IRC17:33
*** notnownikki has joined #openstack-infra17:33
greghaynesseems pretty scary17:34
*** _diana_ has quit IRC17:35
*** jsavak has joined #openstack-infra17:35
fungii would have also accepted "phrenetical" and "heretical"17:36
openstackgerritDrew Thorstensen proposed openstack/requirements: Add pypowervm to global requirements  https://review.openstack.org/19664117:36
EmilienMpabelanger: I saw your question on #openstack-puppet, (we are in #puppet-openstack now) - no we don't plan to support centos6 at all17:36
EmilienMpabelanger: "beaker jobs for unit testing puppet" > they do *functional* testing17:37
*** kiran-r has joined #openstack-infra17:37
*** baoli has joined #openstack-infra17:37
asselintimrc, do you know what this is for? ssh_authorized_key { 'jenkins-master-2014-04-24':  https://github.com/openstack-infra/puppet-jenkins/blob/master/manifests/jenkinsuser.pp#L5817:37
*** jschlueter has left #openstack-infra17:38
*** yfried has joined #openstack-infra17:38
*** amuller is now known as amuller_afk17:38
*** baoli has quit IRC17:38
*** achanda_ has joined #openstack-infra17:38
*** dboik has quit IRC17:39
*** baoli has joined #openstack-infra17:39
*** dboik has joined #openstack-infra17:39
*** patrickeast has joined #openstack-infra17:39
fungiasselin: the jenkins::jenkinsuser class is what gets applied to all our jenkins slaves, and that authorized key allows a jenkins master to ssh into them so it can push and initialize the slave agent17:40
*** spzala has quit IRC17:40
*** abitha has quit IRC17:40
*** achanda has quit IRC17:40
asselinfungi, to be clear, you mean all jenkins slaves, or just static slaves?17:41
fungiasselin: unless someone has changed things around, the jenkins::jenkinsuser doesn't (or didn't previously) get applied on jenkins masters17:41
*** tqtran has joined #openstack-infra17:41
*** vnod has joined #openstack-infra17:41
asselinfungi, correct, it still doesn't17:41
fungiasselin: i think all, but i'll double-check17:41
*** annegentle has joined #openstack-infra17:41
*** achanda has joined #openstack-infra17:42
*** pradk has joined #openstack-infra17:44
fungiasselin: it's included by the jenkins::slave class which even our openstack_project::single_use_slave nodes instantiate17:44
fungiasselin: unless user=false gets passed in17:44
*** achanda_ has quit IRC17:44
fungito be clear, openstack_project::single_use_slave includes openstack_project::slave_common which includes jenkins::slave which includes jenkins::jenkinsuser17:45
AJaegerfungi, are you looking into the translation stuff? I have one idea and can propose a change...17:45
*** mrmartin_ has quit IRC17:45
*** kiran-r has quit IRC17:45
fungiAJaeger: i was about to get started on it, but feel free if you want to give it a try and i'll manually test it by rerunning one of those jobs with your modification in place17:46
*** sputnik13 has joined #openstack-infra17:46
*** sdake_ has joined #openstack-infra17:47
AJaegerthanks, fungi!17:48
AJaegerchange will come in the next 10 mins - after I done some basic testing myself17:48
fungii'll be watching17:48
*** fawadkhaliq has joined #openstack-infra17:50
*** ivar-lazzaro has joined #openstack-infra17:50
*** sdake has quit IRC17:50
*** sdake_ is now known as sdake17:50
*** xarses has quit IRC17:51
*** ihrachyshka has joined #openstack-infra17:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix proposal operations  https://review.openstack.org/19720017:52
AJaegerfungi: here's the patch to use git checkout -B as you suggested17:52
*** vnod has quit IRC17:52
AJaegerfungi: This works fine - but I could not reproduce the failure I got, so I might have tested something else ;(17:52
AJaegerFor me, git review always worked, even in detached state ;( Even when I updated to latest git review from pypi17:53
fungiAJaeger: i'll give it a try. should i rerun nova-propose-translation-update with it, or is there a better job to try first?17:53
*** HeOS has quit IRC17:53
AJaegernrun glance-propose-translation-update - that's a smaller repo and might be faster ;)17:53
fungithanks, giving it a try now17:54
timrcasselin, Sorry I had to resolve an incident in BigCorpLand.  Reading up.17:55
*** mrmartin has joined #openstack-infra17:56
timrcasselin, Ah it looks like fungi already addressed your question?17:56
timrcI wish my phone was as responsive as fungi :)17:56
*** chlong has quit IRC17:56
*** alivigni has joined #openstack-infra17:57
*** enikanorov2 has quit IRC17:57
openstackgerritJames Slagle proposed openstack-infra/irc-meetings: Update the TripleO meeting schedule  https://review.openstack.org/19720217:57
openstackgerritJohn Griffith proposed openstack-infra/project-config: Add config for Cinder LVM Thin configuration  https://review.openstack.org/18249917:57
*** baoli has quit IRC17:58
*** e0ne has joined #openstack-infra17:58
*** yamahata has joined #openstack-infra17:59
*** baoli has joined #openstack-infra17:59
AJaegerfungi, if you look at https://jenkins.openstack.org/job/glance-propose-translation-update/55/consoleFull, you see that it still complains about "A new version of git-review is availble on PyPI. Please update your copy...".17:59
AJaegerfungi: Is there a chance to easily upgrade git-review on the proposal node so that we don't see this warning anymore?17:59
*** amuller_afk is now known as amuller17:59
fungiAJaeger: yeah, we probably specify a version in our puppet manifest18:00
docaedoWho else is looking forward to July 9th when Wo gets back in the office? I can't wait!18:00
nibalizerEmilienM: haha your upgrade jobs leaked into infras beaker use too18:00
*** jsavak has quit IRC18:01
EmilienMnibalizer: is it a big deal?18:01
*** dannywilson has joined #openstack-infra18:01
fungidocaedo: i've already marked it on the calendar18:01
*** tonytan4ever has joined #openstack-infra18:01
EmilienMnibalizer: I honestly think they can help18:01
EmilienMnibalizer: though it requires a fix to work properly (cc crinkle)18:01
nibalizerwhat is the purpose of the upgrade test?18:02
pabelangerEmilienM, okay, perfect18:02
EmilienMnibalizer: 1/ run beaker on top of HEAD^1 2/ run beaker again on top of the patchset18:02
AJaegerfungi: thanks to hound, I found: http://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/slave.pp - requesting 1.1718:02
EmilienMand see if the resources survive18:02
fungiAJaeger: wow, that's crazy ancient18:02
EmilienMit's a grenade like18:02
pabelangerEmilienM, Trying to see what is actually using centos6 within -infra, so far only puppet-apply is what I see18:02
*** otter768 has joined #openstack-infra18:03
* AJaeger prepares a change...18:03
pabelangerwhich will be removed once beaker is working18:03
fungipabelanger: you mean jobs or other things too?18:03
pabelangerfungi, mostly puppet jobs for now18:03
fungipabelanger: we have more than a few python 2.6 unit tests jobs running on centos 6 for example. we also have our asterisk pbx and our git servers and haproxy for them on centos 618:04
pabelangerfungi, however, anything using git in centos6 will be affected with performance issues18:04
pabelangerfungi, right, I knew of asterisk and git using centos 618:04
*** e0ne is now known as e0ne_18:04
*** cindyo has quit IRC18:05
*** notnown20 has joined #openstack-infra18:05
openstackgerritAndreas Jaeger proposed openstack-infra/puppet-jenkins: Update git-review  https://review.openstack.org/19720318:05
*** jlanoux has joined #openstack-infra18:05
*** notnownikki has quit IRC18:06
*** geoffarnold has quit IRC18:06
*** notnownikki has joined #openstack-infra18:07
*** e0ne_ is now known as e0ne18:07
*** xarses has joined #openstack-infra18:07
AJaegerlifeless: we wondered recently about this error message, here's a patch to fix it ^18:07
*** otter768 has quit IRC18:07
pabelangerso, am I missing something or can't you abandon a patchset in gertty?18:08
AJaegerfungi, I see a change from the translation bot!18:08
pabelangerCTRL-a it look slike18:08
AJaegerfungi, is the log file available somewhere or did you run this manually?18:09
nibalizerEmilienM: oh okay18:09
*** ivar-laz_ has joined #openstack-infra18:09
nibalizeri thought it was like try kilo then try liberty18:09
*** ivar-laz_ has quit IRC18:09
*** ivar-lazzaro has quit IRC18:09
fungiAJaeger: disabled puppet and reran manually on that worker with your one-liner patch applied. result just posted at https://review.openstack.org/19720618:10
*** ivar-lazzaro has joined #openstack-infra18:10
openstackgerritMerged openstack-infra/puppet-openstackci: Fix puppet deprecated warning  https://review.openstack.org/19525618:10
*** notnown20 has quit IRC18:10
*** Akshay00 has joined #openstack-infra18:10
EmilienMnibalizer: it's long term goal18:10
nibalizerya that is pretty useful i think18:10
nibalizerI'll have to noodle on it18:10
EmilienMnibalizer: but we don't have beaker in stable/juno18:10
EmilienMnibalizer: I'll propose such a change after our release18:10
*** spzala has joined #openstack-infra18:11
*** SlickNik_ has quit IRC18:11
*** vnod has joined #openstack-infra18:11
*** SlickNik_ has joined #openstack-infra18:12
jeblairyeah, the branch->branch upgrade case is handled fairly well in zuul-cloner because of grenade, but we don't have many (or perhaps any current) jobs doing commit->commit differences18:12
jeblair(that would probably be a good enhancement to zuul-cloner)18:12
AJaegerfungi, thanks a lot for testing!18:12
EmilienMjeblair: yeah that's why it's just for experimentation now18:12
EmilienMjeblair: when we cut our kilo branch, I'll change the workflow to do branch2branch18:13
jeblairoh cool18:13
AJaegerjeblair, could you review https://review.openstack.org/197200 as well to fix the proposal bot scripts, please?18:13
*** annegentle has quit IRC18:13
nibalizerEmilienM: does it leverage catalog-diff?18:13
Akshay00When we try to to run git review we get an error that says we need to update our current contact information. But when we try to enter in our contact info it says it cannot be updated. Would anyone know about this?18:13
*** ericksonsantos has joined #openstack-infra18:14
EmilienMnibalizer: do you mean, test if it's idempotent ?18:14
jeblairAJaeger: er, how did that break?18:14
Akshay00My colleague and I are currently interning at Intel18:14
AJaegerjeblair: see https://jenkins.openstack.org/job/glance-propose-translation-update/55/consoleFull . I don't fully understand the breakage but it seems related to the fact that we start with a detached state18:15
fungiAkshay00: https://ask.openstack.org/question/5672018:15
*** annegentle has joined #openstack-infra18:16
fungiAkshay00: though the short answer is to make sure you're following the instructions listed at http://docs.openstack.org/infra/manual/developers.html#account-setup thoroughly and in the sequence described18:16
*** Sukhdev_ has joined #openstack-infra18:16
AJaegerfungi: regarding the git-review update for https://review.openstack.org/197203: I'm fine with using latest as well. Shall I update?18:17
fungiAJaeger: go for it18:17
*** olaph has joined #openstack-infra18:17
*** e0ne is now known as e0ne_18:18
*** ildikov has joined #openstack-infra18:18
jeblairAJaeger: i feel like i will need to do about an hour of research before i can review that :/18:18
fungialso i have hopes that https://review.openstack.org/190825 may reduce the number of people who end up here asking about contact information update errors from gerrit, if anyone gets a chance to eventually review and approve it18:18
*** puranamr_ has quit IRC18:18
*** jason10258 has joined #openstack-infra18:19
jeblairfungi: oh, huh, i thought i had.  +2.18:19
openstackgerritAndreas Jaeger proposed openstack-infra/puppet-jenkins: Always install latest git-review  https://review.openstack.org/19720318:20
fungiAJaeger: i'm going to test latest git-review with the nova translation updates just to make sure it's not hiding any surprises18:21
*** achanda has quit IRC18:21
*** e0ne_ is now known as e0ne18:23
*** gyee has joined #openstack-infra18:24
AJaegerfungi, thanks!18:25
fungiAJaeger: https://review.openstack.org/197213 but seems to have updated a lot of linenumber comments. is that expected?18:25
Akshay00fungi thank you I will try to work through these links although I have tried the second one already18:25
timrcfungi, Does upstream throttle reviews?  We just had a user create hundreds of reviews (on accident)18:26
*** shardy has quit IRC18:27
timrcAFAICT there has never been a legitimate need to do that... would be good for Gerrit to say "Uh, no" when someone tries to do that.18:27
fungitimrc: we do not, no18:27
*** shardy has joined #openstack-infra18:27
*** jasondotstar has quit IRC18:27
AJaegerfungi: yeah, it is ;( We should review at one point whether we really need the pot files (source). We removed the line numbers from the po files because of that noise a few weeks ago18:27
fungitimrc: if a feature like that existed in gerrit, what threshold would you set it at?18:28
AJaegerfungi, strange things are happening: https://jenkins.openstack.org/job/api-site-manuals-upstream-translation-update/100/console18:28
timrcfungi, !insane... dunno... I would need to look at it heuristically... there is probably some reasonable upper bound.18:28
fungii'm sort of curious. i mean we have devs (who shall remain nameless) intentionally push 50+ change series topic branches somewhat regularly18:29
*** wenlock_ has quit IRC18:29
AJaeger"Exception: A strange error happened."18:29
*** jlanoux has quit IRC18:29
timrcfungi, If there is a legitimate use case then it should be supported, but safe guards to prevents accidents that can bring zuul to a grinding halt would be a Good Thing (tm) in my opinion.18:29
openstackgerritMerged openstack-infra/infra-manual: Update account setup to specify foundation member  https://review.openstack.org/19082518:30
openstackgerritMerged openstack-infra/jenkins-job-builder: Make include_path deterministic  https://review.openstack.org/18707518:30
*** maishsk_afk has joined #openstack-infra18:30
nibalizerasselin: i've completed my voting on openstackci so if you want to try to schedule a time for that stuff I'm in18:30
nibalizeri wont be able to infra meeting today18:30
*** maishsk_afk is now known as maishsk18:30
AJaegerfungi, https://jenkins.openstack.org/job/openstack-manuals-manuals-upstream-translation-update/ failed the same way ;( I'll try to reproduce tomorrow if it still fails...18:31
fungitimrc: out of curiosity, was the user running git-review? it provides a warning when you're pushing more than one commit, so it would have given them the number of new changes they were creating18:31
timrcfungi, Yeah this was done through git-review.18:31
timrcfungi, Merged a bunch of commits to a new branch and then git-review'ed and it created a review per commit.18:32
fungiand asked them to confirm18:32
*** samuelBartel has joined #openstack-infra18:32
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: rename puppet-syntax to puppet-parser-validation  https://review.openstack.org/19721618:32
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: rename puppet-syntax-future to puppet-syntax  https://review.openstack.org/19721718:32
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job  https://review.openstack.org/19721818:32
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches  https://review.openstack.org/19721918:32
EmilienMnibalizer, _ody ^18:32
fungitimrc: well, also merge commits are often a sign something is being done wrong. were your acls set to allow them to propose merge commits?18:32
*** Akshay00 has quit IRC18:32
*** ZZelle_ has joined #openstack-infra18:33
fungitimrc: we _do_ disallow merge commits by default, except generally to release managers maintaining merges in and out of feature branches18:33
timrcfungi, Looks to be.  I'm just now starting to look into root cause of this snafu.  Generally speaking we give people enough rope to hang themselves and the rest of us with it seems :)18:33
*** enikanorov2 has joined #openstack-infra18:34
EmilienMcrinkle: you might want to see https://review.openstack.org/#/q/topic:puppet-jobs-refacto,n,z18:34
*** maishsk_afk has joined #openstack-infra18:34
*** Akshay00 has joined #openstack-infra18:35
*** kushal has quit IRC18:35
*** maishsk has quit IRC18:35
*** rossella_s has quit IRC18:37
*** rossella_s has joined #openstack-infra18:37
*** pradk has quit IRC18:37
*** olaph has quit IRC18:38
*** Akshay00 has quit IRC18:38
fungiAJaeger: that tx/txclib (transifex client?) exception is indeed strange and useless. i'm not sure what to make of it, but i'm going to guess something in the content of those projects is triggering an error in whatever version of txclib ended up on the new proposal worker18:38
*** Akshay00 has joined #openstack-infra18:38
AJaegerfungi: tx is the transifex client. Let me check whether I can reproduce it on my system...18:39
*** davideagnello has quit IRC18:39
*** cindyo has joined #openstack-infra18:39
fungii mean, an error message at least _describing_ what was happening in that function call would be far more useful than "Exception: A strange error happened.18:40
AJaegerfungi: indeed18:40
* AJaeger can't find that error message in the client18:41
*** ivar-lazzaro has quit IRC18:41
fungiAJaeger: it's being raised from a call to push self.get_full_path(sfile) -> do_url_request so i'd first suspect some odd characters in file/directory names which aren't getting properly munged into a url?18:41
fungiAJaeger: the error seems to be coming from txclib which tx calls into18:42
*** ivar-lazzaro has joined #openstack-infra18:42
timrcfungi, I'm not sure this was even a proper merge.  It looks like the person cherry-picked individual changes into a new branch and then git-review'ed em.18:42
fungitimrc: oh! so probably intentional18:43
*** annegentle has quit IRC18:43
*** notnownikki has quit IRC18:43
fungitimrc: we definitely have devs do that though it's generally in the sub-100 change order of scale. and we do have quota of ~900 job workers to chew through them when it happens18:44
timrcfungi, Yeah... but... but why.18:44
*** davideagnello has joined #openstack-infra18:44
fungii've personally scripted pushing 200+ new changes at a time on multiple occasions, though i tend to wait until the ci system is mostly idle18:45
*** xarses has quit IRC18:45
fungiand i don't do them a a dependent change series18:45
fungier, as a18:45
*** xarses has joined #openstack-infra18:45
*** sarob_ has joined #openstack-infra18:45
timrcfungi, What's weird(er) is that 300+ changes hit gerrit but we had something like 79K jobs in zuul (I didn't confirm that myself)18:45
*** xarses has quit IRC18:45
fungithat does sound weird18:46
fungimaybe zuul's event/result queue spiked up to 79k?18:46
*** sdake has quit IRC18:46
*** geoffarnold has joined #openstack-infra18:46
*** sdake has joined #openstack-infra18:47
timrcMore appropriately maybe 79k changesets in the queue -- not sure how that happened :/18:47
AJaegerfungi, reproduced it locally...18:47
*** kushal has joined #openstack-infra18:48
AJaegerfungi, pushing the failing file again, everything is fine.18:48
AJaegerI wonder whether transifex is broken ;(18:49
fungiAJaeger: given it's an opaque proprietary service, it's especially hard to know18:49
AJaegerfungi: yeah ;(18:50
fungigraphite's ipv4 connectivity _just_ came back up18:50
fungino word from phrenetical support yet on what they did to fix it18:51
*** yamamoto has quit IRC18:51
AJaegermorning, jhesketh !18:51
fungijhesketh: enjoyed those 3 hours of sleep i hope?18:51
greghaynesyea, werent you just here jhesketh ?18:52
jheskethfungi: fantastic ;-)18:52
AJaegerfungi: the post queue is still not moving forward, top job is there since 11hours. Do we have a problem?18:52
jheskethI'll probably snooze again after the meeting, otherwise lots of coffee today :-)18:53
fungiAJaeger: i'll check into that next--thanks for the reminder18:53
AJaegerfungi: thanks a lot!18:53
openstackgerritAssaf Muller proposed openstack/requirements: Update pylint pin to 1.4.4  https://review.openstack.org/19722618:53
*** vnod has quit IRC18:54
fungiAJaeger: my guess is we've somehow accumulated a lot of deadweight workers in nodepool or something bringing all the low-priority jobs to a halt since the few available workers are being snatched up by the higher priority queues, but i'm digging in now18:54
AJaegerfungi, now the strange error happened at a different file. I'll keep an eye on this, might need to raise a bug with transifex18:54
greghayneshrm, now im especially confused - my nodepool appears to be uploading a dib image to rax just fine18:55
*** jasondotstar has joined #openstack-infra18:55
*** sarob_ has quit IRC18:56
openstackgerritMerged openstack-dev/cookiecutter: Add insentive to write a long description  https://review.openstack.org/19714618:56
fungiAJaeger: yep, we have ~100 nodes in a ready state for more than an hour at the moment18:56
*** Longgeek has quit IRC18:57
AJaegerfungi, looking at the txlib source code: The Exception comes from an error message from the remote transifex server ;(18:58
fungii'll keep working on this in the background while we're meeting18:58
*** vnod has joined #openstack-infra18:58
AJaegerfungi: mmh, that looks nice18:58
AJaegerfungi, thanks for the reminder of the meeting ;)18:58
*** Longgeek has joined #openstack-infra18:58
*** hamzy has quit IRC18:59
AJaegerfungi, third time pushing the *same* content to transifex, it did not fail but succeeded...19:00
jeblairmeeting time!19:00
fungiAJaeger: yep, i bet transifex is broken19:00
* AJaeger won't bet against fungi on this19:01
*** kgiusti has left #openstack-infra19:01
*** hamzy has joined #openstack-infra19:03
*** dboik has quit IRC19:04
*** dboik has joined #openstack-infra19:04
fungioh wow, >450 floating ips in hpcloud with no associated instance. cleaning those up now19:05
*** fallenpegasus has quit IRC19:05
mordredfungi: yay!19:06
*** vnod has quit IRC19:06
*** Guest80179 has joined #openstack-infra19:06
greghaynesmordred: so, im having a pretty painful time trying to replicate that traceback locally19:08
*** maishsk_afk has quit IRC19:08
greghayneser, context19:08
greghaynesthe nodepool swift traceback19:08
mordredgreghaynes: oh awesome. that's great to hear19:08
*** _nadya_ has joined #openstack-infra19:09
*** achanda has joined #openstack-infra19:10
*** maishsk_afk has joined #openstack-infra19:10
funginot so great for solving our upload failures though19:10
*** geoffarnold has quit IRC19:10
greghaynesI did notice that the only way it happens is if swift client doesnt get an object_storage_url specified in os_options19:10
*** dguerri` is now known as dguerri19:11
*** _nadya__ has joined #openstack-infra19:11
greghaynesand in my shade test that is being set via the shade get_session_endpoint... so I wonder if theres something about our nodepool config which is causing that to fail19:11
*** _nadya_ has quit IRC19:11
greghaynesnot sure if that sends of any lightbulbs...19:11
mordredhrm. lemme go look at something19:12
*** shardy has quit IRC19:13
*** e0ne is now known as e0ne_19:14
*** Guest80179 has quit IRC19:14
*** e0ne_ is now known as e0ne19:15
*** dguerri is now known as dguerri`19:15
greghaynesmordred: btw -  what im referring to re: object_storage_url is line 44 on http://paste.openstack.org/show/321629/ - that only happens if self.url is none which seems like something shade should be passing in19:16
*** tonytan4ever has quit IRC19:16
*** puranamr has joined #openstack-infra19:17
*** puranamr has quit IRC19:18
*** puranamr has joined #openstack-infra19:18
mordredgreghaynes: ZOMG19:18
*** jason10258 has quit IRC19:18
mordredgreghaynes: there is no swift endpoint in ORD19:18
mordredin rax19:19
greghaynesthat would explain why I cant replicate19:19
fungihow do we...19:19
fungii mean...19:19
greghaynesbecause I dont have access to ORD19:19
mordredso -19:19
fungii don'19:19
fungit even19:19
mordrednext quesion - can we import an image into glance in ORD from swift in IAD?19:19
mordrednikhil_k-away: ^^ ???19:19
mordredgreghaynes: so - shade is working19:19
mordredgreghaynes: or - perhaps we should throw an exception there if endpoint is None19:20
mordredso that the error is clearerer19:20
greghaynesYea, definitely - looks like our get endpoint method tries to but for some reason this case it just returns none19:20
greghaynesanyhow, that is amazing19:21
greghaynesmordred: hrm, we dont specify a region or anything AFAIK when we do the glance task-create, so not sure how we would pull from another region...19:22
*** geoffarnold has joined #openstack-infra19:23
*** cindyo has quit IRC19:25
mordredgreghaynes: it's a good question19:26
*** jason10258 has joined #openstack-infra19:26
*** jsavak has joined #openstack-infra19:26
greghaynesok, im going to make a patch to not do the dib.ubuntu-trusty in ORD for now19:26
greghaynesbecause thers not a whole lot else we can do19:26
openstackgerritPaul Belanger proposed openstack-infra/infra-specs: Host trystack.o.o site  https://review.openstack.org/19509819:27
pc_mmtreinish: hi19:29
*** tonytan4ever has joined #openstack-infra19:30
openstackgerritThanh Ha proposed openstack-infra/nodepool: Clarify meaning of Jenkins master configuration settings in target  https://review.openstack.org/16787819:31
*** rm_work is now known as rm_work|away19:31
mtreinishpc_m: hello19:31
pc_mmtreinish: I'm still battling with getting functional jobs running that can reference local neutron repo with changes.19:32
mtreinishpc_m: what was the link to your patch again?19:32
pc_mmtreinish: Here's what I have... committed code passes Jenkins.19:32
pc_mmtreinish: In my local area however, both functional tests fail.19:32
*** rm_work|away is now known as rm_work19:33
mtreinishpc_m: your local env is screwed up then :)19:33
pc_mmtreinish: What I see is that in .tox/dsvm-functional/bin, it is missing many files, especially the neutron-rootwrap and neutron-rootwrap-daemon files that are needed for these tests.19:33
*** HeOS has joined #openstack-infra19:33
*** crc32 has joined #openstack-infra19:33
*** ivar-lazzaro has quit IRC19:34
pc_mmtreinish: Do you know how that bin dir gets populated?19:34
*** armax has quit IRC19:34
*** dimtruck is now known as zz_dimtruck19:34
*** aysyd has quit IRC19:34
mtreinishpc_m: pip will put things in there when it installs a package19:34
openstackgerritThanh Ha proposed openstack-infra/nodepool: Add prerequisites for nodepool on RHEL7  https://review.openstack.org/17070019:35
mtreinishlike when it installs neutron it should install all the console_scripts in neutron's setup.cfg in there19:35
openstackgerritThanh Ha proposed openstack-infra/nodepool: Add MySQL DB creation example  https://review.openstack.org/17069919:35
*** ivar-lazzaro has joined #openstack-infra19:35
openstackgerritgreghaynes proposed openstack-infra/system-config: Remove ubuntu-trusty from rax ORD  https://review.openstack.org/19723819:36
pc_mmtreinish: So, it the working case, where we install neutron via master (https:), it gets placed in .tox area with source is src subdir. Does it also place bin and etc files too?19:36
hogepodgewiki.openstack.org is down19:37
hogepodge(for me at least)19:37
pc_mmtreinish: Wondering how to handle that for this case, where we are not installing neutron in the venv.19:38
hogepodgenevermind, it's back19:38
*** xyang1 has quit IRC19:38
fungihogepodge: i'll check on it anyway just in case19:38
*** xyang1 has joined #openstack-infra19:38
pc_mmtreinish: Thinking out loud. One option may be to copy the needed files over, as part of the rootwrap_deploy.sh script.19:39
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: refactorize puppet-syntax job  https://review.openstack.org/19721619:40
pc_mmtreinish: Another would be to refer to the scripts in the Neutron area via the OS_ROOTWRAP_CMD, but use the venv .conf file (not sure if these files get tweaked though)19:40
pc_mmtreinish: I guess they don't.19:41
mtreinishpc_m: neither of those is anything you want to do.19:41
pc_mmtreinish: Suggestion?19:41
pc_mmtreinish: Currently, the rootwrap_deploy.sh creates the .conf file, and moves the filters into the .tox area.19:42
*** jason10258 has quit IRC19:43
mtreinishpc_m: you'll have to give me a min. I'm in the middle of a couple things19:43
pc_mmtreinish: The tox.ini runs the rootwrap scripts from the .tox/*/bin area and uses the .conf file in etc.19:43
mtreinishI need to take a look at what that script is doing19:43
pc_mmtreinish: sure...19:43
mtreinishbut my gut feeling is that something is wrong in your env (no idea what) but since the gate job passes it probably means that neutron is properly packaging what you need and pip is installing it in the venv for the run19:44
*** nihilifer has quit IRC19:44
*** maishsk_afk is now known as maishsk19:44
pc_mmtreinish: That was another question... why it works in gate job. FYI: On one VM, I wiped out /opt/stack and restacked with DevStack to get a fresh environment. Tested it worked, then patched in 196237, removed .tox and tried and it faile=s there too.19:47
*** tqtran is now known as tqtran-afk19:47
*** john-davidge has quit IRC19:48
*** baoli has quit IRC19:48
*** jsavak has quit IRC19:49
mordredjeblair, fungi, greghaynes: RAX ORD doesn't have swift. we also only have 55 nodes of capacity there ... perhaps we should just kill it19:49
greghaynesmordred: kill it with fire19:49
jeblairmordred: !19:49
greghaynesmordred: does rax support glance import from http url?19:50
*** aysyd has joined #openstack-infra19:50
mordredjeblair: SYD and HKG have swift. we could maybe add one of them if we're worried about the capacity19:50
notmynamemordred: RAX ORD does have swift19:50
*** dboik has quit IRC19:50
mordrednotmyname: it does not show up in my keystone catalog19:51
*** dboik has joined #openstack-infra19:51
notmynamemordred: RAX has swift in ORD, DFW, HKG, SYD, IAD, LON19:51
mordrednotmyname: I show DFW, IAD, HKG and SYD19:51
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job  https://review.openstack.org/19721819:51
*** Rockyg has joined #openstack-infra19:51
*** yamamoto has joined #openstack-infra19:52
mordrednotmyname: oh - weird - I DO see it in the openstackci account. hrm19:52
dstanekjeblair: nice. i didn't realize that all the gertty stuff was getting approved. thx!19:52
*** pblaho has quit IRC19:52
pleia2oops, switching from openstackid-dev to openstackid.org broke zanata login19:53
pleia2^[[0m^[[0m2015-06-30T19:44:43,727Z INFO  [org.openid4java.discovery.html.HtmlResolver] (default task-16) HTML discovery completed on: https://openstackid.org/19:53
pleia2^[[0m^[[0m2015-06-30T19:44:43,727Z INFO  [org.openid4java.discovery.Discovery] (default task-16) Discovered 0 OpenID endpoints.19:53
jeblairdstanek: i merged a couple, but have some comments on the next one19:53
openstackgerritMerged openstack/requirements: Require fixtures 1.3.1.  https://review.openstack.org/19686319:53
mordredgreghaynes: ok - we may have to dig further. we're winding up with endpoint = None I believe19:53
pleia2did I get the endpoint wrong?19:53
greghaynesmordred: oh? You see the endpoint listed somewhere?19:53
dstanekjeblair: i'll take a look and get them addressed later today19:53
mordredgreghaynes: yah19:53
greghaynesmordred: I dont have ORD access at all, is that another just ask nicely to get thing?19:53
mtreinishpc_m: so looking at deploy_rootwrap script itself there is nothing really wrong there19:54
*** yamahata has quit IRC19:54
pc_mmtreinish: Yeah, it seems to be doing the right thing to me.19:54
mordredgreghaynes: maybe?19:54
*** yamahata has joined #openstack-infra19:54
mordredgreghaynes: http://paste.openstack.org/show/329514/19:54
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements  https://review.openstack.org/18682919:54
openstackgerritOpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements  https://review.openstack.org/19724919:54
mtreinishpc_m: but the way your calling it your tox.ini has a hardcoded expectation that neutron is cloned in the dir as neutron-vpnaas19:54
*** achanda has quit IRC19:54
mtreinishwhich is not true in every case19:54
mordredgreghaynes: it's also possible that we need to do something with that tenantId from the catalog19:54
mtreinishI expect you're local env is that way while the gate is not19:55
greghaynesmordred: so easy fix for now is that we can pass in that that url manually19:55
*** aysyd has quit IRC19:55
pc_mmtreinish: I din't quite follow that.19:55
*** achanda has joined #openstack-infra19:55
mordredgreghaynes: are you sure we're getting None for endpoint there?19:56
pc_mmtreinish: did you mean that it has the expectation that neutron is a peer to neutron-vpnaas?19:56
greghaynesmordred: No, I cant actually replicate, but there is a if url is None right above that line in the traceback which causes this19:56
mtreinishpc_m: I'll leave a comment in the review19:56
*** yamamoto has quit IRC19:56
mordredgreghaynes: I don't see if url is None19:57
*** cindyo has joined #openstack-infra19:57
mordredgreghaynes: line number?19:57
mordredoh! gotcha19:58
fungithis is interesting...19:58
pc_mmtreinish: I don't see how that can affect this issue, as, in the local area, neutron is in a peer dir to neutron-vpnaas19:58
mordredgreghaynes: OH!!!!!19:58
mordred                "type": "object-store",19:58
mordredoh - nevermind19:58
greghaynesfungi: Do you know if the other regions worked for our nodepool uploads/19:59
*** fawadkhaliq has quit IRC19:59
greghayneslocally they seem to work for me...19:59
*** zz_dimtruck is now known as dimtruck19:59
fungi"I was able to determine that your IP address had been null routed at our AGGRs due to an apparent glean attack. It appears that while your graphite server was frozen, there was a large influx of traffic. As there was no MAC for this IP address in the AGGR's arp cache, and your cloud server was not responding, all traffic was forwarded to the AGGR's CPU for processing causing performance degradation19:59
fungifor other customers."19:59
*** achanda has quit IRC19:59
fungigreghaynes: it did not appear so, but i can check the upload for, say, iad to confirm20:00
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job  https://review.openstack.org/19721820:00
pleia2mrmartin: perhaps you can help me? having some trouble with openstackid.org20:01
*** abregman has joined #openstack-infra20:01
mtreinishpc_m: you should *not* have the test commands in the tox.ini have a hard coded dependency on the location of the neutron repo20:01
mrmartinpleia2: what's wrong?20:01
mtreinishthat's just going to cause issues20:01
pleia2mrmartin: I switched translate-dev.openstack.org from openstackid-dev.o.o to openstackid.org, now it's not finding endpoints (see my comments about 8 minutes ago)20:01
pc_mmtreinish: OK. How do I determine where neutron is located?20:02
mtreinishyou need to make sure things are properly packaged in neutron, or included in the neutron-vpnaas repo20:02
fungigreghaynes: sorry but same traceback in iad and dfw20:02
mtreinishpc_m: you shouldn't need to know that. That's what I'm saying, requiring a specific path outside of the repo is what is causing your issues20:02
pc_mmtreinish: I think you're starting to lose me...20:02
mrmartinpleia2: is everything ok with the SSL certificates? it is using a different X50920:03
greghaynesthat does help a lot though - so I should be able to replicate20:03
*** agentle has joined #openstack-infra20:03
*** patrickeast_ has joined #openstack-infra20:03
mrmartinpleia2: maybe you should check you can open a simple SSL connection to openstackid.org:44320:03
pleia2mrmartin: it's a self signed cert :\ but it was with openstackid-dev too20:03
*** matbu_ has joined #openstack-infra20:03
mrmartinpleia2: openstackid.org using a real cert I guess, and I'm not sure it is included in java's keystore20:03
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements  https://review.openstack.org/19650420:03
*** otter768 has joined #openstack-infra20:03
*** patrickeast has quit IRC20:04
*** patrickeast_ is now known as patrickeast20:04
pc_mmtreinish: Sure, and our goal is to separate the two, making neutron a library, but we are far from being able to do that currently. How do we deal with it in the interim.20:04
mrmartinpleia2: but it is just a quick tip based on my previous java / ssl experience20:04
pleia2mrmartin: hm, it worked ok with openstackid-dev20:04
AJaegerpleia2: could you review https://review.openstack.org/#/c/197200/ , please?20:04
mtreinishpc_m: that's not what I'm talking about20:04
mtreinishpc_m: this is about where the configuration files for running commands in the test runs live20:04
mrmartinpleia2: I see, dev is using the similar Comodo CA as an issuer20:05
fungipleia2: yeah, not snakeoil/self-signed on the production openstackid.org20:05
mtreinishpc_m: right now in the tox.ini you have a hard coded dependency on the neutron repo living somewhere20:05
mrmartinfungi: but we have a proper cert at openstackid-dev.o.o too20:05
fungipleia2: i installed a cert bought from the same ca i use for our other https sites that aren't snakeoil20:05
*** jasondotstar has quit IRC20:05
pleia2fungi: sorry, I meant translate-dev.o.o itself is self-signed20:05
pc_mmtreinish: Soa re you saying to clone all the .conf, rootwrap scripts from neutron to VPN repo?20:06
mtreinishpc_m: that's not something you want to be doing, you need to make sure neutron either installs that into a known location (so after you pip install neutron in the venv you know where the file lives)20:06
fungipleia2: oh, good point. that may pose a problem since i think openstackid does a callback to the url?20:06
mrmartinpleia2: I see some banned ips, with auth errors on openstackid.org20:06
mtreinishpc_m: or you keep a local copy in the vpnaas repo that you can point to directly (in the alternative you could have a script generate it I guess)20:06
mrmartinpleia2: I did a cleanup now, so please try again to auth20:07
*** armax has joined #openstack-infra20:07
mtreinishpc_m: but saying neutron lives at ../neutron and crossing your fingers is not a good idea20:07
pleia2mrmartin: still failing20:07
pc_mmtreinish: understood, and I was wondering how to handle that.20:07
openstackgerritMonty Taylor proposed openstack-infra/shade: Output some debug information for swift endpoints  https://review.openstack.org/19729220:07
pc_mmtreinish: In this case, however, option #1 is not possible, right?20:07
mrmartinpleia2: I don't have access to that server, but if you have one, what you think about checking the logs at /srv/.../app/storage/ ?20:08
SpamapS\o/ nova talks to me now20:08
* SpamapS builds an image and enrolls a server20:08
pleia2mrmartin: the translate-dev server?20:08
*** otter768 has quit IRC20:08
*** aysyd has joined #openstack-infra20:08
pc_mmtreinish: We're not installing neutron, because we're trying to be able to use an already installed neutron (so that we get the change set that may be present there), right?20:08
mrmartinpleia2: no, the openstackid.org server20:08
*** dimtruck is now known as zz_dimtruck20:08
pleia2mrmartin: ah, sure, sec20:08
mtreinishpc_m: no you're installing neutron, that's what tox_install.sh is ensuring20:09
mrmartinpleia2: maybe you can see some exceptions there20:09
pleia2mrmartin: thanks, looking20:09
mtreinishit either uses a copy already present in the venv, leverages zuul-cloner if it's present to get an up-to-date neutron repo (which is a dirty hack) and installs that in the venv, or locally checks it out from git and installs that in the venv20:09
mrmartinpleia2: the two site have different cert issuer20:10
pc_mmtreinish: OK, well, as it stands right now, in a local environment, when I run the functional test it is not getting installed.20:10
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches  https://review.openstack.org/19729420:10
*** gokrokve has joined #openstack-infra20:10
mtreinishpc_m: which comes back to my original statement, something is weird in your local env :)20:11
mrmartinpleia2: the dev was issued by: PositiveSSL CA 2 and the openstackid.o.o cert was issued by COMODO RSA Domain Validation Secure Server CA20:11
*** erikwilson_ has quit IRC20:11
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches  https://review.openstack.org/19721920:11
mrmartinpleia2: so I still suggest to check the default java cert keystore20:11
pc_mmtreinish: Are you saying that when this runs in the gate... the .tox/dsvm-functional/ area will have neutron source, bin, and etc?20:12
pleia2java itself isn't complaining directly about ssl20:12
*** maishsk has quit IRC20:12
mtreinishpc_m: neutron is being installed in the venv used for running the jobs. tox_install.sh is calling zuul-cloner to update the locally checked out copy of neutron and then installing that in the venv20:13
*** frobware_ has joined #openstack-infra20:13
*** sergmelikyan has joined #openstack-infra20:13
mtreinishtbh, I'm not sure how it has perms to do that in the gate env, but it doesn't appear to error out20:13
mtreinishso I'm assuming it's all working20:14
*** nihilifer has joined #openstack-infra20:14
pc_mmtreinish: ah, so in the local area, doing "pip install -U $*" is not installing.20:14
fungimrmartin: pleia2: yep, the change in issuer is because the ca we're using converted from 1024->2048 bit certs between when i bought those20:14
mrmartinpleia2: and do you know what is this openid discovery process exactly? I guess it must be a simple GET request20:14
*** maishsk has joined #openstack-infra20:14
mtreinishpc_m: well that command is used to install any other requirements20:14
mordredgreghaynes: https://review.openstack.org/197292 is a stab at looking at what's going on if we don't have a swift endpoint - including dumping the catalog to the logs20:14
pc_mmtreinish: I see that script called twice, once with args of -rtest-requirements.txt -rrequirements.txt (with paths), and again with neutron-vpnaas20:14
pleia2mrmartin: I can dig into it, cinerama was handling this previously20:15
mtreinishif something fails before that point it'll never install any requirements20:15
pc_mmtreinish: Never called with neutron20:15
*** changbl has quit IRC20:15
mrmartinpleia2: if you can do the same with curl, maybe it should help to identify the differences between the two deployment20:15
pleia2mrmartin: ok, thanks20:15
cineramahi mrmartin pleia220:16
greghaynesmordred: +120:16
pleia2cinerama: hey! was there a particular reason we were using openstackid-dev rather than openstackid.org?20:16
mordredgreghaynes: I'm trying to replicate touching swift20:16
mrmartinhi cinerama20:16
cineramamrmartin, pleia2: so i submitted a patch to openstackid which got applied to -dev but perhaps not to the main one20:16
greghaynesmordred: hrm?20:16
pleia2cinerama: aha!20:16
mrmartincinerama: eh, it can be an issue20:16
*** Sukhdev_ has quit IRC20:16
mrmartincinerama: what was the patch exactly?20:16
mtreinishfungi, jeblair, mordred: if you get a sec tempest needs a quick reqs bump on tempest-lib to land a patch: https://review.openstack.org/#/c/194951/20:16
openstackgerritgreghaynes proposed openstack-infra/shade: WIP test case for rax image uploads  https://review.openstack.org/19693620:17
pleia2cinerama: thanks, we're finally getting this in the hands of translators to test out so we need to get off the -dev id sever so they can use their regular accounts20:17
greghaynesmordred: I just do ^ with a clouds.yaml that looks like the one in system-config and make an image.vhd file manually using dib20:17
*** sergmelikyan has quit IRC20:17
cineramamrmartin, pleia2: https://review.openstack.org/#/c/165696/20:18
*** claudiub has quit IRC20:18
greghaynesbut, that seems to work just fine... so something is amiss20:18
pc_mmtreinish: it doesn't fail, and gets to the pip install line. How does that line install neutron, if it is never specified as one of the args coming in?20:18
*** sergmelikyan has joined #openstack-infra20:18
mtreinishpc_m: that L37 or L33 is what installs neutron20:18
*** matbu_ has quit IRC20:19
mordredgreghaynes: hrm. yah. it's working for me20:19
cineramamrmartin, pleia2: so zanata is generating a more complex request header than openstackid was expecting - that change is necessary to get zanata to work with openstackid20:19
mtreinishpc_m: it does it under the covers20:19
pleia2cinerama: gotcha, thank you for digging that up20:19
mtreinishpc_m: it's kinda the whole point of adding that script20:19
*** _nadya__ has quit IRC20:19
mtreinishpc_m: it's the hack to get around not actually being able to put neutron in your requirements file20:19
mtreinishfungi: thanks20:19
pc_mmtreinish: Maybe you're missing the point I'm trying to make. Let me rephrase...20:20
*** samuelBartel has quit IRC20:20
pleia2mrmartin: so, do you know how we can get changes applied to openstackid.org?20:20
greghaynesmordred: yea, and so I tried pdb'ing along that traceback and that line in swiftclient is what seems to be divergent from my local setup and that traceback20:20
mrmartinpleia2, cinerama: yeah, I can confirm, that no new release had been rolled out to openstackid.org since march of 13, so cinerama's patch was not applied there20:20
*** matbu has joined #openstack-infra20:20
* pleia2 nods20:20
pleia2just missed the window :)20:21
mordredgreghaynes: oh. I wonder ...20:21
pc_mmtreinish: If I have a machine with neutron and neutron-vpnaas repos installed, have changes in my neutron repo, and want to run the neutron-vpnaas functional test to verify that the changes do not affect neutorn-vpnaas code.20:21
lifelessfungi: so how is requirements check looking now the fun of static is over?20:21
lifelessfungi: and le periodic constraints job?20:22
pleia2AJaeger: approved the proposal fix20:22
pc_mmtreinish: In that local host scenario, how do I get neutron-vpnaas to refer to the neutron code?20:22
mtreinishpc_m: then based on the way that script is written you have to manually install neutron in your venv20:22
AJaegerthanks, pleia2 !20:22
mrmartinpleia2: ok, so what's the plan with that? someone needs to roll out a new release there20:22
fungilifeless: dunno--i haven't looked again yet. everything decided to blow up on meeting day20:22
pleia2mrmartin: yeah, I don't know how that's done :)20:22
mtreinishfrom you local repo with your local changes20:22
mordredgreghaynes: looking to see what v of swiftclient is running there20:23
greghaynesmordred: hah20:23
mrmartinjust need to tag the git repo20:23
mtreinishpc_m: it's basically what that zuul-cloner if does in that script, but since it's your local env you have to do it manually20:23
greghaynesmordred: looked like there hasnt been a swiftclient release in a while20:23
lifelessfungi: boomala boomala20:23
pc_mmtreinish: Would L40 (already installed) every be reached, if running in the gate? IOW, can we assume we are on a local machine at L40?20:23
*** serg_melikyan has joined #openstack-infra20:24
mordredgreghaynes: 2.4.0 is on nodepool20:24
mordredgreghaynes: I have 2.3.1 locally on my laptop20:24
mtreinishpc_m: L40 is always run (unless there is an error)20:24
pleia2mrmartin: ok, so you and smarcet are part of openstackid-release, so does one of you tag it?20:24
greghaynesfor me20:24
pc_mmtreinish: whoops. Duh20:24
*** sergmelikyan has quit IRC20:25
mrmartinpleia2: technically yes20:25
greghaynesmordred: can you paste the whole pip freexe?20:25
mordredgreghaynes: yah. one sec20:25
*** zz_dimtruck is now known as dimtruck20:25
pc_mmtreinish: Trying to see if there is a way to tell if local and then install neutron using reference to existing area.20:25
mordredgreghaynes: http://paste.openstack.org/show/329574/20:25
*** weshay has quit IRC20:26
pleia2mrmartin: well I don't want to force a release into production if it's a problem/not ready/whatever, and I'm not in a good position to judge that, do you want to consult with smarcet and see if it's ok?20:26
*** weshay has joined #openstack-infra20:26
*** erikmwilson_ has joined #openstack-infra20:27
lifelessfungi: ok so can I provoke an eyeball ;)20:27
*** erikmwilson_ has quit IRC20:27
pleia2mrmartin: we definitely need this fix soon, but I didn't realize it until today so that's kind of on me20:27
mrmartinyeah, my only issues that he enrolled a large patchset today: https://github.com/openstack-infra/openstackid/commit/50bbfa7a8f86f1b32af355fb0a4f913b5cf656a320:27
mtreinishpc_m: you have to stop trying to manually automagically using local clones of repos in your scripts. If you have a local repo living somewhere that you want to use in the test env you have to install in the venv yourself.20:27
pleia2mrmartin: ok20:27
mrmartinpleia2: I can talk with smarcet, and can roll out the new release tomorrow if everything works well, I can do some tests with groups.o.o20:27
mtreinishpc_m: the script you're adding allows for that, and it's the only way to sanely work with a local checkout in this framework you've built20:27
pleia2mrmartin: that would be wonderful, thank you20:28
mrmartinpleia2: try to catch me tomorrow20:28
pleia2mrmartin: will do :)20:28
*** hodos|2 has joined #openstack-infra20:28
*** nihilifer has quit IRC20:28
asselinpleia2, nibalizer any reason not to merge this? https://review.openstack.org/#/c/175970/20:29
pc_mmtreinish: So, if working locally, and want to handle the usecase mentioned, I need to manually install neutron into venv?20:29
mtreinishpc_m: yes20:29
mtreinishpc_m: you could modify the script to take in a repo path as an option env var and just run pip install -e env_var or something I guess20:30
greghaynesmordred: nothing interesting in the diff between me and that freeze :(20:30
mtreinishpc_m: but that's more or less the same thing as doing it manually20:30
fungilifeless: i hope so. we'll see how this afternoon goes. i'd like to pay attention in the tc and x-proj meetings but so far too many distractions from people asking for help with things20:30
mordredgreghaynes: me either20:30
*** hodos has quit IRC20:31
pc_mmtreinish: Is there a way to build the venv w/o running the test? Otherwise, I have to run the test, fail, install neutron, rerun the test.20:31
*** dprince has quit IRC20:31
mtreinishpc_m: tox has a --notest arg20:32
mordredgreghaynes: what about keystoneclient?20:32
*** serg_melikyan has quit IRC20:32
greghaynesmordred: 1.6.0 for both20:33
pc_mmtreinish: thanks20:33
openstackgerritMerged openstack-infra/project-config: Fix proposal operations  https://review.openstack.org/19720020:33
*** alivigni has quit IRC20:33
*** sergmelikyan has joined #openstack-infra20:34
*** _nadya_ has joined #openstack-infra20:34
mordredgreghaynes: I can't figure out how it's getting None20:35
*** e0ne has quit IRC20:35
greghaynesmordred: did you merge the debug patch?20:35
mordredgreghaynes: no - running locally20:35
mordredgreghaynes: trying something real quick20:36
mordredgreghaynes: I can do swift operations as the nodepool user on nodepool.o.o against rax iad20:37
*** jtomasek has quit IRC20:37
mordredgreghaynes: and ORD20:37
mordredgreghaynes: so I'm VERY confused\20:38
mordredgreghaynes: http://paste.openstack.org/show/329591/20:38
*** EricGonczer_ has quit IRC20:39
*** _nadya_ has quit IRC20:39
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Modify secgroup rule processing  https://review.openstack.org/19730820:39
*** _nadya_ has joined #openstack-infra20:40
Shrewsmordred: https://review.openstack.org/19730820:40
nibalizerasselin: i think given the content that a jeblair +2 is in order for that patc20:41
nibalizerbut thats just my opinon20:41
*** ChuckC has joined #openstack-infra20:42
greghaynesmordred: get_capabilities is the issue though, not sure get_container will kick that off?20:42
pc_mmtreinish: thanks. I'm trying to test it locally.20:42
mordredoh. hrm20:42
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: refactorize puppet-syntax job  https://review.openstack.org/19721620:42
*** ChuckC has left #openstack-infra20:43
*** mrmartin has quit IRC20:43
*** amit213 has quit IRC20:44
*** amit213 has joined #openstack-infra20:44
*** _nadya_ has quit IRC20:44
*** amit213 has quit IRC20:44
mordredgreghaynes: I just ran get_object_capabilities() and it ran fine too20:44
*** dimtruck is now known as zz_dimtruck20:45
greghaynesmordred: hah20:45
*** zz_dimtruck is now known as dimtruck20:46
*** claudiub has joined #openstack-infra20:47
*** ociuhandu_ has joined #openstack-infra20:48
greghaynesmordred: is it possible that nodepool got restarted in a bad state? like maybe clouds.yaml was missing or we upgraded deps while it was running so it didnt restart and is running against older libs than are installed?20:49
mordredfungi, jeblair, clarkb, pleia2, jhesketh, SergeyLukjanov: can we install requests[security] on the machines that have requests installed via pip?20:49
fungilifeless: resetting my brainstate to where i passed out last night... the requirements check job was breaking on most projects because they still had requirements-py3k.txt files. what was out next step there? mass-propose deletion patches for those? thread on the ml? switch careers to longshoremen?20:49
mordredgreghaynes: I'm not sure- I'm slightly worried that maybe because it's more complex than my test script, state is getting unhappied somewhere20:50
lifelessfungi: jhesketh landed the patch, and built new images.20:50
lifelessfungi: and then static blew its ass out20:50
fungimordred: i'm not opposed to that, but it will require making sure we have the additional library headers that needs installed on those systems20:50
clarkbmordred only if you also install all the C dev packages too20:50
fungiwhat clarkb said20:50
clarkbalso good morning20:50
lifelessfungi: I don't know if the new images are still live or if he rolled them back as part of the troubleshooting steps20:50
mordredfungi, clarkb: cool. I'll work on puppet patch20:51
greghaynesmordred: well, AIUI we assert all the nodepool python deps as latest, but we dont restart nodepool when they change, so pip freeze doesnt necessarially say what the running nodepool is using20:51
fungilifeless: oh, okay. so in theory they should pass now. i'll have a look20:51
*** ZZelle_ has quit IRC20:51
lifelessfungi: https://review.openstack.org/#/c/196930/ might be a good litmus test20:51
lifelessfungi: since the old code blew up on master20:51
fungilifeless: from the looks of https://review.openstack.org/#/q/status:open+topic:openstack/requirements,n,z i expect we're fine now?20:52
lifelessfungi: I don't think so - https://jenkins03.openstack.org/job/gate-ceilometer-requirements/23/console20:52
fungioh, poop20:53
lifelessfungi: I can't tell if that ran the old code or the new20:53
lifelessfungi: I might put up a patch to make that clearer20:53
fungilifeless: 2015-06-30 20:03:37 would have run on images built today. i'll check and see if they actually succeeded20:54
fungilifeless: i guess not. these images all appear to have been the ones i built about 23 hours ago20:55
*** ociuhandu_ has quit IRC20:55
*** dboik has quit IRC20:55
lifelessfungi: yeah, 23 hours is about when I started writing the patch IIRC20:55
lifelessfungi: so yeah. can has nu ones plox.20:55
lifelessfungi: I know jhesketh built some.20:55
lifelessfungi: maybe they are stashed not deleted?20:56
*** dimtruck is now known as zz_dimtruck20:56
*** radez is now known as radez_g0n320:58
*** stevemar has quit IRC20:58
jheskethlifeless,  fungi: nope, sorry, delete during all the debugging.20:58
*** zz_dimtruck is now known as dimtruck20:59
fungijhesketh: so we're okay to roll out new bare-trusty images? i'll start generating those now20:59
jheskethYep, I don't see why jot20:59
lifelesswas there a post-mortem sent to the list? I missed all the recovery excitement20:59
jheskethDeleting them wasn't necessary, I was just being cautious as it was my first time building them21:00
fungii sent a brief reply21:00
fungithe post-mortem is that block storage backends went toes up in rackspace and yanked a cinder volume from the lvm vg to which we write out logs on static.o.o21:01
fungiso i needed to detach/reattach the right one via nova/cinder, then fsck a >12tib fs21:01
*** olaph has joined #openstack-infra21:03
*** armax has quit IRC21:03
mordredgreghaynes: I'm going to beer21:03
*** ddieterly has joined #openstack-infra21:03
greghaynesmordred: that is probably the best idea21:03
greghaynesmordred: I think maybe the thing to try is restart nodepool with the debug patch, will answer two questions at once21:04
greghaynesbut... we can do that post beer21:05
*** Akshay00 has quit IRC21:05
*** aysyd has quit IRC21:06
*** john-davidge has joined #openstack-infra21:07
*** dizquierdo has joined #openstack-infra21:07
*** abregman has quit IRC21:08
*** vahidh has quit IRC21:09
*** puranamr has quit IRC21:10
*** rlandy has quit IRC21:10
*** xnox has quit IRC21:10
pc_mmtreinish: Giving up for today. will try again tomorrow. Even with manually installing neutron, the tests are failing.21:10
pc_mmtreinish: Thanks for all the help hashing through this...21:10
*** armax has joined #openstack-infra21:11
clarkbfungi we still need to do the trove stuff?21:11
*** rlandy has joined #openstack-infra21:11
clarkbI can start with that on etherpad-dev and do the encoding switch21:12
*** xnox has joined #openstack-infra21:12
*** julim has quit IRC21:13
*** olaph has quit IRC21:13
*** olaph has joined #openstack-infra21:13
fungiclarkb: i asked support and they indicated that the migration, while it will cause a ~5 minute outage for database access, won't end up renaming the instance in dns. if we don't want to rebuild some instances and can suffer the 5-minute outage at some random time tomorrow then for most of our systems that's probably a lot less work21:14
fungiif we dump, destroy, create and import in to a new mysql version then we're going to also need to update hiera for the new hostname21:14
fungiso for etherpad we do need to do that for the encoding fix, but it's not necessarily critical that we do it before the db instance migrations happen tomorrow21:15
clarkbI am going to work on etherpad-dev as it needs doing but good to know on the others21:15
*** ldnunes has quit IRC21:15
fungiclarkb: oh, though i did discover we're not backing up the database for openstackid.org yet. i ran a manual backup today and proposed https://review.openstack.org/197083 to solve it longer term21:16
*** claudiub has quit IRC21:17
*** sergmelikyan has quit IRC21:17
*** cindyo has quit IRC21:17
mtreinishpc_m: sure, np21:17
openstackgerritMerged openstack-infra/shade: Correctly name the functional TestImage class  https://review.openstack.org/19689121:20
*** vahidh has joined #openstack-infra21:21
*** tonytan4ever has quit IRC21:22
*** whoops has quit IRC21:25
clarkbI am tempted to suggest we use on instance dbs again though21:25
*** hodos|2 has quit IRC21:25
clarkbtrove seems to just be an additional complicated piece in a thibg that shouldnt be complicated21:25
clarkbonly problem with that is performance nodes arent setup for mich peristent disk21:27
pc_mmtreinish: One quick question...21:27
*** tvanderwiel_ has joined #openstack-infra21:28
pc_mmtreinish: IF I source the activate, and do a "pip install -U /opt/stack/neutron" should that install the src dir, in addition to bin and etc?21:28
*** Ryan_Lane has joined #openstack-infra21:28
Ryan_Laneneat: https://github.com/stackforge/bandit21:28
*** smarcet has quit IRC21:28
*** rfolco has quit IRC21:28
Ryan_Lanejust now seeing that, but whoever made that: thanks :)21:28
fungiRyan_Lane: its developers are part of the openstack security project-team21:29
*** tvanderwiel has quit IRC21:29
clarkbwould be nice if it didnt remake the flake8 wheel but ok21:30
*** maishsk has quit IRC21:30
*** maishsk has joined #openstack-infra21:31
*** otter768 has joined #openstack-infra21:32
*** john-davidge has quit IRC21:32
*** tvanderwiel_ has quit IRC21:33
*** maishsk has quit IRC21:33
*** pc_m has quit IRC21:33
*** teran has joined #openstack-infra21:36
*** thorst has quit IRC21:38
openstackgerritgreghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman  https://review.openstack.org/19517521:38
*** fifieldt has quit IRC21:39
*** ayoung has quit IRC21:39
*** tqtran-afk is now known as tqtran21:39
*** jsavak has joined #openstack-infra21:41
*** peristeri has quit IRC21:41
*** mriedem has quit IRC21:41
fungigreghaynes: ooh!!!21:41
* fungi bounces up and down21:41
greghaynesthats just the start of it - that patch is still sharing db state with nodepool and such but it using gearman to kick off the builds21:42
greghaynesthen im going to start pulling out the shared state21:42
fungii for one welcome our new distributed and heterogeneous image build worker overlords21:43
* SpamapS peruses the list of servers for one to enroll in baremetal021:43
greghaynesSpamapS: just not the bastion!21:43
*** isviridov is now known as isviridov_away21:45
*** puranamr has joined #openstack-infra21:45
*** jsavak has quit IRC21:45
*** ddieterly has quit IRC21:45
*** ZZelle_ has joined #openstack-infra21:46
*** ociuhandu has joined #openstack-infra21:46
*** amitgandhinz has quit IRC21:46
*** koolhead17 has quit IRC21:47
*** puranamr has quit IRC21:47
*** puranamr has joined #openstack-infra21:48
* morganfainberg steals mordred's stash of wet cats. (or is he onto emus now?)21:49
*** skraynev has quit IRC21:49
*** ilyashakhat has quit IRC21:49
*** asahlin is now known as asahlin_away21:50
*** jsavak has joined #openstack-infra21:51
clarkbfungi: I am going to call the new db etherpad-dev-mysql-5.6 so that it is clear which is which21:51
*** fifieldt has joined #openstack-infra21:52
fungiclarkb: that's fine. or you could just delete and replace it unless you really think you need to switch back and forth rapidly21:52
*** tlian has quit IRC21:53
clarkbmore thinking about how the non -dev instance should look21:53
clarkband for that one less comfortable with an extended outage while new db populates21:53
*** VivCheri has quit IRC21:53
SpamapSdoh, default quota for nova is 20 cores21:53
SpamapSone baremetal is 24 .. doh21:53
fungiclarkb: well, we'll want to keep the service offline while we export and import anyway to avoid people adding data that gets undone in the migration21:54
clarkbfungi: true21:54
fungiclarkb: so i'm not sure we can make it much shorter by having two in parallel, but i guess it could shave a couple minutes off21:54
fungiup to you21:55
fungii think the instance name is entirely cosmetic anyway, not even reflected in any of our configs, so it doesn't make a ton of difference if it changes21:55
fungiand we have to update the hostname for the instance in hiera when we cut over regardless21:56
*** fallenpegasus has joined #openstack-infra21:56
clarkbI will preserve the username and password of the old host though21:56
*** timburke has joined #openstack-infra21:56
clarkbunless you think its worth updating that too21:56
fungishouldn't be necessary to change those21:57
nibalizerhow does the openstack-infra/gerrit post stuff work? I would expect to see tags from openstackers in the tag list but I do not21:57
funginibalizer: i have no idea what that combination of words means. can you rephrase>?21:57
*** ddieterly has joined #openstack-infra21:58
*** ihrachyshka has quit IRC21:58
funginibalizer: you mean our backport-enabling local fork of the gerrit source tree?21:58
funginibalizer: and git tags?21:58
*** matbu has left #openstack-infra21:58
clarkbfungi: also this way I can build the non -dev db now, and get to using it later :)21:58
nibalizerfungi: yes21:59
*** agentle has quit IRC21:59
*** puranamr has quit IRC21:59
nibalizerfungi: i mean the gerrit-package-buck job21:59
funginibalizer: we deploy specific ref ids from backport branches like http://git.openstack.org/cgit/openstack-infra/gerrit/log/?h=openstack/2.10.221:59
*** frobware_ has quit IRC21:59
*** ddieterly has quit IRC21:59
nibalizerwhich seems to run in post, but i thought we needed to push tags into openstack-infra/gerrit to make post jobs fire21:59
funginibalizer: zuul fires that job in the post pipeline when we approve a patch on one of those branches, and then uploads the results to http://tarballs.openstack.org/ci/22:00
*** otter768 has quit IRC22:00
funginibalizer: nope, the post pipeline is triggered by ref-updated (similar to change-merged)22:01
funginibalizer: our release, pre-release and tag pipelines trigger on tags22:01
nibalizerso taron and I are looking at hound which is a go thing and wondering if something like that would be a good model22:01
nibalizeroh release is not post, right22:01
clarkbfungi: for 197083 those backups run at 0000UTC and since we don't use a fancy tool likethe percona backup thing it locks a bunch of the DB, for small DBs this isn't an issue but for a DB like silverstripe which is large could that cause cascading service issues?22:01
funginibalizer: right see http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n7722:02
clarkbfungi: mostly not familiar enough with everything tlaking to that db to know what may be affected22:02
*** doug-fish has left #openstack-infra22:02
fungiclarkb: "large" in this case is 40mib compressed. took something like 3 seconds when i ran it manually22:02
fungier, 50mib compressed22:02
clarkbah ok not too bad22:02
lifelessfungi: images building?22:02
clarkbetherpad.o.o is bigger22:02
fungilifeless: about half have completed, some are still going22:03
fungilifeless: the rest seem to be wrapping up, though one needed restarting early on so may be a little bit longer still22:03
*** jsavak has quit IRC22:03
*** spzala has quit IRC22:04
*** dizquierdo has quit IRC22:04
*** yamahata has quit IRC22:04
clarkbfungi: db backups approved22:05
*** puranamr has joined #openstack-infra22:05
*** hdd has quit IRC22:06
*** gokrokve has quit IRC22:07
*** gordc is now known as gordc_afk22:09
*** davideag_ has joined #openstack-infra22:09
*** claudiub has joined #openstack-infra22:12
*** Rockyg has quit IRC22:12
openstackgerritMerged openstack-infra/system-config: Add database backups for openstackid.org  https://review.openstack.org/19708322:12
*** davideagnello has quit IRC22:13
lifelessfungi: ok so here is something odd22:14
lifelessfungi: look at the top22:14
lifelessfungi: then look at the CI messages in detail22:14
*** olaph has quit IRC22:15
*** olaph has joined #openstack-infra22:15
clarkbtaking a lock on hieradata22:17
fungilifeless: what's odd there?22:18
*** dims_ has joined #openstack-infra22:19
lifelessfungi: gate-ceilometer-requirements is missing from the summary at the top22:19
lifelessfungi: but failed and is shown down the bottom22:19
lifelessfungi: so when you look at just the top it says 'jenkins -1' and all the tests are green.22:19
fungi"incompatible requirements" results from the gate-ceilometer-requirements job don't get extracted into the job result summary table because the extended result text doesn't fit our usual success/failure pattern22:19
*** sergmelikyan has joined #openstack-infra22:19
lifelessso, thats the why... its still funny :)22:19
lifelessodd-funny, not haha funny22:20
fungilifeless: we can sacrifice the descriptive message and just make it "success" or "failure" instead22:20
lifelessfungi: what prevents us having schrodinger cake?22:20
clarkbhttps://etherpad-dev.openstack.org/p/clarkb-test is running with mysql5.6 and the special utf8 encoding22:20
fungiand make people pull up the log, or know that a failure of that job means their requirements are probably incompatible22:20
clarkbso please test and throw 4 byte utf8 chars at it22:20
*** rlandy has quit IRC22:20
*** dims has quit IRC22:22
fungilifeless: i can get behind the suggestion that it should just fail consistent with other jobs and then the errors in the job log can instead explain what's wrong and link to appropriate documentation to resolve the issue22:22
*** gokrokve has joined #openstack-infra22:22
fungii think the long-form error message was instated early on when people had no idea what that job was nor why we were testing such a thing22:22
*** pleia2 has quit IRC22:23
asselinnibalizer, any idea what I'm doing wrong her? http://paste.openstack.org/show/329897/22:24
taronclarkb: pasted a bunch at the end and nothing seems to have exploded yet22:24
clarkbtaron: awesome thanks22:25
taronat least not on my end, anyway22:25
asselinfbo, ?? ^^22:25
*** pleia2 has joined #openstack-infra22:25
*** bknudson has quit IRC22:26
*** jsavak has joined #openstack-infra22:28
*** bhunter71 has quit IRC22:28
nibalizerasselin: uh oh did we pyhton22:29
clarkbfor 37k rows with etherpad-dev the utf8 conversion took less than one second22:29
lifelessfungi: well, I guess I'm more curious about the technical undeprinnings22:29
*** ayoung has joined #openstack-infra22:29
asselinnibalizer, trying to validate those lines:   $gerrit_ssh_ident = inline_template('<%= (@gerrit_ssh_host_identity).join(",") %> <%= @gerrit_ssh_host_key %>')22:29
lifelessfungi: like, why can't we say [FAILURE] [details...here]22:29
lifelessfungi: and have the regex for the summary pull just the first bit out22:29
zarois it just me or is review.o.o slow for others as well?22:30
*** jgrimm is now known as zz_jgrimm22:31
*** mriedem has joined #openstack-infra22:32
fungilifeless: also possible. would be an update to http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/gerrit/hideci.js22:33
nibalizerasselin: so it works for me22:33
clarkbok with etherpad-dev done, when is a good time to do etherpad.o.o?22:33
fungilifeless: okay, bare-trusty images are fresh in all providers now22:33
lifelessfungi: (plus presumably a change to the reporting thign ?)22:33
clarkbfungi: pleia2 opinions ^ ?22:33
pleia2clarkb: if you're looking for a quiet window, with the US holiday coming up things will likely be very quiet on thursday22:34
fungilifeless: yeah, adjustment to the failure message can be done at http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n94622:34
asselinnibalizer, ok...will check what's different. thanks22:34
fungiclarkb: i'd say any time. rough estimate of how long you expect we'd need to have it offline?22:35
openstackgerritMerged openstack/requirements: Bump tempest-lib version  https://review.openstack.org/19495122:35
clarkbfungi: let me grab a count of the rows on that db and figure 1 second per 37k rows22:35
clarkbplus overhead to switch22:35
clarkbpleia2: good point22:35
*** soren has quit IRC22:36
*** rossella_s has quit IRC22:36
*** dontalton has quit IRC22:37
*** rossella_s has joined #openstack-infra22:37
openstackgerritClark Boylan proposed openstack-infra/system-config: Bump etherpad version to version running on -dev  https://review.openstack.org/19735822:37
*** ninag has joined #openstack-infra22:38
*** olaph has quit IRC22:38
clarkbfungi: pleia2 ^ that is the first bit of this to make sure we have the correct version in place before restarting everything. Feel free to poke at -dev to test22:38
*** DericHorn-HP has joined #openstack-infra22:38
greghaynesclarkb: might be nice to put the upstream bug in that commit msg :)22:39
greghaynes(where is that btw?)22:39
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: add puppet-unit-latest job  https://review.openstack.org/19721822:40
clarkboh good point let me update22:40
*** zz_jgrimm is now known as jgrimm22:40
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: do not run beaker/upgrade jobs on old branches  https://review.openstack.org/19721922:40
*** ayoung has quit IRC22:41
openstackgerritClark Boylan proposed openstack-infra/system-config: Bump etherpad version to version running on -dev  https://review.openstack.org/19735822:41
clarkbpleia2: fungi rough math based on a 37k rows per second is 317 seconds to do conversion22:42
clarkbso we should probably figure about half an hour to be generous for total downtime22:42
*** jsavak has quit IRC22:42
*** jsavak has joined #openstack-infra22:43
*** julim has joined #openstack-infra22:43
pleia2haven't managed to break etherpad-dev22:43
*** gokrokve has quit IRC22:44
greghayneshttps://etherpad-dev.openstack.org/p/%E2%98%83 works even :)22:46
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements  https://review.openstack.org/19650422:46
clarkbwill anyone be around on friday? if so I am happy to do most of the work then just want to make sure I am the only root not outside in the sun celebrating FREEDOM22:47
*** amuller has quit IRC22:47
clarkbor on the other side of the dateline and otherwise weekending22:47
pleia2I'm out on friday22:47
greghaynesI will be around for moral support22:47
*** sabeen has quit IRC22:48
clarkbI can also do thursday if that is just better22:48
fungii have no friday plans22:48
fungior thursday plans for that matter22:48
fungii'm pretty boring22:49
fungihappy to drive it too if you have the commands worked out22:49
greghaynesfungi: youre not going to go enjoy all the neighbors setting off fireworks? :p22:49
clarkbfungi: cool, let me write it up on etherpad-dev now (since etherpad itself will be down)22:49
fungigreghaynes: saturday we'll walk out to the beach and watch the municipalities a couple miles north and south of us set fireworks off from their piers22:50
*** nelsnels_ has joined #openstack-infra22:50
asselinnibalizer, do you think it makes sense to put that inline_template inside the  if $gerrit_ssh_host_key != '' { ?22:50
fungibut yes, every night for thw next couple of weeks will be nonstop tourists launching fireworks22:51
nibalizerreally i'd prefer to change the api to hard require those things22:51
nibalizerbut i know moving to openstackci is going to be rough enough without doing that so I'm keeping my peace22:51
asselinwhat do you mean by hard-require?22:51
asselinremove the if?22:52
*** sergmelikyan has quit IRC22:52
*** eharney has quit IRC22:53
openstackgerritCody Herriges proposed openstack-infra/project-config: puppet: refactorize puppet-syntax job  https://review.openstack.org/19721622:53
*** nelsnelson has quit IRC22:53
nibalizerasselin: set it to $gerrit_ssh_host_key, in the class parameters22:53
nibalizerso that puppet will fail if you don't pass one in22:53
openstackgerritCatherine Diep proposed openstack-infra/irc-meetings: Add agenda page URL to Refstack team meeting.  https://review.openstack.org/19736522:54
*** larainema has quit IRC22:54
*** jsavak has quit IRC22:54
asselinnibalizer, so get rid of the inline template and pass it in directly? That would be much simpler22:55
asselinnibalizer, and match what I was using before trying to migrate, so easy sell for me :)22:56
*** otter768 has joined #openstack-infra22:56
clarkbfungi: https://etherpad-dev.openstack.org/p/eplite-db-updates22:57
asselinnibalizer, also gerrit_ssh_host_key isn't used anywhereat all22:57
fungiclarkb: awesome--thanks!!!22:58
clarkbfungi: does midday friday (PDT) work for you? mornings have been pretty terrible for me lately but afternoons are much better22:59
clarkbI can send mail to the list about a Noon PDT (1900UTC iirc) etherpad downtime22:59
fungiclarkb: sounds great. and if you can't make it then, it's no big deal. the plan looks clean and dead simple23:00
*** dims_ has quit IRC23:01
*** cindyo has joined #openstack-infra23:04
*** markvoelker_ has quit IRC23:06
*** ashleighfarnham has quit IRC23:06
*** ninag has quit IRC23:08
*** sarob_ has joined #openstack-infra23:08
*** ZZelle_ has quit IRC23:10
*** jsavak has joined #openstack-infra23:13
*** sarob has quit IRC23:14
*** mwagner_lap has quit IRC23:15
*** dimtruck is now known as zz_dimtruck23:15
clarkbfungi: notice sent to dev list23:16
*** sarob_ is now known as sarob23:16
*** mase_x200 has joined #openstack-infra23:16
pleia2thanks clarkb and fungi \o/23:18
*** hichihara has joined #openstack-infra23:20
*** sarob_ has joined #openstack-infra23:21
asselinnibalizer, gave up and put in my -1....23:21
*** claudiub_ has joined #openstack-infra23:23
zaropabelanger: trystack seems broken, http://x86.trystack.org23:23
*** krtaylor has quit IRC23:23
*** jsavak has quit IRC23:24
*** austin81 has left #openstack-infra23:25
*** claudiub has quit IRC23:25
nibalizerasselin: okay23:25
*** sarob_ has quit IRC23:27
asselinnibalizer, what's your opinion? shall I upload another patch or hold off a bit? Really trying to get this done before the break....23:27
nibalizer'the break' ?23:27
*** jsavak has joined #openstack-infra23:27
*** yamahata has joined #openstack-infra23:28
asselinnibalizer, yes...july 4th....guess it doesn't mean much in UTC23:28
nibalizeroh right23:28
nibalizerum im not looking at that exact issue right now23:28
nibalizerill try to circle around and look at it today/this evening23:28
asselinnibalizer, ok thanks23:29
*** DericHorn-HP has quit IRC23:31
tchaypohttp://status.openstack.org/zuul/ gives a lovely overview of things in the queue right now; is there somewhere that shows me the history of jobs that have been run?23:34
*** dannywilson has quit IRC23:35
*** jsavak has quit IRC23:35
*** puranamr has quit IRC23:36
*** mase_x200 has quit IRC23:37
*** dangers is now known as dangers_away23:37
*** puranamr has joined #openstack-infra23:37
clarkbtchaypo: graphite for results over time, logstash if you need it mapped to other data over short term and gerrit for long term23:37
tchaypohttp://logs.openstack.org seems to have some data too23:38
*** Qiming has joined #openstack-infra23:39
*** cindyo has quit IRC23:39
tchaypoin graphite, stats.zuul.pipeline.periodic-stable.job.periodic-tempest-dsvm-neutron-juno only shows SUCCESS and I’m trying to figure out why23:39
clarkbyes though the way it is structured and the vast quantity make it hard to consume for trend data23:39
clarkbetherpad-dev's clarkb-test pad may have exploded23:40
clarkbthe rest of the service continues to run though23:40
*** _nadya_ has joined #openstack-infra23:40
clarkband we need to upgrade anyways so we should move forward with downtime, but what is going on here23:40
clarkbis https://etherpad-dev.openstack.org/p/clarkb-test working for anyone? I wonder if its my client23:41
tchaypologs has data for periodic-tempest-dsvm-neutron-full-juno but not periodict-tempest-dsvm-neutron-juno which makes me think the latter is not a job we run any more23:41
clarkbthere are no errors in the etherpad logs that I can see23:41
*** krtaylor has joined #openstack-infra23:41
Clintclarkb: i can see the chat23:41
tchaypoi can see the chat23:41
clarkbya I see the chat but the content is perpetually loading23:41
pleia2just Loading...23:41
pleia2etherpad is the worst :(23:42
clarkbso anyways we should do the db move on friday but not call the issue fixed fixed until some other time23:42
*** cindyo has joined #openstack-infra23:42
clarkbI bet what happens is the encoding change allows us to write the bytes to mysql, but etherpad lite then fails to pull them out properly23:42
Clintyou put a 4-byte char in?23:43
*** dannywilson has joined #openstack-infra23:43
clarkbtaron did23:44
Clintand the json in the db is sane?23:44
clarkbI am looking at it with chromium dev tools and the content response looks funky23:44
*** _nadya_ has quit IRC23:45
*** dims has joined #openstack-infra23:45
*** dims has quit IRC23:46
greghaynesclarkb: yea, I think your hypothesis is plausible based on error I see23:46
greghaynesseeing Error: Failed assertion: Invalid changeset (checkRep failed in console23:47
*** markvoelker has joined #openstack-infra23:47
greghayneswhich googling leads me to a bug report on an encoding issue23:47
taroninteresting that the top bar disappeared as well23:47
clarkbthis is an improvement in that the other pads served by the server continue to work23:48
clarkbrather than the whole thing dying in a loop23:48
Clintit sure is23:48
*** teran has quit IRC23:48
greghaynesnot that it isnt hilarious to watch etherpad do that23:49
clarkband is that happening in the js vm?23:49
clarkbsocket.io fails due to the encoding issue23:50
greghaynesclarkb: wah?23:50
clarkboh nevermind I was reading too much into the filename23:51
*** marun has quit IRC23:51
clarkbcomments indicate "this file is mostly from the old etherpad" (gread)23:51
clarkbexports.pack takes a string encoding23:53
clarkbso that must not be correct or the source data is being munged23:53
*** sputnik1_ has joined #openstack-infra23:54
*** dannywilson has quit IRC23:54
*** yamada-h has joined #openstack-infra23:54
*** dannywilson has joined #openstack-infra23:54
openstackgerritEmma Barber proposed openstack-infra/project-config: Add hound mirror  https://review.openstack.org/19737623:55
*** dboik has joined #openstack-infra23:56
clarkbI am going to update the upstream bug now but this is beyond my ability to debug on little sleep23:56
clarkbfungi: ^ but we should consider not changing the encoding until this is fixed and just do the db upgrade on friday23:56
*** sputnik13 has quit IRC23:57
*** yamada-h has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!