Wednesday, 2015-09-02

*** achanda has joined #openstack-infra00:00
*** glauco_ has joined #openstack-infra00:00
*** Swami has quit IRC00:00
*** dimtruck is now known as zz_dimtruck00:01
*** camunoz has quit IRC00:01
*** ybathia has quit IRC00:03
*** glauco_ has quit IRC00:04
*** achanda has quit IRC00:04
*** yamamoto has joined #openstack-infra00:05
*** yamamoto has quit IRC00:05
*** yamamoto has joined #openstack-infra00:05
*** sflanigan has quit IRC00:05
fungijgriffith: here's what i find from the zuul logs from when you reapplied the approval... 2015-09-01 18:45:58,768 DEBUG zuul.DependentPipelineManager:   Change <Change 0x7fa73895ae90 212861,11> in project openstack/cinder does not share a change queue with <Change 0x7fa68cbde510 215847,2> in project openstack/python-cinderclient00:07
fungi2015-09-01 18:45:58,769 DEBUG zuul.DependentPipelineManager: Failed to enqueue changes ahead of <Change 0x7fa68cbde510 215847,2>00:07
fungi212861,11 seems to be the most recent patchset for the depends-on00:08
jheskethfungi: er sorry, got a busy morning this morning. Was there something about the swift indexes you wanted to talk about?00:08
fungiwhich didn't merge until 19:29:4400:08
fungijhesketh: nope, clarkb pointed me to the work in progress change for it00:08
*** shashank_hegde has joined #openstack-infra00:09
jheskethfungi: yeah, I broke it last night, hopefully will fix whatever it is this morning00:09
jhesketh(actually it'll have to afternoon)00:09
jheskethI also want to turn some more jobs over to swift logs, so will look at that next00:10
fungijgriffith: so, while that's a known behavior (you can't enqueue a change via approval with a cross-repo dependency on a commit in another repo which doesn't share any of the same job names), i find it odd that cinder and cinderclient wouldn't share a change queue00:10
*** sharikapongubala has quit IRC00:10
fungirather, can00:10
fungi't enqueue it until the dependency merges00:10
fungijgriffith: mtanino: ahh, yep that's correct. cinder and cinderclient really don't share a change queue (they have no jobs in common with each other at all)00:12
*** achanda has joined #openstack-infra00:12
fungijgriffith: mtanino: so anyway, if you have a cinderclient change depends-on a cinder change (or vice versa) you need to wait to approve the second until the first merges00:12
*** lucas-dinner has quit IRC00:12
*** darrenc is now known as darrenc_afk00:12
fungijgriffith: mtanino: now that 212861 has merged, if you remove and readd your approval workflow +1 vote on 215847 it should head into the gate00:13
mtaninofungi: I see. It's a know behaviour...00:13
mtaninofungi: jgriffith is getting back to home so I will ask him later to remove and readd the request. Thank you.00:14
*** fitoduarte has quit IRC00:14
*** salv-orlando has quit IRC00:15
*** dims__ has quit IRC00:16
*** jeadre has quit IRC00:16
*** camunoz has joined #openstack-infra00:17
*** jeadre has joined #openstack-infra00:17
*** pvaneck has quit IRC00:21
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues  https://review.openstack.org/21951400:22
*** sflanigan has joined #openstack-infra00:22
*** ashleighfarnham has quit IRC00:23
openstackgerritK Jonathan Harker proposed openstack-infra/project-config: Create a job to ease manpage reviews  https://review.openstack.org/21849300:26
*** tjones has joined #openstack-infra00:27
*** baoli has quit IRC00:27
*** darrenc_afk is now known as darrenc00:29
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues  https://review.openstack.org/21951400:30
*** zhenguo has joined #openstack-infra00:30
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515100:32
*** wenlock has quit IRC00:33
*** dguitarbite has quit IRC00:37
*** dbite has joined #openstack-infra00:37
*** puranamr has quit IRC00:38
*** claudiub has quit IRC00:39
*** sputnik13 has quit IRC00:39
*** vivekd has quit IRC00:40
*** dannywil_ has quit IRC00:43
*** dannywilson has joined #openstack-infra00:43
*** yamamoto has quit IRC00:44
*** puranamr has joined #openstack-infra00:44
*** jswarren has joined #openstack-infra00:45
*** dannywilson has quit IRC00:45
*** yamamoto has joined #openstack-infra00:46
*** jswarren has quit IRC00:46
*** jswarren has joined #openstack-infra00:47
*** morgabra has quit IRC00:50
*** morgabra has joined #openstack-infra00:50
*** dims__ has joined #openstack-infra00:51
*** tjones has quit IRC00:52
jgriffithfungi: ahh... ok, thank you very muc00:52
jgriffithfungi: I have to admit the cinder dependency didn't register when I looked at it00:53
*** Sukhdev has quit IRC00:55
*** dims__ has quit IRC00:56
*** sabeen1 has quit IRC00:56
*** xyang1 has joined #openstack-infra00:57
*** kaisers has quit IRC01:01
*** kaisers has joined #openstack-infra01:03
*** asettle is now known as asettle-afk01:03
*** r-daneel has quit IRC01:04
*** bknudson has quit IRC01:04
*** puranamr has quit IRC01:05
*** hdd has joined #openstack-infra01:06
*** puranamr has joined #openstack-infra01:06
*** achanda has quit IRC01:09
*** ajmiller has joined #openstack-infra01:12
*** mtanino has quit IRC01:13
*** stevemar has joined #openstack-infra01:15
*** vivekd has joined #openstack-infra01:16
*** ddieterly has joined #openstack-infra01:16
*** btully has joined #openstack-infra01:17
*** thorst has joined #openstack-infra01:17
pabelangeryolanda: rcarrillocruz: Making good progress on ansible-role-nodepool.  I should have something to show you by next week, won't get much time to play with it after today.01:20
*** zeih has joined #openstack-infra01:21
*** skylerberg has quit IRC01:22
*** jeadre has quit IRC01:22
*** tiswanso has joined #openstack-infra01:22
*** armax has quit IRC01:22
*** jeadre has joined #openstack-infra01:23
*** tiswanso has quit IRC01:25
*** mpopow_ has joined #openstack-infra01:25
*** zeih has quit IRC01:26
*** tiswanso has joined #openstack-infra01:26
*** baoli has joined #openstack-infra01:27
*** baoli has quit IRC01:28
*** baoli has joined #openstack-infra01:28
*** sridhar_ram has joined #openstack-infra01:28
*** sridhar_ram has quit IRC01:29
*** mpopow has quit IRC01:29
*** armax has joined #openstack-infra01:35
*** asettle-afk has quit IRC01:36
*** baoli has quit IRC01:37
*** jeadre has quit IRC01:37
*** baoli has joined #openstack-infra01:38
*** jeadre has joined #openstack-infra01:38
*** vivekd has quit IRC01:39
*** jyuso1 has joined #openstack-infra01:40
*** puranamr has quit IRC01:40
*** puranamr has joined #openstack-infra01:40
openstackgerritMerged openstack-infra/project-config: Add docs publish docs for grenade  https://review.openstack.org/21851301:41
*** esmute_ has quit IRC01:41
*** armax has quit IRC01:46
*** sharikapongubala has joined #openstack-infra01:46
*** resker has joined #openstack-infra01:48
*** ddieterl_ has joined #openstack-infra01:48
*** vivekd has joined #openstack-infra01:50
*** ddieterly has quit IRC01:52
*** esker has quit IRC01:52
*** sharikapongubala has quit IRC01:53
*** madhu_ak has quit IRC01:53
*** ashleighfarnham has joined #openstack-infra01:56
*** puranamr has quit IRC01:57
*** hdd has quit IRC01:58
*** dingyichen has quit IRC02:00
*** puranamr has joined #openstack-infra02:00
*** ashleighfarnham has quit IRC02:01
*** wenlock has joined #openstack-infra02:01
*** sharikapongubala has joined #openstack-infra02:01
*** achanda has joined #openstack-infra02:04
*** sflanigan has quit IRC02:07
*** camunoz has quit IRC02:07
*** ddieterl_ has quit IRC02:11
*** jswarren_ has joined #openstack-infra02:13
*** w_ has joined #openstack-infra02:13
*** achanda has quit IRC02:14
*** jswarren has quit IRC02:17
*** olaph has quit IRC02:17
*** thorst has quit IRC02:17
*** shashank_hegde has quit IRC02:18
*** olaph has joined #openstack-infra02:18
*** w_ has quit IRC02:19
*** camunoz has joined #openstack-infra02:20
*** sflanigan has joined #openstack-infra02:22
*** zeih has joined #openstack-infra02:22
*** achanda has joined #openstack-infra02:23
*** zeih has quit IRC02:26
*** FallenPegasus has joined #openstack-infra02:27
*** salv-orlando has joined #openstack-infra02:27
*** BharatK has joined #openstack-infra02:28
StevenKclarkb: Etherpad gone over02:28
*** baoli has quit IRC02:29
*** baoli_ has joined #openstack-infra02:30
*** unicell has quit IRC02:31
*** salv-orlando has quit IRC02:32
*** dingyichen has joined #openstack-infra02:33
*** Sree has quit IRC02:41
*** resker has quit IRC02:41
*** annegentle has quit IRC02:41
*** tiswanso has quit IRC02:42
*** tiswanso_ has joined #openstack-infra02:42
*** annegentle has joined #openstack-infra02:43
clarkbStevenK: thanks, for populating the zanata server with translations we just take what is in git and don't need to touch transifex?02:44
clarkbStevenK: just making sure I understand that bit (not sure if we were able to preserve history or memory or anything like that)02:45
*** Sukhdev_ has joined #openstack-infra02:45
*** gyee has quit IRC02:47
*** achanda has quit IRC02:49
StevenKclarkb: Zanata works out the translation memory from what we import.02:52
StevenKclarkb: populate-zanata-project pulls all translations from Transifex before pushing to Zanata02:53
*** annegentle has quit IRC02:57
clarkbStevenK gotcha and it uses the git repo to know wjere to talk in transifex?03:00
openstackgerritMerged openstack-infra/devstack-gate: Add Rally to projects list  https://review.openstack.org/19060103:00
StevenKclarkb: For that it uses the first argument, but tx (as in the CLI utility) needs somewhere to write to03:01
StevenKclarkb: So it's just easier to just run it from the git repo03:02
clarkbI see03:02
*** achanda has joined #openstack-infra03:02
*** changbl has joined #openstack-infra03:02
StevenKclarkb: And zanata-cli needs to read from the repo, the setup scripts write a zanata.xml into the base directory of it03:04
*** achanda has quit IRC03:08
*** asettle has joined #openstack-infra03:11
*** mmedvede has quit IRC03:11
*** sharikapongubala has quit IRC03:11
*** ddieterly has joined #openstack-infra03:13
anteayaquiet floor complete with crying baby03:14
anteayasounds very small03:14
EmilienManteaya: in MTL?03:15
*** sabeen1 has joined #openstack-infra03:16
anteayayeah03:16
EmilienManteaya: taking the train in 6h03:16
EmilienMI should go sleep eventually03:16
anteayawow early03:16
anteayayou should03:16
EmilienMbut our CI is in a bad shape today03:16
anteayawhat time to you think you will be at the office03:16
anteaya:(03:16
EmilienMI'm trying to get this solved03:16
anteayasleep now03:16
EmilienManteaya: ~9am03:16
anteayasolve tomorrow03:17
EmilienManteaya: come at 9.15am and for sure I'll be here03:17
anteayakk, I'll aim for being there at about 9 then too03:17
EmilienMnice03:17
anteayaah 9:15 then03:17
*** ddieterly has quit IRC03:17
anteayaate at ChuCha, was amazing vegan food03:18
anteayahad such a lovely dinner03:18
*** wenlock1 has joined #openstack-infra03:18
EmilienManteaya: I might have an outstanding request03:18
EmilienMI'm testing now03:19
anteayarequest away03:20
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: enable nova v20 compat testing jobs  https://review.openstack.org/21937003:21
anteayaas soon as I am finished reading the logs from the tc meeting I am signing off though03:21
anteayaI need to sleep too03:21
*** zeih has joined #openstack-infra03:23
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting puppet-keystone beaker on trusty  https://review.openstack.org/21954303:23
EmilienManteaya: ^03:23
EmilienMsorry for asking stuffs so late but this is an outstanding request that will unblock our CI so we can do our Puppet sprint safely tomorrow03:24
EmilienMclarkb: if you're still around ^03:24
*** tiswanso_ has quit IRC03:24
*** glauco has joined #openstack-infra03:26
anteayaI am done reading the tc meeting logs03:27
anteayaEmilienM: did you need me for a thing?03:27
anteayaI'm running out of reasons to stay online03:27
anteayaand I've testing all my pillows and none are feather03:27
anteayaso all eight of them get to spend the time in a stack on the other bed03:27
anteayaand this pillowless bed is looking more and more appealing03:27
*** glauco is now known as 7JTAALLKW03:27
anteayaI don't know why people need so many pillows03:27
anteayabut some folks like many pillows03:27
EmilienManteaya: https://review.openstack.org/21954303:27
* anteaya reviews03:28
*** sharikapongubala has joined #openstack-infra03:28
*** zeih has quit IRC03:28
anteayaEmilienM: +2 from me03:30
EmilienManteaya: thx03:30
anteayahopefully someone like clarkb or jhesketh will wander by soon and shove it in03:30
anteayaget some sleep03:30
anteayasee you tomorrow03:30
anteayag'night03:30
EmilienManteaya: thx03:30
EmilienMsee you tomorrow in person03:30
anteayawelcome :)03:30
* jhesketh 's ears burn03:30
anteaya'til tomorrow03:30
anteayahi jhesketh03:31
anteayag'night jhesketh03:31
*** 7JTAALLKW has quit IRC03:31
jheskethheh, goodnight!03:31
*** dchen has joined #openstack-infra03:32
*** camunoz has quit IRC03:33
*** Sree has joined #openstack-infra03:35
*** esker has joined #openstack-infra03:35
*** sflanigan has quit IRC03:35
*** btully has quit IRC03:36
*** baoli_ has quit IRC03:36
openstackgerritMerged openstack-infra/project-config: zuul/layout: make non-voting puppet-keystone beaker on trusty  https://review.openstack.org/21954303:37
Nakatofungi: clarkb: Could I get you guys to take a look at https://review.openstack.org/207262 ?  It's the project-config part of the constraints work.03:37
*** woodster_ has quit IRC03:39
*** esker has quit IRC03:40
*** salv-orlando has joined #openstack-infra03:41
*** hdd has joined #openstack-infra03:43
*** salv-orlando has quit IRC03:46
*** maishsk has quit IRC03:47
*** links has joined #openstack-infra03:48
*** puranamr has quit IRC03:49
*** camunoz has joined #openstack-infra03:49
*** sflanigan has joined #openstack-infra03:51
*** sflanigan has joined #openstack-infra03:51
*** armax has joined #openstack-infra03:52
*** puranamr has joined #openstack-infra03:52
*** dims__ has joined #openstack-infra04:03
*** jerryz has quit IRC04:03
*** tlian2 has quit IRC04:05
*** unicell has joined #openstack-infra04:10
*** hichihara has joined #openstack-infra04:11
*** bhunter71 has quit IRC04:11
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Remove nova v21 old jobs  https://review.openstack.org/21955304:11
*** ddieterly has joined #openstack-infra04:14
*** unicell has quit IRC04:16
*** xyang1 has quit IRC04:17
*** dims__ is now known as dims04:17
*** ddieterly has quit IRC04:19
*** ihrachys has joined #openstack-infra04:22
*** rlandy has joined #openstack-infra04:22
*** zeih has joined #openstack-infra04:23
*** dims has quit IRC04:24
*** unicell has joined #openstack-infra04:24
*** stevemar has quit IRC04:25
*** zeih has quit IRC04:28
*** esker has joined #openstack-infra04:30
*** ajmiller has quit IRC04:31
*** ajmiller has joined #openstack-infra04:32
zarojhesketh: could you pls help with this gate-dsvm-nodepool failure?? https://review.openstack.org/#/c/200727/04:32
*** btully has joined #openstack-infra04:33
*** sabeen1 has quit IRC04:33
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting puppet-neutron beaker on trusty  https://review.openstack.org/21955604:33
*** yfried has quit IRC04:33
*** esker has quit IRC04:34
*** jyuso1 has quit IRC04:34
jheskethzaro: I'll see what I can do04:35
*** kushal has joined #openstack-infra04:37
*** btully has quit IRC04:37
*** mrmartin has joined #openstack-infra04:39
*** achanda has joined #openstack-infra04:42
*** esker has joined #openstack-infra04:43
*** puranamr has quit IRC04:44
*** Sukhdev_ has quit IRC04:44
*** puranamr has joined #openstack-infra04:45
*** esker has quit IRC04:49
zarojhesketh: i don't really understand the failure.  was wondering if it's a blip or a real failure we need to deal with?04:50
jheskethzaro: http://logs.openstack.org/27/200727/3/check/gate-dsvm-nodepool/3300544/logs/screen-nodepool.txt.gz#_2015-09-01_18_31_31_16404:50
jheskethzaro: trying to figure that out myself.. doing a recheck at the moment04:50
*** ihrachys has quit IRC04:52
*** jcoufal has joined #openstack-infra04:52
SlickNikhi folks — a change in the tarball file names on cloud-images.ubuntu.com is causing a break in the trove gate.04:54
*** mrmartin has quit IRC04:54
SlickNikWe have a fix for this at https://review.openstack.org/#/c/219548 — is there any way to prioritize merging this since the gate is pretty long and without it all other trove changes in flight will fail?04:56
*** salv-orlando has joined #openstack-infra04:56
*** mtreinish has quit IRC04:57
ianwSlickNik: it can be promoted, but maybe jhesketh is the only person around right now that has permissions04:57
SlickNikthanks ianw04:58
SlickNik^ jhesketh: pretty please?04:58
SlickNikit's still in the check queue and should be done in ~7-8 minutes. I'm not sure if you need it to be in the gate queue for it to be promoted.05:00
*** salv-orlando has quit IRC05:01
*** mpopow__ has joined #openstack-infra05:03
*** mrmartin has joined #openstack-infra05:03
*** jyuso1 has joined #openstack-infra05:04
*** mrmartin has quit IRC05:04
*** mtreinish has joined #openstack-infra05:04
*** nadya has joined #openstack-infra05:05
*** rossella_s has quit IRC05:06
*** rossella_s has joined #openstack-infra05:06
jheskethSlickNik, ianw: promoting it will cause all of the current jobs in the integrated queue to be cancelled and re-queued05:06
jheskeththat's quite a burden05:06
*** markvoelker has joined #openstack-infra05:06
jheskethwe might be better just waiting for it to go through05:07
*** mpopow_ has quit IRC05:07
SlickNikugh05:07
*** unicell has quit IRC05:07
SlickNikI agree, it's probably not worth that.05:07
SlickNikGiven the current queue size.05:07
*** rlandy has quit IRC05:07
ianwoh, i had just assumed everything was broken until this got in, so yeah05:07
SlickNikianw: No, just trove changes.05:08
SlickNiksorry if I was unclear.05:08
ianwah, well yeah; np i didn't check :)05:08
*** ayoung is now known as ayoung_ZZzz05:08
openstackgerritRomil Gupta proposed openstack-infra/project-config: Add check-requirements to networking-vsphere  https://review.openstack.org/21955805:09
*** hdd has quit IRC05:09
*** unicell has joined #openstack-infra05:09
openstackgerritRomil Gupta proposed openstack/requirements: Add networking-vsphere to the list  https://review.openstack.org/21818905:09
*** nadya has quit IRC05:09
*** markvoelker has quit IRC05:11
*** markvoelker has joined #openstack-infra05:12
*** ddieterly has joined #openstack-infra05:14
*** Sukhdev_ has joined #openstack-infra05:17
*** ddieterly has quit IRC05:19
*** flepied has joined #openstack-infra05:23
*** puranamr has quit IRC05:23
*** zeih has joined #openstack-infra05:24
*** Sree has quit IRC05:25
*** puranamr has joined #openstack-infra05:26
*** Sree has joined #openstack-infra05:28
*** zeih has quit IRC05:28
openstackgerritIan Wienand proposed openstack/diskimage-builder: Make README.rst a bit more generic  https://review.openstack.org/21804305:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: doc: migrate from README.rst to Sphinx  https://review.openstack.org/19305105:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add DIB_KEEP_PACKAGE_CACHE variable  https://review.openstack.org/20832105:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add DIB_USE_PACKAGE_CACHE variable  https://review.openstack.org/21143405:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add #!/bin/bash to library functions  https://review.openstack.org/21810005:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Remove grub2 in redhat-common/pre-install.d/15-remove-grub  https://review.openstack.org/21112405:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Remove old entries from MANIFEST.in  https://review.openstack.org/21956105:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Handle install with pip -e  https://review.openstack.org/21956205:29
*** yfried has joined #openstack-infra05:33
*** jyuso has joined #openstack-infra05:35
*** yfried_ has joined #openstack-infra05:36
*** yfried has quit IRC05:37
*** camunoz has quit IRC05:37
*** sflanigan has quit IRC05:37
*** jyuso1 has quit IRC05:37
*** btully has joined #openstack-infra05:45
*** jyuso1 has joined #openstack-infra05:49
*** jyuso has quit IRC05:49
*** btully has quit IRC05:49
*** jcoufal has quit IRC05:50
*** camunoz has joined #openstack-infra05:51
*** salv-orlando has joined #openstack-infra05:52
*** amotoki has joined #openstack-infra05:54
*** dchen has quit IRC05:56
*** amotoki has quit IRC05:56
*** dingyichen has quit IRC05:57
*** flepied has quit IRC05:57
*** vadimk_ has quit IRC05:58
*** amotoki has joined #openstack-infra06:00
*** jerryz has joined #openstack-infra06:01
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/21914806:02
*** amotoki_ has joined #openstack-infra06:02
*** Sukhdev_ has quit IRC06:04
*** yfried_ is now known as yfried|afk06:04
openstackgerritHidekazu Nakamura proposed openstack-infra/infra-manual: Fix pip link  https://review.openstack.org/21956706:06
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/21956806:06
*** amotoki has quit IRC06:06
*** yfried|afk is now known as yfried_06:09
*** jtomasek has joined #openstack-infra06:09
*** FallenPegasus has quit IRC06:10
*** HeOS has quit IRC06:10
*** puranamr has quit IRC06:10
*** abregman has joined #openstack-infra06:11
*** ParsectiX has joined #openstack-infra06:11
*** puranamr has joined #openstack-infra06:12
*** ddieterly has joined #openstack-infra06:15
jheskethzaro: the recheck passed fyi06:17
*** ddieterly has quit IRC06:20
*** asettle has quit IRC06:23
*** markvoelker has quit IRC06:23
*** sdake has joined #openstack-infra06:24
*** zeih has joined #openstack-infra06:25
*** ildikov has quit IRC06:25
*** zeih has quit IRC06:29
*** mpaolino has joined #openstack-infra06:30
*** mrmartin has joined #openstack-infra06:31
*** mrunge has joined #openstack-infra06:34
*** flepied has joined #openstack-infra06:37
*** stevemar has joined #openstack-infra06:38
*** yfried_ is now known as yfried|afk06:38
*** stevemar has quit IRC06:42
*** glauco has joined #openstack-infra06:42
*** flepied has quit IRC06:42
*** esmute_ has joined #openstack-infra06:43
*** mrmartin has quit IRC06:45
*** yfried|afk is now known as yfried_06:46
*** glauco has quit IRC06:46
*** achanda has quit IRC06:47
*** mpopow__ has quit IRC06:47
*** puranamr has quit IRC06:49
*** dims has joined #openstack-infra06:50
*** ParsectiX has quit IRC06:51
*** afazekas__ has joined #openstack-infra06:52
*** puranamr has joined #openstack-infra06:53
*** mrmartin has joined #openstack-infra06:53
*** peristeri has joined #openstack-infra06:54
*** dims has quit IRC06:54
*** ildikov has joined #openstack-infra06:58
*** flepied has joined #openstack-infra06:59
*** liam__ has joined #openstack-infra06:59
*** jyuso has joined #openstack-infra07:01
*** esmute_ has quit IRC07:03
openstackgerritMerged openstack-infra/groups: Enable message email notification  https://review.openstack.org/21930707:03
*** jyuso1 has quit IRC07:04
*** zeih has joined #openstack-infra07:08
*** liam__ has quit IRC07:09
*** maishsk has joined #openstack-infra07:09
*** wenlock1 has quit IRC07:10
*** shashank_hegde has joined #openstack-infra07:13
openstackgerritMerged openstack-infra/devstack-gate: remove neutron advanced services from grenade  https://review.openstack.org/20127907:13
*** ParsectiX has joined #openstack-infra07:13
*** markus_z has joined #openstack-infra07:13
*** ddieterly has joined #openstack-infra07:16
*** sshnaidm has joined #openstack-infra07:20
*** ddieterly has quit IRC07:20
*** coolsvap_ has joined #openstack-infra07:21
*** achanda has joined #openstack-infra07:24
openstackgerritSachi King proposed openstack-infra/project-config: Add constraints jobs to Jenkins  https://review.openstack.org/20726207:25
*** shardy has joined #openstack-infra07:25
*** coolsvap_ has quit IRC07:26
*** ifarkas has joined #openstack-infra07:26
*** zeih has quit IRC07:28
*** vivekd has quit IRC07:28
openstackgerritSachi King proposed openstack-infra/project-config: Add non-voting constraints jobs to nova  https://review.openstack.org/21958207:31
yuanyinghi07:32
yuanyingAnyone know, how to merge this https://review.openstack.org/#/c/216933/607:32
yuanying?07:32
*** jerryz has quit IRC07:33
*** armax has quit IRC07:34
*** BharatK has quit IRC07:34
*** eantyshev has joined #openstack-infra07:35
*** shardy_ has joined #openstack-infra07:36
*** jerryz has joined #openstack-infra07:37
*** shardy has quit IRC07:37
*** jyuso has quit IRC07:38
*** arxcruz has joined #openstack-infra07:40
*** fhubik has joined #openstack-infra07:41
*** shardy_ has quit IRC07:41
*** jlanoux has joined #openstack-infra07:41
*** shardy has joined #openstack-infra07:42
*** zeih_ has joined #openstack-infra07:42
*** dims has joined #openstack-infra07:44
*** jlanoux has quit IRC07:45
*** esker has joined #openstack-infra07:48
*** BharatK has joined #openstack-infra07:48
*** puranamr_ has joined #openstack-infra07:48
*** mpopow__ has joined #openstack-infra07:48
*** puranamr has quit IRC07:48
*** dims has quit IRC07:49
*** k4n0 has joined #openstack-infra07:49
openstackgerritMarton Kiss proposed openstack-infra/groups: Refactor Ambassador pages layout  https://review.openstack.org/21958807:50
*** achanda has quit IRC07:51
*** mpopow__ has quit IRC07:52
*** esker has quit IRC07:52
*** BobBall-AWOL is now known as BobBall07:54
*** yfried_ is now known as yfried|afk07:54
*** jyuso1 has joined #openstack-infra07:55
openstackgerritMerged openstack-infra/groups: Refactor Ambassador pages layout  https://review.openstack.org/21958807:56
*** yfried|afk is now known as yfried_07:56
*** vinayyadhav has joined #openstack-infra07:56
*** jlanoux has joined #openstack-infra07:56
*** fhubik has quit IRC07:57
*** larainema has quit IRC07:57
*** b10n1k_ has quit IRC07:57
*** larainema has joined #openstack-infra07:58
*** e0ne has joined #openstack-infra08:01
*** shashank_hegde has quit IRC08:01
*** ildikov has quit IRC08:08
*** hashar has joined #openstack-infra08:09
*** jistr has joined #openstack-infra08:11
*** lucas-dinner has joined #openstack-infra08:12
*** lucas-dinner has quit IRC08:12
openstackgerritMerged openstack/requirements: Add tosca-parser to global requirements  https://review.openstack.org/21799108:12
*** e0ne has quit IRC08:13
*** mquin_ is now known as mquin08:14
*** ddieterly has joined #openstack-infra08:17
*** zeih_ has quit IRC08:19
*** zeih has joined #openstack-infra08:19
*** ddieterly has quit IRC08:21
*** ildikov has joined #openstack-infra08:21
*** hichihara has quit IRC08:21
*** vivekd has joined #openstack-infra08:24
Kennanhi any infra cores here?08:24
Kennanone project hit one issue08:24
Kennanabout projects docs publish08:25
Kennanwe added jenkins zuul jobs about docs publish08:25
Kennanbut we did not find docs published in openstack.org08:25
Kennandoes infra cores know about it ?08:25
*** pelix has joined #openstack-infra08:25
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101108:26
*** fhubik has joined #openstack-infra08:27
*** earlephilhower has quit IRC08:28
*** cdent has joined #openstack-infra08:29
*** vinayyadhav has quit IRC08:31
*** ihrachys has joined #openstack-infra08:33
*** HeOS has joined #openstack-infra08:35
openstackgerritSachi King proposed openstack-infra/project-config: Add non-voting constraints jobs to nova  https://review.openstack.org/21958208:36
openstackgerritSachi King proposed openstack-infra/project-config: Add constraints jobs to Jenkins  https://review.openstack.org/20726208:36
*** ParsectiX has quit IRC08:37
*** dizquierdo has joined #openstack-infra08:37
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip  https://review.openstack.org/21101608:37
*** ihrachys has quit IRC08:37
*** liam__ has joined #openstack-infra08:38
*** liam__ has quit IRC08:38
*** dims has joined #openstack-infra08:38
Kennanlifeless: there?08:42
Kennanor EmilientM ?08:42
*** marzif has joined #openstack-infra08:43
*** puranamr_ has quit IRC08:43
*** dims has quit IRC08:43
*** zeih has quit IRC08:43
*** puranamr has joined #openstack-infra08:45
*** zeih has joined #openstack-infra08:47
openstackgerritKai Qiang Wu(Kennan) proposed openstack/requirements: Add kolla to requirements family  https://review.openstack.org/21814408:47
*** derekh has joined #openstack-infra08:50
*** mrmartin has quit IRC08:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip  https://review.openstack.org/21101608:52
*** markvoelker has joined #openstack-infra08:57
openstackgerritSachi King proposed openstack-infra/project-config: Add non-voting constraints jobs to neutron  https://review.openstack.org/21961008:58
*** amotoki_ has quit IRC08:58
*** peristeri has quit IRC08:58
*** markvoelker has quit IRC09:02
openstackgerritYe-hong Fang proposed openstack/diskimage-builder: Fix grub-efi-amd64-signed install failure  https://review.openstack.org/21961209:05
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-pip: Fix the puppet-pip module to install pip  https://review.openstack.org/21101609:05
lifelessKennan: just ask the question...09:06
*** rossella_s has quit IRC09:06
*** rossella_s has joined #openstack-infra09:06
*** john-davidge has joined #openstack-infra09:07
*** salv-orlando has quit IRC09:07
*** bexelbie_ has joined #openstack-infra09:08
*** mrmartin has joined #openstack-infra09:09
*** jerryz has quit IRC09:10
*** john-davidge_ has joined #openstack-infra09:10
*** john-davidge has quit IRC09:12
*** john-davidge_ is now known as john-davidge09:12
*** ParsectiX has joined #openstack-infra09:15
*** kaisers1 has joined #openstack-infra09:16
*** sanjayu has joined #openstack-infra09:17
*** ddieterly has joined #openstack-infra09:18
*** flepied1 has joined #openstack-infra09:19
*** amotoki has joined #openstack-infra09:19
*** kiran-r has joined #openstack-infra09:20
*** ParsectiX has quit IRC09:20
*** sharikapongubala has quit IRC09:21
*** peristeri has joined #openstack-infra09:21
*** flepied has quit IRC09:21
*** sanjayu_ has quit IRC09:21
*** marzif has quit IRC09:22
*** ddieterly has quit IRC09:22
*** peristeri has quit IRC09:23
openstackgerritLucas Alvares Gomes proposed openstack/diskimage-builder: Map ubuntu versions to match the SHA256SUMs  https://review.openstack.org/21919909:23
Kennanhi lifeless:09:24
KennanI asked about docs publish questions09:24
Kennancould you help that ?09:24
*** peristeri has joined #openstack-infra09:24
Kennanlifeless: we added openstack-server-publish-jobs09:25
Kennanbut seems the docs.openstack.org not publish our dcos09:25
Kennando you know what's the issue ? lifeless:09:25
*** dizquierdo has quit IRC09:26
*** yamamoto has quit IRC09:27
*** puranamr has quit IRC09:28
Kennanlifeless:09:28
Kennanhere is the link https://review.openstack.org/#/c/218137/09:28
*** kushal has quit IRC09:30
*** amotoki has quit IRC09:30
*** e0ne has joined #openstack-infra09:32
EmilienMgood morning infra, I have an outstanding review for Puppet : https://review.openstack.org/#/c/219556/09:32
*** puranamr has joined #openstack-infra09:32
*** dims has joined #openstack-infra09:32
*** esker has joined #openstack-infra09:36
*** dims has quit IRC09:37
*** stevemar has joined #openstack-infra09:38
*** jhesketh has quit IRC09:39
*** andrew has joined #openstack-infra09:40
*** ociuhandu has quit IRC09:41
*** esker has quit IRC09:41
*** jhesketh has joined #openstack-infra09:41
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Decouple openstack_project::server from puppetdb class  https://review.openstack.org/17287109:42
*** glauco_ has joined #openstack-infra09:42
*** stevemar has quit IRC09:42
*** andrew has quit IRC09:44
*** bigtaro has joined #openstack-infra09:46
*** e0ne has quit IRC09:46
*** glauco_ has quit IRC09:46
*** zeih has quit IRC09:50
*** salv-orlando has joined #openstack-infra09:50
*** yamamoto has joined #openstack-infra09:53
*** e0ne has joined #openstack-infra09:55
*** bigtaro has quit IRC09:59
*** hashar_ has joined #openstack-infra09:59
*** bigtaro has joined #openstack-infra10:00
*** fhubik is now known as fhubik_brb10:00
*** fhubik_brb is now known as fhubik10:00
*** fhubik is now known as fhubik_brb10:00
*** regXboi has joined #openstack-infra10:01
*** hashar has quit IRC10:02
*** yamamoto has quit IRC10:03
*** yamamoto has joined #openstack-infra10:04
*** yamamoto has quit IRC10:04
*** yamamoto has joined #openstack-infra10:04
*** ociuhandu has joined #openstack-infra10:05
*** marzif has joined #openstack-infra10:06
*** yamamoto has quit IRC10:06
electricallifeless: finally implementing JJB further at Elastic :-)  have to teach co-workers on how to use it and how the structure works :-)10:06
lifelesselectrical: cool,gl10:07
openstackgerritMerged openstack-infra/project-config: Add stable/liberty crosscheck jobs for reqs  https://review.openstack.org/21730010:07
*** yamamoto has joined #openstack-infra10:12
*** ParsectiX has joined #openstack-infra10:13
*** esker has joined #openstack-infra10:14
*** bigtaro has quit IRC10:16
*** amotoki has joined #openstack-infra10:16
*** ddieterly has joined #openstack-infra10:18
*** esker has quit IRC10:18
*** kiran-r has quit IRC10:19
*** puranamr has quit IRC10:19
*** yamamoto has quit IRC10:20
*** sanjayu_ has joined #openstack-infra10:20
*** ihrachys has joined #openstack-infra10:21
*** puranamr has joined #openstack-infra10:21
*** ihrachys has quit IRC10:22
*** ddieterly has quit IRC10:23
*** hashar has joined #openstack-infra10:23
*** sanjayu has quit IRC10:24
*** hashar_ has quit IRC10:25
*** dims has joined #openstack-infra10:26
*** dims has quit IRC10:31
*** pc_m has joined #openstack-infra10:31
*** wenlock has quit IRC10:33
*** wenlock has joined #openstack-infra10:33
*** gsagie has joined #openstack-infra10:34
gsagieHello anyone, any infrastructure core can review the following patch: https://review.openstack.org/#/c/218575/   will appreciate it alot :)10:35
gsagieeveryone :) not anyone10:35
openstackgerritpangwa proposed openstack-infra/git-review: fix encoding issue on Windows  https://review.openstack.org/21816710:38
*** chmouel has quit IRC10:40
*** akuznetsova has quit IRC10:41
*** chmouel has joined #openstack-infra10:41
*** akuznetsova has joined #openstack-infra10:41
*** yfried_ is now known as yfried|afk10:45
*** fhubik_brb is now known as fhubik10:46
*** ZZelle has quit IRC10:48
*** ZZelle has joined #openstack-infra10:48
*** daemontool_ has joined #openstack-infra10:50
pabelangermorning10:50
*** esker has joined #openstack-infra10:50
*** yamamoto has joined #openstack-infra10:51
*** larainema has quit IRC10:54
*** larainema has joined #openstack-infra10:54
*** fhubik is now known as fhubik_brb10:56
*** fhubik_brb is now known as fhubik10:56
*** claudiub has joined #openstack-infra10:57
*** amotoki has quit IRC10:57
*** trown|outttypeww is now known as trown10:58
openstackgerritGary Duan proposed openstack/diskimage-builder: Contains the directory name of /sys and /proc  https://review.openstack.org/21506010:59
*** yfried has joined #openstack-infra11:01
*** yfried|afk has quit IRC11:01
*** glauco has joined #openstack-infra11:02
*** purp_2 has joined #openstack-infra11:02
*** purp has quit IRC11:02
*** jgriffith has quit IRC11:02
*** eharney has quit IRC11:02
*** yfried_ has joined #openstack-infra11:02
*** yfried_ is now known as yfried|afk11:02
*** jgriffith has joined #openstack-infra11:03
*** eharney has joined #openstack-infra11:04
*** jgriffith is now known as Guest9183411:04
*** amotoki has joined #openstack-infra11:04
*** yfried has quit IRC11:06
*** glauco has quit IRC11:06
*** hashar_ has joined #openstack-infra11:08
*** kushal has joined #openstack-infra11:09
*** puranamr has quit IRC11:10
*** amotoki has quit IRC11:10
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515111:10
*** hashar has quit IRC11:11
*** puranamr has joined #openstack-infra11:13
*** baoli_ has joined #openstack-infra11:13
*** kushal has quit IRC11:14
*** mwagner_lap has joined #openstack-infra11:15
*** mmedvede has joined #openstack-infra11:17
*** yfried|afk has quit IRC11:18
*** ddieterly has joined #openstack-infra11:19
*** fhubik has quit IRC11:20
*** fhubik has joined #openstack-infra11:20
*** dizquierdo has joined #openstack-infra11:20
*** dims has joined #openstack-infra11:21
*** wenlock has quit IRC11:21
*** kiran-r has joined #openstack-infra11:21
*** wenlock has joined #openstack-infra11:22
svashekahi guys could you please take a look at https://review.openstack.org/#/c/219254/1, it fails, but there's no appropriate message in logs11:23
svashekais value too long?11:23
*** kushal has joined #openstack-infra11:23
*** ddieterly has quit IRC11:24
*** esker has quit IRC11:26
*** dims has quit IRC11:26
*** esker has joined #openstack-infra11:26
*** yfried|afk has joined #openstack-infra11:27
*** amotoki has joined #openstack-infra11:28
*** zeih has joined #openstack-infra11:29
*** baoli has joined #openstack-infra11:31
*** yfried|afk has quit IRC11:31
*** yfried has joined #openstack-infra11:31
*** yfried is now known as yfried|afk11:31
*** baoli_ has quit IRC11:33
*** wenlock has quit IRC11:34
*** bharathm has quit IRC11:36
*** gordc has joined #openstack-infra11:38
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515111:42
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix exception lists in functional tests  https://review.openstack.org/21965511:42
*** bapalm has quit IRC11:43
*** vinay_yadhav has joined #openstack-infra11:44
*** ldnunes has joined #openstack-infra11:46
*** bapalm has joined #openstack-infra11:49
*** esker has quit IRC11:52
*** thorst has joined #openstack-infra11:54
*** rfolco has joined #openstack-infra11:54
*** nadya has joined #openstack-infra11:55
*** abregman is now known as abregman|brb11:55
*** puranamr has quit IRC11:57
anteayamorning EmilienM11:58
openstackgerritJulien Danjou proposed openstack/requirements: tooz needs psycopg2>=2.5  https://review.openstack.org/21966311:59
*** puranamr has joined #openstack-infra12:00
*** nadya has quit IRC12:01
*** mmedvede has quit IRC12:01
*** zeih has quit IRC12:02
*** amuller has joined #openstack-infra12:02
anteayamorning pabelanger12:03
anteayapabelanger: when do you expect to be in Montreal?12:03
*** abregman|brb is now known as abregman12:03
*** mmedvede has joined #openstack-infra12:05
*** stevemar has joined #openstack-infra12:05
*** jaypipes has joined #openstack-infra12:08
*** sigmavirus24_awa is now known as sigmavirus2412:08
*** stevemar has quit IRC12:09
*** yfried|afk is now known as yfried12:10
*** mwagner_lap has quit IRC12:10
*** mwagner_lap has joined #openstack-infra12:10
*** jcoufal has joined #openstack-infra12:11
*** bswartz has joined #openstack-infra12:12
*** spzala has joined #openstack-infra12:12
*** dprince has joined #openstack-infra12:13
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add constraints jobs to Jenkins  https://review.openstack.org/20726212:14
*** vinay_yadhav has quit IRC12:15
*** vinay_yadhav has joined #openstack-infra12:16
*** Ala has quit IRC12:16
*** ajmiller has quit IRC12:16
*** aysyd has joined #openstack-infra12:18
*** Ala has joined #openstack-infra12:18
fungiKennan: your docs job isn't working because your tox.ini is incomplete. here's the failing job log for your most recent merge http://logs.openstack.org/34/34937175172e44ed67d271f74540b7f5e0de7c6d/post/kolla-docs/7a9e70c/console.html#_2015-09-02_10_42_45_022 you need to create a section for an env named "venv" in your tox.ini file. see other openstack projects for examples12:19
*** ddieterly has joined #openstack-infra12:20
*** ddieterly has quit IRC12:21
*** ddieterly has joined #openstack-infra12:22
*** jcoufal has quit IRC12:22
*** madong_ has joined #openstack-infra12:23
*** larainema has quit IRC12:23
*** flepied1 has quit IRC12:24
fungisvasheka: that "fatal: remote did not send all necessary objects" failure on our layout jobs is because we're having trouble with at least some shallow clones from our git servers (which that job does). recheck for now, we have a probable solution to try12:25
*** hashar_ has quit IRC12:25
*** ivar-lazzaro has joined #openstack-infra12:26
svashekafungi, thanks12:26
*** mcdonagm has joined #openstack-infra12:26
*** purp_2 is now known as purp12:27
*** david-lyle has quit IRC12:27
*** mordred has quit IRC12:27
*** mfmcdonagh has quit IRC12:28
*** dansmith has quit IRC12:28
*** Swanson has quit IRC12:28
*** timrc has quit IRC12:28
*** haleyb has quit IRC12:28
rakhmerovfungi: not sure if it's your area but I need to land the patch https://review.openstack.org/#/c/194999/ into openstack-requirements12:28
rakhmerovit's been on review for a while12:28
*** david-lyle has joined #openstack-infra12:28
rakhmerovwho would be the best person to talk to about it?12:29
anteayarakhmerov: requirements core reviewers: https://review.openstack.org/#/admin/groups/131,members12:30
fungirakhmerov: the regular requirements-core members are the ones who should be reviewing those https://review.openstack.org/#/admin/groups/131,members (release managers and infra-core are only included groups because sometimes we need to un-break things project-wide by quickly approving requirements changes)12:30
*** hashar has joined #openstack-infra12:31
*** sharikapongubala has joined #openstack-infra12:31
rakhmerovok, thnx12:32
*** dansmith has joined #openstack-infra12:33
*** timrc has joined #openstack-infra12:33
*** ivar-lazzaro has quit IRC12:33
*** dansmith is now known as Guest9133912:33
*** dkranz has joined #openstack-infra12:36
*** bharathm has joined #openstack-infra12:37
*** ddieterly has quit IRC12:37
*** nicodemos has joined #openstack-infra12:38
*** glauco has joined #openstack-infra12:38
*** claudiub has quit IRC12:38
*** vinay_yadhav has quit IRC12:38
*** cburroughs has joined #openstack-infra12:38
*** gsagie has left #openstack-infra12:39
*** yfried is now known as yfried|afk12:40
*** _kiran_ has joined #openstack-infra12:41
*** Swanson has joined #openstack-infra12:41
*** haleyb has joined #openstack-infra12:41
*** bharathm has quit IRC12:41
*** glauco has quit IRC12:42
*** kiran-r has quit IRC12:42
*** _kiran_ is now known as kiran-r12:43
*** yfried|afk is now known as yfried12:44
anteayafungi: can you change the topic in openstack-sprint to reflect the current puppet sprint?12:44
anteayaplease12:44
anteayaEmilienM: ^12:44
*** mpaolino_ has joined #openstack-infra12:44
fungianteaya: sure, what should the new topic be?12:44
*** ivar-lazzaro has joined #openstack-infra12:45
*** mpaolino has quit IRC12:45
fungiEmilienM: ^ ?12:45
*** sigmavirus24 is now known as sigmavirus24_awa12:45
*** dims has joined #openstack-infra12:48
*** bexelbie_ is now known as bexelbie12:49
anteayaI'll let EmilienM say but probably something that includes a link to the etherpad: https://etherpad.openstack.org/p/puppet-liberty-mid-cycle12:50
anteayafungi: he is probably still on the train and should be online again shortly12:50
anteayathanks12:50
* anteaya changes locations12:51
*** jyuso1 has quit IRC12:51
fungii'll put in a temporary guess, thanks12:51
*** alivigni has joined #openstack-infra12:51
*** annegentle has joined #openstack-infra12:51
fungican always adjust it again later if needed12:52
*** e0ne has quit IRC12:53
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove last vestige of glanceclient being different  https://review.openstack.org/21929112:54
*** sanjayu has joined #openstack-infra12:55
*** mpaolino_ has quit IRC12:56
*** bswartz has quit IRC12:58
sdaguefungi: so today's level of fire ....12:58
*** sanjayu_ has quit IRC12:58
sdaguethe neutron grenade job is 100% failing because of a devstack-gate change, as how neutron extension tests work was not well understood. There is a tempest fix, but it's bouncing off of other neutron races.12:59
*** mase_x200 has joined #openstack-infra12:59
sdaguealso heat is 100% fail on their functional job12:59
fungii need to get a fireman's helmet13:00
fungiand send it to you13:00
*** mpaolino has joined #openstack-infra13:00
*** ildikov has quit IRC13:00
fungihow else can i help? still tracking on the same etherpad, or a new one?13:01
*** kushal has quit IRC13:01
*** mwagner_lap has quit IRC13:01
*** bcourt has quit IRC13:01
fungiif nothing else i can try to keep an eye on those patches and try to make sure they get through13:01
*** mpaolino has quit IRC13:01
*** jcoufal has joined #openstack-infra13:02
sdagueso, honestly, https://review.openstack.org/#/c/219632/ is so straight forward, it should just merge13:02
sdaguemy patience on playing recheck games on that is low13:02
*** kiran-r has quit IRC13:02
*** mpaolino has joined #openstack-infra13:02
*** sanjayu_ has joined #openstack-infra13:04
*** Ala has quit IRC13:04
*** vivekd has quit IRC13:04
*** rossella_s has quit IRC13:05
*** rossella_s has joined #openstack-infra13:06
*** ddieterly has joined #openstack-infra13:06
mescanefhello. please review: https://review.openstack.org/217589 https://review.openstack.org/218154 and https://review.openstack.org/218769 :-)13:06
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Add support for listing folder indexes  https://review.openstack.org/21420713:06
*** vivekd has joined #openstack-infra13:06
*** sanjayu has quit IRC13:07
openstackgerritMonty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface  https://review.openstack.org/21424313:09
*** Ala has joined #openstack-infra13:10
sdaguefungi: I'm also writing some scripting to build graphite fail graphs faster than by hand - https://github.com/sdague/failopotamus13:11
*** flepied has joined #openstack-infra13:11
*** mordred has joined #openstack-infra13:11
fungimad props for the name alone13:12
*** rlandy has joined #openstack-infra13:12
*** ildikov has joined #openstack-infra13:12
*** zeih has joined #openstack-infra13:14
fungioh, wow. now that i'm at a computer that can render the zuul status page, i see what you mean13:16
*** tiswanso has joined #openstack-infra13:16
*** bpoulos has joined #openstack-infra13:16
*** mpaolino has quit IRC13:16
*** bcourt has joined #openstack-infra13:17
*** links has quit IRC13:17
*** dims_ has joined #openstack-infra13:17
*** esker has joined #openstack-infra13:18
sdagueyeh, it's kind of been like this for a while13:19
fungiwell, it had caught up well last night13:19
sdaguethe heat thing is just going to be a wrecking ball13:19
sdagueanyway, you can failgraph.py gate-heat-dsvm-functional-orig-mysql13:19
fungido we need to just stop running the heat functional tests in the gate? any details on how complicated the issue is there?13:20
*** esker has quit IRC13:20
sdagueand it opens up a graphite view comparing both check and gate queues13:20
*** dims has quit IRC13:21
sdaguehonestly, I don't know, and I'm about to go for a bike ride, because having to dig into every project's tests to figure out why they've gone into left field is frustrating13:21
sdaguehonestly, I've continued to believe that you should only be able to run tests in the gate which affect multiple projects13:21
*** kushal has joined #openstack-infra13:21
fungiyep, go do that. i need to catch up on the ml to make sure i know what the current status is on people solving these13:21
sdagueand that all the single project tests should never be there13:21
fungibefore i start hounding them in irc13:22
sdaguebut I think I'm of minority opinion on that one13:22
*** ivar-lazzaro has quit IRC13:22
*** vinay_yadhav has joined #openstack-infra13:23
fungiit would likely result in projects continually breaking their own check jobs, though not convinced forcing them to deal with that afterwards is necessarily bad13:24
*** vivekd_ has joined #openstack-infra13:25
*** mriedem_away is now known as mriedem13:25
*** mwagner_lap has joined #openstack-infra13:25
*** vivekd has quit IRC13:25
*** esker has joined #openstack-infra13:25
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Add support for listing folder indexes  https://review.openstack.org/21420713:25
*** vivekd_ is now known as vivekd13:26
*** esker has quit IRC13:26
sdaguefungi: right, then it would be project isolated13:27
*** ajmiller has joined #openstack-infra13:27
*** lucasagomes has joined #openstack-infra13:27
*** esker has joined #openstack-infra13:27
fungiyep13:27
*** ajmiller has quit IRC13:27
*** ajmiller has joined #openstack-infra13:28
shardysdague: Hey I think the heat gate is broken due to bug #149119913:28
openstackbug 1491199 in heat "glance: error: unrecognized arguments: --is-public True --location..." [Critical,In progress] https://launchpad.net/bugs/1491199 - Assigned to Angus Salkeld (asalkeld)13:28
shardysdague: there's a fix posted but not yet merged13:28
shardytl;dr glanceclient released and broke us13:28
sdagueshardy: so, when you identify something like that, it would also be good to message to your core team to stop approving patches until that's fixed13:29
*** zul has quit IRC13:29
fungishardy: thanks, i was digging into the -dev ml looking for signs of some status like that. i can push that change to the front of the gate now to keep all the other heat changes currently in the queue from making a mess13:29
shardysdague: Ok, fair point, sorry I think I'm guilty of approving some things, I thought that was near landing earlier thus was at the head of the queue13:30
shardyapologies13:30
shardyfungi: it's a critical bug on the heat tracker FWIW, reported 12h ago13:30
*** ajmiller_ has joined #openstack-infra13:30
*** glauco has joined #openstack-infra13:31
*** ajmiller has quit IRC13:31
*** kgiusti has joined #openstack-infra13:31
fungiyeah, 6 other heat changes ahead of it which are all going to have to fail before it gets to the front unless i reprioritize it13:31
*** zul has joined #openstack-infra13:32
shardyfungi: Ok, if you can bump https://review.openstack.org/#/c/219533/ to the top of the queue that'd be great, thanks!13:33
sdaguefungi: also, promote - https://review.openstack.org/#/c/21963213:33
sdagueit passed check13:33
sdagueprobably put the heat change in front of it13:33
shardyOut of interest, how hard would it be to give core approves a comment tag which bumps gate-fix patches to the top?13:33
shardygatefix bug ###13:33
fungisdague: yep, had them both queued up in the cli. i'd already pushed that one into the gate anyway in case it didn't pass check sooner13:33
sdagueshardy: hard, plus it would be massively abused13:33
openstackgerritMonty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions  https://review.openstack.org/18317613:34
sdagueif it was going to be done it should be keyed off of something like the elastic recheck bug list13:34
fungiright now we only reprioritize changes in the gate based on conversations between infra and qa13:34
shardysdague: that's sad, I think in our community it'd be effectively self-policed and all core reviewers could be trusted to use it sanely13:34
shardyjust trying to think of ways to spread the load wrt fixing these things13:34
sdagueshardy: really? because I see core team members blindly rechecking patches all the time which is part of what gets us into these high failure rates13:35
fungishardy: if projects were quicker to revoke core reviewership from people who were sloppy, careless, didn't attend meetings, et cetera then maybe13:35
*** ajmiller_ has quit IRC13:35
fungiand if the tc were quicker to punt project-teams out of the tent for not policing themselves, perhaps13:35
*** amotoki has quit IRC13:36
sdaguewe've talked about keying off of the elastic-recheck list, as those are bugs that we know are impacting a number of jobs. However, it's still quite a bit off effort to build that kind of priorty reorganization system.13:36
shardysdague: Ok, cool, I was just reflecting on how this might have been solved sooner, given that the patch was approved 10 hours ago13:37
shardynext time I'll try to reach out to you folks sooner, thanks for the help! :)13:38
*** amitgandhinz has joined #openstack-infra13:38
sdagueshardy: yeh, well that assumes all teams are keeping their test failure rates low13:38
*** kdas_ has joined #openstack-infra13:38
sdaguewhich... as you can see, isn't the case, which is why it's still in the queue :)13:38
shardysdague: well, in this case, we were, but then we got broken by a client release which we couldn't have predicted13:39
sdagueshardy: right, I mean all teams13:39
shardyunfortunately that does happen semi-regularly because we use all the clients :(13:39
*** tlian has joined #openstack-infra13:39
*** fhubik has quit IRC13:39
sdagueso, you could have sped things up by going and plucking all the heat changes above your fix out of the gate13:39
*** fhubik has joined #openstack-infra13:40
shardysdague: Do we do that just by -2ing them?13:40
sdaguethat was a huge reason this was delayed13:40
sdagueno, you have to update the commit13:40
fungieasiest way is to adjust their commit message or rebase them13:40
fungianything that creates a new patchset13:40
*** edmondsw has joined #openstack-infra13:41
*** bcourt has quit IRC13:41
fungibecause then they're immediately ejected from the gate pipeline and have to go back through check again13:41
sdagueor, come into infra and ask. Because if you know you've got patches in the gate that can't pass, it's the team's responsibility to raise that13:41
shardyOk, that just seems like a much harder way to do what I imagined re the gatefix thing, but I'll remember that next time, thanks!13:41
*** kushal has quit IRC13:41
EmilienMfungi, anteaya current topic for #openstack-sprint is perfect - thx13:42
fungimost of the time it's not really needed, but here at the milestone 3 rush when everyone has waited until the last minute to merge their random (and often conflicting) feature implementations, things get really bad really fast13:42
mordredsdague: did something that you know of change in nova? we've got a functional test failing on deleting a floating ip - and it says "invalid id" to the IP we passed - do we need to interact with floating ips by ID now?13:42
*** madong_ is now known as larainema13:43
sdaguemordred: not that I know of, but I'm actually going for that bike ride now. If you write up the issue I'll look this afternoon.13:43
mordredsdague: ok. enjoy the bike13:43
mordredsdague: and I can figure it out - was just asking if you happened to know13:43
*** ajmiller has joined #openstack-infra13:43
*** annegentle has quit IRC13:43
*** glauco has quit IRC13:44
shardyfungi: Ok, thanks, I think I/we just misunderstood the expense of waiting for the fix to merge in this case13:45
shardyfungi, sdague: thanks for the assistance/patience! :)13:45
*** dbite has quit IRC13:45
*** puranamr has quit IRC13:46
*** ajmiller has quit IRC13:46
*** glauco has joined #openstack-infra13:46
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515113:47
*** prad- has joined #openstack-infra13:48
*** puranamr has joined #openstack-infra13:49
*** dguitarbite has joined #openstack-infra13:49
*** BharatK has quit IRC13:49
*** Guest91339 is now known as dansmith13:51
*** dansmith is now known as Guest3643113:51
*** spzala has quit IRC13:51
*** spzala_ has joined #openstack-infra13:51
*** spzala_ is now known as spzala13:52
*** spzala has quit IRC13:54
*** bcourt has joined #openstack-infra13:54
*** kdas_ is now known as kushal13:54
*** kushal is now known as Guest5618813:55
*** Guest36431 is now known as dansmith13:56
*** mriedem is now known as mriedem_meeting13:59
*** mrmartin has quit IRC14:00
*** vinay_yadhav has quit IRC14:00
*** vinay_yadhav has joined #openstack-infra14:01
*** markvoelker has joined #openstack-infra14:01
*** bswartz has joined #openstack-infra14:01
*** Guest56188 is now known as kushal14:03
*** sridhar_ram has joined #openstack-infra14:03
*** kushal has quit IRC14:03
*** kushal has joined #openstack-infra14:03
*** burgerk has joined #openstack-infra14:04
*** julim has joined #openstack-infra14:04
*** BharatK has joined #openstack-infra14:04
*** marzif has quit IRC14:05
*** bharathm has joined #openstack-infra14:06
*** markvoelker has quit IRC14:06
bswartzdoes anyone know how to change the channel topic in IRC for channels managed by the bot?14:06
*** marzif has joined #openstack-infra14:06
*** zz_dimtruck is now known as dimtruck14:06
*** xnox has quit IRC14:07
*** mrmartin has joined #openstack-infra14:09
*** kiran-r has joined #openstack-infra14:09
*** _kiran_ has joined #openstack-infra14:09
*** vivekd_ has joined #openstack-infra14:09
*** xnox has joined #openstack-infra14:09
*** zeih has quit IRC14:09
fungibswartz: what channel?14:10
bswartzfungi: #openstack-manila14:10
*** bharathm has quit IRC14:10
*** _kiran_ has quit IRC14:10
*** mugsie has quit IRC14:10
*** _kiran_ has joined #openstack-infra14:11
bswartzfungi: I'm hoping that I have the power to change the topic since I'm the PTL14:11
fungibswartz: anyone with +t for that channel can just "/topic some topic" or channel administrators can "/msg chanserv topic #openstack-manila some topic"14:11
*** ParsectiX has quit IRC14:11
* fungi checs the channel permissions14:11
*** mugsie has joined #openstack-infra14:11
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101114:12
bswartzfungi: I'm not authorized14:12
*** vivekd has quit IRC14:12
*** vivekd_ is now known as vivekd14:12
lucasagomesclarkb, if you have some time, mind taking a look at https://review.openstack.org/#/c/219351/3/jenkins/jobs/devstack-gate.yaml ?14:12
fungibswartz: yeah, it looks like that channel never got any project-team founders14:12
bswartzfungi: how can we fix that?14:12
fungibswartz: anyway, just a sec and i'll add you14:12
bswartzfungi: ty14:13
lucasagomesclarkb, me and the rest are a bit confused about how things works there14:13
*** Piet has quit IRC14:13
*** kiran-r has quit IRC14:13
fungibswartz: try again14:14
mordredfungi, clarkb: do I remember some issue with devstack as part of the insanity with floating ips?14:14
bswartzfungi: works, ty14:14
clarkbmordred no just that you have to open security groups14:14
*** annegentle has joined #openstack-infra14:14
fungimordred: no recollection14:15
mordredok14:15
mordredclarkb: so nothing about not being able to delete a floating ip by ip address14:15
*** zeih has joined #openstack-infra14:15
clarkbno14:16
fungimordred: fwiw i've only ever manipulated floating ips via id, not address14:16
clarkbthat worked as of the last time I did multinode things14:16
fungididn't know you could use the address too14:16
*** zeih has quit IRC14:16
mordredheh. well, maybe I should just rework the code to always use id14:16
*** zeih has joined #openstack-infra14:17
*** _kiran_ has quit IRC14:18
*** kiran-r has joined #openstack-infra14:18
clarkblucasagomes: I think I understand the question there and yes you need to do a bit extra14:18
clarkblucasagomes: that change has defined a jenkins job template but never instantiates that template with any concrete parameters, this means the job cannot be created in jenkins14:18
clarkblucasagomes: so you need to instantiate the job that usually happens in projects.yaml under a projects jobs: list14:19
clarkblucasagomes: then you haveto tell zuul when you wantto run that job in zuul/layout.yaml14:19
*** kdas_ has joined #openstack-infra14:19
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source  https://review.openstack.org/21935114:19
*** kdas_ has quit IRC14:21
lucasagomesclarkb, gotcha! updated the patch14:21
lucasagomesthanks14:21
*** kushal has quit IRC14:22
*** zeih has quit IRC14:22
*** thiagop has joined #openstack-infra14:22
clarkbfungi: we still need to fix haproxy for the layout job ya?14:23
*** wenlock has joined #openstack-infra14:24
*** zhenguo has quit IRC14:24
*** ajmiller has joined #openstack-infra14:25
*** wenlock has quit IRC14:26
fungiclarkb: yep, was hoping to get around to pushing up a patch for that shortly, unless you want to14:26
clarkbfungi: I am writing it now14:26
*** Guest91834 is now known as jgriffith14:26
mordredShrews, SpamapS: actually, in looking at it - I tihnk our floating ip cleanup in the shade functional tests is wrong14:26
*** wenlock has joined #openstack-infra14:26
mordredwe're passing ips in to cleanup that we do not actually know to be floating ips14:27
openstackgerritClark Boylan proposed openstack-infra/system-config: Use source balancing to transition to centos7  https://review.openstack.org/21970914:27
mordredalthough it's mostly likely that they will be in the envs we've been running in14:27
anteayado we have a gerrit topic for zuul3 yet? http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html14:27
* anteaya can't find one14:27
*** esker has quit IRC14:27
anteayajeblair: ^14:27
*** zeih has joined #openstack-infra14:27
*** sdake has quit IRC14:28
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Make gate-tempest-dsvm-ironic-pxe_ssh vote again  https://review.openstack.org/21971214:28
jrollsdague: curious to hear your thoughts on this ^ :)14:29
jrolllucasagomes: fyi ^14:29
mordredSpamapS, Shrews: I believe instead we should have cleanup_ips get a list of floating ips for the server in question, and then delete those - not just try to delete the floating ip associated with the external ip of the vm14:29
*** fhubik has quit IRC14:29
*** liam__ has joined #openstack-infra14:29
lucasagomesjroll, cheers!14:29
*** jsavak has joined #openstack-infra14:30
*** mtanino has joined #openstack-infra14:30
*** kiran-r has quit IRC14:30
*** afazekas__ has quit IRC14:31
*** sdake has joined #openstack-infra14:32
anteayaperhaps we don't have an inital patch for zuul3 yet so perhaps it hasn't come up14:32
anteayais there a zuul3 etherpad?14:32
clarkbanteaya: I think there is just the spec which should list a topic14:32
anteayaclarkb: I don't see a topic in the spec14:32
* anteaya looks again14:32
dhellmannanteaya, clarkb : if you have a minute, could you help fix up the adoh permissions  so we can do a release? https://review.openstack.org/#/c/218254/14:33
*** jsavak has quit IRC14:34
*** mpopow__ has joined #openstack-infra14:34
anteayadhellmann: it is alarming14:34
anteayaha ha ha14:34
* anteaya shows herself the door14:34
clarkbdhellmann: does the global all-projects acl list allow release managers to push tags?14:34
* clarkb looks it up14:35
dhellmannclarkb: that's my understanding14:35
*** rainya has quit IRC14:35
dhellmannclarkb: I feel like I looked at a view in gerrit that showed that, but couldn't actually find the configuration file14:35
*** mase_x200 has quit IRC14:35
clarkbya I see it, refs/* has a rule for releaes managers to push signed tags14:35
* dhellmann wonders what was in anteaya's tea this morning14:35
anteaya:)14:36
anteayamontreal14:36
*** zeih has quit IRC14:36
dhellmannclarkb: cool, thanks for double checking14:36
*** eharney has quit IRC14:36
dhellmannanteaya: that must be a large tea mug14:36
anteayaha ha ha14:36
*** liam__ has quit IRC14:36
*** masayukig has quit IRC14:36
*** kushal has joined #openstack-infra14:37
anteayathanks clarkb14:37
openstackgerritLucas Alvares Gomes proposed openstack/diskimage-builder: Deprecate the deploy-ironic element  https://review.openstack.org/20907914:37
anteayaaway it goes14:37
clarkbfungi: jenkins04 still has a healthy thread count, I am going to check the others too but so far everything is good14:38
*** mpopow__ has quit IRC14:38
*** e0ne has joined #openstack-infra14:39
*** esker has joined #openstack-infra14:39
*** glauco has quit IRC14:41
*** dkranz has quit IRC14:41
clarkbfungi: 01-07 all look good14:41
*** jsavak has joined #openstack-infra14:42
fungiawesome. i have the replacement etherpad-dev building now (finally) and will swap dns over once puppet runs clean on it14:42
*** puranamr has quit IRC14:42
*** alexpilotti has joined #openstack-infra14:42
*** markvoelker has joined #openstack-infra14:42
*** markvoelker has quit IRC14:42
clarkbwoot, I never left that comment about updating the sha1 for the prod server to what the dev server ends up installing and running with14:43
clarkbwill do that now14:43
*** markvoelker has joined #openstack-infra14:43
*** masayukig has joined #openstack-infra14:43
*** puranamr has joined #openstack-infra14:44
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete floating ip by ID instead of name  https://review.openstack.org/21972514:44
fungii was going to anyway, just hadn't gotten around to updating the change14:44
*** wshao has joined #openstack-infra14:44
*** wenlock_ has joined #openstack-infra14:44
clarkbnp I just want to make sure that the comment is recorded so we don't end up forgetting and I left it just now14:45
*** spzala has joined #openstack-infra14:45
*** ildikov has quit IRC14:46
openstackgerritMerged openstack-infra/project-config: Fix Aodh permissions to let the release team do its job  https://review.openstack.org/21825414:46
*** sputnik13 has joined #openstack-infra14:47
*** hdd has joined #openstack-infra14:48
*** hdd has quit IRC14:50
*** hdd has joined #openstack-infra14:50
*** vinay_yadhav has quit IRC14:51
*** glauco has joined #openstack-infra14:52
*** asselin__ has quit IRC14:52
*** yamahata has joined #openstack-infra14:52
*** glauco has quit IRC14:53
*** esker has quit IRC14:53
*** dkranz has joined #openstack-infra14:54
*** k4n0 has quit IRC14:54
*** mriedem_meeting is now known as mriedem14:54
*** glauco has joined #openstack-infra14:55
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout: make non-voting puppet-neutron beaker on trusty  https://review.openstack.org/21955614:55
*** sabeen has joined #openstack-infra14:57
*** maishsk has quit IRC14:59
*** sabeen2 has joined #openstack-infra14:59
*** sridhar_ram has quit IRC14:59
*** krotscheck is now known as kro_afk14:59
*** glauco has quit IRC15:00
*** vivekd has quit IRC15:00
*** MarkAtwood has joined #openstack-infra15:00
*** tiswanso has quit IRC15:00
*** mwagner_lap has quit IRC15:00
*** tiswanso has joined #openstack-infra15:01
*** vinayyadhav has joined #openstack-infra15:01
*** sabeen has quit IRC15:01
*** julim_ has joined #openstack-infra15:02
anteayaso this patch failed twice on the same issue that hit zm02 yesterday, today it is zm08 same failure both times: https://review.openstack.org/#/c/219556/15:02
*** bhunter71 has joined #openstack-infra15:02
dhellmannclarkb, anteaya : the adoh acl changes merged; is there a delay before those take effect in gerrit?15:02
*** yamahata has quit IRC15:02
*** yamahata has joined #openstack-infra15:03
clarkbdhellmann: yes have to wait for the next puppet run to kick off the gerrit config update15:03
*** dizquierdo has quit IRC15:04
anteayadhellmann: https://review.openstack.org/#/admin/projects/openstack/aodh,access15:04
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source  https://review.openstack.org/21935115:04
*** bcourt has quit IRC15:04
*** kro_afk is now known as krotscheck15:04
*** unicell1 has joined #openstack-infra15:04
dhellmannclarkb, anteaya : ok, I'll check back in later today. Thanks for the assist!15:04
*** julim has quit IRC15:05
anteayadhellmann: sure15:05
*** unicell has quit IRC15:06
anteayaso should I just keep rechecking 219556 to get a different zm than 08?15:06
anteayathat feels wrong15:07
*** ildikov has joined #openstack-infra15:07
fungianteaya: see my comment on your story, that error is nothing to do with the zuul mergers after all and we suspect it has to do with splitting related git requests across different versions of git servers15:08
anteayaah15:08
* anteaya goes back to her story15:08
*** gildub has quit IRC15:08
fungianteaya: hopefully 219709 solves it while we finish the git server upgrades15:08
anteayawonderful thank you15:10
*** vinayyadhav has quit IRC15:10
anteayaso within about 40ish minutes or so, after it merges and a puppet run picks it up15:10
timrcfungi, clarkb, jeblair Looking for some guidance.  We hit a fairly weird edge case in zuul just now -- http://paste.openstack.org/show/441456/15:11
openstackgerritMerged openstack-infra/system-config: Use source balancing to transition to centos7  https://review.openstack.org/21970915:11
timrcfungi, clarkb, jeblair What happened is a user git review'ed two commits and made the child a draft.15:11
*** ddieterly is now known as deklan15:11
timrcThe check jobs ran on patchset-created for 82592,1 but fell over on 'recheck'15:12
jeblairtimrc: drafts are unworkable; we disabled them15:12
*** dims_ has quit IRC15:12
*** jistr is now known as jistr|call15:12
fungitimrc: yep, that's a known behavior (zuul can't enqueue if it can't read all changes in the chain, both child and parent)15:12
timrcSo why would tests start on upload but not on recheck?15:12
timrcThat seems inconsistent to me.15:13
fungitimrc: probably because when zuul saw the parent the first time, the child hadn't hit gerrit yet15:13
timrcFWIW he added zuul as reviewer to give it the necessary visibility to unblock him.  I'm happy to document that as what you need to do should you do something like that.15:13
fungifor a split second15:13
fungiit's one of the reasons we disabled drafts in our gerrit15:14
timrcfungi, Good to know.  Maybe something we should consider as well.15:14
*** dannywilson has joined #openstack-infra15:14
clarkbthey dont really provide any vlaue over a WIP state since they are very leaky and shouldn't be treated as a secure way to hide information15:14
*** bexelbie has quit IRC15:14
clarkband they break gerrit in all sorts of spectacular ways15:14
anteayafungi: I updated the story 2000341 to include a link to clarkb's patch and the git backend postulation15:15
clarkbone of my favorites is pusing a draft patchset after a non draft patchset, confuses reviewers when they cannot vote but are looking at what they see is the latest patchset15:15
timrcclarkb, Yeah.15:15
clarkbanother good one is two changes where there parent is made a draft15:15
clarkbnow the child can't merge and no one knows why15:15
clarkb(this is all independent of zuul too fwiw, gerrit's draft implementation is just confusing)15:16
*** dannywil_ has joined #openstack-infra15:16
*** dannywilson has quit IRC15:16
rcarrillocruzs/draft//15:16
fungithat too15:17
openstackgerritDoug Hellmann proposed openstack/requirements: use os-brick 0.4.0  https://review.openstack.org/21973615:18
openstackgerritDoug Hellmann proposed openstack/requirements: use python-novaclient 2.27.0  https://review.openstack.org/21973715:19
*** pc_m is now known as pc_m_AWAY15:19
clarkbfungi: let me know when new etherpad-dev is ready for some testing15:19
fungiclarkb: it's up and dns is changed, but ttl may still lag15:20
clarkbfungi: and when we rebuild the prod server don't forget to make it bigger15:20
openstackgerritDoug Hellmann proposed openstack/requirements: use python-heatclient 0.7.0  https://review.openstack.org/21973815:20
clarkbfungi: 104.130.252.212 looks like the new one ya?15:20
*** bcourt has joined #openstack-infra15:20
fungiclarkb: 2001:4800:7818:103:be76:4eff:fe04:9718 104.130.252.212 is the new one, yeah15:20
fungiright, yep15:21
jeblairclarkb: make what bigger?15:21
fungietherpad.o.o replacement15:21
jeblairwhy?15:21
clarkbjeblair: beacuse it ran out of memory during the ops summit15:21
*** Sukhdev_ has joined #openstack-infra15:21
clarkbjeblair: we couldn't even run puppet on it without invoking OOMkiller15:21
jeblairi thought that was because of runaway abiword15:21
clarkbjeblair: no, nodejs and puppet wre the biggest ocnsumers15:22
*** eharney has joined #openstack-infra15:22
clarkbabiword was eating cpu not memory15:22
jeblairah ok.  2 -> 4G then?15:22
jeblairor jumping to 8?15:22
openstackgerritDoug Hellmann proposed openstack/requirements: use python-ironicclient 0.8.0  https://review.openstack.org/21974015:22
fungiand bup was eating disk i/o, so it's possible the memory balloon was related to other performance degredation, but still 4g seems reasonable15:22
clarkbya I don't think we need to go huge, 4GB is probably a good step up15:23
jeblairhttp://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=117&rra_id=4&view_type=&graph_start=1408154147&graph_end=144120733115:23
kfox1111Any more reviewers for: https://review.openstack.org/#/c/218898/ ?15:23
jeblairthat's an interesting graph15:23
jeblairclarkb: i don't think the memory usage was related to etherpad usage15:24
jeblairit looks more like a slow leak over time15:24
jeblairhttp://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=117&rra_id=4&view_type=&graph_start=1435922638&graph_end=1441121005&graph_height=120&graph_width=500&title_font_size=1015:24
mordredthat just happened to hit its peak at the wrong time15:24
jeblairyeah15:24
jeblairin the second link, you can see that it was oom during the ops summit, but after that first restart, it didn't jump back up15:25
fungis/its peak/capacity/15:25
*** peristeri has quit IRC15:25
*** jswarren_ is now known as jswarren15:25
clarkbhttps://etherpad-dev.openstack.org/p/woo-trusty-woo15:25
fungiit still seems to be able to effectively fill all available memory with cache, so it might see some performance benefit from a ram increase regardless15:26
openstackgerritArun Kant proposed openstack/requirements: Updating pycadf to use recent taxonomy changes  https://review.openstack.org/21974215:26
jeblairanyway, i don't think we _need_ an increase, but it won't hurt.  it will mean we only have to restart it every 2 months instead of every month.  :)15:26
clarkblooks like it started back in april ish which is when we upgraded to more modern etherpad15:27
clarkbto address security concerns15:27
fungimoar security==moar leaky, apparently15:27
anteayajust moar15:27
mordredjeblair: so - reading some articles on node.js heap profiling and management15:27
mordreda case could be made that with a leaky app more memory will actually make it worse15:28
mordredbecause of how the v8 GC system works15:28
anteayaawesome15:28
*** mwagner_lap has joined #openstack-infra15:28
fungipuppetmaster hasn't spotted the new dns entry for etherpad-dev yet, so i've not updated its cached host key but i'll do that after lunch15:28
fungigotta scoot, back in a little while15:29
mordredas it fills memory the GC gets more aggressive, which means it runs more and takes more CPU - but also if there is more traverse while GCing, then the cost of the rampant GC cycles is going to be higher15:29
jeblairmordred: ugh15:29
clarkbmordred: when you say fills memory the GC gets more aggressive is that absolute "fill" or relative to available memory?15:29
*** wshao has quit IRC15:30
jeblairso let's strike "won't hurt" from my summary.  :)15:30
mordredjeblair: things I've read so far: http://stackoverflow.com/questions/16586705/configuring-v8s-memory-management-to-be-smart-for-a-node-js-process https://strongloop.com/strongblog/node-js-performance-garbage-collection/ https://strongloop.com/strongblog/node-js-performance-heap-profiling-tip/15:30
*** wshao has joined #openstack-infra15:30
mordredclarkb: the one that talked about aggressive GC did not indicate which15:30
jeblairclarkb: in our case don't those numbers approach the same value?15:31
*** puranamr has quit IRC15:31
mordredthe strongloop articles are annoying becuase they're trying to sell their enterprise node stuff15:31
mordredbut there's still good internals info in there15:31
clarkbjeblair: they do but adding 2GB won't be an immediate issue if its relative15:31
clarkbmordred: the one article says it sa 1.7GB heap cap anyways15:32
clarkbmordred: so adding memory won't hurt unless I misunderstand something15:32
jeblairi'm going to putz with websockets on dev15:33
clarkbmordred: but may give puppet breathing room15:33
*** puranamr has joined #openstack-infra15:34
jeblairReferenceError: io is not defined in https://etherpad-dev.openstack.org/javascripts/lib/ep_etherpad-lite/static/js/pad.js?callback=require.define at line 232'15:34
jeblairneat, eplite is broken with websockets?15:34
mordredawesome15:34
mordredhttps://hacks.mozilla.org/2012/11/tracking-down-memory-leaks-in-node-js-a-node-js-holiday-season/ <-- node-memwatch might be worth playing with15:35
j^2I’m starting to build tho OpenStack Operators launchpad, i accidently gave OpenStack Administrators Maintainer rights and removed my own for the set up, can someone help me out here?15:35
clarkbmordred: so ya if I read this correctly we are capped at 1GB by default or 1.7GB if we fiddle with settings, in either case bumping the system memory shouldn't affect node but will give puppet memory to run15:35
*** markus_z has quit IRC15:35
jeblairclarkb: that sounds good to me15:35
mordredclarkb: ++15:36
*** jsavak has quit IRC15:36
*** MarkAtwood has quit IRC15:36
*** geoffarnold has joined #openstack-infra15:36
jeblairclarkb: are we running latest sha of eplite?15:36
clarkbmordred: I think any memory management for node.js ahs to be independent of the host memory space, so good thing to investigate further but I dont expect bumping host size to factor in much other than allowing more processes to be happy15:36
clarkbjeblair: on dev we are15:36
clarkbjeblair: thats their presribed deployment method15:36
*** ayoung_ZZzz is now known as ayoung15:36
*** tonytan4ever has joined #openstack-infra15:37
*** geoffarnold has quit IRC15:38
*** jistr|call is now known as jistr15:38
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101115:39
*** geoffarnold has joined #openstack-infra15:39
clarkbjeblair: that error looks like its just missing a dependency? maybe we need to add something ot the npm install list?15:39
*** glauco has joined #openstack-infra15:39
clarkbor maybe it failed to get the files over websockets and websockets in general dont work15:39
*** armax has joined #openstack-infra15:40
jeblairclarkb: both of those seem plausible15:40
jeblairrcarrillocruz: did you run into this?15:41
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete floating ip by ID instead of name  https://review.openstack.org/21972515:42
*** sshnaidm has quit IRC15:42
clarkbmordred: out of curiousity why not continue to delete by ip?15:42
clarkbmordred: since you discovered the ip was probably unrelated15:42
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-zuul: Add acceptance tests for puppet-zuul  https://review.openstack.org/21773215:43
mordredclarkb: well, I could - but everyone else seemed to work with ids - and with that new code I _have_ the ids, so why not?15:43
*** Ala has quit IRC15:43
mordredclarkb: I don't have as strong a feeling about that as I do about not assuming that the public IP is a floating IP15:44
clarkbmordred: oh I work with IPs15:44
clarkbbecause in theory they are unique (because globally routable) and its an extra lookup to find the id15:44
mordredyah. I tend to work with IPs too15:44
mordredbut in this particular loop, I have the fip object15:45
*** wshao has quit IRC15:45
*** cdent has quit IRC15:45
jeblairclarkb: i think it's the second thing (failed to get file)15:45
mordred(and it's a cleanup, not a test of functionality - I think if we want to test that we can delete a floating ip by address, there should be a test that creates a floating ip and then deletes it)15:46
*** dprince has quit IRC15:47
clarkbjeblair: I agree if you dig under *_modules far enough you find that there is a socket.io present15:47
*** hodos has joined #openstack-infra15:47
clarkbjeblair: which I believe is the require happening in the error you pasted15:48
clarkbmordred: +115:48
*** jsavak has joined #openstack-infra15:48
*** mpopow__ has joined #openstack-infra15:50
jeblairthe proxy config was wrong15:51
jeblairrcarrillocruz: ^15:51
*** amitgandhinz has quit IRC15:51
clarkbmordred: are the two fixed_ip and fixed_ip_address keys from the neutron/nova apis? that seems funny but I know better than to blame shade for it :)15:51
jeblairdisabling puppet on epdev15:51
*** dims has joined #openstack-infra15:51
*** doude has quit IRC15:52
jeblair"101 Switching Protocols" :)15:52
*** kiran-r has joined #openstack-infra15:53
clarkbtake one down pass it around15:53
*** bharathm has joined #openstack-infra15:54
*** mpopow__ has quit IRC15:54
*** regXboi has quit IRC15:55
*** amuller is now known as amuller_afk15:55
*** yfried has quit IRC15:55
*** esker has joined #openstack-infra15:57
*** mtanino has quit IRC15:57
*** gyee has joined #openstack-infra15:57
*** ivar-lazzaro has joined #openstack-infra15:57
SpamapSmordred: OpenStack doesn't automatically deallocate floating ips when you delete a server? or am I missing something?15:58
*** bharathm has quit IRC15:59
*** esker has quit IRC15:59
clarkbSpamapS it doesnt16:00
jeblairSpamapS: when you say deallocate, do you mean "remove from server" or do you mean "remove from user's account"?16:00
*** jsavak has quit IRC16:00
clarkbit disassociates but does not delete16:00
*** jsavak has joined #openstack-infra16:00
openstackgerritGlauco Oliveira proposed openstack-infra/puppet-zuul: Add acceptance tests for puppet-zuul  https://review.openstack.org/21773216:00
rcarrillocruzjeblair: what's up16:00
jeblairSpamapS: the first one should happen, the second should not (so that floating ips can be re-used as permanent service addresses)16:01
rcarrillocruzmay i please get some eyes on https://review.openstack.org/#/c/219181/ and https://review.openstack.org/#/c/219185/16:01
jeblairrcarrillocruz: we're trying websockets on etherpad-dev and the apache config for them didn't work; i had to modify it.  i was wondering if you are using websockets with etherpad-lite?16:01
rcarrillocruzdependant changes16:02
rcarrillocruzwhat error you have?16:02
*** unicell1 has quit IRC16:02
*** dims has quit IRC16:02
*** kiran-r has quit IRC16:02
*** dims has joined #openstack-infra16:03
rcarrillocruzi still may have the vagrant vm i spinned for the etherpad websockets thing16:03
*** markvoelker has quit IRC16:03
jeblairrcarrillocruz: it was a 404 on sockte.io.js; i had to change "ProxyPass        /socket.io http://localhost:9001" to "ProxyPass        /socket.io http://localhost:9001/socket.io"16:03
jeblair                               socket.io.js16:04
*** cburroughs has quit IRC16:04
*** jlanoux has quit IRC16:04
*** dims_ has joined #openstack-infra16:04
rcarrillocruzthat's on trusty, right?16:05
openstackgerritJJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s)  https://review.openstack.org/21976016:05
rcarrillocruz-dev i mean16:05
jeblairrcarrillocruz: yes16:05
rcarrillocruzk, cos i still have the two vagrant vms i tested this16:05
rcarrillocruzprecise and trusty16:05
rcarrillocruzi'll check it out and let you know16:05
*** bknudson has joined #openstack-infra16:06
jeblairrcarrillocruz: also, the manifest does not install mod_proxy_wstunnel by default, i had to manually do that16:06
*** dims has quit IRC16:07
*** spzala has quit IRC16:07
rcarrillocruzthat's a bug, i'll push up something16:07
*** mtanino has joined #openstack-infra16:08
*** andersonvom has joined #openstack-infra16:10
*** stevemar has joined #openstack-infra16:11
*** bharathm has joined #openstack-infra16:12
mordredclarkb: yes - it's called one thing in nova and one in neutron16:13
mordredclarkb: clearly that particularly chunk of cleanup logic code should probably become a shade API at some point16:13
mordred"get_floating_ips_for_server" or somethin16:13
mordredSpamapS: the number of things that don't happen that should boggle the mind16:14
clarkbmordred: thanks16:14
*** esker has joined #openstack-infra16:14
*** angela-s has joined #openstack-infra16:14
jeblairhrm, firebug has no visibility into websocket traffic :(16:14
mordredSpamapS, jeblair: I'd argue that the floating-ip interface should be similar to the cinder volume one - you can create a volume on server boot and mark it as "delete when server goes away"16:14
jeblairmordred: i would not dispute that (as an option)16:15
mordredwhich is what we'd do with nodepool vms - since we know that we do not, in fact, want the ips to live16:15
jeblairya16:15
mordredbut then, for things where we DO want the floating-ip to be an object we manage16:15
mordredwe do not want openstack to delete them for us16:15
mordredfor those reasons16:15
jeblairapparently chromium devtools can show them, but first i need to figure out how to get chromium to permanently accept the ssl cert16:16
*** afazekas__ has joined #openstack-infra16:16
*** devlaps has joined #openstack-infra16:16
clarkbjeblair: you have to accept it system wide because chromium16:16
clarkbor in your user settings for whatever desktop system you use16:16
*** john-davidge has quit IRC16:17
*** julim_ has quit IRC16:17
*** alexpilotti has quit IRC16:17
*** deklan has quit IRC16:20
*** deklan has joined #openstack-infra16:21
clarkbwe are using the source balance method now, if anyone sees the layout job fail due to git clones please let us know here16:21
zaromorning16:22
clarkbjeblair: mordred now that we are balancing based on source https://review.openstack.org/#/c/219427/ should be safe to go in16:22
clarkbzaro: morning16:22
mordredclarkb: k. +316:22
*** esker has quit IRC16:23
*** jtomasek has quit IRC16:23
*** jsavak has quit IRC16:23
*** FallenPegasus has joined #openstack-infra16:24
*** jsavak has joined #openstack-infra16:24
*** burgerk_ has joined #openstack-infra16:24
*** abregman has quit IRC16:25
*** spzala has joined #openstack-infra16:26
sdaguejroll: is it working ?16:27
*** andersonvom has quit IRC16:28
zaroclarkb: yo! getting closer to finding the root of the jgit issue.  looks like the problem occurs when jgit attmepts removePack when there are pack errors https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java#L56216:28
nibalizermordred: i agree about fips16:28
*** Piet has joined #openstack-infra16:28
sdaguejroll: so the prefered direction is the ipa job right? I'd say change all the non ironic / dib bits over to ipa as well16:28
nibalizercleaning up floating ips forom servers that were deleted is annoying16:28
nibalizerplus I always forget to do it16:28
*** doude_ has joined #openstack-infra16:28
zaroclarkb: that causes the repo to be inaccessable.16:28
clarkbzaro: that sounds right, basically there is an error due to a diff timeout and that error is interpreted as a corrupt pack file16:29
sdaguemordred: do you have logs for where this nova thing went wrong?16:29
jeblairclarkb: export and then: certutil -d sql:$HOME/.pki/nssdb -A -t P -n <certificate nickname> -i <certificate filename>16:29
jeblairclarkb: i love google user interfaces!16:29
SpamapSjeblair: I meant deallocate from the server, not delete from the user. :)16:29
clarkbnibalizer: the trick is to delete it first but you do have to remember16:29
mordredsdague: yes - although I'm now not sure it was a server error16:30
akuznetsovahi guys!16:30
sdagueif those logs are up somewhere, I can look16:30
*** sergmelikyan has joined #openstack-infra16:30
*** sergmelikyan has quit IRC16:30
zaroclarkb: so yeah, i think the error should not be handled as a corrupted pack.  i'm thinking gerrit should do something else in that case, or maybe nothing at all?16:30
clarkbzaro probably just log the timeout and send error to http cliemt?16:31
mordredsdague: http://logs.openstack.org/55/219655/1/check/gate-shade-dsvm-functional-nova/1154770/console.html#_2015-09-02_12_10_56_11316:31
clarkbbut definitely dont mark thr packfile as corrupt16:32
nibalizerclarkb: this is hwat I do these days https://github.com/nibalizer/dotfiles/blob/master/.bashrc#L28016:32
nibalizerannoying and dumb but functional16:32
zaroclarkb: that makes sense.16:32
*** sergmelikyan has joined #openstack-infra16:32
*** mpopow__ has joined #openstack-infra16:32
akuznetsovaam I correct that if some project is specified in devstack-vm-gate-wrap.sh like PROJECTS="openstack/project $PROJECTS" then I need to add some fixes in project congig project.yaml and export           export BRANCH_OVERRIDE={branch-override}if [ "$BRANCH_OVERRIDE" != "default" ] ; then16:32
akuznetsova              export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE16:32
akuznetsova          fi ?16:32
*** hashar_ has joined #openstack-infra16:33
akuznetsovaoops, sorry for ugly formatting16:33
*** sergmelikyan has quit IRC16:34
*** afazekas__ has quit IRC16:35
*** hashar has quit IRC16:35
jeblairclarkb: http://i.imgur.com/QhOJ5JV.png16:35
jeblairwebsocket frames!16:35
clarkbneat16:37
jeblairthere is a heartbeat in there -- client sends "2" server responds "3".  so at least client side, it can still do disconnection detection16:37
jeblairhopefully there's a sever-side timeout so we don't end up with lots of stuck-open connections when conference wifi reboots16:38
*** earlephilhower has joined #openstack-infra16:38
sdaguemordred: yeh, so the REST resource should have always been by id16:38
sdagueI'm curious when that worked16:39
openstackgerritMerged openstack-infra/system-config: Swap out git07 to new centos7 node  https://review.openstack.org/21942716:39
mordredsdague: it's pretty much worked constantly until just now16:39
nibalizerjeblair: thats pretty slick16:39
sdaguehttps://github.com/openstack/nova/blob/7a1ef0b53874d3a596583aa5ad451ae3c72637f2/nova/network/api.py#L128 - that line dates back to feb16:40
*** julim has joined #openstack-infra16:40
*** dprince has joined #openstack-infra16:40
sdaguebut it dates back further than that16:40
*** julim has quit IRC16:40
sdaguemordred: nova-net or neutron backend?16:41
*** julim has joined #openstack-infra16:41
*** hashar_ has quit IRC16:41
mordredsdague: nova-net - all of our nova-net functional tests started failing this morning with that error16:41
mordredsdague: they'd al been passing up until then16:41
*** cdent has joined #openstack-infra16:42
*** FallenPegasus has quit IRC16:42
*** ashleighfarnham has joined #openstack-infra16:42
*** ashleighfarnham has quit IRC16:42
*** sergmelikyan has joined #openstack-infra16:42
sdaguehmm16:43
jeblairmordred: wait, did shade just catch a nova api break?16:43
*** marzif has quit IRC16:43
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Ignore noop jobs when combining ChangeQueues  https://review.openstack.org/21951416:43
*** marzif has joined #openstack-infra16:43
*** jistr has quit IRC16:44
*** amuller_afk is now known as amuller16:44
*** angdraug has joined #openstack-infra16:44
mordredjeblair: it's possible. although the docs do not indicate that it should have worked before16:45
*** larainema has quit IRC16:45
*** sergmelikyan has quit IRC16:45
jeblairmordred: which may explain the lack of a test for that.16:45
jeblairso, more like it caught an undefined api break :)16:46
*** larainema has joined #openstack-infra16:46
*** sergmelikyan has joined #openstack-infra16:46
*** unicell has joined #openstack-infra16:47
*** ifarkas has quit IRC16:47
*** pc_m_AWAY is now known as pc_m16:48
*** esmute_ has joined #openstack-infra16:48
sdagueyeh, it definitely wasn't meant to work the way shade was using it16:48
jeblairclarkb: do we need to re-tune apache to increase the number of simultaneous connections it can support, since websockets will maintain long-running connections?16:48
*** afazekas__ has joined #openstack-infra16:49
sdaguehowever, I'm still really curious to figure out what changed16:49
sdaguebecause, from all the code that I'm looking at, this should have thrown an error for at least a year16:49
clarkbjeblair possibly, and also file descriptor limits16:49
sdaguemordred: you happen to have the first test run where this failed to narrow it down?16:50
*** jtomasek has joined #openstack-infra16:50
*** MIDENN_ has quit IRC16:50
*** mdenny has quit IRC16:50
*** lucasagomes is now known as lucas-dinner16:51
*** MIDENN_ has joined #openstack-infra16:52
*** mdenny has joined #openstack-infra16:52
sdagueoh, so I actually know why this failed16:52
mordredoh! yay!16:52
sdagueor I have a good guess16:52
sdagueso can you file a bug on this16:52
mordred(I mean, even if it was a behavior change, it was a buggy cleanup and the new version I like better anyway)16:53
mordredsdague: sure!16:53
mordredsdague: on nova?16:53
sdagueyeh16:53
*** mpopow__ has quit IRC16:53
sdaguewe should evaluate16:53
greghaynesSpamapS: I've added a property to some of our inventory/mac.yaml files of reserved: true for the nodes that gh2bf should skip over16:53
sdaguethis started failing about 12 hours ago?16:53
greghaynesSpamapS: for crinkle's nodes and for the new bastion im spinning up16:53
mordredsdague: yeah - give of take16:53
mordredsdague: my tests yesterday were passing nova but failing neutorn - but that was due to a bug in my code16:54
sdaguehttps://review.openstack.org/#/c/214085/ - we changed the default paste pipelines. The new one was supposed to work correctly in this scenario, however we apparently missed a thing16:54
jeblairclarkb: i think our existing tuning is probably a sufficient starting place; it looks like you sized it assuming everyone's making GET requests at the same time anyway16:54
mordredthen, today, I fixed my bug and started hitting the nova things16:54
*** jerryz has joined #openstack-infra16:54
mordredsdague: ah - that would explain it16:54
sdagueyeh, so, I don't think the old code actually worked16:54
sdagueI think it just failed silently16:54
mordredAH. LOVELY16:55
mordredso, really, fixing it in our code is more better16:55
sdagueyeh, but we should evaluate the fact that those old requests didn't explode before, and if we care in that case.16:55
sdagueso, you definitely want to fix shade to use ids, because that's actually the API16:56
*** earlephilhower has quit IRC16:56
clarkbjeblair 2k iirc16:56
jeblairclarkb: ypp16:56
*** mcdonagm has quit IRC16:56
*** mcdonagm__ has quit IRC16:56
*** shashank_hegde has joined #openstack-infra16:56
sdagueand I'll take the bug into the nova team and evaluate if we want to care about this particular break16:56
*** tonytan4ever has quit IRC16:57
greghaynesSpamapS: I am also assigning the 15.x ip's to the nodes starting at 15.184.52.100, we have less than 100bm nodes so that should make us within the bm /2416:57
mordredsdague: https://bugs.launchpad.net/nova/+bug/149151216:57
openstackLaunchpad bug 1491512 in OpenStack Compute (nova) "Behavior change with latest nova paste config" [Undecided,New]16:57
*** salv-orlando has quit IRC16:57
sdaguemordred: thanks16:57
*** puranamr has quit IRC16:57
*** derekh has quit IRC16:57
mordredsdague: thanks for looking at it!16:57
*** earlephilhower has joined #openstack-infra16:58
mordredwoot! the new test worked16:58
*** julim has quit IRC16:58
mordredjeblair, SpamapS: https://review.openstack.org/#/c/219725/ +3 me?16:58
Shrewsmordred: got it. thx sdague16:59
*** puranamr has joined #openstack-infra16:59
sdagueno prob, thanks for raising the issue16:59
*** sharikapongubala has quit IRC17:00
sdaguemordred: do you have logs of runs that are a day old, so I can see what this looked like in the service logs back then17:00
*** SotK has quit IRC17:01
*** deklan has quit IRC17:01
*** julim has joined #openstack-infra17:01
*** annegentle has quit IRC17:01
*** Zara has quit IRC17:01
jeblairclarkb, fungi: the thing with the proxypass retry i was worried about should not be a problem -- because the proxypass line is now no longer exactly the same as the the mod_rewrite proxy line, the retry=0 should take effect17:02
openstackgerritMonty Taylor proposed openstack-infra/shade: Migrate neutron to the common client interface  https://review.openstack.org/21424317:02
openstackgerritMonty Taylor proposed openstack-infra/shade: Move keystone to common identity client interface  https://review.openstack.org/21424417:02
openstackgerritMonty Taylor proposed openstack-infra/shade: Pass timeout to session, not constructors  https://review.openstack.org/21929017:02
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove last vestige of glanceclient being different  https://review.openstack.org/21929117:02
mordredsdague: yup. one sec17:02
jeblairclarkb, fungi: apache is not complaining about it, and i have performed a simulation of the 500 error propogation that we saw during the ops summit and verified that it works as expected17:02
*** ashleighfarnham has joined #openstack-infra17:02
sdaguemordred: also, I would add a check at the end of - https://github.com/openstack-infra/shade/blob/ceb6a1d36ea197d6489d981c112ec187cf29a961/shade/tests/functional/test_floating_ip.py#L23217:03
jeblairclarkb, fungi: (i was also able to reproduce the 500 error propogation by removing the retry=0 line; so i'm fairly confident in my test)17:03
mordredsdague: this is about 2 weeks old: http://logs.openstack.org/44/214244/2/check/gate-shade-dsvm-functional-nova/2009d01/17:03
sdaguethat actually makes sure the floating ip is removed17:03
jeblairclarkb, fungi: i will write up my local changes now17:03
mordredsdague: yeah - I was thinking the same thing17:03
sdaguebecause, I'm betting it just never was17:03
openstackgerritJJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s)  https://review.openstack.org/21976017:03
*** woodster_ has joined #openstack-infra17:03
mordredsdague: here's from yesterday http://logs.openstack.org/43/214243/3/check/gate-shade-dsvm-functional-nova/bd70567/17:04
sdaguemordred: cool17:04
*** cburroughs has joined #openstack-infra17:04
*** rossella_s has quit IRC17:05
*** Sukhdev_ has quit IRC17:06
*** rossella_s has joined #openstack-infra17:06
sdaguemordred: are you ignore 404?17:06
sdaguehttp://logs.openstack.org/43/214243/3/check/gate-shade-dsvm-functional-nova/bd70567/logs/screen-n-api.txt.gz#_2015-09-01_19_18_35_37917:06
*** arxcruz has quit IRC17:07
pc_mCan infra cores please review updates to https://review.openstack.org/#/c/209887/  for me? Neutron team decided to have two jobs, one for core neutron-client tests and one for advanced services based tests (which will enable the plugins in hook script).17:07
sdaguemordred: yeh, you were being handed a 404 100% of the time before17:07
sdaguehttp://logs.openstack.org/43/214243/3/check/gate-shade-dsvm-functional-nova/bd70567/logs/screen-n-api.txt.gz#_2015-09-01_19_18_47_06617:07
*** sridhar_ram has joined #openstack-infra17:08
sdagueI guess now you get a 40017:08
sdaguebecause we validate first17:08
sdaguebut in that run from yesterday, it did not return success17:08
openstackgerritJames E. Blair proposed openstack-infra/puppet-etherpad_lite: Fix websocket configuration  https://review.openstack.org/21979017:08
clarkbits nova client receiving it so possible client bug17:09
*** skylerberg has joined #openstack-infra17:09
jeblairclarkb, fungi, rcarrillocruz: ^ that's on-system on etherpad-dev and working17:09
mordredsdague: sweet!17:09
sdaguesame 2 weeks ago http://logs.openstack.org/44/214244/2/check/gate-shade-dsvm-functional-nova/2009d01/logs/screen-n-api.txt.gz17:09
sdaguesearch for os-floating-ips/17217:09
sdagueand you'll see they were 404s17:09
sdagueso I'm surprised that test passed17:09
*** sridhar_ram has quit IRC17:09
jeblairthis is cool; etherpad is going to use websockets at the next summit! :)17:09
sdaguewell, at least that's the whole story, previously this made invalid API call, it was generating a 404 on the server, which isn't getting bubbled up in shade as an error. With new code we validate before letting it go hit the db, and just tell you it's not possible that this is a valid request via a 40017:11
jeblairsdague: because 404 in a rest api means "this object does not exist" which is the desired end state anyway.17:12
*** e0ne has quit IRC17:12
mordredright. so a 404 would have been pleasing17:13
sdagueright, except it's a 404 for the wrong reasons. The thing does not exist, because it's not possible for that thing to ever exist17:13
mordredright17:13
sdaguebecause you tried to delete the wrong resource17:13
jeblair(and of course it's perfectly natural to get a 404 on an object you thought existed a moment ago because it's an _asynchronous_ api)17:13
openstackgerritEmma Barber proposed openstack-infra/puppet-hound: Configure and run apache on top of hound  https://review.openstack.org/21164017:13
sdaguejeblair: right, except, the 404 happened before the delete17:14
*** afazekas__ has quit IRC17:14
sdagueso, unless you have a time machine :)17:14
jeblairsdague: yep.  i like the validation.  i think it's an improvement; helps clarify this situation17:14
mordredtotally17:14
mordredit's good stuff17:14
*** glauco has quit IRC17:14
mordredI MUCH prefer getting an error so I can decide if, in my workflow, an error on the thing is something I expect or something I do not expect17:15
*** jtomasek has quit IRC17:15
mordredsdague: so, just for the record, I'm fine if that bug gets closed EWONTFIX17:15
sdaguemordred: ok, cool17:16
sdagueanyway, at least we understand now. I just don't like things changing without understanding :)17:16
mordredI do not believe I was counting on the broken behavior anywhere real17:16
mordredand, even if I was, I would prefer to get the new errors and break for a second17:16
sdagueyeh17:16
*** sridhar_ram has joined #openstack-infra17:17
sdagueit may be a while before you get that new behavior in production clouds. They actually need to make a recommended paste change to get there17:17
*** wshao_ has joined #openstack-infra17:17
sdaguethough, if you start using the /v2.1 nova endpoint if available, you'll always get the new behavior17:17
openstackgerritJJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s)  https://review.openstack.org/21976017:18
*** trown is now known as trown|lunch17:19
*** marzif has quit IRC17:20
*** shashank_hegde has quit IRC17:20
*** pvaneck has joined #openstack-infra17:20
jeblairoh, i think i hear the radiation detecting helicopter.17:21
openstackgerritAnastasia Kuznetsova proposed openstack-infra/project-config: Remove export of 'openstack/rally' to  in jobs' configs  https://review.openstack.org/21979517:21
*** maishsk has joined #openstack-infra17:23
*** cdent has quit IRC17:24
jrollsdague: yes, pxe_ssh is working now, pxe_ipa is the future... need to track down the rest and change them over17:24
*** achanda has joined #openstack-infra17:24
openstackgerritAnastasia Kuznetsova proposed openstack-infra/project-config: Remove export of 'openstack/rally' to PROJECTS list in jobs' configs  https://review.openstack.org/21979517:26
*** wshao_ has quit IRC17:26
*** glauco has joined #openstack-infra17:27
dhellmannI'm seeing a lot of requirements job failures with "ERROR: branch not allowed by features matrix: liberty", does anyone have any idea what that means? I think we turned on some new jobs for the requirements repo today?17:28
dhellmannlifeless, ttx: ^^17:28
*** sharikapongubala has joined #openstack-infra17:28
dhellmannclarkb: ^^17:29
clarkbdhellmann: gah yes I know what that means17:30
clarkbdhellmann: patch inconing17:30
dhellmannclarkb: thanks17:30
clarkbwe should write this patch as soon as the release name is solid17:30
clarkbhrm looks like we did have it but it was incomplete17:31
openstackgerritMonty Taylor proposed openstack-infra/shade: Start using keystoneauth for keystone sessions  https://review.openstack.org/18317617:31
mordredmorgan: ^^ I think that'll pass this time17:31
morganLooking. We did land some changes yesterday re: loading.17:32
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Make liberty actually allowed in features.yaml  https://review.openstack.org/21979617:32
sdaguedhellmann: things started branching today?17:32
clarkbdhellmann: ^ you can depends on that to fix or wiat for stuff to merge17:32
morganmordred: i think we are all but done for 1.0 though. I think the changes shouldnt impact you since you use auth_type17:32
clarkbsdague: no, we added cross branch tests to requirements to prevent master requirements from breaking stable/liberty on other things17:32
*** sharikapongubala has quit IRC17:32
clarkbsdague: and this was a missed piece of d-g config17:32
dhellmannsdague: no, no branches yet17:33
*** dims_ has quit IRC17:33
dhellmannsdague: the new jobs are supposed to ensure we test changes to master requirements against any new stable/liberty branches when they are created17:33
sdagueclarkb: ah gotcha17:33
sdagueanyway +2 on the d-g patch17:33
dhellmannclarkb, sdague : why do we have a hard-coded list of branch names in there?17:33
dhellmannclarkb: want to go ahead and add mitaka now?17:34
jeblairclarkb, sdague: why does q-vpn get added?17:34
mordredmorgan: yeah. I agree17:35
openstackgerritDoug Hellmann proposed openstack/requirements: use python-novaclient 2.27.0  https://review.openstack.org/21973717:35
openstackgerritDoug Hellmann proposed openstack/requirements: use os-brick 0.4.0  https://review.openstack.org/21973617:35
*** markvoelker has joined #openstack-infra17:35
openstackgerritDoug Hellmann proposed openstack/requirements: use python-heatclient 0.7.0  https://review.openstack.org/21973817:35
*** markvoelker has quit IRC17:35
*** dims has joined #openstack-infra17:35
*** abdelwas has joined #openstack-infra17:36
clarkbjeblair: because it is part of kilo so maintaining forward compat lists17:36
openstackgerritDoug Hellmann proposed openstack/requirements: use python-ironicclient 0.8.0  https://review.openstack.org/21974017:36
clarkbdhellmann: because the testing changes depending on branch17:36
*** rlandy has quit IRC17:36
*** markvoelker has joined #openstack-infra17:36
dhellmannclarkb: and I guess we don't want branches of devstack-gate itself?17:36
dhellmannclarkb: anyway, thanks for fixing it!17:36
jeblairclarkb: i guess i don't understand why it's not based on master17:36
*** rlandy has joined #openstack-infra17:36
clarkbjeblair: oh hrm17:37
jeblairdhellmann: since it drives cross-branch testing, having branches itself would be exceedingly complicated17:37
armaxhi folks, it seems that change https://review.openstack.org/#/c/219477/ failed because of a switf error, is tehre any chance it can be promoted over all the other neutron changes in the gate? It addresses a race condition17:37
clarkbwait thats so weird17:37
clarkbso we default have vpn in kilo but not liberty or juni17:37
dhellmannjeblair: right, makes sense17:37
clarkb*juno17:37
clarkbjeblair: but I Think you are correct I will update17:37
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Make liberty actually allowed in features.yaml  https://review.openstack.org/21979617:38
sdaguejeblair: because it's not in juno17:38
clarkbsdague: but its not part of master either17:38
clarkbsdague: so right now kilo is the only branch doing vpn which is fine I guess, just odd17:38
sdagueclarkb: hmmmm ok17:39
*** puranamr has quit IRC17:39
*** wshao_ has joined #openstack-infra17:39
sdagueyeh, right, it's a plugin now so configured outside of devstack17:39
*** puranamr has joined #openstack-infra17:39
armaxI would be grateful, and so would be the gate :)17:40
*** angdraug has quit IRC17:40
sdagueclarkb: +A17:40
*** ashleighfarnham has quit IRC17:40
*** jsavak has quit IRC17:41
*** yamahata has quit IRC17:44
*** ashleighfarnham has joined #openstack-infra17:45
*** pelix has quit IRC17:45
*** shashank_hegde has joined #openstack-infra17:45
openstackgerritMerged openstack-infra/shade: Delete floating ip by ID instead of name  https://review.openstack.org/21972517:45
asselinare infra  nodepool images updating ok?17:46
asselinours are 5 days old17:46
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Don't run swift on ironic jobs  https://review.openstack.org/21980117:46
*** Swami_ has joined #openstack-infra17:48
*** Swami has joined #openstack-infra17:48
asselinours keep failing here: http://paste.openstack.org/show/441848/17:48
greghaynescrinkle: SpamapS Question about the new bm deploy host im spinning up - did we decide that we arent going to use puppet to install bifrost for now because of the need to install from source? (I think I remember that convo but not 100%)17:48
*** ybathia has joined #openstack-infra17:48
SpamapSgreghaynes: no we didn't decide that17:48
SpamapSgreghaynes: puppet pulls it fromg it, no?17:49
* greghaynes looks @ crinkle17:49
*** rockyg has joined #openstack-infra17:49
greghaynesI thought it was pretty well hard coded to only use packages17:49
*** sharikapongubala has joined #openstack-infra17:49
greghaynesbut crinkle im sure will school us ;)17:49
*** dkranz has quit IRC17:50
*** jsavak has joined #openstack-infra17:50
clarkbasselin: thats a known issue with dib/ubuntu repos iirc17:50
asselinclarkb, is there a workaround?17:51
clarkbasselin: the dib devs may be able to confirm but it was being worked last I saw17:51
mordredasselin: greghaynes ^^17:51
* greghaynes looks17:51
greghaynesasselin: the fix for that merged I think17:52
greghaynesasselin: it was up yesterdy17:52
asselingreghaynes, ok cool. do you have a link?17:52
crinklegreghaynes: puppet pulls it from source17:53
greghaynesasselin: looking17:53
fungiokay, back and reading the ~300 lines of lunchtime scrollback17:53
clarkbsdague: http://logs.openstack.org/77/219477/5/check/gate-grenade-dsvm-neutron/742745e/logs/grenade.sh.txt.gz#_2015-09-02_16_44_59_479 is the thing armax's change ran into above. I would be surprised if a neutron change affected that file being used as a block device17:53
crinklegreghaynes: http://git.openstack.org/cgit/openstack/puppet-ironic/tree/manifests/bifrost.pp#n193 is how it does it17:53
clarkbarmax: have you filed a bug for that at least?17:53
armaxclarkb: not yet...I will17:54
sdagueman, haven't seen that one in a while17:54
armaxclarkb: I have other fires to put out17:54
clarkbarmax: also the gate has a large series of things that appear to be passing currently so I will hold off on promoting that change until we get to a reset17:54
greghayneswhere did that dib patch go...17:54
armaxclarkb: ok sure17:54
sdagueclarkb: which change is that ?17:55
greghaynesasselin: oh, the patch was abandoned because apparently upstream (ubuntu) fixed it17:55
clarkbsdague: 21947717:55
greghaynesasselin: https://review.openstack.org/#/c/219199/17:55
greghaynescrinkle: awesome, do you have any time to help me get set up with that module?17:56
crinklegreghaynes: sure, we can do that today17:56
greghaynesasselin: If that isnt the case please let me know and we can resurrect that patch17:56
greghaynescrinkle: w00t17:56
sdagueclarkb: ok, gotcha17:56
*** armax has quit IRC17:57
*** dzamboni has joined #openstack-infra17:57
asselingreghaynes, so what do I need to update in my system? or just rerun the image build?17:58
*** HeOS has quit IRC17:58
greghaynesasselin: just rebuild17:58
asselingreghaynes, ok thanks17:58
greghaynesasselin: assuming you arent caching that file somewhere17:58
*** mpopow__ has joined #openstack-infra17:58
greghaynes(the sha256sums file)17:58
asselingreghaynes, I'm caching whater dib's caching17:59
sdagueclarkb: it is interesting that - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIndyb25nIGZzIHR5cGUsIGJhZCBvcHRpb24sIGJhZCBzdXBlcmJsb2NrIG9uIC9kZXYvXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDEyMTY3MzM3NjR917:59
greghaynesasselin: ok, dib should invalidate that fine17:59
asselincool17:59
sdaguesuperblock fail is only on neutron grenade jobs17:59
*** maishsk has quit IRC17:59
clarkbsdague: that is odd18:00
sdagueSep  2 16:44:59 devstack-trusty-hpcloud-b4-4720563 kernel: [ 2466.205514] XFS (loop2): Filesystem has duplicate UUID c30d9726-bab6-4c4e-9952-16577f49d514 - can't mount18:00
*** kmartin_ has joined #openstack-infra18:02
*** madong_ has joined #openstack-infra18:02
*** Swami__ has joined #openstack-infra18:02
*** kmartin has quit IRC18:03
*** mdenny has quit IRC18:03
*** MIDENN_ has quit IRC18:03
*** Swami_ has quit IRC18:03
*** Swami has quit IRC18:03
*** larainema has quit IRC18:03
*** mdenny has joined #openstack-infra18:03
*** MIDENN_ has joined #openstack-infra18:03
*** Swami has joined #openstack-infra18:03
*** sridhar_ram has quit IRC18:04
*** jsavak has quit IRC18:04
*** dkranz has joined #openstack-infra18:05
*** jsavak has joined #openstack-infra18:05
*** maishsk has joined #openstack-infra18:05
*** markvoelker has quit IRC18:05
sdagueclarkb: apparently it's just a thing that xfs does from time to time - https://bugs.launchpad.net/diskimage-builder/+bug/144370618:06
openstackLaunchpad bug 1443706 in diskimage-builder "Failed to mount: Filesystem has duplicate UUID" [Medium,Fix released] - Assigned to Steve Baker (steve-stevebaker)18:06
sdaguedisk image builder even put in a work around18:07
sdagueso once someone files a bug on the issue, I'll put the work around into grenade and devstack18:07
fungiweird. wonder if there's some birthday paradox problem with the uuid generation there18:08
anteayatoo much cake?18:08
fungilike maybe they're not doing a great job of picking an initial seed18:08
fungiseed cake!18:08
fungimmm18:08
clarkbpickle the initial seed cakes18:09
anteayapickled cakes?18:09
*** marzif has joined #openstack-infra18:09
anteayaI keep kicking pabelanger18:09
anteayaI have shoes and he has flipflops18:09
slaglewhen we saw that in dib it was b/c we were running an instance of the cloud image which was trying to build a dib image using that same cloud image18:10
*** Sukhdev_ has joined #openstack-infra18:11
anteayasdague: did someone say they were going to file a bug on that?18:12
anteayaarmax has left irc18:12
anteayaI mean I can if we want me to18:12
*** sthillma has joined #openstack-infra18:12
*** mpopow__ has quit IRC18:13
anteayaoh armax said he will18:13
anteayaafter he deals with other fires18:13
*** sdake_ has joined #openstack-infra18:13
*** rockyg has quit IRC18:16
*** sdake has quit IRC18:17
*** maishsk has quit IRC18:17
*** sthillma_ has joined #openstack-infra18:18
*** amitgandhinz has joined #openstack-infra18:18
*** sthillma has quit IRC18:18
*** sthillma_ is now known as sthillma18:18
*** annegentle has joined #openstack-infra18:20
clarkbI am going to enqueue 219477 into the gate now then promote it when the current head of the gate list is processed, we actually did merge a few thngs18:20
*** sdake has joined #openstack-infra18:21
anteayayay merged things18:21
*** maishsk has joined #openstack-infra18:22
fungithanks clarkb18:22
*** julim has quit IRC18:22
anteayaI thought I saw about 8 or so pass at the head of the gate18:22
*** marzif has quit IRC18:22
*** tonytan4ever has joined #openstack-infra18:22
*** marzif has joined #openstack-infra18:22
clarkbanteaya: yup and there wee a few before this batch18:22
anteayayay for things working18:22
clarkbthis is why I didn't immediately promote it wanted to give them a chance since they were all passing18:22
anteayagood call18:23
clarkbwhen thats done my next thing will be deleting old git07 and making a new git0618:24
clarkbgonna try to pick up the pace on that so we can go back to leastconn balancing18:25
*** sdake_ has quit IRC18:25
*** sridhar_ram has joined #openstack-infra18:25
*** ybathia has quit IRC18:26
*** sharikapongubala has quit IRC18:27
*** puranamr has quit IRC18:27
clarkbfungi: did you see that jeblair got websockets owrking on etherpad-dev?18:27
clarkbfungi: so we can enable them on prod server when we rebuild too :)18:27
fungiclarkb: yep, commented on the review18:27
mordredyay! the shade changes are merging again18:28
* fungi is excited18:28
*** puranamr has joined #openstack-infra18:28
clarkbjeblair: fungi has a good point about precise, I am not sure that mod_wstunnel exists on precise so it may just explode18:28
*** sharikapongubala has joined #openstack-infra18:28
*** marzif has quit IRC18:28
clarkbderp its already wrapped18:29
fungiahh, yep18:29
clarkbfungi: so that should be fine on precise18:29
clarkb+2 from me18:29
fungii missed it the first time through as well, forgot we branched on ubuntu release18:29
*** yfried has joined #openstack-infra18:29
clarkbya the nested block isn't super clear in the diff18:30
fungiand the vhost config is conditional on the module existing18:30
fungiso lgtm18:30
*** trown|lunch is now known as trown18:30
anteayaclarkb: how many more git servers to upgrade to centos7?18:30
docaedoyolanda: have you had a chance to look at the changes to https://review.openstack.org/#/c/218898/ yet?18:31
fungianteaya: 618:31
anteayafungi: thanks18:31
*** armax has joined #openstack-infra18:31
anteayadocaedo: I'm guessing she is probably offline until tomorrow18:31
fungiclarkb: any objection to me blowing away the old etherpad-dev now?18:31
openstackgerritJJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s)  https://review.openstack.org/21976018:31
docaedoanteaya: ah thank you18:32
clarkbfungi nope18:32
fungiclarkb: you may need the extra bit of headroom to build replacement git servers since we're close to quota18:32
clarkbanteaya 618:32
anteayadocaedo: welcome, she is in Europe18:32
anteayaclarkb: thanks18:32
clarkbfungi +118:32
*** kdas_ has joined #openstack-infra18:32
*** sthillma_ has joined #openstack-infra18:33
docaedoanteaya: perhaps you would like to take a look at https://review.openstack.org/#/c/218898/ for us? we would sure like it merged if it's good to go :)18:33
fungialso puppetmaster is ssh'ing into new etherpad-dev successfully as of a little while ago18:33
anteayaclarkb fungi speed for taking a git server out of rotation, upgrading and putthing it back in is about one per day?18:33
fungianteaya: can be much faster if we want18:33
anteayadocaedo: I don't have +2 on that repo18:33
*** ybathia has joined #openstack-infra18:33
clarkbanteaya no I think I can do the whole set today if I dont get too distracted18:33
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Switch Rally gates to new DevStack plugin style  https://review.openstack.org/21982318:33
fungianteaya: we were just starting slow in case we had to roll back, but once we're in for a penny...18:34
* anteaya clears space for clarkb18:34
anteayafungi: /me nods18:34
boris-42fungi: hi could you please take a look at that patch ^18:34
*** esker has joined #openstack-infra18:34
boris-42fungi: rally gates are terrible broken (they just test master)18:34
clarkbok that stack of gate changes merged, I am promoting now18:34
anteayaclarkb: yay18:34
docaedoanteaya: no problem - if you happen to know anyone who might feel like reviewing it though, please let me know (hint hint fungi clarkb crinkle )18:35
*** kushal has quit IRC18:35
*** esker has quit IRC18:35
*** sthillma has quit IRC18:35
*** sthillma_ is now known as sthillma18:35
anteayadocaedo: clarkb is upgrading git servers today18:35
anteayaand crinkle is in a puppet sprint18:35
anteayabut hopefully someone will review soon18:35
clarkbI had forgotten how this works18:35
docaedoanteaya: well, I think all things considered, my app catalog patch is probably much more important than that, right?18:35
clarkbit actually clears out the queue then redoes it right?18:35
clarkbfungi: ^18:35
*** esker has joined #openstack-infra18:35
anteayadocaedo: I have no idea how important your app catalog patch is18:36
fungiclarkb: it cancels running jobs and reorders the queue18:36
docaedohahaha18:36
anteayaand won't speak for others and how they manage my time18:36
fungiclarkb: so in a manner of speaking, i guess18:36
clarkbfungi: and status.json shows them go out of queue but will redo it ya?18:36
fungiclarkb: correct18:36
anteayaexcept that I will be clearing space for clarkb18:36
clarkbah yup everything is back again18:36
docaedoanteaya: I'm just kidding, the stuff clarkb is working on is more important by a long shot at the moment18:36
anteayaas having git servers running different oss has problems so it is best if we complete the work as soon as we can18:36
anteayadocaedo: oh okay18:37
anteayathanks for understanding18:37
fungidocaedo: anteaya: yep, with puppet mid-cycle sprint going on, we're down a lot of our usual puppet-centric backup teammates18:37
anteayahow they manage their time18:37
rcarrillocruzjeblair: sweet, was working that on my old vm, so probably a new thing on latest socketio versions. In regards to the wstunnel module, i *think* i didn't put cos by the time the servers of system-config were on precise/2.2, which does not come with that module18:38
rcarrillocruzso i left it for a later commit when it was upgraded18:38
rcarrillocruzbut now that i'm thinking i could've probably installed it conditionally per distrelease18:38
fungircarrillocruz: which is what the manifest is doing now18:39
rcarrillocruzhappy to see etherpad on websockets btw, it performed better for my on my tests with websockets enabled18:39
fungiat least as soon as the change merges in a few moments18:39
rcarrillocruzyeah, just saw the comment from jeblalir and the change, is already approved18:40
*** burgerk has quit IRC18:40
fungiand as soon as we upgrade our production server we'll be using it there18:40
j^2I’m starting to build tho OpenStack Operators launchpad, i accidently gave OpenStack Administrators Maintainer rights and removed my own for the set up, can someone help me out here?18:40
rcarrillocruz++18:40
fungij^2: probably. what's your lp username?18:41
j^2fungi: d-jj18:41
fungij^2: and what's the project shortname (the part in the url)?18:41
j^2osops18:41
fungithanks, looking now18:41
*** sdake_ has joined #openstack-infra18:41
j^2rockon thanks fungi18:41
slagleif anyone has a spare moment, tripleo is looking for reviews on https://review.openstack.org/#/c/215186/18:41
*** ihrachys has joined #openstack-infra18:42
*** jsavak has quit IRC18:42
*** e0ne has joined #openstack-infra18:42
openstackgerritMerged openstack-infra/puppet-etherpad_lite: Fix websocket configuration  https://review.openstack.org/21979018:42
fungij^2: i've switched it back to you for now so you can finish setting what you need18:43
j^2fungi: perfect, thanks again.18:43
* clarkb updates schedule and eats food before digging into server replacements18:43
*** jsavak has joined #openstack-infra18:43
openstackgerritMerged openstack-infra/shade: Pass timeout to session, not constructors  https://review.openstack.org/21929018:43
*** sdake has quit IRC18:45
fungiclarkb: huh... nova list in our openstackci tenant on rax is suddenly giving me an unauthorized error from the puppetmaster18:45
clarkbfungi: let me check18:45
rcarrillocruzfungi: when you get a sec, could you have a look at https://review.openstack.org/#/c/219181/ and https://review.openstack.org/#/c/219185/ ? infra-ansible is not passing tests cos I set it up with the ansible job template, but that's only meant for ansible roles18:45
fungicreds file hasn't been touched since may18:46
clarkbcan confirm18:46
clarkbmaybe it really is time for lunch :)18:46
fungiyep, get to it18:47
*** austin81 has joined #openstack-infra18:47
fungii can log in via the webui at least18:47
fungioh yeah, looks like our node count has gone to pot too. same problem with openstackjenkins18:48
fungii suspect rackspace just tanked their caystone api18:49
anteaya:(18:50
*** btully has joined #openstack-infra18:52
nibalizerhttps://status.rackspace.com/ seem that they don't think they have an issue18:52
fungiyeah, and while they've got a migration maintenance going on today for dfw and a (supposedly solved) issue for ord, i'm even seeing it in iad18:52
*** olaph has quit IRC18:53
*** kdas_ is now known as kushal18:53
*** kushal has quit IRC18:53
*** kushal has joined #openstack-infra18:53
*** harlowja has quit IRC18:54
*** olaph has joined #openstack-infra18:55
*** julim has joined #openstack-infra18:56
openstackgerritsebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation  https://review.openstack.org/19845818:58
*** harlowja has joined #openstack-infra18:58
*** Rockyg has joined #openstack-infra18:58
fungihuh, i don't find myself poking around the rax dashboard except when their api goes kaput... looks like they started doing ping monitoring by default? any of the instances we've launched in the past ~month have it turned on, but the rest do not18:59
*** cdent has joined #openstack-infra19:00
*** markvoelker has joined #openstack-infra19:00
*** dkranz has quit IRC19:00
fungioh! i see what it is19:00
* anteaya wants to know what it is19:00
*** HeOS has joined #openstack-infra19:01
fungino, wait, i don't19:01
anteaya:(19:01
mordredanteaya: it is what it is19:01
fungiit's what i thought at first. they really do have ping checks going on the newest instances we've booted19:02
anteayaso I'm told19:02
*** bcourt has quit IRC19:02
openstackgerritMorgan Fainberg proposed openstack-infra/project-config: Add announcements for keystoneauth to #openstack-sdks  https://review.openstack.org/21984219:03
*** mrunge has quit IRC19:03
*** julim_ has joined #openstack-infra19:03
fungijroll: you seem to have your finger on the pulse on all things rackspace. any idea why all of our tenants suddenly (in the past 20-30 minutes) started getting authorization errors from the api for all regions?19:04
funginothing on the status page seems to correspond19:04
*** julim has quit IRC19:06
clarkbfungi: nodepool must be affected?19:06
fungiclarkb: i expect so, the graph looked pretty wonky19:06
*** coolsvap_ has joined #openstack-infra19:09
*** hyakuhei has joined #openstack-infra19:10
*** SotK has joined #openstack-infra19:10
jeblairfungi: nodepool _seems_ to be talking to rax still19:11
fungiyeah, maybe it's already authenticated19:11
jeblairthat could be due to an existing session19:11
*** beboj has joined #openstack-infra19:11
fungii agree tailing the debug log it seems working19:11
hyakuheiAny docs people around?19:11
jeblairfungi: 'nodepool alien-list rax-dfw' works19:11
jeblairfungi: that's suspicious19:12
fungihuh, weird. i wonder if something's different about the puppetmaster19:12
jeblairthat should be entirely new sessions, etc.19:12
*** e0ne has quit IRC19:12
fungimaybe one of the new *client releases is tripping us up?19:12
fungithere were several while i was at lunch, apparently19:12
*** mrmartin has quit IRC19:13
*** dkranz has joined #openstack-infra19:13
mordredfungi: did we restart nodepool?19:13
fungimordred: nope19:13
fungithough the cli should be authenticating fresh anyway19:13
mordredok. so none of the most recent shade changes should be to blame19:13
mordredyeah19:13
jeblairmordred: nodepool is the thing working19:14
mordredah19:14
jeblairmordred: python-novaclient on puppetmaster is not19:14
fungiyeah, where we're seeing it is novaclient on the puppetmaster19:14
mordredAH19:14
mordredsorry19:14
mordredhadn't read the entire scrollback19:14
fungilooks like we got 2.27.019:14
mordredfirst panic is "I just landed things to how we call client constructors in shade"19:14
*** shashank_hegde has quit IRC19:14
fungithat's today's release19:14
fungismart money's on latest novaclient wanting different envvars19:15
fungi(again!)19:15
jeblairfungi: 0.26 works 0.27 fails19:15
fungii just confirmed the same19:15
anteayaskylerberg: you don't need a spec to put your code in openstack/19:15
mordredjeblair: I concur19:15
mordredthis is the same behavior locally on my laptop19:15
anteayaskylerberg: just create a patch to project-config to create the repo and add your code to it19:15
jeblairbut that's unpossible with semver!19:16
fungichecking release notes now19:16
*** bcourt has joined #openstack-infra19:17
* mordred is comparing http debug19:17
*** annegentle has quit IRC19:17
*** sharikapongubala has quit IRC19:18
mordredthe HTTP transactions for "list" are definitely different19:19
mordredso - in the working version, novaclient gets the token, then does a POST to /v2.0/tokens19:20
mordredthen does some iso8601 processing19:20
mordredthen does a GET of /v2/{project_id}/servers/detail19:21
*** beboj has left #openstack-infra19:21
mordredin 2.27.0 it does the post to /v2.0/tokens19:21
mordredit then does a get of /v219:21
mordredwhich gets a 40119:21
mordredit then does another post and another get19:22
mordredwhich then propagates the 40119:22
mordredso -I'm not sure what it's trying to accomplish with the GET of /v219:22
mordredbut it clearly is not working19:22
mordredsdague, jroll: ^^19:22
*** bharathm has quit IRC19:23
sdaguemordred: you got example code and the failure?19:23
mordredsdague: yeah19:23
mordredit's "nova list" on the command line with 2.27.019:23
mordredsdague: http://paste.openstack.org/show/442203/ <-- 2.26.0 against Rackspace19:24
mordredsdague: http://paste.openstack.org/show/442204/ <-- 2.27.0 against Rackspace19:25
clarkbiirc rax does more precision on timestamps than keystone does19:25
mordredsame creds19:25
clarkbif iso8601 is related that may be part of it19:25
mordred2.27.0 doesn't seem to be doing the timestamp processing19:25
mordredat all19:25
mordredor - it might be - and we might have stopped logging iso860119:25
*** BharatK has quit IRC19:26
sdagueprobably stopped logging it19:26
fungii'm testing against hpcloud for comparison, but there it seems to probably be timing out19:27
sdagueyeh, it's the version discovery going arry19:27
sdaguemordred: what's the env (minus secrets) that's set?19:28
mordredsdague: one sec19:28
fungii bet this is the best novaclient yet for use with public clouds that run devstack ;)19:28
mordredsdague: http://paste.openstack.org/show/442205/19:29
mordredsdague: also OS_PASSWORD is set19:29
sdagueman paste.o.o is so slow19:29
*** puranamr has quit IRC19:29
sdaguemordred: what's the service catalog look like for rax?19:30
mtreinishmordred: without OS_PASSWORD in that paste, how am I supposed to know whether you used the right creds? :)19:30
mordredsdague: one sec19:30
mordredmtreinish: :)19:30
*** wshao_ has quit IRC19:30
*** puranamr has joined #openstack-infra19:33
mordredsdague: http://paste.openstack.org/show/442206/19:34
*** annegentle has joined #openstack-infra19:35
sdagueyeh, well their service catalog is wrong ....19:35
sdaguesigh, ok, pondering the right work around19:35
sdaguethey've explicitly said "versionInfo |    https://dfw.servers.api.rackspacecloud.com/v2  "19:36
sdagueis where you should get the versionInfo19:36
mordredyes. they have said that.19:36
mordredit does not work19:36
fungihuh, i think there may be a mass mailer bouncing their execution through http://paste.openstack.org/show/437782/ to avoid locally installing their code. i see hundreds of get requests a second from a single ip address for that page19:36
*** austin81 has left #openstack-infra19:37
mordredfungi: we should edit the code on that paste19:37
fungithat would be hilarious. set the recipient to the fbi crime reporting address19:38
mordred++19:38
sdaguemordred: can you do a favor for me and live hack a thing19:38
lifelessKennan: I don't know, sorry.19:38
*** austin81 has joined #openstack-infra19:38
sdaguemordred: just print out the value of url there - https://github.com/openstack/python-novaclient/blob/893bf9313c2efb78d30fc5df3b67313eec154caa/novaclient/v2/versions.py#L4319:38
lifelessdhellmann: I see clarkb was on it. clarkb for my edification, where does that patch go ?19:38
sdaguethat's where things go wrong for you19:38
*** amitgandhinz has quit IRC19:38
sdagueI no longer have rax creds, so I can't test this thing19:39
fungilifeless: i answered Kennan's question back around 12:19 utc19:39
fungibad tox.ini19:39
mordredsdague: sure. one sec19:39
jeblairfungi: https://review.openstack.org/gitweb?p=openstack%2Fzaqar.git;a=commitdiff;h=e9dd687983dfc8dcab0f8413dcedd3294c9143f919:39
sdaguemordred: I guess I'd also love the value of get_endpoint raw before the splitting19:40
fungijeblair: hah, did you use hound to find that?19:40
jeblairfungi: no :( google19:40
fungiand yeah, the offending source address seems to be on telstra broadband in nz, so fbi probably isn't the best option to crank whoever that is19:42
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add db migration to align column size to 255  https://review.openstack.org/20325119:42
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Compact existing schema and start 1.x.x release branch  https://review.openstack.org/20325219:42
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: SUPER WIP: Stop using oslo anything  https://review.openstack.org/20330819:42
mordredsdague: ('get_current:endpoint:', u'https://dfw.servers.api.rackspacecloud.com/v2/956868')19:42
fungibut i am going to drop that ip address in our iptables penalty box for now19:42
mordredsdague: and I get the error on line 4319:42
clarkblifeless context?19:44
*** gyee has quit IRC19:44
*** coolsvap_ has quit IRC19:45
*** SumitNaiksatam has joined #openstack-infra19:45
*** shashank_hegde has joined #openstack-infra19:47
*** esker has quit IRC19:48
*** bpoulos has left #openstack-infra19:49
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: SUPER WIP: Stop using oslo anything  https://review.openstack.org/20330819:50
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Compact existing schema and start 1.x.x release branch  https://review.openstack.org/20325219:50
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add db migration to align column size to 255  https://review.openstack.org/20325119:50
*** ildikov has quit IRC19:51
SumitNaiksatamhi, we are seeing an issue with our gerrit patches from a stackforge project not entering the merge queue19:52
fungiSumitNaiksatam: let's see an example and i should be able to give you the cause19:52
openstackgerritgreghaynes proposed openstack-infra/system-config: First stab at making things bifrosty  https://review.openstack.org/20875119:52
SumitNaiksatamfungi: https://review.openstack.org/#/c/219397/19:52
SumitNaiksatamfungi: thanks, the last patch before the merged roughly a day back19:53
fungiSumitNaiksatam: its parent commit isn't approved, and isn't even passing tests19:53
*** b10n1k_ has joined #openstack-infra19:53
*** hdd has quit IRC19:53
fungiSumitNaiksatam: see the linked change in the dependencies box there in that page19:53
SumitNaiksatamfungi: ah bummer, did notice the dependency issue, sorry about the false alarm!19:53
sdaguemordred: so on actual nova you can get the top of the version URL and get information about the service19:53
SumitNaiksatam*didnt19:53
SumitNaiksatamfungi: thanks!19:54
fungiSumitNaiksatam: you're welcome19:54
mordredsdague: yah. which is cool19:54
sdaguehttp://paste.openstack.org/show/442210/19:54
sdaguewe clearly did not account for RAX's not actually nova19:54
mordredsdague: how silly of you19:54
*** hdd has joined #openstack-infra19:54
mordredsdague: perhaps trapping for a 401 on the version list and defaulting to something old and dumb?19:55
sdagueyeh, so clearly we need a work around19:55
openstackgerritgreghaynes proposed openstack-infra/system-config: Add bifrost inventory management procedures  https://review.openstack.org/21427919:55
sdaguethere is a nova meeting in the morning, we'll huddle up there, if you can pin back temporarily19:56
*** pc_m has quit IRC19:56
*** julim_ has quit IRC19:56
mordredsdague: we can - we're fine - just mainly wanted to make sure that nova was aware19:56
sdagueyep, thanks much19:56
*** ZZelle_ has joined #openstack-infra19:57
lifelessclarkb: the matrix thing19:57
clarkblifeless: oh, thats a file in devstack-gate that determines what services we enable when testing specific branches19:58
clarkblifeless: so I updated features.yaml in devstack-gate to allow liberty as a valid branch19:58
clarkbsdague: would this affect any nova too old to have microversions or too old to have that functionality?19:59
lifelessclarkb: thanks19:59
clarkbsdague: or is rax extra special here ?19:59
sdagueno, this is part of v2.0 for ever19:59
*** puranamr_ has joined #openstack-infra20:00
*** puranamr has quit IRC20:00
*** erhudy has joined #openstack-infra20:00
*** shardy has quit IRC20:00
*** bharathm has joined #openstack-infra20:01
jrollmordred: fungi so tl;dr latest novaclient is broken with our keystone? or did I skim too quick20:01
clarkbold git07 is deleting now20:02
clarkbgoing to boot two new ones one each for 06 and 05 if quota will let me20:02
fungijroll: possibly broken with all non-devstack clouds. hard to tell just yet20:02
sdaguefungi: no, just with rax20:02
sdagueit's not even the nova code, it's something in their repose config probably20:03
fungisdague: oh, interesting that it's timing out with hpcloud then20:03
*** geoffarnold is now known as geoffarnoldX20:03
*** bharathm has quit IRC20:03
fungiERROR (ConnectionRefused): Unable to establish connection to http://region-b.geo-1.compute-internal.hpcloudsvc.com/v2/20:03
*** bharathm has joined #openstack-infra20:03
fungii guess that could be unrelated20:03
sdagueyeh, I hope so. This really should exist everywhere20:04
*** amitgandhinz has joined #openstack-infra20:04
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: SUPER WIP: Stop using oslo anything  https://review.openstack.org/20330820:04
fungithat's what i get after several minutes of nova list hanging at the console20:04
*** ildikov has joined #openstack-infra20:04
jrollfungi: oh fun, let me know if I can help :)20:05
mordredjroll: yes20:05
mordredsdague: I believe HP also does not allow access to that endpoint20:06
*** Sree has quit IRC20:06
jeblairfungi, sdague: i agree, 2.26 works with hpcloud and 2.27 does not.20:06
*** baoli has quit IRC20:06
sdagueo_O20:06
mordredso - it's broken with RAX and HP20:06
*** glauco has quit IRC20:06
*** gyee has joined #openstack-infra20:06
mordredsdague: /v2 raw hadn't been used by any tools for anything - so I imagine it's not opt-in'd to the apigee or lbs or whatever20:06
sdagueffs people, we put code into the project, and then you block it by proxy ?20:06
mordredyeah20:06
fungiwhat are the odds of it being broken _only_ on rackspace and hpcloud but working for other major service providers?20:07
mordredthat is the predominant mode of deployment20:07
mordredfungi: wnat me to check?20:07
sdagueyes please20:07
* mordred has all the accounts20:07
fungimordred: i don't have an easy way to check, so it was mostly rhetorical20:07
fungibut if you can make it non-rhetorical, all the better20:07
mordreddreamhost is hanging20:07
anteaya:(20:07
jeblairi feel like having a non-theoretical answer will help in future keynote rants.20:07
mordredvexxhost works, as expected20:07
mordredauro I get 40120:08
sdaguejeblair: I am all for non theoretical answers20:08
fungifour... four clouds! hahaha20:08
fungi</the_count>20:08
mordredactually - let me make a quick venv so that I can verify that 26 works on each of these20:08
clarkbmordred: what about runabove?20:09
mordredclarkb: they were hanging before the thing which is why I'm checking 2620:09
mordredyeah. runabove is not responding. I'm guessing htey're working on transitioning to ovh cloud20:10
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add openstack-modules.env  https://review.openstack.org/20961720:10
openstackgerritColleen Murphy proposed openstack-infra/system-config: (WIP) Create playbook to puppet InfraCloud nodes  https://review.openstack.org/20963220:10
*** Sukhdev_ has quit IRC20:11
openstackgerritRobert Clark proposed openstack-infra/project-config: Changes to allow publishing of Anchor developer documentation  https://review.openstack.org/21986520:11
EmilienMcrinkle: why don't you target stable/kilo for ironic module on https://review.openstack.org/#/c/209617/14/openstack-modules.env,cm ? because of bifrost ?20:12
sdaguemordred: so for these clouds "nova version-list" works on 2.26?20:13
mordredsdague: unitedstack works with 26 and hangs with 2720:13
mordredsdague: checking20:13
*** jtomasek has joined #openstack-infra20:13
mordred| v2.0 | CURRENT | 2011-01-21T11:33:21Z |20:13
mordredoh!20:13
mordredit works on unitedstack both ones20:13
mordredlet me try a few more things20:13
mordredso - vexxhost and unitedstack both just wor20:14
openstackgerritColleen Murphy proposed openstack-infra/system-config: First stab at making things bifrosty  https://review.openstack.org/20875120:14
*** Swami has quit IRC20:14
*** cburroughs has quit IRC20:14
*** glauco has joined #openstack-infra20:14
openstackgerritgreghaynes proposed openstack-infra/system-config: First stab at making things bifrosty  https://review.openstack.org/20875120:14
openstackgerritgreghaynes proposed openstack-infra/system-config: Add bifrost inventory management procedures  https://review.openstack.org/21427920:15
mordredsdague: on hp, version-list works, but nova list hangs20:15
clarkbI have git05 and git06 replacements building now20:15
*** Swami__ has quit IRC20:15
*** austin81 has left #openstack-infra20:15
mordred| v2.0 | CURRENT | 2011-01-21T11:33:21Z |20:15
mordredthat's hp version list20:15
sdagueyeh, they should all be the same20:15
sdaguerax?20:15
mordredrax version-list fails 40120:15
*** sdake_ is now known as sdake20:15
mordredon both 26 and 2720:15
sdagueare they the only one?20:15
EmilienMcrinkle: if yes, I would suggest to backport the patches to stable/kilo20:16
mordredsdague: yes20:16
sdagueok20:16
mordreddreamhost and hp work with version-list with 2.6 and fail with 2.720:16
mordredauro:20:17
mordred('get_current:endpoint:', u'https://api.auro.io:8774/v2/0776b1156efa42b4b3e14fca96cabf00')20:17
sdagueok, so that gives me a path. We can get the same info off the resource that version-list calls20:17
mordredERROR (ConnectionRefused): Unable to establish connection to http://api.auro.io:8774/v2/20:17
mordredbut also works with version-list in 2620:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental puppet-unit job for single node  https://review.openstack.org/21986620:17
sdagueyeh, so version list is actually doing a GET of http://api.auro.io:877420:17
*** geoffarnoldX is now known as geoffarnold20:19
*** glauco has quit IRC20:19
sdaguemordred: thanks for all of that20:21
*** markvoelker has quit IRC20:21
*** mwagner_lap has quit IRC20:23
clarkbI have updated ssh rsa host keys to match the old hosts and will flip DNS now then do git repo replication20:23
clarkbI probably should puppet the host keys but it will likely take longer than just doing it20:23
*** ildikov has quit IRC20:23
nibalizerya probably20:23
nibalizeralso there is a bootstrap problem there likely because of known-hosts on the puppet master and ansible ya?20:24
mordredclarkb: just do it20:24
clarkbzaro: ^ maybe there is a way to teach gerrit to reload its known hosts file if it hits X number of host key mismatches?20:27
clarkbzaro: or maybe make it a admin command?20:27
*** sharikapongubala has joined #openstack-infra20:27
mordredclarkb, sdague: runabove list works with 2.26, hangs with 2.27. version-info hangs in both20:27
greghaynesSpamapS: How did you make that inventory erb on the bm deploy host?20:28
greghaynesSpamapS: I see, you commited it20:29
*** jsavak has quit IRC20:29
*** sharikapongubala has quit IRC20:29
clarkbok DNS is done now just waiting for it to resolve properly on review.o.o20:29
*** bhunter71 has quit IRC20:30
asselinis this the proper way to restart zuul? we always use sudo service zuul stop/start/restart https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart20:30
*** jsavak has joined #openstack-infra20:30
clarkbasselin: it depends on your init system, systemd for example would do something like systemctl restart zuul20:30
sdakecould a core reviewer ack this review for kolla infra so I can fix up the project rename review scheduled for the 11th?20:30
sdakehttps://review.openstack.org/#/c/217926/20:30
clarkbasselin: with upstart `restart zuul` is the coorect incantation iirc20:31
sdakealready has a +2 from Ian20:31
*** bswartz has quit IRC20:31
sdakefwiw, this gate is failing all the time (its a removal patch)20:31
asselinok thanks20:31
sdakeso it would also free up more resources20:31
sdakeas an aded plus ;)20:31
clarkbI have about 45 minutes until DNS is correct on review.o.o20:31
clarkbsdake: there is a disconnect between removing the f21 job and the rest of the commit message. How are they related?20:32
amullerquick question - I recall that the neutron functional job had at some point a mechanism that would fail a test if it used sudo directly without neutron-rootwrap, that was controlled by setting the sudo settings on the stack user correct? It seems like that's gone, and the stack user that is executing the tests can actually just call random sudo commands now20:33
clarkbamuller: it was done by the run-tox.sh script20:34
clarkbamuller: it noted when the job started in the auth log and checked for any instances of sudo after that20:34
clarkbif found the job exited 120:34
dimtruckgood afternoon!  i was wondering if there's a way to tie a node in a nodepool to a specific openstack project20:34
clarkbso could be a number of things not working there, have an example job run we can look at?20:35
amullerclarkb: Thanks I'll look in to it20:35
amulleryes20:35
*** jsavak has quit IRC20:35
amullerclarkb: http://logs.openstack.org/03/216603/4/check/gate-neutron-dsvm-functional/9ed19a4/console.html, ctrl-f for '['sudo20:35
amullerclarkb: I know exactly why it's using sudo directly in that specific test20:35
dimtrucke.g. run nova jenkins jobs on a specific instance ... i tried http://docs.openstack.org/infra/nodepool/configuration.html but couldn't find anything20:35
amullerand I'm going to fix that20:35
*** e0ne has joined #openstack-infra20:35
amullerclarkb: but I first want to re-enable the code that checks for direct sudo usage without rootwrap20:35
*** jsavak has joined #openstack-infra20:36
*** sharikapongubala has joined #openstack-infra20:36
fungimordred: sdague: is there a novaclient bug report open for the currently observed behavior, or should i go ahead and create one?20:36
clarkbamuller: oh, so that actually runs in devstack-gate which uses a different mechanism, we remove the stack users sudo privs there20:36
sdaguefungi: mordred is creating one right now20:36
sdagueit will include george carlin level language20:37
clarkbamuller: and the job did fail, are we sure that this isn't working properly?20:37
fungiokay, i'll hold off following up to the release announcement in that case20:37
amullerclarkb: it failed for a different reason20:37
amullerclarkb: there's plenty of tests in that run that are actually using sudo without rootwrap20:37
amullerand passed20:37
*** markvoelker has joined #openstack-infra20:37
*** Hal has joined #openstack-infra20:37
*** Hal has quit IRC20:37
sdaguefungi: and I'll look at fixes in the morning20:37
clarkbamuller: I see what it is, that test overrides the gate_hook which is where we remove sudo powers normally20:38
*** rfolco has quit IRC20:38
clarkbamuller: so the test has gone off and done its own thing without preserving that functionality20:38
amullerclarkb: ok, I'll add it back20:38
mordredfungi: https://bugs.launchpad.net/nova/+bug/149157920:38
openstackLaunchpad bug 1491579 in OpenStack Compute (nova) "against all sanity, nova needs to work around broken public clouds" [Undecided,New]20:38
clarkbamuller: look for DEVSTACK_GATE_REMOVE_STACK_SUDO in devstack-gate to see how we do it typically20:39
amullerclarkb: Much appreciated :)20:39
fungimordred: thanks, i'll reply on the -dev ml with a link to that unless you're already doing so20:39
mordredis there a thread?20:39
*** Hal has joined #openstack-infra20:39
clarkbmordred: sdague it is common to put layer 7 firewall type things in front of web services and they have to be trained to accept traffic, usually you do this by putting it in permissive type mode and hitting it with all the client functionality, then you tell it to enforce20:39
clarkbmordred: sdague so the client not exercising valid APIs there may have bitten us20:40
*** vahidh has joined #openstack-infra20:40
mordredclarkb: what does that gain?20:40
clarkbwhether or not thats sane is another question but in my experience very common for deploying HTTP services20:40
fungimordred: i replied to the novaclient release announcement20:40
sdakeclarkb perhaps I busted the commit let me look second20:40
*** sharikapongubala has quit IRC20:40
fungimordred: because that's a good way to start a -dev ml thread about breakage ensuing in new releases20:41
sdagueclarkb: yeh, so my plan is, create a series of work arounds in novaclient so people can work again20:41
*** sharikapongubala has joined #openstack-infra20:41
sdaguethen add tempest tests to touch all these urls, and ensure they get into the defcore definition20:41
sdagueso you just won't be openstack if you don't expose this20:41
mordredsdague: ++20:41
sdakeclarkb we are remvoing this gate :   - name: gate-kolla-functional-f2120:41
clarkbsdake: yes but what does that have to do with bash vs python? there is a giant paragraph of stuff that has nothing to do with f2120:42
*** dprince has quit IRC20:42
sdakeclarkb yes i understand now the commit log is confusing20:42
sdakeclarkb probbly has something to do with the 36 hour days i've been pulling over the last 3 weeks ;)20:43
sdakewould you prefer i udpate the commit log?20:43
*** julim has joined #openstack-infra20:43
sdakeI just want to get er done before the 11th so i can get the renames done in one go20:43
clarkbsdake: well first just want to check that the change does what you want it to do and if it does then yes please update the commit message, otherwise updatethe change to do the thing that is described20:44
clarkbsdake: because as is I have no idea what the correct thing is20:44
sdakeso basically functional-f21 uses a bash script tobulid20:44
sdakethe other gates use a python tool20:44
sdakefunctional-f21 is really misnamed it only does bah script building tet20:44
sdakebut the bash scripts have been removed from the repo20:44
clarkbI see, in that case a small update to the commit message that says "removing the functional-f21 job removes the only use of the old bashscript"20:44
sdakeso its a dead gate20:44
sdakewill do20:45
*** Piet has quit IRC20:45
sdakein a ocnversation in tripeo atm, but after that I'll udpate20:45
sdakethanks for the recommendation - hopefully can get a quck review after20:45
crinkleEmilienM: yes we should backport it20:45
*** harlowja has quit IRC20:45
*** wshao has joined #openstack-infra20:46
*** signed8bit has joined #openstack-infra20:47
*** puranamr_ has quit IRC20:48
*** markvoelker has quit IRC20:48
jasondotstarI've got a system-config qn20:50
jasondotstarqn is about running a local.pp file to setup a VM as a test node for review.o.o20:50
mordredjasondotstar: yes! it's definitely a thing to do20:51
*** wshao has quit IRC20:51
jasondotstartried using this: http://paste.openstack.org/show/442387/20:51
*** dkranz has quit IRC20:52
openstackgerritEmma Barber proposed openstack-infra/puppet-hound: Configure and run apache on top of hound  https://review.openstack.org/21164020:52
jasondotstarand basically following this: http://docs.openstack.org/infra/system-config/sysadmin.html#sysadmin20:52
jasondotstarto kick it off, I created this Vagrantfile: http://paste.openstack.org/show/442388/20:53
jasondotstari get what it's *suppose* to do... but it's not running cleanly20:53
jasondotstarlemme see if I can grab my output...20:54
boris-42mordred: hi20:54
*** ociuhandu has quit IRC20:54
*** kushal has quit IRC20:56
*** adrian_otto has joined #openstack-infra20:56
rm_workhey all, could we get this merged? https://review.openstack.org/#/c/217330/20:57
rm_workthe code it depends on is merged now in Barbican20:57
anteayarm_work: we are having a hard time merging anything right now20:57
*** julim has quit IRC20:57
rm_workwould like to get the jobs running for a bit20:57
rm_workahhh ok20:57
rm_workproblems? :(20:57
anteayaas both host clouds are having issues20:57
rm_workok20:57
anteayahttps://bugs.launchpad.net/nova/+bug/149157920:58
openstackLaunchpad bug 1491579 in python-novaclient "against all sanity, nova needs to work around broken public clouds" [Critical,Confirmed]20:58
anteayaand the backscroll has details20:58
rm_workthanks, I will come back ... later :) give you some time to work that out20:58
anteayarm_work: http://lists.openstack.org/pipermail/openstack-dev/2015-September/073535.html20:59
anteayarm_work: thanks for understanding, its a fun one20:59
boris-42anteaya: hi20:59
anteayahow can I help you boris21:00
*** armax has quit IRC21:00
*** stevemar has quit IRC21:00
*** armax has joined #openstack-infra21:00
*** trown is now known as trown|outttypeww21:01
openstackgerritSteven Dake proposed openstack-infra/project-config: kolla: Remove kolla-functional-f21 job  https://review.openstack.org/21792621:01
rm_workanteaya: lol, I love that bug report -- doesn't seem very professional, but made me laugh none-the-less21:02
annegentlemordred: around?21:02
anteayarm_work: the first time you have witnessed mordred's work I take it21:02
anteayaannegentle: hey21:02
boris-42anteaya: so in Rally project we are facing issues21:02
boris-42anteaya: currently all DSVM jobs are testing master (instead of patches)21:03
boris-42anteaya: this is fix https://review.openstack.org/#/c/219823/21:03
anteayaannegentle: if you could review dhellmann's patch both he and I would be very grateful: https://review.openstack.org/#/c/219420/21:03
boris-42anteaya: could you please take a look?21:03
anteayaannegentle: if you can work it on your stack21:03
anteayaboris-42: I can take a look21:03
boris-42anteaya: thank you21:03
anteayaboris-42: be aware our host clouds are both in a bad way21:03
anteayaas I just told rm_work21:03
rm_workyes, i mean, we should have known everything would be broken -- it's right about code freeze time, so this is to be expected21:04
*** btully has quit IRC21:04
rm_workthe gods enjoy taunting us right at code-freeze21:04
zaroclarkb: sorry, was afk.  gerrit loading known hosts is in regards to replication correct?21:05
*** rossella_s has quit IRC21:05
*** rossella_s has joined #openstack-infra21:06
anteayarm_work: now you've got it21:07
*** baoli has joined #openstack-infra21:07
*** glauco has joined #openstack-infra21:07
anteayaboris-42: you have two jobs that have two plugins enabled in the job, both sahara and rally and both manilla and rally21:07
annegentleanteaya: sure21:07
fungianteaya: the novaclient issue isn't (yet) impacting nodepool and we can pin it on the server if needed as a workaround should it start to do so21:07
anteayaannegentle: thank you21:07
*** Rockyg has quit IRC21:08
boris-42anteaya: as far as I remember we have more21:08
*** baoli has quit IRC21:08
boris-42anteaya: e.g. zaqar & rally, designate & rally21:08
anteayafungi: ah sorry, I thought you were seeing rax and hp issues and that the novaclient was the issue21:08
anteayafungi: but yes I did wonder how it could affect us so quickly21:08
openstackgerritSpencer Krum proposed openstack-infra/project-config: Run beaker tests on Puppet 3.8 and 4.2  https://review.openstack.org/21987921:08
*** baoli has joined #openstack-infra21:08
*** Swami has joined #openstack-infra21:08
*** Swami_ has joined #openstack-infra21:08
*** achanda has quit IRC21:08
anteayaboris-42: okay so I'm about to break for dinner and you can't believe how I hesitate to ask this question21:09
anteayabut if you have two devstack plugins enabled for the same job21:09
anteayahow can you know what you are testing?21:09
clarkbzaro yes21:09
*** thorst has quit IRC21:09
jasondotstarmordred: k sorry for the long wait but here's how I ran it: https://asciinema.org/a/2rdb51mh5lq7jm8xxr95ru9qa21:09
*** stevemar has joined #openstack-infra21:09
boris-42anteaya: so like we are running rally agaisnt zaqar21:10
boris-42anteaya: so you need to have rally & zaqar installed21:10
boris-42anteaya: that is why we are adding 2 plugins..21:10
*** tiswanso has quit IRC21:10
anteayabut the plugins pull in the tip of master, yes?21:11
*** pfallenop has quit IRC21:11
anteayato get the patch in, yes?21:11
anteayaisn't that what your commit message states?21:11
boris-42anteaya: sooo if we are trigerring rally job for rally patch21:12
*** glauco has quit IRC21:12
* anteaya listens21:12
boris-42anteaya: it should use zaqar master and rally + patch21:12
boris-42anteaya: if it is in zaqar gate it should run zaqar + patch and rally master21:12
boris-42anteaya: that's the idea=)21:12
boris-42it worked before21:13
boris-42but now we are trying to switch to new plugin base21:13
anteayayou weren't testing the patch before21:13
zaroclarkb: does gerrit load known hosts?  i thought ssh was the thing that loads it?  i could be wrong, but seems to me that load/reload of known hosts is outside Gerrit's scope.21:13
anteayathat is what your commit message states21:13
anteayathe job ran before21:13
*** Sukhdev_ has joined #openstack-infra21:13
anteayabut it wasn't testing what you thought it was testing21:13
anteayathat is what you tell me is the point of this patch21:13
boris-42anteaya: I know exactly that before the new devstack plugins we tested everything properly21:14
*** pfallenop has joined #openstack-infra21:14
clarkbzaro: gerrit uses a java ssh implementation (MINA iirc) and loads the file on start21:14
clarkbzaro: you hvae to restart gerrit to get it to reload the file21:14
boris-42anteaya: e.g. if it is in rally (it is rally+patch against master) if it is in other project other project + rally maser21:14
*** unicell has quit IRC21:14
anteayaboris-42: can you update your commit message then please?21:14
*** unicell has joined #openstack-infra21:14
*** ldnunes has quit IRC21:15
anteayabecause I am going by what your commit message says21:15
pabelangerclarkb: fungi: jeblair: When you have a moment, I wold be interested in feedback of: https://review.openstack.org/#/c/217346/21:15
boris-42anteaya: okay let me try to rephrase21:15
anteayawhich you now tell me is not an accurate reflection of reality21:15
pabelangerinterested in what you thing of the organization variable, or if you see another way21:15
anteayaI just want what you say you want to do to be reflected in the commit message21:15
pabelangerWe are looking, EmilienM and team, to get that merged to fix some depends-on issue we have with beaker-jobs and our integration job21:15
clarkbpabelanger: we would prefer to not have organization vars back in jjb21:16
clarkbpabelanger: can you use ZUUL_PROJECT instead?21:16
*** jtomasek has quit IRC21:16
*** bcourt has quit IRC21:17
pabelangerclarkb: don't think I can.  I would want ZUUL_PROJECT=foo to zuul clone project bar21:17
pabelangerand don't see how I could get that info21:17
*** chutwig has joined #openstack-infra21:18
clarkbpabelanger: ok well we need some way to not reencode the github thing back into all our config21:18
*** tonytan4ever has quit IRC21:18
clarkbin theory ZUUL_PROJECT is that thing but I may have misread what zuul cloner is doing21:18
zaroclarkb: ahh, ok.  i'll write up a bug to remind myself to look into that.21:19
*** dims has quit IRC21:19
pabelangerclarkb: okay, let me look21:19
asselingreghaynes, fyi, my images built sucessfully now w/o any updates. thanks21:19
*** dims has joined #openstack-infra21:19
pabelangerI couldn't see a way to extract out the organization info from zuul21:20
*** baoli has quit IRC21:20
clarkbpabelanger: you are setting up org/name with zuul cloner, the thing that encodes that in the zuul parameters is ZUUL_PROJECT21:21
clarkbpabelanger: so I don't understand why you can't use that parameter21:21
*** harlowja has joined #openstack-infra21:21
*** edmondsw has quit IRC21:21
*** chutwig is now known as erhudy121:22
clarkbits the zuul cloner default too iirc21:22
clarkbso it should just work (tm)21:22
* anteaya is off for dinner21:22
jeblairclarkb, pabelanger: use the macro "zuul-git-prep" to clone $ZUUL_PROJECT; use the macro "zuul-clone" to clone a _specific_ project21:23
*** dims has quit IRC21:24
jeblairbut that change is functionally equivalent to what's there right now21:24
*** aysyd has quit IRC21:25
jeblairas nibalizer points out21:25
*** tonytan4ever has joined #openstack-infra21:26
*** stevemar has quit IRC21:26
*** ociuhandu has joined #openstack-infra21:29
openstackgerritJJ Asghar proposed openstack-infra/project-config: Creation of the OpenStack Operators Project(s)  https://review.openstack.org/21976021:29
nibalizerjeblair: paul corrected me a bit, if the gate-keystone-... test is added to another repo then this change does the right thing21:29
*** ociuhandu has quit IRC21:30
clarkbnibalizer: pabelanger if thats how you want to do it you should have a single job that just does the correct thing, similar to how devstack gate works21:30
clarkbyou don't want many many integration tests, you just want one21:31
*** ociuhandu has joined #openstack-infra21:31
*** e0ne has quit IRC21:32
nibalizeror http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n12221:32
*** ociuhandu has quit IRC21:32
jeblairnibalizer: if that job were instantiated for the "foo" jjb project, then it would create "gate-foo-puppet..." and checkout foo.  it would do that regardless of using gerrit-git-prep or zuul-clone21:32
*** kgiusti has left #openstack-infra21:33
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Switch Rally gates to new DevStack plugin style  https://review.openstack.org/21982321:33
*** e0ne has joined #openstack-infra21:33
*** derekh has joined #openstack-infra21:33
boris-42anteaya: ^ better ?21:33
*** cdent has quit IRC21:33
fungipabelanger: clarkb: jeblair: nibalizer: wow, 217346 is a blast from the past. what was old really is new again21:33
*** ociuhandu has joined #openstack-infra21:33
jeblairfungi: if only it had shown up as a revert? :)21:34
fungisrsly21:34
nibalizeroh did we used to attach a 'project' to each repo?21:34
clarkbreplication is almost done to new 05 and 0621:34
nibalizerer organization21:34
clarkbonce done I will do some test clones and push up a change to put them in use21:34
funginibalizer: yep, even used the same parameter name21:35
clarkbnibalizer: yes21:35
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515121:35
fungiit was a pbd when we were finally able to rip it out21:35
nibalizerfungi: clarkb haha21:35
fungiwell, actually it was called github-org for a while i think21:35
fungii can't remember now if it got renamed to organization before being ripped out though... that was so many beers ago21:36
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for new /api directory and gzip  https://review.openstack.org/21889821:36
*** lucas-dinner has quit IRC21:36
nibalizerfungi: more beers more commits :)21:36
*** achanda has joined #openstack-infra21:37
*** dannywil_ has quit IRC21:40
*** pvaneck_ has joined #openstack-infra21:40
*** jsavak has quit IRC21:40
*** glauco has joined #openstack-infra21:41
*** e0ne has quit IRC21:41
openstackgerritMatt Riedemann proposed openstack-infra/system-config: Index logs/screen-m-shr.txt for debugging manila runs in the gate  https://review.openstack.org/21988821:42
*** olaph has quit IRC21:43
*** e0ne has joined #openstack-infra21:43
*** pvaneck has quit IRC21:44
*** marzif has joined #openstack-infra21:44
clarkbmriedem: should you do the other manila services too?21:45
mriedemclarkb: i thought about it21:45
mriedemi can21:45
mriedemmtreinish: seen this? http://logs.openstack.org/35/219835/2/check/gate-novaclient-dsvm-functional/a887156/console.html#_2015-09-02_20_48_59_36121:45
mriedemmelwitt: ^21:45
clarkbmriedem: I think people will expect all of them if one is indexed21:45
*** baoli has joined #openstack-infra21:46
*** alivigni has quit IRC21:46
openstackgerritMatt Riedemann proposed openstack-infra/system-config: Index logs/screen-m-shr.txt for debugging manila runs in the gate  https://review.openstack.org/21988821:46
mriedemlet me rename that21:46
openstackgerritMatt Riedemann proposed openstack-infra/system-config: Index manila service logs for debugging manila runs in the gate  https://review.openstack.org/21988821:47
*** dannywilson has joined #openstack-infra21:47
*** achanda has quit IRC21:47
*** FallenPegasus has joined #openstack-infra21:49
*** glauco has quit IRC21:49
mtreinishmriedem: http://lists.openstack.org/pipermail/openstack-dev/2015-August/072982.html21:50
*** amuller has quit IRC21:50
mriedemmtreinish: ok, i'm going to e-r query that thing21:50
mtreinishmriedem: it shouldn't be breaking anything21:50
mriedemit is21:50
mtreinishdoes the hook err exit on that?21:51
openstackgerritClark Boylan proposed openstack-infra/system-config: Switch to centos7 git05 and git06 hosts  https://review.openstack.org/21989221:51
clarkbanteaya: ^ tha will get us to half centos721:51
melwittmriedem: I find 20 hits the past 24h in logstash, one of them is success the rest are failures21:51
clarkbonce those are in I can delete the old centos6 hosts and do another batch (semi limited by rax quota here)21:51
mriedemmtreinish: melwitt: http://goo.gl/IQLHW121:52
clarkbmriedem: should nova consider releasing a 2.28 that is 2.26 rereleased?21:52
clarkbmriedem: then nova can discuss what to do with behavior change and maybe release a 3.0?21:52
mriedemclarkb: i was going to do 2.27.121:52
mriedemfor this https://review.openstack.org/#/c/219835/21:52
*** dzamboni has quit IRC21:52
*** e0ne has quit IRC21:52
*** prad- has quit IRC21:53
mriedemand then bump g-r min to that21:53
clarkbmriedem: right but that won't fix the break every public cloud behavior right?21:53
mriedemclarkb: no, we don't have a fix for that yet21:53
clarkband in theory a behavior change like that requires a 3.0 if we are semvering21:53
clarkbmriedem: right so maybe we release 2.26 as 2.28 so people don't get hit with that?21:53
mriedemmordred: ^ thoughts?21:53
mriedemi thought it sounded like we kind of expect clouds to expose that21:53
*** achanda has joined #openstack-infra21:54
mriedemplus i need to leave RSN to pickup my kid from daycare21:54
clarkbI think we do but it won't happen any time soon if I know anyhting abut change control21:54
mriedemyeah21:54
clarkband in the mean time not breaking everyone is nice21:54
mriedemlet them eat cake!21:54
*** glauco_ has joined #openstack-infra21:54
mriedemso,21:54
mriedemcan i double back on this in about an hour when i'm home?21:55
mriedemdansmith: ^ thoughts?21:55
clarkbsure21:55
mriedemnot that having a 4 year old in my ear is going to help...21:55
* dansmith reads21:55
mriedemi'll check back in when i'm home21:56
*** mriedem has quit IRC21:56
*** amitgandhinz has quit IRC21:56
dansmithclarkb: I wasn't really part of the discussion of that21:57
dansmithclarkb: I just threw unhelpful thoughts in the middle21:57
*** mtanino has quit IRC21:57
clarkbok, I was just throwing the idea out there since 2.27 provably doesn't work against the majority of public clouds that we (mordred) have access to21:58
clarkband at least one major onsumer of these clouds is going to have to either pin novaclient everywhere or halt config managment or both to work around this until the clouds catch up21:58
dansmithI think that the proposed change fixes it for everyone, right?21:59
*** Hal has quit IRC21:59
*** glauco_ has quit IRC21:59
dansmiththe correct behavior is to not pass it at all if it's not specified, which should satisfy the older and newer code, AFAIK21:59
fungidansmith: no, there is no proposed change for bug 149157921:59
openstackbug 1491579 in python-novaclient "against all sanity, nova needs to work around broken public clouds" [Critical,Confirmed] https://launchpad.net/bugs/149157921:59
*** burgerk_ has quit IRC21:59
fungidansmith: which is what clarkb is referring to22:00
dansmithfungi: not this? https://review.openstack.org/#/c/219835/22:00
clarkbdansmith: two unrelated bugs aiui22:00
funginope. that one's small potatoes22:00
dansmithI thought they were related22:01
fungidansmith: 1491579 is latest novaclient not working because most large service providers have decided to filter/break version discovery22:01
fungidansmith: so the most likely fix is a variety of workarounds to accommodate the different ways they've broken their nova implementations in production22:02
*** wshao has joined #openstack-infra22:02
*** wshao has quit IRC22:03
dansmithI see, I thought this was related to the device thing in terms of being able to determine what the proper behavior is22:03
mtreinishfungi: my fix for the version discovery thing is to get hogepodge to fail them if they block it22:03
mtreinishI'm working on getting that test into tempest to enable that22:03
dansmithI was really only tangentially paying attention to both of those conversations22:03
mtreinishbut it's sort of a long term fix22:03
clarkbmtreinish: I think we should do that too22:03
dansmithmy feeling is that the most graceful thing we can/should do is assume that if you fail to return versions, then we assume 2.0, and you never get any new features until you provide that22:04
dansmithand if you hang, then whatever, you're stupid22:04
clarkbin the mean time it would be pragmatic to have novaclient work against clouds, file bugs against the clouds for their brokeness, and work on the long term fixes from there22:04
dansmithsince this versioned API has been like three years in the making, I'm not sure I expect those bugs to go anywhere in a reasonable amount of time22:05
*** mtanino has joined #openstack-infra22:05
openstackgerritsebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation  https://review.openstack.org/19845822:05
clarkbdansmith: given how people setup http proxies I am assuming the fix is as simple as go update your rulesets and that will take however long change control for $cloud takes22:06
clarkbthat may or may not be a reasonable amount of time22:06
fungidansmith: yes, the alternative is to make a big stink in public and shame the offending providers into fixing their s...tuff22:06
*** adrian_otto1 has joined #openstack-infra22:06
fungiand let users suffer in the interim22:06
*** ivar-lazzaro has quit IRC22:06
*** Sree has joined #openstack-infra22:07
dansmithclarkb: I dunno, maybe22:07
*** adrian_otto has quit IRC22:07
*** jsavak has joined #openstack-infra22:07
dansmithfungi: yeah, I don't want to punch users in the face, but I also think that if we have to wait another six months to a year to get them all resolved, that will suck pretty hard22:07
*** signed8bit has quit IRC22:08
fungiyeah22:08
clarkbright so that is why I suggest a 3.0 release22:08
*** annegentle has quit IRC22:08
melwittdansmith: yeah, sdague is going to work on implementing that tomorrow he said22:08
clarkbmake it clear that this is not necessarily backward compatible with existing clouds and let users footgun if they want22:08
fungidansmith: and the thing is, if you could name 3 or 4 of the largest openstack-based public providers off the top of your head, it's likely they're all not working with novaclient 2.2722:08
dansmithclarkb: 3.0 is enough signaling for you?22:09
*** btully has joined #openstack-infra22:09
*** ams__ has quit IRC22:09
dansmithmelwitt: cool22:09
clarkbdansmith: ya its not perfect but according to semver would be a more correct way to handle this22:09
dansmithfungi: yeah, I saw the paste22:09
melwittdansmith: the fall back on v2.0 if getting versions fails or times out and use a short time out, that is22:09
dansmithclarkb: I don't understand at all how this is a semver-able thing, from the library side22:09
dansmithclarkb: are you saying "meh, a bunch of clouds are broken" is a semantic difference?22:10
clarkbdansmith: the client has made a change that is not backward compatible with existing clouds22:10
*** jaypipes has quit IRC22:10
dansmithclarkb: but only because those clouds are stupid, not because we're doing anything that is not legit22:10
clarkbdansmith: sure22:11
clarkbbut still not backward compatible22:11
clarkbas a user that is what you will see22:11
clarkbthe technicaly details don't matter much when novaclient stops working everywhere22:11
melwittin the ideal case we could just do a new release with that workaround in it and not have to re-release an older client as the newest (essentially a revert)22:11
clarkbmelwitt: that wont fix the problem22:11
clarkbit fixes a problem but not the big one22:12
dansmithclarkb: well, if we look for that behavior and dump a message with the phone number of the CEO of said cloud....22:12
clarkbdansmith: hahaha22:12
*** Sree has quit IRC22:12
dansmithclarkb: I dunno, I feel like the client moving forward is the most leverage we'll have22:12
clarkbdansmith: right I say move forwad with a 3.0 release22:12
clarkbthen users know hey upgrading to this may not work I have some testing to do22:12
*** jsavak has quit IRC22:13
clarkbrather than user upgrading point release and breaking everything grumping and never using openstack again22:13
dansmithI disagree with clouds being stupid being able to force us to make a versioning change like this, but if that's what people want as a compromise then that's fine I guess22:13
*** jsavak has joined #openstack-infra22:13
*** btully has quit IRC22:13
dansmithlike, in a less all-clouds-are-apparently-stupid situation, what percentage of them being stupid warrants us doing this?22:14
*** SumitNaiksatam has quit IRC22:14
dansmithlike, all the deployed redhat, mirantis, metacloud clouds (assuming) are fine...22:14
*** jcoufal has quit IRC22:14
*** Rockyg has joined #openstack-infra22:14
clarkbdansmith: its possible they are fine, depends on how they deploy the nova api22:15
fungii could get behind recommending the non-broken providers to every user who complains about this problem ;)22:15
dansmith"4 out of 5 dentists agree" -> semantic version bump22:16
clarkbbut clearly there is a majority of deployments that are publicly available doing the crazy thing so would not be surprised if private deployments fall into the same trap22:16
dansmithfungi: yeah, "sorry, your cloud is broken, but https:///foobar.com is not"22:16
dansmithclarkb: I'm not so sure22:16
*** SumitNaiksatam has joined #openstack-infra22:17
fungii mean, ultimately that's refstack/marketplace... "here's the list of non-broken providers. when you start running a non-broken openstack we'll be happy to add you (back) to the list!"22:17
rm_workas someone peripherally involved only, I vote for whatever makes it so we can merge changes soonest <_<22:18
*** marzif has quit IRC22:18
dansmithis this blocking the gate?22:18
rm_workI was told we can't merge stuff really right now because of this by anteaya22:18
clarkbdansmith: no, but if nodepool upgrades we will no longer boot any test VMs22:18
clarkbwhich means to fix that I have to go fix nodepool and a half dozen other places we use novaclient22:18
rm_workor maybe it was just that "everyone is busy looking at this, so we don't have time to review"22:18
rm_worki could have misunderstood22:19
dansmithrm_work: yeah, doesn't sound like that's right22:19
clarkbrm_work: I think it was more we don't have time to review22:19
*** hdd has quit IRC22:19
rm_workalright22:19
dansmithclarkb: can't we cap whatever nodepool is doing to avoid that terrible situation?22:19
dansmith"if nodepool upgrades" makes it sound like it has a mind of its own.. :)22:19
clarkbdansmith: yes and shade, and the puppetmaster, and ansible (though in theory will be shade there?),22:19
clarkband its not just us using rackspace and hpcloud and friends22:20
dansmithsure, but tomorrow is a deadline and if we know about it, then being proactive would be good22:21
clarkbI get that the public clouds are silly, but I also know that novaclient users won't care, they want a novaclient that works with clouds that exist22:21
fungibasically we can update our tools to only support python-novaclient<2.7.022:21
fungibut that seems like not a great long-term solution22:21
dansmithfungi: no, not a long-term solution22:22
dansmithfungi: a solution to avoid mutiny in the next 48 hours because we have a deadline and we knew about the iceberg early enough to make sure we didn't hit it22:22
* dansmith just watched a titanic documentary22:22
clarkbfungi: jenkins04 still happy with the thread counts22:22
*** sabeen2 has quit IRC22:23
*** sigmavirus24_awa is now known as sigmavirus2422:23
*** achanda has quit IRC22:24
*** achanda_ has joined #openstack-infra22:24
dansmithclarkb: given that this is going to happen now or later, has anyone filed a bug with the two clouds we use? presumably that is the most useful if someone does it "as the client"22:24
*** tonytan4ever has quit IRC22:24
clarkbdansmith: I have not, I was hoping mordred would considering he had all the accounts (iirc filing bugs requires accounts)22:24
dansmithright, okay22:24
clarkbjroll: onmetal question, does nova client just work with it ? (assuming we use a client version that works with rax)22:25
armaxclarkb: fwiw https://bugs.launchpad.net/grenade/+bug/127453922:25
openstackLaunchpad bug 1274539 in grenade "Failure in /.../grenade/projects/30_swift/upgrade.sh" [Undecided,New]22:25
clarkbjroll: eg can I nova boot --flavor onmetal --image somethinginglance clarkbmachine?22:25
*** eharney has quit IRC22:26
*** glauco has joined #openstack-infra22:26
*** thorst has joined #openstack-infra22:27
*** jsavak has quit IRC22:27
clarkbjroll: I ask beacuse the docs tell me to use this supernova thing http://www.rackspace.com/knowledge_center/article/using-onmetal-cloud-servers-through-api22:29
*** puranamr has joined #openstack-infra22:29
dansmithisn't supernova generally recommended for rax stuff?22:29
clarkbdansmith: we only use the upstream libs and other than the current issue and their ssl cert being something requests doesn't like it just works22:30
dansmithright, I meant,22:30
dansmithI thought that they document using supernova is probably just their SOP, and would expect the nova client to work as much as it does with vms22:30
clarkbalso we should totally rebug about the requests cert thing when we complain about the nova version api thing22:30
dansmithwhen I was playing with onmetal pre-ga,22:30
openstackgerritsebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation  https://review.openstack.org/19845822:30
*** btully has joined #openstack-infra22:31
dansmiththere were instructions to use the novaclient, IIRC22:31
*** puranamr has quit IRC22:31
clarkbdansmith: ok, so it should work or at least did work when you were testing it out?22:31
*** thiagop has quit IRC22:31
dansmithI didn't try it, but pretty sure there were instructions for it22:31
j^2Is Craig Bryant still around?22:31
dansmithI just used the gui because it was the quickest way to get a box to watch things go fast on :)22:31
clarkbgotcha22:33
fungij^2: seems to still be active on monasca as of last week https://review.openstack.org/#/q/owner:craig.bryant%2540hp.com,n,z22:33
dansmithclarkb: so basically, completely unhelpful response, but until jroll comes around, you're bored anyway, right? :)22:33
clarkbassuming supernova doens't do any fancy switching on the flavor type then novaclient would work22:33
j^2fungi: ah he posted a question on one of my reviews, i wanted to make sure i understood it: https://review.openstack.org/#/c/219760/522:34
dansmithhttp://docs.rackspace.com/servers/api/v2/cs-gettingstarted/content/overview.html22:34
dansmith^ mentions onmetal and nova client, makes no caveats, that I can see22:34
clarkbaha I needed out of the knowledge center to get the real docs22:35
clarkbgoogle you failed me22:35
*** glauco has quit IRC22:35
*** btully has quit IRC22:35
*** taron2 has quit IRC22:35
*** dannywilson has quit IRC22:35
clarkbyup and onmetal flavors are in that list22:35
dansmithyeah22:35
*** gordc has quit IRC22:36
*** dannywilson has joined #openstack-infra22:37
*** mwagner_lap has joined #openstack-infra22:37
clarkbok I have to go buy groceries for dinner, back in a bit to check up on git server backends22:38
fungiclarkb: i've approved that change22:40
clarkbthanks22:40
fungiconfirmed the existing 5 and 6 are handling no requests any longer22:40
clarkbI did disable them in haproxy22:42
*** unicell has quit IRC22:43
*** unicell has joined #openstack-infra22:43
armaxlifeless: ping22:43
lifelessarmax: pong22:43
armaxlifeless: I was wondering if you could help me understand how I can render testrepository.subunit.gz in a human readable format22:44
*** unicell has quit IRC22:44
clarkbchanging their IPs reenables them so no explicit enable needed22:44
*** unicell has joined #openstack-infra22:44
armaxlifeless: forgive the silly question22:44
openstackgerritMerged openstack-infra/system-config: Switch to centos7 git05 and git06 hosts  https://review.openstack.org/21989222:45
lifelessarmax: is this the file attached to a failed CI run in OpenStack ?22:45
armaxyes22:45
lifelessok, so there is a web render done automatically22:45
armaxlike this one22:45
armaxhttp://logs.openstack.org/01/214601/13/check/gate-tempest-dsvm-full/677ee89/logs/testrepository.subunit.gz22:45
reedthe fact that some people don't want to learn how to use IRC and pretend to force on others to learn how to use $ANY_OTHER_TOOL is getting more annoying by the day22:45
*** thorst has quit IRC22:45
reedbad channel to rant about that, sorry22:46
lifelesshmm, its not glued in for tempest, odd.22:46
armaxbut I notice that the subunit file has a lot more information than the html one22:46
lifelessarmax: sure yes. So -22:46
reedI came here to ask about shade though :)22:46
reedmordred, does shade have its own graphic logo or something to identify it visually?22:46
lifelessarmax: its a subunit V1 file currently. so: gunzip -c < testrepository.subunit.gz | subunit1to2 | subunit2html22:46
lifelessarmax: or whatever filter you want to use22:47
lifelessarmax: the subunit package has a number, or you can import it into your local testrepository::22:47
lifelessgunzip -c < testrepository.subunit.gz | subunit1to2 | testr load22:47
armaxlifeless: ok let me give it a shot22:47
openstackgerritsebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation  https://review.openstack.org/19845822:47
lifelessarmax: 'human readable' formats - there are so many - junitxml might be useful for you, or csv22:48
lifelessarmax: depends what sort of querying you want to do really22:48
armaxlifeless: plain text would be fine22:49
armaxlifeless: I am mainly interested in the test logging output22:49
armaxlifeless: and I see that the html version doesn’t have it all22:49
*** Rockyg has quit IRC22:49
clarkbarmax I just open the subunitv1 files with my text editor22:50
*** rlandy has quit IRC22:50
armaxclarkb: there’s binary chars in the ones I am looking at22:50
*** derekh has quit IRC22:50
clarkbthats v2 then run it through subunit2to122:51
clarkbthe output should be all text with v122:51
*** shashank_hegde has quit IRC22:51
jheskethMorning22:52
armaxclarkb: that’ll do I think22:52
armaxclarkb, lifeless: thanks22:52
*** ihrachys has quit IRC22:53
*** shashank_hegde has joined #openstack-infra22:53
*** ayoung has quit IRC22:53
lifelessclarkb: v1 isn't text22:54
lifelessclarkb: it only looks like it22:54
*** dimtruck is now known as zz_dimtruck22:54
lifelessclarkb: because its framing is somewhat human readable. Chunk delimiters and the embedded content are 8-bit22:54
*** mriedem has joined #openstack-infra22:55
lifelessarmax: in subunit's git repo there is a new filter22:55
lifelessarmax: subunit2disk22:55
lifelessarmax: it will explode all the attachments onto disk so you can look at them with a text editor22:55
armaxlifeless: that’s good to know I’ll give it a shot22:55
mriedemclarkb: dansmith: so what did you guys come up with for the novaclient thing?22:56
dansmithmriedem: when we detect the situation,22:56
dansmithmriedem: we print a banner to the console with the home phone number of the CEO for the cloud being contacted22:56
mriedemdansmith: can it flash?22:57
*** asettle has joined #openstack-infra22:57
dansmithit *must* flash22:58
clarkband scroll22:58
dansmithflashing marquis22:58
mtreinishdansmith: shell out to sl and print it as a banner on the trains22:59
dansmith....Your cloud is broken ...... let the CEO know ..... CALL NOW! ..... 777-555-1212 .....22:59
dansmithheh22:59
*** mtanino_ has joined #openstack-infra22:59
openstackgerritMerged openstack/requirements: Add django-formtools to global requirements  https://review.openstack.org/21922222:59
rm_worklol23:01
rm_workwell, once again I will plug https://review.openstack.org/#/c/217330/ if anyone has a moment, would love to get those jobs running ASAP23:01
*** mtanino has quit IRC23:02
rm_worktwo new nonvoting jobs23:02
rm_workcoming from one that was experimental and already passing23:02
rm_worksince I guess people will be busy with this novaclient thing for ... an "indeterminate" amount of time, heh23:03
*** mriedem has quit IRC23:03
*** mriedem has joined #openstack-infra23:04
*** SumitNaiksatam has quit IRC23:09
*** ayoung has joined #openstack-infra23:09
*** angela-s has quit IRC23:10
*** SumitNaiksatam has joined #openstack-infra23:10
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for missing subunit2html bug 1491646  https://review.openstack.org/21990923:11
openstackbug 1491646 in OpenStack-Gate ".tox/functional/bin/python: can't open file '/usr/local/jenkins/slave_scripts/subunit2html.py': [Errno 2] No such file or directory" [Undecided,New] https://launchpad.net/bugs/149164623:11
*** achanda_ has quit IRC23:12
*** wenlock_ has quit IRC23:13
*** Sukhdev_ has quit IRC23:13
*** baoli has quit IRC23:14
boris-42clarkb: please could you take a look as well at the patch: https://review.openstack.org/#/c/219823/ Rally gates are really badly broken23:14
*** sdake has quit IRC23:15
*** wenlock has quit IRC23:15
*** gildub has joined #openstack-infra23:15
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add bifrost manifest  https://review.openstack.org/20875123:16
boris-42clarkb: I mean all dsvm jobs test Rally master instead of propsed patch23:16
*** sputnik1_ has joined #openstack-infra23:16
*** shashank_hegde has quit IRC23:17
*** Piet has joined #openstack-infra23:17
openstackgerritColleen Murphy proposed openstack-infra/system-config: First stab at making things bifrosty  https://review.openstack.org/20875123:17
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add openstack-modules.env  https://review.openstack.org/20961723:17
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add bifrost inventory management procedures  https://review.openstack.org/21427923:17
*** shashank_hegde has joined #openstack-infra23:18
*** b10n1k_ has quit IRC23:19
*** erhudy has quit IRC23:19
*** b10n1k_ has joined #openstack-infra23:19
*** bknudson has left #openstack-infra23:20
*** sputnik13 has quit IRC23:20
*** bknudson has joined #openstack-infra23:20
*** baoli has joined #openstack-infra23:20
*** achanda has joined #openstack-infra23:21
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add bifrost inventory management procedures  https://review.openstack.org/21427923:21
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add bifrost manifest  https://review.openstack.org/20875123:23
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add bifrost inventory management procedures  https://review.openstack.org/21427923:23
crinkle^ painful rebase23:24
*** sharikapongubala has quit IRC23:24
*** baoli has quit IRC23:25
*** pfallenop has quit IRC23:26
*** sdake has joined #openstack-infra23:27
*** pfallenop has joined #openstack-infra23:27
*** yamamoto has quit IRC23:31
*** dannywilson has quit IRC23:31
notmynamewhere could I go to read about or learn how infra is doing the logstash and swift stuff? ie you're storing logs in swift and also indexing it with ELK. I'd like to learn more about that plubming23:31
notmyname(actually i want to hand those docs to someone else ;-)23:32
notmynamejhesketh: ^23:32
*** wshao has joined #openstack-infra23:34
*** krotscheck is now known as kro_paternity23:34
*** wshao has quit IRC23:35
*** bhunter71 has joined #openstack-infra23:35
nibalizernotmyname: here is a start http://docs.openstack.org/infra/system-config/logstash.html23:38
nibalizerafaik swift and logstash don't really interact23:39
notmynameah, ok23:39
notmynamethanks23:39
*** baoli has joined #openstack-infra23:40
fungiyeah, we have a "proxy" in front of swift (a nova instance running apache) and the logstash workers just download the files they want through that apache server23:40
fungiin theory we could probably find a way to hand off the direct swift urls to the logstash worker layer23:41
fungiexcept today we're in a split transitional situation where some logs are in swift and some are on the filesystem of the logserver (that apache server)23:42
*** harlowja has quit IRC23:42
*** harlowja has joined #openstack-infra23:43
*** morgabra has quit IRC23:45
*** ZZelle_ has quit IRC23:45
*** morgabra has joined #openstack-infra23:45
*** markvoelker has joined #openstack-infra23:46
jrollclarkb: supernova is just a wrapper around nova that deals with env variables23:46
jrollclarkb: so yes, novaclient works with onmetal, do a flavor-list / image-list | grep OnMetal for flavors and images23:47
nibalizerdownstream we run without swift, and the architecture is the same23:47
jheskethnotmyname: sorry was in a meeting.. but yes, that's the link I would have sent23:47
jheskethand yes, swift and logstash don't interact23:48
fungiyou know, just like that loverboy song... "pig and elephant dna just won't splice"23:49
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix exception lists in functional tests  https://review.openstack.org/21965523:49
Kennanhi fungi:23:51
Kennancould you help one question?23:51
Kennanwe added  publish docs jobs in project-config23:52
*** achanda has quit IRC23:52
Kennanbut we did notice openstack.org site have such documents23:52
Kennandid you know about it ? why23:52
*** sergmelikyan has quit IRC23:52
*** wshao has joined #openstack-infra23:53
Kennanclarkb: or lifeless: do you know about it ?23:53
sc68calfungi: lol wow that's a reference that takes me back23:54
*** ajmiller has quit IRC23:56
lifelessKennan: I believe fungi answered your question about that about 12 hours back23:56
Kennanlifeless: maybe IRC was flooded with many message, I did not notice, OK, let me check IRC logs on yesterday23:57
*** dangers is now known as dangers_away23:57
openstackgerritMerged openstack-infra/puppet-jenkins: Bump puppetlabs-apt to 2.1.0  https://review.openstack.org/21577923:57
*** wshao has quit IRC23:57
*** ybathia has quit IRC23:57
*** markvoelker has quit IRC23:58
*** shashank_hegde has quit IRC23:58
*** olaph has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!