Tuesday, 2016-10-04

*** amotoki has quit IRC00:00
*** pvaneck has quit IRC00:01
*** amotoki has joined #openstack-infra00:01
*** lezbar has quit IRC00:03
*** spzala has quit IRC00:03
*** lezbar has joined #openstack-infra00:04
openstackgerritDan Kolb proposed openstack-infra/jenkins-job-builder: corrected minor mispelling  https://review.openstack.org/38134500:04
*** amotoki has quit IRC00:05
*** number80 has quit IRC00:08
*** ijw has joined #openstack-infra00:08
*** kaisers has quit IRC00:08
*** kaisers has joined #openstack-infra00:09
*** kaisers2 has joined #openstack-infra00:09
*** kaisers1 has quit IRC00:09
*** amotoki has joined #openstack-infra00:13
*** ijw has quit IRC00:13
*** verdurin_ has joined #openstack-infra00:14
*** sflanigan_afk has joined #openstack-infra00:16
*** verdurin has quit IRC00:17
*** thcipriani is now known as thcipriani|afk00:18
tonybCan I get an infra-core to have a quick look at: https://review.openstack.org/#/q/is:open+project:openstack-infra/irc-meetings+owner:tony%2540bakeyournoodle.com+status:open+label:Code-Review%253E%253D2 rather then self-approve ?00:18
jheskethtonyb: I'll take al ook00:20
tonybjhesketh: Thanks.00:21
*** Apoorva has quit IRC00:23
jheskethtonyb: lgtm00:23
tonybjhesketh: Thanks00:23
openstackgerritMerged openstack-infra/irc-meetings: I18n: change time and chair information  https://review.openstack.org/38099800:24
openstackgerritMerged openstack-infra/irc-meetings: Update meeting chair for Nova Bugs Team  https://review.openstack.org/38131000:24
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Add translation jobs to neutron-lbaas-dashboard  https://review.openstack.org/38134700:24
openstackgerritMerged openstack-infra/irc-meetings: Expand the valid characters in chair names  https://review.openstack.org/37991200:26
*** ijw has joined #openstack-infra00:26
*** ijw has quit IRC00:27
*** ijw_ has joined #openstack-infra00:28
*** tuannguyen has joined #openstack-infra00:29
*** ijw_ has quit IRC00:29
*** markvoelker has joined #openstack-infra00:29
*** adrian_otto has quit IRC00:29
*** thorst has quit IRC00:32
*** asselin__ has quit IRC00:33
*** thorst has joined #openstack-infra00:33
*** tuannguyen has quit IRC00:34
*** markvoelker has quit IRC00:34
*** gouthamr has joined #openstack-infra00:38
*** mtanino has joined #openstack-infra00:40
*** thorst has quit IRC00:42
*** Rockyg has quit IRC00:44
pabelangerclarkb: More info, looks like this issue is limited to providers that are doing DHCP00:44
armaxclarkb: ping00:44
armaxsorry it took me so long to get back to you00:44
pabelangerclarkb: which makes sense, since I am only seeing failures in rackspace and ovh right now00:45
armaxposted a comment on https://bugs.launchpad.net/devstack/+bug/1629133/comments/900:45
openstackLaunchpad bug 1629133 in devstack "New neutron subnet pool support breaks multinode testing." [Critical,In progress] - Assigned to Dr. Jens Rosenboom (j-rosenboom-j)00:45
clarkbpabelanger: huh00:45
clarkbarmax ok00:45
pabelangerclarkb: it looks like SSH is started before dhclient has finished being assigned an IP address, then /etc/network/if-up.d/openssh-server is restarting sshd00:45
pabelangeronce the IP is assigned00:46
*** tqtran has quit IRC00:46
pabelangerhowever, I am able to establish a connection, because I get the banner failure00:46
*** xarses has joined #openstack-infra00:46
pabelangershowing the restart of SSHd00:47
*** baoli has joined #openstack-infra00:47
*** amitgandhinz has joined #openstack-infra00:48
*** baoli_ has joined #openstack-infra00:49
openstackDebian bug 502444 in openssh-server "sshd fails at boot-time following reload by /etc/network/if-up.d/openssh-server presumably due to race condition" [Grave,Fixed]00:50
pabelangeroriginally reloaded sshd00:50
*** ijw has joined #openstack-infra00:50
pabelangerbut there was race conditions00:50
pabelangerso, they changed it to a restart00:50
*** adrian_otto has joined #openstack-infra00:50
armaxclarkb: mind you that was the initial value I had set for the subnet pool :)00:51
*** adrian_otto has quit IRC00:52
*** baoli has quit IRC00:52
clarkbarmax: the problem is while they may be different this is a backward inconpatible change with no docs and no signaling how to fix it00:52
armaxyou mean your change?00:53
clarkbin that situation you should use the existing method for avoiding subnet range conflicts00:53
clarkbno the change yo devstack00:53
armaxoh, you mean https://review.openstack.org/#/c/282559/00:53
*** adrian_otto has joined #openstack-infra00:53
clarkbmy change us an attemot to pick the pieces up by using existing information available in drvstack to make a better choice at pool range00:53
armaxor strictly speaking assaf's?00:54
*** markvoelker has joined #openstack-infra00:54
*** ijw has quit IRC00:54
clarkbwhatever change to devstack set that range with new variables00:54
armaxfwiw I’d rather revert change https://review.openstack.org/#/c/28328900:54
clarkbthat is also fine by me00:55
armaxI am not sure I understand why you think the change is bw incompat00:55
*** adrian_otto has quit IRC00:56
armaxa user can still create his/her own networks without having subnet pools interfere00:56
armaxhow does the multinode breakage happen?00:56
clarkbits not backward compat because it is on by default and breaks anyone using 10/800:57
clarkbthr breakage happens because the change takes over 10/8 and applies a route for it on br-ex00:57
clarkbmeaning you cannot use this change and 10/8 unless you set q bunch of new config00:57
clarkbthats not backward xompat00:57
*** tiswanso has joined #openstack-infra00:57
*** adrian_otto has joined #openstack-infra00:58
*** rfolco has quit IRC00:58
*** zz_dimtruck is now known as dimtruck00:58
clarkbanother alternative is to not set a route for 10/800:59
*** spzala has joined #openstack-infra00:59
*** adrian_otto has quit IRC00:59
*** amotoki has quit IRC00:59
armaxI suppose the route is installed during the tempest run?01:00
armaxnot at setup time, right?01:00
*** amotoki has joined #openstack-infra01:00
*** purp has quit IRC01:00
armaxI mean, this happens as soon as the tests that make use of subnetpools get executed01:00
clarkbIts in devstack iirc01:00
pabelangerclarkb: I'm going to try a custom upstart script in the morning for openssh-server, I think we could do something like: start on runlevel [2345] and net-device-up IFACE!=lo for ssh.conf, which means SSH would only start once any other interface is only01:00
clarkbbut I didnt dig into confirm01:00
armaxbecause up until that point, I can’t see how that can happen01:01
armaxeither way I agree that /8 is a poor choice01:01
*** claudiub has quit IRC01:01
clarkbarmax: if it does happen as a side effect of tempest then neutron should stop doing that and only set routes for the actual subnets used not the entire pool01:01
armaxI’ll have to check with carl_baldwin about that because he’s the mastermind behind subnetpools01:02
armaxbut as a stopgap we should do something right now01:02
*** amotoki has quit IRC01:04
*** spzala has quit IRC01:04
mordredI agree with all suggestoins that are correct01:04
mordredbut let me go on record saying that I think all the incorrect suggestions for fixing this are wrong01:05
armaxmordred: my suggestions are always sensible01:05
armaxif that’s what you’re implying :)01:05
armaxclarkb: I’ll talk to carl_baldwin tonight as funnily enough he’s in town01:07
armaxclarkb: if you can point me out to how this fails in the gate, that would help me understand deeper what’s going on01:07
*** amotoki has joined #openstack-infra01:08
armaxclarkb: as for https://review.openstack.org/#/c/379543/ I would personally hardcode new values (my original ones) rather than rely on the FIXED_RANGE_*, but I am not strongly opinionated on it01:09
*** baoli_ has quit IRC01:09
*** baoli has joined #openstack-infra01:09
clarkbarmax: the overlsy for multinode doesnt work because it runs over 10/8 on eg eth1 not br-ex01:09
clarkbjust look at any neutron multinode job on osic01:10
armaxclarkb: how can I tell?01:10
armaxI mean which one is a neutron multinode job on osic01:10
clarkbyou can ask logstash, its also recorded at the beginning of the console log01:11
clarkblet me find anexample log01:12
*** amitgandhinz has quit IRC01:12
armaxclarkb: don’t worry, I’ll dig one myself01:12
armaxI need to dash now01:12
*** tuanluong has joined #openstack-infra01:13
*** adrian_otto has joined #openstack-infra01:13
clarkbhttp://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/7f82862/console.html#_2016-10-03_21_14_28_763192 is one way it fails01:13
clarkbbasically ansible cant sshto subnodes because broken routes01:13
clarkbhave alsoseen grenade and other jobs fail in other ways01:14
* mordred apologizes for incoming patch bomb01:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Make sure we're matching image status properly  https://review.openstack.org/37716401:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Normalize images  https://review.openstack.org/37716301:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Add helper properties to generate location info  https://review.openstack.org/38133901:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Start splitting normalize functions into a mixin  https://review.openstack.org/38135201:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Add location to server record  https://review.openstack.org/38135301:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Move image normalize calls to _normalize  https://review.openstack.org/38135401:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Move normalize_flavors to _normalize  https://review.openstack.org/38135501:15
openstackgerritMonty Taylor proposed openstack-infra/shade: Add location field to flavors  https://review.openstack.org/38135601:15
armaxclarkb: ack01:15
armaxI’ll follow up01:16
*** adrian_otto has quit IRC01:16
*** adrian_otto has joined #openstack-infra01:16
armaxclarkb: as I said, because of the severity of the issue I am happy to see the default values change01:16
armaxclarkb: I’ll talk to carl_baldwin to understand better how subnetpools fiddle with the host routes and get back to you01:17
cloudnullarmax: I'd be happy to kick a couple of tests nodes from within the osic if you need.01:17
openstackgerritTony Breeds proposed openstack-infra/project-config: Add non-voting python3 jobs to requirements  https://review.openstack.org/38135701:18
*** thorst has joined #openstack-infra01:21
*** thorst has quit IRC01:21
*** thorst has joined #openstack-infra01:22
*** adrian_otto has quit IRC01:22
*** amotoki has quit IRC01:24
*** amotoki has joined #openstack-infra01:24
*** adrian_otto has joined #openstack-infra01:26
*** adrian_otto has quit IRC01:27
*** number80 has joined #openstack-infra01:27
openstackgerritBrian Haley proposed openstack-infra/devstack-gate: Support an IPv6 underlay network  https://review.openstack.org/34304101:28
*** amotoki has quit IRC01:29
*** adrian_otto has joined #openstack-infra01:29
*** thorst has quit IRC01:30
*** adrian_otto has quit IRC01:32
*** kaisers_ has joined #openstack-infra01:38
*** tuannguyen has joined #openstack-infra01:40
*** rfolco has joined #openstack-infra01:40
*** esikachev has joined #openstack-infra01:41
*** kaisers_ has quit IRC01:42
*** esikachev has quit IRC01:46
*** thorst has joined #openstack-infra01:47
*** adrian_otto has joined #openstack-infra01:48
*** thorst has quit IRC01:48
*** tphummel has quit IRC01:49
*** adrian_otto has quit IRC01:51
*** adrian_otto has joined #openstack-infra01:53
*** tuannguyen has quit IRC01:54
*** spzala has joined #openstack-infra01:54
*** adrian_otto has quit IRC01:55
*** tuannguyen has joined #openstack-infra01:56
*** spzala has quit IRC01:59
*** amotoki has joined #openstack-infra01:59
*** EricGonczer_ has joined #openstack-infra02:03
*** Swami__ has joined #openstack-infra02:04
*** Swami_ has quit IRC02:08
*** amitgandhinz has joined #openstack-infra02:09
*** timello has quit IRC02:09
*** spzala has joined #openstack-infra02:10
*** timello has joined #openstack-infra02:13
*** timello has quit IRC02:19
mordredarmax, cloudnull: is this related: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html#_2016-10-04_01_59_52_598273 ?02:23
mordredthe order of the fixed ips returned seems to have changed, and it was stable before02:24
mordredit's also possible it was 'stable' accidentally and I should not just do an assertEqual there ...02:24
*** david-lyle has joined #openstack-infra02:25
*** jamesdenton has joined #openstack-infra02:25
*** rfolco has quit IRC02:25
*** timello has joined #openstack-infra02:26
openstackgerritBrian Haley proposed openstack-infra/devstack-gate: Support an IPv6 underlay network  https://review.openstack.org/34304102:26
cloudnullmordred: i dont think so .02:31
cloudnullwithin the OSIC the subnet order should be consistent.02:31
cloudnullin the rax cloud they flip flow02:32
*** sputnik13 has joined #openstack-infra02:32
mordredcloudnull: hrm - those should be devstack subnets though, not underlying cloud subnets02:33
cloudnullin OSIC cloud I believe the ordering is 10/8, ::/64, fe8002:33
cloudnulland I believe the assignment of the 10/8 within the devstack subnets is the culperit causing multi-node gate brokeness.02:34
mordred(I'm thinking I should just make that test more resilient)02:34
clarkbyes and its breaking a bunch if other random.stuff02:34
*** rossella_s has quit IRC02:34
cloudnullthough I could just be tired and sick, and sick and tired and plain wrong.02:34
*** rossella_s has joined #openstack-infra02:35
cloudnullso I got that going for me, which is nice :)02:35
mordredcloudnull: at least you have something going for you02:35
*** hongbin has joined #openstack-infra02:35
clarkbyou can depends on the workarou d change to see if that fixes it02:35
*** esberglu has quit IRC02:35
mordredI think I already am depends-on that change :)02:35
cloudnullmordred:  this is true.02:36
* cloudnull drinks more more of that half full glass. :)02:36
mordredoh - wait - no, I'm not - that was a nodepool change I made depends-on the workaround02:36
mordredI think02:36
*** yamahata has quit IRC02:36
cloudnulls/more more/some more/02:36
EmilienMclarkb: can you confirm that experimental jobs are non-voting anyway (without -nv thing) ?02:38
*** david-lyle has quit IRC02:38
jrollEmilienM: they are02:39
mordredEmilienM: yah. if they're in the experimental pipeline, they only report back results, they don't vote02:39
EmilienMthanks :-)02:39
EmilienMwhat do you think of this kind of change? https://review.openstack.org/#/c/381322/02:40
*** amitgandhinz has quit IRC02:43
*** tqtran has joined #openstack-infra02:43
*** tqtran has quit IRC02:48
*** thorst has joined #openstack-infra02:48
*** david-lyle has joined #openstack-infra02:49
*** dimtruck is now known as zz_dimtruck02:52
*** zz_dimtruck is now known as dimtruck02:52
*** david-lyle has quit IRC02:55
*** amotoki_ has joined #openstack-infra02:55
*** sputnik13 has quit IRC02:57
*** thorst has quit IRC02:57
*** sputnik13 has joined #openstack-infra02:57
openstackgerritJonathan Como proposed openstack-infra/jenkins-job-builder: Implement a publisher for HockeyApp  https://review.openstack.org/37117802:58
*** amotoki has quit IRC02:58
*** caowei has joined #openstack-infra03:00
*** caowei has quit IRC03:01
*** dimtruck is now known as zz_dimtruck03:02
*** tqtran has joined #openstack-infra03:02
openstackgerritJonathan Como proposed openstack-infra/jenkins-job-builder: Fix reference repo configuration for Git SCM  https://review.openstack.org/37118103:03
*** tuannguyen has quit IRC03:05
openstackgerritBrian Haley proposed openstack-infra/devstack-gate: Support an IPv6 underlay network  https://review.openstack.org/34304103:06
*** tqtran has quit IRC03:06
*** ijw has joined #openstack-infra03:10
*** links has joined #openstack-infra03:10
*** ijw_ has joined #openstack-infra03:10
*** 7F1ACHYUT has joined #openstack-infra03:12
*** ijw has quit IRC03:14
*** Swami__ has quit IRC03:14
*** sflanigan_afk is now known as sflanigan03:15
*** hongbin has quit IRC03:19
*** baoli has quit IRC03:20
*** kaisers_ has joined #openstack-infra03:27
*** sputnik13 has quit IRC03:27
*** sputnik13 has joined #openstack-infra03:27
openstackgerritJonathan Como proposed openstack-infra/jenkins-job-builder: Implement a publisher for HockeyApp  https://review.openstack.org/37117803:27
*** spzala has quit IRC03:27
*** kaisers_ has quit IRC03:32
*** kzaitsev_mb has quit IRC03:33
*** EricGonczer_ has quit IRC03:38
*** amitgandhinz has joined #openstack-infra03:39
*** jamesdenton has quit IRC03:42
*** tiswanso has quit IRC03:45
*** 7F1ACHYUT has quit IRC03:52
sdakehey folks, in kolla we are splitting repos of kolla into a couple of repos03:53
sdakei am curiosu about the upstream: flag - wil lthat copy the branches intact?03:53
sdake(I dont want the branches copied intact in some cases in others do)03:53
*** vikrant has joined #openstack-infra03:54
clarkbit will so prep thr branches the way you want them to turn out03:54
clarkbtags too03:54
*** thorst has joined #openstack-infra03:55
*** yuanying has quit IRC04:01
*** yuanying_ has joined #openstack-infra04:01
*** thorst has quit IRC04:02
*** mtanino has quit IRC04:03
*** yamamoto_ has joined #openstack-infra04:11
*** lezbar has quit IRC04:13
*** amitgandhinz has quit IRC04:14
*** lezbar has joined #openstack-infra04:14
*** amotoki has joined #openstack-infra04:18
*** amotoki_ has quit IRC04:21
*** salv-orlando has joined #openstack-infra04:29
*** salv-orlando has quit IRC04:34
*** adriant has quit IRC04:35
*** senk__ has joined #openstack-infra04:39
*** senk__ has quit IRC04:39
*** sputnik13 has quit IRC04:40
*** woodster_ has quit IRC04:40
*** Sukhdev has joined #openstack-infra04:41
*** pgadiya has joined #openstack-infra04:43
*** rbrndt has joined #openstack-infra04:44
*** sputnik13 has joined #openstack-infra04:46
*** kzaitsev_mb has joined #openstack-infra04:49
*** lezbar has quit IRC04:50
*** lezbar has joined #openstack-infra04:50
*** narayrak has joined #openstack-infra04:52
*** tuannguyen has joined #openstack-infra04:52
*** tuannguyen has quit IRC04:58
*** rwsu has quit IRC04:59
*** sputnik13 has quit IRC05:01
*** thorst has joined #openstack-infra05:01
*** Ravikiran_K has joined #openstack-infra05:05
*** salv-orlando has joined #openstack-infra05:07
*** thorst has quit IRC05:08
*** amitgandhinz has joined #openstack-infra05:10
*** kaisers_ has joined #openstack-infra05:14
*** spzala has joined #openstack-infra05:28
*** rbrndt has quit IRC05:32
*** spzala has quit IRC05:32
*** Sukhdev has quit IRC05:35
*** sputnik13 has joined #openstack-infra05:38
*** sandanar has joined #openstack-infra05:40
*** nadya has joined #openstack-infra05:42
*** esikachev has joined #openstack-infra05:42
*** amitgandhinz has quit IRC05:45
*** jaosorior has joined #openstack-infra05:46
*** esikachev has quit IRC05:47
*** tnovacik has joined #openstack-infra05:48
*** flaper87 has joined #openstack-infra05:49
*** flaper87 has quit IRC05:49
*** flaper87 has joined #openstack-infra05:49
*** armax has quit IRC05:49
*** ArxCruz has joined #openstack-infra05:51
*** e0ne has joined #openstack-infra05:52
*** ArxCruz has quit IRC05:52
*** ArxCruz has joined #openstack-infra05:53
*** adrian_otto has joined #openstack-infra05:53
*** ArxCruz has quit IRC05:54
*** ArxCruz has joined #openstack-infra05:54
*** ArxCruz has quit IRC05:54
*** ArxCruz has joined #openstack-infra05:55
*** tuannguyen has joined #openstack-infra05:55
*** ArxCruz has quit IRC05:55
*** ArxCruz has joined #openstack-infra05:55
*** ArxCruz has quit IRC05:56
*** ArxCruz has joined #openstack-infra05:56
*** ArxCruz has quit IRC05:57
*** ArxCruz has joined #openstack-infra05:57
*** ArxCruz has quit IRC05:58
*** ArxCruz has joined #openstack-infra05:58
*** ArxCruz has quit IRC05:58
*** ArxCruz has joined #openstack-infra05:59
*** yamahata has joined #openstack-infra05:59
*** ArxCruz has quit IRC05:59
*** tuannguyen has quit IRC05:59
*** ArxCruz has joined #openstack-infra06:00
*** john-davidge has joined #openstack-infra06:00
*** ArxCruz has quit IRC06:00
*** ArxCruz has joined #openstack-infra06:00
*** ArxCruz has quit IRC06:01
*** ArxCruz has joined #openstack-infra06:01
*** ArxCruz has quit IRC06:02
*** ArxCruz has joined #openstack-infra06:02
*** ArxCruz has quit IRC06:02
*** ArxCruz has joined #openstack-infra06:03
*** ArxCruz has quit IRC06:03
*** drifterza has quit IRC06:03
*** ArxCruz has joined #openstack-infra06:04
*** ArxCruz has quit IRC06:04
*** ArxCruz has joined #openstack-infra06:04
*** thorst has joined #openstack-infra06:05
*** ArxCruz has quit IRC06:05
*** john-davidge has quit IRC06:05
*** ArxCruz has joined #openstack-infra06:05
*** tqtran has joined #openstack-infra06:05
*** ArxCruz has quit IRC06:06
*** watersoul has quit IRC06:06
openstackgerritTony Breeds proposed openstack-infra/project-config: Add linters to the governance gate  https://review.openstack.org/38142506:06
*** ArxCruz has joined #openstack-infra06:06
*** watersoul has joined #openstack-infra06:06
*** ArxCruz has quit IRC06:07
*** ArxCruz has joined #openstack-infra06:07
*** ArxCruz has quit IRC06:07
*** nadya has quit IRC06:08
*** ArxCruz has joined #openstack-infra06:08
*** ArxCruz has quit IRC06:08
*** ArxCruz has joined #openstack-infra06:08
*** armax has joined #openstack-infra06:09
*** ArxCruz has quit IRC06:09
*** tqtran has quit IRC06:10
*** thorst has quit IRC06:12
*** armax has quit IRC06:12
*** armax has joined #openstack-infra06:12
*** armax has quit IRC06:13
*** Jeffrey4l__ has quit IRC06:14
*** Jeffrey4l__ has joined #openstack-infra06:14
*** Jeffrey4l__ has quit IRC06:14
*** Jeffrey4l_ has joined #openstack-infra06:15
*** Jeffrey4l_ is now known as Jeffrey4l06:15
*** winggundamth has quit IRC06:17
*** sputnik13 has quit IRC06:20
*** salv-orlando has quit IRC06:21
*** esikachev has joined #openstack-infra06:22
*** AJaeger has quit IRC06:23
*** esikachev has quit IRC06:26
*** esikachev has joined #openstack-infra06:29
*** Ravikiran_K has quit IRC06:29
*** tqtran has joined #openstack-infra06:31
*** nadya has joined #openstack-infra06:33
*** esikachev has quit IRC06:34
*** tqtran has quit IRC06:35
sdakeclarkb have to hit rack06:36
sdakeclarkb can we chat in morning06:36
sdakeclarkb i didn't quite parse your msg - been up since 4am working with contractors06:36
sdakeand its 1106:36
sdakethat could be part of the problem :)06:36
openstackgerritMerged openstack-infra/project-config: Shorten OSA role job names  https://review.openstack.org/38118506:38
*** tqtran has joined #openstack-infra06:38
*** AJaeger has joined #openstack-infra06:39
AJaegerjeblair: we try already publishing of developer.openstack.org document to AFS - a side effect of adding the afs publisher to some macros. Please check http://logs.openstack.org/e6/e6e7d4a523fef49cb44efba6e1763b2dfcc44d2b/post/ironic-api-ref/a40b53f/_zuul_ansible/ansible_log.txt at bottom.06:40
*** amitgandhinz has joined #openstack-infra06:41
AJaegerjeblair: do you want to enable that directory for developer.o.o as well, ignore this for now - or should we fix the macro so that this only works with docs.o.o?06:41
*** lezbar has quit IRC06:42
*** lezbar has joined #openstack-infra06:42
*** e0ne has quit IRC06:43
*** amotoki has quit IRC06:43
*** andreas_s has joined #openstack-infra06:43
*** rcernin has joined #openstack-infra06:43
*** e0ne has joined #openstack-infra06:44
*** winggundamth has joined #openstack-infra06:44
*** nadya has quit IRC06:45
*** adrian_otto has quit IRC06:45
*** ccamacho has joined #openstack-infra06:47
*** chem has joined #openstack-infra06:48
*** adrian_otto has joined #openstack-infra06:48
*** amotoki has joined #openstack-infra06:49
*** rlu has joined #openstack-infra06:49
*** pilgrimstack has quit IRC06:50
*** tesseract- has joined #openstack-infra06:52
*** pilgrimstack has joined #openstack-infra06:53
*** e0ne has quit IRC06:55
*** rlu has quit IRC06:56
*** yamahata has quit IRC06:56
*** nadya has joined #openstack-infra07:00
*** rlu has joined #openstack-infra07:02
*** senk_ has joined #openstack-infra07:04
*** kzaitsev_mb has quit IRC07:04
*** senk_ has quit IRC07:06
*** esikachev has joined #openstack-infra07:08
*** thorst has joined #openstack-infra07:10
*** salv-orlando has joined #openstack-infra07:11
*** amitgandhinz has quit IRC07:14
*** matrohon has joined #openstack-infra07:14
*** dragonmaster_ has joined #openstack-infra07:15
*** adrian_otto1 has joined #openstack-infra07:16
*** yaume has joined #openstack-infra07:16
*** thorst has quit IRC07:16
*** adrian_otto has quit IRC07:19
*** markvoelker has quit IRC07:24
*** amotoki_ has joined #openstack-infra07:24
*** dtantsur|afk is now known as dtantsur07:26
*** Naeil has joined #openstack-infra07:27
*** amotoki has quit IRC07:27
*** ihrachys has joined #openstack-infra07:28
*** salv-orlando has quit IRC07:33
*** Naeil has quit IRC07:33
*** pilgrimstack1 has joined #openstack-infra07:34
*** pilgrimstack has quit IRC07:34
*** adrian_otto1 has quit IRC07:37
*** hashar has joined #openstack-infra07:37
*** adrian_otto has joined #openstack-infra07:38
*** Adri2000 has quit IRC07:42
*** Adri2000 has joined #openstack-infra07:43
*** drifterza has joined #openstack-infra07:44
*** jpich has joined #openstack-infra07:44
*** vincentll has joined #openstack-infra07:45
*** salv-orlando has joined #openstack-infra07:46
*** adrian_otto has quit IRC07:46
vincentllIs there an issue with Jenkins at the moment? Our patchset is stuck on "Needs verified" eventhough Jenkins gave a +1/verified (https://review.openstack.org/#/c/381092/)07:47
*** adrian_otto has joined #openstack-infra07:49
*** jak-jak has joined #openstack-infra07:49
*** Naeil has joined #openstack-infra07:49
*** jak-jak is now known as Guest4522007:49
*** adrian_otto has quit IRC07:50
*** jtomasek|afk is now known as jtomasek07:51
*** jpena|off is now known as jpena07:53
hasharvincentll: from   ssh -p 29418 review.openstack.org 'gerrit query 381092 --submit-records'07:55
hasharthe change shows as status: NOT_READY07:55
hasharVerified: NEED07:55
hasharseems to be a chain of dependencies07:56
hasharultimately landing on patchset 1 of https://review.openstack.org/#/c/380997/07:56
hasharwhich now has a patchset 4.  Maybe try to rebase?07:56
*** yamahata has joined #openstack-infra07:56
*** tuannguyen has joined #openstack-infra07:56
*** pilgrimstack has joined #openstack-infra07:57
*** jistr has joined #openstack-infra07:57
*** pilgrimstack1 has quit IRC07:57
hasharand https://review.openstack.org/#/c/380997/  got CR+2/W+1  on patchset 3, but a new patchset got sent which cleared the votes07:57
*** yamahata has quit IRC07:58
*** zzzeek has quit IRC08:00
*** jed56 has joined #openstack-infra08:00
*** zzzeek has joined #openstack-infra08:01
*** tuannguyen has quit IRC08:01
*** rwsu has joined #openstack-infra08:03
*** amoralej|off is now known as amoralej08:03
*** jordanP has joined #openstack-infra08:06
*** yamahata has joined #openstack-infra08:07
*** Ravikiran_K has joined #openstack-infra08:09
*** tqtran has quit IRC08:09
*** e0ne has joined #openstack-infra08:10
*** amitgandhinz has joined #openstack-infra08:11
*** salv-orlando has quit IRC08:12
*** yamahata has quit IRC08:13
*** yamahata has joined #openstack-infra08:13
*** thorst has joined #openstack-infra08:15
*** amotoki_ has quit IRC08:18
*** salv-orlando has joined #openstack-infra08:19
vincentllthanks hashar !08:19
*** amotoki has joined #openstack-infra08:21
*** salv-orlando has quit IRC08:21
*** salv-orlando has joined #openstack-infra08:21
*** thorst has quit IRC08:22
*** kzaitsev_mb has joined #openstack-infra08:24
*** markvoelker has joined #openstack-infra08:24
*** markvoelker has quit IRC08:29
*** tosky has joined #openstack-infra08:31
*** kzaitsev_mb has quit IRC08:36
*** yamahata has quit IRC08:37
*** admcleod has joined #openstack-infra08:38
*** admcleod has joined #openstack-infra08:38
*** admcleod_ has quit IRC08:39
*** kzaitsev_mb has joined #openstack-infra08:40
dragonmaster_morning all08:41
*** amitgandhinz has quit IRC08:45
yolandagood morning08:48
*** mhickey has joined #openstack-infra08:48
*** ijw_ has quit IRC08:49
*** dtantsur is now known as dtantsur|bbl08:52
*** dtardivel has joined #openstack-infra08:52
*** kzaitsev_mb has quit IRC08:53
*** dizquierdo has joined #openstack-infra08:54
dragonmaster_anyone that could possibly point me to a zuul setup / deployment doc? we’re trying to implement the openstack-infra gating solution for our internal projects08:54
*** john-davidge has joined #openstack-infra08:54
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/38155908:55
*** electrofelix has joined #openstack-infra09:00
AJaegerdragonmaster_: http://docs.openstack.org/infra/system-config/third_party.html - and there's a third party meetign every week that you should attend09:02
AJaegerdragonmaster_: see http://eavesdrop.openstack.org/#Third_Party_Working_Group_Meeting09:03
dragonmaster_cool, thanks AJaeger09:03
*** derekh has joined #openstack-infra09:04
*** strigazi_AFK is now known as strigazi09:04
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Disable building ramdisk for ironic-multitest job.  https://review.openstack.org/38157009:05
*** kzaitsev_mb has joined #openstack-infra09:06
*** verdurin_ is now known as verdurin09:07
AJaegerfungi, looking at deb-glare, the upstream repo points master to " remotes/origin/HEAD -> origin/debian/newton"09:11
AJaegerfungi, zigo, this is now the 3rd or fourth repository that failed to import with our tools and had this setup.09:11
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Move the opensuse mkinitrd script to the zypper element  https://review.openstack.org/38157409:11
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add zypper-minimal element  https://review.openstack.org/38157509:11
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element  https://review.openstack.org/38157609:11
AJaegerfungi, is there a bug in our import - or should we require that HEAD points to master?09:12
openstackgerritPawel Koniszewski proposed openstack-infra/project-config: Add new job to test live migration with grenade  https://review.openstack.org/36480909:12
openstackgerritMerged openstack-infra/project-config: Add non-voting python3 jobs to requirements  https://review.openstack.org/38135709:14
openstackgerritMerged openstack-infra/project-config: Update the gate-tempest-neutron-dsvm-src-oslo.messaging test  https://review.openstack.org/37150209:20
*** thorst has joined #openstack-infra09:20
openstackgerritMerged openstack-infra/project-config: layout: update tempest jobs coverage for puppet  https://review.openstack.org/38128509:20
*** sambetts|afk is now known as sambetts09:21
openstackgerritMerged openstack-infra/project-config: Create glare-specs project  https://review.openstack.org/37074909:26
*** thorst has quit IRC09:27
vsaienk0AJaeger, yolanda: please help to merge https://review.openstack.org/#/c/381570/ 1 line patch that disables building ramdisk for ironic-multitest experimental job.09:29
AJaegervsaienk0: I would appreciate dtantsur|bbl or jroll to +1 it.09:30
openstackgerritMerged openstack-infra/project-config: Add new project: devstack-plugin-libvirt-qemu  https://review.openstack.org/37792009:32
openstackgerritMerged openstack-infra/project-config: Kolla Kubernetes Ansible test  https://review.openstack.org/38084809:32
*** ijw has joined #openstack-infra09:35
*** lezbar has quit IRC09:36
*** lezbar has joined #openstack-infra09:36
*** bhavik1 has joined #openstack-infra09:39
*** ramishra has quit IRC09:39
*** ijw has quit IRC09:40
*** amitgandhinz has joined #openstack-infra09:41
*** ramishra has joined #openstack-infra09:42
*** shardy has joined #openstack-infra09:47
*** dtantsur|bbl is now known as dtantsur09:50
*** amotoki has quit IRC09:51
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add zypper-minimal element  https://review.openstack.org/38157509:57
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element  https://review.openstack.org/38157609:57
vsaienk0AJaeger, just got +1 from Dmitry https://review.openstack.org/#/c/381570/09:57
*** tuannguyen has joined #openstack-infra09:58
AJaegervsaienk0: then you have my +2 ;)09:59
*** jistr is now known as jistr|mtg09:59
vsaienk0AJaeger: thanks!09:59
*** zigo has quit IRC10:01
*** tuannguyen has quit IRC10:02
*** zigo has joined #openstack-infra10:04
*** zigo is now known as Guest8478010:05
*** amotoki has joined #openstack-infra10:07
*** dizquierdo has quit IRC10:11
*** senk_ has joined #openstack-infra10:11
*** senk_ has quit IRC10:12
*** Guest84780 has quit IRC10:14
*** senk_ has joined #openstack-infra10:15
*** amitgandhinz has quit IRC10:16
*** zigo_ has joined #openstack-infra10:16
*** senk_ has quit IRC10:21
*** _degorenko|afk is now known as degorenko10:22
*** thorst has joined #openstack-infra10:25
*** markvoelker has joined #openstack-infra10:26
*** yamamoto_ has quit IRC10:27
*** kzaitsev_mb has quit IRC10:27
*** vincentll has quit IRC10:27
*** number80 has quit IRC10:28
*** markvoelker has quit IRC10:31
*** hashar is now known as hasharFood10:31
*** thorst has quit IRC10:32
*** ijw has joined #openstack-infra10:36
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces  https://review.openstack.org/38014210:36
*** amotoki has quit IRC10:39
*** amotoki has joined #openstack-infra10:40
*** ijw has quit IRC10:42
*** salv-orlando has quit IRC10:42
*** spzala has joined #openstack-infra10:44
*** kaisers has quit IRC10:46
*** kaisers2 has quit IRC10:46
*** EricGonczer_ has joined #openstack-infra10:48
*** spzala has quit IRC10:49
*** bhavik1 has quit IRC10:50
*** jkilpatr has quit IRC10:52
*** jkilpatr has joined #openstack-infra10:53
*** ralonsoh has joined #openstack-infra10:53
*** psilvad has joined #openstack-infra10:54
ralonsohHi. I need to be added as member to devstack-plugin-libvirt-qemu-core and devstack-plugin-libvirt-qemu-release. My email is rodolfo.alonso.hernande@intel.com10:54
ralonsohSorry, my mail is rodolfo.alonso.hernandez@intel.com10:54
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Rework ironic-lib tempest jobs  https://review.openstack.org/38165210:55
dtantsurjroll, ^^^10:55
*** kzaitsev_mb has joined #openstack-infra10:57
*** rcernin has quit IRC10:57
*** rcernin has joined #openstack-infra10:58
*** psilvad has quit IRC11:00
*** jistr|mtg is now known as jistr11:00
*** dprince has joined #openstack-infra11:01
*** nadya has quit IRC11:01
*** coolsvap is now known as coolsvap_11:01
*** jkilpatr has quit IRC11:03
*** Naeil has quit IRC11:05
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add option to disable default routes  https://review.openstack.org/38166011:08
*** rtheis has joined #openstack-infra11:09
*** jpena is now known as jpena|lunch11:09
*** senk_ has joined #openstack-infra11:10
*** tpsilva has joined #openstack-infra11:11
*** amitgandhinz has joined #openstack-infra11:12
*** amoralej is now known as amoralej|lunch11:13
*** baoli has joined #openstack-infra11:14
*** ccamacho is now known as ccamacho|lunch11:14
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add zypper-minimal element  https://review.openstack.org/38157511:14
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element  https://review.openstack.org/38157611:14
*** baoli has quit IRC11:15
*** rhallisey has joined #openstack-infra11:15
*** sputnik13 has joined #openstack-infra11:18
*** Naeil has joined #openstack-infra11:18
*** jkilpatr has joined #openstack-infra11:19
*** kashyap has left #openstack-infra11:21
*** breton has joined #openstack-infra11:22
*** zigo_ is now known as zigo11:23
*** gildub has quit IRC11:24
*** sputnik13 has quit IRC11:25
*** EricGonczer_ has quit IRC11:26
*** sputnik13 has joined #openstack-infra11:27
*** Naeil has quit IRC11:27
*** markvoelker has joined #openstack-infra11:27
*** markvoelker has quit IRC11:32
*** mhickey has quit IRC11:32
*** ijw has joined #openstack-infra11:38
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Do Not Merge - Test Undercloud upgrade mitaka -> newton  https://review.openstack.org/38130911:38
*** Naeil has joined #openstack-infra11:39
*** thorst has joined #openstack-infra11:40
*** asselin__ has joined #openstack-infra11:40
*** EricGonczer_ has joined #openstack-infra11:40
*** breton has left #openstack-infra11:40
*** salv-orlando has joined #openstack-infra11:41
*** Goneri has quit IRC11:43
*** ijw has quit IRC11:44
*** ldnunes has joined #openstack-infra11:44
*** salv-orlando has quit IRC11:46
*** amitgandhinz has quit IRC11:46
*** rfolco has joined #openstack-infra11:47
*** esikachev has quit IRC11:48
*** ldnunes has quit IRC11:55
*** baoli has joined #openstack-infra11:55
*** nadya has joined #openstack-infra11:56
*** ldnunes has joined #openstack-infra11:57
*** amotoki has quit IRC11:57
*** tuannguyen has joined #openstack-infra11:59
*** lucas-afk is now known as lucasagomes11:59
*** krtaylor has quit IRC12:01
*** moravec has quit IRC12:01
*** flepied has quit IRC12:04
*** tuannguyen has quit IRC12:04
*** tuanluong has quit IRC12:05
*** hasharFood is now known as hashar12:06
*** mhickey has joined #openstack-infra12:09
*** raildo has joined #openstack-infra12:09
*** psilvad has joined #openstack-infra12:10
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces  https://review.openstack.org/38014212:12
*** amotoki has joined #openstack-infra12:12
openstackgerritErlon R. Cruz proposed openstack-infra/project-config: Adds new experimental Cinder migration job  https://review.openstack.org/38173712:14
*** kevinbenton has quit IRC12:15
*** furlongm has quit IRC12:15
*** Ravikiran_K has quit IRC12:15
*** furlongm has joined #openstack-infra12:16
*** kevinbenton has joined #openstack-infra12:17
*** psilvad has quit IRC12:17
*** arxcruz has joined #openstack-infra12:17
*** esikachev has joined #openstack-infra12:17
*** kashyap has joined #openstack-infra12:18
*** edmondsw has joined #openstack-infra12:18
kashyapIs Gerrit down, or is it just me?12:18
AJaegerkashyap: just checked - works for me.12:18
*** akshai has joined #openstack-infra12:18
kashyapAJaeger: Thanks, indeed.  It's my VPN that's behaving12:19
*** flepied has joined #openstack-infra12:20
*** esikachev has quit IRC12:22
*** trown|outtypewww is now known as trown12:22
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element  https://review.openstack.org/38157612:22
AJaegerkashyap: that's sad to hear, hope you can heal it12:23
dtantsurjroll, are you ok with switching ironic-lib jobs to xenial completely? AJaeger demands it, and honestly I'm tired of arguing12:24
jrolldtantsur: why not?12:25
dtantsurjroll, we don't run many jobs on xenial so far, so this might cause a breakage on master and/or newton12:25
jrolldtantsur: if the iscsi jobs work on xenial, I doubt others would break too hard12:26
dtantsurjroll, we only run postgres job on xenial now.. and if we switch ironic-lib, maybe we should switch ironic and IPA too... though I've no clue if image buildings works out of box on Xenial..12:27
jrolldtantsur: good point... ironic-lib is pretty low volume, it wouldn't be much trouble if it broke there first12:28
AJaegerdtantsur: I ask for it for new jobs on master.12:28
dtantsurAJaeger, well, for some definition of "new"...12:28
jrolldtantsur: I believe that IPA gate is still down due to the problem with osic cloud external networking12:28
*** salv-orlando has joined #openstack-infra12:28
dtantsurugh, wonderful.. I guess switch to xenial won't hurt it any more12:29
dtantsurAJaeger, next time I'd love to see something in place to avoid duplicating all the jobs with -xenial and -whatever prefixes12:30
*** mriedem has joined #openstack-infra12:30
*** ralonsoh_ has joined #openstack-infra12:30
kashyapAJaeger: Hehe, healed (with a crude workaround, restarted OpenVPN :-( )12:30
AJaegerdtantsur: we all hoped it would be easier ;( We had a long discussion on the pros and cons. In the past we did some magic and switched directly all jobs on stable/X to old Ubuntu version and everything newer to new Ubuntu version. But that surprised quite a few....12:31
pabelangerhopefully zuulv3 will make this easier next time we need to add a new OS to master only12:32
dtantsurI'd prefer to be surprised in this case, rather than figuring it out in every patch I make while trying to fix our CI situation12:32
*** markvoelker has joined #openstack-infra12:33
*** ralonsoh has quit IRC12:34
*** yamamoto has joined #openstack-infra12:34
AJaegerdtantsur: understood. The other solution had different pitfalls. Let's see what options we have next time...12:34
*** akshai has quit IRC12:35
*** number80 has joined #openstack-infra12:36
openstackgerritMerged openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces  https://review.openstack.org/38014212:36
openstackgerritMerged openstack-infra/puppet-infracloud: Add option to disable default routes  https://review.openstack.org/38166012:36
*** fguillot has joined #openstack-infra12:37
openstackgerritRichard Theis proposed openstack-infra/project-config: Move networking-ovn multinode job to check queue  https://review.openstack.org/37803212:37
*** amoralej|lunch is now known as amoralej12:37
openstackgerritRichard Theis proposed openstack-infra/project-config: Move networking-ovn multinode job to check queue  https://review.openstack.org/37803212:39
*** ijw has joined #openstack-infra12:40
*** pgadiya has quit IRC12:41
*** burgerk has joined #openstack-infra12:41
openstackgerritMajor Hayden proposed openstack-infra/project-config: Add gate jobs for RHEL 7 STIG testing  https://review.openstack.org/38177512:41
*** senk_ has quit IRC12:42
*** jpena|lunch is now known as jpena12:42
*** ccamacho|lunch is now known as ccamacho12:42
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix test_queries regression  https://review.openstack.org/38177612:42
*** amitgandhinz has joined #openstack-infra12:43
*** yolanda has quit IRC12:44
*** baoli_ has joined #openstack-infra12:44
*** tuannguyen has joined #openstack-infra12:45
*** ijw has quit IRC12:45
*** EricGonczer_ has quit IRC12:46
*** yolanda has joined #openstack-infra12:46
*** drawsmcgraw has left #openstack-infra12:46
*** baoli has quit IRC12:47
dtantsurjroll, do we want to switch to ipmitool jobs simultaneously?12:47
*** amitgandhinz has quit IRC12:47
jrolldtantsur: one part of me says one thing at a time, the other says we're overdue to switch12:47
dtantsurjroll, well, one thing at a time is good, but in this case I'll have to duplicate all jobs in this patch or ignore ipmitool for now12:48
jrolldtantsur: yeah let's just do it12:48
*** amotoki has quit IRC12:49
*** jcoufal has joined #openstack-infra12:50
mriedemare jobs in the experimental queue automatically non-voting?12:51
mriedemi think they are12:51
AJaegermriedem: implicitely ;)12:51
AJaegerthey are voting jobs but the vote is ignored ;)12:52
*** gordc has joined #openstack-infra12:52
AJaegerSo, "check experimental" does not change the Jenkins vote one way or the other.12:52
*** yamamoto has quit IRC12:53
openstackgerritCyril Roelandt proposed openstack-dev/hacking: Add a check to make sure the right assert* method is used  https://review.openstack.org/35418512:53
*** amotoki has joined #openstack-infra12:54
*** ralonsoh_ is now known as ralonsoh12:54
*** yamamoto has joined #openstack-infra12:55
ralonsoh Hola yolanda. I need to be added as member to devstack-plugin-libvirt-qemu-core and devstack-plugin-libvirt-qemu-release. My email is rodolfo.alonso.hernandez@intel.com12:55
yolandahola ralonsoh :) ok, going to add you12:56
ralonsohyolanda gracias12:56
*** shardy is now known as shardy_mtg12:56
yolandaralonsoh, de nada, hecho12:58
*** chandankumar has joined #openstack-infra12:59
*** moravec has joined #openstack-infra13:01
*** akshai has joined #openstack-infra13:01
*** Goneri has joined #openstack-infra13:02
*** yamamoto has quit IRC13:02
*** akshai has quit IRC13:02
*** yolanda has quit IRC13:02
*** moravec has quit IRC13:02
openstackgerritMajor Hayden proposed openstack-infra/project-config: Add gate jobs for RHEL 7 STIG testing  https://review.openstack.org/38177513:03
*** amitgandhinz has joined #openstack-infra13:03
*** amitgandhinz has quit IRC13:04
*** akshai has joined #openstack-infra13:04
openstackgerritJames Slagle proposed openstack-infra/project-config: Add check-tripleo-undercloud-jobs template  https://review.openstack.org/38179013:04
*** amitgandhinz has joined #openstack-infra13:05
pabelangermgagne: morning, we are seeing some issues with internap-mtl01 this morning, can you confirm?13:05
pabelangeroutage maybe?13:05
openstackgerritNate Johnston proposed openstack-infra/project-config: Bifurcate neutron-fwaas tempest gate for v2  https://review.openstack.org/37174913:06
*** yolanda has joined #openstack-infra13:07
*** spzala has joined #openstack-infra13:07
*** BobBall has quit IRC13:07
*** arxcruz has quit IRC13:07
*** fguillot has quit IRC13:08
openstackgerritJames Slagle proposed openstack-infra/project-config: Add check-tripleo-undercloud-jobs template  https://review.openstack.org/38179013:08
*** mdrabe has joined #openstack-infra13:09
*** vikrant has quit IRC13:11
*** esikachev has joined #openstack-infra13:11
*** rbrndt has joined #openstack-infra13:15
*** akshai has quit IRC13:16
*** yamamoto has joined #openstack-infra13:17
*** mriedem has quit IRC13:18
AJaegerpabelanger: we still have both http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rate and http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rates - could you delete the older one, please13:19
*** arxcruz has joined #openstack-infra13:19
*** yamamoto has quit IRC13:20
*** akshai has joined #openstack-infra13:20
*** tiswanso has joined #openstack-infra13:21
*** lseki has joined #openstack-infra13:23
openstackgerritHironori Shiina proposed openstack/diskimage-builder: Fix a command in Developer Documentation  https://review.openstack.org/38183313:24
*** yamamoto has joined #openstack-infra13:24
*** yamamoto has quit IRC13:25
*** chandankumar has quit IRC13:25
*** eharney has joined #openstack-infra13:26
*** yamamoto has joined #openstack-infra13:28
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit  https://review.openstack.org/37511313:29
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Fix test_queries regression  https://review.openstack.org/38177613:29
*** ralonsoh has quit IRC13:30
*** ralonsoh_ has joined #openstack-infra13:30
*** daemontool has joined #openstack-infra13:33
*** jaosorior has quit IRC13:34
*** jaosorior has joined #openstack-infra13:34
*** esikachev has quit IRC13:35
*** matt-borland has joined #openstack-infra13:35
*** pilgrimstack has quit IRC13:36
EmilienMGerrit is super slow for me, maybe I'm wrong13:36
dtantsurfor me too, waiting for my patch to be submitted13:36
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Rework ironic-lib tempest jobs  https://review.openstack.org/38165213:36
dtantsurjroll, AJaeger ^^^13:36
dtantsur"502 proxy error in web UI"13:37
*** yamamoto has quit IRC13:37
jrolldtantsur: why would you tell me to click that when gerrit is so slow :P13:38
dtantsurhehe :)13:38
jrollgot it now, will review13:38
zigome too ...13:38
*** amotoki has quit IRC13:38
*** rodrigods has quit IRC13:40
*** rodrigods has joined #openstack-infra13:40
*** ijw has joined #openstack-infra13:41
*** tuannguyen has quit IRC13:41
*** chandankumar has joined #openstack-infra13:41
*** tuannguyen has joined #openstack-infra13:42
ianychoime too....13:42
fricklerinfra-core: gerrit looks like it might be flatlining again http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all13:45
*** ijw has quit IRC13:46
*** burgerk has quit IRC13:46
*** chandankumar has quit IRC13:47
fungii'll take a peek at the javamelody graphs in a moment, and if it's the usual java gc death spiral i'll restart it13:48
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Missing location_id on Location images  https://review.openstack.org/38185113:48
fungiafter the release this week, we're planning to upgrade to a newer jgit version that in theory has fixed this leak13:48
*** links has quit IRC13:49
fungijust worried to create unnecessary disruption leading into release day13:49
*** jamesdenton has joined #openstack-infra13:49
*** pilgrimstack1 has joined #openstack-infra13:49
*** mariojv has joined #openstack-infra13:50
* zigo goes to upload 237 packages to Debian unstable ... :)13:51
zigo(not jocking, this really is my todo...)13:51
AJaegerzigo, will that trigger 237 patches in our CI? ;)13:52
zigoAJaeger: That part is already done! :)13:52
openstackgerritMerged openstack-infra/openstackid-resources: Missing location_id on Location images  https://review.openstack.org/38185113:52
zigoMostly, editing debian/changelog to put "unstable" there ...13:52
* AJaeger noticed - a new record of jobs in post queue at one day13:52
zigoTook me the whole day ...13:52
anteayaAJaeger: what is the current record for post jobs?13:53
*** hongbin has joined #openstack-infra13:53
AJaegeranteaya: you were counting last time, weren't you? Wasn't it 370?13:53
anteayawas it that day last week? I think we were over 40013:53
AJaegerseems ages ago ;(13:53
anteayaI was yeah, I will have to go back to the logs13:53
anteayabut after i get back from errands13:53
anteayaI'm out for most of today, I am going to try to get back for the meeting13:54
*** senk_ has joined #openstack-infra13:54
*** hamzy has quit IRC13:54
mariojvhi - is there something wrong with gerrit? i'm getting intermittent slowness and 502s13:54
anteayaif I miss, the only thoughts I have about ianychoi's item is that we don't have bugs expected on both storyboard and launchpad13:54
dasmmariojv: it seems so. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all13:55
dasmmariojv: already mentioned by frickler, EmilienM, dtantsur and others.13:55
mariojvthanks - wasn't in the channel but just wanted to add 1 more report13:55
zigomariojv: Yeah, there's something wrong ... memory leak as always ! :)13:56
*** moravec has joined #openstack-infra13:56
*** esikachev has joined #openstack-infra13:57
*** chandankumar has joined #openstack-infra13:57
*** yamamoto has joined #openstack-infra13:58
fungiconfirmed, gerrit's jvm started spending a lot of time on garbage collection right at 13:30. i'll go ahead and restart it13:58
*** chandankumar1 has joined #openstack-infra13:58
dasmfungi: thanks13:58
*** yamamoto has quit IRC13:59
*** cardeois has joined #openstack-infra13:59
fungi#status notice The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily13:59
openstackstatusfungi: sending notice13:59
*** bin_ has joined #openstack-infra13:59
*** coolsvap_ is now known as coolsvap13:59
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily14:00
*** xyang1 has joined #openstack-infra14:01
ianychoianteaya, actually infra IRC meeting is 4 AM here so it is too difficult for me to participate in the meeting. I asked to ildikov to explain and discuss storyboard and LP for openstack-dev/sandbox. She will lead upstream training on Barcelona.14:01
fungithe webui has returned now14:01
*** EricGonczer_ has joined #openstack-infra14:01
fungiianychoi: sounds good. she's already been asking me for some background on the current situation with sb14:02
openstackstatusfungi: finished sending notice14:02
*** salv-orlando has quit IRC14:02
*** esberglu has joined #openstack-infra14:03
*** chandankumar has quit IRC14:03
ianychoifungi, oh I see I will read meeting notes when I will get up tomorrow!14:03
*** senk_ has quit IRC14:03
*** kgiusti has joined #openstack-infra14:04
*** shardy_mtg is now known as shardy14:06
*** matrohon has quit IRC14:06
*** daemontool_ has joined #openstack-infra14:06
*** yamamoto has joined #openstack-infra14:08
*** yamamoto has quit IRC14:09
*** daemontool has quit IRC14:10
*** EricGonc_ has joined #openstack-infra14:10
*** xarses has quit IRC14:10
*** jheroux has joined #openstack-infra14:11
*** yamamoto has joined #openstack-infra14:12
*** claudiub has joined #openstack-infra14:12
*** burgerk has joined #openstack-infra14:13
*** avladu has joined #openstack-infra14:13
*** arxcruz has quit IRC14:13
*** EricGonczer_ has quit IRC14:14
*** gouthamr has quit IRC14:14
*** fguillot has joined #openstack-infra14:14
avladuhello, I get some failures during the docs generation on the cloudbase-init project. This is the test that fails: gate-cloudbase-init-docs-ubuntu-xenial14:15
avladuand this is an example of patch with that failure14:15
*** chandankumar1 is now known as chandankumar14:15
*** yamamoto has quit IRC14:16
*** mgould has joined #openstack-infra14:17
*** amotoki has joined #openstack-infra14:17
mgouldhi everyone! Could I please have some reviews on https://review.openstack.org/#/c/346923 ?14:18
mgould^^^ trivial patch, just adds a comment explaining something we found confusing in the Ironic team14:18
*** tosky has quit IRC14:20
*** matrohon has joined #openstack-infra14:21
*** tosky has joined #openstack-infra14:21
*** moravec has quit IRC14:21
pabelangeravladu: AJaeger: this looks to be an issue with our upcoming migration to afsdocs, we'll look into it shortly14:22
*** rlu has quit IRC14:23
*** rlu has joined #openstack-infra14:24
*** rlu has quit IRC14:24
pabelangerclarkb: fungi: jeblair: So, I am currently testing out 381883 Custom ssh.conf (upstart) for openssh-server to fix our sshd restart issue.  I'm open to suggestions on how better to solve this, including making nodepool a little more forgiving if the SSH connection is reset when we first bring a node online14:25
openstackgerritDaniel Castellanos proposed openstack-infra/project-config: Update Jenkins jobs for craton-dashboard  https://review.openstack.org/38132714:25
*** yamamoto has joined #openstack-infra14:26
fungipabelanger: yeah, i'm starting to get the distinct impression that there are no real guarantees in boot-time that once sshd is up and listening, it won't be suddenly restarted by something else in the boot process14:27
*** hamzy has joined #openstack-infra14:27
*** yamamoto has quit IRC14:27
*** Naeil has quit IRC14:27
pabelangeryes, it is a battle for sure14:27
*** yamamoto has joined #openstack-infra14:28
*** nadya has quit IRC14:28
*** drawsmcgraw has joined #openstack-infra14:28
pabelangerso far I've found /etc/network/if-up.d/openssh-server14:28
fungimaybe we should have a very final initscript or whatever on all our nodes that touches something like /booted and then have nodepool keep checking for that and allow (some limited number of) ssh retries or whatever until that file exists14:29
clarkbmgould: that is a grenade change whoch is a qa project. You may have better luck getting reviews if you ask in #openstack-qa14:29
clarkbpabelanger: is it glean fighting the rest of startup? maybe we just need to remove that from glean?14:30
pabelangerclarkb: no, the patch I did yesterday was not the actually issue.  infact, we can abandon it I think14:30
pabelangerits the same issue as always, sshd is restarted because of X14:31
fungiclarkb: it's interface configuration changes causing sshd restarts in that case14:31
pabelangerand with trusty SSH is started pretty early in the process14:31
pabelangersystemd tries to solved this14:31
pabelangerand explains why we see less failures now14:32
pabelangerfungi: a marker might help, but having nodepool retry a limited amount of times is likely the key to fixing this14:32
clarkbthough the resets are due to sshd running without host keys not the restarts right?14:33
pabelangernot any more, we fixed that14:33
clarkb(it should generally be safe to restart sshd but not to run it without the required keys)14:33
cloudnulljroll: RE: "I believe that IPA gate is still down due to the problem with osic cloud external networking" -- this is still the devstack+neutron network issue clarkb and a few others have mentioned before, right? There's not some new connectivity issue in the OSIC?14:33
* cloudnull reading scrollback14:33
*** apetrich has quit IRC14:34
pabelangerclarkb: the command is invoke-rc.d ssh restart >/dev/null 2>&1 || true, which kills our ssh connection14:34
mgouldclarkb: d'oh, thanks!14:34
*** apetrich has joined #openstack-infra14:34
*** rossella_s has quit IRC14:34
*** mgould has left #openstack-infra14:34
clarkbpabelanger: I dont think that should kill connections. Are we sure host keys are generated at that point?14:34
pabelangerclarkb: yes, keys have been created14:35
pabelangerI can see ssh-keygen server start and stop successfully14:35
pabelangerthen, sshd is started14:35
*** rossella_s has joined #openstack-infra14:35
*** spzala has quit IRC14:35
*** gouthamr has joined #openstack-infra14:37
*** annegentle has joined #openstack-infra14:37
clarkbcloudnull: I am not aware of any new issues with networking. I think that ironic using 10/8 to address their fale baremetal nodes in testing so the route update makes those hosts uncommunicative14:39
cloudnullmakes sense .14:39
*** rook_ is now known as rook14:39
cloudnulljust checking.14:39
*** yamamoto has quit IRC14:40
* cloudnull making sure cloud1 isn't being a bad actor14:40
*** senk_ has joined #openstack-infra14:42
*** tuannguyen has quit IRC14:42
pabelangerWe could add an except handler for [Errno 104] Connection reset by peer http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodeutils.py#n47 like we did for cloud user created after sshd is up14:42
*** ijw has joined #openstack-infra14:42
*** tuannguyen has joined #openstack-infra14:42
*** john-davidge has quit IRC14:44
*** john-davidge has joined #openstack-infra14:44
*** burgerk_ has joined #openstack-infra14:46
*** yamamoto has joined #openstack-infra14:46
*** nadya has joined #openstack-infra14:47
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Switch coverage tox job to use pbr test command  https://review.openstack.org/38189814:47
*** ijw has quit IRC14:47
*** mtanino has joined #openstack-infra14:48
*** john-davidge has quit IRC14:49
*** burgerk has quit IRC14:50
*** dragonmaster_ has quit IRC14:50
*** yamamoto has quit IRC14:51
*** Naeil has joined #openstack-infra14:51
*** tuannguy_ has joined #openstack-infra14:51
*** salv-orlando has joined #openstack-infra14:51
*** avladu has quit IRC14:52
*** tuannguyen has quit IRC14:52
*** nadya has quit IRC14:52
clarkbpabelanger: random brainstorming we could try to make it so firewall doesn't open port 22 until things are actually ready (this is hard because on boot the firewall is wide open then we restrict it from there)14:54
*** salv-orlando has quit IRC14:56
AJaegeravladu, pabelanger : cloudint setup is broken, they do not publish docs to doc/build/html.14:58
*** drifterza is now known as notdrwahl14:59
jeblairpabelanger, clarkb: i'm also curious why that restart affects existing ssh connections... does it perhaps only affect connections that are not fully established yet or something?14:59
AJaegerpabelanger: still, we should not abort in that case. Let me try fixing both sides15:00
clarkbjeblair: that was my theory before we ran into the no host keys behavior. Was thinking that maybe we are catching the connection before the subprocess has forked off and reparented15:00
clarkbjeblair: however i Have not been able to confirm that, and the no host keys behavior fit what we were seeing perfectly15:00
*** rajinir has joined #openstack-infra15:01
rajinirfatal: unable to access 'https://review.openstack.org/openstack-dev/devstack/': The requested URL returned error: 50315:01
rajinirThis was working last week15:02
pabelangerjeblair: clarkb: not much, but here is the traceback when the connection resets: http://paste.openstack.org/show/584234/15:02
jeblairpabelanger, clarkb: i could probably get behind a retry for connection reset15:02
jeblairrajinir: what received that error?15:03
AJaegerrajinir: and when?15:03
rajinirThis is in our CI pre_test_hook, we have to fetch an upstream patch15:03
*** r-mibu has quit IRC15:04
*** r-mibu has joined #openstack-infra15:04
AJaegerrajinir: we restarted gerrit an hour ago due to some issues, did it occur at that time?15:04
*** HeOS has quit IRC15:04
*** HeOS has joined #openstack-infra15:05
*** spzala has joined #openstack-infra15:05
rajinirAJaeger: seems intermittent15:05
*** avladu has joined #openstack-infra15:05
rajinirAJaeger: Saw some errors in the last hour15:06
openstackgerritEric Harney proposed openstack-infra/project-config: Cinder: don't skip changed tempest tests  https://review.openstack.org/38191015:06
AJaegerpabelanger, jeblair: there was a comment that https://review.openstack.org/#/c/333877 failed the docs job due to the "echo >  doc/build/html/.root-marker". Looking at the repo, build_sphinx was misconfigured. Do we care about that - or should and thus should I make run-docs.sh robust to not fail in this case - or should we let it fail, so that people integrate nicely?15:07
*** electrofelix has quit IRC15:07
fungijeblair: theory... nodepool attempts to connect right at the moment that the network interface configuration is updated and sshd restart has been triggered, so no listening socket15:07
AJaegerwithout that path, docs-draft publishing will not work.15:07
*** electrofelix has joined #openstack-infra15:07
jeblairfungi: well, it's a 'connection reset', so i expect it's at least started the tcp handshake...15:08
*** notdrwahl is now known as drifterza15:09
fungihrm... maybe. as long as "connection reset" in this case doesn't mean tcp/rst in response to tcp/syn15:09
jeblairAJaeger: regarding developer-docs -- i think we just haven't made the afs volume for it yet.  i think we can go ahead and do that now, yeah?15:09
fungiwhich would of course be the normal reaction when attempting to establish a connection with no listening socket15:10
*** rcernin has quit IRC15:10
*** kmartin has quit IRC15:10
AJaegerjeblair: I'm fine with creating it, all jobs should be ready.15:10
jeblairfungi: it's certainly different behavior from 'ssh isn't running right now'15:10
AJaegerjeblair: but I'll double check15:10
fungigot it15:10
jrollcloudnull: yes, the 10/8 thing15:11
*** esikachev has quit IRC15:12
cloudnullok. just making sure.15:12
jeblairAJaeger: i'm in favor of standardization, so i'd leave run-docs.sh as-is.15:12
AJaegerjeblair: yes, you added all the afs jobs already to api-jobs, so we should be fine on that side. And checking the jobs, the root-markers are fine as well...15:13
AJaegerjeblair: that was my second thought as well ;) I only fixed the broken cloudbase-init ;). Let's see what they think...15:14
avladuAJaeger: so now it s all fine on that gate run?15:14
*** moravec has joined #openstack-infra15:14
AJaegeravladu: cloudbase-init is broken, I sent a patch for it.. https://review.openstack.org/38190715:15
*** yamahata has joined #openstack-infra15:16
jeblairAJaeger: what kind of documentation will the 'developer-docs' volume contain?15:16
AJaegeravladu: you publish to a location that is different from what our infra scripts expect.15:16
*** lucasagomes is now known as lucas-hungry15:16
jeblairAJaeger: (it's *not* docs.o.o/developer -- i know that)15:16
avladuAJaeger: got it, thanks15:16
*** mriedem has joined #openstack-infra15:16
AJaegerjeblair: http://developer.openstack.org/ - so api-ref, api-quick-start (from api-site), firstapp application, api-guide documents15:16
avladuAJaeger: I will +2 asap15:16
AJaegerAnd python-openstacksdk15:16
jeblairAJaeger: got it, thx15:17
*** daemontool_ has quit IRC15:19
*** armax has joined #openstack-infra15:22
AJaegerpabelanger, jeblair , we added a vhost for docs.o.o in https://review.openstack.org/#/c/380483/15:23
AJaegerWhat do you think of changing that to docs-beta (or similar) so that we can browse it properly? files.openstack.org has all the URLs wrong, they are relative to / and thus don't work...15:23
*** akshai has quit IRC15:24
jeblairAJaeger, pabelanger: maybe we can make docs-beta a serveralias15:24
* AJaeger does not care about the name, just wants to check it properly ;)15:25
strigazihi clarkb, I tried your patch (379543) and the jobs in magnum succeed in osic. However the change is not merging. Any advice?15:25
clarkbstrigazi: we need to wait for neutron/devstack to make a change globally that fixes the issue15:25
clarkbjeblair: fungi pabelanger https://review.openstack.org/#/c/370442/1 is a fix for nova net multinode testing15:25
*** tuannguy_ has quit IRC15:26
strigaziclarkb, so until then no patch merges for magnum15:26
JayFWhere is the right place to go for TC Election feedback? I'm pretty sure I was just emailed a second ballot.15:26
jrollclarkb: is that "global change" you mention this or something else (is it up yet)? https://review.openstack.org/#/c/380023/115:26
jrollJayF: tonyb / njohnston / tristanC15:26
*** yaume has quit IRC15:26
clarkbstrigazi: unless you reconfigure your job to stop using 10/8 or get lucky and jobs run on not osic15:26
JayFtonyb: njohnston: tristanC: I am pretty sure I just got mailed a second TC Election ballot.15:27
clarkbjroll: I have sort of somewhat intentionally tried to hand that off back to the responsible teams so I am not currently sure of what tactic they want to take15:27
*** tuannguyen has joined #openstack-infra15:27
clarkbjroll: revert has been mentioned though that requires some unwinding of a test or two that depends on the feature15:27
*** daemontool has joined #openstack-infra15:27
clarkb(I don't want to bandaid this in infra, this needs to be fixed where it was broken)15:27
JayFtonyb: njohnston: tristanC: oh; apparently just duplicate email w/same voter key. It won't let me revote. Disregard.15:27
jrollclarkb: how would one stop using 10/8?15:27
strigaziclarkb, ok I'll change out job. Is there a WIP in neutron/devstack to follow and revert my change later?15:27
*** xarses has joined #openstack-infra15:28
*** Teresita-Warrior has joined #openstack-infra15:28
jeblairJayF: (the election officials will have the authoritative answer, but if it was sent to the same email address, it's likely the same ballot just resent because of some rax delivery issues)15:28
clarkbstrigazi: as I just said I don't know you will need to talk to them15:28
strigaziclarkb, thanks15:28
clarkbjroll: strigazi in magnums case they are connecting to rabbit over the 10/8 address they could just use localhost I think15:28
JayFjeblair: yeah, that's what it looks like. Just weird because I know better than to use my Rackspace email address for emails I want to recieve :P15:28
jrollclarkb: ah, I see. no hacks for external connectivity?15:28
njohnstonJayF: We resent ballots to everyone because there were numerous reports of email issues.  As jeblair said, it should have been sent to the same email address, which means it is the same ballot just reissued; you still only have one vote. :-) http://lists.openstack.org/pipermail/openstack-dev/2016-October/105002.html15:28
fungiJayF: yeah, see tonyb's announcement to the -dev ml yesterday/today (depending on what you call midnight). a lot of people didn't receive their ballots the first time around, so they resent copies of them all just tp be on the safe side15:29
*** ilyashakhat_mobi has joined #openstack-infra15:29
clarkbjroll: strigazi ironic's case is more complicated beacuse you are doing networking in and out of neutron. You would need to make your network subnets outside of neutron not overlap with 10/8 so you could use 192.168/16 or 172.whateverthatrangeisIforget15:29
AJaegerjeblair: I'm not a puppet expert, is this the right syntax ? https://review.openstack.org/38192715:29
jrollclarkb: got it, thanks15:29
jeblairAJaeger: sorry, i'm digging into something, i'll pop back up to that in a few minutes15:29
AJaegerinfra-root, we lost gerritbot - anybody has time to help, please?15:29
AJaegerjeblair: sure, not urgent for now.15:29
funginjohnston: worth noting, i didn't receive a second copy. and my friend who didn't get a ballot the first time still hasn't seen one (i've confirmed i can contact the address he has in the electoral roll)15:29
*** rohanion has joined #openstack-infra15:30
JayFclarkb: 172.16/1215:30
clarkbjroll: and I think the lesson here is we should be using reasonably sized ranges that meet our needs rather than the biggest possible range in hopes that we someday need it all15:30
*** adrian_otto has joined #openstack-infra15:31
clarkbso for ironic testing you probably only need a /2415:31
jrollclarkb: indeed15:31
clarkb(not sure what you are currently using that may already be the case)15:31
njohnstonfungi: Please encourage your friend to reach out to Tony, Tristan, or myself either directly or on #openstack-election, and we can work with your friend to ensure their ballot is received.15:31
*** moravec has quit IRC15:32
avladuAJaeger: thanks all for the help, change merged15:32
*** akshai has joined #openstack-infra15:32
AJaegeravladu: I rechecked the failing job, should be fine soon...15:32
openstackgerritNate Johnston proposed openstack-infra/project-config: Bifurcate neutron-fwaas tempest gate for v2  https://review.openstack.org/37174915:33
AJaegeroh, openstackgerrit is mirally back. Thanks!15:34
funginjohnston: yep, it's hard enough to encourage people to vote, so getting them to e-mail someone so they can get a ballot is tough. seeing how convincing i can be15:34
*** flepied1 has joined #openstack-infra15:35
njohnstonfungi: Thanks, I appreciate it.  I'm passionate about elections - I'm an elections officer IRL as well - so if there is anything I can do to assist please let me know.15:35
*** esikachev has joined #openstack-infra15:35
AJaegerfungi, njohnston : I didn't get a second email either - but the first arrived.15:36
*** flepied has quit IRC15:37
funginjohnston: appreciated! but yeah, as i feared the resend seems to have been about as spotty as the original batch of ballots. so if 80% of people received ballots each time then we have 10% of the electorate who didn't receive a ballot either time (numbers picked out of thin air for purposes of demonstration)15:38
AJaegerfungi, my math says 16 %...15:39
fungiAJaeger: your math is more precise than my estimates ;)15:39
pabelangerclarkb: fungi: jeblair: I'm also going to try having nodepool use host key checking (ssh-keyscan) to collect more data. Curious to see what a happens with sshd in that instance15:40
ajo_any idea of why is this failing sometimes ?15:40
pleia2good morning15:40
ajo_(fetching logs from subnodes in a multinode)15:40
clarkbajo_: yes, beacuse devstack and neutron have set an invalid route for breaking that ipv4 connectivity between hosts in multinode testing15:40
*** sandanar has quit IRC15:40
openstackgerritNate Johnston proposed openstack-infra/project-config: Trifurcate neutron-fwaas tempest gate for v2  https://review.openstack.org/37174915:40
*** salv-orlando has joined #openstack-infra15:40
ajo_clarkb, is there any way around that ?15:41
ajo_clarkb, where is that set?15:41
clarkbajo_: not right now no15:41
AJaegerajo_: clarkb is working on it AFAIU - feed him ;)15:41
clarkbajo_: devstack is setting the range via the new subnetpool feature support and I am not sure if neutron or devstack is setting the route15:41
pabelangerAlso testing connection failure retry logic15:41
clarkbNo! I am trying my best to not work on it :)15:41
AJaegerclarkb: Sorry;(15:42
clarkbI have helped identify and diagnose the problem along with a couple others but would really really like devstack and neutron to fix it15:42
ajo_clarkb, I'd be happy to help with that if I can/know how (I'm a neutron core)15:42
AJaegerclarkb: understood15:42
clarkbajo_: let me get the bug15:42
ajo_clarkb, awesome15:42
openstackLaunchpad bug 1629133 in devstack "New neutron subnet pool support breaks multinode testing." [Critical,In progress] - Assigned to Dr. Jens Rosenboom (j-rosenboom-j)15:42
SpamapSjeblair: did we ever figure out why geard doesn't work with the upstream gearman CLI worker?15:43
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/puppet-iptables: Change Cacti IP address for Iptables rules  https://review.openstack.org/38193615:43
* SpamapS is poking at things left on the backburner for way too long15:43
*** ijw has joined #openstack-infra15:43
avladuAJaeger: all good now, much appreciated15:43
pabelangeravladu: as a follow up, it is usually better just to ask your questions here, then PM.  So, others can see and answer them15:44
*** eharney has quit IRC15:44
SpamapSlooks like maybe geard doesn't understand GEARMAN_GRAB_JOB_ALL15:45
*** eharney has joined #openstack-infra15:47
*** thcipriani|afk is now known as thcipriani15:47
*** matrohon has quit IRC15:47
*** ijw has quit IRC15:48
*** adrian_otto has quit IRC15:48
rcarrillocruzhaha, jesusaur i just pushed https://review.openstack.org/#/c/381936/1 while thinking 'oh my, why we have this hardcoded!' and just saw on the right column of the change you pushed something already to parameterize it15:49
rcarrillocruzso ++15:49
rcarrillocruzhappy to land your patch15:49
*** akshai has quit IRC15:51
mtreinishfungi: so it looks like my o-h change merged last night but health.o.o is still giving out 500s15:51
mtreinishfungi: is it still the same error message as before?15:51
ajo_clarkb, I wonder if using a LLA range would work (
fungimtreinish: [Tue Oct 04 15:52:07.278906 2016] [:error] [pid 31032:tid 139914604582656] [remote] TypeError: __init__() got an unexpected keyword argument 'es_url'15:53
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/gear: Add py35 to tox.ini default envs  https://review.openstack.org/38194415:53
*** armax has quit IRC15:53
fungimtreinish: the timestamp there was just a few seconds ago, so i guess it is15:53
clarkbajo_: maybe but that wouldn't be a valid use of that range right? We jsut need to stop using all encompassing ranges15:53
clarkbajo_: we should use a /20 not a /815:53
clarkbthen have a utility that will find you the next available /20 based on routes on the host15:54
clarkbbut I think step one is stop breaking everyone15:54
fungimtreinish: "Fix elasticsearch querying with recent e-r changes" is the current checkout in /opt/openstack-health so i'm seeing what's causing it to hold onto the old version15:54
clarkbajo_: and also only write routes for subnets actually in use15:55
mtreinishfungi: ok, that's what I figured was happening. Maybe apache just needs a restart?15:55
clarkb(pretty sure I captured my argument in the bug already)15:55
*** yamahata has quit IRC15:55
ajo_clarkb, that's a valid use of the range I think, if for testing we just need it to be accessible locally to the controller...15:55
fungimtreinish: probably, but want to check where openstack-health is installed first15:55
*** hashar is now known as hasharAway15:56
ajo_clarkb, the dynamic routes is more a deployment thing,15:56
clarkbajo_: I thought that address range was for when you had no address managment15:56
ajo_the routes must be there before neutron l3 agent kicks in15:56
clarkbajo_: neutron is managing addresses and even runs a dhcp so that wouldn't be a valid use of the range15:56
mtreinishfungi: it's in a venv /opt/openstack-health/.venv iirc15:56
*** tiswanso has quit IRC15:56
fungimtreinish: `/opt/openstack-health.venv/bin/pbr freeze` says openstack-health==0.0.1.dev680  # git sha c03378f but your change merged as fa9c80415:57
ajo_clarkb, let me investigate the thing a little more, I will even send a dummy patch to check if LLA works15:57
fungimtreinish: checking puppet logs next to see why it failed to upgrade15:57
*** mdrabe has quit IRC15:57
clarkbajo_: "are only used where stable, routable addresses are not available" from the rfc15:57
clarkbajo_: so using that range would be a violation I think15:58
fungimtreinish: yeah, c03378f is the merge commit prior to your change15:58
mtreinishfungi: oh, right the puppet does a pip install -U /opt/openstack-health exec (there was weirdness with the entrypoints and -e)15:58
ajo_clarkb, we don't need them to be routed, right?15:58
clarkbajo_: you do15:58
*** adrian_otto has joined #openstack-infra15:58
clarkbajo_: at least I was told that was the whole reason we are in this mess in the first place :)15:58
ajo_clarkb, ok, let me do my homework the, (checking how's this set'd up)15:59
clarkbajo_: if this weren't routable we would not have problems and it would likely just work15:59
ajo_clarkb, for routable, i mean, you need them to go out of the node itself for testing15:59
*** tiswanso has joined #openstack-infra15:59
*** tiswanso has quit IRC15:59
clarkbajo_: yes, for multinode you do15:59
ajo_like for floating ips, we just attack from tempest to local15:59
ajo_yeah, in DVR may be15:59
*** moravec has joined #openstack-infra15:59
clarkbajo_: not always. DVR and nova net need them to be routable15:59
clarkbDVR being the case that matters here16:00
pleia2good morning16:00
*** akshai has joined #openstack-infra16:00
ajo_undertood :/16:00
fungimtreinish: Oct  3 21:58:39 health puppet-user[302]: (/Stage[main]/Openstack_health::Api/Exec[requirements]/returns)     ImportError: Building pandas requires cython16:00
ajo_whatever XD16:00
SpamapSjeblair: well I created this https://storyboard.openstack.org/#!/story/2000737 ... but I can't seem to find the description I submitted. :-P16:00
ajo_I don't know how to write anymore16:00
*** tiswanso has joined #openstack-infra16:00
mtreinishfungi: huh, wasn't expecting that. Heh, I guess that's the downside with -U16:01
mtreinishI didn't even realize pandas added a cython buildep16:01
SpamapSSeriously.. what happens to the description when creating a new story?16:01
SotKSpamapS: oops, its there, just breaking the markdown renderer16:02
*** jaosorior has quit IRC16:02
SpamapSSotK: hah, doh16:02
fungiSotK: i wonder if it would make sense to wrap markdown or similar parsers in a try/except and fall back to displaying raw content16:02
SpamapSno fall back to plaintext eh16:03
*** amotoki has quit IRC16:03
SotKfungi: it probably would make sense to do that :)16:03
*** mdrabe has joined #openstack-infra16:03
SpamapSalso should fail saving16:03
fungimight be a cheaper solution than alternatives anyway16:03
SpamapS"Markdown doesn't like this."16:03
*** amitgandhinz has quit IRC16:03
SotKSpamapS: try indenting any logs you've pasted in the description by 4 characters to make them code blocks, that might help16:04
fungimeh, if it fell back on rendering plaintext when the other format parsers break, then i'd be fine with saving still working16:04
*** BobBall has joined #openstack-infra16:04
fungiat least that way it's discoverable16:04
SpamapSSotK: Ok, however, that's.. not easy ;)16:04
ZaraI think ``` should also work to make a codeblock16:05
persiaSpamapS: `sed 's/^/    /' ?16:05
SotKwrapping in <pre> tags may work too16:05
*** andreas_s has quit IRC16:06
fungioh, yep, i've done it in sb with <pre></pre> successfully before16:06
SpamapSpersia: not easy. In my browser.16:06
*** ilyashakhat_mobi has quit IRC16:06
SpamapSpre's would be better yah16:06
persiaOh, ugh, I don't know the special JS trick for that.16:06
Zara(for backticks, there's an example here; I tend to use those most: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code-and-syntax-highlighting)16:06
SpamapSbut I just vim'd things16:06
ihrachysAJaeger: that bindep.txt thingy, is it safe to land one in e.g. neutron? can't it break any other jobs in other gates?16:06
SpamapSand it's especially confused by the logs that are variable-left-space-filled16:07
ihrachysAJaeger: seems like devstack jobs are not affected? anything else to be cautious about?16:07
*** drifterza has quit IRC16:07
SpamapSah no it was a missing blank line16:07
jeblairSpamapS: ah, yep.  that'll be because GRAB_JOB_ALL does not appear in http://gearman.org/protocol/ so i didn't know to implement it :)16:08
SpamapSSotK: ok, so yeah if you want a test case.. take the leading spaces out of those logs. :)16:08
SpamapSjeblair: I think it does appear there now16:08
SpamapSor not16:08
jeblairSpamapS: i don't see it even after madly hitting shift-reload ;)16:08
SpamapSjeblair: well guess who maintains that website now ;-)16:08
fungiihrachys: right now the only jobs that use bindep lists are project-specific jobs. probably the only thing to worry about is if you have file filters or skip-if for neutron jobs in zuul's layout.yaml causing some of your jobs not to run on the change adding bindep.txt16:08
jeblairSpamapS: is it me? ;)16:08
SpamapSjeblair: it can be!16:08
SpamapS--> https://github.com/gearman/gearmand16:09
mtreinishfungi: so I guess I should add cython to the puppet?16:09
SpamapSactually it's possible that /protocol just isn't up to date16:09
ihrachysfungi: I would be very surprised if we would have such a skip-if.16:09
* SpamapS checks16:09
ihrachysfungi: cool, thanks for the info16:09
jeblairSpamapS: cool -- though i think it might be best for not-me to write the docs for grab_job_all since i dunno what it is :)16:09
jeblairSpamapS: "Erm, it grabs all the jobs or something"16:09
SpamapSaha! yes I didn't realize I have to manually copy the new version into the website16:10
fungimtreinish: that's probably the easiest solution yeah. cython-dev or whatever16:10
SpamapShttps://github.com/gearman/gearmand/blob/master/PROTOCOL <-- up to date16:10
SpamapSand even in a release, 1.1.1316:10
jeblairoh it's a mapreduce thing16:11
*** dragonmaster has joined #openstack-infra16:11
*** tphummel has joined #openstack-infra16:11
jeblairAJaeger: http://files.openstack.org/docs-developer/ exists now16:11
*** tongli has joined #openstack-infra16:11
*** chandankumar has quit IRC16:11
*** jcoufal_ has joined #openstack-infra16:11
clarkbjeblair: do we intend on moving docs-draft to afs as well?16:12
*** amitgandhinz has joined #openstack-infra16:12
* clarkb should go pull up the spec16:12
*** avladu has quit IRC16:12
ihrachysfungi: hm, one more thing. this is the bindep.txt file I have for neutron: https://review.openstack.org/#/c/381906/3/bindep.txt (I assume it misses some needed deps, but that's what I start with)16:12
clarkbwas just thinking that that migration could give us a little headroom on the logs siguation if we can recoup that disk space for logs properl16:12
ihrachysfungi: it passes bindep test locally16:12
*** jcoufal has quit IRC16:12
ihrachysfungi: but in gate I catch this: http://logs.openstack.org/06/381906/3/check/gate-neutron-pep8-ubuntu-xenial/148dcd8/console.html#_2016-10-04_15_57_36_84669916:13
ihrachysif I call 'bindep -b -f bindep.txt test' in the local venv it passes too16:13
openstackgerritMatthew Treinish proposed openstack-infra/puppet-openstack_health: Install cython with the openstack-health api server  https://review.openstack.org/38196016:13
mtreinishfungi: ^^^16:13
SpamapSjeblair: _now_ refresh http://gearman.org/protocol/16:13
*** moravec has quit IRC16:13
jeblairclarkb: out of scope for spec, but it certainly could be done16:13
*** ArxCruz has joined #openstack-infra16:14
jeblairclarkb: we may want to think about whether running a find on all of docs-draft is a good idea though -- we may need a variant of the afs publisher that doesn't do that16:14
*** Ahharu has joined #openstack-infra16:15
jeblairclarkb: (i am certain running a find on *logs* is a bad idea)16:15
Ahharuare you aware of "2016-10-04 10:50:19.020907 |   Failed to fetch ec2 uri 500 Internal Server Error" ?16:15
Ahharuits bombing out the beaker tests16:15
clarkbjeblair: ya16:15
*** weshay is now known as weshay_lunch16:15
SpamapSjeblair: the reducer stuff needs some fleshing out, as I understand it finally.16:15
clarkbjeblair: well we do it weekly for deletes and compression but its infrequent and on weekends16:15
*** Swami has joined #openstack-infra16:16
jeblairclarkb: i think i'd probably go ahead and make the 'non-find' variant of the afs publisher before moving either docs-draft or logs16:16
jeblairAJaeger: 381927 lgtm.  pabelanger want to aprv?16:17
*** armax has joined #openstack-infra16:17
*** links has joined #openstack-infra16:17
*** amitgandhinz has quit IRC16:18
SpamapSjeblair: but basically, it ties in with the coalescing. When the first client submits the job, they submit the 'reducer' argument along with the job data and unique, and workers can use that to submit the result of the work to that function.16:18
*** amitgandhinz has joined #openstack-infra16:18
jeblairSpamapS: that makes sense16:18
*** links has quit IRC16:18
SpamapSso geard doesn't have to do anything special16:18
SpamapSkrow was working on some built-in thread-workers that would automatically do reducers though.16:19
jeblairsounds like the whole kit should be easy to add16:19
SpamapSjeblair: yeah, almost done. ;)16:19
SpamapSjeblair: also about 90% done on coalescing16:19
jeblairclarkb: looking at https://review.openstack.org/381944 should we make py35 voting?16:20
SpamapSthough I think there's a missing coordination piece that's allowing coalesced jobs to stay in queue after completion.16:20
*** Ahharu has quit IRC16:20
clarkbjeblair: I think so esepcailyl since 34 is and there aren't a ton of differences in that direction of transition16:20
SpamapSEven though I know.. you're just about done removing gearman from the zuul/nodepool world. ;)16:20
*** arxcruz_ has joined #openstack-infra16:21
*** claudiub has quit IRC16:21
jeblairSpamapS: well, gearman is going to stick around for the initial version of v3 -- my plan to remove it completely is very long range :)16:21
*** lucas-hungry is now known as lucasagomes16:21
*** david-lyle has joined #openstack-infra16:21
jeblairSpamapS: (zk for all of nodepool and nodepool<->zuul interaction; but zuul->zuul-workers is gearman)16:21
Shrewsv4 includes death star plans16:22
Shrews(oops, did i just let that slip?)16:22
TheJuliaShrews: you did16:22
jeblairShrews: when v4 is fully operational you will see a demonstration of its power...16:23
SpamapSShrews: that's no cloud16:23
*** tiswanso has quit IRC16:23
SpamapSjeblair: Ahh, forgot about the worker comms16:23
*** nt has quit IRC16:23
*** dragonmaster has quit IRC16:24
*** nt has joined #openstack-infra16:24
ihrachysfungi: is it even correct to call -b here https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/install-distro-packages.sh#L48 ?16:24
ihrachysseems to me like -b makes the tool to not write any stdout16:24
strigaziclarkb, is FIXED_RANGE= going to work?16:24
*** dragonmaster has joined #openstack-infra16:25
strigaziclarkb, or FIXED_RANGE= ?16:25
clarkbstrigazi: we set FIXED_RANGE to in the gate. Unfortunately this isn't sufficient because the subnetpools ignore the fixed range16:25
*** tiswanso has joined #openstack-infra16:25
fungiihrachys: -b is brief/batch mode. it will output a list of all packages missing on the system. no output means nothing missing16:26
jeblairremote:   https://review.openstack.org/381969 Gear: make python3.5 jobs voting16:26
ihrachysfungi: well, if I execute without -b it returns me a package16:26
ihrachysBad versions of installed packages:16:26
ihrachys    sqlite version 3.7.17-8.el7 does not match >=3.816:26
ihrachysbut with -b, it's just return code = 116:26
*** openstackgerrit has quit IRC16:26
ihrachysbut no output16:26
pabelangerjeblair: AJaeger: +316:27
fungiihrachys: oh, that may be a bug in the version parsing, which is not used heavily yet16:27
*** openstackgerrit has joined #openstack-infra16:27
ihrachysfungi: yeah, I noticed it seems like not used anywhere ;)16:27
ihrachysfungi: was trying to find an example16:27
*** baoli_ has quit IRC16:27
ajo_armax, clarkb,  I sent a middle ground patch: https://review.openstack.org/#/c/381965/16:28
strigaziclarkb, what if I set SUBNETPOOL_PREFIX_V4=
ajo_strigazi, 172.0.x.x is public16:28
fungiihrachys: version specifications were roughed in originally but since most distros tend to not have multiple versions of the same package name in the same release there's not been a lot of call to improve it yet16:28
clarkbajo_: the problem with that is we already tell devstack what range to use and devstack ignores it...16:28
ajo_you would prevent any host on that range to be reached16:28
*** openstackgerrit has quit IRC16:28
ajo_clarkb, what do you mean?16:28
*** tesseract- has quit IRC16:29
ihrachysfungi: well, we have a case where unit tests fail with old sqlite16:29
clarkbajo_: I would like to see something that incorporates existing range selection signalling so that other people that have to set a range will continue to work16:29
*** openstackgerrit has joined #openstack-infra16:29
ihrachysfungi: I see a bug in code16:29
clarkbajo_: every devstack job in the gate sets fixed range to to avoid conflicts with clouds16:29
clarkbajo_: this is the range that should be used.16:29
armaxajo_: I talkred to carl_baldwin yesterday, we’re gonna look into ways in which subnet_pools don’t interfere with host routes16:29
clarkbexcept it isn't anymore16:29
ihrachysfungi: the way we handle error[0] == 'badversion', it does not output if brief == True16:29
*** ArxCruz has quit IRC16:29
ajo_armax, clarkb copy that16:29
*** eharney has quit IRC16:29
*** openstackgerrit has quit IRC16:30
fungiihrachys: thanks. reviews in the usual place. since -b is designed to be inlined or piped to platform-specific package managers, i'm open to ideas on how that should behave for version mismatches16:30
*** openstackgerrit has joined #openstack-infra16:30
ihrachysfungi: I assume we should join name and version specifier. checking for yum.16:31
strigaziajo_ clarkb would the combination FIXED_RANGE= and SUBNETPOOL_PREFIX_V4= work?16:31
fungiihrachys: luckily in the case of our ci, at least, we always upgrade to the most recent versions of any installed or requested packages the system knows about so this shouldn't bite us there16:32
*** amitgandhinz has quit IRC16:32
*** e0ne has quit IRC16:32
ihrachysfungi: well, latest centos does not deliver needed sqlite16:32
clarkbstrigazi: you shouldn't use 12/8 like that. ATT apparently own at least part of that range16:32
ihrachysfungi: so if we would have unit test job for the platform, it would fail with the proposed bindep16:32
*** amitgandhinz has joined #openstack-infra16:32
fungiihrachys: right, and bindep won't fix that in our ci so you'd either need a newer centos release or a custom job to add those packages from somewhere16:33
*** mdrabe has quit IRC16:33
*** mdrabe has joined #openstack-infra16:33
ihrachysfungi: right. it's not going to fix anything, but it will at least give some clue to devs16:34
jesusaurrcarrillocruz: ah yeah, no reason to hardcode that in the iptables module16:34
fungiihrachys: agreed16:34
clarkbstrigazi: for your jobs specifically I think you just need to update how you talk to rabbit to use localhost16:35
clarkbstrigazi: unless you have multinode jobs too16:35
strigaziclarkb, we don't have multinode jobs16:35
*** daemontool has quit IRC16:35
jesusaurrcarrillocruz: do you want to push a change that passes the new ip addresses from system-config and depends-on my iptables change?16:35
fungiihrachys: anyway, you originally asked about use of -b in jenkins/scripts/install-distro-packages.sh so it's not really a concern there. if you have a convenience bindep env in your tox.ini i recommend not using -b so you have more useful and verbose output from the tool for local developer use anyway16:36
strigaziclarkb, and in conf we use to talk to rabbit16:36
ihrachysfungi: well, the thing is, once I land the file, gate starts using it :)16:36
ihrachysfungi: and since there is no output, it fails the jobs16:36
ihrachysfungi: I will send a patch for bindep16:36
strigaziclarkb, see here: http://logs.openstack.org/06/374906/7/check/gate-functional-dsvm-magnum-k8s/c63884d/logs/etc/magnum/magnum.conf.txt.gz16:36
clarkbstrigazi: are you sure that that is being used or that some other service isn't what is failing/16:37
rcarrillocruzjesusaur: yeah, i rather land your patch and then i push a new patch that sets new cacti via site.pp16:37
*** csomerville has joined #openstack-infra16:38
*** yamahata has joined #openstack-infra16:38
strigaziclarkb, I think yes, I keep getting timeout when the is a request in magnum's api server. The reason is timeout while waiting for the message16:39
fungiihrachys: can you point to the failure? if there's a job running bindep on centos 7 and you request for it to ensure sqlite>=3.8 is present, it's going to fail regardless (but i agree that a more verbose failure would be useful at any rate)16:39
clarkbstrigazi: could it be the remote end failing to respond rather than magnum itself? Basically this does work in the normal single node neutron jobs so I don't think its a general issue when you avoid the routing problem16:40
clarkbwhich implies to me that somehow we are getting hit by the routing problem here16:40
*** trown is now known as trown|lunch16:40
ihrachysfungi: any job fails, because -b returns 1 while no output16:40
ihrachysfungi: http://logs.openstack.org/06/381906/3/check/gate-neutron-pep8-ubuntu-xenial/148dcd8/console.html#_2016-10-04_15_57_36_84669916:40
fungiihrachys: oh! i completely misunderstood where this was breaking. that traceback is useful16:41
*** Ravikiran_K has joined #openstack-infra16:41
fungii thought you were saying it just silently reports a nonzero exit code in centos 7 jobs16:42
fungiihrachys: i think version comparisons may not even be properly implemented across all platforms, so we're going to be breaking new ground by trying to include them in lists. may need to make it more clear that's an experimental feature of the tool that still needs a lot of work16:43
openstackgerritIhar Hrachyshka proposed openstack-infra/bindep: Fixed bindep -b mode for version mismatch errors  https://review.openstack.org/38197916:43
ihrachysfungi: the fix would be similar to ^16:43
ihrachysfungi: checked locally for centos16:44
fungifor example, i think bindep currently uses rpm to query package lists on rh-specific platforms, so doesn't know whether there is even an available version to meet the requested requirement16:44
ihrachysseems to return what I want16:44
*** ijw has joined #openstack-infra16:44
ihrachysanyhow, need to run. please review if you have cycles. I will add tests later.16:44
fungiit can see what version is installed, but if for example sqlite isn't installed there yet then there is no version number known for it16:44
ihrachysfungi: ack I will need to check that16:45
*** fguillot has quit IRC16:45
fungithe feature was originally added back when bindep was debian-only, and how other distros track package versions wasn't taken into account at the time16:45
*** amitgandhinz has quit IRC16:45
AJaegerthanks, jeblair and pabelanger !16:45
fungiihrachys: so when we extended support to additional distros, version handling was left as something to add down the road when people needed it and figured out how to make it work16:46
ihrachysfungi: so here I am... I will need it :)16:46
fungiihrachys: yep! i'm excited to see us get it working better16:47
ihrachysfungi: I will chew it in next days16:47
*** ihrachys has quit IRC16:48
*** yamahata has quit IRC16:49
*** ijw has quit IRC16:50
*** Apoorva has joined #openstack-infra16:50
*** tiswanso has quit IRC16:50
*** sputnik13 has quit IRC16:53
mtreinishfungi, clarkb, rcarrillocruz: https://review.openstack.org/#/c/381960/ should be good to go I think16:54
mtreinishhopefully that's the last issue to unbreak health.o.o16:54
*** weshay_lunch is now known as weshay16:54
openstackgerritJordan Pittier proposed openstack-infra/shade: remove_router_interface: check subnet_id or port_id is provided  https://review.openstack.org/38198116:54
*** tnovacik has quit IRC16:55
*** yamahata has joined #openstack-infra16:57
*** weshay is now known as weshay_mtg16:58
*** ccamacho has left #openstack-infra16:59
*** vsaienko has joined #openstack-infra16:59
*** amitgandhinz has joined #openstack-infra17:00
*** strigazi is now known as strigazi_AFK17:00
*** narayrak has quit IRC17:00
*** ralonsoh_ has quit IRC17:00
*** derekh has quit IRC17:00
clarkbmtreinish: mriedem have you had a chance to look at why e-r is unhappy?17:00
openstackgerritThomas Goirand proposed openstack-infra/project-config: track-upstream for deb-{glare,zaqar-ui}  https://review.openstack.org/38198417:00
vsaienkohello clarkb, could you please check my comment at L142 and let me know what do you think about it: https://review.openstack.org/#/c/352790/14/devstack-vm-gate.sh@14217:00
openstackgerritJordan Pittier proposed openstack-infra/project-config: Enable py35 voting on project shade  https://review.openstack.org/38198517:01
openstackgerritJordan Pittier proposed openstack-infra/shade: List py35 in the default tox env list  https://review.openstack.org/38198717:03
mtreinishclarkb: I haven't yet. Do you have the paste handy again. I was afk for most of yesterday17:03
*** vhosakot has joined #openstack-infra17:03
*** adrian_otto has quit IRC17:03
jeblairpabelanger, AJaeger: http://docs-beta.openstack.org/17:03
fungimtreinish: did you confirm for 381960 that the cython package gets pandas sdists installing correctly?17:04
AJaegerjeblair: Woot! Thanks17:04
fungimtreinish: if not, i'll take a stab at making sure it works17:04
*** jordanP has quit IRC17:04
*** tuannguyen has quit IRC17:05
clarkbmtreinish: mriedem http://paste.openstack.org/show/584064/17:05
fungijeblair: that's served out of afs? snappy!17:05
mtreinishfungi: I don't have a trusty env up and running right now to easily confirm that17:05
clarkbvsaienko: ya I think its fine if you want to have a generic way to signal roles to devstack and devstack plugins17:06
pabelangerjeblair: AJaeger: looks good, thanks17:06
fungimtreinish: no worries, just making sure i'm not wasting my time by double-checking it17:06
clarkbvsaienko: re the ssh generic key thing, your comment doesn't really explain what it is used for17:06
jeblairfungi: yeah, the afs root is served from http://files.openstack.org/17:06
* AJaeger now fixes a few absolute references to docs.o.o ;)17:06
clarkbvsaienko: I am still confused at why we need another ssh key there. Devstack-gate is configuring both sides of the bridge for you using the test system ssh key17:06
clarkbvsaienko: is that needed because neutron has to talk to ironic using some sort of ssh driver and it is the key for that?17:07
jeblairfungi: (docs-beta is rooted one level down at /docs so folks can verify the site with /urls)17:07
vsaienkoclarkb: yes, neutron should talk via ssh to subnode17:07
clarkbvsaienko: I see17:07
*** krtaylor has joined #openstack-infra17:08
*** tiswanso has joined #openstack-infra17:08
vsaienkoI will update localrc variable names to be more generic17:08
clarkbvsaienko: ok so much of this setup is there to allow interaction between neutron on the controller and ironic on the subndoe?17:08
*** vsaienko has quit IRC17:08
*** jpich has quit IRC17:09
openstackgerritMerged openstack-infra/zuul: Combine shared queues explicitly  https://review.openstack.org/35800917:10
*** strigazi_AFK is now known as strigazi17:11
*** vsaienko has joined #openstack-infra17:11
*** tuannguyen has joined #openstack-infra17:11
strigaziclarkb, with FIXED_RANGE= and SUBNETPOOL_PREFIX_V4=$FIXED_RANGE all jobs succeeded in osic see here http://paste.openstack.org/show/584265/ Is it by accident?17:12
clarkbstrigazi: no, that likely would make it happy. My point earlier about fixed range is that we already set it to you shouldn't be overriding it to
clarkbstrigazi: the key is getting devstack to continue to honor that data rather than ignoring it17:13
vsaienkoclarkb: yes, we need ssh keys for two things. 1st ironic-conductor is talking to libvirt on subnode via ssh and manage VMs on subnode. 2nd neutron talks to subnode and change ovs settings17:14
*** vhosakot has quit IRC17:15
*** vhosakot has joined #openstack-infra17:15
clarkbvsaienko: is there ar eason the second one isn'g neutron talking to an agent over the existing message bus to manage that? I am guessing because those ovs settings are for the fake baremetal not for neutron itself?17:15
strigaziclarkb, is it an acceptable as a temporary solution until the problem is fixed by devstack/neutron ? I can't think of anything else :(17:16
clarkbstrigazi: if you want to work around it taht way don't change the FIXED_RANGe just assign it to the subnet pool17:16
clarkbI have been avoiding bandaiding over it like that because thsi really needs to be fixed in devstack imo17:16
strigaziclarkb, I comletely agree with that ^^17:17
vsaienkoclarkb: neutron ml2 driver is designed to talk via ssh, normally it talks to real switches. But on the gates we emulate ironic nodes by VMs, and ml2 driver talks to subnode via ssh17:17
strigaziclarkb, but I would like to follow this path as a temporary solution17:18
clarkbstrigazi: ok17:18
*** dtantsur is now known as dtantsur|afk17:18
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints  https://review.openstack.org/37804017:19
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add support for testing predictable placement  https://review.openstack.org/37801417:19
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test hostname map  https://review.openstack.org/37801717:19
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Fall back to previous failure list for older releases  https://review.openstack.org/38199817:19
fungimtreinish: looks like pandas has an implicit dependency on cython, the python library. apt-get installing that won't help without system site packages enabled in the venv, so instead we may need cython added to the requirements.txt for openstack-health. i'm testing that next17:19
vsaienkoclarkb if you are ok to setup ssh keys could you please vote here: https://review.openstack.org/#/c/36483017:20
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Do Not Merge - Test Undercloud upgrade mitaka -> newton  https://review.openstack.org/38130917:20
mtreinishfungi: :/ sigh, it's never simple17:21
openstackgerritMerged openstack-infra/shade: Add test for os_group Ansible module  https://review.openstack.org/37985417:21
*** ArxCruz has joined #openstack-infra17:21
*** strigazi is now known as strigazi_AFK17:22
*** tiswanso has quit IRC17:22
fungimtreinish: oh, worse, since pandas expects to find cython at install time, simply declaring it as a runtime requirement is likely insufficient. we probably need it as a setup_requires in fact to avoid a race where pip might try to install cython after pandas17:22
*** eharney has joined #openstack-infra17:24
mtreinishfungi: it's probably just simple to break the venv and system install openstack-health17:24
mtreinish*break out of17:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref  https://review.openstack.org/38200217:25
AJaegerannegentle: ^17:25
*** vsaienko has quit IRC17:26
*** tiswanso has joined #openstack-infra17:26
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add provider upload framework  https://review.openstack.org/37979517:26
*** vsaienko has joined #openstack-infra17:27
*** vhosakot_ has joined #openstack-infra17:27
*** baoli has joined #openstack-infra17:29
fungidstufft: stupid pip user question... we have a package with an install_requires on pandas. in turn, pandas expects cython to be available in the env to build its sdist but doesn't declare that. is it sufficient for our package to declare an install_requires on cython before pandas, or do we need to make cython a setup_requires in our package to ensure it's installed before pandas?17:29
*** vsaienko has quit IRC17:30
*** sambetts is now known as sambetts|afk17:30
*** vhosakot has quit IRC17:30
*** vsaienko has joined #openstack-infra17:32
*** esikachev has quit IRC17:33
*** flepied has joined #openstack-infra17:34
*** vsaienko has quit IRC17:36
*** EricGonc_ has quit IRC17:36
fungimtreinish: i'm going to submit a pr to pandas to add this as a setup_requires, but in the meantime https://review.openstack.org/38201017:37
*** flepied1 has quit IRC17:37
*** tosky has quit IRC17:37
*** baoli has quit IRC17:37
*** baoli has joined #openstack-infra17:38
mtreinishfungi: ok I'm +A on that, but I'm quickly gonna add a comment inline about it17:38
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref  https://review.openstack.org/38200217:38
fungimtreinish: feel free to push another patchset if you want, or i can in a moment17:39
fungihacking on pandas packaging now17:39
mtreinishfungi: actuall I don;t think that'll fix our issue17:39
*** eranrom has joined #openstack-infra17:39
mtreinishfungi: because we install requirements explicitly first17:39
*** pilgrimstack1 has quit IRC17:39
fungiindeed we do17:39
mtreinishfungi: http://git.openstack.org/cgit/openstack-infra/puppet-openstack_health/tree/manifests/api.pp#n10817:40
fungii merely confirmed that a pip install cython followed by pip install . worked correctly17:40
*** arxcruz_ has quit IRC17:40
*** jschwarz has quit IRC17:40
mtreinishfungi: we can put cython in o-h requirements right before panda17:40
*** piet_ has joined #openstack-infra17:40
mtreinishthat way we'll always install it first17:40
mtreinishor we can just cap pandas to a version before the implicit cython dep17:40
*** hamzy has quit IRC17:41
*** weshay_mtg is now known as weshay17:41
*** ArxCruz has quit IRC17:41
*** jschwarz has joined #openstack-infra17:42
fungii'm unsure that it's possible to have them in the same requirements list. i think pip may run setup.py on pandas before cython is fully installed into the env. or at least there used to be some races around trying to treat ordering as a substitute for setup_requires17:42
clarkbcloudnull: on further inspection it seems that part of the issue here may be that the ipv4 subnet in use on osic is a /8 as well17:42
clarkbcloudnull: which means that a new route for 10/8 supercedes the old one rather than being a less specific route that doesn't affect a smaller range17:43
mtreinishfungi: ok, then I'm fine just capping pandas17:43
fungithat may be the easiest route, yes17:43
mtreinishfungi: I think it's only an issue with pandas 0.19.0 which was released 2 days ago17:44
sc68caljeblair: ping - regarding raw IPv6 links via telnet - I swear there's an RFC - happy eyeballs? where we're supposed to give the ability to dual stack17:44
fungiin the meantime, the api seems to be working again with cython having been manually installed. we'll want to blow away .venv before we land a change with the pandas cap17:44
sc68caljeblair: and that raw IPv6 addrs are not really good user experience. Think maybe we can design a dual stack proxy?17:44
cloudnullclarkb: so then the only way around that issue is to use a different subnet. yes?17:45
clarkbcloudnull: yes a different smaller subnet17:45
clarkbfungi: ^ may have opinions on that too17:45
*** Goneri has quit IRC17:46
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add provider upload framework  https://review.openstack.org/37979517:47
*** yamamoto has joined #openstack-infra17:47
mtreinishfungi: https://review.openstack.org/#/c/382019/17:47
*** xyang1 has quit IRC17:47
jeblairsc68cal: there will be a websockets proxy in zuulv317:48
sc68caljeblair: ack. if I can help in any way hit me up17:49
*** drifterza has joined #openstack-infra17:49
*** fguillot has joined #openstack-infra17:49
jeblairsc68cal: (but as an aside, the hosts in question are not dual stack hosts -- they are actually ipv6 only)17:50
*** tnovacik has joined #openstack-infra17:51
*** claudiub has joined #openstack-infra17:51
sc68caljeblair: right - which is causing a UX issue since not everyone has v6 yet17:51
*** EricGonczer_ has joined #openstack-infra17:51
*** yamamoto has quit IRC17:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref  https://review.openstack.org/38200217:52
jeblairsc68cal: sure, i just want to be clear, there is no rfc that says "you must run all services on both ipv4 and ipv6".  the one you're thinking about discusses ways to mitigate delays on dual stack hosts with non-working v6 (which is not the situation we are in).  that was to address the whole "my browser is hung" thing.17:54
*** hamzy has joined #openstack-infra17:55
cloudnullclarkb: couldn't devstack simply use the fixed range or the both of which have been carved out of the osic 10/8 for folks that need a 10.x.x.x network?17:55
jeblairsc68cal: there are (slowly) increasing numbers of sites on the internet that are only accessible over ipv6 -- after all, that's really the crux of ipv4 address exhaustion17:55
clarkbcloudnull: yup, I think we should absolutely do that too17:55
cloudnullI think those are the subnets.17:55
* cloudnull goes to look17:55
clarkbcloudnull: but generally you don't want to have an l2 on a /817:55
jeblairsc68cal: having said *that* -- i'm not saying that we should start making our servers ipv6 only -- we try to dual-stack as many of them as possible17:55
*** tqtran has joined #openstack-infra17:55
sc68caljeblair: I'd have to remember the RFC - but basically having sites only accessible over v6 is sort of broken17:56
jeblairsc68cal: and so the zuul status page, gerrit, etc, are all ipv4 and ipv6 --17:56
jeblairsc68cal: well, i mean, i agree that if you want people to use your sites, it's probably a bad idea at this stage17:56
jeblairsc68cal: but i don't think there's anything *broken* about it -- it's really the whole point of ipv6 is to replace v4 eventually, and it's happening now regardless of what we do here :)17:56
sc68calright, and I think having raw IPv6 links to get event streams from the CI system causes UX frustration17:57
clarkband from a pragmatic standpoint this gives us access to 1k more test instances than we would have otherwise17:57
jeblairsc68cal: https://tools.ietf.org/html/rfc6555  is about the user experience of clients on dual-stack hosts17:57
jeblairsc68cal: i agree it causes frustration17:57
clarkb*we wouldn't have17:57
*** Sukhdev has joined #openstack-infra17:58
*** sdake has quit IRC17:58
sc68calclarkb: right, not arguing that - I just think having raw IPv6 links to the CI logs can be fixed with a proxy17:58
jeblairsc68cal: yep, we agree on this point17:58
clarkbugh did I get my english right the first time then I broke it?>17:58
clarkbI should find some brunch17:58
*** rcernin has joined #openstack-infra17:58
sc68calanyway I'd like to help if I am able17:58
jeblairsc68cal: but i also think that this is a great opportunity for folks to get an ipv6 connection -- tunnel brokers abound17:59
*** shamail has joined #openstack-infra17:59
AJaegerfungi, did you see my earlier comment about deb-glare?17:59
*** ccamacho has joined #openstack-infra18:00
*** moravec has joined #openstack-infra18:00
jeblairsc68cal: i think a lot of people see ipv6 as being very theoretical, and it's not -- as clarkb says, we have 1000 workers that we would not otherwise because of ipv618:00
AJaegermight have been the usual github issue as well - https://github.com/openstack/deb-glare is empty but git.openstack.org exists...18:00
*** cardeois has quit IRC18:00
sc68caljeblair: ok, I think though getting people to tunnel for v6 just to see CI logs is .... not ideal18:00
sc68calI mean hey, I'm all for v6 - remember who I am - but just want to avoid frustration18:00
AJaegersc68cal: they can get CI logs from logs.openstack.org via gerrit once the run finishes18:00
*** degorenko is now known as _degorenko|afk18:01
*** zz_ja is now known as zz_zz_ja18:01
AJaegersc68cal: the IPv6 addresses do not block anybody from getting work done. The streaming is nice and might help some to monitor what's going on - but we offer logs to everybody18:01
jeblairsc68cal: yep, and we would have put the websocket proxy in place already if the websocket situation in python2 wasn't so terrible18:01
*** tphummel has quit IRC18:02
*** tphummel has joined #openstack-infra18:02
jeblairsc68cal: but it's a big enough project that we deferred it to zuulv3, since the logs are accessible on v4 and v6 after the job completes18:02
*** cardeois has joined #openstack-infra18:02
*** tiswanso has quit IRC18:02
AJaegerinfra-root, could somebody fix https://github.com/openstack/deb-glare setup, please?18:03
*** harlowja has quit IRC18:03
jeblairsc68cal: (the current system was put in place very quickly thanks to some surprises with the old one)18:03
*** tiswanso has joined #openstack-infra18:03
AJaegermordred: https://review.openstack.org/#/c/381985 makes python35 voting on shade, hope that's ok for you...18:04
*** tuannguyen has quit IRC18:05
*** electrofelix has quit IRC18:06
*** adrian_otto has joined #openstack-infra18:06
*** drifterza has quit IRC18:07
sc68caljeblair: ack - thanks for bringing me up to speed18:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref  https://review.openstack.org/38200218:07
*** cardeois_ has joined #openstack-infra18:07
* AJaeger needs far too many iterations for such a change ;( ^18:07
* AJaeger calls it a day now and waves good-night18:08
fungimtreinish: well, an alternative would be to upgrade to pip 8.1.0 or later since they provide manylinux1 wheels on pypi18:08
jeblairsc68cal: thanks for the feedback.  and let me know how much time you have to work on zuulv3.  :)18:08
*** drifterza has joined #openstack-infra18:08
*** adrian_otto has quit IRC18:08
sc68caljeblair: ok let me pull down the code and start grepping :)18:08
fungimtreinish: also, my discussion with the maintainer in https://github.com/pydata/pandas/issues/14204 suggests that they're openly hostile to setup_requires and would rather building their sdist simply break than rely on them18:09
sc68caljeblair: is it just master branch of openstack-infra/zuul ?18:09
*** cardeois has quit IRC18:09
jeblairsc68cal: it's feature/zuulv3, and we're at the stage were there are no small jobs, only big refactorings happening.  but mordred has actually done quite a bit of the work on the websocket thing, it's just not the highest priority right now.18:10
jeblairsc68cal: (i don't think his work is in tree yet, we really need to focus on things like "execute jobs" first)18:11
*** pvaneck has joined #openstack-infra18:13
*** pilgrimstack has joined #openstack-infra18:13
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit  https://review.openstack.org/37511318:13
*** Goneri has joined #openstack-infra18:13
*** tuannguyen has joined #openstack-infra18:15
*** bin_ has quit IRC18:17
*** akshai has quit IRC18:18
mtreinishfungi: hmm, we don't have pip 8.1.0 on health.o.o?18:19
dragonmasterasselin__: thanks for the help18:19
fungimtreinish: nope, it's still on 7.1.018:19
*** EricGonczer_ has quit IRC18:19
dragonmasteri see that zuul has a state_dir setting, and this is a configurable item in the config18:19
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref  https://review.openstack.org/38200218:20
dragonmasteri think it is what i was missing18:20
fungimtreinish: and well, the version of virtualenv is what's important here as it bundles its own copy of pip, but it's bundling pip 7.1.018:20
*** e0ne has joined #openstack-infra18:20
mtreinishfungi: the more I think about this, the more I'm thinking we want to stop using a venv there18:20
mtreinishit just complicates things18:20
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Add tool for aborting jobs in jenkins  https://review.openstack.org/38202918:20
fungimtreinish: makes sense. i'm happy going either way with it18:20
mtreinishit was a low prio todo for me to stop that, maybe I'll just do that after I find some food18:21
*** trown|lunch is now known as trown18:21
mtreinishfungi: we'll still want to upgrade the pip version, but it makes doing upgrades simpler if we don't have to reconstruct the venv all the time18:21
fungimtreinish: regardless, upgrading to a newer pip/virtualenv will allow us to get manylinux1 wheels pf pandas rather than needing to preinstall cython and numpy18:21
jeblairfungi: aren't manylinux1 things compiled lowest-common-denominator?18:22
drifterzaheya dragonmaster :)18:22
jeblair(so perhaps preinstalling those and letting things be compiled is better?)18:22
fungijeblair: yeah, i think the spec is loosely "rhel 5 with only glibc and a handful of very common libraries"18:23
dragonmasterhey drifterza18:23
drifterzayou winning ?18:23
*** kzaitsev_mb has quit IRC18:26
*** rohanion has quit IRC18:26
*** dtardivel has quit IRC18:27
*** shardy is now known as shardy_afk18:27
*** Naeil has quit IRC18:28
*** EricGonczer_ has joined #openstack-infra18:29
*** docaedo has quit IRC18:29
*** amoralej is now known as amoralej|off18:29
*** docaedo4 has joined #openstack-infra18:29
*** moravec has quit IRC18:29
*** jpena is now known as jpena|off18:30
dragonmastergetting there slowly18:30
*** tuannguyen has quit IRC18:30
*** Ravikiran_K has quit IRC18:31
*** adrian_otto has joined #openstack-infra18:31
*** hasharAway is now known as hashar18:32
fungiAJaeger: i'm looking into the deb-glare failure now18:32
*** senk_ has quit IRC18:33
*** bnemec has quit IRC18:34
clarkbfungi: ya https://www.python.org/dev/peps/pep-0513/ the list is in there and relatively small and very old18:35
*** tuannguyen has joined #openstack-infra18:35
openstackgerritFrédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible  https://review.openstack.org/37159018:36
clarkbfungi: which is why the pyerasurecoode project shouldn't publish a wheel since liberasurecode is not part of that list18:39
openstackgerritMerged openstack-infra/elastic-recheck: Switch coverage tox job to use pbr test command  https://review.openstack.org/38189818:39
openstackgerritMerged openstack-infra/elastic-recheck: Remove discover from test-requirements  https://review.openstack.org/34579418:40
fungimtreinish: seems i've talked a pandas maintainer into the need for a setup_requires on cython finally, fwiw, so will go ahead with a pr after meetings18:41
clarkbfungi: mtreinish catching up basically system install of cython won't help beacuse health runs in a venv and thats where we need cython?18:42
*** caphrim007 has quit IRC18:42
*** Rockyg has joined #openstack-infra18:43
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add reno jobs for python-tripleoclient  https://review.openstack.org/38204118:45
openstackgerritMerged openstack-infra/zuul: Re-enable the shared/independent queue test  https://review.openstack.org/35801018:45
openstackgerritMerged openstack-infra/zuul: Handle a project-template with no jobs  https://review.openstack.org/36078318:45
openstackgerritMerged openstack-infra/zuul: Add an OpenStack test  https://review.openstack.org/36078418:45
openstackgerritMerged openstack-infra/zuul: Pass assigned nodes to the launcher  https://review.openstack.org/36078518:45
openstackgerritMerged openstack-infra/zuul: Add a python27 job variant to the OpenStack test  https://review.openstack.org/36078618:46
openstackgerritMerged openstack-infra/zuul: Re-enable test_failed_change_at_head  https://review.openstack.org/36382118:46
openstackgerritMerged openstack-infra/zuul: Add nodeset as a top-level configuration item  https://review.openstack.org/36508918:46
*** Teresita-Warrior has quit IRC18:47
*** ijw has joined #openstack-infra18:47
mriedemclarkb: on that e-r failure i'm seeing this18:47
mriedem"[FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]"18:47
openstackgerritMerged openstack-infra/zuul: Use NodeSet objects in jobs and NodeRequests  https://review.openstack.org/36509918:47
*** salv-orlando has quit IRC18:48
mriedemmtreinish: ^18:48
*** vhosakot_ has quit IRC18:48
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/gear: Support _ALL and _REDUCE_ jobs from protocol  https://review.openstack.org/38204218:48
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/gear: WIP: Handle new STATUS commands from protocol  https://review.openstack.org/38204318:48
SpamapSjeblair: ^^ 382042 talks to libgearman again. :-D18:49
*** vhosakot has joined #openstack-infra18:49
clarkbmriedem: oh ok maybe I need to make sure indexes are claened up again18:49
SpamapSmorgan: ^ you're playing with gear too right? 38204218:49
mriedemclarkb: i'll work on fixing that IndexError18:50
clarkbmriedem: I amseeing that in the ES logs too18:50
*** akshai has joined #openstack-infra18:50
mriedemclarkb: yeah i'm able to recreate locally18:50
mriedemtox -r -e venv -- elastic-recheck-uncategorized --verbose18:50
openstackgerritMerged openstack-infra/elastic-recheck: Enable more reasonable logging  https://review.openstack.org/36016418:50
clarkbmriedem: I deleted our oldest index18:51
*** spzala has quit IRC18:51
clarkbit was really small and doesn't seem to have corrected the issue18:51
clarkbwent ahead and did the next one too since it was much larger18:52
*** ijw has quit IRC18:52
*** spzala has joined #openstack-infra18:52
*** thorst has quit IRC18:54
*** thorst has joined #openstack-infra18:55
anteayanjohnston: the understanding that I was left with from the discussion about having a private channel for election questions was that queries would be made in public in the #openstack-dev channel with people emailing allo the officials directly for private matters18:55
anteayatonyb: was going to remove the reference to the private channel from the documentation18:55
*** adrian_otto has quit IRC18:55
clarkbmriedem: I haven't seen anymore instances of that error in the logs. As for handling it e-r side I don't expect your query to end up being happy but thats per index I think. So one way to potentially handle it is log the error but then go on to the next index and just try to do what you can18:56
clarkbmriedem: basically if that error shows up on an index it will likely always persist on that index but moving to the next index may end up working18:56
*** spzala has quit IRC18:57
njohnstonanteaya: Yes, that is my understanding as well.  I had changes in to add logging to the elections channel, but those were abandoned.18:57
njohnstonasa result of those discussions18:57
anteayanjohnston: then why are you instructing people to use the private channel to communicate with election officials regarding missing ballots?19:00
fungiit's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour19:00
*** spzala has joined #openstack-infra19:01
*** rwsu_ has joined #openstack-infra19:01
anteayaand for what its worth I also did not recieve my second ballot, by way of a data point, but I did recieve my first one so can still vote19:01
*** docaedo has joined #openstack-infra19:02
*** eranrom has quit IRC19:02
*** vhosakot has quit IRC19:03
*** vhosakot has joined #openstack-infra19:03
*** rwsu has quit IRC19:04
*** spzala has quit IRC19:05
*** xyang1 has joined #openstack-infra19:05
njohnstonanteaya: Glad that you got a ballot!  I contacted Andrew Myers, the person who runs CIVS, and he said that it seems the Cornell University SMTP servers are either delaying or discarding some outbound email.19:06
*** esikachev has joined #openstack-infra19:07
anteayacan you get any logs from him to find out what email is delayed or discarded?19:07
njohnstonanteaya: And about #openstack-election, the notice that tonyb sent out referred people to http://governance.openstack.org/election/#election-officials which still indicates #openstack-election; I did not want to cause confusion by differing from those instructions.19:07
anteayanjohnston: well how about the election officials get together and discuss?19:08
anteayasince the understand I currently hold is that the private channel isn't to be used for queries19:08
njohnstonanteaya: Will do.19:09
anteayaso it would be nice to have some consistentcy amoungst the election officials19:09
anteayathank you19:09
anteayaand thank you for emailing andrew myers, if he is willing to share logs of our email runs we can see if we can identify which email addresses never recieved a ballot19:10
njohnstonanteaya: I've emailed him, hopefully he will be able to differentiate our emails from the other elections CIVS runs and be able to furnish us some logs without impinging on the privacy of someone else19:13
njohnston's poll19:13
openstackgerritDaniel Castellanos proposed openstack-infra/project-config: Update Jenkins jobs for craton-dashboard  https://review.openstack.org/38132719:13
anteayanjohnston: well if you give him all the details especially the timestamps the ballots were run, I am guessing that would help him narrow down our runs19:13
anteayaand thank you19:14
njohnstonanteaya: My pleasure.19:14
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: Full quickstart gate run on OVB  https://review.openstack.org/38109419:14
mriedemclarkb: i'm working on a patch19:15
*** tuannguyen has quit IRC19:20
*** david-lyle has quit IRC19:22
*** cardeois_ has quit IRC19:22
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Skip hits that don't have data  https://review.openstack.org/38204919:23
mriedemclarkb: ^ is hacky but i'm leaving soon19:23
mriedemand fixes the immediate breakage19:23
mriedemmtreinish: ^19:23
*** cardeois_ has joined #openstack-infra19:26
mtreinishclarkb: yeah we need cython in the venv to build pandas19:27
*** vhosakot has quit IRC19:27
mtreinishmriedem: ok I'll take a look19:27
*** spzala has joined #openstack-infra19:28
*** rkukura has quit IRC19:29
*** cardeois_ has quit IRC19:32
*** mriedem has quit IRC19:32
openstackgerritFrédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible  https://review.openstack.org/37159019:33
clarkbmtreinish: mriedem e-r looks like it may still be failing after the index cleanup19:33
clarkbI still get an index out of range error19:33
clarkbbut no new logs about the memory issue. Maybe those two things are unrelated?19:34
mat128fguillot: lgtm :)19:34
*** ijw has joined #openstack-infra19:35
openstackgerritDaniel Castellanos proposed openstack-infra/project-config: Update Jenkins jobs for craton-dashboard  https://review.openstack.org/38132719:36
*** cardeois has joined #openstack-infra19:37
*** amitgandhinz has quit IRC19:39
*** ijw has quit IRC19:40
*** matrohon has joined #openstack-infra19:40
asselin__dragonmaster, cool, glad to hear you found the config setting19:40
*** amitgandhinz has joined #openstack-infra19:40
mnaseris tarballs.openstack.org having issues?  it seems like the connectivity is being dropped mid-file download19:41
*** shardy_afk is now known as shardy19:41
mnaser"2016-10-04 19:39:41 (1.57 MB/s) - Connection closed at byte 150487655. Retrying."19:41
*** jkilpatr has quit IRC19:42
*** cardeois_ has joined #openstack-infra19:42
*** cardeois has quit IRC19:44
*** woodster_ has joined #openstack-infra19:45
anteayamnaser: can you share more of your log?19:45
mnaserits just a simple wget that closed, happened when i wget'd the file on my local machine too19:45
*** Apoorva has quit IRC19:47
*** Apoorva has joined #openstack-infra19:48
*** Apoorva has quit IRC19:48
*** Apoorva has joined #openstack-infra19:48
*** jkilpatr has joined #openstack-infra19:48
morganSpamapS: I was... Haven't been lately.19:49
openstackgerritFrédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible  https://review.openstack.org/37159019:50
clarkbmnaser: I see a small number of segfaults in the apache log19:50
clarkbthats interesting19:50
*** moravec has joined #openstack-infra19:51
*** salv-orlando has joined #openstack-infra19:51
SpamapSmorgan: ah cool. It doesn't get many reviewers, so if you have a chance, take a gander.19:51
*** rkukura has joined #openstack-infra19:51
mtreinishclarkb: I just tried mriedem's change locally and it seems to workaround the issue19:51
SpamapSI'm looking at coalescing now19:51
mtreinishclarkb: my only concern with it is that it is silent about the bad indexes19:51
*** Teresita-Warrior has joined #openstack-infra19:52
clarkbmtreinish: ok, I am seeing the error continue on status.o.o but the bad indexes should be gone19:52
clarkbmtreinish: so maybe its some other issue too?19:52
mtreinishclarkb: hmm, yeah I'm not sure19:53
*** nadya has joined #openstack-infra19:54
*** eharney has quit IRC19:55
*** salv-orlando has quit IRC19:55
*** moravec has quit IRC19:57
*** salv-orlando has joined #openstack-infra19:59
*** kzaitsev_mb has joined #openstack-infra19:59
*** ArxCruz has joined #openstack-infra19:59
*** salv-orlando has quit IRC19:59
*** salv-orlando has joined #openstack-infra20:00
*** dprince has quit IRC20:01
*** csomerville has quit IRC20:02
*** shardy has quit IRC20:03
*** Teresita-Warrior has quit IRC20:03
*** ArxCruz has quit IRC20:04
*** jkilpatr has quit IRC20:06
*** karthikp has quit IRC20:07
*** david-lyle has joined #openstack-infra20:07
*** eharney has joined #openstack-infra20:08
*** ccamacho has left #openstack-infra20:08
*** cardeois has joined #openstack-infra20:08
*** Goneri has quit IRC20:09
*** e0ne has quit IRC20:09
rcarrillocruzcan i get more eyes on https://review.openstack.org/#/c/362490/1 ? if there are no objections i can babysit20:09
*** cardeois_ has quit IRC20:09
*** adrian_otto has joined #openstack-infra20:09
*** karthikp has joined #openstack-infra20:09
*** shamail has quit IRC20:10
*** cardeois_ has joined #openstack-infra20:11
*** pilgrimstack has quit IRC20:11
*** cardeois has quit IRC20:13
pleia2rcarrillocruz: lgtm, feel free to +A at will20:14
*** harlowja has joined #openstack-infra20:15
rcarrillocruzjesusaur: ^ approving your parameterize snmp stuff for puppet-iptables20:15
*** mixos has joined #openstack-infra20:16
*** cardeois has joined #openstack-infra20:17
*** cardeois_ has quit IRC20:18
*** david-lyle has quit IRC20:19
*** cardeois has quit IRC20:19
*** david-lyle has joined #openstack-infra20:19
*** jkilpatr has joined #openstack-infra20:20
*** cardeois has joined #openstack-infra20:20
*** tuannguyen has joined #openstack-infra20:21
openstackgerritMerged openstack-infra/puppet-iptables: Parameterize SNMP source hosts  https://review.openstack.org/36249020:21
*** cardeois_ has joined #openstack-infra20:21
*** phschwartz has quit IRC20:22
*** mordred has quit IRC20:23
*** cardeois has quit IRC20:25
*** tuannguyen has quit IRC20:25
jesusaurawesome, thanks20:26
*** esikachev has quit IRC20:30
*** coolsvap has quit IRC20:32
*** dfflanders has joined #openstack-infra20:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update IPs and metadata for compute057.chocolate  https://review.openstack.org/38206120:32
dstufftfungi: uhhh20:33
dstufftfungi: I think the answer is neither, at least if it works it only works on accident20:33
*** Teresita-Warrior has joined #openstack-infra20:33
openstackgerritFrédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible  https://review.openstack.org/37159020:33
*** nadya has quit IRC20:34
*** Jeffrey4l_ has joined #openstack-infra20:34
*** cardeois has joined #openstack-infra20:35
*** john-davidge has joined #openstack-infra20:35
*** Jeffrey4l has quit IRC20:35
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update IPs and metadata for compute055.chocolate  https://review.openstack.org/38206520:36
*** cardeois_ has quit IRC20:37
fungidstufft: got it. after some back and forth i think i convinced a pandas maintainer that adding cython to their setup_requires rather than just aborting setup.py after attempting to import it was slightly better for their users20:37
*** cardeois_ has joined #openstack-infra20:39
*** shardy has joined #openstack-infra20:39
*** john-davidge has quit IRC20:39
*** cardeois has quit IRC20:39
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update IPs and metadata of compute056.chocolate  https://review.openstack.org/38206620:39
*** cardeois has joined #openstack-infra20:42
*** cardeois_ has quit IRC20:43
*** dragonmaster has quit IRC20:45
EmilienMhi, any chance to have a last review on https://review.openstack.org/#/c/381322 please?20:45
*** kgiusti has quit IRC20:47
clarkbEmilienM: quick question about that. Does tripleo install nova from source?20:48
clarkbif not that job isn't going to help much20:48
EmilienMclarkb: yes20:48
EmilienMclarkb: we build an rpm by using a tool called delorean20:48
pleia2EmilienM: lgtm20:48
EmilienMpleia2: merci!20:48
*** cardeois has quit IRC20:49
clarkbEmilienM: and that rpm is built using the nova change under test and not just eg master?20:49
*** phschwartz has joined #openstack-infra20:49
EmilienMclarkb: right20:49
clarkblooks like pleia2 got the approval so I have just responded to mriedems question about voting20:49
EmilienMthanks you :)20:49
pleia2he mentioned in the comments that he'd talked to someone too20:50
*** cardeois has joined #openstack-infra20:52
EmilienMclarkb: if you're interested by how delorean works, here's the source code https://github.com/openstack-packages/DLRN20:52
EmilienMclarkb: we build packages at every commit in OpenStack https://trunk.rdoproject.org/centos7-master/report.html20:53
clarkbEmilienM: ya I have vague familiarity with what it does20:53
*** mordred has joined #openstack-infra20:53
openstackgerritMerged openstack-infra/project-config: Add TripleO multinode in Nova experimental jobs  https://review.openstack.org/38132220:53
rcarrillocruzi have 6 nodes that textcons interface shows 'Monitor is in graphics mode'20:53
*** cardeois has quit IRC20:53
clarkbEmilienM: was curious if the jobs take into account the change to eg nova under test20:54
rcarrillocruzi need some jumphost to use the iLO graphical console to see what's going on20:54
*** chem has quit IRC20:54
clarkbrcarrillocruz: time to get a vga cable?20:54
*** cardeois has joined #openstack-infra20:54
EmilienMclarkb: yes, even with Depends-On other projects20:54
EmilienMclarkb: like Cinder, etc20:54
EmilienMclarkb: so if you submit a patch in nova and use Depends-On python-novaclient, it will build both packages20:54
*** gyee has joined #openstack-infra20:55
clarkbEmilienM: perfect20:56
clarkbEmilienM: also someone pointed out to me that delorean is using review.o.o instaed of git.o.o for git clones? maybe we can change that?20:56
clarkbjeblair: mordred the stack starting at https://review.openstack.org/#/c/371679/6 could probably use some attention to get in finally20:56
clarkbrcarrillocruz: ^ may interest you as well20:57
openstackgerritMerged openstack-infra/system-config: Update IPs and metadata for compute057.chocolate  https://review.openstack.org/38206120:57
EmilienMclarkb: I didn't know that. Maybe dmsimard knows?20:57
rcarrillocruzlet me see20:57
*** adriant has joined #openstack-infra20:57
* dmsimard reading20:57
clarkbrcarrillocruz: there was a great big push on that right before we all got on planes to germany then that caused the work to stall20:58
clarkbor at least the reviews20:58
rcarrillocruzi miss all the fun!20:58
* rcarrillocruz sobs20:58
dmsimardclarkb, EmilienM: yeah I saw that too todya ? I have no clue where that comes from, our config definitely uses git.o.o.20:58
dmsimardIf it uses review.o.o it's a bug and we'll address it but we have to find where it's picking that up20:59
*** pilgrimstack has joined #openstack-infra20:59
clarkbfungi: pabelanger tyty for reviewing the change to fix nova net multinode testing20:59
dmsimardbkero: can you let me know where you saw the git clone from review.openstack.org ?20:59
bkerodmsimard: How's this? https://github.com/redhat-openstack/ansible-role-tripleo-gate/blob/master/tasks/dlrn-build.yml#L2921:00
dmsimardwell that's not delorean at all :)21:00
rcarrillocruzclarkb: i've done mental variable substitutions and lgtm21:00
bkerodmsimard: Yeah, had to go read the code to determine what was actually doing the cloning.21:01
dmsimardbkero: adarazs is the guardian of that I think, I'll poke him21:01
bkerogood luck21:01
*** mhickey has quit IRC21:01
clarkbrcarrillocruz: ya I kind of wanted jeblair to do a once over on it just because he was the one exercising it when it came up21:02
*** krtaylor has quit IRC21:02
*** rfolco has quit IRC21:02
*** pilgrimstack has quit IRC21:04
*** rkukura_ has joined #openstack-infra21:04
*** david-lyle has quit IRC21:04
*** tuannguyen has joined #openstack-infra21:04
clarkbrcarrillocruz: ansible 2.2 final realse should happen at the end of the week or early next week?21:05
rcarrillocruzhmm, i expect at least a couple more RCs, mordred Shrews  ?21:05
rcarrillocruzsuperexcited about the include_role as a task thing21:05
clarkb(I am going off of their release process doc which says 5 days for RC1 and 2 days for subsequent RCs21:05
dmsimardrcarrillocruz: what?21:06
clarkbI just want the bug fix for the emtpy inventory group :)21:06
dmsimardrcarrillocruz: an include_role task ?21:06
rcarrillocruzthat efectively allows to call a role from a role21:06
dmsimardrcarrillocruz: omg21:06
dmsimardrcarrillocruz: not just that21:06
dmsimardrcarrillocruz: it'd allow you to wrap roles inside a block21:06
*** rkukura has quit IRC21:06
*** rkukura_ is now known as rkukura21:06
dmsimardwith rescues21:06
clarkbdmsimard: oh thats shiny21:06
rcarrillocruzdmsimard: https://docs.ansible.com/ansible/include_role_module.html :D21:06
rcarrillocruzand it allows to also change the entry point of a role21:06
rcarrillocruze.g. go to main.yml or go to main2.yml21:07
*** cody-somerville has joined #openstack-infra21:07
dmsimardrcarrillocruz: https://github.com/ansible/ansible/issues/1358721:07
clarkbbasically I think we will be ready to turn on tls in the multinode jobs as soon as ansible 2.2 is available as it fixes a bunch of bugs necessary to do the setup cleany21:07
rcarrillocruzhehe yeah21:07
rcarrillocruzand it's also promising their claim of performance21:07
dmsimardrcarrillocruz: that is super awesome and could've been easy to miss, thanks so much for letting me know about that .. it'll allow me to clean up so much of my code21:08
*** tuannguyen has quit IRC21:09
rcarrillocruzi'm going to make use of this for the launcher...will use it for my long standing pre/post create/delete tasks per resource action item21:10
*** thorst has quit IRC21:10
clarkbthough if people are willing to do reviews today I have hacked around the 2.1 ansible issues21:10
*** rtheis has quit IRC21:11
clarkbhttps://review.openstack.org/#/c/373219/13 is the change needed to bootstrap tls for multinode and ansible 2.1, its child change uses the current 2.2 rc1 to test it simplified with ansible 2.221:11
clarkband https://review.openstack.org/#/c/372374/ incorporates the whole stack with tls forced on to see that it actually does work21:12
*** trown is now known as trown|outtypewww21:12
jeblairclarkb, rcarrillocruz: i'm game for +3ing pansible if we're willing to risk a puppetmaster stoppage if something goes wrong21:13
jeblairi will be around for several more hours to deal with potential fallout21:13
clarkbjeblair: I don't know of any very urgent puppet related things right now21:13
Shrewsclarkb: rcarrillocruz: depends on what issues are found. i would expect a few more RCs, personally21:13
clarkband its a couple days before release when we likely should prvent that. All that to say I think its reasonably safe and we will know what to revert if anything pop ups21:14
openstackgerritMerged openstack-infra/system-config: Update IPs and metadata for compute055.chocolate  https://review.openstack.org/38206521:14
rcarrillocruzi will be around at least one hour21:14
clarkbjeblair: I will be around too, if you want to go ahead and +A the stack21:14
rcarrillocruzso we can probably just go for it21:14
jeblairclarkb: all approved21:15
*** shardy has quit IRC21:15
*** tiswanso has quit IRC21:15
*** matrohon has quit IRC21:15
*** raildo has quit IRC21:15
openstackgerritMerged openstack-infra/system-config: Update IPs and metadata of compute056.chocolate  https://review.openstack.org/38206621:15
fungiyeah, fine by me. we're not changing anything ci-related (worst case we might break job updates)21:15
jeblairaww, ansible-role-puppet doesn't share a queue with system-config21:15
*** ldnunes has quit IRC21:16
jeblair(i think we discussed that maybe we should throw an integration test into that repo though)21:16
clarkbShrews: ok I will go ahead and get this working with 2.1 then21:16
jeblairso we'll need to poke the system-config changes after the role changes land21:16
openstackgerritNate Potter proposed openstack-infra/project-config: Rename RSC project to Valence  https://review.openstack.org/38207221:17
openstackgerritMerged openstack-infra/ansible-role-puppet: Make sure environment winds up in basemodulepath  https://review.openstack.org/37167921:17
openstackgerritMerged openstack-infra/ansible-role-puppet: Pass environment to puppet module  https://review.openstack.org/37171021:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Allow any logdest  https://review.openstack.org/37176421:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Add noop argument  https://review.openstack.org/37177421:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Simplify puppet cmd generation  https://review.openstack.org/37177821:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Add debug argument  https://review.openstack.org/37181121:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Don't set detailed-exitcodes when noop  https://review.openstack.org/37182721:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Sync puppet module changes from upstream ansible  https://review.openstack.org/37183521:18
openstackgerritMerged openstack-infra/ansible-role-puppet: Remove basemodulepath from environment config  https://review.openstack.org/37181221:18
*** vhosakot has joined #openstack-infra21:19
*** thorst has joined #openstack-infra21:19
*** r-daneel has joined #openstack-infra21:20
*** senk_ has joined #openstack-infra21:20
*** docaedo4 has quit IRC21:21
*** thorst has quit IRC21:24
*** dfflanders has quit IRC21:25
*** adam_g` is now known as adam_g21:25
*** jkilpatr has quit IRC21:26
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Do Not Merge - Test Undercloud upgrade mitaka -> newton  https://review.openstack.org/38130921:27
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Undercloud upgrade for mitaka and newton  https://review.openstack.org/38128621:27
*** dfflanders has joined #openstack-infra21:28
*** burgerk__ has joined #openstack-infra21:29
*** jheroux has quit IRC21:30
*** akshai has quit IRC21:30
erlonclarkb: Im trying to understand, one of those tests, despite of being checked out in the gate are not running21:30
erlonclarkb: does the tag: @test.attr(type='slow') affects in something?21:31
erlonclarkb: if not what does it implies?21:31
*** mixos has quit IRC21:31
erlonclarkb: sure, if you know21:31
clarkberlon: I think that means its a "slow" test for the long periodic run21:32
clarkbwhich may be filtered out for the shorter gate runs21:32
erlonclarkb: long periodic run?21:32
rcarrillocruzk, three more deploys from servers i rescued from limbo, aka they were not inventoried yet on hiera21:32
*** burgerk_ has quit IRC21:32
clarkbthere is a job or two tha run the long tests periodically (like once a day) that aren't run on every change21:33
*** burgerk__ has quit IRC21:33
*** david-lyle has joined #openstack-infra21:33
*** vhosakot_ has joined #openstack-infra21:34
*** vhosakot has quit IRC21:35
erlonclarkb: hmmm that makes sense if it is not running21:35
*** moravec has joined #openstack-infra21:35
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add getProjectBranches to Source  https://review.openstack.org/38207621:36
openstackgerritJames E. Blair proposed openstack-infra/zuul: Implement branch-specific in-repo job config  https://review.openstack.org/38207721:36
jeblairclarkb: the system config patches failed21:37
* clarkb digs up system config patches21:38
jeblairclarkb: https://review.openstack.org/37176221:38
jeblairclarkb: http://logs.openstack.org/62/371762/14/gate/gate-infra-puppet-apply-centos-7/a20fd49/console.html21:38
*** amitgandhinz has quit IRC21:38
*** senk_ has quit IRC21:38
*** mixos has joined #openstack-infra21:39
clarkbcompile error?21:39
*** eharney has quit IRC21:40
clarkbnewer ansible relies on newer paramiko which needs cryptography21:40
clarkband before we weren't using ansible in the apply test21:40
clarkbso I think we can just put a bindep update to include libffi-dev under that stack21:40
jeblairand that changed since 2 weeks ago?21:40
*** cody-somerville has quit IRC21:40
clarkbI don't think the ansible side changes that recently but our bindep side may have21:41
*** annegentle has quit IRC21:41
jeblairother jobs seem to be failing for different reasons?21:41
*** matt-borland has quit IRC21:41
pabelangerI can confirm the libffi-dev / libssl-dev issue with ansible21:42
*** jamesdenton has quit IRC21:42
*** david-lyle has quit IRC21:42
clarkbah ya the ubuntu ones seem to be getting past that issue (I am guessing that a wheel or similar may be helping us work there)21:42
*** vhosakot_ has quit IRC21:42
jeblaircentos7 and f24 failed on the compile error21:42
jeblairthe others are failing on not finding the puppet manifest file (???)21:43
clarkbjeblair: http://logs.openstack.org/62/371762/14/gate/gate-infra-puppet-apply-ubuntu-xenial/d55b79d/console.html#_2016-10-04_21_20_01_054895 we seem to still hit it on ubuntu21:43
clarkbits just not fatal for some reason which is a little weird21:43
jeblairclarkb: weird... i think i still suspect a second problem21:45
*** jcoufal_ has quit IRC21:45
jeblaircause it doesn't seem like that error would cause the 'manifest file not found' error21:45
jeblair(but maybe?  through some very convoluted process?)21:45
*** david-lyle has joined #openstack-infra21:47
*** krtaylor has joined #openstack-infra21:47
clarkbjeblair: all of the failed ones also have a .final file21:47
*** david-lyle has quit IRC21:48
*** akshai has joined #openstack-infra21:48
*** david-lyle has joined #openstack-infra21:48
jeblairclarkb: yeah, i think that's just part of the process of generating the input21:48
jeblairclarkb: (eg, ".final" is what we give to puppet after doing sed, etc)21:48
*** docaedo4 has joined #openstack-infra21:48
*** akshai has quit IRC21:49
jeblairclarkb: and the 1:2 between .final and .final.out.FAILED means "all puppet runs are failing"21:49
jeblairer 1:121:49
clarkbya but only 3 of them are in that state (have a .final and also happen to have a .FAILED)21:49
clarkboh I understand21:49
clarkbthsoe are the only ones that matched the distro release specifier21:49
jeblairclarkb: probably means only 3 node defs matched21:49
clarkbya I agree I think there is something else going on there21:50
clarkbit does look like its looking for a relative path to the manifest file, are we sure that when asnible does its thing the resulting pwd is in that dir?21:50
*** gordc has quit IRC21:51
openstackgerritJames E. Blair proposed openstack-infra/ansible-role-puppet: DNM: revert some commits  https://review.openstack.org/38208121:52
openstackgerritJames E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test  https://review.openstack.org/37176221:52
jeblairclarkb: two commits were added to the end of the ansible-role-puppet stack, but i don't think they were tested21:52
jeblairthat reverts them to see if they were the cause21:52
clarkbjeblair: ah ok21:53
clarkbreading where we run ansible there is a sudo but that doesn't change pwd21:53
clarkband the role doesn't seem to do anything magical to get the dir set21:54
clarkbits possible we want to set a full path for that file21:54
*** Rockyg has quit IRC21:55
jeblairokay, not the result of those commits21:55
*** lezbar has quit IRC21:55
*** thorst has joined #openstack-infra21:55
jeblairi'll work up a change to set the absolute path21:56
*** lezbar has joined #openstack-infra21:56
clarkbI think we want a change to bindep.txt to add in the proper libffi-dev deps and ^21:56
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: enable undercloud/ssh on multinode jobs  https://review.openstack.org/38208221:56
clarkbI can work on bindep.txt if you want21:56
jeblairclarkb: ++21:56
*** fguillot has quit IRC21:58
*** piet_ has quit IRC21:58
openstackgerritJames E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test  https://review.openstack.org/37176221:58
*** mixos has quit IRC21:58
clarkbjeblair: where should I "place" the change?21:58
*** spzala has quit IRC21:59
*** adrian_otto has quit IRC21:59
jeblairclarkb: bottom of the stack?21:59
jeblair(i had to make the pwd change in 762)21:59
jeblairso let's rebase 762 on the bindep change21:59
*** adrian_otto has joined #openstack-infra22:00
*** thorst has quit IRC22:00
openstackgerritClark Boylan proposed openstack-infra/system-config: Use ansible for puppet apply test  https://review.openstack.org/37176222:01
openstackgerritClark Boylan proposed openstack-infra/system-config: Add libffi dev packages needed for ansible install  https://review.openstack.org/38208622:01
clarkbjeblair: ^ there reabsed22:01
*** senk_ has joined #openstack-infra22:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Implement branch-specific in-repo job config  https://review.openstack.org/38207722:02
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add getProjectBranches to Source  https://review.openstack.org/38207622:02
mnaserfyi tarballs.openstack.org still dropping connections for $some_reason22:02
mnaser4 failures during the download of the manila image (wget resume helps but yea)22:02
clarkbmnaser: does it always fail at the same spot?22:03
mnasergood question let me check logs clarkb22:03
*** baoli has quit IRC22:03
*** EricGonczer_ has quit IRC22:03
clarkbmnaser: and file is http://tarballs.openstack.org/manila-image-elements/images/manila-service-image-master.qcow2 ?22:03
openstackgerritJames E. Blair proposed openstack-infra/system-config: Stop managing puppet config in puppet  https://review.openstack.org/37171422:04
openstackgerritJames E. Blair proposed openstack-infra/system-config: Run ansible apply test in parallel  https://review.openstack.org/37182322:04
mnaserclarkb, http://paste.openstack.org/show/584321/ -- and yes22:04
mnaserthats 2 failures (out of 2 downloads).22:05
clarkbok so its not consistent in location22:05
clarkbmnaser: and any idea if that is via ipv4 or ipv6? I am ipv4 locally and so far it hasn't broken for me22:05
clarkbmnaser: maybe you can force v4 or v6 if the other was used and failing? (we serve via apache mod auto index so its not a very complicated setup that should fail much)22:06
*** adrian_otto has quit IRC22:06
mnaserConnecting to tarballs.openstack.org (tarballs.openstack.org)||:80... connected.22:06
mnaser (for all failures)22:06
*** hashar has quit IRC22:06
*** mdrabe has quit IRC22:07
mnaseryeah its kinda confusing as to why its happening22:07
clarkbheh /me sees a slackbot in the logs for when I pasted the link above presumably22:08
openstackgerritJames E. Blair proposed openstack-infra/system-config: DNM: squashed pansible  https://review.openstack.org/38208722:08
mnaseri'll see if this repeats, thats weird, dont have any issues with other servers22:09
jeblairclarkb: hooray for information leakage22:09
jeblairclarkb, mnaser: we saw apache segfaults on the mirrors a while back.  i don't think anyone solved that.22:09
mnasermaybe that could have to do with it22:09
mnaserim troubleshooting another fun issue at the moment so ill give feedback a bit later22:10
*** lseki has quit IRC22:10
*** edmondsw has quit IRC22:10
clarkbthee are segfaults in the error log22:12
clarkbnow trying to map that back to the requests in the access log22:13
jeblairclarkb: in 382087 i squashed all the changes together and depended on the revert -- that is making the non-redhat jobs pass22:13
*** gouthamr has quit IRC22:13
jeblairclarkb: so i think the "rc1" error may have been introduced by changes to ansible-role puppet.  no idea how the 'pwd' error happened.  and it's looking like maybe the bindep thing isn't helping?22:13
openstackgerritJames E. Blair proposed openstack-infra/system-config: DNM: squashed pansible2  https://review.openstack.org/38209022:15
clarkbjeblair: at least on debuntu libffi-dev fixes the error installing ansible http://logs.openstack.org/87/382087/1/check/gate-infra-puppet-apply-ubuntu-xenial/8dfc529/console.html#_2016-10-04_22_09_55_92786722:15
jeblair(that's the same squashed change without the depends-on)22:15
jeblairclarkb: maybe also need to install something for openssl?22:16
clarkbya looks like I need ssl dev headers on centos22:16
clarkblet me update the change with those22:16
jeblair"openssl-devel"  ?22:17
clarkband probably should include them on ubuntu for completeness (worried we are getting them transitvely without us noticing)22:17
clarkbhttps://apps.fedoraproject.org/packages/openssl-devel and http://packages.ubuntu.com/xenial/libssl-dev22:19
openstackgerritClark Boylan proposed openstack-infra/system-config: Add libffi dev packages needed for ansible install  https://review.openstack.org/38208622:20
openstackgerritJames E. Blair proposed openstack-infra/ansible-role-puppet: Revert "Remove basemodulepath from environment config"  https://review.openstack.org/38209222:20
openstackgerritJames E. Blair proposed openstack-infra/ansible-role-puppet: Revert "Sync puppet module changes from upstream ansible"  https://review.openstack.org/38209322:20
clarkb(unrelated and unscientific observation seems like the github PR closer bot has been very busy the last few days)22:21
openstackgerritJames E. Blair proposed openstack-infra/system-config: Stop managing puppet config in puppet  https://review.openstack.org/37171422:21
openstackgerritJames E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test  https://review.openstack.org/37176222:21
openstackgerritJames E. Blair proposed openstack-infra/system-config: Run ansible apply test in parallel  https://review.openstack.org/37182322:21
openstackgerritJames E. Blair proposed openstack-infra/system-config: DNM: squashed pansible  https://review.openstack.org/38208722:22
openstackgerritClark Boylan proposed openstack-infra/project-config: Turn on tls-proxy in nova net multinode job  https://review.openstack.org/38209422:22
clarkbwoot http://logs.openstack.org/74/372374/22/check/gate-tempest-dsvm-multinode-full-ubuntu-xenial/3dd7f29/ passes which means that ^ is ready for review22:22
openstackgerritJames E. Blair proposed openstack-infra/system-config: DNM: squashed pansible2  https://review.openstack.org/38209022:22
JayFclarkb: awesome, and ty for the patches to ironic to make us respect all the proper vars22:23
JayFTLS FTW22:24
clarkbJayF: still have to figure out a good way to deal with the cert on the ipa image (and other images)22:24
clarkbJayF: maybe a toggle that says "don't verify ssl because testing" though that makes the image less good for production22:24
*** tpsilva has quit IRC22:24
JayFIf we added that toggle, we'd trigger it via kernel command line22:24
JayFand only configure it on during testing22:24
JayFIPA is a strange beast, because you're bootstrapping with an inherently untrusted protocol (PXE) we've been loathe to implement even some of the basic auth measures because it's all subject to simple attack if the network is compromised22:25
JayFlet me see if I can whip up something for you real quick22:26
clarkbJayF: I will defer to ironic on how they want to handle that then, but I expect that ironic's jobs will work with tls too at this point once that hurdle is jumped22:26
clarkbJayF: you can use https://review.openstack.org/372374 as a depends on (in either direction) to show that it works22:26
jeblairclarkb: things are pointing toward https://review.openstack.org/382092 being the remaining culprit22:28
jeblairI'm going to go aheand and depends-on the stack to that22:28
openstackgerritJames E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test  https://review.openstack.org/37176222:29
*** xarses has quit IRC22:29
openstackgerritJames E. Blair proposed openstack-infra/system-config: Stop managing puppet config in puppet  https://review.openstack.org/37171422:29
openstackgerritJames E. Blair proposed openstack-infra/system-config: Run ansible apply test in parallel  https://review.openstack.org/37182322:29
*** pcrews has quit IRC22:29
clarkboh right that was the one we had trouble with before too but nibalizer confirmed we should be able to remove it so :/22:33
*** pcrews has joined #openstack-infra22:35
*** rossella_s has quit IRC22:35
*** rossella_s has joined #openstack-infra22:35
jeblairyeah, i think we've got it -- some jobs are still running, but one of each has passed somewhere in the current stack22:35
openstackgerritJames E. Blair proposed openstack-infra/ansible-role-puppet: Revert "Remove basemodulepath from environment config"  https://review.openstack.org/38209222:36
*** Teresita-Warrior has quit IRC22:36
clarkbjeblair: +222:36
jeblairclarkb: everything in topic:pansible should be ready now (i've abandoned or updated commit msgs as appropriate)22:37
clarkbok will review the remainder22:37
jeblairi think the abandon caused some -1s, i'm rechecking22:38
*** baoli has joined #openstack-infra22:39
*** xarses has joined #openstack-infra22:40
jeblairclarkb: i'm confused because https://review.openstack.org/371832 seems to have worked a while ago22:40
clarkbjeblair: one msall issue in https://review.openstack.org/#/c/371823/922:40
*** senk_ has quit IRC22:40
clarkbjeblair: at least for bindep I think ist relatively recent that we stopped using the fallback file in system-config22:40
jeblairclarkb: that changed moved the pwd for logdest out a bit on to line 1822:41
clarkbas for the pathing things maybe new behavior in newer ansible?22:41
clarkboh so it did22:41
clarkbvote updated22:41
jeblair(i may or may not have had a good reason for doing that originally... <shrug>)22:42
*** cardeois has quit IRC22:43
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add puppet apply jobs to ansible-role-puppet  https://review.openstack.org/38209622:44
jeblairclarkb: also ^22:44
*** Teresita-Warrior has joined #openstack-infra22:44
clarkbnow with all that in place should we be upstream for that role again? I bet our CI is much more comprehensive for it :)22:45
jeblairmordred: https://review.openstack.org/382092  FYI -- i don't understand it yet.  but it seems to be necessary.  it will be easier to revisit after we add the test jobs to the repo22:46
*** pradk has quit IRC22:48
jeblairclarkb: https://review.openstack.org/371823 is green :)22:48
openstackgerritMerged openstack-infra/ansible-role-puppet: Revert "Remove basemodulepath from environment config"  https://review.openstack.org/38209222:48
*** ijw has joined #openstack-infra22:48
jeblairi've reapproved the stack, so it should go in in about as long as it takes to run those test jobs serially :)22:49
*** rcernin has quit IRC22:50
*** gildub has joined #openstack-infra22:50
mordredjeblair: kk. I was mostly following spencer's lead there ... so ... nibalizer ^^22:51
*** r-daneel has quit IRC22:52
clarkbmordred: jeblair with that mostly sorted out can I beg for reviews on https://review.openstack.org/382094 and its depends-on?22:52
*** ijw has quit IRC22:53
openstackgerritMerged openstack-infra/system-config: Add libffi dev packages needed for ansible install  https://review.openstack.org/38208622:55
*** rhallisey has quit IRC22:55
*** senk_ has joined #openstack-infra22:55
*** yamahata has quit IRC22:55
fungiinfra-root: (and anyone else) just fyi a mandatory evacuation is looking likely here. in an effort to still be able to go on our vacation we're planning on leaving friday morning leaving me only a couple days to get the house storm-ready. i'm still planning to be around through the release and hack on summit prep, but not expecting to get any other work done before i head out of town22:57
*** yamamoto has joined #openstack-infra22:57
clarkbfungi: gl22:57
pleia2fungi: stay safe22:58
*** sdake has joined #openstack-infra22:58
pleia2fungi: feel free to brain dump anything you need to me, if such a thing is needed :)22:58
fungipleia2: appreciated, and i definitely will!22:58
jeblairclarkb: lgtm23:00
clarkbjeblair: ty23:01
JayFclarkb: put up a quick POC patchset(s): https://review.openstack.org/382106 is the one that would pass devstack tests if it worked23:01
*** salv-orlando has quit IRC23:01
JayFclarkb: actually, come to think of it, IPA gate is busted if we hit osic cloud, so that might not pass anyway... but we'll see :D23:01
clarkbJayF: oh right that23:01
mordredfungi: yah. stay safe. last storm that hit somewhere named after a person I know turned into a super storm23:01
pabelangerfungi: good luck, will be on standby for releases23:02
*** xyang1 has quit IRC23:02
*** john-davidge has joined #openstack-infra23:02
mordredfungi: given olaph's real name, I have my concerns for this one23:02
ianwclarkb: that segfault ...23:02
fungimordred: yeah, i heard noaa named this after olaph because they knew him so well23:02
ianwaddr2line -e ./mod_mpm_event.so  0x65e023:02
ianw /build/apache2-Rau9Dr/apache2-2.4.7/server/mpm/event/event.c:106423:02
clarkbianw: oh did you get a coredump to inspect?23:03
ianwafter digging around, that puts it right in the middle of https://github.com/apache/httpd/commit/d62d4c3f5001d44cae2d5b023b36a9ebc0a8e6b523:03
ianwactually this line https://github.com/apache/httpd/commit/d62d4c3f5001d44cae2d5b023b36a9ebc0a8e6b5#diff-895d7e9f8add746606c82027dabc04d4L117723:03
clarkbianw: so in theory fixable with a newer apache23:04
*** hongbin has quit IRC23:04
anteayafungi: travel safely to all your destinations23:04
*** mixos has joined #openstack-infra23:04
ianwclarkb: no, that's from the offsets in the segfault log.  there is probably a case for a trusty bug to include that patch23:05
mordredclarkb: +A23:05
*** rajinir has quit IRC23:05
*** sdake_ has joined #openstack-infra23:05
*** Apoorva_ has joined #openstack-infra23:05
clarkbmordred: woot thanks23:05
clarkbianw: there are more verbose logs than [Tue Oct 04 23:05:36.185405 2016] [core:notice] [pid 3677:tid 140379199530880] AH00052: child pid 31920 exit signal Segmentation fault (11) ?23:06
clarkbianw: wondering where you found 0x65e023:06
*** john-davidge has quit IRC23:06
*** tuannguyen has joined #openstack-infra23:06
clarkbin any case cool that it is tracked down23:07
ianwclarkb: /var/log/kern.log -- subtract the ip from the base address23:07
*** Apoorva has quit IRC23:08
clarkbianw: aha thanks23:08
*** Teresita-Warrior has quit IRC23:09
*** yamamoto has quit IRC23:09
*** tiswanso has joined #openstack-infra23:09
*** sdake has quit IRC23:09
*** amitgandhinz has joined #openstack-infra23:09
ianwbasically, it seems like good luck reproducing it, though.  "possible crashes under load" i.e. good as calling random()23:09
*** tuannguyen has quit IRC23:11
*** jamielennox is now known as jamielennox|away23:12
*** cody-somerville has joined #openstack-infra23:12
*** jamielennox|away is now known as jamielennox23:13
*** tiswanso has quit IRC23:13
*** amitgandhinz has quit IRC23:14
ianwclarkb: unfortunately, but unsurprisingly, that patch doesn't apply cleanly.  anyway, i'll report a bug since something is obviously going on23:16
clarkbianw: sounds good, thanks23:17
mordredclarkb: so - this failure: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html ... is from two different neutron API calls for the same port23:17
*** yamahata has joined #openstack-infra23:17
*** senk_ has quit IRC23:17
clarkbmordred: either side of the assert is a different api call?23:18
mordredclarkb: the 'reference' dict is the one returned from create. the other dict is the one returned from get - getting via the id returned from create ... so the list of fixed_ips is just indeterminate order23:18
clarkbyour fixed_ip order is different23:18
*** gouthamr has joined #openstack-infra23:18
clarkbmordred: yup23:18
clarkbmordred: you could return a set to express that more explicitly23:18
openstackgerritMerged openstack-infra/system-config: Use ansible for puppet apply test  https://review.openstack.org/37176223:18
clarkbbut that might not be backward compat23:18
openstackgerritMerged openstack-infra/system-config: Stop managing puppet config in puppet  https://review.openstack.org/37171423:18
openstackgerritMerged openstack-infra/system-config: Run ansible apply test in parallel  https://review.openstack.org/37182323:19
clarkbyou could sort (ugh) or just make the assert smart enough to ignore order on that?23:19
mordredclarkb: yah - I think I'm going to have to just make a smart assert call23:19
mordredclarkb: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/tests/functional/test_port.py#n75 is the test23:19
mordredarmax: if you're around ... ^^ is the lack of determinism in the fixed_ips list between the ipv4 and ipv6 address intentional/expected?23:20
armaxmordred: I don’t believe determinism was ever intended23:20
mordredarmax: mostly asking in case it's not and a bug filed would be useful - I can work around it on my side just fine23:21
mordredso fixed_ips is a list, but the order of the ips in the list is not expected to be stable between api calls23:21
clarkbmordred: also you should review the depends on for that one change you approved :) https://review.openstack.org/#/c/373219/23:21
mordredclarkb: bah23:21
clarkbmordred: it would be neat to clean those things up and clearly specify them as sets vs lists if we ever get to make a new api23:21
clarkb(I don't think json can express that though)23:22
armaxmordred: no, the ordering of the list of IPs is not something that’s embossed in the API contract23:22
clarkbarmax: it is implicitly by returning a list23:22
mordredarmax: ok. cool. thanks for clarifying!23:22
armaxmordred: when have you started seeing lack of determinism?23:22
clarkbarmax: lists are ordered... you should use unordred types if that is teh case23:22
clarkbbut that ship may have sailed23:22
mordredarmax: it just randomly happened recently - but it only happened on 2 out of 10 calls I made23:22
mordred2 out of 10 test runs, that is23:23
mordredit had been stable before23:23
mordredor, had seemed that way23:23
armaxclarkb: I am not sure I understand your claim23:23
* olaph always wanted to be a major storm23:24
fungiolaph: your wish is my command23:24
armaxlist are ordered how?23:24
fungiarmax: python preserves order in its list datatype23:24
fungiarmax: as opposed to, say, dict keys which are considered unordered23:25
armaxright, but how data is packed in the list is where the lack of determinism comes from23:25
clarkbthe definition of a list is that it is ordered23:25
*** mixos has quit IRC23:25
clarkbif you can't reliably order contents of a list you should use an unordered type23:25
armaxright, I understand that23:26
armaxone thing is saying once I create a list of [1, 2, 3] is always going to stay like that23:26
clarkbby definition it should23:26
clarkbbecause it is a list23:26
armaxand that I agree23:26
armaxthe source of non determinsm is how that list is built23:27
armaxonce the list if built, it won’t change23:27
mordredarmax: in this case, the first instance of the list is from the return of "POST /ports" ... the second instance of the list is from GET /ports/{port_id}23:27
clarkbright and I am saying that if you cannot garuntee that the list will be built deterministically that way, then that should be properly expressed by using an unordered type (I totally get that json makes this hard and python makes the other thing easy)23:28
clarkbbeacuse from the end users perspective they don't care how you ahve implemented it23:28
clarkbto them that data structure is the same (or should be the same) in both places23:28
mordredI tihnk what I'm going to do shade-side is always sort the list of fixed_ips in a port by the port_id ... it's still arbitrary, but it'll be stable23:29
armaxclarkb: as an API client I would not expect any order in the list of IPs that are returned to me23:29
armaxbut that’s a different point23:29
armaxmordred: do you have an instance of the test failure?23:30
mordredarmax: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html23:30
mordredarmax: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html#_2016-10-04_01_59_52_598273 more specifically23:30
armaxyup I see23:30
mordredarmax: here's teh test: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/tests/functional/test_port.py#n7523:30
mordredanywho - I can deal with this in my world. if it's behaving right from your end, alls good23:32
*** pahuang has joined #openstack-infra23:32
armaxmordred: I suppose we can address the lack of non-determinism23:32
armaxon the neutron side23:32
clarkbarmax: I guess from a client perspective if I get a datastructure from an API I would expect the same data structure back again if requested a second time with no changes made23:33
armaxthough, for the nature of the elements stored in the fixed_ips, I wonder what’s going to buy us23:33
clarkbarmax: and changing the order of the list is not the same thing. However if you returned an unordered type that would be fine23:33
armaxclarkb: yeah, it looks like mordred is saying that the change occurs between a POST and a GET23:34
armaxI wonder what happens in susequent GETs23:34
armaxI suspect the reason lies on how the create response is built vs the read one23:35
armaxand I can think of which code change might have altered that23:35
armaxclarkb, mordred: I bet that subsequent GETs will alway return the list in the same order23:37
armaxI think that’s the culprit: https://review.openstack.org/#/c/37358223:38
armaxI can have kevinbenton look into restoring peace in the middle earth23:38
anteayahe is good at that23:39
* clarkb waves https://review.openstack.org/#/c/373219/ in front of mordred one more time before going to attach shelves to walls23:39
anteayaclarkb: you are offically out of the boxes?23:40
clarkbanteaya: not entirely23:40
clarkbya pretty close23:42
*** senk_ has joined #openstack-infra23:42
*** mixos has joined #openstack-infra23:43
mordredarmax: you are like an encyclopedia of what's gone on in neutron23:44
*** Swami has quit IRC23:44
armaxmordred: not sure about that, it’s more likely I was the one creating the damage :)23:44
armaxmordred: let me make it up to you: bug 163041023:49
openstackbug 1630410 in neutron "fixed_ips list out of order" [Undecided,New] https://launchpad.net/bugs/163041023:49
*** thorst has joined #openstack-infra23:49
mordredwoot! armax is teh ossum23:50
anteayakevinbenton shall fix this23:51
anteayathank you kevinbenton23:51
mordredthank you kevinbenton23:51
armaxmordred: having said that my suggestion is to try and avoid the implicit assumptions in relation to ordering23:56
mordredyup. also doing that23:57
armaxmordred: we had other examples in the past were it was assumed that the IP allocation be deterministic and more specifically that IPs were handed out from the lowest IP to the highest23:57
mordredoh my.23:57
armaxwhen in that there’s nothing in the API that says that’s the IP allocation policy23:57
*** kaisers_ has quit IRC23:57
*** mixos has quit IRC23:57
armaxespecially when pluggable IPAM backends are brought into the picture23:58
armaxmordred: that’s not exactly what you experienced23:58
armaxmordred: but you get the idea :)23:58
mordredarmax: I'm actually going through the things we return at the moment and making sure we know what the contract is for all of the things (it's super fun, let me assure you)23:59
armaxmordred: I want to make sure I minimize your fun23:59
armaxmordred: so do not hesitate to shout if we can make take some of that fun away23:59
mordredarmax: yes, I appreciate that. as little fun as I can have the better23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!