Friday, 2017-04-28

pabelangerrolled back ubuntu-trusty dib00:00
*** ijw has quit IRC00:00
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for operator_cloud functional tests  https://review.openstack.org/46078400:01
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for operator_cloud functional tests  https://review.openstack.org/46078400:02
openstackgerritMerged openstack-infra/project-config master: Force /bin/bash for jenkins user  https://review.openstack.org/46078100:02
*** esikachev has quit IRC00:04
*** lamt has joined #openstack-infra00:08
*** petems has joined #openstack-infra00:10
*** esikachev has joined #openstack-infra00:11
*** ihrachys has joined #openstack-infra00:12
*** xgerman has joined #openstack-infra00:14
*** puiterwijk|cld has joined #openstack-infra00:15
pabelangerianw: seen a presentation about linuxkit last night, like DIB but driven from containers00:15
pabelangerinteresting concept00:15
fungiclarkb: EmilienM: i think branch deletion is waiting for gerrit upgrade where there will be explicit permission for an api call to do that00:15
*** esikachev has quit IRC00:16
fungiso for now branch deletion is basically still manual as far as i'm aware00:16
*** esikachev has joined #openstack-infra00:17
*** ijw has joined #openstack-infra00:20
ianwpabelanger: hmm, probably the interesting bit of dib is the "d" part00:22
*** esikachev has quit IRC00:22
Sukhdev_fungi clarkb : why do I get requirements mismatch on a new project - https://review.openstack.org/#/c/460767/00:22
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests  https://review.openstack.org/46078400:23
openstackgerritPaul Van Eck proposed openstack-infra/puppet-refstack master: Fix puppet lint errors  https://review.openstack.org/46078700:24
pabelangerianw: disk?00:24
*** ijw has quit IRC00:24
ianwyeah, partitioning, bootloaders, lvm, encryption etc etc00:24
*** ihrachys has quit IRC00:25
pabelangerianw: that's the thing, linuxkit does that. It creates a qcow2 image for you, but picked from container bits00:25
pabelangerit's pretty crazy00:25
*** Julien-zte has joined #openstack-infra00:26
openstackgerritMerged openstack-infra/devstack-gate master: Redirect stderr to stdout for some ansible calls  https://review.openstack.org/45868900:33
*** xyang1 has quit IRC00:37
*** xyang1 has joined #openstack-infra00:38
clarkbSukhdev_: the job logs tell you00:38
*** sflanigan has quit IRC00:38
clarkbSukhdev_: hacking requirement doesn't match00:38
*** salv-orlando has joined #openstack-infra00:39
*** pvaneck has quit IRC00:40
*** markvoelker has joined #openstack-infra00:41
*** rfolco has joined #openstack-infra00:41
*** rfolco has quit IRC00:41
Sukhdev_clarkb: I saw that but, am confused as to what is that is needed though - the only requirement I see in the requirements.txt is pbr>-2.000:41
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add in a bunch of TODOs about interface=admin  https://review.openstack.org/46079000:42
clarkbSukhdev_: https://review.openstack.org/#/c/460767/2/test-requirements.txt line 500:43
*** salv-orlando has quit IRC00:44
Sukhdev_clarkb : got it - thanks00:46
*** markvoelker has quit IRC00:46
pabelangerfungi: sent out ML post about summit dinner, atleast get some numbers00:50
clarkboh good someone else is goign to do it this time around :P00:51
clarkbpabelanger: as far as picking places I think it worked well where I essentially asserted a place and asked people to veto. Didn't get any veto00:51
pabelangerclarkb: cool, I'll take a look on some food sites, see what they recommend00:51
clarkbpabelanger: there is also a guide on the summit site. Thats what I used for ptg00:52
*** esikachev has joined #openstack-infra00:52
pabelangerk00:52
fungipabelanger: thanks! replied00:54
*** sflanigan has joined #openstack-infra00:55
*** sflanigan has joined #openstack-infra00:55
*** ihrachys has joined #openstack-infra00:55
*** kgiusti has joined #openstack-infra00:56
*** esikachev has quit IRC00:57
*** yee379 has quit IRC00:58
*** yee379 has joined #openstack-infra00:58
*** liusheng has quit IRC01:03
*** ihrachys has quit IRC01:05
*** SotK has quit IRC01:06
*** SotK has joined #openstack-infra01:06
*** jamesmca_ has joined #openstack-infra01:08
*** kgiusti has quit IRC01:09
*** afred312 has joined #openstack-infra01:09
*** Apoorva_ has joined #openstack-infra01:16
*** annegentle has joined #openstack-infra01:16
*** armax has quit IRC01:17
*** Gorian has quit IRC01:18
*** tphummel has quit IRC01:18
*** erlon has joined #openstack-infra01:19
*** Gorian has joined #openstack-infra01:19
*** Apoorva has quit IRC01:20
*** liusheng has joined #openstack-infra01:20
*** yolkfull has joined #openstack-infra01:20
*** Apoorva_ has quit IRC01:21
*** xarses_ has joined #openstack-infra01:21
*** xarses_ has quit IRC01:21
*** xarses_ has joined #openstack-infra01:21
*** spzala has quit IRC01:22
*** xarses_ has quit IRC01:22
*** xarses_ has joined #openstack-infra01:22
*** spzala has joined #openstack-infra01:23
*** spzala has quit IRC01:27
*** xarses_ has quit IRC01:28
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Fix warning from useradd  https://review.openstack.org/46079901:29
pabelangerclarkb: ianw: ^ interesting one, because /home does exist, useradd will complain and create a warning01:29
*** zz_dimtruck is now known as dimtruck01:29
pabelangerianw: clarkb: maybe DIB should create /home out of the box?01:30
ianwhmmm, minimal doesn't maybe?  devuser must?01:31
*** tuanluong has joined #openstack-infra01:31
*** liujiong has joined #openstack-infra01:31
*** hongbin has joined #openstack-infra01:31
*** Julien-zte has quit IRC01:31
*** xarses_ has joined #openstack-infra01:32
ianwsurely it must be there from the base packages01:32
pabelangeractually, maybe that isn't the issue01:32
pabelangerlet me google more01:32
ianwpabelanger: isn't that saying /home/jenkins already exists01:33
ianwso it's not going to copy anything over01:33
pabelangermaybe?01:33
pabelangeris the groupadd creating it?01:33
ianwi doubt it01:34
pabelangerya, not sure what would be creating it now01:34
pabelangerlet me test quick01:34
ianwmkdir -p ~jenkins/cache/files01:35
ianwmaybe running prior?01:35
*** yamahata_ has quit IRC01:35
pabelangerdon't think so01:35
pabelangerpretty sure it is after01:35
ianwboth 20 in install.d ?01:36
*** Julien-zte has joined #openstack-infra01:36
ianwwhy do we even need the groupadd?01:37
*** yanyanhu has joined #openstack-infra01:37
*** xarses_ has quit IRC01:38
*** gardlt has quit IRC01:38
*** markvoelker has joined #openstack-infra01:38
pabelangeropensuse I think01:39
pabelangerI added it to be safe01:39
pabelangerwas also what puppet-jenkins did01:39
ianwgroupadd doesn't touch homedir on my test trusty01:40
ianw# useradd -g jenkins -m jenkins01:40
ianwuseradd: group 'jenkins' does not exist01:40
ianwbut, if you just leave the "-g jenkins" out, it creates the group01:41
pabelangerthe behavior of useradd will depend on the USERGROUPS_ENAB variable in01:43
pabelanger           /etc/login.defs01:43
pabelangerwhich, I think opensuse was disabled01:43
clarkbya suse does users as default group01:43
pabelangerwe can only do groupadd for opensuse if we want01:43
pabelangerbut testing something now01:43
*** spzala has joined #openstack-infra01:44
*** gouthamr has quit IRC01:44
ianwpabelanger: 2017-04-27 07:00:48,402 in http://nb01.openstack.org/dib.ubuntu-trusty.log01:44
ianwpuppet creating?01:44
*** gouthamr has joined #openstack-infra01:45
pabelangerhttp://nb02.openstack.org/dib.ubuntu-trusty.log is latest01:45
*** salv-orlando has joined #openstack-infra01:45
EmilienMfungi: if you can proceed to the branch deletion, when you can, that would be cool01:46
EmilienMgood night folks01:46
*** jamesmca_ has quit IRC01:47
pabelangerconfirmed /home is empty01:48
pabelangertesting again01:48
*** rkukura has joined #openstack-infra01:48
*** spzala has quit IRC01:49
*** ramishra has joined #openstack-infra01:49
*** EricGonczer_ has joined #openstack-infra01:49
*** Julien-z_ has joined #openstack-infra01:49
*** salv-orlando has quit IRC01:50
*** spzala has joined #openstack-infra01:50
*** esikachev has joined #openstack-infra01:50
*** Julien-zte has quit IRC01:50
*** jamesmca_ has joined #openstack-infra01:51
*** Sukhdev_ has quit IRC01:52
*** EricGonc_ has joined #openstack-infra01:53
ianwwe just do "useradd -m zuul" later01:53
*** EricGonczer_ has quit IRC01:53
*** gildub has joined #openstack-infra01:54
*** spzala has quit IRC01:55
*** esikachev has quit IRC01:55
pabelangerI'll have to dig into it tomorrow01:55
*** mat128 has joined #openstack-infra01:55
*** yolanda has quit IRC01:55
*** jamesmca_ has quit IRC01:56
*** Goneri has quit IRC01:58
*** gildub has quit IRC01:59
ianwhmm, looking at the source, it's not like stat() /home/jenkins -> throw that message02:00
ianwit might be a red-herring02:00
*** esikachev has joined #openstack-infra02:00
ianwhttps://github.com/shadow-maint/shadow/blob/master/src/useradd.c#L1943 ... we don't see the mkdir failing.  maybe it's the access() call that fails, maybe /home perms are wrong?02:02
*** zhurong has joined #openstack-infra02:04
*** esikachev has quit IRC02:05
*** Nakato has quit IRC02:06
*** ramishra has quit IRC02:07
*** ramishra has joined #openstack-infra02:08
*** Nakato has joined #openstack-infra02:09
*** spzala has joined #openstack-infra02:11
*** rlandy|bbl is now known as rlandy02:12
*** esikachev has joined #openstack-infra02:13
*** spzala has quit IRC02:16
*** gildub has joined #openstack-infra02:16
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921502:16
*** esikachev has quit IRC02:17
*** SotK has quit IRC02:22
*** SotK has joined #openstack-infra02:25
*** erikmwilson has quit IRC02:26
*** erikmwilson has joined #openstack-infra02:26
*** dave-mccowan has quit IRC02:27
*** yee379 has quit IRC02:31
*** yee37917 has joined #openstack-infra02:31
*** rkukura has quit IRC02:32
*** spzala has joined #openstack-infra02:32
*** Nakato has quit IRC02:32
*** Nakato has joined #openstack-infra02:33
*** rkukura has joined #openstack-infra02:35
*** spzala has quit IRC02:36
*** thorst has quit IRC02:36
*** tphummel has joined #openstack-infra02:37
*** yolkfull has quit IRC02:44
*** yolkfull has joined #openstack-infra02:44
*** armax has joined #openstack-infra02:45
*** salv-orlando has joined #openstack-infra02:46
ianwyes of course adding a user has lead me down the path of converting things to python302:48
*** spzala has joined #openstack-infra02:48
*** salv-orlando has quit IRC02:51
*** yolkfull_ has joined #openstack-infra02:51
*** spzala has quit IRC02:53
*** yolkfull__ has joined #openstack-infra02:53
*** yolkfull has quit IRC02:54
*** EricGonc_ has quit IRC02:55
*** yolkfull_ has quit IRC02:56
*** Julien-zte has joined #openstack-infra02:57
*** yolkfull has joined #openstack-infra02:57
*** yolkfull__ has quit IRC02:59
*** Julien-z_ has quit IRC03:00
*** spzala has joined #openstack-infra03:01
*** psachin has joined #openstack-infra03:06
*** thorst has joined #openstack-infra03:07
*** yolkfull_ has joined #openstack-infra03:09
*** yolkfull__ has joined #openstack-infra03:10
*** thorst has quit IRC03:11
*** yolkfull has quit IRC03:12
*** hongbin_ has joined #openstack-infra03:13
*** hongbin has quit IRC03:13
*** yolkfull_ has quit IRC03:14
*** mat128 has quit IRC03:15
openstackgerritRabi Mishra proposed openstack-infra/project-config master: Don't run tempest tests in heat grenade job  https://review.openstack.org/46081003:16
*** armax has quit IRC03:18
*** david-lyle is now known as dklyle03:19
*** dklyle is now known as david-lyle03:20
*** david-lyle is now known as dklyle03:21
*** nicolasbock has quit IRC03:22
*** dklyle is now known as david-lyle03:23
*** yolkfull__ is now known as yolkfull03:25
ianwpabelanger: yep ... /home/jenkins/cache/files is what's pre-created the jenkins homedir03:26
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Zun] Add zun-ui server release job  https://review.openstack.org/46081503:29
*** yamahata_ has joined #openstack-infra03:35
*** thorst has joined #openstack-infra03:38
*** zhurong has quit IRC03:38
*** Julien-zte has quit IRC03:40
*** john-davidge has joined #openstack-infra03:42
*** thorst has quit IRC03:45
*** rkukura has quit IRC03:47
*** mat128 has joined #openstack-infra03:47
*** john-davidge has quit IRC03:47
*** gouthamr has quit IRC03:50
*** gouthamr has joined #openstack-infra03:52
*** mriedem has quit IRC03:52
*** gouthamr has quit IRC03:56
*** Julien-zte has joined #openstack-infra04:05
*** ykarel has joined #openstack-infra04:06
*** yolkfull_ has joined #openstack-infra04:07
*** yolkfull has quit IRC04:10
*** yolkfull__ has joined #openstack-infra04:10
*** yolkfull_ has quit IRC04:12
*** jcoufal has joined #openstack-infra04:13
*** amotoki has joined #openstack-infra04:14
openstackgerritAshwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests  https://review.openstack.org/46082204:20
*** jcoufal has quit IRC04:20
*** esikachev has joined #openstack-infra04:26
*** mat128 has quit IRC04:27
*** jaypipes has quit IRC04:29
*** hongbin_ has quit IRC04:29
*** Sukhdev has joined #openstack-infra04:30
*** esikachev has quit IRC04:31
*** aviau has quit IRC04:32
*** aviau has joined #openstack-infra04:32
*** thorst has joined #openstack-infra04:41
*** jaypipes has joined #openstack-infra04:41
*** thorst has quit IRC04:45
*** salv-orlando has joined #openstack-infra04:47
*** armax has joined #openstack-infra04:48
*** bhavik1 has joined #openstack-infra04:48
*** esikachev has joined #openstack-infra04:52
*** salv-orlando has quit IRC04:52
*** esikachev has quit IRC04:56
*** zhurong has joined #openstack-infra05:00
*** xyang1 has quit IRC05:02
*** dimtruck is now known as zz_dimtruck05:03
*** gyee has quit IRC05:03
*** zz_dimtruck is now known as dimtruck05:03
ianwwell this turns out to be a right pickle05:04
*** beslemon has joined #openstack-infra05:08
*** dimtruck is now known as zz_dimtruck05:12
*** erlon has quit IRC05:13
*** Hal has joined #openstack-infra05:13
*** Sukhdev has quit IRC05:13
openstackgerritMerged openstack-dev/cookiecutter master: Remove testsenarios from test-requirements.txt  https://review.openstack.org/45027905:24
*** prateek has joined #openstack-infra05:24
*** Hal has quit IRC05:24
openstackgerritAshwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests  https://review.openstack.org/46082205:25
openstackgerritIan Wienand proposed openstack-infra/project-config master: Move jenkins cache to /opt/cache  https://review.openstack.org/46083505:28
ianwpabelanger: ^ i think that solves the case of the mystery /home/jenkins.  http://paste.openstack.org/show/608278/05:29
ianwin the mean time, i've discovered a bunch of python3 issues, and a bunch of dead code ... all of which i'll clean up another day.  that just about fried my brain :/05:29
*** zul has quit IRC05:34
*** zul has joined #openstack-infra05:35
*** salv-orlando has joined #openstack-infra05:48
*** salv-orlando has quit IRC05:53
*** esikachev has joined #openstack-infra05:55
*** lamt has quit IRC05:56
*** yuanying_ has quit IRC05:57
*** sree has joined #openstack-infra06:00
openstackgerritIan Wienand proposed openstack-infra/project-config master: Move jenkins cache to /opt/cache  https://review.openstack.org/46083506:01
*** zhurong has quit IRC06:04
*** adrian_otto has joined #openstack-infra06:05
*** beslemon has quit IRC06:08
*** andreas_s has joined #openstack-infra06:08
*** tjones has quit IRC06:09
*** zhurong has joined #openstack-infra06:12
*** adrian_otto has quit IRC06:15
*** kzaitsev_ws has joined #openstack-infra06:19
*** Julien-zte has quit IRC06:22
*** zhurong has quit IRC06:23
*** Julien-zte has joined #openstack-infra06:24
*** kzaitsev_ws has quit IRC06:28
*** yuanying has joined #openstack-infra06:36
*** matrohon has quit IRC06:36
*** pcaruana has joined #openstack-infra06:36
*** yolkfull_ has joined #openstack-infra06:37
*** ccamacho has joined #openstack-infra06:40
*** yolkfull__ has quit IRC06:40
*** yolkfull has joined #openstack-infra06:40
*** thorst has joined #openstack-infra06:42
*** yuanying_ has joined #openstack-infra06:42
*** yuanying has quit IRC06:42
*** yolkfull_ has quit IRC06:43
*** yanyanhu has quit IRC06:45
*** e0ne has joined #openstack-infra06:46
*** yuanying_ has quit IRC06:46
*** thorst has quit IRC06:46
*** yuanying has joined #openstack-infra06:47
*** jamesmca_ has joined #openstack-infra06:47
*** salv-orlando has joined #openstack-infra06:49
AJaegerianw: thanks, fixed zuul ^06:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Allways add zuul group to zuul user  https://review.openstack.org/46085006:52
AJaegerdirk, cmurphy ^06:52
*** jamesmca_ has quit IRC06:52
*** tphummel has quit IRC06:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Always add zuul group to zuul user  https://review.openstack.org/46085006:53
*** rkukura has joined #openstack-infra06:53
*** camunoz has quit IRC06:54
*** salv-orlando has quit IRC06:54
*** shardy has joined #openstack-infra06:56
*** salv-orlando has joined #openstack-infra06:57
*** matrohon has joined #openstack-infra06:58
*** flepied has joined #openstack-infra06:59
*** andreas-f has quit IRC06:59
*** tie has joined #openstack-infra07:03
*** tesseract has joined #openstack-infra07:05
*** pgadiya has joined #openstack-infra07:11
*** ggnel_t has joined #openstack-infra07:17
*** aviau has quit IRC07:19
*** aviau has joined #openstack-infra07:19
*** e0ne has quit IRC07:19
*** salv-orlando has quit IRC07:23
*** markus_z has joined #openstack-infra07:30
*** jpich has joined #openstack-infra07:32
*** kornica has joined #openstack-infra07:33
*** kornica is now known as kornicameister07:33
*** tuanluong has quit IRC07:34
*** gildub has quit IRC07:39
*** yamamoto has quit IRC07:40
*** jpena|off is now known as jpena07:42
*** thorst has joined #openstack-infra07:43
*** kornica has joined #openstack-infra07:44
*** kornicameister has quit IRC07:46
*** kornicameister has joined #openstack-infra07:48
*** kornica has quit IRC07:48
*** kornicameister has quit IRC07:49
*** kornica has joined #openstack-infra07:50
*** Julien-zte has quit IRC07:53
*** flepied has quit IRC07:53
*** makowals has joined #openstack-infra07:54
*** kornica is now known as kornicameister07:54
*** armax has quit IRC07:55
*** sree_ has joined #openstack-infra07:57
*** sree_ is now known as Guest9037707:57
*** kornica has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** sree has quit IRC08:00
*** kornicameister has quit IRC08:02
*** andreaf has quit IRC08:02
*** thorst has quit IRC08:02
*** andreaf has joined #openstack-infra08:04
*** aarefiev_afk is now known as aarefiev08:07
*** asselin__ has joined #openstack-infra08:08
*** asselin has joined #openstack-infra08:10
*** asselin_ has quit IRC08:11
*** asselin__ has quit IRC08:13
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin  https://review.openstack.org/46050608:15
*** stakeda has quit IRC08:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Always add zuul group to zuul user  https://review.openstack.org/46085008:16
*** dtantsur|afk is now known as dtantsur08:17
*** florianf has joined #openstack-infra08:17
*** dizquierdo has joined #openstack-infra08:17
*** lucas-afk is now known as lucasagomes08:17
*** Julien-zte has joined #openstack-infra08:21
*** flepied has joined #openstack-infra08:23
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin  https://review.openstack.org/46050608:28
*** rossella_s has joined #openstack-infra08:36
*** kornica is now known as kornicameister08:40
*** derekh has joined #openstack-infra08:40
*** makowals has quit IRC08:42
*** tuanluong has joined #openstack-infra08:43
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin  https://review.openstack.org/46050608:44
openstackgerritJakub Libosvar proposed openstack-infra/project-config master: neutron: add ovs firewall tempest job to check queue  https://review.openstack.org/46040908:46
*** electrofelix has joined #openstack-infra08:46
*** Julien-zte has quit IRC08:48
*** jaosorior_away is now known as jaosorior08:49
*** Julien-zte has joined #openstack-infra08:50
*** mhickey has joined #openstack-infra08:51
*** e0ne has joined #openstack-infra08:52
*** thorst has joined #openstack-infra08:59
*** deafult has quit IRC09:00
*** deafult has joined #openstack-infra09:01
*** waynr has quit IRC09:01
*** yolkfull_ has joined #openstack-infra09:03
*** hamzy has joined #openstack-infra09:04
*** chenying_ has joined #openstack-infra09:06
*** yolkfull has quit IRC09:06
*** yamahata_ has quit IRC09:08
*** sflanigan has quit IRC09:08
*** chenying__ has quit IRC09:09
*** liusheng has quit IRC09:09
*** liusheng has joined #openstack-infra09:09
*** thorst has quit IRC09:10
*** john-davidge has joined #openstack-infra09:10
*** kornicameister has quit IRC09:11
*** kornica has joined #openstack-infra09:12
*** MasterOfBugs has quit IRC09:14
*** markvoelker has quit IRC09:16
*** odyssey4me_ is now known as odyssey4me09:20
openstackgerritArne Recknagel proposed openstack/diskimage-builder master: Add zipl support for s390 architecture with SCSI boot  https://review.openstack.org/44354809:20
openstackgerritEvgeni Golov proposed openstack-infra/jenkins-job-builder master: properly support private builds in the registry  https://review.openstack.org/46088909:21
*** adriant_ has quit IRC09:23
*** Julien-zte has quit IRC09:24
*** dhajare has quit IRC09:24
openstackgerritArne Recknagel proposed openstack/diskimage-builder master: Add zipl support for s390 architecture with SCSI boot  https://review.openstack.org/44354809:24
*** Guest90377 has quit IRC09:28
*** sree has joined #openstack-infra09:30
*** sambetts|afk is now known as sambetts09:31
*** jtomasek has quit IRC09:34
*** tosky has joined #openstack-infra09:37
*** jtomasek has joined #openstack-infra09:37
openstackgerritEvgeni Golov proposed openstack-infra/jenkins-job-builder master: conditional_publisher: add support for Matrix Aggregation  https://review.openstack.org/46089809:40
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/46089909:41
*** adarazs has left #openstack-infra09:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: OSA: Output ZUUL variables to a file  https://review.openstack.org/46090709:55
odyssey4meproject-config cores, as is evidenced by http://logs.openstack.org/39/458839/11/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/21b9266/console.html I broke our gate job with https://review.openstack.org/458890 - I've proposed https://review.openstack.org/460907 to fix it and would appreciate prompt review09:56
odyssey4meclarkb ^ that's the change we discussed previously09:57
*** sdague has joined #openstack-infra10:00
*** jaosorior is now known as jaosorior_away10:00
openstackgerritDmitry Tantsur proposed openstack-infra/project-config master: Make gate-ironic-inspector-tox-func3 voting  https://review.openstack.org/46091410:02
*** lyarwood has quit IRC10:03
*** nicolasbock has joined #openstack-infra10:03
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/46089910:04
*** tesseract has quit IRC10:04
*** thorst has joined #openstack-infra10:07
*** tesseract has joined #openstack-infra10:08
openstackgerritDmitry Tantsur proposed openstack-infra/project-config master: Make gate-ironic-inspector-tox-func3 voting  https://review.openstack.org/46091410:09
*** thorst has quit IRC10:11
*** mkebich has joined #openstack-infra10:11
*** markvoelker has joined #openstack-infra10:17
*** sree has quit IRC10:17
*** bhavik1 has quit IRC10:17
*** markvoelker has quit IRC10:22
openstackgerritMichal Dulko proposed openstack-infra/project-config master: Add docs job for openstack-helm  https://review.openstack.org/46092510:28
*** liujiong has quit IRC10:41
*** jkilpatr has quit IRC10:42
*** mat128 has joined #openstack-infra10:44
openstackgerritSean Dague proposed openstack-infra/devstack-gate master: Use compat names for journal files  https://review.openstack.org/46094710:44
*** jamesmca_ has joined #openstack-infra10:49
*** tuanluong has quit IRC10:52
*** jamesmca_ has quit IRC10:53
*** BobBall_AWOL is now known as BobBall10:54
*** dhajare has joined #openstack-infra10:58
*** rossella_s has quit IRC11:04
*** rossella_s has joined #openstack-infra11:05
*** dhajare has quit IRC11:05
odyssey4meAJaeger to answer your question in https://review.openstack.org/#/c/460907/1/jenkins/jobs/openstack-ansible-jobs.yaml the purpose of the if is to cover the case where the ZUUL_ vars may not be set... although I'm not sure that will ever be the case in the gate?11:05
*** yamamoto has joined #openstack-infra11:05
odyssey4meare those vars always present?11:05
*** yamamoto_ has joined #openstack-infra11:06
AJaegerodyssey4me: ZUUL_ vars are always set. Depending on the queue you run in - post, periodic, check/gate, there might be a few different variables.11:07
*** yolkfull_ has quit IRC11:08
AJaegerodyssey4me: but your code would still work if ZUUL_ is not set, doesn't it?11:08
odyssey4meok, if there is always at least one ZUUL_ var, then I don't need to fence off a possible failure there11:08
odyssey4meif no ZUUL_ var is set, then 'printenv | grep ^ZUUL' will results in an error11:09
odyssey4methe grep returns a fail :)11:09
odyssey4meI suppose I could use awk instead, which won't11:09
AJaegerodyssey4me: indeed, see it.11:09
*** makowals has joined #openstack-infra11:09
AJaegeryou could work around that but I would just leave it for now.11:09
*** yamamoto has quit IRC11:10
*** makowals has quit IRC11:10
odyssey4meI was thinking of doing this instead: printenv | grep ^ZUUL > zuul.env || true11:10
AJaegerodyssey4me: but if you want to test, why not use "-n ${{SOME_VAR}}" ?11:11
odyssey4mebut I think there might be a precedence issue there which may not do exactly what's intended11:11
*** jkilpatr has joined #openstack-infra11:11
odyssey4meyeah, that could work too11:12
odyssey4mebut it's unnecessary to cage it, so I'll remove the test entirely11:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: OSA: Output ZUUL variables to a file  https://review.openstack.org/46090711:14
*** dizquierdo has quit IRC11:15
*** makowals has joined #openstack-infra11:17
*** yamamoto_ has quit IRC11:18
*** shardy is now known as shardy_lunch11:22
*** kornicameister has joined #openstack-infra11:26
*** Julien-zte has joined #openstack-infra11:26
*** dizquierdo has joined #openstack-infra11:27
*** kornica has quit IRC11:27
*** rhallisey has joined #openstack-infra11:30
*** ldnunes has joined #openstack-infra11:34
*** psachin has quit IRC11:37
*** csomerville has joined #openstack-infra11:40
*** cody-somerville has quit IRC11:40
*** gildub has joined #openstack-infra11:44
*** thorst has joined #openstack-infra11:44
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: CI test - never merge  https://review.openstack.org/46100111:45
openstackgerritSean Dague proposed openstack-infra/os-loganalyze master: fix missing space in systemd logs  https://review.openstack.org/46100211:45
sdagueclarkb: I'm out for the day starting soon, but https://review.openstack.org/461002 and https://review.openstack.org/460947 should get looked at.11:46
*** sree has joined #openstack-infra11:48
*** gordc has joined #openstack-infra11:49
*** tesseract has quit IRC11:50
*** psachin has joined #openstack-infra11:51
*** sree has quit IRC11:52
*** shardy_lunch is now known as shardy11:55
*** dave-mccowan has joined #openstack-infra11:55
*** tinwood is now known as tinwood_lunch11:56
*** ccamacho is now known as ccamacho|lunch11:57
*** psachin has quit IRC11:59
*** d0ugal_ has joined #openstack-infra12:02
*** zhenguo has quit IRC12:02
*** d0ugal has quit IRC12:04
*** rodrigopaiva has joined #openstack-infra12:04
*** tesseract has joined #openstack-infra12:05
kornicameisterif anyone has time to review python-openstackclient12:06
kornicameistersorry12:06
kornicameisterhttps://review.openstack.org/#/c/460506/12:06
kornicameisterthat one12:06
kornicameisterthat'd be great12:06
*** kornicameister has quit IRC12:06
*** EricGonczer_ has joined #openstack-infra12:06
openstackgerritJiri Stransky proposed openstack-infra/project-config master: TripleO: experimental container upgrade job on more repos  https://review.openstack.org/46100712:07
*** coreycb has quit IRC12:09
*** lucasagomes is now known as lucas-hungry12:11
*** dprince has joined #openstack-infra12:11
*** yamamoto has joined #openstack-infra12:15
*** yamamoto has quit IRC12:15
*** yamamoto has joined #openstack-infra12:18
*** coreycb has joined #openstack-infra12:19
*** coreycb has quit IRC12:19
*** coreycb has joined #openstack-infra12:20
*** chlong has quit IRC12:21
*** d0ugal_ is now known as d0ugal12:21
*** coreycb_ has joined #openstack-infra12:21
*** d0ugal is now known as Guest804912:21
*** coreycb_ has quit IRC12:21
*** Guest8049 has quit IRC12:21
*** d0ugal_ has joined #openstack-infra12:22
openstackgerritMichal Dulko proposed openstack-infra/project-config master: Add docs job for openstack-helm  https://review.openstack.org/46092512:23
openstackgerritMerged openstack-infra/project-config master: OSA: Output ZUUL variables to a file  https://review.openstack.org/46090712:23
*** pgadiya has quit IRC12:24
*** Kevin_Zheng has quit IRC12:27
openstackgerritHongbin Lu proposed openstack-infra/project-config master: [Zun] Add zun-ui server release job  https://review.openstack.org/46081512:29
*** jpena is now known as jpena|lunch12:32
*** markvoelker has joined #openstack-infra12:38
*** gcb has quit IRC12:40
*** trown|outtypewww is now known as trown12:40
*** rlandy has joined #openstack-infra12:41
weshayianw, looking for an infra review on https://review.openstack.org/#/c/458054/12:41
*** mriedem has joined #openstack-infra12:41
leifmadsenzxiiro-away: ah no problem, if I haven't figured it out in two weeks, I'll probably have lost interest :)12:41
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix CleanupWorker exception messages  https://review.openstack.org/46101912:44
*** yamamoto has quit IRC12:47
*** askb has quit IRC12:49
*** tinwood_lunch is now known as tinwood12:53
*** tinwood has quit IRC12:54
*** sree has joined #openstack-infra12:54
*** tinwood has joined #openstack-infra12:54
*** esikachev has quit IRC12:57
*** mat128 has quit IRC12:58
EmilienMfungi: would it be possible to remove the stable/mitaka branches for tripleo projects please?12:58
*** mat128 has joined #openstack-infra13:00
*** ihrachys has joined #openstack-infra13:01
*** jamesdenton has joined #openstack-infra13:03
*** Goneri has joined #openstack-infra13:04
*** lamt has joined #openstack-infra13:04
*** andreas_s has quit IRC13:04
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Fix repos for promotion jobs  https://review.openstack.org/46055813:04
*** jcoufal has joined #openstack-infra13:05
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci master: Do not merge: Test Ceph Jewel 10.2.7  https://review.openstack.org/46012113:06
*** john-davidge has quit IRC13:07
*** john-dav_ has joined #openstack-infra13:07
*** ccamacho|lunch is now known as ccamacho13:07
*** lucas-hungry is now known as lucasagomes13:08
*** lamt has quit IRC13:09
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add optional error_message to adapter.request  https://review.openstack.org/46102513:13
*** lamt has joined #openstack-infra13:14
*** dizquierdo has quit IRC13:15
*** mdrabe has joined #openstack-infra13:16
*** jaypipes is now known as leakypipes13:16
openstackgerritSean Dague proposed openstack-infra/devstack-gate master: Use compat names for journal files  https://review.openstack.org/46094713:20
rm_workIs there some log file from a gate run that shows the configuration for the $PROJECTS var?13:22
*** gildub has quit IRC13:22
rm_workAh nm just found it, reproduce.sh13:23
*** kgiusti has joined #openstack-infra13:25
*** d0ugal_ has quit IRC13:26
*** d0ugal has joined #openstack-infra13:27
*** d0ugal has quit IRC13:27
*** d0ugal has joined #openstack-infra13:27
*** bauzas is now known as bauwser13:27
openstackgerritAdam Harwell proposed openstack-infra/project-config master: Add python-octaviaclient to neutron-lbaas jobs  https://review.openstack.org/46103013:27
*** madhuvishy has quit IRC13:30
*** madhuvishy has joined #openstack-infra13:31
* AJaeger has two changes for infra-manual to update links https://review.openstack.org/#/c/459842/ and https://review.openstack.org/#/c/459844 - could anybody review these, please?13:32
dmelladoAJaeger: o/ we've been seeing a lot of failures in kuryr gates13:33
dmelladosuch as in here https://review.openstack.org/#/c/459292/613:33
dmelladoPOST_FAILURE13:33
dmelladoI wanted to check that out but without logs, is there any way13:33
dmelladoor would you recommend any13:33
AJaegerwhen did those start?13:33
dmelladoto how to debug these13:33
dmelladoI was tracking a 'suspect' commit13:33
dmelladowhich was the latest one that affected project.config some days ago13:34
AJaegerSo, it's always gate-install-dsvm-default-kuryr-kubernetes - it sometimes fails, sometimes works.13:34
dmelladohttps://github.com/openstack-infra/project-config/commit/a69a9880b16de3ffc2ed6bc9ddda7e2316e9424d13:35
dmelladothat one, and also /url 113:35
AJaegerdmellado: POST_FAILUREs often happen if your tests take down the machine, e.g. no network possible.13:35
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests  https://review.openstack.org/46078413:35
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests  https://review.openstack.org/46078413:36
dmelladoAJaeger: also gate-install-dsvm-dragonflow-kuryr-kubernetes-nv13:36
dmelladohave been failing a lot13:36
dmelladobut my question is13:36
dmelladoisn't is weird that it only happens 'sometimes'13:36
dmellado?13:36
dmelladoshould this be happing all the time or none13:36
AJaegerdmellado: it could happen in some cloud providers and not in others13:36
dmelladoAJaeger: I see, so it could depend on which cloud does the job end13:37
*** Guest24728 is now known as zeus13:37
*** zeus has quit IRC13:37
*** zeus has joined #openstack-infra13:37
AJaegerSo, a recheck brought this live, you can now look at logs via http://status.openstack.org/zuul/13:37
AJaegerThe gate job links to telnet://198.72.124.178:19885  , so I run nc (instead of telnet 198.72.124.178 19885 - and wait ;)13:38
dmelladoI'll check that, thanks for the tip AJaeger13:38
AJaegerdmellado: I do not have access to other logs myself, you need one of the infra-roots for that but nobody has spoken up here in the last few hours. Perhaps one of them has a better idea13:38
dmelladowill follow-up with this13:39
dmellado(or get yolanda at some point)13:39
AJaegerdmellado: yolanda is currently not around13:39
dmelladoAJaeger: I know ;)13:39
dmelladobut I guess I'll be able to reach her xD13:39
AJaegerdmellado: so, this job runs on ubuntu-xenial-internap-mtl01-8644644 - that's internap. See second line of output13:39
AJaegerwould be interesting to see where the jobs run that *succeeded*13:40
dmelladoAJaeger: also, dummy question, is there any way to log those logs?13:40
dmelladototally13:40
AJaegerdmellado: nc 198.72.124.178 19885 |tee -a mylogfile13:40
AJaegerdmellado: in general those are logged but only safed if the job completed13:41
rm_workhttps://review.openstack.org/#/c/461030/ would be great :) pretty trivial13:41
AJaegerdmellado: for the nodes used for successful runs - see the second line of log file13:41
AJaegerrm_work: what kind of change is that?13:41
rm_workproject-config, adds our client to the gates so we can merge the change that adds it to our devstack plugin without the gates barfing13:42
dmelladoyes, I see13:42
*** EricGonc_ has joined #openstack-infra13:42
dmelladoI'll need to check what's up with the underneath infra13:42
rm_workit's a copy of a patch I did yesterday that already merged, I had only done it for Octavia but also needs to be done for Neutron-LBaaS13:43
*** srobert has joined #openstack-infra13:43
*** EricGonczer_ has quit IRC13:43
AJaegerrm_work: LGTM13:43
*** jpena|lunch is now known as jpena13:45
*** EricGonc_ has quit IRC13:45
*** mat128 has quit IRC13:45
*** Dinesh_Bhor has quit IRC13:46
rm_workthanks :)13:47
*** jcoufal_ has joined #openstack-infra13:48
*** jamesmca_ has joined #openstack-infra13:48
*** yamamoto has joined #openstack-infra13:48
*** lamt has quit IRC13:49
*** jcoufal has quit IRC13:50
*** eharney has joined #openstack-infra13:50
*** lamt has joined #openstack-infra13:50
AJaegerdmellado: heisenbug - you tail the logs and it works ;)13:52
*** jamesmca_ has quit IRC13:52
*** chlong has joined #openstack-infra13:53
*** sree has quit IRC13:55
openstackgerritBrad P. Crochet proposed openstack-infra/project-config master: Make the TripleO job non-voting on Mistral for now  https://review.openstack.org/46103913:55
dmelladolol xD13:56
pabelangermorning13:57
fungiEmilienM: any of our infra-root gerrit admins can delete branches, but i'll see if i can script up something for a mass branch deletion for you here in a bit13:57
EmilienMfungi: thanks, i've asked multiple times but I guess folks are just busy. Really, it's just a very few projects13:57
AJaegerfungi, jhesketh has done the liberty retirement - perhaps he has a script?13:58
*** mdrabe has quit IRC13:58
*** baoli has joined #openstack-infra13:59
fungiwell, it's more just that i need to make sure i have all their deliverables cloned, check for stable/mitaka branches in them, check that there are mitaka-eol tags on them, and then try to push an empty ref... can be done in a one-liner shell loop without too much trouble. though if it's only a handful of repos maybe using the webui is easier14:00
*** makowals has quit IRC14:01
fungiEmilienM: did you send a list of which ones you want deleted to one of the mailing lists so i have a quick reference, or should i just loop through all the tripleo deliverables in governance?14:01
*** apuimedo has joined #openstack-infra14:01
pabelangerdmellado: AJaeger: POST_FAILURE usually are issues with infrastructure, however in the case of your job, it was aborted for job timeout14:02
EmilienMfungi: stable/mitaka branches for instack-undercloud, tripleo-heat-templates, tripleo-puppet-elements, puppet-tripleo, python-tripleoclient and tripleo-common14:02
dmelladopabelanger: it was a job timeout?14:03
dmelladohmmm14:03
dmelladoI'm trying to nc now here14:03
dmelladoto check what's going on14:03
dmelladotelnet://[2001:4800:1ae1:18:f816:3eff:fecd:1bcc]:1988514:03
* dmellado needs to check if his ipv6 stack is on14:03
pabelangerdmellado: it ran for 2h 26m 10s14:03
dmelladobut I'd say tihs is OSIC14:03
fungidmellado: `nc 2001:4800:1ae1:18:f816:3eff:fecd:1bcc 19885`14:03
EmilienMfungi: also, https://github.com/openstack/instack has old branches like feature/V2 and mgt-kilo. You can also remove them safely.14:03
dmelladofungi: as I said14:04
AJaegerpabelanger, dmellado: Argh, confused it with RETRY_LimiT ;/14:04
dmelladosadly, no route to host14:04
dmelladoargh14:04
dmelladoon Ipv614:04
dmelladopabelanger: AJaeger fungi should I assume IPv6==OSIC?14:04
fungidmellado: no working ipv6 maybe? i can reach it14:04
dmelladofungi: most probably that14:04
fungidmellado: or is your isp blocking egress to 19885/tcp?14:04
dmelladofungi: nope, that worked just before14:05
dmelladofungi: would you be so kind of logging that execution?14:05
pabelangerAJaeger: RETRY_LIMIT is also likely infra related, so far we lost SSH access to remote node, or zuul-launcher couldn't clean up a job propelry14:05
pabelangerdmellado: fungi: nc WFM14:05
fungidmellado: sure, i can tail that to a file locally and then upload it somewhere. right now it's executing stack.sh as of ~3.5 minutes ago14:06
*** eharney has quit IRC14:06
*** claudiub has quit IRC14:07
*** chlong has quit IRC14:07
*** felipemonteiro has joined #openstack-infra14:07
dmelladomigt be IPv6 issue on my end14:08
dmelladofungi: thanks!14:08
dmelladowe've been hitting such errors for a while on a couple of patches14:08
dmelladoand I'd like to dig this out14:08
*** felipemonteiro_ has joined #openstack-infra14:09
fungiEmilienM: i see an ARG-PRINT-TEMPLATES branch on os-apply-config, do you want that removed too?14:09
EmilienMfungi: please14:09
EmilienMfungi: old legacy things, I wasn't born at this time.14:10
*** jaosorior_away has quit IRC14:10
fungiyeah, probably an import artifact14:11
fungisome topic branch someone had on the origin before moving into gerrit14:11
dmelladoEmilienM: maybe you could find BASIC code within that branch xD14:11
fungidmellado: i was hoping for algol-6014:12
*** felipemonteiro has quit IRC14:12
EmilienMdmellado: lol14:12
dmelladofungi: I had to search that out14:12
dmelladobut maybe you'd prefer logos14:12
dmelladoI do recall playing with that drawing turtle xD14:13
fungiyeah, logo was an amusing language14:13
EmilienMfungi: once mitaka branches are removed, we can review https://review.openstack.org/#/c/460347/14:13
*** dizquierdo has joined #openstack-infra14:13
*** rbrndt has joined #openstack-infra14:13
fungiEmilienM: i see that tripleo-quickstart-extras has a branch called "develop" (last updated 5 months ago)14:14
fungiwant that added to the deletion queue too?14:14
AJaegerEmilienM: I'll happily +2 it once fungi gives the green light ;914:14
EmilienMweshay: ^ can we go ahead?14:14
*** prateek has quit IRC14:14
EmilienMAJaeger: thanks !14:14
*** ltomasbo has joined #openstack-infra14:15
*** yamamoto has quit IRC14:16
weshayah.. remotes/gerrit/develop look at that.. EmilienM AJaeger yes14:16
EmilienMfungi: ^ shoot it14:17
weshaythanks14:17
openstackgerritBrad P. Crochet proposed openstack-infra/project-config master: Make the TripleO job non-voting on Mistral for now  https://review.openstack.org/46103914:17
AJaegerpabelanger: could you +2A https://review.openstack.org/460850 as well to sync useradd commands, please?14:17
*** esikachev has joined #openstack-infra14:18
AJaegerThanks!14:18
*** aarefiev is now known as aarefiev_afk14:19
openstackgerritMerged openstack-infra/project-config master: Move jenkins cache to /opt/cache  https://review.openstack.org/46083514:19
*** chlong has joined #openstack-infra14:20
fungiEmilienM: so i _think_ these are all the branches you want deleted... can you confirm? http://paste.openstack.org/show/60833714:21
*** eharney has joined #openstack-infra14:21
EmilienMfungi: that is perfect to me14:22
*** tphummel has joined #openstack-infra14:22
EmilienMfungi: I still see them on github but I guess it's not synced yet?14:22
fungiEmilienM: i haven't deleted anything yet. i'm still prepping14:23
fungiEmilienM: i see there are no "mataka-eol" tags pushed to the tips of those stable/mitaka branches. do you want those tags added too?14:23
fungier, "mitaka-eol" tags i mean, of course14:23
*** beslemon has joined #openstack-infra14:23
*** dingyichen has quit IRC14:23
*** gcb has joined #openstack-infra14:24
EmilienMfungi: if you can do it, that would be awesome14:25
fungiokay, i'll push those too14:25
openstackgerritMerged openstack-infra/project-config master: Always add zuul group to zuul user  https://review.openstack.org/46085014:25
*** yamahata_ has joined #openstack-infra14:25
EmilienMthanks14:25
*** hongbin has joined #openstack-infra14:26
AJaegerpabelanger: https://review.openstack.org/460252 can be abandoned, and https://review.openstack.org/460728 used instead - don't you think?14:28
*** wolverineav has joined #openstack-infra14:28
*** adrian_otto has joined #openstack-infra14:30
*** rvba has quit IRC14:30
*** armax has joined #openstack-infra14:33
*** ggillies has quit IRC14:34
pabelangerAJaeger: ya, just reviewing that patch now14:34
*** ggillies has joined #openstack-infra14:35
*** annegentle has quit IRC14:36
AJaegerthanks, pabelanger14:37
*** booleanfour has joined #openstack-infra14:40
*** booleanfour has quit IRC14:40
fungiEmilienM: mitaka-eol tagging and branch deletion has now been completed for official tripleo deliverable repositories. please double-check that everything looks correct to you14:40
EmilienMfungi: ok, I'm looking now14:41
AJaegerfungi: do you want to +2A https://review.openstack.org/#/c/460347/ then?14:41
AJaeger(once EmilienM is happy;)14:41
EmilienMfungi, AJaeger: it looks perfect to me14:41
*** adrian_otto has quit IRC14:44
*** dmsimard is now known as dmsimard|cavern14:44
*** gouthamr has joined #openstack-infra14:45
AJaegercmurphy:  mkdir -p ~jenkins/cache/files is done in change Ibca6867f29b257a5110cb5522a5cca3a97fa9377 - isn't it?14:45
*** ramishra has quit IRC14:46
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: DNM - deploy overcloud with debug  https://review.openstack.org/45703714:46
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: DNM - deploy overcloud with debug  https://review.openstack.org/45703714:46
*** esikachev has quit IRC14:46
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests  https://review.openstack.org/46078414:48
cmurphyAJaeger: aha, looks like14:48
*** annegentle has joined #openstack-infra14:48
*** gcb has quit IRC14:48
*** gcb has joined #openstack-infra14:49
AJaegercmurphy, pabelanger, so do we need to remove the mkdir to fix our images?14:49
pabelangerAJaeger: fix our images?14:50
AJaegerpabelanger: building our images14:50
cmurphyAJaeger: I thought 460835 fixed whatever the problem was?14:50
pabelangerAJaeger: sorry, not sure what the issue current is. Could you explain14:50
*** dizquierdo has quit IRC14:50
AJaegercmurphy: I'm tired and confused right now, so assume I'M wrong ;/14:50
openstackgerritMerged openstack-infra/project-config master: Reproduce non-puppet parts from install_puppet.sh  https://review.openstack.org/46071314:50
* AJaeger grabs a link...14:50
openstackgerritMerged openstack-infra/project-config master: tripleo: EOLing Mitaka  https://review.openstack.org/46034714:50
cmurphyi don't think running mkdir -p twice will hurt14:50
AJaegerI see http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/jenkins-slave/install.d/20-jenkins-slave#n1914:51
fungithe -p makes it idempotent, yes14:52
*** esikachev has joined #openstack-infra14:52
AJaegermkdir -p /home/jenkins/cache; ln -sf /opt/cache/files /home/jenkins/cache/files14:52
AJaegerand we have also "mkdir -p ~jenkins/cache/files"14:52
AJaegerso, a symbolic link vs a directory14:52
*** dprince has quit IRC14:53
pabelangerya, we should drop mkdir -p ~jenkins/cache from prepare-node. I don't think we need / want that any more. We added it to jenkins-slave, to ensure we create the home directory first with adduser14:53
pabelangerother wise, we get a warning14:53
pabelangeruseradd: warning: the home directory already exists.14:53
fungiyou _can_ tell useradd not to create the homedir14:54
fungibut granted, you can also rely on it to do so as long as you move any steps relying on its existence to happen afterward14:54
EmilienMfungi, AJaeger: thank you again14:54
*** jamesmca_ has joined #openstack-infra14:54
fungiEmilienM: you're welcome! apologies for the delay14:54
mtreinishclarkb, fungi, pabelanger, mordred: are you around today to help get the last 2 things publishing on firehose? https://review.openstack.org/#/q/status:open++topic:more-firehose14:55
EmilienMfungi: no worries at all14:55
pabelangerfungi: yes, that is what we are doing now in elements14:55
mtreinishI promise these will be the last service adds until after the summit at least14:55
fungimtreinish: yes, i am. it's rising in priority for me ;_14:55
fungi;)14:55
mtreinishfungi: heh, ok cool :)14:55
fungi(along with my other cnoference-related obligations)14:55
fungic-no-ference14:56
AJaegerfungi, that bad? ;)14:56
cmurphypabelanger: I was going to run some reproduce.sh's on https://review.openstack.org/#/c/460728/ but I don't think I'll have time tonight, if you have the time and desire to do that today feel free otherwise I'll get to it over the weekend14:56
fungiAJaeger: at least bad enough i can't type straight anymore14:56
cmurphypabelanger: also if you want to remove the prepare-node in that patch go ahead14:56
mtreinishI'm looking forward to the ansible callback. Because I can hopefully use that to actually see when my puppet changes are applied14:56
mtreinishsince puppetboard isn't a thing anymore14:56
sbezverkpabelanger: recent changes of infra, could impact connectivity between two VMs running on the same gate job node?14:56
pabelangercmurphy: sure, I don't think we're merge today, we can aim for monday14:57
pabelangerI'll see about running a test too14:57
sbezverkpabelanger: until very recently we could ping between two VMs we spawn inside of a gate job14:57
pabelangersbezverk: yes, it is possible ssh host key check might fail. Do you have a log file?14:58
dmelladofungi: hmm I bet that telnet://[2001:4800:1ae1:18:f816:3eff:fecd:1bcc]:19885 got stuck14:58
*** andreww has joined #openstack-infra14:58
dmelladois there something suspicious on the log?14:58
fungisbezverk: pabelanger: maybe recent refactoring of how we set up iptables on job nodes has impacted some connectivity between nested instances?14:58
sbezverkpabelanger: ssh key part is fixed but what is failing is just a regular ping..14:58
*** bnemec is now known as beekneemech14:59
fungidmellado: this is the last line in the console log...14:59
pabelangeryes, iptables setup was changed last week too14:59
fungi2017-04-28 14:02:50.100084 | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:main:L729:   /tmp/ansible/bin/ansible primary -f 5 -i /home/jenkins/workspace/gate-install-dsvm-default-kuryr-kubernetes/inventory -m shell -a 'cd '\''/opt/stack/new/devstack'\'' && sudo -H -u stack DSTOOLS_VERSION=0.3.0 stdbuf -oL -eL ./stack.sh 2>&1 executable=/bin/bash'14:59
*** aarefiev_afk has quit IRC14:59
fungidmellado: so nothing output on the console for nearly an hour14:59
AJaegerfungi, a new telnet to that line does not give any output14:59
AJaegerfungi, dmellado, so might be we killed the network?14:59
fungii'll see when zuul lost contact with it15:00
sbezverkpabelanger: so what should I do then?15:00
pabelangersbezverk: which job is this? Do you have a log?15:00
sbezverkpabelanger: if I change iptables inside a job would it be sufficient?15:00
pabelangersbezverk: lets see if we can figure out the issue first15:01
*** xarses_ has joined #openstack-infra15:02
dmelladoAJaeger: could be15:02
*** andreww has quit IRC15:02
sbezverkpabelanger: http://logs.openstack.org/16/461016/1/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/383e702/console.html#_2017-04-28_14_20_41_84343215:03
dmelladoAJaeger: fungi but what I'm interested into is, in that case, why did that happen?15:03
fungidmellado: yeah, i was hoping the timing might tell us something15:04
pabelangersbezverk: http://logs.openstack.org/63/461063/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/de04fbb/console.html#_2017-04-28_14_54_31_484663 is an example of ssh host key failure, you likley need https://review.openstack.org/#/c/460783/15:04
pabelangersbezverk: looking at log you linked now15:04
*** rossella_s has quit IRC15:04
AJaegerdmellado: I can't help you further on this...15:04
*** zz_dimtruck is now known as dimtruck15:04
dmelladoAJaeger: np! thanks for your help in any case15:04
*** rossella_s has joined #openstack-infra15:05
dmelladoI'll try to reproduce something locally15:05
dmelladothe only difference within the working/non working that comes to my mind15:05
dmelladois the cloud provider and IPv4/IPv615:05
dmelladoI'll check what happens exactly on those commands15:05
fungidmellado: i've held the node, but it seems i get no response on its ip address15:06
fungiso something is taking out its network connectivity (at least ipv6 connectivity since that's all we have ingress for that provider)15:06
*** dprince has joined #openstack-infra15:06
clarkbdmellado: fungi possibly another case of disabling RAs?15:06
fungicertainly possible15:07
dmelladoclarkb: RAs?15:07
fungirouter announcements15:07
dmelladooh, on IPv615:07
clarkbits how you know what your ipv6 address is in some cases15:07
fungii'll hop into another running instance there and see if i can reach its linklocal address15:07
clarkb(and its what we use in our case)15:07
dmelladoyeah15:07
dmelladoclarkb: thanks!15:07
fungis/announcements/advertisements/15:07
sbezverkpabelanger: I added ssh key workaround in this ps and it helped15:07
clarkbwe've seen neutron related testing break that in the past15:07
clarkbby disabling processing of them in the linux kernel15:08
sbezverkpabelanger: but ironic job is constantly failing so worried to merge it..15:08
clarkbsbezverk: pabelanger that looks like the job is creating an overlay network for 172.21.0.0/24 and the instance is booting on the subnode then the overlay network can't carry the traffic to it for some reaosn15:08
clarkbsbezverk: pabelanger perhaps the overlay isn't set up properly or routes don't exist etc15:08
fungiyeah, ra support is generally disabled for devices configured as routers, so that it doesn't compete with your more advanced routing protocols15:09
pabelangerya, that is what I am thinking too.  I don't see any mention of it in iptables logs15:09
dmelladoclarkb: fungi hm I see, that would explain why it worked just fine on the IPv4 node15:10
dmelladoso it looked like a 'cloudroulette' xD15:10
fungilooks like it might actually be breaking ipv6 entirely15:12
fungii can't reach its linklocal v6 address but _can_ reach its rfc-1918 address from another instance there15:12
sbezverkclarkb: that job was working 100% before the last week15:12
pabelangersbezverk: do you know when it started failing?15:12
clarkbfungi: check sysctl ra values?15:13
pabelangerwe can see which image it was using15:13
*** rvba has joined #openstack-infra15:13
*** rvba has quit IRC15:13
*** rvba has joined #openstack-infra15:13
fungiclarkb: well, need to work out how to get myself access to it15:13
clarkbfungi: ssh -A through mirror?15:13
fungibut it's definitely not participating in neighbor discovery on its linklocal15:13
fungii'll see if i can get that working15:14
AJaegerclarkb, pabelanger, I have a small change to remove single node lists in projects.yaml - could you put that on your review queue, please? https://review.openstack.org/45904915:15
clarkbsbezverk: pabelanger ovs.txt and ovs-init.txt are both empty files. Not sure if that is used for the overlay and expected to have content15:16
clarkbit also looks like the logs are primarily from the primary node so we can't really see what the subnode did?15:16
pabelangerHmm, is it possible this could be related to switching to centos octca from rdo ?15:16
pabelangerit would be good to see when it started failing15:17
sbezverkclarkb: internal "net" is built using linux bridges15:17
clarkbsbezverk: is that where 172.21.0.0/24 is being used/15:17
*** openstackgerrit has quit IRC15:17
clarkbsbezverk: and can you share where that is being configured?15:18
sbezverkclarkb: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate_common.sh#L3215:20
dmelladoclarkb: fungi let me know if I can help you debugging that in any another way that doesn't require infra-core powers15:21
dmellado:D15:22
fungidmellado: i've gotten into it and confirmed ipv6 seems entirely broken on the node. i'm getting some details for you now15:22
clarkbhaving logs from the subnode(s) would be very helpful in this case too. I see the primary node ip listing which show br-tenants and 172.21.0.10/24 which is where we are pinging 172.21.0.110/24 so the routes are configured properly on that side at least but not sure about remote15:22
sbezverkclarkb: check this out just run gatecheck http://logs.openstack.org/63/461063/2/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/b408799/console.html#_2017-04-28_15_16_44_15562015:23
clarkbsbezverk: unrelated but that script could use indentation :P15:23
sbezverkclarkb: for sure :)15:24
dmelladofungi: thanks!15:24
clarkbsbezverk: net2 is 172.22.0.0/24 not 172.21.0.0/24. 172.21.0.0/24 is on br-tenants15:24
fungidmellado: http://paste.openstack.org/show/60834315:24
fungidmellado: even linklocal addressing is gone from ens3 now15:25
dmellado:\15:25
fungidmellado: it's like something completely disabled ipv6 support on this machine15:25
sbezverkclarkb: it seems connectivity issue is intermittent15:25
dmelladoI'll try to spawn an ipv6 dsvm and see if I can reproduce this15:25
dmelladobut that's quite weird!15:26
clarkbsbezverk: do you know where br-tenants is configured? and how it is overlaying between test nodes?15:26
fungisbezverk: net.ipv6.conf.all.disable_ipv6 and net.ipv6.conf.default.disable_ipv6 are set to 1, so that's why, the bigger question is what's setting that?15:27
fungidmellado: ^15:28
fungi(sorry sbezverk)15:28
dmelladoOo15:28
clarkbsbezverk: as for being intermittent my guess is that it works when your ping test is to an instance on the priamry node and it fails when it is on the subnode so depends on how the scheduling happens15:28
dmelladofungi: I'll need to check that15:28
fungidmellado: so something seems to be explicitly turning off ipv6 support for you15:28
*** pcaruana has quit IRC15:28
clarkbcan't have ipv615:29
clarkbtoo retro15:29
fungiwelcome to the 20th century!15:29
*** mdrabe has joined #openstack-infra15:29
sbezverkclarkb: https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/deploy_iscsi_common.sh#L24615:29
sbezverkclarkb: ironic job is single node job, everything happens on the same gate VM15:30
clarkbsbezverk: http://logs.openstack.org/16/461016/1/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/383e702/console.html#_2017-04-28_13_41_51_248665 no its a two node job15:30
clarkbsbezverk: so thats my best guess. Its two node job. You don't actually have an overlay on that l2, just a local bridge. If thing schedules to subnode it breaks15:31
sbezverkclarkb: could you please clarify where you see two nodes? If it is 2 node job it is mistake somewhere as we need a signle node for it..15:32
clarkbsbezverk: on the link above, see the hostname15:32
clarkbcentos-7 2-node rax-ord NNNNNNNN15:32
sbezverkclarkb: oh man I thought 7-2 means centos 7.215:33
sbezverkclarkb: regardless in the job there is no logic to deal with second node. it is always uses main node.. I guess I will have to push a change to switch it to a single node..15:34
*** chlong has quit IRC15:34
dmelladofungi: clarkb I'll try to reproduce this and get back to you, thanks for your help!15:35
fungidmellado: you're welcome. i'll keep this specific node held for a little longer in case there are particular logs you want checked15:35
clarkbsbezverk: yes I think thats step 0. If problem persists because second node was just unused and issue is in single node too then I'm going to guess the ironic container isn't being attached to that bridge15:35
clarkbsbezverk: so your next step would be to list all the things attached to the bridge15:36
*** sdague has quit IRC15:36
clarkbsbezverk: I think brctl show br-tenants is what you want for doing that15:39
clarkbmaybe also brctl show15:40
sbezverkclarkb: could you point where I can specify that only single node is required for the job?15:42
*** yolanda has joined #openstack-infra15:43
clarkbsbezverk: change the node in your jjb config to centos-7 instead of centos-7-2-node15:43
*** Apoorva has joined #openstack-infra15:43
*** dkehn has quit IRC15:44
sbezverkclarkb: got it, thanks15:45
*** openstackgerrit has joined #openstack-infra15:47
openstackgerritKhai Do proposed openstack/python-jenkins master: Should use fullname instead of name  https://review.openstack.org/45908715:47
*** jpena is now known as jpena|off15:48
openstackgerritSerguei Bezverkhi proposed openstack-infra/project-config master: Switch kolla-kubernetes ironic job for single node  https://review.openstack.org/46108515:49
clarkbactually wait its an ovs bridge?15:49
clarkbso brctl wont work15:49
clarkband I always have to go re remember ovs things but same idea15:50
sbezverkclarkb: https://review.openstack.org/46108515:50
*** eranrom_ has quit IRC15:50
sbezverkclarkb: we noticed as soon as ironic job lands on rax-ord there is a good chance it will fail15:50
*** eranrom has joined #openstack-infra15:51
sbezverkclarkb: so hopefully once it is switched to single node, it will not land there anymore :)15:51
dmelladofungi: clarkb I think I found the issue15:51
dmelladowill work on it next week15:51
clarkbwell rax-ord is valid for all the labels15:51
dmelladohttps://github.com/openstack/kuryr-kubernetes/blob/master/devstack/plugin.sh#L113-L11615:52
*** gyee has joined #openstack-infra15:52
*** eranrom has quit IRC15:53
*** eranrom has joined #openstack-infra15:53
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Rename gate-dox-docker-save to gate-dox-buildimages-nv  https://review.openstack.org/46108715:55
clarkbsbezverk: is the ironic host a VM?15:56
clarkbsbezverk: this host http://logs.openstack.org/16/461016/1/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/383e702/logs/vm-1.log ? there is no log which seems supsicious15:57
sbezverkclarkb: inside on VM infra gives us I create a nested VM using virsh15:57
sbezverkand use it as a "baremetal" node15:57
*** mhickey has quit IRC15:58
clarkbgotcha so thats the other thing I would check15:58
clarkbwhy is there no log for the vm?15:58
*** dkehn has joined #openstack-infra15:59
sbezverkclarkb: well virsh and nova clearly indicates that VM sucessfully started15:59
clarkbsbezverk: yup so maybe you are getting the wrong log file?15:59
clarkbbut I bet console logs from that VM will also be useful15:59
clarkbyou can see if it bootstrapped networking properly and such16:00
sbezverkclarkb: right I should be able to get those logs from /var/lib/libvirt orsomething..16:00
clarkbyup16:00
*** david-lyle has quit IRC16:00
clarkbpossibly even the nova api?16:00
sbezverkclarkb: check this out, it landed on a different node and now success :) http://logs.openstack.org/16/461016/2/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/5dc77b2/console.html#_2017-04-28_16_00_32_15703316:02
* fungi wonders if the "breaks on rax-something" is tied more to running on xen. does the job maybe rely on working nested-virt acceleration support?16:02
clarkbthe xml says domain type = qemu16:03
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Remove old jobs for Tripleo Heat Templates  https://review.openstack.org/46109016:04
EmilienMfungi, pabelanger, AJaeger: please review this one if you can ^ I broke tripleo CI16:04
EmilienMwith the mitaka EOL I forgot to remove these 2 jobs16:04
*** tie has quit IRC16:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use the security context of the playbook when checking out roles  https://review.openstack.org/46109216:05
clarkbfungi: sbezverk one thing I notice is that the first 6 pings fail. maybe in the rax case you aren't waiting long enough for it to all be working properly?16:05
clarkbalso there are significantly fewer logs on the successful job so hard to compare16:06
*** spzala has quit IRC16:06
fungiEmilienM: is there corresponding cleanup for that in jenkins/jobs/projects.yaml too?16:06
*** spzala has joined #openstack-infra16:06
AJaegerEmilienM: quick suggestion16:06
clarkbsbezverk: maybe try a longer ping and see if it comes up16:06
sbezverkclarkb: right, we do not collect logs on successful run16:06
EmilienMfungi: no, it's a template that is used by puppet ci16:07
*** trown is now known as trown|lunch16:07
fungiEmilienM: but do you instantiate that template for the project?16:07
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Remove old jobs for Tripleo Heat Templates  https://review.openstack.org/46109016:07
EmilienMAJaeger: done16:07
* fungi checks16:07
*** ltomasbo is now known as ltomasbo|away16:08
*** mdrabe has quit IRC16:08
*** ykarel has quit IRC16:09
*** adrian_otto has joined #openstack-infra16:09
EmilienMfungi: no I don't16:10
fungiEmilienM: oh, i see it's a job-group you're adding16:10
EmilienMyeah16:10
fungiso we'll still be generating those jobs for now unless we create a different job-group or explicitly specify the templates to instantiate for the project16:11
fungigood enough16:11
*** mhickey has joined #openstack-infra16:11
*** spzala has quit IRC16:11
EmilienMthanks for the quick review. It should be good otherwise16:11
openstackgerritMartin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode  https://review.openstack.org/45415216:12
AJaegercould somebody +2A this project-config change to unblock monasca, please? https://review.openstack.org/46050616:14
*** shardy has quit IRC16:14
*** derekh has quit IRC16:15
*** mdrabe has joined #openstack-infra16:16
*** chlong has joined #openstack-infra16:16
AJaegerproject-config cores, could you review these, please? They fix existing jobs: https://review.openstack.org/458392 https://review.openstack.org/458391 https://review.openstack.org/457484 https://review.openstack.org/439840 https://review.openstack.org/459827 https://review.openstack.org/460810 https://review.openstack.org/46103016:17
*** esikachev has quit IRC16:17
*** jcoufal has joined #openstack-infra16:20
*** esikachev has joined #openstack-infra16:21
*** matrohon has quit IRC16:21
*** jcoufal__ has joined #openstack-infra16:22
*** jcoufal_ has quit IRC16:22
openstackgerritMerged openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin  https://review.openstack.org/46050616:24
*** jcoufal has quit IRC16:24
openstackgerritMerged openstack-infra/project-config master: Remove old jobs for Tripleo Heat Templates  https://review.openstack.org/46109016:25
*** esikachev has quit IRC16:26
*** eharney has quit IRC16:28
*** annegentle has quit IRC16:28
*** annegentle has joined #openstack-infra16:30
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Create stdlib.prepare-workspace role  https://review.openstack.org/45906616:31
*** beslemon has quit IRC16:32
sbezverkclarkb: or pabelanger: could you please ack this one  https://review.openstack.org/46108516:32
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Create stdlib.prepare-workspace role  https://review.openstack.org/45906616:32
*** jcoufal has joined #openstack-infra16:33
*** ykarel has joined #openstack-infra16:34
*** jcoufal__ has quit IRC16:34
*** jascott1 has joined #openstack-infra16:36
*** JayF has joined #openstack-infra16:36
*** dave-mccowan has quit IRC16:36
JayFIs there an easy way to unsubscribe from all reviews in gerrit, or disabling email notifications of review?16:36
*** jpich has quit IRC16:37
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Clean up driverfixes branch jobs  https://review.openstack.org/46110216:37
clarkbJayF: there are a subset of reviews that gerrit doesn't let you unsubscribe from iirc. Basically events you have pushed patchsets for?16:38
*** rossella_s has quit IRC16:38
*** esikachev has joined #openstack-infra16:38
clarkbJayF: otherwise look in your account settings under watched projects to remove those16:38
JayFclarkb: ah, that's probably most of mine then. So I should just filter them to /dev/null until every patch I've touched is abandoned for filtered through?16:38
JayFs/for/or/16:38
clarkbJayF: ya16:38
JayFAight, sensible enough. Just a periodic sad coming straight into my inbox, lol16:39
*** erikmwil_ has joined #openstack-infra16:41
*** eharney has joined #openstack-infra16:41
openstackgerritMerged openstack-infra/project-config master: Switch kolla-kubernetes ironic job for single node  https://review.openstack.org/46108516:42
clarkbpabelanger: fungi: simple review on osloganalyze to make systemd things happier https://review.openstack.org/#/c/461002/ much appreciated if you can review it16:42
*** erikmwilson has quit IRC16:42
*** esikachev has quit IRC16:42
*** esikachev has joined #openstack-infra16:43
*** eranrom_ has joined #openstack-infra16:43
pabelanger+316:44
*** dtantsur is now known as dtantsur|afk16:45
openstackgerritMonty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests  https://review.openstack.org/46078416:45
*** tesseract has quit IRC16:45
clarkbthanks16:46
*** markus_z has quit IRC16:46
*** eranrom has quit IRC16:46
openstackgerritKendall Nelson proposed openstack-infra/storyboard master: Handle Branches from Launchpad in Migration  https://review.openstack.org/43746916:47
*** jcoufal_ has joined #openstack-infra16:47
clarkbmtreinish: i opened your gerrit topic thinking it would just be a couple changes :P16:47
*** flepied has quit IRC16:47
*** ildikov is now known as coffee_cat16:47
*** ldnunes has quit IRC16:47
openstackgerritMerged openstack-infra/os-loganalyze master: fix missing space in systemd logs  https://review.openstack.org/46100216:47
*** esikachev has quit IRC16:48
openstackgerritAshwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests  https://review.openstack.org/46082216:48
*** ldnunes has joined #openstack-infra16:48
rm_workproject-config change could use a +A :) https://review.openstack.org/#/c/461030/16:49
*** david-lyle has joined #openstack-infra16:50
*** jcoufal has quit IRC16:50
*** JayF has left #openstack-infra16:50
openstackgerritMerged openstack-infra/project-config master: Do not use single item lists for nodes  https://review.openstack.org/45904916:50
*** esikachev has joined #openstack-infra16:51
*** tosky has quit IRC16:52
*** jamesmca_ has quit IRC16:52
openstackgerritAshwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests  https://review.openstack.org/46082216:53
*** lucasagomes is now known as lucas-afk16:54
*** jascott1_ has joined #openstack-infra16:54
openstackgerritMerged openstack-infra/project-config master: Adding periodic job running tempest in tripleo ocata release  https://review.openstack.org/45805416:54
*** ijw has joined #openstack-infra16:54
clarkbrm_work: I went ahead and approved it, but keep in mind devstack default should continue to be installing that client from pypi so I think that change is a noop?16:55
*** ijw has quit IRC16:56
*** esikachev has quit IRC16:56
rm_workit allows us to run a git_clone as part of our devstack script16:56
rm_workbecause it'll already be there16:56
clarkbyes, but you shouldn't do that for libs/clients unless libs from git is set for that lib16:56
rm_workhmm... following the model of another project16:56
*** ijw has joined #openstack-infra16:56
clarkb(and if its set we automagically add the project to the projects list for you)16:56
openstackgerritAshwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests  https://review.openstack.org/46082216:57
rm_workbut outside of the gate, that doesn't work16:57
rm_workif you set LIBS_FROM_GIT in devstack outside of a gate job, it doesn't automatically install it, all it does is fail a check16:57
*** jascott1 has quit IRC16:57
clarkbwhat?16:57
openstackgerritMerged openstack-infra/irc-meetings master: Updating co-chair of Neutron L3 meeting  https://review.openstack.org/46066016:57
clarkbLIBS_FROM_GIT is entirely about do I install from git source or do I use pypi16:58
clarkband then error if I detect the wrong one was used16:58
rm_workerr, wait i think i'm mixing up the two approaches i was taking16:58
rm_workbecause i tried that first, and then scrapped it and went to this16:58
rm_workbut I think maybe a mixture would do what you're saying16:58
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Validate flavor specification in config  https://review.openstack.org/45187516:58
clarkband devstack-gate now automatically adds you to PROJECTS list if in LIBS_FROM_GIT16:59
clarkbas a convenience16:59
rm_workthough this is the model that Barbican is using currently, I just used their pattern16:59
*** jamesmca_ has joined #openstack-infra16:59
rm_worklet me look at some other projects16:59
*** dave-mccowan has joined #openstack-infra16:59
*** rbrndt has quit IRC17:00
*** rbrndt has joined #openstack-infra17:00
*** rbrndt has quit IRC17:00
rm_workmmm yeah I see Magnum is doing it the LIBS_FROM_GIT way17:00
rm_worklet me try switching to that17:01
*** thorst has quit IRC17:01
*** ijw has quit IRC17:01
*** shadow-x has joined #openstack-infra17:01
*** trown|lunch is now known as trown17:02
*** shadow-x has quit IRC17:02
*** jamesmca_ has quit IRC17:02
openstackgerritMerged openstack-infra/project-config master: Add python-octaviaclient to neutron-lbaas jobs  https://review.openstack.org/46103017:03
rm_workclarkb: do you know if there is an easy way to specify a patch for a client lib if you use LIBS_FROM_GIT?17:03
clarkbrm_work: I don't. In the gate case being in the PROJECTS list ensures you have the correct commit checked out17:04
*** rbrndt has joined #openstack-infra17:04
clarkbrm_work: but for local stuff you'd likely have to check it out yourself locally or have some other variable to use a specific ref17:04
*** Sukhdev has joined #openstack-infra17:05
*** harlowja has quit IRC17:06
*** jcoufal has joined #openstack-infra17:07
*** e0ne has quit IRC17:08
*** mhickey has quit IRC17:09
*** asselin_ has joined #openstack-infra17:09
*** jcoufal_ has quit IRC17:09
mtreinishclarkb: heh, I did say a couple services :)17:10
mtreinishthe 2 big ones are the same that I linked to the other day17:11
*** asselin has quit IRC17:11
mtreinishfor the ansible callback and the logstash worker17:11
mtreinishthe logstash worker needed a few patches to get everything setup though17:11
*** yamahata_ has quit IRC17:11
*** thorst has joined #openstack-infra17:11
rm_workclarkb: ok, I switched to using that method, though it does actually make it more difficult to get the correct version if that's what you want17:12
rm_workbut, it's cleaner from a devstack perspective I guess17:12
*** annegentle has quit IRC17:15
odyssey4meclarkb are the reverse proxies for RDO in effect yet and ready for broader use?17:16
rm_workah actually, I think I see how the patch selection works, derp17:17
rm_workk hopefully this is good17:17
clarkbodyssey4me: they are there but are opt in17:18
clarkbpabelanger: EmilienM ^ do you have more info/17:18
odyssey4meok awesome, how do we opt in ?17:18
* fungi sets out on a quest for a midday repast, but shall return forthwith17:18
rm_workclarkb: thanks for the direction on that, I guess I picked the wrong project to emulate for clientlib stuff :P17:21
clarkbodyssey4me: I thoguht pabelanger sent email about it so searching for it (without much initial luck)17:22
rm_workso I guess neither of those project-config patches were necessary, but they also don't hurt, so it shouldn't really matter. in both cases we would have added it to PROJECTS anyway via LIBS_FOR_GIT, now it is just explicit17:22
rm_workso as you say, noop17:23
clarkbyup17:23
clarkbodyssey4me: but basically you use http:// $regionmirror.openstack.org/rdo as your rdo root url17:24
*** gcb has quit IRC17:24
*** asselin__ has joined #openstack-infra17:25
*** amotoki has quit IRC17:25
odyssey4mewould that perhaps be http://lists.openstack.org/pipermail/openstack-dev/2017-April/115137.html ?17:26
pabelangerI thought I did one for RDO too17:26
pabelangerhttp://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n5417:26
pabelangertd;dr source /etc/ci/mirror_info.sh; echo $NODEPOOL_RDO_PROXY17:26
clarkboh good17:27
clarkband yes I remember that now because intially the var was called _MIRROR and I asked to chagne ti to be specfiic17:27
*** asselin_ has quit IRC17:28
*** electrofelix has quit IRC17:28
pabelangerodyssey4me: http://lists.openstack.org/pipermail/openstack-dev/2017-April/114908.html17:29
odyssey4meso to use it, all we do is set the URL for the repo to http://mirror.dfw.rax.openstack.org:8080/rdo ?17:30
odyssey4meobviously the dns name changes per region17:30
*** persia has quit IRC17:30
pabelangerodyssey4me: ya, baseurl for repo file17:31
pabelangerlet me find patch to tripleo that set it up17:31
clarkbodyssey4me: ya you can source the mirror_info.sh script and use the $NODEPOOL_RDO_PROXY var to get the data17:31
odyssey4meok, sweet thanks - we're looking to speed up our CentOS builds and this may be very helpful17:31
pabelangerkolla is doing it with: http://git.openstack.org/cgit/openstack/kolla/tree/tools/setup_gate.sh#n5717:32
*** jamesdenton has quit IRC17:32
pabelangersame would apply, but in ansible :)17:32
odyssey4mecool, thank you very much sir17:33
pabelangerclarkb: just checked osic mirror, also see traffic on docker proxy. Maybe next week we can work on bumping our apache_cache a little more17:34
clarkbpabelanger: ya we should poll freespace on /opt across the nodes and go to something like 3/4 min remaining space17:35
*** jascott1_ has quit IRC17:37
*** Swami has joined #openstack-infra17:37
*** ijw has joined #openstack-infra17:37
*** jascott1 has joined #openstack-infra17:39
*** thorst has quit IRC17:41
*** ijw has quit IRC17:42
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove useless quotes for nodes  https://review.openstack.org/45906517:43
*** erikmwil_ has quit IRC17:44
*** rbrndt has quit IRC17:44
rm_workclarkb: ah, it's NOT a noop for neutron-lbaas17:47
*** sambetts is now known as sambetts|afk17:47
openstackgerritMonty Taylor proposed openstack/os-client-config master: Fix interactions with keystoneauth from newton  https://review.openstack.org/46111417:47
rm_workbecause it doesn't seem to add it to the PROJECTS list when it's from a sub-project17:47
*** flepied has joined #openstack-infra17:48
rm_worklike..17:48
clarkbrm_work: what is your LIBS_FROM_GIT?17:48
rm_workhttp://logs.openstack.org/99/460499/6/check/gate-neutron-lbaasv2-dsvm-api-ubuntu-xenial/1b6006c/17:48
rm_worklooks empty17:49
rm_workbecause17:49
*** neiljerram has quit IRC17:49
rm_workwe add it in octavia's devstack plugin, not neutron-lbaas17:49
clarkbyou have to set it on the devstack-gate side17:49
clarkbso that PROJECTS is set up properly17:49
rm_workah, so what magnum does isn't right either?17:49
clarkbmaybe? I haven't looked17:49
*** yamahata_ has joined #openstack-infra17:49
rm_workhttps://github.com/openstack/magnum/blob/master/devstack/plugin.sh#L1917:50
rm_worksorry was trying to pull that up17:50
rm_workthey add it to LIBS_FROM_GIT in their plugin.py17:50
rm_workerr17:50
rm_workplugin.sh17:50
clarkbhttps://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/nodepool.yaml#n111 is how we do it in nodepool17:51
rm_workso we need to do it in... our gate_hook?17:51
rm_workoh, so the same place I made the project-config change before17:51
rm_workjust a different thing17:51
clarkbno you have to do it before anything else runs eg at top level so that PROJECTS is set properly and the code is checked out17:51
clarkbgate_hook is too late as that has already checked out code17:51
rm_workaugh k17:51
rm_workso another project-config change17:51
clarkbalso you likely don't want to ahrdcode libs from git in your devstack plugin because otherwise the only way to install is from source17:52
rm_workIMO If you're using our plugin ... that's fine17:52
rm_workbut17:52
* rm_work shrugs17:52
clarkbrm_work: the reason its not that way by default is opeople running your software do it from releases17:52
clarkbso the idea was test that by default and make sure it works, then possibly depending on need have a separate test that does the source install to check that HEAD is fine17:53
rm_workhmm17:53
rm_workalright, well let me get that project-config change done then17:53
rm_workagain >_>17:53
*** esikachev has joined #openstack-infra17:53
*** jamesmca_ has joined #openstack-infra17:54
openstackgerritAshwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests  https://review.openstack.org/46082217:54
rm_workclarkb: wait are you sure we can't do it in the gate_hook?17:54
clarkbrm_work: PROJECTS determines what code is checked out which includes your gate_hook17:55
rm_workwe're editing devstack config and LIBS_FROM_GIT there already17:55
clarkbif you set it in gate_hook well your gate_hook won't even exist necessarily17:55
clarkboh we might do it wrong with nodepool too let me find the canonical example which is oslo17:55
rm_workhttps://github.com/openstack/octavia/blob/master/octavia/tests/contrib/gate_hook.sh#L18-L2317:55
rm_workI know those work17:55
clarkbyes because they only affect devstack17:55
rm_workand LIBS_FROM_GIT below17:55
clarkbwhat I'm talking about is setting a thing that affects devstack and devstack-gate17:56
rm_workhmm17:56
clarkbhttps://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n173 bah of course its a different var name but there it is17:56
*** atarakt has quit IRC17:57
clarkbsetting ^ translates into PROJECTS and LIBS_FROM_GIT being set appropriately17:57
rm_workok so17:57
*** e0ne has joined #openstack-infra17:57
rm_workI should set DEVSTACK_PROJECT_FROM_GIT=python-octaviaclient17:57
rm_workin the project-config job for us17:57
rm_workcan-do17:58
rm_workhow about:17:58
rm_workexport DEVSTACK_PROJECT_FROM_GIT="python-octaviaclient $DEVSTACK_PROJECT_FROM_GIT"17:58
rm_workis that safer?17:59
*** rkukura has quit IRC17:59
*** flepied has quit IRC17:59
*** atarakt has joined #openstack-infra18:00
*** atarakt has quit IRC18:01
clarkbya either way should work18:01
openstackgerritAdam Harwell proposed openstack-infra/project-config master: Use DEVSTACK_PROJECT_FROM_GIT for python-octaviaclient  https://review.openstack.org/46111618:01
rm_workclarkb: see if that is what you like ^^18:01
clarkbI don't expect anything in a calling context to set it too18:01
rm_workk18:01
*** atarakt has joined #openstack-infra18:01
clarkboh except the format is comma delimited?18:01
clarkbbeacuse I think thats how LIBS_FROM_GIT is18:02
rm_workah is it18:02
rm_worklol k18:02
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Clean up driverfixes branch jobs  https://review.openstack.org/46110218:02
*** aspiers has quit IRC18:02
rm_workwhatever, fine18:02
rm_workwon't try to re-use18:02
rm_workcurrently trying to parse what the branch override thing below it is doing18:02
*** trown is now known as trown|brb18:02
rm_workdo you think that's *to allow* for us overriding the branch, or to prevent it?18:02
*** thorst has joined #openstack-infra18:03
AJaegerfungi, clarkb, pabelanger I have two small changes for infra-manual to update links https://review.openstack.org/#/c/459842/ and https://review.openstack.org/#/c/459844 - could anybody review these, please?18:03
rm_workor tldr, do I need to add that too18:03
clarkbrm_work: branch override is that you can test eg master $client against newton cloud18:04
clarkbrm_work: its a way of constructing tests to ensure your client doesn't regress against old clouds18:04
rm_workok18:04
rm_workso18:04
rm_workwhat is THAT block doing18:04
rm_workun-doing the branch override?18:04
rm_workor actually fixing it to do it correctly18:05
*** ijw has joined #openstack-infra18:05
clarkbrm_work: can you link to the specific block?18:05
rm_workline 175-17918:05
rm_workhttps://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n17518:05
*** neiljerram has joined #openstack-infra18:06
clarkbrm_work: you need it if you are doing the thing just above it with BRANCH_OVERRIDE18:06
*** yolanda has quit IRC18:07
clarkbrm_work: so BRANCH_OVERRIDE=stable/newton will apply to cloud but then you can get master of lbaasclient18:07
rm_workwhich we do18:07
clarkbotherwise it will newton both of them18:07
rm_workok18:07
rm_workso probably we want that?18:07
clarkbya I think so18:07
rm_workoh except i don't need all of that code18:08
rm_workbecause it's a specific project :P18:08
rm_workso i don't need to detect it18:08
rm_workjust need to figure out what exactly it's doing to the formatting18:09
rm_workPYTHON_OCTAVIACLIENT18:09
rm_workcool18:09
rm_workexport OVERRIDE_PYTHON_OCTAVIACLIENT_PROJECT_BRANCH=$ZUUL_BRANCH18:10
rm_workis actually what I need then18:10
clarkbya18:10
rm_workwhat will $ZUUL_BRANCH be18:10
rm_workyou know what18:10
rm_workI assume this will work18:10
*** ijw has quit IRC18:11
rm_workok18:12
openstackgerritAdam Harwell proposed openstack-infra/project-config master: Use DEVSTACK_PROJECT_FROM_GIT for python-octaviaclient  https://review.openstack.org/46111618:12
rm_workclarkb: ^^ would you be so kind as to take a gander? :)18:12
rm_workappreciate the direction18:12
*** neiljerram has quit IRC18:13
rm_workgetting devstack+gates to play nicely always feels a bit like a carnival smoke and mirrors funhouse18:13
clarkbZUUL_BRANCH is set by zuul to indicate to the test env what branch the change under test belongs to18:13
*** BobBall has quit IRC18:14
clarkbrm_work: I think thats because we haev a lot of wheel reinventing going on18:14
clarkbso any copy pasta inherits that wheels shape18:14
rm_workyeah...18:14
clarkbside effect of pushing more and more of the work out to the laeves...18:15
*** yolanda has joined #openstack-infra18:16
rm_workalright, guess I'll wait for that to merge now18:16
*** rbrndt has joined #openstack-infra18:17
*** dprince has quit IRC18:19
*** jamesmca_ has quit IRC18:20
*** jamesmca_ has joined #openstack-infra18:21
openstackgerritDavid Shrewsbury proposed openstack-infra/shade master: Add in a bunch of TODOs about interface=admin  https://review.openstack.org/46079018:23
openstackgerritIhar Hrachyshka proposed openstack-infra/elastic-recheck master: Added query for bug 1687044  https://review.openstack.org/46112118:24
openstackbug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/168704418:24
*** david-lyle_ has joined #openstack-infra18:24
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add optional error_message to adapter.request  https://review.openstack.org/46102518:25
*** david-lyle has quit IRC18:26
rm_workok, could use +A then on this patch which should be "correct": https://review.openstack.org/#/c/461116/18:27
*** trown|brb is now known as trown18:29
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add support for specifying key-name per label  https://review.openstack.org/45546418:30
*** gordc has quit IRC18:33
*** gordc has joined #openstack-infra18:33
*** gordc has quit IRC18:34
*** gordc has joined #openstack-infra18:36
*** ihrachys_ has joined #openstack-infra18:36
*** gordc has quit IRC18:36
*** gordc has joined #openstack-infra18:36
* fungi hath returned18:37
*** ihrachys has quit IRC18:38
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Move makemergeritem into model  https://review.openstack.org/46071418:38
*** harlowja has joined #openstack-infra18:38
*** andreww has joined #openstack-infra18:39
*** funzo has quit IRC18:40
*** john-dav_ has quit IRC18:41
*** funzo has joined #openstack-infra18:41
* mordred hands fungi a fatted calf18:42
Shrews"and there was much rejoicing"18:43
*** xarses has quit IRC18:44
*** odyssey4me has quit IRC18:44
*** dpaterson_ has joined #openstack-infra18:45
openstackgerritMerged openstack-infra/infra-manual master: Update git-review link  https://review.openstack.org/45984218:45
*** odyssey4me has joined #openstack-infra18:46
*** jamesmc__ has joined #openstack-infra18:48
*** jamesmca_ has quit IRC18:48
fungiShrews wins by conjuring images of mpfc in my brain18:48
*** rbrndt has quit IRC18:49
openstackgerritMerged openstack-infra/infra-manual master: Use https in many more places  https://review.openstack.org/45984418:49
*** ijw has joined #openstack-infra18:51
*** ykarel has quit IRC18:52
pabelangerfungi: clarkb: where did we end up with maybe having AFS in vexxhost too?  The email about sahara artifacts just me thinking about it again18:57
*** _ari_ is now known as _ari_|gone18:57
clarkbI think we should continue to work through hosting there too, but I have no idea how practical it will be to host in another country if we can't do a few states over with afs18:58
clarkbI guess that is mostly determined by volume size?18:58
*** e0ne has quit IRC18:58
fungipabelanger: i think we were hoping to meet up with mnaser in boston to hash out details on moving the static.o.o volumes there, and the amout of space sahara's asking for seems somewhat reasonable for tarballs.o.o (though moving tarballs.o.o into afs also makes sense)18:59
*** rodrigopaiva has quit IRC18:59
clarkboh right talking about straight up move not dual homing18:59
mordredyah18:59
clarkb(sorry distracted by cinder lvm things)18:59
mordredit's just the WAN sync that sucked18:59
mnaseri'm free to talk at boston, perhaps we could setup a little email thread to find a good time like we did at austin :)19:00
pabelangerit would also be good to discuss what a job artifact site would look like too, for temporary things too. Today we upload them to tarballs.o.o, but don't delete them when no longer needed19:01
*** ihrachys_ is now known as ihrachys19:02
*** rodrigopaiva has joined #openstack-infra19:02
fungiwell, we mostly rely on jobs being designed in such a way that they continually replace artifacts that don't need to persist indefinitely19:02
pabelangermordred: also, did you want to try upload vhd-utils again for xenial PPA? Might start work on nodepool-builder.o.o for xenial this weekend19:02
pabelangerfungi: this would be for the deb packaging pipeline, and even docker images19:02
pabelangerbecause we need to pass untrusted to trusted workers19:03
*** Goneri has quit IRC19:03
openstackgerritMerged openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests  https://review.openstack.org/46078419:04
openstackgerritIhar Hrachyshka proposed openstack-infra/elastic-recheck master: Added query for bug 1687044  https://review.openstack.org/46112119:05
openstackbug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/168704419:05
*** david-lyle_ is now known as david-lyle19:09
fungipabelanger: specifically for deb packages and docker images which don't need to persist indefinitely?19:10
fungican't we do the same thing we do with branch tarballs in that case and just replace them?19:10
fungirename to a common filename when stashing temporarily, rename back to whatever specific filename they need when the trusted worker job operates on them?19:12
pabelangerfungi: today we upload the artifacts by zuul_commit hash, so we know which one to download on trusted node. But maybe we need to optimize that better19:12
fungiif we're integrating an ephemeral file storage into zuul then we need to integrate an ephemeral file cleanup into zuul along with it19:12
fungiif we're not integrating ephemeral file storage into zuul, then we should either design a self-cleaning storage platform or replace files on each upload19:13
fungi(noting that file-replacing is also racy if you have any concurrent generators/consumers19:15
fungi)19:15
*** xyang1 has joined #openstack-infra19:15
*** EricGonczer_ has joined #openstack-infra19:20
*** florianf has quit IRC19:20
*** rodrigopaiva has quit IRC19:22
rm_workplug for a project-config +A: https://review.openstack.org/#/c/461116/19:23
mtreinishpabelanger: is this something related to the ruby/puppet removal from the test nodes?: http://logs.openstack.org/70/460570/1/gate/gate-openstackci-beaker-centos-7/2c3fa2d/console.html#_2017-04-28_19_18_37_68708419:23
*** mhickey has joined #openstack-infra19:25
*** fandi has joined #openstack-infra19:25
cmurphymtreinish: my first guess is that a gem changed its dependencies19:25
*** EricGonczer_ has quit IRC19:27
*** erikmwilson has joined #openstack-infra19:27
dmsimard|cavernproject-config cores, still looking for easy jjb reviews on https://review.openstack.org/#/c/459913/ .. thanks19:28
fungimtreinish: ooh, fun given that we don't do any centos-based stuff in those patches19:29
mtreinishcmurphy: hmm, I can check if there was a new release recently19:30
fungiyeah, centos 7 is probably running an older ruby release than that or something19:30
mtreinishcmurphy: the last release was 0.3.0 on Feb 16, 201719:31
fungiPackage ruby-devel.x86_64 0:2.0.0.648-29.el7 will be installed19:31
cmurphyi don't think we ever had ruby 2.3 on centos 7, even in epel, so i don't think we could have accidentally taken it away19:31
fungiyeah, we're still installing ruby19:31
*** florianf has joined #openstack-infra19:32
fungisomething probably upped its dependency on that gem to a version which needs newer ruby19:32
fungimany possible layers of indirection here19:32
cmurphyhas it been broken since february and we didn't notice?19:32
fungino, these changes just passed the same jobs in check19:33
fungiwithin the past 24 hours19:33
openstackgerritMerged openstack-infra/elastic-recheck master: Added query for bug 1687044  https://review.openstack.org/46112119:33
openstackbug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/168704419:33
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in router commands  https://review.openstack.org/46051819:34
mtreinish:( all my firehose changes are failing19:34
*** rbrndt has joined #openstack-infra19:36
*** adrian_otto has quit IRC19:37
cmurphysomething just added xmlrpc as a dependency19:37
cmurphyhttp://logs.openstack.org/70/460570/1/check/gate-openstackci-beaker-centos-7/9fc5806/console.html doesn't install it19:37
mtreinishthis is where what-broke.py but for gems whould be useful :)19:38
cmurphynone of the gems listed in http://logs.openstack.org/70/460570/1/gate/gate-openstackci-beaker-centos-7/2c3fa2d/console.html have different versions so it's one of the gems that would have been installed after that broken point19:38
clarkbI love how its really what released but essentially ehat broke19:39
*** ijw has quit IRC19:39
cmurphyeverything is terrible19:39
mtreinishheh19:39
fungiproof that the python package ecosystem is not the only one with these sorts of problems ;)19:40
fungiit's immediately after the message which says19:41
fungi"Installing docker-api 1.33.4"19:41
fungihrm, release date on that is april 1019:41
fungiso another dead end19:41
*** ijw has joined #openstack-infra19:41
*** ijw has quit IRC19:47
cmurphyfound it https://github.com/fog/fog-core/commit/ffcce1066602b41b603675b751650d427bfec57619:49
*** flepied has joined #openstack-infra19:49
cmurphy3 hours ago19:49
* fungi sounds the fog horn19:49
*** thorst has quit IRC19:49
*** thorst has joined #openstack-infra19:50
fungiwhat did you do to track that down? i was going through the add_dependency lines in puppet-openstack_infra_spec_helper.gemspec, digging down through their runtime dep crosslinks on the rubygems site and looking for any with xmlrpc19:51
fungihoping to hear there's a "faster" way ;)19:51
*** thorst_ has joined #openstack-infra19:51
*** annegentle has joined #openstack-infra19:51
openstackgerritMerged openstack-infra/shade master: Add in a bunch of TODOs about interface=admin  https://review.openstack.org/46079019:52
cmurphyfungi: i have ruby2.3 on tumbleweed so i just ran bundle install and compared the list to the last successful beaker run19:52
fungiahh19:52
fungithat makes much more sense ;)19:52
cmurphyit doesn't entirely make sense though, it shouldn't have triggered that dependency https://github.com/fog/fog-core/blob/master/fog-core.gemspec#L2419:53
fungicmurphy: am i reading this right? https://github.com/fog/fog-core/blob/v1.44.0/fog-core.gemspec#L2419:53
*** tphummel has quit IRC19:53
*** sree has joined #openstack-infra19:53
fungilooks like it would only add xmlrpc if the installed ruby interpreter is 2.4 or later?19:53
cmurphyyeah that is confusing19:53
fungilooks like we both ended up there at the same time19:54
*** thorst has quit IRC19:54
cmurphyxmlrpc installed for me using ruby 2.319:54
cmurphyso i think maybe spec.add_dependency doesn't work like they think it does19:55
fungialso that line definitely appeared between 1.43.0 and 1.44.019:55
fungiso it's pretty certain to be the trigger19:55
*** thorst_ has quit IRC19:56
fungido we need to add a skip for fog!=1.44.0 in puppet-openstack_infra_spec_helper?19:56
*** sree has quit IRC19:57
fungicmurphy: https://github.com/fog/fog-core/issues/206 seems to confirm19:58
fungisafety in numbers!19:59
cmurphyexcellent19:59
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstack_infra_spec_helper master: Pin fog-core  https://review.openstack.org/46114119:59
pabelangerand back20:00
pabelangerhad to run out for beer errand20:00
fungipabelanger: just in time to help us polish rubies20:00
fungicmurphy: i take it < is preferable to != in this scenario (or do specfiles even have skips?)20:01
rcarrillocruzdmsimard|cavern: heya, merged https://github.com/ansible/ansible/pull/2412220:01
rcarrillocruzpabelanger: has the infra dinner been settled yet?20:01
cmurphyfungi: hmm i just copied the one above it, i'm not entirely sure20:01
rcarrillocruzi can do any day except thursday, flying back that day20:01
* rcarrillocruz should probably just respond to that email20:02
pabelangerrcarrillocruz: not yet, but Monday / Wednesday seems to be the front runners20:02
pabelangerI too leave on thursday20:02
rcarrillocruzweeeee20:02
*** Apoorva has quit IRC20:02
fungihah, three reviewers just independently suggested linking the github issue on that change ;)20:02
rcarrillocruzi'm in then20:02
*** jcoufal_ has joined #openstack-infra20:03
cmurphyfungi: looks like != is possible, that's probably better https://www.devalot.com/articles/2012/04/gem-versions.html20:03
pabelangerrcarrillocruz: Yay, going to search this weekend for eats and reply back to thread20:03
rcarrillocruz\o/ thx sie20:03
rcarrillocruzsir too20:03
fungicmurphy: our tests should tell. and given that the intent in their code seems to have been counter to this behavior and there's already an open issue odds are 1.44.1 won't have this problem20:04
mtreinishrcarrillocruz: ooh, you're the perfect person to review my ansible firehose patch: https://review.openstack.org/#/c/458248/ :)20:04
* rcarrillocruz flees20:04
rcarrillocruz:P20:04
mtreinishrcarrillocruz: I have the plugin there also submitted as a pr on ansible proper, but I need to do some work to fix dmsimard|cavern's comments on that PR20:04
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstack_infra_spec_helper master: Pin fog-core  https://review.openstack.org/46114120:04
cmurphyfungi: agree ^20:05
* rcarrillocruz totally forgot all puppet he once knew20:05
fungicmurphy: thanks!20:05
mtreinishrcarrillocruz: heh, you can concentrate on the ansible/python bits :)20:05
*** jcoufal has quit IRC20:05
fungircarrillocruz: i'll happily approve take responsibility for reviewing the puppet bits on tat one, it's just the ansible-isms i wasn't so sure about20:06
* fungi is typing sideways apparently. that made next to no sense20:06
mtreinishfungi: to be fair like nothing in the callback plugin api was documented. I managed to get that working by copy and paste/trial and error20:06
*** thorst has joined #openstack-infra20:07
dmsimard|cavernmtreinish: ++20:08
dmsimard|caverntell me about reverse engineering ansible internals20:08
*** beslemon has joined #openstack-infra20:09
*** jamesmc__ has quit IRC20:09
*** Apoorva has joined #openstack-infra20:09
*** jamesmca_ has joined #openstack-infra20:10
*** dave-mccowan has quit IRC20:10
fungicmurphy: latest patchset just passed the previously broken job, approving!20:12
rcarrillocruzyah20:12
cmurphywoohoo \o/20:12
rcarrillocruzactually i think the docs say 'hey, check it out other plugins to see how it works to write your own'20:12
rcarrillocruz:D20:12
mtreinishrcarrillocruz: heh, yeah I think you're right20:12
mtreinishthe fun thing was none of the packaged plugins did anything the same way20:13
rcarrillocruzi don't spot anything bad, so +220:13
fungimtreinish: isn't that usually what happens with undocumented interfaces?20:13
mtreinishyep20:13
* rcarrillocruz goes open an amber beer now20:13
mtreinishrcarrillocruz: cool, thanks20:13
fungircarrillocruz: thanks, and happy beer!20:13
rcarrillocruz++20:13
fungican't wait to hang out some in boston20:14
rcarrillocruzindeed20:14
rcarrillocruzsucks i missed PTG :/20:14
mtreinishrcarrillocruz: oh, you're gonna be in boston, cool.20:14
rcarrillocruzyah20:14
rcarrillocruzthx to robyn :-)20:14
rcarrillocruzi'll give a talk on ansible day20:14
fungishould convince him to attend the firehose talk ;)20:14
rcarrillocruztalk i have not even started20:15
rcarrillocruzweeee20:15
mtreinishrcarrillocruz: heh, I've got 2 talks like that :)20:15
fungisame here (2 or 3)20:15
rcarrillocruzo robyn is here20:15
mtreinishrcarrillocruz: you just helped with one of them20:15
rcarrillocruzrbergeron: ^20:15
*** esikachev has quit IRC20:16
fungiall good talks are written on the plane flight to the conference, right? ;)20:16
mtreinishfungi: btw, I think you'll appreciate this: https://github.com/mtreinish/handbrakecloud I wrote that for my budget cloud talk (to show what you could do with a cloud in your closet)20:16
mtreinishbut after I finished it I realized I just wrote a really crappy nodepool20:16
clarkbfungi: I'm sort of working on mine.20:16
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper master: Pin fog-core  https://review.openstack.org/46114120:17
fungimtreinish: hah! fun20:17
*** morgabra has left #openstack-infra20:17
*** florianf has quit IRC20:20
*** esikachev has joined #openstack-infra20:20
*** jcoufal has joined #openstack-infra20:21
*** jcoufal_ has quit IRC20:23
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add optional error_message to adapter.request  https://review.openstack.org/46102520:26
*** florianf has joined #openstack-infra20:26
*** EricGonczer_ has joined #openstack-infra20:28
*** jamesmc__ has joined #openstack-infra20:28
*** jamesmca_ has quit IRC20:29
openstackgerritMerged openstack-infra/puppet-logstash master: Remove logstash mqtt plugin install  https://review.openstack.org/46072920:31
*** esikachev has quit IRC20:31
openstackgerritMerged openstack-infra/puppet-subunit2sql master: Move paho-mqtt package install to worker node  https://review.openstack.org/46057020:32
*** esikachev has joined #openstack-infra20:34
openstackgerritMerged openstack-infra/system-config master: Enable ansible callback events on firehose  https://review.openstack.org/45824820:34
fungimtreinish: i'm going to start paying attention to our ansible periodic run just to make sure we haven't broken it with that ^20:35
*** rlandy is now known as rlandy|brb20:36
*** esikachev has quit IRC20:38
mtreinishfungi: ok20:38
mtreinishfungi: I'm about to drive home, so hopefully nothing will break in the next 10-15min :)20:39
mtreinishbut I assume something will break eventually20:39
fungimtreinish: if it does, it's not a super-urgent issue as long as we're paying attention20:39
mtreinishit wouldn't feel like an infra patch from me if it didn't20:39
fungiwe all break things all the time20:40
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Remove neutronclient mocks from ports tests  https://review.openstack.org/46114920:41
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Adding CI jobs to monasca-kibana-plugin  https://review.openstack.org/45951120:42
*** ijw has joined #openstack-infra20:43
*** gordc has quit IRC20:43
fungifirst puppet run since 458248 merged is about to kick off in 60 seconds. i'm tailing the log20:44
openstackgerritMerged openstack-infra/storyboard-webclient master: Rewrite the query string when removing browse criteria  https://review.openstack.org/45502420:47
*** asselin has joined #openstack-infra20:48
*** jcoufal_ has joined #openstack-infra20:48
*** vhosakot has joined #openstack-infra20:48
fungicmurphy: aha, some enlightenment has appeared in that github issue... apparently the conditional is evaluated at build time not install time, and the gem was built against newer ruby so baked in the xmlrpc dep unconditionally20:49
*** asselin__ has quit IRC20:50
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add some more debugging to the post_test_hook  https://review.openstack.org/46077920:50
*** jcoufal has quit IRC20:50
cmurphythat is fantastic20:50
*** ijw has quit IRC20:51
fungicertainly one of the better words for it20:51
*** mhickey has quit IRC20:51
*** eharney has quit IRC20:52
clarkbthats like when lxml wheel broke20:53
fungipretty much exactly, yep20:54
*** florianf has quit IRC20:54
fungisdists build on install, wheels are prebuilt. shifts assumptions about whether you can test for certain conditions at "installation"20:55
*** tphummel has joined #openstack-infra20:55
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Remove neutronclient mocks from quotas tests  https://review.openstack.org/46115020:59
*** Apoorva_ has joined #openstack-infra21:01
*** Sukhdev has quit IRC21:03
openstackgerritMerged openstack-infra/puppet-mqtt_statsd master: Update subscribed mosquitto metrics  https://review.openstack.org/45895321:03
*** jcoufal has joined #openstack-infra21:03
*** Apoorva has quit IRC21:04
mtreinishfungi: did anything eventful happen in the puppet run?21:05
openstackgerritMerged openstack-infra/puppet-subunit2sql master: Add source_url to mqtt payload  https://review.openstack.org/46031321:05
*** jamesmc__ has quit IRC21:05
openstackgerritMerged openstack-infra/system-config master: Add subunit gearman worker mqtt info to firehose docs  https://review.openstack.org/46032321:05
*** jcoufal_ has quit IRC21:05
fungimtreinish: it's still underway21:05
fungi(and i'm still watching)21:05
fungiwe're only 20 minutes in and it can take >3021:06
*** dimtruck is now known as zz_dimtruck21:06
fungi(started at 20:45 utc)21:06
*** jamesmca_ has joined #openstack-infra21:06
mtreinishok, I just opened an mosquitto_sub on the ansible/# topic21:06
mtreinishhopefully something will pop up there21:06
*** dpaterson_ has quit IRC21:07
mtreinishit's supposed to publish something when a playbook starts, stops, and on each task finishing21:07
*** beslemon has quit IRC21:07
fungigood news is it doesn't look like we broke puppeting our servers21:07
mtreinishsmall victories :)21:07
fungibut there are tasks being run since you started gulping from the firehose, so if you're not seeing anything then we're likely still missing something21:08
*** thorst_ has joined #openstack-infra21:08
fungiokay, i think the full pass just completed. next will start at 21:15z21:08
mtreinishyeah, I'm not seeing anything on the other end of the firehose21:08
*** ijw has joined #openstack-infra21:09
fungimtreinish: oh! it's possible we needed to wait through one pass21:09
*** thorst has quit IRC21:09
mtreinishfungi: heh, yeah I was just thinking that21:09
mtreinishbecause we rely on puppet to put the plugin in the repo21:09
fungisince the puppetmaster updates itself, the pass which updated it may have started before it put its files in place, yeah21:10
*** trown is now known as trown|outtypewww21:10
fungii'm going to double-check stuff is there now21:10
*** ijw has quit IRC21:10
*** ijw has joined #openstack-infra21:10
fungiit has /etc/mqtt_ca_cert.pem.crt and /etc/mqtt_client.yaml with what look like correct contents21:11
fungiand /opt/ansible/lib/ansible/plugins/callback/mqtt.py too21:11
mtreinishfungi: cool, that's a good sign21:11
fungiso we should be all set for when the next pass begins in ~3 minutes21:12
* mtreinish sits on the edge of his seat21:12
fungimy standing workstation makes that harder than it should be21:12
mtreinishlol21:12
*** thorst_ has quit IRC21:12
fungijust realized the only sitting furniture in my lab are a kneeling chair and a barstool21:13
*** Goneri has joined #openstack-infra21:13
*** rlandy|brb is now known as rlandy21:13
*** ldnunes has quit IRC21:14
fungiand here it goes!21:15
mtreinishnot seeing anything so far21:15
fungii guess if i open up a packet sniffer looking for packets destined for the firehose, i should see something assuming it's publishing21:16
*** esberglu has quit IRC21:17
*** harlowja has quit IRC21:17
*** harlowja has joined #openstack-infra21:17
mtreinishyeah that would work. Wireshark has mqtt filters too21:17
mtreinishalthough we should be encrypting the packets, so that's not really useful :)21:17
mtreinishI'm still not seeing anything on my end21:18
fungiwell, it at least tells me whether any communication is being attempted at all21:18
mtreinishyeah21:18
fungiand so far tcpdump is showing no packets from puppetmaster to the ipv4 nor ipv6 addresses of firehose0121:18
fungiis ansible supposed to automatically pick up any plugins we drop into /opt/ansible/lib/ansible/plugins/callback/ or do they need to be registered with it somehow?21:19
mtreinishfungi: they should as long as 'CALLBACK_NEEDS_WHITELIST = True' isn't set in the module21:20
*** ijw has quit IRC21:20
mtreinishbut one sec I can pull up the config docs on it21:20
*** dmsimard|cavern is now known as dmsimard21:20
mtreinishfungi: these 3 options: http://docs.ansible.com/ansible/intro_configuration.html#bin-ansible-callbacks21:21
mtreinishbut it should set lib/ansible/plugins/callback to be the default path21:21
mtreinishotherwise it wouldn't pick up the preinstalled ones21:21
fungithere are definitely a ton of other *.py files in that directory21:22
mtreinishfungi: a simple way to test it would be to comment out the mqtt_hostname field in the config yaml and run a noop playbook locally21:24
mtreinishit should print a warning saying it can't find the hostname variable and disable itself21:24
mtreinishfungi: oh wait is paho-mqtt installed?21:25
*** flepied has quit IRC21:26
fungiwould be a pypi package, right?21:26
mtreinishI was just looking at the patch and I don't think I put that in the puppetmaster.pp21:26
mtreinishyeah21:26
funginot in the global environment at least21:26
mtreinishalthough if it is missing it should warn that it's not installed at the start of the play21:26
mtreinishdo we run ansible from a venv?21:26
*** askb has joined #openstack-infra21:27
fungithat's what i'm checking now21:27
* mtreinish isn't super familiar with the puppetmaster setup21:27
fungii don't believe we do21:27
*** srobert has quit IRC21:27
mtreinishheh, well I'll push a patch to add paho to the node21:27
*** cody-somerville has joined #openstack-infra21:27
*** cody-somerville has joined #openstack-infra21:27
mtreinishbut I'm surprised we're not seeing the warning21:27
*** thorst has joined #openstack-infra21:29
openstackgerritMatthew Treinish proposed openstack-infra/system-config master: Add paho-mqtt install if we turn mqtt on  https://review.openstack.org/46115621:29
*** csomerville has quit IRC21:29
mtreinishfungi: ^^^21:29
fungihttp://git.openstack.org/cgit/openstack-infra/puppet-ansible/tree/manifests/init.pp seems to globally pip install ansible (from a local source tree in our case)21:29
mtreinishok cool, that makes it easier to get right21:30
fungia little leery about pip installing things on puppetmaster.o.o, but we're doing that now for shade, ansible and openstackclient21:33
*** thorst has quit IRC21:33
*** ijw has joined #openstack-infra21:34
*** asselin has quit IRC21:34
*** ijw has quit IRC21:34
*** ijw has joined #openstack-infra21:34
*** zz_dimtruck is now known as dimtruck21:35
mtreinishfungi: if it helps paho has 0 deps: https://github.com/eclipse/paho.mqtt.python/blob/v1.2.3/setup.py#L1321:35
openstackgerritMonty Taylor proposed openstack/os-client-config master: Fix interactions with keystoneauth from newton  https://review.openstack.org/46111421:35
* mtreinish didn't realize that until just now21:35
fungimaybe we'll interest someone in getting paho-mqtt into debian/ubuntu though21:35
*** harlowja has quit IRC21:36
mtreinishfungi: it might be in there already21:36
funginot under any name including the string "mqtt" at least21:36
fungihttp://packages.debian.org/mqtt http://packages.ubuntu.com/mqtt21:36
*** Apoorva_ has quit IRC21:36
mtreinisherr yeah the only hit on ubuntu I see is: http://packages.ubuntu.com/yakkety/golang-eclipse-paho-dev21:37
*** mriedem has quit IRC21:37
*** Apoorva has joined #openstack-infra21:37
mtreinishfungi: ubuntu does package the old mosquitto python package: http://packages.ubuntu.com/precise-updates/python-mosquitto21:38
mtreinishbut that was replaced by paho21:38
mtreinishhttps://mosquitto.org/documentation/python/21:38
*** chlong has quit IRC21:39
fungiyeah, in debian up through jessie21:39
fungidropped for stretch21:39
mtreinishfungi: I've never done debian packaging before, but if I find the time I'd be willing to push that (unless I can find someone else to do it, which would probably be easier)21:39
mtreinishhow much harder can it be than maintaing arch packages :P21:40
fungithe debian python modules team (dpmt) has some fairly straightforward packaging guidelines and is usually willing to team-maintain stuff for you. also there's a tool for generating debs from sdists as a starting point21:40
*** lin_yang has joined #openstack-infra21:41
clarkbmtreinish: arch is much much much easier imo21:41
clarkbdebian has rules21:42
fungiheh21:42
fungi(it even has a file _called_ "rules" in each package)21:42
fungi((though it's really just a makefile))21:43
fungii'll consider trying to package it for debian once the stretch release freeze is over21:43
mtreinishclarkb: lol, yeah arch definitely doesn't. Especially for the AUR where all of my packages live21:43
fungiubuntu will generally just suck packages like that straight in from debian/testing anyway21:44
mtreinishfungi: so was there no warning at the top of the playbook output?21:45
jeblairfungi: huh, my irc client cut you off mid-sentence there.  weird.21:45
mtreinishfungi: because I wonder if we've got 2 issues, it's not being loaded and paho was missing21:46
fungimtreinish: i'm looking for that tiny needle in this haystack of output now21:46
*** ijw has quit IRC21:47
*** mdrabe has quit IRC21:47
mtreinishfungi: I'll spin up my test vm to get an example21:47
*** ijw has joined #openstack-infra21:48
fungimtreinish: not spotting it in the log (which i think is just redirected stdout from our wrapper script)21:48
*** ijw has quit IRC21:48
*** ijw has joined #openstack-infra21:48
fungino matches for "warn" (case-insensitive)21:49
*** baoli has quit IRC21:49
v1k0d3nhello everyone21:50
v1k0d3nis there anyone who can help with this one? https://review.openstack.org/#/c/460925/21:50
*** jcoufal_ has joined #openstack-infra21:50
v1k0d3ntrying to get docs sorted before the summit21:50
*** jcoufal has quit IRC21:50
mtreinishfungi: it goes to stderr21:51
mtreinishI just tested it locally21:51
mtreinishfungi: http://paste.openstack.org/show/608371/21:51
openstackgerritClark Boylan proposed openstack-infra/elastic-recheck master: Extend query for bug 1687044  https://review.openstack.org/46116621:52
openstackbug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/168704421:52
mtreinishI also just copied the plugin from system-config into my local ansible checkout which seemed to work without configuration21:52
*** deafult has quit IRC21:52
fungiokay21:52
fungiour cronjob does indeed >> logfile 2>&1 so we should be getting stderr in there too21:54
*** felipemonteiro_ has quit IRC21:54
mtreinishfungi: hmm, then I wonder if it's really not getting loaded21:55
*** esikachev has joined #openstack-infra21:56
mtreinishfungi: if your up for manually testing it, I was using this as my test playbook: http://paste.openstack.org/show/608372/21:56
openstackgerritClark Boylan proposed openstack-infra/elastic-recheck master: Extend query for bug 1687044  https://review.openstack.org/46116621:57
openstackbug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/168704421:57
fungimtreinish: i wonder if ansible is only loading plugins from /usr/local/lib/python2.7/dist-packages/ansible/plugins and not also from /opt/ansible/lib/ansible/plugins?21:57
*** rlandy has quit IRC21:57
mtreinishfungi: how do we install it?21:57
mtreinishfungi: I did pip install -e locally nad it seemed to work21:57
openstackgerritMerged openstack-infra/tripleo-ci master: Redirect stack-show output in tripleo.sh  https://review.openstack.org/45925721:57
fungiwhatever the openstack_pip provider does, which is _probably_ not editable?21:58
mtreinishok, that might do it21:58
fungibecause the file we've added under the source tree isn't showing up under the installed location for the package21:58
mtreinishalthough if we pip install again it should move it21:58
mtreinishsince it's in the package21:59
mtreinishah, that would do it21:59
mtreinishdo you want me to update the path in the puppet?21:59
fungii think we only trigger pip install when the source git ref changes21:59
*** cmurphy has quit IRC21:59
fungii'm wondering whether ansible provides an additional default path for local plugin additions (or an envvar you're supposed to set for a plugin search path or something)21:59
mtreinishfungi: you can specify a path in the config file (which I think also gets exposed as a env var)22:00
jeblairfungi: you can set the plugin path in the config file22:00
jeblairya that22:00
fungijust got there in the ansible docs myselg22:00
fungimyself22:00
jeblairi don't know if it has automatic entry points stuff?  (i don't think so)22:00
mtreinishfungi: http://docs.ansible.com/ansible/intro_configuration.html#callback-plugins22:00
mtreinishjeblair: there aren't any entrypoints that I could see when I was looking at the code before22:01
fungiyeah, that's the section i was looking at22:01
mtreinishjust the search paths22:01
jeblairmtreinish: so do you have an ansible.cfg with your source or install directory for it?22:01
jeblairmtreinish: so do you have an ansible.cfg with your source or install directory added to the callback plugin path?22:01
jeblair(sorry about first nonsense question :)22:01
mtreinishjeblair: when I ran it I just did git clone ansible && pip install -e ansible && cp mqtt.py ansible/lib/ansible/plugin/callbacks/.22:02
mtreinishI didn't even bother creating an ansible.cfg22:02
jeblairah yeah, so that's why it found it locally22:02
fungithere's also "You can activate a custom callback by either dropping it into a callback_plugins directory adjacent to your play or inside a role or by putting it in one of the callback directory sources configured in ansible.cfg." http://docs.ansible.com/ansible/dev_guide/developing_plugins.html#configuring-callback-plugins22:02
mtreinishyep22:02
*** esikachev has quit IRC22:02
mtreinishjeblair: I wrongly assumed it was also a editable install when I wrote the puppet to copy it into the source tree22:03
*** jcoufal has joined #openstack-infra22:03
jeblairso we could put this in system-config, adjacent to the playbook.  but if we want to maintain it as a separate git repo (yes, i rather think), then we may just want to install it like we install puppet modules.  'git clone /opt/....', then add that to ansible.cfg.22:04
*** cmurphy has joined #openstack-infra22:04
jeblairoh there's puppet to copy it somewhere?22:04
jeblairmtreinish: link to that ^?22:04
mtreinishjeblair: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/puppetmaster.pp#n36622:05
*** Apoorva has quit IRC22:05
mtreinishoh sry L383 more specifically22:05
*** jcoufal_ has quit IRC22:05
fungiyeah, i approved that without noticing it was installing it into the git source tree on disk rather than somewhere ansible knew to look22:05
*** Apoorva has joined #openstack-infra22:05
mtreinishjeblair: my longer term plan with this was to put it in ansible proper (I have an open PR for it now)22:05
jeblairah.  sorry, some of my earlier statements were under the misapprehension the mqtt plugin was a python package, but you meant 'pip install ansible'22:06
fungijust a bare plugin (not packaged)22:07
mtreinishyeah22:07
jeblairthat actually makes doing system-config/playbooks/callback_plugins/mqtt.py a little more feasible.22:07
openstackgerritMerged openstack-infra/elastic-recheck master: Extend query for bug 1687044  https://review.openstack.org/46116622:08
openstackbug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/168704422:08
fungido the playbooks in system-config only ever run on the puppetmaster?22:08
jeblair(though, there may still be reasons to install it to neutral location and add it to ansible.cfg -- like if we think there are circumstances where people would be surprised that plugin is getting pulled in)22:08
jeblairfungi: ^ i think you may be getting at what i was saying there22:08
fungiyep22:08
mtreinishfungi: does it matter? woudln't that just more events for firehose :p22:09
mtreinishwow, I really can't write today...22:09
jeblairmtreinish: i probably don't have the right creds on my desktop.  at least i hope i don't.22:09
fungimtreinish: more likely it would be more warning messages from not having the dependency and config present22:09
jeblairfungi: i *think* so.  but i kind of lean toward explicit ansible.cfg so we don't have to worry about it in the future.22:09
fungithat seems fine22:10
jeblair(i like levers for this sort of thing too)22:10
fungishould we stick it somewhere in /etc? /opt? /usr/local? bikeshed time22:10
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/gear master: Provide TextJob and TextWorker for convenience  https://review.openstack.org/46117222:11
*** thorst has joined #openstack-infra22:11
jeblairfungi: imma gonna let you pait this one22:11
fungii don't know where the "more correct" location in our filesystem is since i'm not aware of any good prior art22:11
fungibut happy to just pick one22:11
jeblairwow.  i typed 'pain' at first and made it halfway to 'paint'.22:11
mtreinishdo we write out the ansible.cfg somewhere? or is that a manual step22:11
*** jascott1 has quit IRC22:12
fungiwe have an /etc/ansible/ansible.cfg22:12
mtreinishfungi: I would probably just drop it in /opt/callback_plugins22:12
fungi_probably_ put there by openstack-infra/puppet-ansible22:12
*** thorst has quit IRC22:12
jeblairlooks like it22:13
fungiwe have roles_path=/etc/ansible/roles in it, so maybe we want /etc/ansible/plugins?22:13
mtreinishthat works too22:13
mtreinishfungi: yep: https://github.com/openstack-infra/puppet-ansible/blob/master/templates/ansible.cfg.erb22:13
*** jascott1 has joined #openstack-infra22:14
*** gouthamr has quit IRC22:14
fungii need to take a break from this soon to go do some yardwork before i lose the daylight, but can pick it back up after22:14
*** jamesmca_ has quit IRC22:15
jeblairmtreinish: if you want to write changes, i'm around for a little longer and can review22:15
fungihappy to babysit it some more this evening and see if it's any closer22:15
mtreinishok, I'm working on writing up the puppet-ansible patch now22:16
*** jamesmca_ has joined #openstack-infra22:16
fungiawesome22:16
mtreinishand I'll push up the system-config depends on after that22:16
*** xyang1 has quit IRC22:18
*** EricGonc_ has joined #openstack-infra22:19
openstackgerritMatthew Treinish proposed openstack-infra/puppet-ansible master: Set an ansible callback directory  https://review.openstack.org/46117422:19
*** asselin has joined #openstack-infra22:19
mtreinishfungi, jeblair: ^^^22:19
*** jamesmca_ has quit IRC22:20
*** EricGonczer_ has quit IRC22:20
openstackgerritMatthew Treinish proposed openstack-infra/system-config master: Copy ansible mqtt callback plugin into the plugins dir  https://review.openstack.org/46117522:22
mtreinishfungi, jeblair: ^^^22:22
*** jcoufal has quit IRC22:22
*** EricGonc_ has quit IRC22:24
jeblairmtreinish: +3 on first; tentative -1 on second22:26
* jeblair imagines fungi mowing lawn with HUD22:26
*** vhosakot has quit IRC22:27
jeblairwhat could possibly go wrong22:27
fungiweapons of yard destruction!22:27
* fungi hasn't actually gone out to do yardwork yet22:27
mtreinisherr, I have lint on the first change22:28
*** ijw has quit IRC22:28
fungiapply ample quantities of lint remover22:28
*** adrian_otto has joined #openstack-infra22:29
jeblairwow, it sure looked like it had enough whitespace22:30
openstackgerritMatthew Treinish proposed openstack-infra/puppet-ansible master: Set an ansible callback directory  https://review.openstack.org/46117422:30
mtreinishjeblair: off by 122:30
*** jkilpatr has quit IRC22:30
fungiyou anger the whitespace gods with your inadequate use of the spacebar22:30
mtreinishjeblair: for the system-config patch can I have a requires on the file if it's inside a different module?22:31
mtreinishjeblair: I wasn't sure about that which is why I left it out22:31
jeblairmtreinish: i believe so22:31
mtreinishok, I'll add that then22:31
mtreinishI always screw up puppet-scoping so I'm never sure22:31
fungishould be able to, and then it'll wait to process that block until the other module has created the dependency22:32
fungiit merges them all into one ginormous manifest and then generates the graph22:32
*** Apoorva_ has joined #openstack-infra22:32
fungias i understand it22:32
fungiwhich is also why we have so many silly-looking conditional guards around resources to prevent duplicate definitions22:33
openstackgerritMatthew Treinish proposed openstack-infra/system-config master: Copy ansible mqtt callback plugin into the plugins dir  https://review.openstack.org/46117522:34
mtreinishfungi: ^^ I also added the absent to save you the rm22:34
fungithanks22:34
*** esikachev has joined #openstack-infra22:35
*** Apoorva has quit IRC22:36
jeblair+622:36
fungithe boxcar22:37
fungiokay, i'm going to go do battle with the lawn while those churn through22:37
fungiback soon22:37
mtreinishfungi: enjoy22:37
clarkbtomorrow is my yard battle day22:39
*** esikachev has quit IRC22:39
*** thorst has joined #openstack-infra22:43
openstackgerritMerged openstack-infra/puppet-ansible master: Set an ansible callback directory  https://review.openstack.org/46117422:43
*** baoli has joined #openstack-infra22:45
*** baoli has quit IRC22:45
*** wolverineav has quit IRC22:46
*** adrian_otto has quit IRC22:46
*** baoli has joined #openstack-infra22:46
*** esikachev has joined #openstack-infra22:47
*** adrian_otto has joined #openstack-infra22:48
openstackgerritMerged openstack-infra/system-config master: Copy ansible mqtt callback plugin into the plugins dir  https://review.openstack.org/46117522:51
*** ijw has joined #openstack-infra22:51
*** esikachev has quit IRC22:51
*** Sukhdev has joined #openstack-infra22:53
mtreinishjeblair: oh there is one more patch we'll need before this can work: https://review.openstack.org/#/c/461156/22:55
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger  https://review.openstack.org/46117622:55
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor  https://review.openstack.org/46117722:55
mtreinishotherwise we'll be running the plugin without the mqtt libs installed22:55
*** annegentle has quit IRC22:55
*** ijw has quit IRC22:56
*** asselin_ has joined #openstack-infra22:58
jeblairmtreinish: +322:58
mtreinishthanks22:59
jeblairmtreinish: i'm afk for a bit now, will check in again soon22:59
*** asselin__ has joined #openstack-infra22:59
mtreinishok22:59
*** thorst has quit IRC22:59
*** lamt has quit IRC23:01
*** asselin has quit IRC23:01
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger  https://review.openstack.org/46117623:02
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor  https://review.openstack.org/46117723:02
*** lamt has joined #openstack-infra23:02
*** rbrndt has quit IRC23:02
*** rhallisey has quit IRC23:02
openstackgerritMorgan Fainberg proposed openstack-infra/shade master: Convert test_role_assignments to requests mock  https://review.openstack.org/44831323:03
*** asselin_ has quit IRC23:03
openstackgerritMerged openstack-infra/system-config master: Add paho-mqtt install if we turn mqtt on  https://review.openstack.org/46115623:04
*** xarses_ has quit IRC23:05
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/gear master: Provide TextJob and TextWorker for convenience  https://review.openstack.org/46117223:05
*** lamt has quit IRC23:08
*** ijw has joined #openstack-infra23:09
*** adrian_otto has quit IRC23:09
*** esikachev has joined #openstack-infra23:10
*** asselin has joined #openstack-infra23:10
*** adrian_otto has joined #openstack-infra23:11
*** asselin__ has quit IRC23:12
*** ijw has quit IRC23:13
*** asselin_ has joined #openstack-infra23:13
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add some more debugging to the post_test_hook  https://review.openstack.org/46077923:15
*** adrian_otto has quit IRC23:15
fungiclarkb: tomorrow is _also_ my yard battle day ;)23:15
fungijust trying to not fall behind23:16
*** esikachev has quit IRC23:17
*** asselin has quit IRC23:17
asselin_anyone know where the docs are to connect a new gerrit instance to authenticate with launchpad? how do you get the access token/secret/consumer key? https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n7423:19
clarkbasselin_: I think thats for the hook scripts not for authentication with openid23:21
clarkber not for gerrit's openid auth23:21
asselin_oh...so maybe nothing needs to be done? just use the default setting?23:21
mordredfungi: every day is battle day23:21
clarkbif you don't need launchpad bugs to be updated by the hook scripts thats fine23:21
fungimtreinish: we have paho-mqtt (1.2.3) on puppetmaster.o.o now, and callback_plugins=/etc/ansible/callback_plugins in /etc/ansible/ansible.cfg and the plugin is present in that dir and already has a corresponding mqtt.pyc file indicating ansible loaded it roughly 2 minutes after it was installed (so likely took effect partway into the run)23:21
clarkbotherwise you have to do the launchpadlib thing23:22
clarkbwhich is documented in launchpadlib iirc23:22
asselin_clarkb, no...just looking for openid for users to create accounts in gerrit....maybe I don't need to do anything. Will give it a try. thanks!23:22
fungiasselin_: right, not necessary for that case23:23
fungiasselin_: just make sure to set the openid configuration in the gerrit.conf and it should just work23:23
mtreinishfungi: ok cool, I've got my mosquitto_sub still open in a window23:23
mtreinishhopefully it'll start emitting the next go around23:23
asselin_fungi, yeah...seems that's already in puppet...so the defaults might 'just work'23:23
*** esberglu has joined #openstack-infra23:24
fungiasselin_: specifically the lp auth stuff is for having gerrit be able to comment on and reassign lp bugs and blueprints23:24
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks  https://review.openstack.org/43983423:24
fungias clarkb mentioned23:24
asselin_ack...no need for that use case23:24
fungimtreinish: it's already trying to use it, and i see warnings in the logs23:25
fungimtreinish: i'll see if i can collect them all into a paste trimming out all the noise23:26
mtreinishfungi: ooh, ok23:26
*** esikachev has joined #openstack-infra23:27
*** esberglu has quit IRC23:28
fungimtreinish: these are the unique errors i found http://paste.openstack.org/show/60837323:29
fungilooks like they're probably all the same problem though?23:29
*** sree has joined #openstack-infra23:29
mtreinishfungi: yeah I think I missed a string cast somewhere23:29
mtreinishalthough why I didn't hit that locally I can't say23:29
fungiseems likely23:29
fungipython version?23:30
mtreinishfungi: if we ran with -vvv that should print the tracebacks23:30
fungiPython 2.7.6 on puppetmaster.o.o23:30
mtreinishI was running on ubuntu with 2.7 so I would expect the same23:30
fungiyeah23:30
mtreinish2.7.1223:30
*** harlowja has joined #openstack-infra23:30
fungihow were you invoking your hello world?23:31
mtreinishansible-play -vvv hello_world.yml23:31
mtreinishs/ansible-play/ansible-playbook/23:31
mtreinishfungi: I think I found it23:32
*** esikachev has quit IRC23:32
fungiyeah, it gets me the same warnings23:32
openstackgerritMatthew Treinish proposed openstack-infra/system-config master: Cast the playbook uuid as a string  https://review.openstack.org/46118023:32
mtreinishfungi: ^^^23:32
*** sree has quit IRC23:33
mtreinishfungi: I'm thinking it might be my local ansible version, I'm running a random checkout on the current master23:34
mtreinishso > 2.3.0 which is the latest release23:34
mtreinishfungi: which ansible version is installed on puppetmaster.o.o?23:36
fungimtreinish: extra verbosity still didn't show me any python tracebacks, but ansible 2.2.1.023:37
fungi(pinned a couple lines from where you edited the manifest)23:37
fungimtreinish: anyway, that patch seems to do the trick23:40
mtreinishyou tested it locally23:41
mtreinish?23:41
fungion the puppetmaster, with your hello world playbook23:41
mtreinishhmm, I didn't see any mqtt events23:41
fungiwhere i was able to reproduce previously23:42
mtreinishoh nope there they are23:42
mtreinishI had the wrong terminal open23:42
fungii still get one failure from that playbook but it seems to be unrelated23:42
fungithe echo task at the end fails23:42
*** jamesmca_ has joined #openstack-infra23:42
*** Swami has quit IRC23:42
mtreinishhttp://paste.openstack.org/show/608375/23:42
mtreinishyeah there is an exit 1 there23:42
mtreinishI did that on purpose to see the error code :)23:42
fungiaha, yep, i missed it before23:42
fungiokay, so working as intended23:43
fungii've approved the one-liner fix to the plugin so we should see this working shortly23:43
openstackgerritColleen Murphy proposed openstack-infra/project-config master: Let the jenkins user own the cache files  https://review.openstack.org/46118123:44
mtreinish\o/ that was easier than I was expecting23:44
cmurphyi think jobs are going to start failing tomorrow morning without that ^23:44
mtreinishnow I guess I'll need to write docs23:44
mtreinishugh, describing the schema is gonna be annoying23:44
mtreinishooh, I'm seeing lots of ansible events now23:45
mtreinishand there is a nice spike in grafana: http://grafana.openstack.org/render/dashboard-solo/db/mosquitto-status?panelId=6&fullscreen&from=1493419594595&to=1493423194595&width=1000&height=50023:46
*** jamesmca_ has quit IRC23:47
openstackgerritMerged openstack-infra/system-config master: Cast the playbook uuid as a string  https://review.openstack.org/46118023:47
mtreinishlol, I really like that the facts end up in the output so we get a bunch of xeon model names in the json23:48
*** Apoorva_ has quit IRC23:48
*** jamesmca_ has joined #openstack-infra23:48
fungiyeah, i think we discussed getting rid of the fact reporting there since it's mostly useless noise to be generating on every single run23:49
fungithere have also been security bugs in the past where module authors didn't understand that custom facts are not a safe place to put things like service passwords23:51
*** gildub has joined #openstack-infra23:53
*** jamesmca_ has quit IRC23:53
*** jamesmca_ has joined #openstack-infra23:54
mtreinishheh, I was a bit worried about that actually23:54
mtreinishthe blob was so big even on my hello world playbook I couldn't inspect it for secrets by eye23:54
mtreinishbut I figured it was a well known enough space it was probably safe23:55
mtreinish(probably a foolish assumption to make)23:55
*** harlowja has quit IRC23:55
*** thorst has joined #openstack-infra23:56
*** jkilpatr has joined #openstack-infra23:56
mtreinishthere sure are a lot of infracloud events23:56
fungilots of machines in there23:57
*** slagle has quit IRC23:57
*** esikachev has joined #openstack-infra23:58
mtreinishooh looks like the most recent run failed: {"status": "FAILED", "playbook_name": "compute*.ic.openstack.org:!disabled", "playbook_id": "2f16fa1a-d9fd-44ad-b717-cf83750fafa2"}23:58
*** jamesmca_ has quit IRC23:59
*** harlowja has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!