Thursday, 2017-10-26

openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance  https://review.openstack.org/51135200:01
*** boden has joined #openstack-infra00:02
*** baoli has quit IRC00:03
*** andreas_s has joined #openstack-infra00:04
*** claudiub|2 has quit IRC00:04
*** dingyichen has joined #openstack-infra00:05
*** rkukura has quit IRC00:07
openstackgerritClark Boylan proposed openstack-infra/zuul feature/zuulv3: Make the git web url a template  https://review.openstack.org/51516800:08
*** andreas_s has quit IRC00:08
*** andreas_s has joined #openstack-infra00:09
*** boden has quit IRC00:10
openstackgerritClark Boylan proposed openstack-infra/zuul feature/zuulv3: Make the git web url a template  https://review.openstack.org/51516800:11
openstackgerritIan Wienand proposed openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3  https://review.openstack.org/51523100:14
ianwpabelanger: ^00:14
*** andreas_s has quit IRC00:15
*** andreas_s has joined #openstack-infra00:15
*** Swami has quit IRC00:16
pabelangerianw: left comment00:16
*** gildub has joined #openstack-infra00:16
ianwpabelanger: but only want to add them when they're up right?00:17
ianwor does that only do firewall config00:18
pabelangerianw: yah, that just opens firewall00:18
pabelangershould be safe to do now00:19
openstackgerritIan Wienand proposed openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3  https://review.openstack.org/51523100:20
ianwok, maybe if names don't resolve it's temporary failiure.  ^00:20
*** florianf has quit IRC00:20
*** andreas_s has quit IRC00:21
*** andreas_s has joined #openstack-infra00:21
*** links has joined #openstack-infra00:23
*** wolverineav has quit IRC00:26
*** andreas_s has quit IRC00:27
*** wolverineav has joined #openstack-infra00:27
*** cody-somerville has joined #openstack-infra00:27
*** cody-somerville has joined #openstack-infra00:27
ianwinfra-root: ^^ anyone able to sanity check ... don't want to bring up new hosts and have puppet roll out the zuulv2 treatment to them00:27
*** andreas_s has joined #openstack-infra00:27
*** csomerville has quit IRC00:29
jeblairianw: one nit.  otherwise lgtm.  we'll need to restart the relevant firewalls after the hosts are re-added to dns.00:30
jeblair(and after dns has propagated)00:31
*** mummer has quit IRC00:31
*** wolverineav has quit IRC00:31
*** andreas_s has quit IRC00:32
*** Apoorva_ has joined #openstack-infra00:32
openstackgerritIan Wienand proposed openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3  https://review.openstack.org/51523100:32
ianwjeblair: ^ ... firewalls is in the plan (https://etherpad.openstack.org/p/zuulv2-removal) :)00:33
*** andreas_s has joined #openstack-infra00:33
*** Apoorva has quit IRC00:35
*** Apoorva_ has quit IRC00:36
*** baoli has joined #openstack-infra00:37
*** andreas_s has quit IRC00:44
dhineshwhat is the ideal location for ssh keys when building an image using nodepool-builder, so that i can ssh to that instance?00:48
ianwdhinesh: well ... that depends00:49
ianware you using devuser to add a user, or all the infra elements?00:49
*** bobh has joined #openstack-infra00:51
dhinesh    images:00:52
dhinesh      - name: dpc00:52
dhinesh        min-ram: 819200:52
dhinesh        diskimage: dpc00:52
dhinesh        username: jenkins00:52
dhinesh        private-key: '/home/nodepool/.ssh/id_rsa'00:52
dhineshi currently have a nodepool.yaml, with entries like above00:52
openstackgerritCao Xuan Hoang proposed openstack-infra/project-config master: Add docs job for VPNaaS and VPNaaS dashboard  https://review.openstack.org/51490600:52
dhineshianw: sorry for the weird paste00:52
dhinesh ianw: but this does not seem to work00:52
*** andreas_s has joined #openstack-infra00:53
ianwdhinesh: it's going to pick up the ~/.ssh/id_rsa of the building user IIRC00:54
*** kiennt26 has joined #openstack-infra00:55
ianwbut, if you're using glean, that should allow you to put other keys into the config-drive that will be added to root00:55
*** namnh has joined #openstack-infra00:55
ianwthat's how we get the infra root's keys on there00:55
*** cuongnv has joined #openstack-infra00:58
mnaseris there a way of knowing if zuulv3 is running with a specific change merged?00:58
mnasermore specifically, https://review.openstack.org/#/c/514744/00:58
*** LindaWang has joined #openstack-infra00:59
clarkbianw: dhinesh it will pick up the ~/.ssh/id_rsa.pub01:00
clarkbimage build sshould never touch te private key only ssh'ing into instances does that01:00
clarkbmnaser: you can check the zuul version on the status page01:01
clarkbmnaser: thta change is running iirc01:01
mnaseroo01:01
mnaserid have to somehow match the version which seems like a lot of work :p but ill take your word for it and report back if it breaks01:01
*** rwsu has joined #openstack-infra01:02
clarkbmnaser: git describe is the magic to check typically01:02
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance  https://review.openstack.org/51135201:03
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524901:03
mnaserdarn01:04
mnaserjeblair maybe i'm missing things but it looks like the zuulv3 issue we reported about job variants was moved to the fixed etherpad but i still see it occur01:04
* mnaser has been out of the loop for a day or two01:04
jeblairmnaser: yeah, restarted with that about 10 hours ago.01:05
mnaseri just tried a recheck on https://review.openstack.org/#/c/512835 and it tried to run puppet-openstack-integration-4-scenario004-tempest-{centos-7,ubuntu-xenial} again01:05
mnaserit hasn't reported back but i can see it in the zuul status page01:05
*** dave-mccowan has quit IRC01:05
mnaserand just reported back01:06
jeblairmnaser: you may be running into the problem that i tracked down with EmilienM earlier.  is it a job that inherits from another job that has file exclusion matchers and is on multiple branches?01:06
*** andreas_s has quit IRC01:06
mnaserthe job inherits another job indeed (re-using the base job which handles common logic) but no file exclusion matchers, however, we have a branch matcher on scenario00401:08
*** andreas_s has joined #openstack-infra01:08
mnaserjeblair sorry, i lied, the base job has file exclusion matchers too01:09
mnaserso yeah, looks like its the same issue01:09
jeblairmnaser: okay, that's almost certainly it.  the good news is that i just added a test for that to my WIP change.  the bad news is it probably won't be in production before friday.01:09
*** tinwood has quit IRC01:10
mnaserjeblair its ok, as long as we have forward progress :D01:10
openstackgerritTovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.versionedobjects job  https://review.openstack.org/51178401:10
mnaserjeblair good luck and thanks again :)01:10
*** tinwood has joined #openstack-infra01:11
jeblairmnaser: if it's urgent, we can probably make it work by adding or removing file matchers, but if you can wait, it will reduce the amount of temp work we have to undo01:11
*** dave-mccowan has joined #openstack-infra01:12
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance  https://review.openstack.org/51135201:12
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524901:12
mnaserjeblair i know the issue is currently blocking the version bump for queens milestone 1 for some modules which puts us behind, but mwhahaha takes care of release so it's his call - https://review.openstack.org/#/q/topic:prep-release-queens-1+status:open --- or perhaps infra can merge this for us (if its possible and if mwhahaha prefers so we can move with release)01:13
mnaseri'll leave that up to him :>01:13
dhineshianw: in that case, is the key stored as part of the image build, if so what is the location?01:13
*** andreas_s has quit IRC01:13
openstackgerritTovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.db job  https://review.openstack.org/51141201:14
*** andreas_s has joined #openstack-infra01:14
ianwdhinesh: i'm not sure exactly what you mean.  but, if you try, from the host & user you build images with "ssh -i ~/.ssh/id_rsa jenkins@remote.host.com" you might have luck01:17
*** salv-orl_ has joined #openstack-infra01:18
dhineshianw: ok let me give it a shot, thanks01:20
*** esberglu has joined #openstack-infra01:20
*** esberglu has quit IRC01:20
openstackgerritMerged openstack-infra/system-config master: Switch zm[1-4].o.o to zuulv3  https://review.openstack.org/51523101:20
openstackgerritMerged openstack-infra/irc-meetings master: Change Congress meeting time  https://review.openstack.org/51346701:21
dhineshianw: i meant the location within the VM, where the key will be stored.01:21
*** salv-orlando has quit IRC01:21
*** kiennt26 has quit IRC01:23
*** andreas_s has quit IRC01:24
*** huanxie has joined #openstack-infra01:24
ianwdhinesh: the key in ~/.ssh/id_rsa is added to authorized-keys by one of the infra build elements -- see https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user01:25
*** andreas_s has joined #openstack-infra01:26
*** daidv has joined #openstack-infra01:27
*** liujiong has joined #openstack-infra01:29
*** esberglu has joined #openstack-infra01:30
dhineshianw: ok, from the code it looks like, the node yaml entry of private-key: '/home/nodepool/.ssh/id_rsa' would be overwritten in any case01:31
clarkbno those are separate things01:32
*** raissa has quit IRC01:32
clarkbthe yaml private key is telling nodepool what key to use to ssh into the VMs that have been booted01:32
clarkbthe disk image builder element is telling dib what public key to write on disk01:32
clarkbultimately those two have to line up but one is explicitly the private key and the other is explicitly the public key (they just have to be a pair)01:33
*** yamahata has joined #openstack-infra01:33
*** kiennt26 has joined #openstack-infra01:34
*** andreas_s has quit IRC01:35
*** andreas_s has joined #openstack-infra01:36
*** anup has quit IRC01:37
*** luzC has quit IRC01:38
dhineshclarkb: got it, thanks01:38
*** psachin has joined #openstack-infra01:41
*** Qiming has quit IRC01:43
*** luzC has joined #openstack-infra01:44
*** anupn has joined #openstack-infra01:44
*** andreas_s has quit IRC01:45
*** esberglu has quit IRC01:46
*** raissa has joined #openstack-infra01:46
*** andreas_s has joined #openstack-infra01:47
*** dhinesh has quit IRC01:47
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Switch to late-binding inheritance  https://review.openstack.org/51135201:48
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524901:48
*** gus has quit IRC01:50
*** gus has joined #openstack-infra01:51
*** gyee has quit IRC01:52
*** yamahata has quit IRC01:55
*** andreas_s has quit IRC01:56
*** Guest12599 has quit IRC02:00
*** lindycoder__ has joined #openstack-infra02:00
*** lindycoder__ has quit IRC02:00
*** Qiming has joined #openstack-infra02:01
*** spectr has quit IRC02:03
*** fanzhang__ is now known as fanzhang02:04
*** spectr has joined #openstack-infra02:05
*** mat128 has joined #openstack-infra02:05
*** Qiming has quit IRC02:05
*** camunoz has quit IRC02:08
*** rwsu has quit IRC02:11
*** ramishra has joined #openstack-infra02:11
*** rwsu has joined #openstack-infra02:14
openstackgerritzhongshengping proposed openstack-infra/openstack-zuul-jobs master: Remove useless legacy-puppet-check-jobs  https://review.openstack.org/51526302:16
*** bobh has quit IRC02:18
*** felipemonteiro_ has joined #openstack-infra02:20
*** Sukhdev has joined #openstack-infra02:20
*** vhosako__ has quit IRC02:22
*** vhosakot has joined #openstack-infra02:22
*** dhinesh has joined #openstack-infra02:24
*** zerick has quit IRC02:26
*** hongbin has joined #openstack-infra02:27
*** zerick has joined #openstack-infra02:27
*** masuberu has quit IRC02:34
*** gongysh has joined #openstack-infra02:35
*** gongysh has quit IRC02:36
*** mrodden has quit IRC02:41
*** larsks has quit IRC02:42
*** zerick has quit IRC02:43
*** zerick has joined #openstack-infra02:43
*** mrodden has joined #openstack-infra02:44
*** larsks has joined #openstack-infra02:44
*** vhosakot_ has joined #openstack-infra02:47
*** vhosakot has quit IRC02:48
*** rosmaita has quit IRC02:53
*** rwsu has quit IRC02:54
*** masber has joined #openstack-infra02:55
*** masber has quit IRC02:57
*** cuongnv has quit IRC03:00
*** cuongnv has joined #openstack-infra03:01
tonybWhat is the right way to to clone a repo that isn't a required project?  In the past I'd use zuul-cloner and point it at the /opt/git cache in the nodepool image03:02
*** Qiming has joined #openstack-infra03:03
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364503:07
*** Qiming has quit IRC03:07
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364503:09
*** dave-mccowan has quit IRC03:10
clarkbtonyb: you probably want to clone from cache then update origin and update head?03:11
clarkbit may be worth having a script/tool for that03:11
*** Qiming has joined #openstack-infra03:12
tonybclarkb: Yeah I think that's what I need.  I was thinking I could get way without respecting Depends-On but I can't :(03:14
*** annp has joined #openstack-infra03:14
tonybclarkb: where is the cache? still in /opt/git?03:15
*** Qiming has quit IRC03:16
clarkbtonyb: yes03:17
clarkbtonyb: if you need to respect depends on then maybe just use required projects?03:17
tonybclarkb: I can't :(03:18
tonybclarkb: It's for validating $project uses check-requirements so I just need to grovel around in the global and per project zuul config03:19
tonybclarkb: it's similar to what we're doign for releases and publish jobs03:19
tonybfrickler: Modified the tool to look in project-config but the per-projects stuff is a todo item03:20
tonybclarkb: I guess we could punt but that seems poor form03:20
clarkbI see03:20
*** nicolasbock has quit IRC03:22
*** yamamoto has joined #openstack-infra03:22
*** gouthamr has quit IRC03:25
*** armaan has quit IRC03:31
*** thorst has joined #openstack-infra03:35
*** thorst has quit IRC03:41
*** Qiming has joined #openstack-infra03:50
*** iyamahat has joined #openstack-infra03:54
*** sree has joined #openstack-infra03:56
ianwhey clarkb -- the error you mentioned in http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-08-16.log.html#t2017-08-16T23:58:2503:59
ianwpuppet-agent[9836]: Could not request certificate: Failed to open TCP connection to puppet:814003:59
ianwwas that a red-herring?03:59
ianwi'm seeing creation seem to get stuck on "copy system puppet modules"04:00
*** erlon has quit IRC04:00
ianwbut it's worked for two prior hosts04:00
*** felipemonteiro_ has quit IRC04:01
clarkbianw: ya it went on to work04:03
clarkbianw: I htink what is happening is the xenial distro package for puppet3 tried to start the service04:03
*** baoli has quit IRC04:05
*** hongbin has quit IRC04:05
ianwclarkb: hmm, maybe i'll add a manual stop to launcher after it does install_puppet.sh04:06
*** dhinesh has quit IRC04:08
*** armaan has joined #openstack-infra04:11
*** huanxie has quit IRC04:20
*** armaan has quit IRC04:20
ianwthat helps, it's still stuck syncronising puppet repos for some reason though04:22
*** udesale has joined #openstack-infra04:22
*** markvoelker has quit IRC04:23
*** ykarel has joined #openstack-infra04:24
*** dhajare has joined #openstack-infra04:28
*** udesale has quit IRC04:34
*** mat128 has quit IRC04:35
*** kiennt26 has quit IRC04:36
ianwdo i want to know why "virtualenv" creates a python3 virtualenv on puppetmaster by default?04:39
tonybianw: forward thinking? :D04:40
*** Hunner has quit IRC04:41
*** Hunner has joined #openstack-infra04:41
*** Hunner has quit IRC04:41
*** Hunner has joined #openstack-infra04:41
clarkbit uses the python it is installed under by default iirc04:41
clarkbso maybe it got pip3 installed?04:41
jeblairtonyb: the only way to get depends-on is via required-projects.04:48
*** dhinesh has joined #openstack-infra04:49
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Switch to late-binding inheritance  https://review.openstack.org/51135204:49
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524904:49
*** claudiub|2 has joined #openstack-infra04:49
jeblairtonyb: which job are you working on?04:50
tonybjeblair: Thanks.  We can't use required-projects.  In practise it wont be anyt different to waht we had pre-zuulv304:50
tonybjeblair: validate-projects for requirements04:50
tonybjeblair: what it did pre-v3 was lok in layout.yaml to see if the project had check-requirements listed in the template section04:51
tonybjeblair: (for every project in projects.txt)04:51
openstackgerritIan Wienand proposed openstack-infra/system-config master: Stop puppet during install  https://review.openstack.org/51527904:52
tonybjeblair: We can look in zuul/projects.txt which has been done but I'm pretty sure that'd break longer term if $project was doing in repo config04:52
ianwclarkb: ^^ re that puppet-agent warning error04:53
tonybjeblair: The Depends-On thing is a nice to have but based on what you said it's a non-starter04:53
jeblairtonyb: we're planning on, very soon, having a rest api where we should be able to interrogate zuul's running config.  i wonder if that might be the best approach.  (it still won't have depends-on support, but may be easier than the alternative)04:54
*** dhajare is now known as dhajare|mtg04:54
tonybjeblair: For me that'd be close to perfect04:54
*** markvoelker has joined #openstack-infra04:55
tonybjeblair: I asked yesterday if such a thing existed ... I should have been more forward thinking ;P04:55
jeblairtonyb: maybe go with checking project-config for short-term, don't worry about the in-repo config and hopefully we'll have the api before it's a problem?  :)04:55
tonybjeblair: That sir sounds like a bonza[1] plan04:56
tonyb[1] https://www.urbandictionary.com/define.php?term=Bonza04:56
*** armaan has joined #openstack-infra04:57
tonybjeblair: I'd offer to help but I suspect I'd just slow y'all down04:57
andreafjeblair, mordred: I still get no real logging from zuul about what happens to my post-tempest play here http://logs.openstack.org/72/509272/13/check/devstack-tempest/d8d2021/job-output.txt.gz#_2017-10-25_17_10_46_09887404:58
andreafjeblair, mordred: I'm trying to consume on tempest side the stage-output roles via depends-on - I probably did some silly mistake in my patch but without logging is hard to tell04:59
*** huanxie has joined #openstack-infra04:59
jeblairandreaf: wow the post-tempest playbook doesn't even show up in ara or the json05:00
jeblairlemme see what i can dig up off the executor05:00
tonybandreaf: aren't you just copying them over themselves?05:01
tonybandreaf: logs_root and stage_dir are the same thing05:02
andreaftonyb: they go from /opt/stack/tempest/... into /opt/stack/logs05:03
jeblairandreaf: "ERROR! the role 'stage-output' was not found in ..."05:03
jeblairandreaf: i added an entry to the etherpad for mordred so when he's back around he can look at why that didn't end up in the job log05:03
andreafjeblair: ok thanks05:04
andreafjeblair: oh, the depends-on points to the top patch of the series that is not rebased, that's why the name is not visible05:04
*** vhosakot_ has quit IRC05:04
andreafjeblair: thanks05:04
*** vhosakot has joined #openstack-infra05:05
andreafjeblair, tonyb: I assumed the staging area could still be /opt/stack/logs... but it doesn't have to, it can be anywhere05:05
tonybandreaf: okay.  I see how that works now05:06
jeblairokay, it's past my bedtime.  'night :)05:06
andreafjeblair: 'night :)05:07
*** vhosakot_ has joined #openstack-infra05:12
*** armaan has quit IRC05:12
*** yamahata has joined #openstack-infra05:13
*** armaan has joined #openstack-infra05:13
*** vhosakot has quit IRC05:14
*** gcb has joined #openstack-infra05:16
ianw#info zm[0-4].o.o rebuilt to xenial and added to zuulv305:16
*** armaan has quit IRC05:17
*** yamamoto has quit IRC05:17
ianw#log zm[0-4].o.o rebuilt to xenial and added to zuulv305:17
ianw#status log zm[0-4].o.o rebuilt to xenial and added to zuulv305:18
openstackstatusianw: finished logging05:18
ianwdurr05:18
*** markvoelker has quit IRC05:24
*** rwsu has joined #openstack-infra05:24
*** armax has quit IRC05:24
*** armax has joined #openstack-infra05:24
*** markvoelker has joined #openstack-infra05:24
*** armax has quit IRC05:25
*** armax has joined #openstack-infra05:25
*** armax has quit IRC05:25
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Move to a common lock-file directory  https://review.openstack.org/51407905:28
*** markvoelker has quit IRC05:28
*** jbadiapa_ has quit IRC05:32
*** thorst has joined #openstack-infra05:37
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364505:39
*** thorst has quit IRC05:41
*** nikhil has quit IRC05:45
openstackgerritMerged openstack-infra/zuul-jobs master: Remove pep8/pyflakes  https://review.openstack.org/51504105:55
openstackgerritzhongshengping proposed openstack-infra/openstack-zuul-jobs master: Remove migrated legacy puppet openstack check jobs  https://review.openstack.org/51526305:56
*** aeng has quit IRC05:59
openstackgerritzhongshengping proposed openstack-infra/openstack-zuul-jobs master: Remove migrated legacy puppet openstack check jobs  https://review.openstack.org/51526305:59
*** sree has quit IRC06:00
*** sree has joined #openstack-infra06:00
*** salv-orlando has joined #openstack-infra06:03
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Test binary data in zuul stream  https://review.openstack.org/51528306:03
*** salv-orl_ has quit IRC06:04
*** sree_ has joined #openstack-infra06:05
*** sree has quit IRC06:05
*** sree_ is now known as Guest1391406:05
*** cshastri has joined #openstack-infra06:05
*** e0ne has joined #openstack-infra06:10
*** dhinesh has quit IRC06:14
*** liusheng has joined #openstack-infra06:15
*** Sukhdev has quit IRC06:19
*** andreas_s has joined #openstack-infra06:20
*** vsaienk0 has joined #openstack-infra06:20
*** kjackal_ has joined #openstack-infra06:21
*** e0ne has quit IRC06:28
*** udesale has joined #openstack-infra06:30
*** janki has joined #openstack-infra06:31
*** kiennt26 has joined #openstack-infra06:40
*** LindaWang1 has joined #openstack-infra06:45
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer  https://review.openstack.org/51504306:46
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Test binary data in zuul stream  https://review.openstack.org/51528306:47
*** LindaWang has quit IRC06:49
*** LindaWang1 is now known as LindaWang06:49
*** yamamoto has joined #openstack-infra06:49
*** gongysh has joined #openstack-infra06:52
*** ociuhandu has joined #openstack-infra06:53
*** florianf has joined #openstack-infra06:56
*** jbadiapa_ has joined #openstack-infra06:57
*** martinkopec has joined #openstack-infra06:57
*** jbadiapa_ is now known as jbadiapa06:57
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer  https://review.openstack.org/51504307:00
*** iyamahat has quit IRC07:04
*** rcernin has quit IRC07:04
*** ociuhandu has quit IRC07:05
*** rossella_s has joined #openstack-infra07:08
*** tesseract has joined #openstack-infra07:09
*** apetrich has quit IRC07:09
*** salv-orlando has quit IRC07:12
*** Hal has joined #openstack-infra07:12
*** Hal is now known as Guest3195407:12
*** salv-orlando has joined #openstack-infra07:12
*** e0ne has joined #openstack-infra07:15
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test containers update  https://review.openstack.org/51117507:15
*** salv-orlando has quit IRC07:16
*** Goneri has joined #openstack-infra07:16
openstackgerritMerged openstack-infra/project-config master: Fix neutron-failure-rate grafana dashboard  https://review.openstack.org/50929107:18
*** aviau has quit IRC07:19
*** ralonsoh has joined #openstack-infra07:19
*** aviau has joined #openstack-infra07:19
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Test binary data in zuul stream  https://review.openstack.org/51528307:23
*** markvoelker has joined #openstack-infra07:25
openstackgerritAndrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic stage-output role  https://review.openstack.org/50923307:25
*** yolanda has joined #openstack-infra07:27
*** gongysh has quit IRC07:28
*** huanxie has quit IRC07:31
fricklertonyb: do you think we are ready to make the reqs job voting again? https://review.openstack.org/51500707:34
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer  https://review.openstack.org/51504307:35
*** amoralej|off is now known as amoralej07:36
tonybfrickler: No I think we shoudl leave it non-voting until we can do the per-project query which jeblair says we'll be able to do via the REST api tomorrow (well maybe not that soon)07:37
tonybfrickler: I'm okay with manually verifying any failures, but you shoudl probably get an answer from prometheanfire07:37
AJaegertonyb: we recommend that projects have check-requirements in project-config07:38
tonybAJaeger: okay, but not require right?07:38
AJaegertonyb: that's the policy we have documented, so I would be fine with the check as is and require that projects leave it in there07:38
*** yamamoto_ has joined #openstack-infra07:38
AJaegertonyb: read https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert07:39
AJaegertonyb: so we require it for governance projects. And the requirements team could require it for everybody ;)07:39
*** gongysh has joined #openstack-infra07:39
AJaegerwould make life easier - and I have seen seldom a need to customize this at all, so no need to move in-repo.07:39
*** huanxie has joined #openstack-infra07:39
tonybAJaeger: okay.07:40
AJaegertonyb: but discuss with prometheanfire and rest of team.07:40
frickleralso, if we allow a project to configure and drop it on their own power, that check seems to not make sense anymore at all to me07:40
AJaegertonyb: we could also for now make the job voting and you could wait and see...07:40
tonybAJaeger: sure.  I guess if it's documented when it fails we just say "you did it wrong" and point them at the docs07:42
tonybAJaeger, frickler: I've +1'd the review07:42
*** yamamoto has quit IRC07:42
*** salv-orlando has joined #openstack-infra07:42
AJaegertonyb: works for me ;)07:42
*** jpena|off is now known as jpena07:43
AJaegerI'll approved the revert now...07:43
*** ccamacho has joined #openstack-infra07:43
tonybfrickler, AJaeger: Thanks07:43
AJaegerfrickler: care to +2 https://review.openstack.org/#/c/514911/ some time today, please?07:47
* AJaeger likes to get rid of newton jobs quickly ;)07:47
*** stakeda has joined #openstack-infra07:47
*** kiennt26 has quit IRC07:48
*** gildub has quit IRC07:50
openstackgerritMerged openstack-infra/project-config master: Revert "Make legacy-requirements-tox-validate-projects non-voting until fixed"  https://review.openstack.org/51500707:50
jd_howdy07:51
*** kiennt26 has joined #openstack-infra07:51
jd_we still have our Zuul playbooks issue we don't really understand on https://review.openstack.org/512700 where zuul complains about missing playbooks whereas we remove the jobs :) if anybody has an idea07:52
openstackgerritChandan Kumar proposed openstack-infra/shade master: Remove enabling IDENTITY_V2 API  https://review.openstack.org/51530407:55
vsaienk0folks after zuulv3 ennoblement we found that ironic-python-agent post job  didn't upload images to tarballs.openstack.org. Looks like path was changed with migration can you please check? http://logs.openstack.org/8b/8bf0629749ac4947762b5307b8048087f928ff13/post/ironic-python-agent-buildimage-tinyipa/eedc93a/job-output.txt.gz#_2017-10-25_23_18_59_57493007:57
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Default change and patchset to NULL in SQLReporter  https://review.openstack.org/51442307:58
*** markvoelker has quit IRC07:59
*** gcb has quit IRC08:01
*** namnh has quit IRC08:04
*** namnh has joined #openstack-infra08:04
*** gildub has joined #openstack-infra08:07
fricklerjd_: I'm only guessing but it may be due to the way zuul processes this patch, it is trying to read the playbooks before noticing that the job definition is removed. the solution then would be to split the patch, first remove the job definition only and do the playbook removal in a second patch08:07
*** vsaienk0 has quit IRC08:08
jd_frickler: hm but won't it runs the target in the first patch anyway? that'd be a problem08:08
*** armaan has joined #openstack-infra08:08
fricklerjd_: because they are failing? then maybe you can make them non-voting first08:10
*** vsaienk0 has joined #openstack-infra08:11
*** ykarel is now known as ykarel|lunch08:11
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.versionedobjects job  https://review.openstack.org/51178408:11
jd_frickler: O_O if that's it, it really sounds like a nifty Zuul bug/limitation08:11
jd_AJaeger: opinion?08:11
*** apetrich has joined #openstack-infra08:12
*** slaweq has joined #openstack-infra08:14
fricklerjd_: I agree that this looks like a bug or missing feature at least. can you submit a change with only your patch to .zuul.yaml to verify? we can then add it to our list at https://etherpad.openstack.org/p/zuulv3-issues for further evaluation08:18
*** yamamoto_ has quit IRC08:18
*** slaweq has quit IRC08:18
*** CrayZee has joined #openstack-infra08:21
*** david-lyle has quit IRC08:24
*** hamzy has quit IRC08:24
*** david-lyle has joined #openstack-infra08:25
*** hamzy has joined #openstack-infra08:26
*** dbecker has joined #openstack-infra08:27
openstackgerritTovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.db job  https://review.openstack.org/51141208:27
*** yamamoto has joined #openstack-infra08:28
*** huanxie has quit IRC08:29
*** shardy has joined #openstack-infra08:29
*** gildub has quit IRC08:31
openstackgerritFlavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo  https://review.openstack.org/51531608:31
openstackgerritFlavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo to zuul.d  https://review.openstack.org/51531708:31
*** d0ugal has quit IRC08:32
*** d0ugal_ has joined #openstack-infra08:32
*** d0ugal_ has quit IRC08:33
*** yamamoto has quit IRC08:33
*** d0ugal has joined #openstack-infra08:33
*** d0ugal has quit IRC08:33
*** d0ugal has joined #openstack-infra08:33
*** huanxie has joined #openstack-infra08:35
*** hashar has joined #openstack-infra08:41
*** lucas-afk is now known as lucasagomes08:41
*** dingyichen has quit IRC08:41
openstackgerritFlavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo  https://review.openstack.org/51531608:42
openstackgerritFlavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-tripleo to zuul.d  https://review.openstack.org/51531708:42
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Updated templates for heat-dashboard  https://review.openstack.org/51489508:42
*** derekh has joined #openstack-infra08:45
*** ociuhandu has joined #openstack-infra08:47
*** hashar is now known as hasharAway08:49
*** iyamahat has joined #openstack-infra08:53
*** yamahata has quit IRC08:56
*** markvoelker has joined #openstack-infra08:57
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Updated templates for heat-dashboard  https://review.openstack.org/51489508:57
*** ykarel|lunch is now known as ykarel08:58
*** bhavik1 has joined #openstack-infra09:00
*** yamamoto has joined #openstack-infra09:01
*** electrofelix has joined #openstack-infra09:03
*** bhavik1 has quit IRC09:05
*** huanxie has quit IRC09:05
*** tosky has joined #openstack-infra09:05
vsaienk0infra-team please help to investigate https://bugs.launchpad.net/ironic-python-agent/+bug/1727652 after migration to zuulv3 we found that ironic-python-agent images are not uploaded to tarballs09:06
openstackLaunchpad bug 1727652 in ironic-python-agent "post job doesn't work after migration to zuulv3" [Undecided,New]09:06
*** panda|ruck|off is now known as panda|ruck09:06
jd_frickler: I think your theory does not hold because the same problem happens on the next patch https://review.openstack.org/51309809:07
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Make version_string() lazy load version info  https://review.openstack.org/45091809:08
*** iyamahat has quit IRC09:10
andreafAJaeger, frickler: I just noticed that the ARA folder is not copied in legacy-tempest-dsvm-neutron-full - is that a known issue?09:16
vsaienk0AJaeger: frickler could you please help to investigate issue described above https://bugs.launchpad.net/ironic-python-agent/+bug/1727652 ^09:17
openstackLaunchpad bug 1727652 in ironic-python-agent "post job doesn't work after migration to zuulv3" [High,Confirmed]09:17
fricklerandreaf: do you have a link to a failure handy?09:18
andreaffrickler: http://logs.openstack.org/72/509272/13/check/legacy-tempest-dsvm-neutron-full/cc0bfed/09:19
andreaffrickler: there's no failure in the sense of error in the logs, I just don't see the ARA folder there09:19
*** gongysh has quit IRC09:23
*** namnh has quit IRC09:24
*** namnh has joined #openstack-infra09:24
*** pgadiya has joined #openstack-infra09:26
*** david-lyle has quit IRC09:26
*** dklyle has joined #openstack-infra09:26
*** hamzy has quit IRC09:27
*** huanxie has joined #openstack-infra09:28
*** hamzy has joined #openstack-infra09:28
*** markvoelker has quit IRC09:29
*** eumel8 has joined #openstack-infra09:30
*** wolverineav has joined #openstack-infra09:30
fricklerandreaf: looks like it may be generated only when the job is failing, the folder does exist e.g. for http://logs.openstack.org/80/348080/23/check/legacy-tempest-dsvm-neutron-full/42bcc78/09:31
*** dtantsur|afk is now known as dtantsur09:31
andreaffrickler: I see... that's a bit strange though - it's always generated for new style jobs09:32
andreaffrickler: but ok...09:32
fricklerandreaf: I think this may have been changed to save space/inodes on the zuul+log servers, introduced in https://review.openstack.org/51191809:35
andreaffrickler: ok got it09:35
openstackgerritWitold Bedyk proposed openstack-infra/project-config master: Add nodejs4-publish jobs  https://review.openstack.org/51533209:35
andreaffrickler: thanks09:36
*** wolverineav has quit IRC09:37
*** yamamoto has quit IRC09:38
*** thorst has joined #openstack-infra09:39
*** sshnaidm is now known as sshnaidm|afk09:39
*** thorst has quit IRC09:43
*** pgadiya has quit IRC09:43
openstackgerritMerged openstack-infra/tripleo-ci master: Match job-output as console logs for getthelogs  https://review.openstack.org/51456009:45
*** gongysh has joined #openstack-infra09:49
openstackgerritNiraj Singh proposed openstack-infra/project-config master: Add masakari-dashboard project  https://review.openstack.org/51533709:49
openstackgerritgongysh proposed openstack-infra/project-config master: Remove Tacker legacy job  https://review.openstack.org/51534109:52
*** rossella_s has quit IRC09:56
*** erlon has joined #openstack-infra09:57
andreykurilinhi folks09:59
andreykurilinwe had a critical issue09:59
*** pgadiya has joined #openstack-infra10:00
andreykurilinI missed that one of our rally branch was mentioned here http://lists.openstack.org/pipermail/openstack-dev/2017-September/122381.html (there was not rally tag in the subject)10:00
andreykurilinAnd an active branch (a stable branch for previous rally release) was removed yesturday10:00
*** armaan has quit IRC10:00
andreykurilinthis branch was created just a half year ago and I'm not sure why it was marked as old10:01
andreykurilinThe removing stable/0.9 affected some of our customers10:02
andreykurilintonyb^10:03
*** sdague has joined #openstack-infra10:03
andreykurilinI have this branch at local repo. Does it possible to push it back?10:04
andreykurilinAlso, the tag 0.9-oel should be removed. I do not know how to do it10:04
andreykurilinAny thoughts on the subject or it is better to raise the questions at openstack-releases channel?10:04
openstackgerritgongysh proposed openstack-infra/openstack-zuul-jobs master: Remove tacker legacy job  https://review.openstack.org/51534610:05
openstackgerritMerged openstack-infra/tripleo-ci master: Mirror images from RDO server  https://review.openstack.org/51036210:07
openstackgerritgongysh proposed openstack-infra/project-config master: Remove Tacker legacy job  https://review.openstack.org/51534110:07
*** cuongnv has quit IRC10:09
*** sambetts|afk is now known as sambetts10:09
*** jklare_ has joined #openstack-infra10:09
*** liujiong has quit IRC10:10
*** jklare has quit IRC10:12
*** jklare_ is now known as jklare10:12
*** andreas_s has quit IRC10:12
*** andreas_s has joined #openstack-infra10:13
*** namnh has quit IRC10:16
*** kiennt26 has quit IRC10:16
*** andreas_s has quit IRC10:17
*** bhavik1 has joined #openstack-infra10:17
*** sshnaidm|afk is now known as sshnaidm10:18
*** pgadiya has quit IRC10:21
*** andreas_s has joined #openstack-infra10:24
AJaegerandreykurilin: the tag was [all]. Best to followup via email to the removal email. AFAIK it's possible to revert parts of this (the tag is distributed, even if we delete it, everybody will have it) but best to discuss later with rest of the team. Right now nobody is around that can help.10:25
*** markvoelker has joined #openstack-infra10:26
*** ldnunes has joined #openstack-infra10:27
*** e0ne has quit IRC10:28
*** andreas_s has quit IRC10:29
*** gmann is now known as gmann_afk10:29
andreykurilinAJaeger: I'm ok about that someone already obtain the tag. I just do not want to continue distributing it (at pypi at least).10:29
andreykurilin*already obtained10:30
*** e0ne has joined #openstack-infra10:30
*** e0ne has quit IRC10:30
*** e0ne has joined #openstack-infra10:31
*** e0ne has quit IRC10:32
*** e0ne has joined #openstack-infra10:33
*** e0ne has quit IRC10:33
*** pgadiya has joined #openstack-infra10:34
*** Guest13914 has quit IRC10:34
*** vhosakot_ has quit IRC10:34
*** sree has joined #openstack-infra10:35
*** andreas_s has joined #openstack-infra10:36
*** boden has joined #openstack-infra10:36
*** yamamoto has joined #openstack-infra10:39
*** sree has quit IRC10:40
*** apetrich has quit IRC10:41
*** pbourke has quit IRC10:42
*** gcb has joined #openstack-infra10:43
AJaeger#status log we lost the docs.o.o central home page, somehow our publishing is broken10:43
openstackstatusAJaeger: finished logging10:43
*** pbourke has joined #openstack-infra10:44
*** yamamoto has quit IRC10:44
*** ramishra has quit IRC10:45
*** yamamoto has joined #openstack-infra10:47
*** ramishra has joined #openstack-infra10:47
*** andreas_s has quit IRC10:47
*** andreas_s has joined #openstack-infra10:49
*** stakeda has quit IRC10:50
*** gongysh has quit IRC10:52
openstackgerritWitold Bedyk proposed openstack-infra/project-config master: Add nodejs4-publish jobs  https://review.openstack.org/51533210:52
*** andreas_s has quit IRC10:54
*** andreas_s has joined #openstack-infra10:54
*** nicolasbock has joined #openstack-infra10:54
*** jpena is now known as jpena|lunch10:59
*** markvoelker has quit IRC10:59
*** bhavik1 has quit IRC11:02
*** amoralej is now known as amoralej|lunch11:03
*** huanxie has quit IRC11:03
*** vsaienk0 has quit IRC11:06
*** nicolasbock has quit IRC11:06
*** vsaienk0 has joined #openstack-infra11:07
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci master: WIP, test CI with experimental undercloud upgrade  https://review.openstack.org/51536111:07
*** gildub has joined #openstack-infra11:09
*** andreas_s has quit IRC11:10
smcginnisRuh roh. I thought the home page was a static, manually generated portion of the site.11:12
*** udesale has quit IRC11:13
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Do not publish docs when building fails  https://review.openstack.org/51536511:15
AJaegerinfra-root, config-core, let's make our publishing robust ^11:15
AJaegersmcginnis: it is build from jinja-templates and we just removed it due to a failure in tox run ;(, see 515365 for more details11:16
smcginnisOuch.11:19
smcginnisAJaeger: Yeah, good change. There's probably more places where we should be checking that the previous step was a success.11:20
*** nicolasbock has joined #openstack-infra11:20
AJaegersmcginnis: yes11:20
*** gildub has quit IRC11:21
AJaeger#status log docs.o.o index page was lost due to broken build and publishing of broken build, suggested fix in https://review.openstack.org/51536511:22
openstackstatusAJaeger: finished logging11:22
*** tpsilva has joined #openstack-infra11:24
* AJaeger disappears again11:25
*** gongysh has joined #openstack-infra11:26
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Do not publish Zaqar to pypi  https://review.openstack.org/51536611:26
*** andreas_s has joined #openstack-infra11:27
*** janki has quit IRC11:31
andreafmordred, jeblair: so this works now https://review.openstack.org/#/c/509233 and it's kind of proven by https://review.openstack.org/#/c/509272/11:31
andreafmordred, jeblair: but I was wondering if we should do some kind of unit testing of these roles - to validate the various conditions in the roles11:32
andreafon my machine I use a simple play to test things, it should be possible to have something like that in the gate11:33
*** links has quit IRC11:36
*** kjackal_ has quit IRC11:36
*** andreas_s has quit IRC11:36
*** andreas_s has joined #openstack-infra11:38
*** _ari_ has quit IRC11:38
*** smatzek has joined #openstack-infra11:39
*** lucasagomes is now known as lucas-hungry11:40
*** _ari_ has joined #openstack-infra11:43
*** annp has quit IRC11:46
*** andreas_s has quit IRC11:48
*** andreas_s has joined #openstack-infra11:50
*** edmondsw has joined #openstack-infra11:50
*** dave-mccowan has joined #openstack-infra11:51
AJaegerSo, https://docs.openstack.org is up again - this time build worked without failure11:51
AJaegerinfra-root, please delete http://files.openstack.org/docs/publish-docs/ - this was broken publishing and is all wrong...11:52
*** bobh has joined #openstack-infra11:54
*** markvoelker has joined #openstack-infra11:57
*** rossella_s has joined #openstack-infra11:58
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Remove ansible-role-jobs template from openstack-ansible-ops  https://review.openstack.org/51537711:59
*** bobh has quit IRC12:00
*** ociuhandu_ has joined #openstack-infra12:01
*** thorst has joined #openstack-infra12:01
*** ociuhandu has quit IRC12:01
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Build tarballs for networking-{bagpipe,bgvpn}  https://review.openstack.org/51538112:02
*** ociuhandu_ has quit IRC12:02
*** salv-orl_ has joined #openstack-infra12:06
*** salv-orlando has quit IRC12:09
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Switch service projects to not publish to pypi  https://review.openstack.org/51536612:13
*** kjackal_ has joined #openstack-infra12:13
*** jpena|lunch is now known as jpena12:13
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Switch service projects to not publish to pypi  https://review.openstack.org/51536612:18
*** shardy has quit IRC12:20
*** jcoufal has joined #openstack-infra12:20
*** dsariel has joined #openstack-infra12:21
*** sree has joined #openstack-infra12:24
dsarielhi!  https://review.openstack.org/#/c/515377/ has a list of gates ( openstack-tox-linters ) triggered by change request. Where it is specified what is the list of gates what should be triggred unit +1 is given12:25
*** cody-somerville has quit IRC12:25
*** cody-somerville has joined #openstack-infra12:25
*** rlandy has joined #openstack-infra12:27
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Switch service projects to not publish to pypi  https://review.openstack.org/51536612:28
*** sree has quit IRC12:29
*** markvoelker has quit IRC12:29
*** wolverineav has joined #openstack-infra12:32
*** lucas-hungry is now known as lucasagomes12:34
fricklerdsariel: I'm not sure I understand your question correctly. did you read https://docs.openstack.org/infra/manual/zuulv3.html ? that should answer most questions regarding how checks are being defined now12:36
*** markvoelker has joined #openstack-infra12:40
*** LindaWang has quit IRC12:40
dmsimardAJaeger: for https://review.openstack.org/#/c/515365/, the only thing I wonder is if we should handle that at the role level rather than the playbook level12:41
*** gcb has quit IRC12:42
*** gcb has joined #openstack-infra12:43
*** amoralej|lunch is now known as amoralej12:43
*** sree has joined #openstack-infra12:43
*** dprince has joined #openstack-infra12:44
*** sree_ has joined #openstack-infra12:44
*** sree_ is now known as Guest1842612:45
openstackgerritMerged openstack-infra/tripleo-ci master: Use infra proxy server for trunk.r.o in delorean-deps  https://review.openstack.org/50888412:45
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Switch service projects to not publish to pypi  https://review.openstack.org/51536612:46
*** vsaienk0 has quit IRC12:47
dsarielfrickler, thanks for the helpful reply! :q12:47
*** sree has quit IRC12:48
*** gcb has quit IRC12:48
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Switch service projects to not publish to pypi  https://review.openstack.org/51536612:49
*** bh526r has joined #openstack-infra12:49
dsariel:q12:50
fricklerdmsimard: I wondered about that, too, but it seems its mostly jobs referencing the playbooks directly and I didn't see how one could handle the condition there12:52
dmsimardfrickler: in the roles' tasks/main.yaml, check for zuul_success12:53
dsarielfrickler, one more question if you don't mind openstack-zuul-jobs/playbooks/legacy/ have definitions of what is performed by those gates. is there somewhere a definition which gates are obligatory to pass in order to have +1 from gerrit?12:53
*** baoli has joined #openstack-infra12:54
*** LindaWang has joined #openstack-infra12:54
*** shardy has joined #openstack-infra12:57
*** bobh has joined #openstack-infra12:58
openstackgerritRicardo Noriega proposed openstack-infra/project-config master: Add new repo for Neutron L2GW Tempest Plugin  https://review.openstack.org/50982312:59
*** Goneri has quit IRC12:59
*** apetrich has joined #openstack-infra13:00
*** vsaienk0 has joined #openstack-infra13:00
*** gongysh has quit IRC13:00
*** rhallisey has joined #openstack-infra13:00
mkolesnihey, someone know why the gate jobs for py27/py35 are sporadically failing?13:01
mkolesniits really interfering with getting changes merged13:01
*** camunoz has joined #openstack-infra13:02
fricklerdmsimard: oh, you mean within the write-root-marker and upload-afs roles. well, I was assuming that they were too generic to be coupled with zuul_success unconditionally13:02
fricklermkolesni: do you have links to failures? I'm not aware of any global issues regarding these13:03
*** e0ne has joined #openstack-infra13:03
dmsimardfrickler: yeah, I don't have a strong opinion13:03
mkolesnifrickler, this one failed on POST_FAILURE even though py27 itself ran - http://logs.openstack.org/31/514931/2/gate/openstack-tox-py27/2f78d5b/13:03
*** yolanda has quit IRC13:04
mkolesnifrickler, on recheck, this one will fail on py35 but still hanging right now - http://zuulv3.openstack.org/static/stream.html?uuid=bcfe8e0605e8405a92ec00874a6eee7a&logfile=console.log13:04
fricklerdsariel: it is the set of all checks that are applicable for a patch, except those marked with "voting: false"13:04
AJaegerdmsimard: if you teach me how to do it differently ... I'm an ansible novice and need your help. Feel free to take over the chagne or propose a superior one separately13:05
* AJaeger is busy the next 4 hours13:06
fricklermkolesni: this looks like there might be networking issues in one of our clouds, I guess some infra-root will have to look into this13:06
mkolesnifrickler, ok do you know who should i contact?13:07
* fungi will see if he can track down the intermittent network connectivity issues to a particular provider/region13:08
dsarielfrickler, maybe I will rephrase my question: how is it ensured that gerrit will not vote +1 until all non-voting gates are passed?13:08
fricklerfungi: citycloud-kna1 was the one with ssh failure here http://logs.openstack.org/31/514931/2/gate/openstack-tox-py27/2f78d5b/job-output.txt.gz#_2017-10-26_08_40_47_77742213:09
*** camunoz has quit IRC13:09
*** camunoz has joined #openstack-infra13:10
fricklermkolesni: fungi is one of them, so it is already being handled13:10
*** mriedem has joined #openstack-infra13:11
fricklerdsariel: iiuc not gerrit is voting, but zuul. and the definition of how and when zuul does this is here https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml13:12
*** Goneri has joined #openstack-infra13:12
mkolesnifrickler, ok thanks :)13:12
*** hemna_ has joined #openstack-infra13:14
*** lindycoder has joined #openstack-infra13:15
*** lindycoder has quit IRC13:16
*** jaosorior has quit IRC13:16
fungifrickler: yeah, i think we only recently reenabled some citycloud regions after having disabled them a while back for network instability. the old issues there may still be present13:16
*** ykarel is now known as ykarel|away13:17
*** lindycoder has joined #openstack-infra13:18
fungithough looks like that kna1 region has been enabled for a while (months at least)13:20
*** eharney has joined #openstack-infra13:20
*** sbezverk has joined #openstack-infra13:20
*** vsaienk0 has quit IRC13:20
*** ykarel|away has quit IRC13:21
*** vsaienk0 has joined #openstack-infra13:23
dsarielfrickler, thanks!13:23
fungicitycloud hasn't reached out to notify us of any issues there, though that's fairly typical for public cloud providers we work with so not especially surprising13:23
*** mat128 has joined #openstack-infra13:24
*** yamamoto has quit IRC13:26
openstackgerritRicardo Noriega proposed openstack-infra/project-config master: Add new repo for Neutron L2GW Tempest Plugin  https://review.openstack.org/51540813:26
*** dhajare|mtg has quit IRC13:27
*** smatzek has quit IRC13:34
EmilienMinfra-root: outstanding request to promote https://review.openstack.org/#/c/515388/ in the gate, that would help us to sort things out in our gate - thank very much13:34
*** rosmaita has joined #openstack-infra13:35
*** spzala has joined #openstack-infra13:35
*** gouthamr has joined #openstack-infra13:36
openstackgerritRicardo Noriega proposed openstack-infra/project-config master: Add new repo for Neutron L2GW Tempest Plugin  https://review.openstack.org/51540813:38
*** martinkopec has quit IRC13:39
*** felipemonteiro_ has joined #openstack-infra13:40
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Revert "Disable voting on 2 multinode tripleo jobs"  https://review.openstack.org/51541113:40
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Revert "Disable voting on 2 multinode tripleo jobs"  https://review.openstack.org/51541113:41
fricklerfungi: openstack-health data seems to be stale, last updated 7am yesterday, makes it difficult to see whether we might have a larger issue currently13:41
fungifrickler: yep, i'm at a more effective computer now and starting to see if i can spot some patterns the old-fashioned way13:42
EmilienMproject-config core: please review https://review.openstack.org/515411 when you can so we can re-enable voting on 2 important jobs. Thanks!13:43
*** pgadiya has quit IRC13:45
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: Remove usage of 'tripleo' gate queue  https://review.openstack.org/51507013:46
*** dosaboy has quit IRC13:48
*** annp has joined #openstack-infra13:50
*** e0ne has quit IRC13:53
*** ykarel|away has joined #openstack-infra13:53
*** jaosorior has joined #openstack-infra13:53
*** spzala has quit IRC13:54
fungiEmilienM: infra-root: i've enqueued tripleo-quickstart-extras patchset 515388,2 into the gate pipeline and promoted it to the head of the tripleo change queue13:54
*** hongbin has joined #openstack-infra13:56
fungimkolesni: frickler: scanning the running jobs right now, i'm not seeing evidence of a spate of job failures, so maybe this was a temporary condition which has since resolved itself? or maybe the frequency of incidence is just really low?13:56
*** ihrachys has joined #openstack-infra13:57
mkolesnifungi, i dont know its happening since yesterday but seems to be hitting different jobs13:57
EmilienMfungi: thanks very much13:58
mkolesnifungi, the problem is the ci is broken because of some external changes which im trying to fix with this patch to unblock all others but these failures have stopped me from merging it for a day and a half now..13:58
*** smatzek has joined #openstack-infra13:59
*** ramishra has quit IRC14:00
*** dosaboy has joined #openstack-infra14:00
*** smatzek has quit IRC14:00
*** smatzek has joined #openstack-infra14:01
*** smatzek has quit IRC14:01
*** smatzek has joined #openstack-infra14:01
prometheanfireAJaeger: ya, centralized project config should enforce some tests, like check-reqs14:02
*** ramishra has joined #openstack-infra14:02
fungimkolesni: is it at all possible that something about the unit tests actually impacts networking on the node where the tests run? are these tests fully mocked or do they interact with the kernel/perform actual networking tasks?14:06
mkolesnifungi, no theyre fully mocked, its an sdn controller and its not installed during these tests14:06
*** spzala has joined #openstack-infra14:06
fungimkolesni: because it really looks in the currently timing-out job like the executor lost connectivity to the test node entirely14:06
mkolesnifungi, i gtg but please talk to mpeterson hes working with me on this14:07
fungiokay, just making sure. in that case we may have nova instances dropping off the network at random, but normally zuul should retry such jobs14:07
*** esberglu has joined #openstack-infra14:07
*** jbadiapa has quit IRC14:07
* mpeterson is catching up with what you were talking about 14:08
*** rcernin has joined #openstack-infra14:08
fungimpeterson: currently failing job's console stream is at http://zuulv3.openstack.org/static/stream.html?uuid=bcfe8e0605e8405a92ec00874a6eee7a&logfile=console.log14:08
*** pblaho has quit IRC14:08
mpetersonfungi: yeah, it's not the first time we see a job stuck at that phase14:09
fungipeering closer, looks like it the executor lost connectivity to the test node during a pre phase task, and each subsequent task is trying for 30 minutes before timing out. trying to track down potential connectivity issues in one or more of our donor providers' regions14:09
mpetersonyes, I've seen that as well. Where the `run` can't run because `pre` phases timeout or have a network issue14:11
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Remove unreachable code  https://review.openstack.org/50613714:12
mpetersonfungi: is there a way to detect those issues though and retry the checks? otherwise it adds like long delays14:12
fungimpeterson: well, in this case i fully expect zuul to reschedule this particular build automatically14:13
*** marst has joined #openstack-infra14:13
fungiif it does not, that will significantly alter my expectations14:13
mpetersonfungi: I think I've never seen zuul reschedule builds on this kind of issues before. I might not have observed too closely though, as you know the ins and outs :)14:14
*** dklyle has quit IRC14:15
*** Guest31954 has quit IRC14:15
fungimpeterson: you would only notice if watching the status page, you'd see a job with some percentage of completion go back to queued temporarily and then start its completion bar at 0 again once it got scheduled to a replacement node14:15
fungithough it might be nice if it didn't try each and every post task for 30 minutes before giving up (though i don't know how we'd actually go about correctly detecting that we should do that in cases like this one where it's completely lost connectivity to the test node)14:16
fungii expect it to tick over to the next task any second now, it's been on the current one for 30 minutes14:17
fungiand there it goes14:17
*** catintheroof has joined #openstack-infra14:17
mpetersonlike a swiss clock14:18
fungias long as the rest of these are localhost tasks it should hopefully wrap up14:18
*** armax has joined #openstack-infra14:18
fungiokay, logs for that are now at http://logs.openstack.org/31/514931/2/gate/openstack-tox-py35/bcfe8e0 but hopefully the build got rescheduled to a new node automagically and 514931,2 is still in the gate... checking14:20
mpetersonyes14:20
mpetersonas you expected14:20
*** rbrndt_ has joined #openstack-infra14:21
*** Swami has joined #openstack-infra14:22
*** vsaienk0 has quit IRC14:23
mpetersonthings seem to be going well now14:24
fungithat's good at least14:24
fungias for the build that was timing out, that one seems to have run in inap14:25
*** yamamoto has joined #openstack-infra14:26
mpetersondamn14:27
mpetersonfailed14:27
fungithat was quick14:28
mpetersonthe job didn't fail yet, but I can see that the tox command failed installing packages14:28
*** eumel8 has quit IRC14:28
fungigot that stream url handy?14:29
mpetersonfungi: http://zuulv3.openstack.org/static/stream.html?uuid=2601c9f64c224b00a9b64b5b61917916&logfile=console.log14:29
fungithere's a failure-looking message pip is emitting under tox when installing from local source14:29
mpetersonbut it's running the tests now mmm weird14:29
fungiscrolling back in the stream is slow14:30
fungiso much output from the tests14:30
mpetersonmmm I'll get you the line that I meant once I can read the logs14:30
fungiyeah, i have a feeling it was the one where pip complains because it doesn't find an origin remote in the git repo. if so, that's scary-looking yet entirely benign14:31
mpetersonI actually wrote me a note early today to silence all this logging14:31
*** yamamoto has quit IRC14:32
fungii sort of wonder if the higher incidence of post failures for those unit tests is due to the massive console log14:32
fungithough that doesn't explain the node in inap we just saw disappear out from under zuul in the pre phase14:32
mpetersonthat could actually be an issue?14:32
mpetersonyeah, specially because that inap node never failed in a different phase14:33
fungimpeterson: well, we have to collect logs over the network, and network interactions are the flakiest part of this entire process because, you know, clouds14:33
mpetersonyeah, wth, who are the crazy guys behind those...14:33
mpetersonfungi: failed again in the log collection14:34
fungi;)14:34
fungi:(14:34
mpetersonperhaps it's bening?14:34
mpetersonbecause a merged message was issued14:34
fungiwell, now that we have logs, i can check and tell you14:34
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524914:35
mpetersonokey, so let me find you those lines...14:35
fungii'm already digging around in the console log14:35
mpetersonfungi: one is this one http://logs.openstack.org/31/514931/2/check/openstack-tox-py35/336c1b8/job-output.txt.gz#_2017-10-26_10_54_23_15127314:35
mpetersonoh, okey14:35
fungi845kb compressed? fun14:35
mtreinishfungi, clarkb: I haven't seen any launchpad firehose events go by in a while (I was watching to check for something) can you check if the lpmqtt daemon is running14:36
mpetersonoh that's nothing... on the java part we usually get like 123M compressed14:36
fungimpeterson: the "falling back to uneditable format,Could not determine repository location" messages are indeed benign, a side effect of pip getting confused by the git repos we've pushed onto the test nodes since they don't have an origin remote14:37
mpetersonfungi: okey, and the skipping tests part is weird but is obviously related14:38
fungiit still installs them, as noted there, it just can't provide detailed git urls for them but we don't need it to anyway14:38
*** gordc has joined #openstack-infra14:38
fungimpeterson: the "py35: skipped tests" part is because the task that ran under wasn't the actual test run task but rather one named "tox-siblings : Run tox without tests"14:39
jeblair13:58 < tobiash> jkt: you probably will want to clone it from the merger and update (clone if the merger doesn't have it cached) it from upstream14:39
jeblair14:12 < openstackgerrit> Merged openstack-infra/nodepool feature/zuulv3: Remove unreachable code  https://review.openstack.org/50613714:39
jeblair14:35 < openstackgerrit> James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524914:39
jeblairderp sorry14:39
fungimtreinish: i'll take a look at the server now and see if it's still going14:39
fungimtreinish: lpmqtt   14396  0.1  1.3 283444 26972 ?        Ssl   2016 605:19 /usr/bin/python /usr/local/bin/lpmqtt /etc/lpmqtt.conf14:40
fungimtreinish: maybe we have some sort of e-mail delivery problem with it? when do you last remember seeing an lp event in the stream?14:40
mtreinishfungi: hmm, I wonder if the email stopped14:40
mtreinishfungi: oh, just saw a batch14:41
*** nikhil has joined #openstack-infra14:41
fungioh, good14:41
* fungi stands down14:41
mtreinishI think we might have too long of a retry window14:41
mtreinishand I'm just impatient14:41
mpetersonfungi: mmm there was an error in the stream that just doesn't appear here :/14:41
mtreinishfungi: I'll check the config in the puppet and maybe decrease the wait time a bit14:41
mtreinishfungi: oh, and more alarming is the warning on: http://status.openstack.org/openstack-health/#/14:42
*** martinkopec has joined #openstack-infra14:42
fungimtreinish: yeah, frickler pointed that out a little bit ago14:42
fungii wonder if something is awry with the subunit2sql workers14:43
*** csomerville has joined #openstack-infra14:43
mtreinishfungi: yeah, I would assume it's something on the workers14:43
fungimtreinish: i'm tailing /var/log/subunit2sql/subunit-worker-A-debug.log but... nothing showing up yet14:44
*** scottda_ has quit IRC14:44
fungiand i see why14:44
fungilooks like it's not running14:44
*** cshastri has quit IRC14:45
*** ianychoi_ has joined #openstack-infra14:45
mtreinishoh, did it crash?14:45
*** martinkopec has quit IRC14:45
fungithough i also may be looking at the wrong logfiles... these were last modified in may14:45
mtreinishhah14:45
*** cody-somerville has quit IRC14:46
fungiwhere is it supposed to log?14:46
openstackgerritMerged openstack-infra/system-config master: Add notes on Release Managers abandon permissions  https://review.openstack.org/51486314:46
*** vsaienk0 has joined #openstack-infra14:47
mtreinishfungi: looking at the puppet that looks like the right place14:47
mpetersonwhat is MERGER_FAILURE?14:48
mtreinishfungi: /var/log/subunit2sql/subunit-worker<%= suffix %>-debug.log14:48
*** ianychoi has quit IRC14:48
mtreinishis what's in the puppet template for the init file14:48
fungimtreinish: yeah, jenkins-subunit-worker-A initscript does say -d /var/log/subunit2sql/subunit-worker-A-debug.log14:48
*** david-lyle has joined #openstack-infra14:48
mtreinishfungi: are you on subunit-worker 01 or 02?14:48
fungi01. didn't realize we have an 0214:48
fungior, rather, forgot14:48
mtreinishwe stopped using 01 because it's precise iirc14:49
jeblairmpeterson: can you point me at the change where you saw that?14:49
fungiokay, now tailing the log on the active server, and it is indeed running14:49
mtreinishand there was an issue with ssl on it14:49
fungi"ValueError: Python 2.7.9 and 3.2 are the minimum supported versions for TLS."14:49
mpetersonjeblair: https://review.openstack.org/#/c/515057/214:49
fungimtreinish: methinks we got a new paho-mqtt14:49
mtreinishfungi: oh, fun14:50
jeblairnew things are always better!14:50
AJaegerjeblair: unless the break everything ;)14:50
mpetersonjeblair: there are two errors there actually the MERGER_FAILURE and the first job of that batch14:50
fungimtreinish: http://paste.openstack.org/show/624721/14:50
*** armax has quit IRC14:50
jeblairmpeterson: indeed.  i bet neutron really does have a master branch.  :)14:51
*** burgerk has joined #openstack-infra14:51
mtreinishfungi: hmm, the last paho release was 10/9: https://pypi.python.org/pypi/paho-mqtt14:51
*** felipemonteiro__ has joined #openstack-infra14:52
fungimtreinish: and pip freeze says that's what we've got14:52
fungisubunit-gearman-worker.py last restarted Oct17 too according to the process list14:53
fungiso you would think we'd have no data since at least 9 days if this was the cause14:53
*** felipemonteiro_ has quit IRC14:55
mtreinishfungi: hmm, well it says the last data is from the 25th14:55
fungiright, which is why i think this must not be (completely at least) breaking it14:55
mpetersonjeblair: I guess I need to ignore both those errors and see what the recheck does then?14:55
jeblairmpeterson: i'm looking into them14:55
mpetersongood, thanks14:56
mtreinishfungi: are we running multiple worker processes on subunit-worker02?14:56
mtreinishI can't remember if we ever got that working14:56
funginope, just the one14:57
mtreinishfungi: could we have been running jobs in the future? (time sync issue)14:57
fungia remote possibility i suppose14:57
mtreinishthat timestamp in the warning comes from the date inserted into the db which it gets from the subunit iirc14:58
fungialso i don't think it's a gearman restart, since the scheduler was restarted 6+ hours after the data stops14:58
jeblairmtreinish: what warning?  i don't see anything on o-h14:58
fungijeblair: tan box near the top: " The last data in the subunit2sql database, from "2017-10-25T08:17:15.000Z", is >1 day old. There might be an issue with result collection."14:58
jeblairnope.  don't see it.14:59
*** ykarel|away has quit IRC14:59
fungiweird14:59
mtreinishjeblair: https://i.imgur.com/WQewf57.png14:59
openstackgerritBrian Haley proposed openstack-infra/project-config master: Update neutron-failure-rate grafana dashboard  https://review.openstack.org/51543914:59
*** xarses has joined #openstack-infra14:59
jeblairmtreinish: https://screenshots.firefox.com/lm7FkiWlAJozvQUE/status.openstack.org15:00
*** slaweq_ has quit IRC15:00
mtreinishjeblair: timezones? I might have screwed up the JS and its comparing localtime to utc15:00
fungiahh15:01
fungii would buy that15:01
jeblairah15:01
fungiwell, except that 08:17z is longer ago than the offset between us and jeblair15:02
*** spectr has quit IRC15:02
mtreinishheh, I'm in prague right now so all my TZs are screwed up :)15:02
fungiahh, you mean it's earluer than 08:17pdt15:02
*** jaosorior has quit IRC15:02
fungiwhich it technically is for 15 more minutes15:02
*** baoli has quit IRC15:02
fungiso, yeah, maybe that explains the missing message at least15:02
*** gordc has left #openstack-infra15:03
jeblairi'll check in 13m.15:04
AJaegermtreinish: prague? That has a very nice timezone, keep it ;)15:04
*** witek has quit IRC15:04
*** rcernin has quit IRC15:04
*** baoli has joined #openstack-infra15:05
*** witek has joined #openstack-infra15:06
AJaegerjeblair, mordred, is https://review.openstack.org/#/c/515365/ the right approach - or should I do those changes directly on the roles involved?15:06
mtreinishjeblair, fungi: https://github.com/openstack/openstack-health/blob/master/app/js/directives/fresh-check.js#L19-L28 I don't see anything about tz in there15:06
mtreinishAJaeger: heh, yeah for open source summit europe. It's been a nice trip so far15:07
mtreinishfungi: so should I push a patch to pin paho, or should we spin up a new worker on a newer host os to get python >2.7.9?15:07
jeblairmpeterson: the executor that ran that job ran out of space.  that shouldn't happen very often, though it may happen occasionally until the container/image projects change their workflow so we're not using the executor as an intermediary.  a recheck should be good.15:08
mpetersonjeblair: thank you!15:10
mpetersonfungi: also you!15:10
mpetersonyou guys are great15:10
jeblairthough also, we may have extra directories laying around from some debugging work i did yesterday taking up space15:10
jeblairi'll work on cleaning those up to reduce the chances of this15:10
fungimtreinish: i can try to replace the old 01 with xenial since it's unused anyway15:10
fungiwe don't still need anything off subunit-worker01's filesystem i'm assuming? it's just been sitting there idle since may-ish15:11
mtreinishfungi: I don't think so15:11
fungii'll try to get started on it as soon as tc office hour winds up15:12
fungior winds down. whichever15:12
jeblairfungi: find /var/lib/zuul/builds -maxdepth 1 -cmin +480 -exec rm -fr {} \;15:12
fricklerfungi: seems openstack-health is indeed still processing data, just being well behind. the timestamp for "latest data" seems to be continually increasing, just slowly15:12
jeblairfungi: how's that look for deleting all build dirs older than 8 hours?15:12
*** annp has quit IRC15:13
*** baoli has quit IRC15:13
fungijeblair: any reason not to use -type d?15:14
fungiworried the timestamps on the directories may be older than contents several levels deeper?15:14
fungijeblair: maybe use -type f with -delete and then do a separate pass looking for -type d -empty?15:15
*** e0ne has joined #openstack-infra15:15
fungioh, right, with -maxdepth 1 you presumably only get directories anyway15:16
fungijeblair: yeah, i'm done talking to myself now. lgtm ;)15:16
jeblair\o/15:17
*** Guest18426 has quit IRC15:17
*** sree has joined #openstack-infra15:18
*** baoli has joined #openstack-infra15:18
mtreinishfungi, frickler: hmm yeah: http://paste.openstack.org/show/624727/ there are some new ones in there from what o-h was previously reporting15:18
*** catintheroof has quit IRC15:20
mtreinishfungi: is the worker really stuck there in a loop reporting the paho issue, or could it sporadically inserting data between the mqtt errors?15:20
*** catintheroof has joined #openstack-infra15:20
fungimtreinish: there's lots of other logging going on about gearman work requests/notifications15:20
fungiso it does appear to at least be trying to do stuff. this error message _may_ simply be benign if we don't have it configured to actually do ssl/tls15:21
fungioh, or, yeah... it's just failing to report to the firehose15:22
fungibut possible these mqtt reporting failures are also bogging it down, no idea really15:22
*** sree has quit IRC15:22
mtreinishfungi: if there are log messages like: "Converting Subunit V2 stream: %s to SQL" that means it's running the db insert stuff15:23
fricklercan this also be related to e-r being backed up because it is still waiting for some console.log to show up? or is that a completely different beast?15:23
mtreinishthe worker is configured to handle exceptions in the outerloop by logging them but not exit on it15:24
*** sree has joined #openstack-infra15:24
*** catintheroof has quit IRC15:25
fungi2017-10-26 15:23:32,825 Converting Subunit V2 stream: <_io.BytesIO object at 0x7f7a42ea9770> to SQL15:25
fungiso, yeah15:25
mtreinishfrickler: that's likely related to zuul v3 changing the filename of the output15:26
fungilogging into the rackspace dashboard is always a fun experience. never fails to have an interesting trouble ticket or several waiting for us15:27
mtreinishfungi: hah, nice15:27
mtreinishfrickler: although the logstash workers will likely be hit by the same paho mqtt issue on restart15:27
mtreinishas will any of our python things running mqtt on trusty15:28
*** yamamoto has joined #openstack-infra15:28
*** dhinesh has joined #openstack-infra15:28
*** sree has quit IRC15:29
*** ykarel has joined #openstack-infra15:29
*** ijw has joined #openstack-infra15:30
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Make tripleo undercloud-containers voting  https://review.openstack.org/51316315:31
*** ijw has quit IRC15:32
fungi#status log Provider maintenance is scheduled for 2017-10-30 between 06:00-09:00 UTC which may result in up to a 5 minute connectivity outage for the production Gerrit server's Trove database instance15:32
openstackstatusfungi: finished logging15:32
fungiinfra-root: ^15:32
dmsimardfungi: should we pre-emptively shut down services before the maintenance window in order to avoid corruption or otherwise an unclean crash ?15:33
*** yamamoto has quit IRC15:33
dmsimardfungi: or is it just network ?15:34
clarkbfwiw gerrit survived the unclean shutdown of live migration gone wrong not too long ago15:34
clarkbit will likely just 500 error while it can't hit the database15:35
clarkband be fine afterwards15:35
*** ykarel has quit IRC15:35
clarkband the db will be backed up at 0000UTC15:35
*** fultonj_ has joined #openstack-infra15:35
*** fultonj_ has quit IRC15:36
clarkbfrickler: ya the elasticsearch indexing is almost completely independent. And really it is indexing we are just looking for the wrong things in e-r15:36
clarkbdmsimard: re ^ did a chance get pushed to support a config switch?15:36
dmsimardclarkb: I sent a patchset with another approach15:37
dmsimardwhich seemed ok to me15:37
dmsimardsec15:37
mtreinishclarkb: iirc it's already configurable we should just need to update the puppet15:37
clarkbdmsimard: we should probably have mtreinish review it if mtreinish isn't too busy in prague :)15:37
fungidmsimard: yeah, i would mostly just make sure we keep an eye on things... maybe ianw can double-check things are still in sane condition before he hits eod on the 30th15:37
clarkbmtreinish: the problem is that console.html and the "last" log line were hardcoded iirc15:37
dmsimardclarkb: https://review.openstack.org/#/c/509313/ oh, frickler added some comments -- I missed that15:37
dmsimardmtreinish: ^ see that review15:37
mtreinishclarkb: I think that was one of the things ramy did for his hpe downstream e-r usage15:38
*** fultonj has quit IRC15:38
dmsimardclarkb: the fact that gerrit survived the unclean shutdown doesn't mean it will every time :p15:38
*** baoli has quit IRC15:38
*** fultonj has joined #openstack-infra15:38
*** baoli has joined #openstack-infra15:39
clarkbdmsimard: this is true15:39
fungidmsimard: in this case it's not the gerrit server going offline, it's the trove database instance it talks to15:39
clarkbmtreinish: I know the base url and stuff was configurable15:39
clarkbmtreinish: in any case https://review.openstack.org/#/c/509313/ is dmsimard's change to address some of this15:39
dmsimardI'll check out frickler's comments and address them later today, need to wrap up something first15:40
clarkbdmsimard: thanks15:40
mtreinishclarkb: ok, I'll try to review it tomorrow15:40
mtreinishor late tonight depending on my mood then :)15:41
clarkbmtreinish: just drink the lager and not the pils they say its lighter ;)15:41
fungilighter colored? lighter bodied? lighter in abv?15:42
jeblairless dense?15:42
fungithese things matter to fungi15:42
*** witek has quit IRC15:42
clarkbfungi: the four options in every place were "light pils wheat dark"15:42
clarkbyou didn't get a choice of brewery or whatever15:43
clarkbthe light and the pils were pretty indistinguishable to me15:43
fungijeblair: lower specific gravity i suppose15:43
*** baoli has quit IRC15:43
clarkbya I'm guessing its basically the same beer as the pils just scaled down a bit15:43
*** gyee has joined #openstack-infra15:43
* fungi will eventually find his way to prague15:44
clarkbI kind of enjoy a simple beer menu like that, makes ordering easy. But the equivalent here is bud light, bud, blue moon, guiness and not sure I could live in that world15:45
jeblairto be fair, you're very likely to get 'bud' there too.  but the real bud.15:46
*** andreas_s has quit IRC15:46
*** andreas_s has joined #openstack-infra15:47
*** makowals has quit IRC15:47
*** witek has joined #openstack-infra15:47
mtreinishclarkb: I had "Dalešické májové" the other day. I don't know whether it's lager or pils (or something else) but I liked it15:48
*** vsaienk0 has quit IRC15:48
mtreinishI also have no idea how to pronounce it15:49
*** felipemonteiro_ has joined #openstack-infra15:49
*** baoli has joined #openstack-infra15:49
fungijeblair: itym "the real kind"15:50
*** catintheroof has joined #openstack-infra15:51
*** andreas_s has quit IRC15:51
jeblairindeed15:51
*** felipemonteiro__ has quit IRC15:52
*** andreas_s has joined #openstack-infra15:53
*** baoli has quit IRC15:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add implied branch matchers on 'master'  https://review.openstack.org/51445915:54
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch to late-binding inheritance  https://review.openstack.org/51135215:54
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524915:54
*** hemna_ has quit IRC15:54
*** catintheroof has quit IRC15:55
clarkbdmsimard: I left a comment on that change too. I think that approach should work though15:56
*** rkukura has joined #openstack-infra15:58
*** vsaienk0 has joined #openstack-infra15:58
*** Apoorva has joined #openstack-infra15:59
*** catintheroof has joined #openstack-infra16:01
*** andreas_s has quit IRC16:03
*** Hal has joined #openstack-infra16:03
*** andreas_s has joined #openstack-infra16:03
*** Hal is now known as Guest8718516:04
fungiinfra-root: the fix to get e-mail reporting working again merged earlier today; when would be a good time for a scheduler restart to pick that up?16:07
*** kiennt26 has joined #openstack-infra16:07
*** baoli has joined #openstack-infra16:08
clarkbfungi: I'm good with whenever, probably depends more on release team than anything else? jeblair did just +3 a change of mine that will allow us to configure cgit urls16:08
jeblairfungi: any time i think16:08
clarkblet me push up a config change now so that we can get both things in at the same time16:09
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Make the git web url a template  https://review.openstack.org/51516816:09
*** andreas_s has quit IRC16:09
*** vsaienk0 has quit IRC16:09
*** LindaWang has quit IRC16:09
fungicool, i'll see when the release team has achieved quiescence16:09
clarkbfungi: also all three of your pre release signed tarballs fix changes merged I think we just need to confirm they work as expected then can mark them off the list16:09
fungioh, yeah i believe they confirmed that yesterday while i was on the road. i'll double-check that too thanks16:10
*** andreas_s has joined #openstack-infra16:10
*** armax has joined #openstack-infra16:10
*** dprince has quit IRC16:12
pabelangerable to support a scheduler restart too16:12
pabelangeralso, I think ianw finished the rebuild of zm01 to zm04 servers. Going to see if he started them or not16:13
*** sree has joined #openstack-infra16:14
openstackgerritClark Boylan proposed openstack-infra/system-config master: Set cgit url in zuul connection config  https://review.openstack.org/51546016:15
clarkbthere is the config change to use cgit urls16:15
clarkbif we can bundle that into the restart that would be excellent16:15
pabelangerHmm, for some reason we didn't create swap again on zm01-zm0416:16
*** slaweq has joined #openstack-infra16:16
*** baoli has quit IRC16:16
pabelangerI'm going to work on that now16:16
*** derekh has quit IRC16:17
*** psachin has quit IRC16:17
*** lucasagomes is now known as lucas-afk16:17
jeblairthe cgit url change is safe to approve now -- it's okay to add unused things to the config file16:17
jeblairi have +2d16:18
fungimy attempt at building the replacement subunit-worker01 failed when trying to gather facts during the "remove cloud-init" task. weird... Could not detect which package manager to use."16:18
*** andreas_s has quit IRC16:19
openstackgerritMerged openstack-infra/project-config master: Revert "Disable voting on 2 multinode tripleo jobs"  https://review.openstack.org/51541116:19
*** sree has quit IRC16:20
clarkbfor ansible file resources that are state directory will it create multiple directories if parent dirs are not created yet? or do we have to sequentially list each directory?16:20
fungirelease team is cool with us restarting zuul any time now (and excited to have their e-mail error reporting return)16:21
*** baoli has joined #openstack-infra16:21
*** baoli has quit IRC16:22
*** mummer has joined #openstack-infra16:23
clarkb"all immediate subdirectories will be created if they do not exist" <- I'm not sure I understand what that means16:24
clarkbdoes that mean just one level?16:24
clarkbor anything in the path?16:24
* clarkb goes with it16:24
*** andreas_s has joined #openstack-infra16:24
fungihuh, trying to launch the new subunit-worker01 again, same failure gathering facts on the same cloud-init removal task16:25
dmsimardclarkb: needs recurse: true16:25
fungii'll try running with ansible from a virtualenv instead of the one we have globally-installed and see if that works any better16:25
dmsimardclarkb: file: path: /some/deep/path state: directory recurse: yes16:26
openstackgerritMartin André proposed openstack-infra/tripleo-ci master: Use local cirros image in tempest  https://review.openstack.org/51546216:26
clarkbdmsimard: thanks16:26
fungiugh, i think the launch progressed further by using a virtualenv'd install of ansible i found in ~root, but it's failing to get the paho-mqtt library installed on the server now. rerunning with --keep to see if i can figure out from its syslog what transpires16:30
*** yamamoto has joined #openstack-infra16:30
pabelangerokay, zm01-zm04 have swap again, it looks like ianw had an old version of make_swap.sh when he created new servers16:30
*** ccamacho has left #openstack-infra16:31
openstackgerritClark Boylan proposed openstack-infra/openstack-zuul-jobs master: Create dir needed to publish gerrit plugins  https://review.openstack.org/51546316:31
clarkbdmsimard: like that ^?16:31
*** baoli has joined #openstack-infra16:32
*** iyamahat has joined #openstack-infra16:32
*** bhavik1 has joined #openstack-infra16:33
dmsimardclarkb: yes16:34
dmsimardclarkb: that allows you to create a nested tree, but also to do things like set permissions and stuff.16:34
*** Goneri has quit IRC16:34
*** yamamoto has quit IRC16:34
fungiargh. --keep doesn't help because the manifest hasn't gotten far enough to add our admin shell accounts yet :/16:35
*** andreas_s has quit IRC16:36
clarkbfungi: I think you can use the ssh key launch uses in that case16:36
clarkbto ssh as root?16:36
fungioh, as root from the puppetmaster? yeah maybe16:36
funginope, prompting for root's password now16:37
fungidoes the launch script use a different ssh key than the normal root ssh key on the puppetmaster?16:37
clarkbya it generates one iirc16:38
fungii see some files in .tmp which end in id_rsa so maybe16:38
clarkbbut I think if you --keep it doesn't delete it16:38
fungier, in /tmp16:38
fungiooh, those uuids are the nova instance ids. perfect16:38
fungicool, that gets me in. thanks for the idea clarkb16:39
fungioh, i think the log entry about the missing paho-mqtt lib is just a warning16:41
*** rossella_s has quit IRC16:42
fungiit's still failing on "TASK [set_hostname : Remove cloud-init]"16:42
fungifatal: [subunit-worker01.openstack.org]: FAILED! => {"changed": false, "failed" true, "msg": "Could not detect which package manager to use. Try gathering facts or setting the \"use\" option."}16:42
*** e0ne has quit IRC16:43
fungiany clue what could cause that?>16:43
*** Goneri has joined #openstack-infra16:43
*** e0ne has joined #openstack-infra16:44
clarkbfungi: I think we likely used the generic package module in ansible rather than package manager specific modules to remove that package so ansible needs facts to switch automatically on the right one16:44
openstackgerritBrian Haley proposed openstack-infra/project-config master: Fix Grafana periodic counters for neutron-lib dashboard  https://review.openstack.org/51480116:44
clarkbfungi: likely just needs to gather facts on that playbook16:44
*** e0ne has quit IRC16:44
*** baoli has quit IRC16:44
fungiseems like that would be impacting all our server launching though16:44
pabelangerwhich playbook is that?16:44
*** ijw has joined #openstack-infra16:44
clarkbpabelanger: the launch one I think16:45
clarkbfungi: its a fairly new change (I didn't realize we had merged it even)16:45
*** baoli has joined #openstack-infra16:45
pabelangerah, puppetmaster.o.o16:45
*** rkukura has quit IRC16:45
pabelangermaybe we should setup fact cache and use smart facts there16:45
*** Swami has quit IRC16:45
*** e0ne has joined #openstack-infra16:45
clarkbpabelanger: I'm not sure that would change anything since this is fire and forget launch node right?16:46
fungiokay, so this being an extremely recent regression makes sense i suppose16:46
*** e0ne has quit IRC16:46
clarkbfungi: ya16:46
*** cody-somerville has joined #openstack-infra16:46
fungilike in between when ianw finished launching the replacement zuul mergers and now16:46
pabelangerclarkb: Oh, this is launch-node.py, now i understand16:47
*** andreas_s has joined #openstack-infra16:47
*** e0ne has joined #openstack-infra16:47
clarkbfungi: oh not sure if it is that recent16:47
pabelangerwhich verson of ansible are use using fungi ?16:47
*** e0ne has quit IRC16:47
clarkbsystem-config/playbooks/roles/set_hostname/tasks/main.yml is the playbook I think16:48
*** csomerville has quit IRC16:48
*** slaweq has quit IRC16:48
clarkbI don't see it overriding fact gathering16:48
*** slaweq has joined #openstack-infra16:48
rajinirDoes recheck still work for Zuul Gate upstream?16:48
*** bhavik1 has quit IRC16:49
clarkbrajinir: should16:49
rajinir"reacheck" same as before?16:49
*** ijw has quit IRC16:49
fungipabelanger: i tried the one globally installed on puppetmaster, and then the one in the virtualenv in ~root, both with the same result16:49
*** baoli has quit IRC16:49
clarkbrajinir: yes "recheck" same as before16:49
rajinirclarkb: thanks16:51
fungipabelanger: so both 2.2.1.0 and 2.1.1.016:51
*** andreas_s has quit IRC16:51
pabelangerodd16:51
pabelangerianw launched 4 new servers yesterday, don't think he seen that issue16:51
pabelangerpossible he has a different ansible.cfg in is home directory that always gather facts, but not sure16:52
*** dprince has joined #openstack-infra16:52
clarkbgathering facts is the default though right?16:53
clarkbnothing seems to turn that off in the playbook16:53
*** slaweq has quit IRC16:53
clarkbor in the launch.py ansible invocation? maybe its in the /etc/ansible config?16:53
pabelangermaybe fungi has it disabled in ansible.cfg?16:53
openstackgerritBrian Haley proposed openstack-infra/project-config master: Fix Grafana neutron-lib dashboard  https://review.openstack.org/51480116:53
*** florianf has quit IRC16:53
pabelangerlocal ansible.cfg16:53
fungii don't have a local ansible.cfg afaik16:54
fungihere's what i'm doing (as a non-root user): http://paste.openstack.org/show/624732/16:54
pabelangeryah, looks correct16:55
pabelangerit is also possible ianw wasn't using latest master, as his version of make_swap.sh was out of date16:55
fungiyou're welcome to repeat the exercise if you like. unfortunately the fact that it spams teh entire openstack inventory on stdout makes it pretty impossible to find what transpired before the last few steps unless you have a really huge terminal buffer16:55
*** e0ne has joined #openstack-infra16:56
clarkbcloud init removal merged 10/916:56
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging  https://review.openstack.org/51547016:56
clarkbI think our use of ansible-playbook rather than ansible may be part of this?16:58
clarkbwe don't pass a hosts directive in our playbook16:58
*** sambetts is now known as sambetts|afk16:58
clarkboh because it is a role?16:58
* clarkb reads more16:59
*** jpena is now known as jpena|off16:59
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging  https://review.openstack.org/51547016:59
clarkbaha!16:59
clarkbsystem-config/playbooks/set_hostname.yml16:59
clarkbfungi: ^ that says gather facts no. I think you can just delete that line or comment it out and rerun and it should work16:59
clarkband if it does push a patch I guess17:00
fungiis the playbook shared by other stuff besides the launch script? wondering if that's otherwise useful, or whether we simply merged a completely untested change for our server launching17:00
*** slaweq has joined #openstack-infra17:00
clarkbgrep says we don't use it anywhere else17:01
fungistrange17:01
fungiwell, running on a separate clone of system-config in my homedir now with that line removed17:02
fungifingers crossed17:02
*** Swami has joined #openstack-infra17:02
*** claudiub|2 has quit IRC17:03
*** ralonsoh has quit IRC17:04
fungii'm going to enqueue 515460 into the gate so we don't have to wait for the check backlog to burn down before we can restart zuul with it in place17:04
clarkbsounds good17:05
*** dsariel has quit IRC17:06
*** andreas_s has joined #openstack-infra17:06
*** ijw has joined #openstack-infra17:07
pabelanger++17:08
fungii think the launch has made it past where it was erroring before, so that's a good sign17:08
*** baoli has joined #openstack-infra17:09
*** kjackal_ has quit IRC17:09
*** kjackal_ has joined #openstack-infra17:10
*** cody-somerville has quit IRC17:11
fungiwell, we made it to running puppet anyway17:11
*** hemna_ has joined #openstack-infra17:11
*** e0ne has quit IRC17:12
*** baoli has quit IRC17:12
*** kiennt26 has quit IRC17:13
funginow to figure out what's regressed in this manifest17:13
*** rwsu has quit IRC17:13
fungiahh, something's breaking with `pip install -q --upgrade subunit2sql`17:14
fungilooks like pip's having trouble building netifaces17:15
fungilikely missing a dev package or something17:15
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Gather facts in set_hostname playbook  https://review.openstack.org/51547817:16
*** andreas_s has quit IRC17:17
*** baoli has joined #openstack-infra17:17
*** jamesdenton has quit IRC17:17
*** jamesden_ has joined #openstack-infra17:17
*** jamesden_ is now known as jamesdenton17:18
*** rbrndt_ has quit IRC17:18
*** yamahata has joined #openstack-infra17:18
johnsomIs the current recommendation to remove the "queue: " configuration in our projects?  I thought I saw some talk of that here the other day.17:19
*** iyamahat has quit IRC17:19
*** Goneri has quit IRC17:20
fungijohnsom: i think the tripleo team (temporarily?) undid their shared change queue in an attempt to get changes for more of their different projects merging in parallel (at the risk of wedging some of them which are normally tested together)17:21
johnsomOk, if it's not a general thing, I will leave ours.  I don't think it's been a big problem for us.17:21
*** dtantsur is now known as dtantsur|afk17:22
*** trown is now known as trown|lunch17:22
*** slaweq has quit IRC17:22
*** Goneri has joined #openstack-infra17:22
pabelangerfungi: no, they just made jobs non-voting but have reverted that17:23
*** rkukura has joined #openstack-infra17:25
*** rwsu has joined #openstack-infra17:26
fungipabelanger: oh, i saw a change fly by earlier for removing the tripleo shared queue from zuul17:26
* fungi finds17:26
fungipabelanger: https://review.openstack.org/515070 but i guess it hasn't merged yet17:27
fungiand (justifiable) has opposition voiced on it17:27
fungier, justifiably17:27
*** panda|ruck is now known as panda|ruck|off17:30
*** yamamoto has joined #openstack-infra17:31
*** kjackal_ has quit IRC17:32
pabelangerfungi: thanks, commented too. And agree17:32
*** kjackal_ has joined #openstack-infra17:33
*** ijw has quit IRC17:33
*** yamamoto has quit IRC17:36
*** ijw has joined #openstack-infra17:36
*** andreas_s has joined #openstack-infra17:38
AJaegermordred, pabelanger, is https://review.openstack.org/#/c/515365/ the right approach - or should I do those changes directly on the roles involved?17:38
*** ijw has quit IRC17:40
*** andreas_s has quit IRC17:43
pabelangerAJaeger: left comments about create / delete afs token, we can likely skip that too17:44
openstackgerritMerged openstack-infra/system-config master: Set cgit url in zuul connection config  https://review.openstack.org/51546017:44
AJaegerpabelanger: yes, we could - is there a way to skip the whole section? Or do I need this for each one?17:45
*** tesseract has quit IRC17:45
pabelangerAJaeger: you'd need to add them for each. If we moved to include_role, we could use a block and single conditional17:46
*** amoralej is now known as amoralej|off17:46
AJaegerok17:46
fungiokay, 515460 merged a couple minutes ago, so getting prepped for scheduler restart now17:47
*** dhajare|mtg has joined #openstack-infra17:47
fungijeblair: do we have any custom patches that need to be cherry-picked in, or just a normal restart is expected to be fine at this point?17:48
*** sree has joined #openstack-infra17:48
fungiany patched libs i should check that haven't been upgraded to non-patched versions or anything like that?17:48
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524917:49
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add pragma directive  https://review.openstack.org/51548317:49
pabelangerAJaeger: I'm not sure if we'd add it to zuul or not, but one idea would be to have a post-run playbook on success flag, then we wouldn't need to sprinkle the conditions inside a playbook.17:49
jeblairfungi: the only thing is the relp patch i've been manually adding, but i think i can live without it for a bit, so feel free to just use system-installed zuul at this point.17:49
*** yamahata has quit IRC17:49
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging  https://review.openstack.org/51547017:49
pabelangerright now, I don't think you can write a playbook to have hosts: foo, when: something and skip all tasks / roles17:50
fungijeblair: great, thanks. grabbing check/gate contents and will then restart17:50
pabelangeryou'd just not run ansible-playbook on that host17:50
*** andreas_s has joined #openstack-infra17:50
jeblairfungi: mind restarting iptables while it's down?17:50
fungigladly!17:50
*** Apoorva has quit IRC17:51
fungiwhat's the service name for it? not iptables nor iptables-persistent i guess17:51
AJaegerpabelanger: ok, will continue with the way for now17:51
*** dhinesh has quit IRC17:52
clarkbfungi: I think nftables17:52
clarkbon xenial /me looks17:52
*** sree has quit IRC17:52
clarkbfungi: netfilter-persistent17:53
fungiperfect, thanks clarkb!17:53
clarkbI guess that is because iptables is a lie at this point and it is all netfilter under the covers?17:53
funginetfilter-persistent restarted, zuul-scheduler coming up now17:54
*** ccamacho has joined #openstack-infra17:55
fungianything in particular i should wait to see in the logs before i start reenqueuing jobs?17:55
*** andreas_s has quit IRC17:55
openstackgerritMichael Johnson proposed openstack-infra/project-config master: Remove migrated octavia zuul v3 jobs  https://review.openstack.org/51548617:55
clarkbmaybe just double check iptables -L -n shows the expected rules?17:55
fungii did, they're as expected17:56
clarkbcool17:56
SamYaplecan i get some love on https://review.openstack.org/#/c/512941/ and https://review.openstack.org/#/c/513591/ ? they have been sitting for a week17:56
jeblairfungi: the 5k cat runs will take a few minutes.  during that time, you can submit a zuul enqueue but it won't complete until they do17:56
clarkbas for the cgit thing we need post jobs to run to know that is working so nothing else to check on my end17:56
fungilooks like the scheduler is about done loading configuration maybe17:56
jeblairfungi: so it's okay to go ahead and starte re-enqueing any time, or you can wait until it's done config.17:56
fungiokay, cool17:56
*** andreas_s has joined #openstack-infra17:57
jeblair(just be aware if you start early, it's going to hang for a few mins anyway)17:57
fungiyeah, seems to be processing events already. i'll go ahead and reenqueue17:57
jlvillalDid something just happen to Zuul? The status page now shows 3 total jobs???17:57
jeblairjlvillal: yeah we're restarting it.  we're re-enqueing the old changes17:57
* jlvillal wonders if chat about re-enqueing is realted17:57
jeblairjlvillal: shouldn't need to do anything17:57
jlvillalrelated17:57
fungijlvillal: you'll see them back in the next few minutes17:57
jlvillaljeblair: fungi: Awesome!17:58
fungii saved the contents of check, gate and check-tripleo just before stopping and am putting them all back in now17:58
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: remove implied run  https://review.openstack.org/51548717:58
*** Apoorva has joined #openstack-infra17:58
*** slaweq has joined #openstack-infra18:00
*** tosky has quit IRC18:00
clarkbjeblair: re SamYaple's changes do you know if gerrit publishes a list of files on ref-updated events?18:01
*** slaweq has quit IRC18:01
*** andreas_s has quit IRC18:02
clarkblooks liek they aren't, are file inclusion/exclusion relevant in that case with zuul (it could look up the diff itself I guesss)18:03
*** dhajare|mtg has quit IRC18:03
SamYapleclarkb: oh thats a good question. i hadnt considered that18:05
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Copy Gerrit delay to GitHub driver  https://review.openstack.org/51522418:05
SamYapleclarkb: only to save infra resources. i really only need to trigger on upper-constraints.txt changes, but changing on all doesnt matter to me18:06
*** iyamahat has joined #openstack-infra18:06
clarkbSamYaple: I think it woudl require zuul to inspect the diff directly. With v2 I'm pretty sure it was only supported if the gerrit event had the data though18:06
SamYapleits in the post queue, so low priority and jeblair has some post queue optimizations18:06
jeblairclarkb: i believe you're correct18:06
jeblairi really want change-merged to have the merge commit sha :(18:06
sdaketonyb "This will NOT be a discussion on how we implement an LTS but the topic will certainly come up ;P" :-)18:07
fungi#status log zuul scheduler restarted and check/check-tripleo/gate pipeline contents successfully reenqueued18:07
openstackstatusfungi: finished logging18:07
jeblairfungi, pabelanger: do the firewalls on the new zm servers look right?18:08
*** rbrndt_ has joined #openstack-infra18:08
fungijeblair: the rules on zm01 appear a bit... minimalist18:08
*** eroux has joined #openstack-infra18:09
* SpamapS is sad that OpenStack is moving away from industry best practice of CD, and toward old world LTS releasing.18:09
clarkbSpamapS: so I think we'll have to drop the files restriction based on jeblair's comment. I'll leave a note on the change18:09
clarkber SamYaple ^18:09
* SpamapS S'niped clarkb's reply18:10
SamYapleclarkb: will update it, thanks :)18:10
jeblairfungi: hrm, i thought we had a reboot at the end of the launch which should take care of it.18:10
fungiis it safe to stop the zuul-merger daemon on one and reboot it?18:12
*** ijw has joined #openstack-infra18:12
jeblairfungi: i think/hope so?18:12
fungijust to see what transpires18:12
fungii'll do that on zm01 now18:12
jeblairi'm pretty sure i put in a think that waits till the current merge is complete before shutting down, so should be transparent.18:12
jeblaira thing18:12
fungiyeah, it's not running after a stop, so it stopped at any rate18:13
clarkbSamYaple: commented on the other change too18:13
fungirebooting now18:13
openstackgerritSam Yaple proposed openstack-infra/project-config master: Publish requirements loci images to DockerHub  https://review.openstack.org/51294118:13
*** catintheroof has quit IRC18:13
smcginnisWe have a couple of release-related project-config changes I'd like to see get through.18:14
*** electrofelix has quit IRC18:14
smcginnishttps://review.openstack.org/515381 and https://review.openstack.org/51536618:14
smcginnisWhen someone has the time...18:14
SamYapleclarkb: dang. i even updated the wrong section. that patch was for ceph, not docker18:14
fungijeblair: the firewall rules look the same to me after a reboot too18:15
*** andreas_s has joined #openstack-infra18:15
fungido we normally allow anything to them besides ping, ssh and cacti snmp?18:15
jeblairfungi: are we not enabling the pseudo-service or something?18:15
*** jkilpatr has joined #openstack-infra18:16
*** jkilpatr has quit IRC18:16
jeblairfungi: oh, for mergers, no18:16
openstackgerritSam Yaple proposed openstack-infra/system-config master: Add stretch mirror for ceph  https://review.openstack.org/51359118:16
smcginnisThanks jeblair18:16
fungijeblair: oh, well then the firewall rules before the reboot looked as expected18:16
jeblairfungi: that should be correct.  when you said minimal, i thought you meant 'open'.  :)18:16
fungino, no. the rules were just our minimal set18:16
jeblairfungi: thanks for testing reboot too!18:16
fungiindeed18:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Newton-EOL: Remove obsolete jobs and branches  https://review.openstack.org/51549218:16
jeblaircool, so it sounds like there isn't a systemic problem18:16
*** trown|lunch is now known as trown18:16
fungijeblair: merger daemon started back up on its own... i suppose that's to be expected?18:16
jeblairfungi: yep18:17
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Newton-EOL: Remove obsolete jobs  https://review.openstack.org/51549318:17
*** andreas_s has quit IRC18:20
*** yamahata has joined #openstack-infra18:22
*** andreas_s has joined #openstack-infra18:22
fungicool, i'll let the release team know they're open for business again18:23
*** Sukhdev has joined #openstack-infra18:25
*** aviau has quit IRC18:25
*** aviau has joined #openstack-infra18:25
clarkbfungi: pabelanger ianw I think tomorrow is the day I'm going to set aside for working on the various infra presentations at the summit. Probably going to start with fungi's onboarding presentty rst since thats mostly done then work on the infra update18:26
*** slaweq has joined #openstack-infra18:26
clarkbI think getting to the point where we have something then iterating in gerrit on infra publications is the way to go?18:26
fungiwfm18:27
fungii'll be around most of tomorrow, though need to go do early voting at some point since i'll be out of the country on election day18:27
clarkboh right I need to drop a ballot in the mail18:27
fungiyou and your fancy state from the future18:27
pabelangerclarkb: wfm18:28
clarkbwe use ancient technology to vote. Postal system, paper, ink18:28
fungiover here you still need to gouge your votes into clay tablets18:29
clarkbin washington postage is prepaid too iirc. But not in oregon18:29
fungior at least show up in person anyway18:29
*** slaweq has quit IRC18:29
openstackgerritMerged openstack-infra/project-config master: Build tarballs for networking-{bagpipe,bgvpn}  https://review.openstack.org/51538118:30
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Do not publish docs when building fails  https://review.openstack.org/51536518:30
AJaegerpabelanger: ^18:30
AJaegeris this good now?18:30
*** andreas_s has quit IRC18:32
openstackgerritMerged openstack-infra/project-config master: Switch service projects to not publish to pypi  https://review.openstack.org/51536618:32
*** yamamoto has joined #openstack-infra18:32
*** Sukhdev has quit IRC18:33
*** sree has joined #openstack-infra18:35
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Simplify queue names  https://review.openstack.org/51549618:37
*** andreas_s has joined #openstack-infra18:38
*** yamamoto has quit IRC18:38
*** sree has quit IRC18:40
*** andreas_s has quit IRC18:42
*** dhinesh has joined #openstack-infra18:42
openstackgerritMerged openstack-infra/system-config master: Gather facts in set_hostname playbook  https://review.openstack.org/51547818:42
AJaegerinfra-root, could one of you remove http://files.openstack.org/docs/publish-docs/ , please? That content is a publishing error that is fixed now18:43
clarkbAJaeger: the entier publish-docs directory needs to be deleted?18:44
AJaegerclarkb: yes18:44
AJaegerclarkb: missing / in rsync, so we published everything to publish-docs instead to /18:45
AJaegerthe same content is just available one level higher up as well18:45
clarkbgotcha18:45
clarkbI'm about to grab lunch but can work to remove it afterwards18:45
AJaegerthanks, clarkb18:45
*** jcoufal has quit IRC18:48
*** tosky has joined #openstack-infra18:49
*** catintheroof has joined #openstack-infra18:51
*** catintheroof has quit IRC18:56
dhineshi get the error, "SSHException: not a valid EC private key file", when trying to launch machines with nodepool18:58
dhineshi do have the key pairs set on the user that i am running from18:59
dhineshand also part of nodepool.yml, there is an entry "private-key: /home/nodepool/.ssh/id_rsa"19:00
dhineshis there somewhere else i have to populate the keys19:00
*** Goneri has quit IRC19:00
fungidoes that file have the usual begin and end markers in it? is it owned by the nodepool user?19:03
fungiand is the .ssh directory owned and readable by the nodepool user?19:05
*** catintheroof has joined #openstack-infra19:05
*** slaweq has joined #openstack-infra19:06
dhineshfungi: https://pastebin.com/deQPcGAV this is the nodepool.yml i am using19:06
fungiby "the file" i meant your /home/nodepool/.ssh/id_rsa file19:07
fungiand please don't let us know the contents of /home/nodepool/.ssh/id_rsa just double check that it's properly formatted with begin/end markers in it19:08
*** catintheroof has quit IRC19:09
fungiyou should see the first and last lines are something like -----BEGIN OPENSSH PRIVATE KEY----- and -----END OPENSSH PRIVATE KEY-----19:11
dhineshfungi: yes that is -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----19:11
dhineshshould it be openssh instead?19:11
fungiit's probably fine19:11
fungialso when you generated the key you used an empty passphrase so it wouldn't be encrypted, right19:12
dhineshyeah i did not add a keyphrase to it19:12
*** cody-somerville has joined #openstack-infra19:12
*** cody-somerville has joined #openstack-infra19:12
pabelangertrying using the ssh private key with ssh -v -i /home/nodepool/.ssh/id_rsa user@ip manually, see what SSH logs19:14
fungiright, openssh client tends to have a little better error reporting than the paramiko library19:15
*** jcoufal has joined #openstack-infra19:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Fuel branch cleanup  https://review.openstack.org/51550119:15
*** sree has joined #openstack-infra19:17
*** HeOS has joined #openstack-infra19:20
dhineshfungi: pabelanger: these are the last four lines19:21
dhineshdebug1: Offering RSA public key: /home/nodepool/.ssh/id_rsa19:21
dhineshdebug1: Authentications that can continue: publickey19:21
dhineshdebug1: No more authentication methods to try.19:21
dhineshPermission denied (publickey).19:21
tobiashdhinesh: does the node have the according public key in its authorized_keys file?19:22
fungidhinesh: okay, that suggests that either the username or key nodepool is trying to ssh in with isn't valid for the image you booted19:22
*** eharney has quit IRC19:23
pabelangeragree19:24
dhineshtobiash: yes i created the pair usinf ssh-keygen19:24
pabelangershould be able to look in dib build logs for your image to see which SSH public key was setup19:24
pabelangeror are you using glean to inject the keys?19:25
AJaegerconfig-core, could you review my newton-eol cleanups, please? https://review.openstack.org/#/q/status:open+topic:newton-eol I like to get those in quickly to avoid merge conflicts19:25
*** sree has quit IRC19:25
*** hemna_ has quit IRC19:26
*** trown is now known as trown|brb19:26
dhineshfungi: pabelanger: ok let me check the logs, no i dont have glean setup yet19:27
*** eumel8 has joined #openstack-infra19:27
*** catintheroof has joined #openstack-infra19:28
tobiashdhinesh: if you have the public key in the image maybe the permissions of the authorized_keys file are too open19:29
dhineshtobiash: but is authorized_keys file required in /home/nodepool/.ssh/ ?19:31
tobiashit's required in the home directory of the user you want connect to19:31
*** gordc has joined #openstack-infra19:32
gordchi, had a question re:zuul. https://review.openstack.org/#/c/515327 remove job from .zuul.yaml but for some reason it still gets run in check?19:34
gordcis there a secondary place the jobs are?19:34
*** yamamoto has joined #openstack-infra19:34
*** trown|brb is now known as trown19:35
pabelangergordc: which job is that?19:37
gordcpabelanger: the tox-py27-* jobs19:37
pabelangergordc: check project-config, they might be defined there19:38
AJaegerpabelanger: looks like zuul v3 does not remove jobs, when they are deleted.19:38
AJaegerpabelanger: jd_ reported something similar earlier19:38
AJaegerpabelanger: those jobs are not in project-config19:38
gordcAJaeger: oh. it's probably the same thing thing.19:38
pabelangerAJaeger: you mean removal from .zuul.yaml?19:38
pabelangerI don't see why it wouldn't19:39
*** yamamoto has quit IRC19:39
pabelangerdo you have .zuul.yaml on other branches?19:39
gordcpabelanger: all of them.19:39
gordcwe did manage to remove jobs in another patch: https://review.openstack.org/#/c/512286/12/.zuul.yaml19:39
pabelangersounds like it could be related to the work jeblair is working on with branch matchers19:40
gordcall of them (that isn't eol that is)19:40
pabelangerhttp://logs.openstack.org/27/515327/1/check/ceilometer-tox-py27-mongodb/d761ab3/zuul-info/inventory.yaml19:40
pabelangerjob seems to be coming from stable/ocata19:40
gordchmm.. i guess i'll have to wait for jeblair work19:41
dhineshfungi: pabelanger: https://pastebin.com/mtxkMmJ0 these are the only entries on the logs present when i searched for "id_rsa"19:41
gordcthere isn't a patch i could track is there?19:41
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Use native v3 publishing for project-team-guide  https://review.openstack.org/51405319:42
pabelangergordc: there is a ML post about the currently branch issues on openstack-infra, might be worth reading that. We also have zuulv3-issues etherpad, you can look and maybe add to19:42
gordcpabelanger: cool cool. thanks for the info19:42
dhineshfungi: pabelanger: looks like it is not copying the private key?19:42
* gordc going to try and remove branch references in branches to see if that works.19:43
pabelangerdhinesh: what users did you use to ssh with?19:43
AJaegergordc: jeblair said yesterday to expect something by Friday - can you wait another day or two?19:43
gordcAJaeger: yep :)19:44
dhineshpabelanger: i tried nodepool and jenkins19:44
pabelangerdhinesh: try zuul user19:45
pabelangerthat code path is http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user19:45
dhineshpabelanger: cool that worked19:45
pabelanger\o/19:46
*** harlowja has quit IRC19:46
dhineshpabelanger: so does nodepool try logging in with a different user19:46
pabelangerdhinesh: depends on what you have setup in nodepool.yaml19:46
dhineshpabelanger: yes got it19:46
pabelangerusername: zuul19:47
pabelangershould be your fix now19:47
*** ijw has quit IRC19:48
clarkblooks like afs doc publishing is via a secret now. Is there an easy way for me to get the keytab file so that I can delete the dir AJaeger wants deleted?19:49
clarkboh I suppose I can auth as my own user on one of the afs machines19:50
clarkb(no local afs for me)19:50
pabelangeryah19:50
dhineshpabelanger: ok let me try it out19:53
*** tpsilva has quit IRC19:53
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Switch kuryr back to publish-to-pypi  https://review.openstack.org/51551119:55
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Copy Gerrit delay to GitHub driver  https://review.openstack.org/51522419:56
*** hasharAway has quit IRC19:56
*** ijw has joined #openstack-infra19:56
*** jcoufal has quit IRC19:56
smcginnisAny issues going on that would be causing POST_FAILURES right now?19:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Set .root-marker variables for openstack-manuals  https://review.openstack.org/51551219:56
clarkbdoes 'aklog: a pioctl failed while obtaining tokens for cell openstack.org` mean I am on the wrong host for doing afs things? afs01.dfw.openstack.org fwiw19:58
clarkbsmcginnis: not that I am aware of19:58
AJaegersmcginnis: where do you get one?19:59
smcginnisclarkb: OK, just had a bunch of release failures due to the releasenotes jobs all geting POST_FAILURE.19:59
smcginnisAnd this is a new error to me: http://logs.openstack.org/9d/9da719b17a84b93f1725cf358ec4a2ef1752e0bb/tag/publish-openstack-releasenotes/1666a37/job-output.txt.gz#_2017-10-26_19_57_41_74898820:00
clarkbhuh ya /afs/openstack.org doesn't exist there I am guessing that is it20:01
*** vsaienk0 has joined #openstack-infra20:01
AJaegersmcginnis: mmh, zuul.branch not set in the pipeline?20:01
clarkbAJaeger: that is the tag pipeline I'm not sure if it will be present there20:01
AJaegerneed some zuul expert help on https://review.openstack.org/#/c/515512/1/zuul.d/jobs.yaml - why does this fail?20:02
jeblairyeah, there's no branch in the tag pipeline20:02
AJaegerso, how to fix http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/write-root-marker/tasks/main.yaml#n4 ?20:03
jeblairthat's just for humans, let's stick a default filter on it20:03
clarkbI'm going to use ze01 for afs dir deletion20:03
clarkbsince that is where things get afs anyways during job execution20:03
jeblairAJaeger: "{{ zuul.branch | default('') }}"  should do it20:04
AJaegerjeblair: shall I fix it or are you on it already?20:04
jeblairor put anything else in there20:04
jeblairAJaeger: if you could that'd be great20:04
AJaegeron it...20:04
clarkbAJaeger: to double check I am deleting /afs/.openstack.org/docs/publish-docs and we don't need a backup copy or anything?20:05
AJaegerclarkb: correct, this is all one level above - only content published fro msecurity-doc, openstack-manuals and api-site.20:05
clarkbAJaeger: ok going to run rm now20:05
AJaegerjeblair: and the other variables are all set? zuul.build and zuul.ref?20:06
AJaegerclarkb: thanks20:06
clarkbok that didn't work20:06
*** Apoorva has quit IRC20:06
clarkboh right we have separate admin principals right?20:06
AJaegerclarkb: are you on the read-write volume?20:06
clarkbI don't use kerberos users often enough20:06
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Hande write-root-marker in tag pipeline  https://review.openstack.org/51551720:08
*** eharney has joined #openstack-infra20:08
AJaegersmcginnis, pabelanger, jeblair ^20:08
clarkbthat appears to have been it. delete in progress now20:09
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Switch octavia back to publish-to-pypi  https://review.openstack.org/51551820:09
*** slaweq has quit IRC20:09
smcginnisAJaeger: Thanks!20:09
*** slaweq has joined #openstack-infra20:10
clarkbAJaeger: ls: cannot access 'publish-docs': No such file or directory <- I'm inclined to let the next publish job do the afs publish if only to avoid racing on that (not sure if we can do concurrent publishes20:10
clarkbAJaeger: so it may be a bit before that reflects on the RO volume but it is done on RW now20:10
AJaegerclarkb: http://files.openstack.org/docs/publish-docs/ is gone, so I'm happy ;)20:10
jeblairhappens every 5m if the volume is updated20:11
clarkbperfect20:11
jeblairclarkb: so you probably sucessfully waited for the next automatic publish20:11
AJaegerjeblair: yep - so, perfect timing ;)20:11
*** vsaienk0 has quit IRC20:11
clarkband btw it was I kinit'd the non admin account first time around20:11
clarkbonce that was sorted everything worked great20:11
jeblairyay perms are correct20:11
AJaegerjeblair: could you tell me why zuul encounters a syntax error on https://review.openstack.org/#/c/515512/1/zuul.d/jobs.yaml , please?20:12
clarkbI should make my local fileserver an afs client since it runs ubuntu20:12
jeblairAJaeger: you need quotes around {{}}20:13
AJaegerAh! thanks20:13
jeblairwe should probably intercept that error and handle it specially20:14
*** slaweq has quit IRC20:14
clarkboh I just approved the change20:14
*** hamzy has quit IRC20:14
clarkbor are we talking about a different change20:14
clarkbI approved 51551720:14
jeblairclarkb: different: 51551220:14
clarkb2 and 7 too similar at first glance20:14
jeblairbecause the yaml parser is confused (it thinks it's constructing yet another mapping started by the {{ characters )20:15
jeblairthat's why the message ("unhashable key") doesn't make sense20:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Set .root-marker variables for openstack-manuals  https://review.openstack.org/51551220:15
AJaegerjeblair: I see, thanks. Trying again ^20:15
jeblairclarkb: that's why new versions of pep8 don't allow the use of those numbers20:15
*** ldnunes has quit IRC20:15
AJaegerjeblair: that's an upcoming pep8 ;)20:16
jeblairbig problem on those clocks too, where the 7 is an upside down 220:16
AJaegerOur publishing deleted by accident the docs.o.o index page due to uploading docs after build that failed. Could I get a second +2 on https://review.openstack.org/#/c/515365 , please? This way we only upload after successfull builds20:17
jeblairAJaeger: done20:19
AJaegerthanks, jeblair20:19
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Update testenv-worker for newer novaclient  https://review.openstack.org/51551920:19
clarkbfungi: it looks like zuul.conf may have updated 2 minutes after zuul was restarted? I'm not seeing the cgit urls apply properly, current guess is that the restart happened just a little too early20:22
ianwpabelanger / fungi : yeah, doh ... i was using my branch of launch-node.py from https://review.openstack.org/#/c/440928/ (which i actually find quite useful but not sure anyone else agrees)20:22
clarkbfungi: do you know if you checked that puppet had run and update the conf before it was restarted? want to make sure I don't blame that too early20:23
clarkbeg do I need to keep debugging or just wait for next restart20:23
*** sree has joined #openstack-infra20:24
*** smatzek has quit IRC20:25
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add provider defaults to launch-node.py  https://review.openstack.org/44092820:25
openstackgerritMerged openstack-infra/project-config master: Do not publish docs when building fails  https://review.openstack.org/51536520:26
*** armax has quit IRC20:26
*** camunoz has quit IRC20:26
*** anupn has quit IRC20:26
*** anupn has joined #openstack-infra20:27
*** smatzek has joined #openstack-infra20:27
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Silence ansible-lint  https://review.openstack.org/51552020:28
*** sree has quit IRC20:29
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Silence ansible-lint  https://review.openstack.org/51552020:29
*** jamielennox has quit IRC20:30
*** rbrndt_ has quit IRC20:30
*** smatzek has quit IRC20:32
*** harlowja has joined #openstack-infra20:33
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Silence ansible-lint  https://review.openstack.org/51552020:33
fungiclarkb: i probably forgot to make sure it had successfully applied, sorry :/20:33
clarkbfungi: no problem I am sure there will be another restart :)20:33
clarkbjust wanted to make sure I didn't need to debug more20:34
AJaegerpabelanger: testing ansible-lint in 515520...20:34
*** burgerk has quit IRC20:34
* AJaeger waves good night20:34
*** jamielennox has joined #openstack-infra20:34
*** Apoorva has joined #openstack-infra20:35
fungiclarkb: yeah, let's just assume it's waiting for the next restart for now20:35
*** yamamoto has joined #openstack-infra20:35
jeblairwhich will hopefully be tomorrow with the changes i'm writing20:35
dmsimardok, finished extinguishing various fires..20:35
dmsimardlooking at elastic recheck patch now20:36
jeblairhi, if folks wanted to start looking at the inheritance changes, i think they're ready for review.  i'm finishing up the implied-run "bonus" change now.  but the others (514459, 511352, 515249, 515483) are ready.20:36
clarkbjeblair: AJaeger does https://review.openstack.org/#/c/515512/2/zuul.d/jobs.yaml need the same branch deafult fix that 515517 got?20:37
clarkbbasically default to empty string if no value is present20:37
*** ijw has quit IRC20:37
jeblairgood question.  if it gets used in tag, yes it will.20:37
AJaegerclarkb: it only runs in post, so shouldn't.20:37
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Use rdoproject-based settings for containers and dlrn_hash for images  https://review.openstack.org/51466820:38
clarkbAJaeger: thanks20:38
clarkbI guess we don't tag our manuals20:38
*** ijw has joined #openstack-infra20:38
AJaegerwe could make it robust in case it gets used there at one day - right now we don#t tag...20:38
AJaegerbye for today20:39
*** yamamoto has quit IRC20:40
clarkbgood night20:41
*** ijw has quit IRC20:44
fungiianw: tonyb: not sure whether you're around yet, but you're probably best equipped to address http://lists.openstack.org/pipermail/openstack-dev/2017-October/124030.html20:46
*** andreas_s has joined #openstack-infra20:48
*** catintheroof has quit IRC20:49
ianwtonyb: hmm, i imagine just putting back eol tags ... ping me and we can do that20:50
*** thorst has quit IRC20:50
*** jcoufal has joined #openstack-infra20:52
openstackgerritMerged openstack-infra/project-config master: Set .root-marker variables for openstack-manuals  https://review.openstack.org/51551220:53
*** jamielennox has quit IRC20:54
clarkbianw: ya the tag is there could just branch off of that20:55
clarkband point back at hte mailing list I guess20:55
*** jamielennox has joined #openstack-infra20:55
*** smatzek has joined #openstack-infra20:55
fungithe only catch is that when it does finally come time to eol that branch later, you need a slightly different tag20:56
fungi9.0-eol-no-really-we-mean-it-this-time20:56
pabelangerha20:56
clarkbeolv220:57
jeblairi guess we need a stricter explicit sign-off process20:58
*** andreas_s has quit IRC20:58
*** jcoufal has quit IRC20:59
*** andreas_s has joined #openstack-infra20:59
*** smatzek has quit IRC21:01
openstackgerritMerged openstack/os-client-config master: Consume the osc-functional-devstack-tips job  https://review.openstack.org/51484721:01
*** HeOS has quit IRC21:02
tonybandreykurilin: So sorry.  We can lean on infra to correct that21:03
*** HeOS has joined #openstack-infra21:03
*** mat128 has quit IRC21:03
dmsimardJust curious, I remember a discussion at the PTG around moving the hardware from infracloud elsewhere (HPE datacenter to an university or something like that?), did that ever happen ?21:03
tonybandreykurilin: My next question is what could I have reasonably done to ensure your saw and responed to my email?21:03
andreykurilintonyb: nothing is lost, so np :)21:04
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Hande write-root-marker in tag pipeline  https://review.openstack.org/51551721:04
andreykurilintonyb: there were too many lines with listed branches, so I even didn't read them. Mention all affected project can fix "attention"21:05
clarkbfwiw there is nothing on pypi related to that eol21:05
clarkbwe can recreate the stable branch based on the tag21:06
*** trown is now known as trown|outtypewww21:06
tonybandreykurilin: ok.21:06
tonybandreykurilin: you could have searched for rally ... that would have gottent passed the too many line issue21:06
jeblair"[rally]" did appear to be missing from the subject line, so it's understandable21:07
clarkbhttps://pypi.python.org/simple/rally/ for pypi listing21:07
andreykurilinclarkb: cool. Also, I pushed the branch to my own public repo if needed21:07
pabelangerdmsimard: http://lists.openstack.org/pipermail/openstack-infra/2017-September/005585.html21:07
tonybjeblair: It's true I didn't call it out specifically.  I'll just make the subject longer next time21:07
jeblairtonyb: heh, yeah, all or none would probably be best :)21:08
pabelangerdmsimard: next step is to find hardware21:08
tonybclarkb: I shoudl be able to crate a branch as a Release manager, shouldn't I?21:08
clarkbtonyb:  maybe? I'll look21:08
clarkbit looks like rally-core can create branches themslves21:09
clarkbrally-core has create reference on refs/heads21:09
clarkbhttps://review.openstack.org/#/admin/projects/openstack/rally,access21:09
*** andreas_s has quit IRC21:10
clarkbrelease managers has this perm too so either group21:10
*** mummer has quit IRC21:10
andreykurilintonyb: people usually lazy and if nothing attracts attention, no action will be performed. The topic attracted me, but I just saw tags for several projects and it led me ignoring global tag [all]21:10
tonybandreykurilin: again I'm sorry.  Do you want me to (re-)create the branch for you or will you do it?21:11
*** pblaho has joined #openstack-infra21:11
andreykurilintonyb: how can I do it?21:11
tonybandreykurilin: okay, next time I'll include all affected project teams.21:11
andreykurilinCan I just push the branch via gerrit? Do I have enough rights for this?21:12
clarkbno you can't push them21:12
clarkbyou have to create them via the gerrit ui21:12
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Use rdoproject-based settings for containers and dlrn_hash for images  https://review.openstack.org/51466821:12
jeblairand use "0.9-eol" for the value21:13
clarkbhttps://review.openstack.org/#/admin/projects/openstack/rally,branches is where you do it21:14
*** lindycoder has quit IRC21:14
clarkbbranch name field should be stable/0.9 I think and then initial revision is 0.9-eol21:14
*** anupn has quit IRC21:15
andreykurilinit worked21:15
andreykurilinthanks21:15
tonyb\o/21:15
tonybandreykurilin: What is the exptect EOL for that branch?21:16
*** anupn has joined #openstack-infra21:16
tonybandreykurilin: FWIW I now have the branch here21:17
andreykurilintonyb: no special date. we just released a new major version(0.10.0) and need to support both for awhile, since 0.10.0 contains a lot of refactoring...21:18
andreykurilinActually, I hope that I'll not any more 0.9.* releases and currect state of the branch will be the last21:18
tonybandreykurilin: okay are you thinking 6months 12months ?21:18
tonybandreykurilin: or will it approximately match the queens-eol date?21:19
andreykurilinideally, < 6 month21:19
*** andreas_s has joined #openstack-infra21:19
tonybAhh okay, well when you're ready let me know and I'll include it in the closest batch if that works for you21:19
tonybthis was a one off clean all the "cruft" out so from now on it shoudl be more targetted21:20
andreykurilintonyb: our branches doesn't correlate with openstack releases. the latest rally has several jobs with mitaka openstack and it still work. even with latest openstack clients:)21:20
andreykurilintonyb: I'll ping you as soon as we decide that 0.9 is not required any more. as I mentioned, hope it will be soon21:22
*** armax has joined #openstack-infra21:22
andreykurilintonyb: and what about 0.9-oel tag. does it possible to remove it?21:22
tonybandreykurilin: Cool.21:22
*** rlandy is now known as rlandy|brb21:23
clarkbtypically we don't delete tags because downstreams wno't pick up a deleted tag21:25
clarkbI'd be inclined to leave it particularly since chances are good we don't actually change that branch between now and actual eol21:25
clarkbin which case we acn just delete the branch and be done21:26
*** aeng has joined #openstack-infra21:27
*** andreas_s has quit IRC21:28
openstackgerritMerged openstack-infra/project-config master: Newton-EOL: Remove obsolete jobs and branches  https://review.openstack.org/51549221:28
*** dprince has quit IRC21:30
*** andreas_s has joined #openstack-infra21:30
openstackgerritMerged openstack-infra/project-config master: Simplify queue names  https://review.openstack.org/51549621:31
cloudnullqq - https://docs.openstack.org/pike/deploy/ that shows links to the kolla and osa deployment guides however it just links on-to itself, is there something we should do to fix that ?21:32
*** CrayZee has quit IRC21:32
smcginnisNew POST_FAILURES on releases: http://logs.openstack.org/2f/2f8ed39c4e19fc0afd22525fa36f9de3cc02ca38/tag/publish-openstack-releasenotes/3fe77fb/job-output.txt.gz#_2017-10-26_21_29_41_77884021:33
smcginnisDoesn't like the conditional publishing that just merged?21:33
jeblairsmcginnis: ya21:34
*** andreas_s has quit IRC21:34
jeblairoh, missing a :21:34
jeblairpabelanger, jlk: how's the ansible-lint project coming?  :)21:34
jeblairi'll work on a fix21:35
*** CrayZee has joined #openstack-infra21:36
pabelangerjeblair: needs some reviews: https://review.openstack.org/512715/21:36
smcginnisjeblair: Thanks21:36
smcginnislinting sounds useful there. ;)21:36
*** yamamoto has joined #openstack-infra21:37
pabelangersmcginnis: that is my mistake, I reviewed that change from AJaeger21:37
jeblairpabelanger: is there an effort to create a generic ansible linter any repo can use?21:37
jeblairansible linter job, that is21:38
ianwclarkb: https://review.openstack.org/#/c/515347 :/ we all missed that (ENABLE_KSM not using $).  i guess it wasn't working21:38
smcginnispabelanger: I looked at it too, but that wasn't obvious.21:38
clarkbianw: oh good they pushed a patch up21:38
clarkbI got a personal email asking about it ans I suggested the patch it :)21:39
cloudnullalso looks like our docs to the draft deploy guide are being redirected https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/ :(21:39
pabelangerjeblair: yah, we could document how a project could update tox.ini today, but I think mordred see us creating ansible-linters playbook job that any project could run, without tox.ini changes21:39
*** askb has joined #openstack-infra21:39
jeblairpabelanger: yes, is there a reason we need tox?21:40
pabelangerjeblair: no, it was an easy way to manage pip dependencies mostly21:40
jeblairpabelanger: what are the pip depes we need?21:40
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Newton removal: Remove periodic-newton  https://review.openstack.org/51491121:41
pabelangerjeblair: just ansible-lint21:41
clarkbcloudnull: I want to say all those redirects are controlled by htpasswd files in the projects themselves21:41
pabelangerwith zuulv3, would be a simple playbook21:41
clarkbcloudnull: so I would look in the project deploy guide and kolla21:41
jeblairpabelanger: ok.  i +2d that change because if we land it, it will save us so much time.  but yeah, i think we should replace it with a job that doesn't use tox, just installs ansible-lint, and adds all required-projects to the ansible-roles-path automatically.21:41
jeblairpabelanger: then we could use the same job for any single repo or sets of repos21:42
jeblairclarkb: can you +3 512715?21:42
*** baoli has quit IRC21:42
* clarkb looks21:42
dhinesham getting an error "HTTPError: HTTP Error 405: Method Not Allowed" while running nodepool, is there a way to see logs of HTTP requests given by nodepool?21:42
*** yamamoto has quit IRC21:42
dhineshi was not able to find it in the debug logs21:43
*** baoli has joined #openstack-infra21:43
*** baoli has quit IRC21:43
cloudnullclarkb: it doesn't look like there's any htaccess file in https://github.com/openstack/kolla-ansible/tree/master/deploy-guide/source or https://github.com/openstack/openstack-ansible/tree/master/deploy-guide/source somewhere else I might need to look?21:44
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Fix missing "role:" mapping keys  https://review.openstack.org/51552921:46
jeblairclarkb, smcginnis, pabelanger: ^21:46
clarkbcloudnull: https://github.com/openstack/kolla-ansible/tree/stable/pike is where I would look because https://docs.openstack.org/project-deploy-guide/kolla-ansible/pike/ is what is being redirected21:47
dmsimardfungi: I hear you had estimated the costs for shipping the infracloud hardware to OSU OSL ?21:48
clarkbcloudnull: but I don't see one there, could it be project-deploy-guide?21:48
clarkbcloudnull: dhellmann and AJaeger would probably undersatnd that best though21:48
*** andreas_s has joined #openstack-infra21:48
clarkbdhinesh: where are you seeing that error?21:48
smcginnisHey, I got to tour the OSU OSL once.21:48
cloudnullthanks clarkb! I'll keep poking around.21:49
openstackgerritKendall Nelson proposed openstack-infra/system-config master: Add #openstack-women to meetbot channels  https://review.openstack.org/51553121:50
*** rlandy|brb is now known as rlandy21:51
dmsimardsmcginnis: nice place? :)21:52
smcginnisdmsimard: Yeah, it was pretty cool.21:53
clarkbdhinesh: as for seeing the logs of http requests you can edit your logging config to log python requests as a debug level to get everything. Be warned this may include raw passwords/tokens/etc21:53
dhineshclarkb: its part of the debug.log, as part of nodepool, its able to ssh into the machine, and then it throws this error.21:54
clarkbdhinesh: do you have a full traceback you can share?21:55
*** jascott1 has joined #openstack-infra21:55
dhineshclarkb: yes i do https://pastebin.com/Q23ZV3pF21:57
jeblairclarkb, pabelanger: can you review https://review.openstack.org/515529 ?  it's urgent for release21:58
*** andreas_s has quit IRC21:58
*** gouthamr has quit IRC21:59
*** marst has quit IRC22:00
*** thorst has joined #openstack-infra22:00
clarkbdhinesh: iirc jenkins updated their API and changed some things from get to post22:01
*** rbrndt_ has joined #openstack-infra22:01
*** jascott1 has quit IRC22:01
clarkbdhinesh: I thought that latest nodepool master switched most jenkins api calls to python-jenkins which added support for this?22:01
clarkbbkero: ^ I seem to recall you were involved in that? do you know?22:01
dmsimardjeblair: looking22:01
dmsimardjeblair: our gate jobs should have caught that :(22:02
clarkbapproved22:02
jeblairdmsimard: we just approved the ansible lint job that would have22:02
dmsimardack22:02
clarkbdhinesh: what version of nodepool and what version of python-jenkins do you have?22:03
bkeroclarkb: that was a while ago. I think it uses it, but not for everything22:03
bkeroIIRC there are some features of python-jenkins that are duplicated by nodepool, although there was a big commit to get rid of a lot of that22:03
hongbinhi infra team, could anyone take a look at: https://review.openstack.org/#/c/515511/ ? we needs it to release the library22:03
clarkbbkero: ya pretty sure that big commit merged22:03
clarkbhttps://review.openstack.org/#/c/259157/ that one22:04
*** andreas_s has joined #openstack-infra22:04
clarkbwhcih based on dhinesh's traceback isn't the code being used22:04
clarkbdhinesh: so I think you need to update22:04
dhineshmy nodepool version is 0.3.122:05
*** thorst has quit IRC22:06
dhineshclarkb: how would you get the version of python-jenkins?22:06
clarkbdhinesh: pip freeze22:06
dhineshclarkb: python-jenkins==0.4.1522:07
*** boden has quit IRC22:07
*** esberglu has quit IRC22:07
clarkbbut I think you need to update nodepool22:08
clarkbsince myjenkins is the cause of the problem22:09
*** MasterPiece has joined #openstack-infra22:09
*** andreas_s has quit IRC22:09
ianwclarkb: https://review.openstack.org/#/c/515463/1/playbooks/legacy/its-storyboard-gerrit-plugin-stable-2.13/post.yaml -- does the recurse actually do anything?22:10
*** smatzek has joined #openstack-infra22:10
clarkbianw dmsimard said it was necessary22:10
clarkbI think to apply the same atrirbutes the whole way?22:11
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Create openstack-zuul-jobs-linters jobs  https://review.openstack.org/51271522:11
dmsimardclarkb: iirc you wanted to use recurse to create a nested directory tree22:11
ianwdmsimard: that's not really my reading of http://docs.ansible.com/ansible/latest/file_module.html, but, on occasion, i have been known to be wrong about things :)22:12
*** spzala has quit IRC22:14
*** spzala has joined #openstack-infra22:15
*** andreas_s has joined #openstack-infra22:15
dmsimardianw: http://paste.openstack.org/raw/624753/22:15
openstackgerritMichael Johnson proposed openstack-infra/project-config master: Switch kuryr back to publish-to-pypi  https://review.openstack.org/51551122:16
openstackgerritMichael Johnson proposed openstack-infra/project-config master: Switch octavia back to publish-to-pypi  https://review.openstack.org/51551822:16
openstackgerritMichael Johnson proposed openstack-infra/project-config master: Remove migrated octavia zuul v3 jobs  https://review.openstack.org/51548622:16
dmsimardianw: you can send a PR to update the docs :) https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/file.py#L1622:18
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524922:18
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add pragma directive  https://review.openstack.org/51548322:18
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove implied run  https://review.openstack.org/51548722:18
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Support file extension in playbook path  https://review.openstack.org/51553522:18
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Update test fixtures to use explicit run  https://review.openstack.org/51553622:18
ianwdmsimard: yeah, but try it without "recurse" and it still works?22:18
*** smatzek has quit IRC22:18
ianw~/tmp/ansible/bin/ansible localhost -m file -a "path=/tmp/some/long/path/with/dirs state=directory"22:18
ianwdid for me, anyway22:18
dmsimardianw: huh, you're right actually o_O22:18
dmsimardTIL22:18
*** gildub has joined #openstack-infra22:19
dmsimardEither I wrongly assumed recurse was required or maybe that changed at some point and I kept using it (which wouldn't surprise me)22:19
*** spzala has quit IRC22:19
dmsimardclarkb: I was wrong!22:20
*** andreas_s has quit IRC22:20
*** andreas_s has joined #openstack-infra22:21
openstackgerritMerged openstack-infra/project-config master: Fix missing "role:" mapping keys  https://review.openstack.org/51552922:23
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Create dir needed to publish gerrit plugins  https://review.openstack.org/51546322:23
*** felipemonteiro_ has quit IRC22:24
dhineshclarkb: i upgraded nodepool using http://git.openstack.org/cgit/openstack-infra/nodepool/tree/README.rst22:27
dhineshthen tried to start nodepool service22:27
dhineshgot this ImportError: No module named nodepool.cmd.nodepoold22:28
*** xarses has quit IRC22:28
*** bobh has quit IRC22:30
*** spzala has joined #openstack-infra22:31
*** bobh has joined #openstack-infra22:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Validate that a job has a run playbook on freeze  https://review.openstack.org/51553722:31
ianwjeblair / clarkb: https://review.openstack.org/#/c/513595/ -- moving swap/opt role into o-z-j ... any opinions here?  this is what we agreed to do in the meeting a week or so ago.  i'd like to call it from dib jobs to make sure /opt is nice and big22:31
jeblairianw: did we discuss maybe putting it in plain-old zuul-jobs?22:32
*** xarses has joined #openstack-infra22:32
jeblairoh i see comments...22:33
*** xarses has quit IRC22:33
*** rlandy has quit IRC22:33
jeblair+222:33
*** gordc has quit IRC22:34
*** xarses has joined #openstack-infra22:35
*** bobh has quit IRC22:36
*** xarses has quit IRC22:36
*** xarses has joined #openstack-infra22:38
*** yamamoto has joined #openstack-infra22:39
*** leyal has quit IRC22:39
*** lihi has quit IRC22:39
clarkbdhinesh: doe that have a traceback too? (it is much easier to debug these things with more context)22:41
*** leyal has joined #openstack-infra22:41
*** lihi has joined #openstack-infra22:41
ianwjeblair: yeah, the only "guarantee" i'd give is that it does something useful on hosts inside clouds that infra uses ... to be z-j i'd think that would "does something useful on every cloud"22:42
jeblairianw: maybe we can get there, but we don't have to do that today22:42
clarkbianw: jeblair configure-swap is the same name in devstack and in ozj, will that cause lookup problems?22:43
clarkbroles can be shared22:43
*** apetrich has quit IRC22:43
jeblairclarkb: there's a lookup order.  in this case it doesn't matter which wins, they're the same.  we'll remove the devstack one when this lands.22:43
ianwthe devstack one should overwite until we remove it22:44
ianwoh, what jeblair said :)22:44
*** yamamoto has quit IRC22:44
clarkbok approved22:44
dhineshsure, https://pastebin.com/mdF4W81m22:45
jeblair(roles are prepended as you go down the inheritance hierarchy, so if i child job specifies a role repo, it will take precedence over its parent.  the roles form part of the playbook context though, so a child can't override the set of roles available to a parent playbook)22:46
*** apetrich has joined #openstack-infra22:46
clarkbdhinesh: that module is certainly there in the git repo22:46
*** sshnaidm is now known as sshnaidm|off22:46
clarkbdhinesh: does pip freeze show it is installed? is the version different?22:47
ianwjeblair / clarkb: anyway, if you'd like https://review.openstack.org/#/c/513597/ removes it from devstack too :)22:47
*** andreas_s has quit IRC22:48
*** claudiub|2 has joined #openstack-infra22:48
*** rbrndt_ has quit IRC22:49
*** andreas_s has joined #openstack-infra22:49
*** jaypipes has quit IRC22:51
dhineshclarkb: when i do pip freeze this shows up "-e git://git.openstack.org/openstack-infra/nodepool@3bd21837001c77873b7cf4827de6d898fed567e1#egg=nodepool"22:52
dhineshbut nodepool --version shows 0.4.122:52
dhineshlooks like it did not install properly22:53
clarkbthose two versions look right actually. 3bd2183 is tip of master and 0.4.0 was the last tag (so we are 0.4.1 for the actual version)22:53
*** salv-orl_ has quit IRC22:53
clarkbwas this installed in a virtualenv or to the system?22:54
*** andreas_s has quit IRC22:54
*** salv-orlando has joined #openstack-infra22:54
dhineshclarkb: it was installed to the system,  i used puppet to initiate it the first time i installed it, maybe it has some dependencies on it22:58
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add configure-swap role  https://review.openstack.org/51359522:58
*** salv-orlando has quit IRC22:59
clarkbdhinesh: if you run python and then import nodepool.cmd.nodepoold does that work?22:59
clarkbmaybe it installed to a different python?22:59
*** andreas_s has joined #openstack-infra23:00
dhineshclarkb: yeah that works fine, python version is 2.723:01
dmsimardclarkb, frickler: for https://review.openstack.org/#/c/509313/3/elastic_recheck/config.py -- in v2 it's not necessary to pick up the failure messages because we can just use build_status:"FAILURE"23:01
dmsimardit's shorter and more reliable IMO23:01
clarkbdmsimard: I think it is using the console log output to make sure we've actually indexed it properly23:02
*** xarses has quit IRC23:02
clarkbbut ya I think in practice using the build status there is probably fine23:03
dmsimardclarkb: that's ok, we're matching [Zuul] Job complete23:03
dmsimardclarkb: I'll reply to the comments and send another patchset23:03
clarkbya that should work23:03
clarkbI still think you want to group teh v2 query bits together and OR them with the v3 query bits23:03
dmsimardyeah I got that, that's fine23:04
dhineshclarkb: i tried upgrading 0.4.1 using puppet, it failed, but when i tried 0.4.0, it works now23:04
*** xarses has joined #openstack-infra23:04
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Update test fixtures to use explicit run  https://review.openstack.org/51553623:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove implied run  https://review.openstack.org/51548723:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Validate that a job has a run playbook on freeze  https://review.openstack.org/51553723:05
dhineshclarkb: https://pastebin.com/dVXAH3ia logs on puppet23:06
clarkbdhinesh: yes 0.4.1 isn't tagged yet23:06
*** mriedem is now known as mriedem_away23:07
clarkbbut since we haev 0.4.0 tagged any commits beyond it have to be at least 0.4.1 so that is what the version shows as when installing from the branch tip23:07
clarkbit doesn't look like 0.4.0 has the myjenkins fix though23:07
dmsimardclarkb: oh, er, frickler has a point in the other file comments.. there's also the *jenkins* version of the v2 to match23:07
dmsimard[Zuul] Job complete and the jenkins equivalent23:07
* dmsimard pouts23:07
*** xarses has quit IRC23:07
clarkbdmsimard: ya so its (filename:"console.html" AND (message:"foo" OR message:"bar"))23:08
*** tosky has quit IRC23:11
*** andreas_s has quit IRC23:11
*** andreas_s has joined #openstack-infra23:13
*** wolverineav has quit IRC23:14
*** wolverineav has joined #openstack-infra23:15
openstackgerritDavid Moreau Simard proposed openstack-infra/elastic-recheck master: Add support for Zuulv3-specific parameters in elastic-recheck  https://review.openstack.org/50931323:19
*** pblaho has quit IRC23:19
*** wolverineav has quit IRC23:19
dmsimardclarkb: ^ I middle fingered pep8 because the darn thing was not readable, I'm sure the legibility improvement is worth it :)23:20
*** wolverineav has joined #openstack-infra23:20
clarkbya ship it23:21
clarkbI just +2'd23:21
dmsimardmtreinish: do you think you can take a look at https://review.openstack.org/#/c/509313/ ?23:21
*** wolverineav has quit IRC23:24
*** andreas_s has quit IRC23:27
*** andreas_s has joined #openstack-infra23:28
*** claudiub|2 has quit IRC23:28
*** jascott1 has joined #openstack-infra23:28
*** jamesdenton has quit IRC23:28
*** thorst has joined #openstack-infra23:29
*** lindycoder has joined #openstack-infra23:30
*** jamesdenton has joined #openstack-infra23:31
*** sdague has quit IRC23:33
*** andreas_s has quit IRC23:33
*** andreas_s has joined #openstack-infra23:35
*** dingyichen has joined #openstack-infra23:36
*** yamamoto has joined #openstack-infra23:40
*** thorst has quit IRC23:43
*** andreas_s has quit IRC23:44
*** yamamoto has quit IRC23:46
*** links has joined #openstack-infra23:46
*** andreas_s has joined #openstack-infra23:47
jeblairremote:   https://review.openstack.org/515552 Zuul: add run attribute23:47
jeblairclarkb, ianw: ^ that might be good to get moving soon23:48
jeblair(i'm scanning all the repos in gerrit to find cases like that)23:49
ianwjeblair: ++ ... it can get hard to follow without that23:49
*** salv-orlando has joined #openstack-infra23:54
*** MasterPiece has quit IRC23:57
*** andreas_s has quit IRC23:57
*** Swami has quit IRC23:59
*** salv-orlando has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!