Tuesday, 2017-10-10

fungiand just in time for the "a little over 35 hours from now" to not be a lie00:00
*** aeng has joined #openstack-infra00:00
*** edmondsw has quit IRC00:04
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci master: Add logs for config-download  https://review.openstack.org/51070900:04
*** mriedem1 is now known as mriedem00:04
*** yamahata has quit IRC00:07
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout  https://review.openstack.org/50951700:10
*** aspiers has quit IRC00:11
fungiianw: i take it the strace sudo error for gerrit2 on review.o.o was you00:13
ianwfungi: yep, sorry00:14
funginp, just making sure00:14
*** ianychoi has joined #openstack-infra00:14
*** dingyichen has joined #openstack-infra00:15
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Retry change queries on a 502 response  https://review.openstack.org/51070600:19
*** cody-somerville has quit IRC00:19
*** cody-somerville has joined #openstack-infra00:20
*** cody-somerville has quit IRC00:20
*** cody-somerville has joined #openstack-infra00:20
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Improve exception handling around lost requests  https://review.openstack.org/51059400:20
openstackgerritAnup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3  https://review.openstack.org/50964100:20
*** Swami has quit IRC00:21
*** gouthamr has joined #openstack-infra00:21
*** xarses has quit IRC00:21
*** ijw has joined #openstack-infra00:27
pabelangerfungi: wfm00:27
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project  https://review.openstack.org/50911900:29
pabelangerianw: I'm getting some 502 back from gerrit this evening too00:32
*** ijw has quit IRC00:32
*** ijw has joined #openstack-infra00:33
ianwpabelanger: yeah, we're moving into ~ hour 4 of killer cpu time00:34
ianwjeblair suggested about 1:30 ago to wait ~ hour and see if it decreases, i'd say it's not00:35
ianwany opposition to a restart?00:35
ianwpabelanger: do you recall anything around ensuring the installation of python-selinux to ensure ansible templates get their permissions setup right?00:37
ianwi'm going to assume it's all working, until proven otherwise00:38
jeblairianw: i agree this doesn't look promising.  i'm +1 on restart.00:38
ianw#status log restarting zuul after prolonged period of high GC activity is causing 502 errors00:39
openstackstatusianw: finished logging00:39
jeblairs/zuul/gerrit/ :)00:40
ianwbah!  zuul on my brain00:40
*** mriedem has quit IRC00:40
*** sree has joined #openstack-infra00:41
*** Apoorva_ has joined #openstack-infra00:41
*** andreas_s has joined #openstack-infra00:42
ianw#status log restart of *gerrit* complete00:43
openstackstatusianw: finished logging00:43
ianwcpu is still suspiciously high i'd say, but let's give it a bit to settle down00:44
*** Apoorva has quit IRC00:44
ianwhmm, settling down to 2-300%, better than the 8-900% before00:45
*** Apoorva_ has quit IRC00:45
*** sree has quit IRC00:45
*** LindaWang has joined #openstack-infra00:46
*** andreas_s has quit IRC00:50
*** andreas_s has joined #openstack-infra00:51
*** ijw has quit IRC00:53
smcginnisIt does seem much more responsive.00:55
*** ijw has joined #openstack-infra00:58
*** jkilpatr has quit IRC00:59
*** s-shiono has joined #openstack-infra01:02
*** gildub has joined #openstack-infra01:04
*** cody-somerville has quit IRC01:04
fungizuulv3 scheduler debug log is nearing 15gib now... mildly worried that come time for log rotation there won't be sufficient room on the rootfs to hold the compressed copy01:10
fungidf claims 17gib free on /01:10
tonybfungi: Bah who needs roatated logs ;P01:10
fungiwith about another 5 hours to go until rotation01:11
fungitonyb: more along the lines of "bah who needs to be able to write to the root filesystem"01:11
tonyb:)01:12
fungiwhich is what processes will be asking themselves once logrotate sucks up every last available block in it01:12
mtreinishfungi: you could just run the rotate job manually (assuming it's got the normal default of cron daily)01:12
fungimtreinish: calling logrotate manually has taken care of it in the past, at any rate01:12
fungitonyb: last time this happened, i had to nova reboot the server because sshd couldn't write to disk any longer01:13
tonybfungi: Oh dear :(01:13
fungiyeah :/01:13
*** andreas_s has quit IRC01:14
fungiinfra-root: any objections to me manually rotating logs for now (we plan to provide more room before wednesday cut-over anyway)01:14
ianwspin spin spin01:15
fungiright round01:15
*** cuongnv has joined #openstack-infra01:17
pabelangerfungi: I think we'll also want to resize our current swap on zuulv3 tomorrow too, and make room for /var/log/zuul on it01:17
pabelangernow I EOD01:18
*** baoli has joined #openstack-infra01:25
*** jascott1 has quit IRC01:26
*** psachin has joined #openstack-infra01:26
*** jascott1 has joined #openstack-infra01:26
*** andreas_s has joined #openstack-infra01:27
*** edmondsw has joined #openstack-infra01:28
*** jascott1 has quit IRC01:30
*** dbecker has joined #openstack-infra01:31
*** xarses has joined #openstack-infra01:31
fungipabelanger: right, that's what i meant by "provide more room"01:31
fungibut this manual rotate is just to get us through until then01:32
*** andreas_s has quit IRC01:36
*** dbecker has quit IRC01:38
*** xarses has quit IRC01:42
fungi#status log manually rotated all logs on zuulv3.openstack.org as a stop-gap to prevent a full rootfs later when scheduled log rotation kicks in; an additional 14gib were freed as a result01:43
openstackstatusfungi: finished logging01:43
*** kiennt26 has joined #openstack-infra01:43
*** cody-somerville has joined #openstack-infra01:43
*** cody-somerville has joined #openstack-infra01:43
*** baoli has quit IRC01:43
*** vhosakot has quit IRC01:44
*** ijw has quit IRC01:44
*** ijw has joined #openstack-infra01:44
*** baoli has joined #openstack-infra01:44
*** ijw has quit IRC01:44
fungithat should be plenty of breathing room until tomorrow01:44
*** ijw has joined #openstack-infra01:44
*** ijw has quit IRC01:44
*** andreas_s has joined #openstack-infra01:45
*** camunoz has quit IRC01:45
*** cody-somerville has quit IRC01:46
SamYapleyou dont want 160gb of swap and think we should use some of that for logs? thats crazy talk01:46
fungiheh01:47
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs  https://review.openstack.org/51072001:48
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs  https://review.openstack.org/51072001:49
*** andreas_s has quit IRC01:50
*** baoli has quit IRC01:50
*** andreas_s has joined #openstack-infra01:50
*** fanzhang has joined #openstack-infra01:51
openstackgerritMerged openstack-infra/irc-meetings master: Moving the Octavia meeting time and channel  https://review.openstack.org/51070101:53
*** andreas_s has quit IRC01:55
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs  https://review.openstack.org/51072001:55
clarkblooks like gerrit is happy now?02:04
SamYapleseems much more responsive02:05
clarkbI bet we can look at the task queue to start tracking that down02:05
tonybfungi: http://logs.openstack.org/85/858c988956fce8ac8ed8a202fe91aadbc0387395/release-post/tag-releases/8b44d50/console.html is that the same issue with mistral-lib ? or just a general git issue?02:05
fungiclarkb: well, it got restarted02:07
fungiso no idea if it would have cleared up on its own02:07
fungitonyb: that one looks different, and likely a transient network issue between signing01.ci.o.o and zm03.o.o02:10
tonybfungi: okay.  Can I re-trigger it? or is that another favor I need to ask?02:10
fungithe issue which hit mistral-lib should not recur on the signing01.ci node unless we rearrange zuul mergers again02:11
*** hongbin has quit IRC02:11
fungitonyb: i expect you just need that ref reenqueued into post right? or did other jobs also run successfully for it in post which cannot safely be rerun?02:11
fungi(well, in release-post)02:12
tonybfungi: I think reenqueued that ref is safe.  that job failed which measn the next was skipped and none of the per-repo jobs started, but to be frank I've never been the one awake to requiest this so I may be wrong.02:16
tonybhttp://lists.openstack.org/pipermail/release-job-failures/2017-October/000535.html  I don't know if that helps or not02:16
clarkbfungi: ah02:17
fungitonyb: cool, so there were only two jobs, the one which failed, and one depending on that which was subsequently skipped. thus, reenqueuing will rerun them both as desired02:19
*** dhinesh has quit IRC02:19
tonybThanks.  I'm glad I'm not too wide of the mark.02:19
*** liujiong has joined #openstack-infra02:20
fungitonyb: i've reenqueued it now02:22
fungithat seems to be the latest change to merge to the releases repo02:22
fungithough i expect that those jobs are idempotent, and so merging another may have just caught it up on whatever needed to be tagged02:23
*** thorst has quit IRC02:23
tonybfungi: Thanks. You're probably right.02:23
*** thorst has joined #openstack-infra02:24
*** neon_icon has quit IRC02:24
*** neon_icon has joined #openstack-infra02:24
SamYapleanyone want to review a bindep patch? https://review.openstack.org/#/c/506502/02:24
SamYaplewho doesnt like Parsley?02:24
*** neon_icon has quit IRC02:26
*** neon_icon has joined #openstack-infra02:27
*** neon_iconstar has joined #openstack-infra02:27
*** neon_icon has quit IRC02:27
*** thorst has quit IRC02:28
*** eharney has quit IRC02:31
*** dave-mccowan has quit IRC02:31
*** thorst has joined #openstack-infra02:37
*** thorst has quit IRC02:37
*** rwsu has quit IRC02:40
*** links has joined #openstack-infra02:44
*** nicolasbock_ has quit IRC02:47
*** kiennt26 has quit IRC02:49
*** pt_15 has quit IRC02:53
*** cody-somerville has joined #openstack-infra02:58
*** csomerville has joined #openstack-infra02:59
*** cody-somerville has quit IRC02:59
*** kiennt26 has joined #openstack-infra02:59
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Ceilometer  https://review.openstack.org/51045903:02
*** ihrachys has quit IRC03:03
*** ihrachys has joined #openstack-infra03:04
*** bobh has joined #openstack-infra03:10
*** bobh has quit IRC03:20
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: Add unbound role  https://review.openstack.org/51072503:21
*** lukebrowning_ has joined #openstack-infra03:23
*** edmondsw has quit IRC03:23
*** lukebrowning has quit IRC03:26
*** andreas_s has joined #openstack-infra03:27
*** andreas_s has quit IRC03:31
*** kiennt26 has quit IRC03:31
*** hongbin has joined #openstack-infra03:32
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Ceilometer  https://review.openstack.org/51045903:32
*** kiennt26 has joined #openstack-infra03:32
*** namnh has joined #openstack-infra03:32
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add unbound role to base job  https://review.openstack.org/51072803:34
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Ceilometer  https://review.openstack.org/51045903:38
*** gildub has quit IRC03:38
ianwhmm, my browser does not like that status page at all03:39
*** lukebrowning_ has quit IRC03:47
*** lukebrowning has joined #openstack-infra03:48
*** cody-somerville has joined #openstack-infra03:50
*** csomerville has quit IRC03:52
*** neon_iconstar has quit IRC03:53
*** ram5391 has joined #openstack-infra03:55
*** gildub has joined #openstack-infra03:56
*** mat128 has quit IRC03:58
SamYapleianw: yea every since zuulv3 dropped ive had to switch to a cli tool because my browser just wont take it03:59
*** jaosorior has joined #openstack-infra04:00
*** ykarel has joined #openstack-infra04:00
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Zaqar  https://review.openstack.org/51073104:04
*** ram5391 has quit IRC04:06
*** claudiub has joined #openstack-infra04:07
*** neon_iconstar has joined #openstack-infra04:07
*** coolsvap has joined #openstack-infra04:09
*** xarses has joined #openstack-infra04:12
*** gouthamr has quit IRC04:14
clarkbfirefox 57 beta mostly seems to keep up (though I cant check right now)04:16
*** neon_iconstar has quit IRC04:17
*** edmondsw has joined #openstack-infra04:21
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Zaqar legacy jobs  https://review.openstack.org/51073204:25
*** edmondsw has quit IRC04:26
*** lukebrowning_ has joined #openstack-infra04:27
*** lukebrowning has quit IRC04:29
*** pcaruana has joined #openstack-infra04:32
*** kiennt26 has quit IRC04:36
*** hongbin has quit IRC04:40
*** lukebrowning_ has quit IRC04:41
*** armaan has quit IRC04:41
*** armaan has joined #openstack-infra04:41
*** csomerville has joined #openstack-infra04:42
*** lukebrowning has joined #openstack-infra04:42
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Test unbound role  https://review.openstack.org/51073404:43
*** lukebrowning_ has joined #openstack-infra04:45
*** cody-somerville has quit IRC04:45
*** lukebrowning has quit IRC04:47
chandankumarAJaeger: hello04:52
chandankumarAJaeger: regarding this review https://review.openstack.org/#/c/510726/04:53
chandankumarAJaeger: the list of repo generates during run time when this script https://github.com/openstack/tempest/blob/master/tools/generate-tempest-plugins-list.py executes04:53
chandankumarAJaeger: and there will be multiple repos, it is a unique case how to handle that?04:54
*** udesale has joined #openstack-infra04:57
*** neon_icon has joined #openstack-infra04:58
*** pcaruana has quit IRC04:58
*** armax has quit IRC04:59
*** armax has joined #openstack-infra05:00
*** rwsu has joined #openstack-infra05:00
*** vsaienk0 has joined #openstack-infra05:03
*** armax has quit IRC05:04
*** jtomasek has joined #openstack-infra05:04
openstackgerritMerged openstack-infra/project-config master: Remove cron entries from nodepool launcher configs  https://review.openstack.org/51067605:07
*** csomerville has quit IRC05:07
*** ykarel is now known as ykarel|afk05:11
*** vsaienk0 has quit IRC05:12
*** yamahata has joined #openstack-infra05:20
openstackgerritChangBo Guo(gcb) proposed openstack-infra/project-config master: Remove project pylockfile from infrastructure systems  https://review.openstack.org/50865605:22
*** csomerville has joined #openstack-infra05:25
*** mwarad has joined #openstack-infra05:26
*** jaosorior has quit IRC05:27
snapiriHi infra, I have the strangest issue - in the following patch: https://review.openstack.org/#/c/508761/ I have tried to remove the zuul-clone for zuul v3 builds. Now both zuul and jenkins builds fail for a strange reason. it seems like git fails to find values that exist in the project git config. can anyone take a look or suggest a way to tackle it?05:27
*** cshastri has joined #openstack-infra05:35
*** neon_icon has quit IRC05:39
*** csomerville has quit IRC05:42
*** e0ne has joined #openstack-infra05:43
*** sree has joined #openstack-infra05:54
*** udesale__ has joined #openstack-infra05:55
*** udesale__ has quit IRC05:57
*** udesale__ has joined #openstack-infra05:57
*** udesale has quit IRC05:57
*** nmathew has joined #openstack-infra05:59
*** dhajare has joined #openstack-infra06:00
tonybsnapiri: I dropped a comment on the review.06:01
AJaegerchandankumar: please ask the rest of the team during US hours, I have no quick fix for you.06:01
chandankumarAJaeger: sure :-)06:01
*** jaosorior has joined #openstack-infra06:03
*** jrist has quit IRC06:03
*** andreas_s has joined #openstack-infra06:05
*** sree has quit IRC06:07
*** edmondsw has joined #openstack-infra06:10
*** sree_ has joined #openstack-infra06:11
*** sree_ is now known as Guest6217206:11
*** eumel8 has joined #openstack-infra06:11
*** yamamoto has quit IRC06:11
snapiritonyb: thanks06:12
eumel8morning06:12
*** edmondsw has quit IRC06:14
*** vsaienk0 has joined #openstack-infra06:15
*** neon_icon has joined #openstack-infra06:15
*** rlandy|bbl is now known as rlandy06:16
*** rlandy has quit IRC06:16
*** jrist has joined #openstack-infra06:16
*** hemna_ has quit IRC06:18
*** Guest62172 has quit IRC06:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove legacy manuals jobs  https://review.openstack.org/51076106:21
AJaegermorning, eumel806:21
*** spectr has joined #openstack-infra06:22
*** sshnaidm|off is now known as sshnaidm06:22
AJaegerianw, if you have some review time, could you review https://review.openstack.org/#/c/510681/ https://review.openstack.org/510761  and https://review.openstack.org/#/c/510623/ , please?06:22
eumel8morning, AJaeger, the man who never sleeps :)06:23
eumel8AJaeger: is the new translation job working?06:24
zhongjunHi anyone could help me to find why manila tempest config options cannot synchronize to new tempest job in new zuul check.06:26
zhongjunThe original tempest conf file in manila-tempest-minimal-dsvm-dummy job belong to  original Jenkins check:06:27
zhongjunhttp://logs.openstack.org/42/510542/1/check/legacy-manila-tempest-minimal-dsvm-dummy/2cb9f4d/logs/tempest_conf.txt.gz06:27
zhongjunCurrently, the tempest conf in manila-tempest-minimal-dsvm-dummy job belong to  new zuul check:  http://logs.openstack.org/42/510542/1/check/legacy-manila-tempest-minimal-dsvm-dummy/2cb9f4d/logs/tempest_conf.txt.gz06:27
zhongjunWe lack of some options about [share] node in tempest conf file in our new tempest gate job for zuul check. Do we have some patches to fix this?06:28
zhongjunThe original tempest conf file in manila-tempest-minimal-dsvm-dummy job belong to  original Jenkins check:  http://logs.openstack.org/42/510542/1/check/gate-manila-tempest-minimal-dsvm-dummy-ubuntu-xenial/81a1434/logs/tempest_conf.txt.gz06:31
*** pt_15 has joined #openstack-infra06:31
AJaegereumel8: zuul has too much work and too little nodes, the jobs are not run yet. So, no idea ;(06:33
eumel8AJaeger: thats okay. I mean in general06:34
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove legacy jobs from Kuryr-libnetwork  https://review.openstack.org/50998706:36
*** aeng has quit IRC06:36
*** ykarel|afk is now known as ykarel06:38
AJaegereumel8: really no idea. It looks good in theory, waiting for practice run ;)06:40
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs  https://review.openstack.org/51076506:40
*** kiennt26 has joined #openstack-infra06:42
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs  https://review.openstack.org/51076506:44
dmelladoAJaeger: ping re https://review.openstack.org/#/c/509383/06:46
dmelladowe'd need to remove the legacy jobs? is that what you're referring about?06:46
*** xarses has quit IRC06:51
AJaegerdmellado: yes, similar to https://review.openstack.org/509987 and https://review.openstack.org/509986 - you need to *move* the jobs around. So, you added them, let's now remove them from central repos to not run them twice.06:51
dmelladoAJaeger: ack, will follow up with a patch, thanks!06:52
AJaegerthanks, dmellado (actually: two patches, please - one for each repo ;)06:54
openstackgerritDaniel Mellado proposed openstack-infra/project-config master: Remove legacy jobs from kuryr-k8s  https://review.openstack.org/51076906:55
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Migrate Kuryr-kubernetes legacy jobs  https://review.openstack.org/51077106:56
AJaegerdmellado: kien is also working on it ^, can you discuss with him who will abandon? ^06:57
openstackgerritKien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Migrate Kuryr-kubernetes legacy jobs  https://review.openstack.org/51077306:57
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Migrate Kuryr-kubernetes legacy jobs  https://review.openstack.org/51077106:58
*** rcernin has joined #openstack-infra06:58
*** gildub has quit IRC06:59
dmelladokiennt26:07:00
dmelladoI don't mind abandoning mine but please do note that we had a blueprint for this07:00
dmelladoand you didn't sign up as an assignee07:00
dmelladoso for next times please dont work on 'rogue' mode07:00
kiennt26dmellado: yeah, I already abandoned my patches xD sr about it07:01
dmelladokiennt26: np ;)07:02
dmelladothanks for your help! but I'd like to address this in a sane mode07:02
dmelladokiennt26: I'll create a small blueprint for all the multinode gates07:02
dmelladolet's split the work there ;)07:02
*** dhajare has quit IRC07:03
*** dhajare has joined #openstack-infra07:03
kiennt26dmellado: cool ;)07:03
AJaegerdmellado, kiennt26, we need 510773 - should we restore it and correct dependency?07:03
dmelladoAJaeger: yah, np from my side07:04
AJaegerkiennt26: can you do that, please? ^07:04
*** pcaruana has joined #openstack-infra07:04
*** jaosorior has quit IRC07:04
kiennt26AJaeger, dmellado: Sure07:04
*** ykarel has quit IRC07:05
openstackgerritKien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Migrate Kuryr-kubernetes legacy jobs  https://review.openstack.org/51077307:06
*** nmathew has quit IRC07:07
kiennt26AJaeger, dmellado: Done! ^^07:07
dmelladokiennt26: thanks!07:07
dmelladoAJaeger: thanks also for all your help on the zuulness so far ;)07:07
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: Add unbound role  https://review.openstack.org/51072507:08
dmelladoich schulde dir ein Bier :P07:08
AJaegerdmellado: Danke! ;)07:09
AJaegerkiennt26: great, thanks.07:09
*** nmathew has joined #openstack-infra07:09
AJaegerianw, yolanda,could you putw https://review.openstack.org/#/c/510681/ https://review.openstack.org/510761  and https://review.openstack.org/#/c/510623/ on your review queues as time permits, please?07:10
* AJaeger will be back later...07:11
*** ykarel has joined #openstack-infra07:12
*** claudiub|2 has joined #openstack-infra07:15
*** e0ne has quit IRC07:17
*** Hal has joined #openstack-infra07:17
*** Hal is now known as Guest7875107:17
*** claudiub has quit IRC07:18
*** yamahata has quit IRC07:18
*** sileht has quit IRC07:23
*** sileht has joined #openstack-infra07:23
openstackgerritThomas Morin proposed openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn)  https://review.openstack.org/51043507:23
openstackgerritMerged openstack-infra/project-config master: Remove legacy jobs from Kuryr-libnetwork  https://review.openstack.org/50998707:24
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Zaqar legacy jobs  https://review.openstack.org/51073207:27
*** tesseract has joined #openstack-infra07:27
openstackgerritMerged openstack-infra/project-config master: use announce-release everywhere  https://review.openstack.org/51068107:28
*** kjackal_ has joined #openstack-infra07:33
*** dizquierdo has joined #openstack-infra07:33
openstackgerritMerged openstack-infra/tripleo-ci master: Fix TripleO CI jobs  https://review.openstack.org/50866007:33
*** jaosorior has joined #openstack-infra07:33
*** jpich has joined #openstack-infra07:35
*** shardy has joined #openstack-infra07:35
*** dhajare has quit IRC07:38
*** dhajare has joined #openstack-infra07:38
*** egonzalez has joined #openstack-infra07:39
*** electrofelix has joined #openstack-infra07:39
*** jpena|off is now known as jpena07:41
*** kiennt26 has quit IRC07:43
*** kiennt26 has joined #openstack-infra07:43
*** ralonsoh has joined #openstack-infra07:44
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Test unbound role  https://review.openstack.org/51073407:44
*** gildub has joined #openstack-infra07:48
*** askb has quit IRC07:49
openstackgerritThomas Morin proposed openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn)  https://review.openstack.org/51043507:55
*** edmondsw has joined #openstack-infra07:58
*** threestrands has quit IRC07:59
*** edmondsw has quit IRC08:02
openstackgerritMerged openstack-infra/project-config master: Remove legacy manuals jobs  https://review.openstack.org/51076108:13
*** sree has joined #openstack-infra08:15
*** sree has quit IRC08:20
*** lin_yang has quit IRC08:21
*** pratik has joined #openstack-infra08:23
*** lucas-afk is now known as lucasagomes08:25
*** s-shiono has quit IRC08:27
*** inc0 has quit IRC08:32
*** pratik has quit IRC08:32
*** dbecker has joined #openstack-infra08:33
*** sree has joined #openstack-infra08:34
*** derekh has joined #openstack-infra08:36
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Add reno for release notes management  https://review.openstack.org/51067908:37
*** askb has joined #openstack-infra08:37
openstackgerritDaniel Mellado proposed openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin  https://review.openstack.org/51078608:38
*** dingyichen has quit IRC08:38
*** inc0 has joined #openstack-infra08:39
*** priteau has joined #openstack-infra08:40
*** e0ne has joined #openstack-infra08:40
*** ryotagami has joined #openstack-infra08:44
*** priteau has quit IRC08:46
*** priteau has joined #openstack-infra08:46
*** tosky has joined #openstack-infra08:48
*** andreas_s has quit IRC08:48
*** ykarel is now known as ykarel|lunch08:48
*** priteau_ has joined #openstack-infra08:49
*** priteau has quit IRC08:51
*** pt_15 has quit IRC09:02
openstackgerritDao Cong Tien proposed openstack-infra/openstack-zuul-jobs master: grenade-dsvm-ironic: Move task 'legacy workspace dir' to first  https://review.openstack.org/51079509:03
*** gildub has quit IRC09:06
*** gildub has joined #openstack-infra09:07
*** andreas_s has joined #openstack-infra09:07
snapirihi infra, I have a job that seems to be stuck on the zuul3. it is already running for over 20 hours and does not seem to progress: patch 509189,8. Can you suggest what to do?09:08
*** aspiers has joined #openstack-infra09:12
dmelladosnapiri: I'd abandon + recover the job just to check if it unblocks it.... happened to me last week09:14
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Make version_string() lazy load version info  https://review.openstack.org/45091809:15
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Zaqar  https://review.openstack.org/51073109:17
*** kiennt26 has quit IRC09:19
snapiridmellado: thanks for the answer, isn't a recheck sufficient?09:19
dmelladosnapiri: if it got really stuck and you don't want or can't wait for the timeout, nope :(09:20
*** sambetts_ is now known as sambetts09:21
snapiridmellado: thanks09:21
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Discover Distribution through the class hierarchy  https://review.openstack.org/39918809:24
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Use super for a cleaner resolution of the superclass.  https://review.openstack.org/41222409:24
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Remove unnecessary 'if True'  https://review.openstack.org/51080609:24
AJaegersnapiri: we have a very high load on Zuul v3, just wait...09:27
*** spiffxp has quit IRC09:28
*** vipul has quit IRC09:28
*** witek has quit IRC09:28
AJaegersnapiri: you're enquing at end of queue then, will have to wait another 20 hours09:28
AJaegersnapiri: you can check status at zuulv3.openstack.org09:28
*** spectr has quit IRC09:29
snapiriAJaeger: will wait. note that the job has some post_failure09:29
*** gildub has quit IRC09:29
*** sree has quit IRC09:29
*** spectr has joined #openstack-infra09:29
*** david-lyle has quit IRC09:29
*** pcrews has quit IRC09:29
AJaegersnapiri: your change is not testing what you expect. A depends-on for project-config will not work, it's a trusted repository and does not allow dynamic reconfiguration.09:31
*** pcrews has joined #openstack-infra09:31
AJaegeryolanda, ianw, could one of you +2A https://review.openstack.org/#/c/510531/ , please?09:31
AJaegersnapiri: just wait until the dependencies are in and then test again, please09:32
*** spectr has quit IRC09:32
dirkdo we need https://review.openstack.org/#/c/510565/ before the zuulv3 switch?09:33
snapiriAJaeger: thanks!09:33
*** spiffxp has joined #openstack-infra09:33
AJaegerdirk: Nice to have working jobs ;) That change misses Zuul v3 +1, let's wait for that - but then get it in, please09:34
*** spectr has joined #openstack-infra09:34
*** armaan has quit IRC09:35
*** armaan has joined #openstack-infra09:35
dirkAJaeger: the zuulv3 check queue is pretty long ;) and I think it doesn't actually run this code (its only executed when other repos use the check-requirements job)09:36
*** vipul has joined #openstack-infra09:36
AJaegerdirk: I see. Let's see what mordred thinks ^09:40
*** nmathew- has joined #openstack-infra09:41
*** thorst has joined #openstack-infra09:41
*** nmathew has quit IRC09:44
*** edmondsw has joined #openstack-infra09:46
toskyAJaeger: sooo, I had two weird failures yesterday on zuulv309:46
*** thorst has quit IRC09:47
toskyin https://review.openstack.org/#/c/510528/, legacy-sahara-dsvm-scenario-neutron (and the wsgi version) shows a strange "Permission denied: '/home/zuul/.cache/pip/wheels/ca/1e/92/53679e3334df7f41c24014e01111f6495a586e003db0ffc7ba'"09:47
toskyand legacy-tempest-dsvm-sahara had a post_failure, so maybe that's infrastructural related09:48
*** mwarad has quit IRC09:48
*** edmondsw has quit IRC09:51
*** kiennt26 has joined #openstack-infra09:52
*** kiennt26 has quit IRC09:53
*** dtantsur|afk is now known as dtantsur09:53
*** alexchadin has joined #openstack-infra09:54
*** liujiong has quit IRC09:54
*** wolverineav has joined #openstack-infra09:55
AJaegertosky: please followup with rest of team later on09:56
toskyack09:58
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project  https://review.openstack.org/50911909:58
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project  https://review.openstack.org/50911910:00
*** nmathew- has quit IRC10:01
*** ykarel|lunch is now known as ykarel10:02
*** andreas_s has quit IRC10:03
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project  https://review.openstack.org/50911910:03
*** tbarron|PTO is now known as tbarron10:04
*** cuongnv has quit IRC10:05
*** wolverineav has quit IRC10:07
robcresswello/ Has anyone run into an issue with Ocata and Centos 7 and libvirt-python failing to build / install ?10:07
*** andreas_s has joined #openstack-infra10:07
*** lukebrowning_ has quit IRC10:11
*** pbourke has quit IRC10:11
*** sdague has joined #openstack-infra10:12
*** pbourke has joined #openstack-infra10:12
openstackgerritDuong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Craton  https://review.openstack.org/51082010:13
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Added pytest support to setup.py test  https://review.openstack.org/43817610:16
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server  https://review.openstack.org/51036210:16
*** andreas_s has quit IRC10:17
*** andreas_s has joined #openstack-infra10:18
openstackgerritDuong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Craton  https://review.openstack.org/51082410:19
kazshAJaeger: thanks for your comment, I've updated my change, please check it when you have time.10:19
*** xinni9e has joined #openstack-infra10:20
ianwjeblair / fungi / dmsimard : as discussed this morning, updates on the unbound configuration in the zuulv3-issues page.  i'm sure it will want some review rounds (ci hasn't reported and i'm out) but don't think it's a blocker10:22
*** lukebrowning has joined #openstack-infra10:22
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove unnecessary scripts from tripleo-ci  https://review.openstack.org/51082810:24
AJaegerthanks, ianw and good night!10:25
*** kjackal_ has quit IRC10:26
*** neon_icon has quit IRC10:26
*** lukebrowning has quit IRC10:27
*** jascott1 has joined #openstack-infra10:27
AJaegerkazsh: you need to split it in two changes10:28
kazshAJaeger: oh ok, will do, thanks!10:29
*** gildub has joined #openstack-infra10:29
*** wolverineav has joined #openstack-infra10:30
*** andreas_s has quit IRC10:31
*** jascott1 has quit IRC10:32
*** andreas_s has joined #openstack-infra10:32
*** panda is now known as panda|relo10:36
sshnaidmdoes zuulv3 run now? I don't see legacy-* jobs anymore10:36
*** andreas_s has quit IRC10:36
toskyit is running, but there is a huge queue10:37
AJaegersshnaidm: it does - but only 1400+ changes in the queue due to the small quota we gave it.10:38
*** kjackal_ has joined #openstack-infra10:38
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add publish-to-pypi-neutron template  https://review.openstack.org/51053110:42
*** andreas_s has joined #openstack-infra10:43
*** pblaho has joined #openstack-infra10:43
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project  https://review.openstack.org/50911910:43
*** alexchadin has quit IRC10:46
*** alexchadin has joined #openstack-infra10:47
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat dashboard creation  https://review.openstack.org/51083310:49
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation  https://review.openstack.org/51083510:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Move infra-gate pipeline after gate  https://review.openstack.org/51083610:58
*** ociuhandu has joined #openstack-infra10:59
alexchadinhi, can I mark own jobs as non voting in .zuul file?11:00
alexchadinif I do so, Zuul throws an error "mapping values are not allowed in this context"11:01
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation  https://review.openstack.org/51083511:03
AJaegeralexchadin: which change?11:03
AJaegeralexchadin: I guess you miss a ":" - let me check...11:03
*** jkilpatr has joined #openstack-infra11:04
*** dtantsur has quit IRC11:04
*** nicolasbock_ has joined #openstack-infra11:04
AJaegeralexchadin: commented on https://review.openstack.org/#/c/50977711:05
AJaegeryou indeed need a colon11:05
*** ykarel_ has joined #openstack-infra11:06
*** alexchadin has quit IRC11:07
*** ykarel has quit IRC11:08
*** dtantsur has joined #openstack-infra11:08
*** dtantsur has quit IRC11:08
*** dtantsur has joined #openstack-infra11:09
*** dtantsur has quit IRC11:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Stop using the cached cirros image  https://review.openstack.org/51083911:09
*** dtantsur has joined #openstack-infra11:13
*** andreas_s has quit IRC11:13
*** Guest78751 has quit IRC11:14
*** lucasagomes is now known as lucas-hungry11:15
*** spiffxp has quit IRC11:16
*** alexchadin has joined #openstack-infra11:16
*** nicolasbock_ has quit IRC11:16
*** vipul has quit IRC11:16
*** pcrews has quit IRC11:16
*** vipul has joined #openstack-infra11:17
*** spiffxp has joined #openstack-infra11:17
*** pcrews has joined #openstack-infra11:17
*** andreas_s has joined #openstack-infra11:18
*** andreas_s has quit IRC11:22
*** andreas_s has joined #openstack-infra11:23
*** lukebrowning has joined #openstack-infra11:23
*** jpena is now known as jpena|lunch11:23
*** ykarel_ is now known as ykarel11:24
*** nicolasbock_ has joined #openstack-infra11:28
*** rhallisey has joined #openstack-infra11:29
*** sree has joined #openstack-infra11:30
*** andreas_s has quit IRC11:32
*** sree has quit IRC11:34
*** andreas_s has joined #openstack-infra11:37
*** ldnunes has joined #openstack-infra11:38
*** yamamoto has joined #openstack-infra11:38
*** jaypipes has joined #openstack-infra11:39
openstackgerritAlexander Chadin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy watcher jobs  https://review.openstack.org/51084811:42
*** yamamoto has quit IRC11:44
*** yamamoto has joined #openstack-infra11:44
openstackgerritAlexander Chadin proposed openstack-infra/project-config master: Remove legacy watcher jobs  https://review.openstack.org/51084911:47
*** yamamoto has quit IRC11:47
*** andreas_s has quit IRC11:48
*** andreas_s has joined #openstack-infra11:48
*** yamamoto has joined #openstack-infra11:48
*** yamamoto has quit IRC11:49
openstackgerritAlexander Chadin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy watcher jobs  https://review.openstack.org/51084811:49
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Google Cal Sync  https://review.openstack.org/51085011:51
openstackgerritAlexander Chadin proposed openstack-infra/project-config master: Remove legacy watcher jobs  https://review.openstack.org/51084911:51
openstackgerritMerged openstack-infra/openstackid-resources master: Google Cal Sync  https://review.openstack.org/51085011:51
*** geekinutah has quit IRC11:54
*** geekinutah has joined #openstack-infra11:54
*** MasterPiece2 has joined #openstack-infra11:54
*** yamamoto has joined #openstack-infra11:55
*** trown|outtypewww is now known as trown11:55
*** panda|relo is now known as panda11:56
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Google Cal Sync  https://review.openstack.org/51085311:58
*** eumel8 has quit IRC12:00
*** edmondsw has joined #openstack-infra12:03
*** edmondsw has quit IRC12:04
*** priteau_ has quit IRC12:04
*** edmondsw_ has joined #openstack-infra12:05
*** tpsilva has joined #openstack-infra12:06
*** zzzeek has quit IRC12:07
*** sree has joined #openstack-infra12:07
openstackgerritMerged openstack-infra/openstackid-resources master: Google Cal Sync  https://review.openstack.org/51085312:08
*** zzzeek has joined #openstack-infra12:10
*** thorst has joined #openstack-infra12:16
*** alexchadin has quit IRC12:17
*** Hal has joined #openstack-infra12:22
*** Hal is now known as Guest8459012:22
*** eharney has joined #openstack-infra12:24
*** namnh has quit IRC12:25
*** lucas-hungry is now known as lucasagomes12:27
*** yamamoto has quit IRC12:27
*** dprince has joined #openstack-infra12:28
*** yamamoto has joined #openstack-infra12:29
*** rlandy has joined #openstack-infra12:32
*** yamamoto has quit IRC12:33
*** mat128 has joined #openstack-infra12:34
*** esberglu has quit IRC12:38
*** alexchadin has joined #openstack-infra12:39
*** makowals has joined #openstack-infra12:40
*** jpena|lunch is now known as jpena12:40
*** Klerisson has joined #openstack-infra12:41
*** udesale__ has quit IRC12:44
mordredmorning AJaeger ! anything urgent/immedate I should look at?12:44
*** Klerisson has left #openstack-infra12:46
*** LindaWang has quit IRC12:46
*** baoli has joined #openstack-infra12:49
*** thorst_ has joined #openstack-infra12:49
*** thorst has quit IRC12:49
*** esberglu has joined #openstack-infra12:52
*** Kevin_Zheng has joined #openstack-infra12:52
*** spectr has quit IRC12:53
AJaegermordred: a couple of jobs seem hanging, see infra-check, but nothing immediate otherwise.12:53
AJaegerlots of changes to review - and a far too long queue in v3, we might need to restart it12:54
*** spectr has joined #openstack-infra12:55
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul-jobs master: Remove integration test playbooks from zuul-jobs  https://review.openstack.org/51086812:56
*** kashyap has left #openstack-infra12:56
*** gildub has quit IRC12:56
*** zhurong has joined #openstack-infra12:57
*** efried has quit IRC12:58
*** shardy has quit IRC13:00
*** edmondsw_ is now known as edmondsw13:01
*** ykarel has quit IRC13:01
*** ykarel has joined #openstack-infra13:02
*** kzaitsev1pi has quit IRC13:05
*** ldnunes has quit IRC13:05
*** david-lyle has joined #openstack-infra13:06
*** kzaitsev_pi has joined #openstack-infra13:06
*** LindaWang has joined #openstack-infra13:11
*** jistr is now known as jistr|mtg13:11
*** shardy has joined #openstack-infra13:11
*** ykarel_ has joined #openstack-infra13:11
*** ykarel has quit IRC13:12
*** ykarel_ is now known as ykarel13:12
*** lihi has quit IRC13:12
*** kgiusti has joined #openstack-infra13:12
*** leyal has quit IRC13:13
AJaegermordred: I proposed https://review.openstack.org/510836 in the hope this changes the display on zuulv3.o.o - think this is usefull for our switch to v3 tomorrow13:13
*** dizquierdo has quit IRC13:13
mordredAJaeger: worth a shot! :)13:14
*** dimak has quit IRC13:14
*** oanson has quit IRC13:14
AJaeger;)13:14
*** dimak has joined #openstack-infra13:15
*** leyal has joined #openstack-infra13:15
*** oanson has joined #openstack-infra13:15
*** lihi has joined #openstack-infra13:16
*** ldnunes has joined #openstack-infra13:18
*** jistr|mtg is now known as jistr13:19
openstackgerritDavid Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Integration tests: run mirror-info before configure-mirrors  https://review.openstack.org/51088113:20
dmsimardianw: thanks for picking up unbound, was on pto yesterday13:21
dmsimardianw: left a batch of reviews13:21
*** mriedem has joined #openstack-infra13:22
*** baoli has quit IRC13:23
fungimordred: i'm still catching up, but if you didn't see during the zuul meeting yesterday we decided to start the cut-over to v3 at 11:00z tomorrow. one of our top priorities for today is to get refreshed versions of the changes we reverted to un-revert (also we decided to keep the infra-check pipeline in the near term, but leave only project-config in it)13:23
dmsimardianw: I tagged your reviews with https://review.openstack.org/#/q/topic:zuulv3-unbound to make them easier to track13:23
*** baoli has joined #openstack-infra13:24
AJaegerfungi, I read the notes - and that's why I pushed 510836 to make it easier to find ;)13:24
fungithanks AJaeger!13:25
*** lbragstad has joined #openstack-infra13:28
*** shardy has quit IRC13:29
*** links has quit IRC13:29
*** ccamacho has left #openstack-infra13:30
mordredfungi: yes! I saw the meeting summary - and also that I was assigned to prepare the revert patches I believe13:31
*** bobh has joined #openstack-infra13:33
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove infra-gate pipelines  https://review.openstack.org/51088513:33
*** dave-mccowan has joined #openstack-infra13:33
fungimordred: i don't know that you were _formally_ assigned, it was just suggested that we should ask you whether you want to work on that today ;)13:33
*** zaneb has quit IRC13:33
AJaegerfungi: pabelanger took the task ^13:34
mordredpabelanger: you might want to rebase https://review.openstack.org/#/c/510885/ on top of https://review.openstack.org/51083613:34
fungithanks pabelanger!13:34
mordredpabelanger: and also thanks!13:34
AJaegermordred: 510836 and 510885 are different repos13:34
fungii have a sort of full morning still, but will hopefully be able to start reviewing between lunch and the infra meeting13:34
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Remove infra-gate pipeline  https://review.openstack.org/51088613:35
pabelangerlooking13:35
toskymordred: hi! I guess that you didn't see my note above, basically related to the two different failures in https://review.openstack.org/#/c/510528/13:35
*** armaan has quit IRC13:35
toskya "Permission denied" writing /home/zuul/.cache/... something for legacy-sahara-dsvm-scenario-neutron, and a post_failure apparently not related to the job for legacy-tempest-dsvm-sahara13:35
*** armaan has joined #openstack-infra13:36
AJaegerpabelanger: just +2A 510836 before you remove the other pipelines ;)13:36
pabelangerAJaeger: mordred: we're removing infra-gate, so does it makesense to move it around in config this morning?13:36
AJaegerpabelanger: we leave infra-gate for project-config!13:37
*** baoli has quit IRC13:37
pabelangerAJaeger: Oh, right13:37
pabelangerblah13:37
pabelangerno coffee yet13:37
AJaegerpabelanger: and thus I'd like to have it nicely grouped and able to find13:37
* AJaeger sends coffee to pabelanger13:37
pabelanger+313:37
AJaegerthanks13:38
*** shardy has joined #openstack-infra13:38
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove infra-gate pipeline  https://review.openstack.org/51088713:39
AJaegerpabelanger: let's WIP that until tomorrow - ok ? ^13:40
mordredAJaeger, pabelanger: wait- we're only supposed to leave infra-check13:41
pabelangerAJaeger: I don't think it will be an issue, will it? We have no other projects using gate on zuulv3 now13:41
*** ramishra has quit IRC13:41
AJaegermordred: Argh, infra-check, not infra-gate. ;(((((((((13:41
AJaegerSorry...13:41
mordredAJaeger, pabelanger: for project-config - because infra-check is high priority vs. normal check13:41
pabelangeryah, that's what I thought13:42
AJaegermordred: now I need coffee ;) You're right...13:42
mordredAJaeger: I need MORE coffee13:42
* AJaeger abandons 51083613:42
*** jstockto2000 has joined #openstack-infra13:42
fungimoar coffieee all around13:42
pabelangerokay, let me fix up13:43
AJaegerpabelanger: if you merge infra-gate -> gate now, nothing will merge there- we have gate disabled. So, let's prepare and WIP.13:43
pabelangerAJaeger: Oh, right. because of voting13:43
pabelangerfollowing now13:43
pabelangerYa, I'll WIP until we are ready13:44
openstackgerritKen Giusti proposed openstack-infra/openstack-zuul-jobs master: Remove most of the oslo.messaging legacy tests  https://review.openstack.org/51066113:44
AJaegerpabelanger: let me do the project-config change and reorder at same time, ok?13:44
pabelangerAJaeger: sure13:44
*** jstockto2000 has quit IRC13:44
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Remove infra-check and infra-gate pipelines  https://review.openstack.org/51088613:45
*** zhurong has quit IRC13:45
*** zaneb has joined #openstack-infra13:45
*** gouthamr has joined #openstack-infra13:45
pabelangerAJaeger: don't forget infra-post13:46
pabelangerwe likely also want to remove that13:46
*** jstockto2000 has joined #openstack-infra13:47
*** chlong_ has joined #openstack-infra13:47
*** vsaienk0 has quit IRC13:47
*** chlong has joined #openstack-infra13:48
*** ihrachys_ has joined #openstack-infra13:48
*** ihrachys_ has quit IRC13:48
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Revert "Use new infra pipelines"  https://review.openstack.org/51088513:48
*** Marx314 has joined #openstack-infra13:49
*** Marx314 has quit IRC13:49
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Revert "Use new infra pipelines"  https://review.openstack.org/51088613:49
*** zaneb has quit IRC13:50
*** Marx314 has joined #openstack-infra13:51
*** jstockto2000 has quit IRC13:52
*** igordc has quit IRC13:52
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Revert "Use new post pipelines"  https://review.openstack.org/51088713:52
*** kiennt26 has joined #openstack-infra13:53
pabelangerI used topic:zuulv3-rollout13:54
*** hongbin has joined #openstack-infra13:54
*** andreas_s has quit IRC13:54
*** aviau has quit IRC13:55
*** aviau has joined #openstack-infra13:55
*** igordc has joined #openstack-infra13:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines  https://review.openstack.org/51089213:56
AJaegerpabelanger: ^13:56
AJaegerpabelanger: should be all good but please carefully review13:56
pabelangerAJaeger: sure, I will fetch coffee first13:57
AJaeger;)13:57
*** armax has joined #openstack-infra13:58
*** andreas_s has joined #openstack-infra13:59
*** vsaienk0 has joined #openstack-infra13:59
*** d0ugal has quit IRC14:00
*** d0ugal has joined #openstack-infra14:00
*** d0ugal has quit IRC14:00
*** d0ugal has joined #openstack-infra14:00
*** d0ugal has quit IRC14:00
*** d0ugal has joined #openstack-infra14:00
*** d0ugal has quit IRC14:01
*** d0ugal has joined #openstack-infra14:02
*** d0ugal has quit IRC14:02
*** d0ugal has joined #openstack-infra14:02
*** alexchadin has quit IRC14:03
*** ijw has joined #openstack-infra14:03
AJaegera few cleanups and updates to approve if anybody has time (all project-config): https://review.openstack.org/#/c/510659/  https://review.openstack.org/510786 https://review.openstack.org/#/c/510347/ https://review.openstack.org/510435 https://review.openstack.org/510769 https://review.openstack.org/510638 https://review.openstack.org/#/c/510659/ https://review.openstack.org/51063614:04
*** lihi has quit IRC14:05
*** baoli has joined #openstack-infra14:05
*** pblaho has quit IRC14:05
*** felipemonteiro has joined #openstack-infra14:05
*** leyal has quit IRC14:06
*** dimak has quit IRC14:06
*** oanson has quit IRC14:06
*** felipemonteiro_ has joined #openstack-infra14:06
*** dimak has joined #openstack-infra14:07
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs  https://review.openstack.org/51076514:07
*** leyal has joined #openstack-infra14:07
*** oanson has joined #openstack-infra14:07
AJaegerand for openstack-zuul-jobs: https://review.openstack.org/509986 and https://review.openstack.org/51063714:07
AJaegerdmsimard: question by jeblair on https://review.openstack.org/#/c/504788/14:08
*** lihi has joined #openstack-infra14:08
dmsimardYes, I'll address it soon. Catching up from 3 day weekend :)14:08
AJaeger;)14:08
*** felipemonteiro has quit IRC14:10
*** andreas_s has quit IRC14:12
pabelangerAJaeger: lets do 2 patches for 510892. 1st move jobs back into check / gate / post, 2nd delete pipelines.14:16
pabelangerthen we can setup Depends-On for 2nd project-config patch14:16
*** hemna_ has joined #openstack-infra14:16
prometheanfireare zuulv3 jobs running? https://review.openstack.org/51056514:16
*** alexchadin has joined #openstack-infra14:16
pabelangerchecking14:17
AJaegerprometheanfire: check zuulv3.status.openstack.org - currently 1600+ changes in queue, it takes ages ;(14:18
prometheanfireoh, lol14:18
*** thorst_ has quit IRC14:18
AJaegerprometheanfire: too small quota14:18
prometheanfireI thought it was keeping up for a while14:19
pabelangerYa, something might be up. I see a node with 19hr in-use14:19
pabelangerI think we have a hung executor ago14:20
pabelangeragain*14:20
pabelangerze03.o.o is hung14:20
pabelangerchecking why14:20
*** pblaho has joined #openstack-infra14:20
AJaegerpabelanger: why two changes? We cannot merge them separetly!14:21
pabelangerzuul     28367  0.0  0.0  21352  3164 ?        S    Oct09   0:00 git fetch-pack --stateless-rpc --stdin --lock-pack --thin --no-progress https://git.openstack.org/openstack/tripleo-quickstart/14:21
pabelangerlooks like git timeout didn't work as expected14:21
AJaegerpabelanger: Once we merge the first, the gate pipeline is disabled and we can only merge the second - with force-merge.14:22
*** kiennt26 has quit IRC14:22
pabelangerAJaeger: okay, then you see to setup depends-on to my reverts above14:22
AJaegerpabelanger: yes, will do that...14:23
pabelangerI'm going to kill the git process and confirm the executor starts working again, if so, we should then get jeblair hard git timeout landed this morning and restart all executors14:23
AJaegerpabelanger: but it *cannot* merge without those either - since the pipelines are in use...14:23
*** zaneb has joined #openstack-infra14:24
AJaegerbut let's make it explicit14:24
pabelangerAJaeger: right, for all this to work, we need to make gate pipeline functional again. But we cannot do that until we stop zuulv314:24
pabelangererr14:24
pabelangerzuulv214:24
pabelangerso, will be a little bit of dancing to land all this14:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines  https://review.openstack.org/51089214:25
pabelangerokay, ze03 working again, which is good14:25
AJaegeryeah!14:25
pabelangerze04 also blocked14:25
*** edmondsw has quit IRC14:25
* AJaeger added all his review requests to the etherpad now14:25
AJaegerpabelanger: argh ;(14:26
*** edmondsw has joined #openstack-infra14:26
AJaegerthanks for checking and restarting, pabelanger14:26
*** andreas_s has joined #openstack-infra14:27
AJaegerwow, 1004 changes in check queue ;(14:27
*** edmondsw_ has joined #openstack-infra14:27
*** srobert has joined #openstack-infra14:27
*** dhinesh has joined #openstack-infra14:28
*** edmonds__ has joined #openstack-infra14:28
pabelangerokay, in total ze03, ze04, ze05 and ze06 were all blocked14:29
pabelangerI have left ze04 blocked so we can audit it when jeblair is online14:29
pabelangerI am not sure why our gitconfig settings didn't work14:29
*** edmondsw has quit IRC14:30
*** camunoz has joined #openstack-infra14:30
jeblairpabelanger: please fix ze0414:31
pabelangerack14:31
*** thorst has joined #openstack-infra14:31
*** edmondsw_ has quit IRC14:31
pabelangerand processing again14:32
*** claudiub|2 has quit IRC14:33
*** jamesdenton has joined #openstack-infra14:34
AJaegeryeah! I see progress on some hung ones ;)14:35
*** chlong has quit IRC14:36
*** chlong_ has quit IRC14:36
*** chlong has joined #openstack-infra14:36
*** alexchadin has quit IRC14:37
openstackgerritMerged openstack-infra/elastic-recheck master: Retry change queries on a 502 response  https://review.openstack.org/51070614:38
*** pblaho has quit IRC14:38
*** pblaho has joined #openstack-infra14:41
*** pblaho has quit IRC14:41
*** pblaho has joined #openstack-infra14:41
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual master: Add announce-release job to stay in project-config  https://review.openstack.org/51090514:42
AJaegermordred: as discussed yesterday ^14:42
*** nikhil has joined #openstack-infra14:44
*** andreas_s has quit IRC14:44
mordredAJaeger: I wonder if we should consider making some project-templates like "python-pti" and "release-managed" or similar (not today)14:44
*** MasterPiece2 has quit IRC14:46
mordredAJaeger: and potentially a "system-required" which is a template into which we put things like the merge-check jobs - but if we decided that there was something else that **all** projects in zuul need to run for logistical or system reasons14:46
AJaegermordred: nice idea - go for it ;)14:46
*** pblaho has quit IRC14:46
prometheanfireAJaeger: looks like the grub2 dib element may not work on ubuntu xenial again14:47
*** jgriffith has quit IRC14:48
AJaegerprometheanfire: are you sure I'm the right person to address? I don't rememeber anything here...14:48
prometheanfireAJaeger: you were the person to last commit to that element in feb14:48
prometheanfireoh, wrong a<tab>14:48
prometheanfireAJaeger: ya, wrong person14:48
*** rcernin has quit IRC14:49
AJaeger;)14:50
prometheanfireansreas around then :P14:50
prometheanfireor flonatel14:51
prometheanfirenot sure which nick is used14:51
*** links has joined #openstack-infra14:51
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Migrate ansible-hardening to zuulv3  https://review.openstack.org/51062314:52
*** dizquierdo has joined #openstack-infra14:54
*** xarses has joined #openstack-infra14:56
*** ykarel is now known as ykarel|afk14:57
*** jgriffith_ has joined #openstack-infra14:58
AJaegerpabelanger, mordred, fungi, could you try going over open reviews today if there're no fires, please? I'd like to see fixes for jobs and removal of unneeded jobs in...15:01
*** thorst has quit IRC15:01
*** pblaho has joined #openstack-infra15:02
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout  https://review.openstack.org/50951715:02
openstackgerritKen Giusti proposed openstack-infra/zuul-jobs master: Parameterize the test-setup role  https://review.openstack.org/51090715:03
*** vsaienk0 has quit IRC15:04
EmilienMmordred, clarkb: afik sshnaidm fixed all blockers to run zuul v3 legacy tripleo jobs (Sagi, tell me if I'm wrong)15:06
*** felipemonteiro_ has quit IRC15:07
AJaegerEmilienM: Woot! \o/15:07
sshnaidmEmilienM, all that I was aware of them15:07
openstackgerritCedric Savignan proposed openstack-infra/project-config master: networking-bgpvpn: add translation jobs  https://review.openstack.org/50917815:07
*** spectr has quit IRC15:07
mhaydenAJaeger: thanks for the help yesterday -- both patches merged for ansible-hardening but i'm still seeing zuulv2 jobs being run for that repo15:08
EmilienMmordred, clarkb: could we try to migrate tripleo jobs to zuulv3 again (running legacy jobs)?15:08
dmsimardEmilienM: legacy jobs are already being run, no ?15:08
AJaegermhayden: we still have v2 running15:09
EmilienMdmsimard: they are, but I meant to say "we're ready to try again"15:09
sshnaidmEmilienM, last patch is in the gate still: https://review.openstack.org/#/c/510839/15:09
EmilienMah this one ok15:09
EmilienMsshnaidm: why did you comment it and not remove? you plan to clean this up later?15:10
dmsimardEmilienM: there's >1k changes in the v3 check queue so it's entirely possible jobs aren't reporting back in a timely manner15:10
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout  https://review.openstack.org/50951715:10
sshnaidmEmilienM, yeah, after we use one version of zuul, and not two together15:11
*** armaan has quit IRC15:11
EmilienMok15:11
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Added openstack/requirements project to required_projects for Omni project  https://review.openstack.org/51001515:12
sshnaidmEmilienM, anyway I'll be keeping eye on switching to v3 tomorrow15:13
*** cshastri has quit IRC15:13
EmilienMsshnaidm: I'm glad we're more ready than last time, thanks15:14
sshnaidmya.. we should have started from parallel run :)15:14
smcginnisQuestion about the zuul switch. Has anyone done a check of querying where jenkins has left +1 but zuul left -1?15:17
smcginnisSeems like the results of that should be pretty small before we decide to flip the switch again.15:17
*** ccamacho has joined #openstack-infra15:17
smcginnisAnd my unsubstantiated impression right now that it will not be a small list.15:17
*** psachin has quit IRC15:18
*** vhosakot has joined #openstack-infra15:19
mnasersmcginnis in my experience, fixing a job generally is very quick and im sure infra will be all hands on deck to help those quick fixes15:19
mnaser99% of the time its a missing required-project or some dependency on a ZUUL_* var15:19
*** links has quit IRC15:20
*** spectr has joined #openstack-infra15:20
smcginnismnaser: Yeah, but I would much rather see those quick fixes happen before the switch rather than after.15:20
mnaserit's extremely hard to do that zuul -1 / jenkins +1 because there isn't a lot of resources available for zuulv3 to do that afaik15:20
mnasersmcginnis i think it's more of a resource problem, we cant allocate enough resources to zuul to do that i think15:20
smcginnismnaser: Not sure what you mean.15:21
mnasersmcginnis: the nodes which are used for testing are not shared across zuulv2 and v315:21
mnaserat the moment, v3 is opearting with 20% of the total donated capacity15:21
smcginnismnaser: But we can query gerrit. There are plenty I've seen where jenkins has +1 and zuul has -1. Not sure how the node capacity fits there.15:21
mnasersmcginnis: true, but the problem is zuul might have not reported for a day or two because it's so backed up.. i know some projects have zuul fixes but the result can't be seen yet cause its jobs are queued for hours now15:22
mnaserand i think we're generating more patches than zuulv3 can run, so the queue will just increase forever15:23
mnaser(with the small subset of resources that it has at the moment at least)15:23
*** thorst has joined #openstack-infra15:23
smcginnismnaser: OK, I get that. Not trying to be a jerk about this, but I think more work needs to be done to make sure we are ready to make the switch. I think everyone appreciates the work that has been/is being done, but there's a lot of "frustration" towards the gate still from the last attempt.15:24
mhaydenAJaeger: well, the odd part is that both of those patches merged, but nothing has changed in the list of jobs that runs: https://gist.github.com/major/8bea45359bf8a3c5a057c2eee138871515:25
mhaydenalso, i noticed that the zuulv2 jobs aren't prefaced with 'legacy-'15:26
*** andreas_s has joined #openstack-infra15:26
fungismcginnis: basically, we _can_ do what you're proposing by reallocating all the zuul v2 resources to run v3 jobs and ruen off v2, but that's basically cutting over to v3 anyway15:26
mnasersmcginnis totally, i get that, but i think part of that frustration was from the zuul issues that made it harder to fix things.  i twas a bit of a compounded problem.  jobs were broken and we couldn't fix them because zuul has issues15:26
fungis/ruen/turn/15:26
mnasernow that zuul runs much smoother, it'll be much easier and quicker to fix jobs, + we are having a dedicated queue for project-config which means that changes to project-config will get faster CI response15:27
smcginnisI'm just saying we should understand these before we flip the switch again: https://review.openstack.org/#/q/label:Verified%253E%253D1%252Cjenkins+label:Verified%253C0%252Czuul15:27
mnaserso iterating fixes will be quicker15:27
mnasersmcginnis TIL about label: in search, thanks :D15:27
smcginnismnaser: It required some googling on my part. ;)15:27
mnaseri do think you raise a good point for us to sift through those to identify potential issues15:28
fungismcginnis: an alternative i suppose is that we ask openstack to stop pushing up changes to projects for a while, so that we actually have enough capacity to run duplicates of all jobs in a timely fashion, but i doubt that would work out particularly well either15:28
smcginnisfungi: Hah, no, that was imposed a week or two ago and it didn't go over well from what I saw. :)15:29
*** edmonds__ has quit IRC15:29
AJaegermhayden: yes, we switch off Zuul v2 tomorrow...15:29
AJaegermhayden: none of the changes we merged disables the v2 jobs15:30
mhaydenah okay -- i'm trying to understand how all of these events impacts things ;)15:30
mnaseri'm pretty confident tomorrow will go over much smoother15:30
*** thorst has quit IRC15:30
mhaydenAJaeger: shouldn't i have more than the docs job in v3?15:30
mhaydenfeel free to flog me if i'm misreading the migration guide ;)15:30
*** gibi has quit IRC15:30
AJaegermhayden: Ah, more - yes, you should - but we have a queue of 1000+ jobs. Did any jobs finish?15:31
AJaeger1000+ changes I mean15:31
AJaegersorry, have to collect my daughter - bbl15:31
*** udesale__ has joined #openstack-infra15:31
mhaydeni've only see one job in the zuulv3 queue ever -- it has finished successfully once though15:31
*** shardy_ has joined #openstack-infra15:32
fungismcginnis: i see you caught my joke ;)15:32
smcginnisfungi: ;)15:32
*** gibi has joined #openstack-infra15:32
*** edmondsw has joined #openstack-infra15:32
*** esberglu has quit IRC15:32
mhaydenso i heard we're supposed to take an openstack vacation this week15:33
*** udesale__ has quit IRC15:33
*** esberglu has joined #openstack-infra15:33
*** thorst has joined #openstack-infra15:34
*** andreas_s has quit IRC15:34
*** edmondsw_ has joined #openstack-infra15:36
*** ccamacho has quit IRC15:37
*** dizquierdo has quit IRC15:37
*** edmondsw has quit IRC15:37
*** esberglu has quit IRC15:37
clarkbsmcginnis: re your last email I think that was the expectation ~ a week ago when we did the soft rollback. Zuulv3 would be reporting on your changes as well as zuulv2, please talk to us about problems with the zuulv3 results (and since then we've continued to help fix job issues in places like docs publishing and tripleo and so on)15:38
*** thorst has quit IRC15:39
clarkbbasically rhere is like half a million lines of configuration here. There are lots of fun unexpected corner cases15:39
clarkband the only way we are ever going to manage a transition is to do it, and fix those problems as they arise15:39
*** shardy_ has quit IRC15:40
clarkbpart of that is due to resource constraints as fungi and mnaser pointed out, but also just due to general size of it15:41
*** edmondsw_ has quit IRC15:41
clarkbthis is why we've been working hard to get this done early in the cycle (same as the gerrit upgrade)15:41
smcginnisAre the POST_FAILURE and RETRY_LIMIT failures cleared up? Or are they a result of the constrained resources?15:41
clarkbsmcginnis: I believe the POST_FAILUREs should largely be cleaned up. RETRY_LIMIT is likely due to bugs in the job pre playbook(s)15:42
*** reed_ has joined #openstack-infra15:42
smcginnisOK, I'll watch for latest results and raise any issues I see.15:42
*** thorst has joined #openstack-infra15:43
*** reed_ has quit IRC15:44
*** yamahata has joined #openstack-infra15:44
fricklerfungi: I haven't found any logs for the renamed v3 periodic jobs, do you know whether they ran at all or whether zuul simply didn't have resources for that?15:44
*** e0ne has quit IRC15:45
*** slaweq has joined #openstack-infra15:45
*** thorst_ has joined #openstack-infra15:46
EmilienMI know the zuulv3 queue is huge but is it expected that UI is freaking slow? http://zuulv3.openstack.org/15:47
jeblairEmilienM: yes.  those things are related.  it's managable at normal loads.  we have plans to fix after cutover.15:47
EmilienMgood to know, thanks15:48
clarkbjeblair: could we also use the existing status page? iirc it performs a bit better?15:48
*** andreas_s has joined #openstack-infra15:49
*** thorst has quit IRC15:49
jeblairclarkb: i'd rather drop it.  it's twice the work.15:49
*** edmondsw has joined #openstack-infra15:50
fungifrickler: since periodic jobs get lowest priority, i expect those simply haven't run yet because v3 has more jobs queued in check than it will be able to catch up with15:51
jeblairclarkb, fungi: should i issue a gear release with https://review.openstack.org/508698 ?15:51
jeblair0.10.1 i guess?15:52
fungijeblair: i'm skimming the commit log15:52
jeblairi think that's the only change since 10.015:52
fungiit's just that one patch, yeah15:52
*** LindaWang has quit IRC15:52
fungiso i guess it depends on whether you consider the addition of a send lock to be a new feature or a bug fix15:52
jeblairbugfix; not user visible15:53
fungi0.19.1 seems fine to me15:53
fungier15:53
fungi0.10.1 i mean15:53
*** andreas_s has quit IRC15:53
clarkbwfm15:53
*** thorst has joined #openstack-infra15:53
jeblairdone15:53
fungiwe're running with that on zuulv3 right?15:53
jeblairnope15:53
*** csomerville has joined #openstack-infra15:53
jeblairit actually only affects the clients15:54
fungiahh, so the mergers and executors have it installed15:54
*** thorst_ has quit IRC15:54
jeblairfungi: no, it's not installed anywhere15:54
pabelangerwhile the backlog is huge, zuulv3 itself is holding up very well now15:54
*** edmondsw has quit IRC15:55
*** edmondsw has joined #openstack-infra15:56
openstackgerritMike Perez proposed openstack-infra/project-config master: Adding the Contributor Guide Project  https://review.openstack.org/50994315:56
openstackgerritMike Perez proposed openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide  https://review.openstack.org/50993715:56
*** egonzalez has quit IRC15:57
thingeehey infra, I think the contributor guide project is ready ^ - I'm aiming to launch the contributor portal on openstack.org/community this week, but we need this.15:58
*** ykarel|afk has quit IRC15:58
*** yamahata has quit IRC16:00
*** dave-mcc_ has joined #openstack-infra16:01
*** rbrndt has joined #openstack-infra16:04
*** dave-mccowan has quit IRC16:04
*** bhavik1 has joined #openstack-infra16:06
mordredpabelanger: that, it turns out, is super important16:06
*** felipemonteiro has joined #openstack-infra16:06
mordredpabelanger: zuul should not degrade when the backlog is huge, otherwise building a huge backlog is very dangerous :)16:07
*** andreas_s has joined #openstack-infra16:07
pabelangerindeed16:07
*** pcaruana has quit IRC16:07
*** felipemonteiro_ has joined #openstack-infra16:07
*** lucasagomes is now known as lucas-afk16:08
*** gouthamr has quit IRC16:10
*** felipemonteiro has quit IRC16:11
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for volume detach 400 bug 1722577  https://review.openstack.org/51092616:12
openstackbug 1722577 in tempest "test_list_get_volume_attachments failing with 400 error on teardown when detaching an already detached volume" [Medium,Triaged] https://launchpad.net/bugs/172257716:12
*** ralonsoh_ has joined #openstack-infra16:13
*** spzala has joined #openstack-infra16:13
*** spzala has quit IRC16:13
*** jascott1 has joined #openstack-infra16:14
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job  https://review.openstack.org/51023716:15
*** ralonsoh has quit IRC16:16
*** armaan has joined #openstack-infra16:18
*** andreas_s has quit IRC16:20
clarkbre thingee's changes above any idea what happened in http://logs.openstack.org/43/509943/7/infra-check/project-config-gerrit/bd9fc7f/ara/ <- mordred jeblair pabelanger ? that resulted in POST_FAILURE16:23
*** liusheng has quit IRC16:23
clarkbthingee: the other failure is http://logs.openstack.org/43/509943/7/infra-check/project-config-frozen-files/31b0555/job-output.txt.gz#_2017-10-10_16_02_34_169545 so I thinkwe'd like to figure that out post v3 migration tomorrow?16:24
clarkbAJaeger: ^ your input owuldbe valuable on that I think16:25
*** andreas_s has joined #openstack-infra16:25
pabelangerclarkb: I think it is possible it could be logstash integration. Hard to say, because we do that task after we upload our logs, and nothing in printed to debug.logs by default16:25
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add release-openstack-javascript job  https://review.openstack.org/50218516:25
pabelangerclarkb: I think we could move http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml#n13 before upload-logs role16:26
pabelangerto help provide more info16:26
clarkbthingee: AJaeger I guess that job is non voting so we can make a judgement call?16:26
clarkbpabelanger: the issue with that is logstash expects the logs to be present when it gets the jobs and they won't be if we flip the order around16:27
thingeemordred: openstack-infra/openstack-zuul-jobs is what I was looking for job definitions. This is where a lot of my confusion was with project-config16:27
thingeewhew, omg the world makes sense again16:27
thingeefor now16:28
mordredthingee: \o/16:28
pabelangerclarkb: right, so we have a potential blind spot in logging I think16:28
mordredthingee: so - we have job documentation generation on a per-repo basis currently, such as https://docs.openstack.org/infra/openstack-zuul-jobs/ ...16:28
pabelanger2017-10-10 16:01:18,261 DEBUG zuul.AnsibleJob: [build: bd9fc7fd8d5b46b18e51c5dd1fb4b8ee] Ansible exit code: 216:28
pabelangerthat is from last post playbook, but debug.log is lacking any other ansible output16:29
mordredthingee: we have a roadmap item to teach zuul how to make a consolidated view of job documentation for all of the jobs it knows about in its combined job config16:29
jeblairpabelanger, clarkb, mordred: i think we're talking about the same class of failure here and in #zuul16:29
openstackgerritJakub Libosvar proposed openstack-infra/project-config master: neutron: Extend windowSize for periodic jobs  https://review.openstack.org/51093416:30
mordredjeblair: agree16:30
fungiclarkb: pabelanger: should we at a minimum make the logstash and subunit tasks always exit 0/succeed even if they fail to do what we want? we should probably instead have some other means of reporting breakages for those tasks without failing the jobs themselves since (in theory) there's nothing a job should be able to do to break those post tasks16:30
pabelangerjeblair: I believe so too16:30
fungijeblair: oh, excellent, i'll change venues16:31
*** armaan_ has joined #openstack-infra16:31
clarkbI'm catching up on scrollback there16:31
pabelangerfungi: ignore_errors: yes should provide that on ansible tasks16:31
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for volume detach 400 bug 1722577  https://review.openstack.org/51092616:31
openstackbug 1722577 in OpenStack Compute (nova) "test_list_get_volume_attachments failing with 400 error on teardown when detaching an already detached volume" [Undecided,Triaged] https://launchpad.net/bugs/172257716:31
jeblairpabelanger: yeah, want to go ahead and do that?16:31
*** dhinesh has quit IRC16:31
pabelangerjeblair: sure16:32
*** armaan has quit IRC16:33
*** jascott1 has quit IRC16:33
Guest66098pabelanger: we are having an emergency that requires putting the inap-mtl01 region offline for a minimum of 24h. Is there something you can do to mitigate the impact on the CI? This should be effective ASAP16:34
*** Guest66098 is now known as mgagne16:34
*** mgagne has quit IRC16:34
*** mgagne has joined #openstack-infra16:34
mgagnepabelanger: was me ^16:34
*** eharney has quit IRC16:35
clarkbmgagne: we'll want to set the max-servers to 0 in that region16:35
clarkbmgagne: do you want us to push that patch up or do you want to do it?16:35
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Update statsd output for tenants  https://review.openstack.org/51058016:35
mgagneI can do it16:35
*** eharney has joined #openstack-infra16:35
*** ociuhandu has quit IRC16:36
clarkbmgagne: ok you might want to grep the provider name and make sure you get it in the nodepool/zuul v2 and v3 configs16:36
mgagneok, I'm not familiar with zuul v316:36
clarkbmgagne: should mostly be the same config its just kept in multiple places while we transition16:36
clarkbso two files to edit instead of one16:36
*** dhinesh has joined #openstack-infra16:37
pabelangernodepool.yaml and nl01.openstack.org.yaml are the files in project-config16:37
*** pvaneck has joined #openstack-infra16:38
mgagnefound them16:38
openstackgerritMathieu Gagné proposed openstack-infra/project-config master: Offline inap-mtl01 region  https://review.openstack.org/51093616:38
*** andreas_s has quit IRC16:38
*** Guest84590 has quit IRC16:39
*** jpich has quit IRC16:39
*** csomerville has quit IRC16:41
*** dhinesh has quit IRC16:41
*** csomerville has joined #openstack-infra16:41
clarkbinfra-root ^ can we get that reviewed for mgagne? make sure I didn't miss anything :)16:44
pabelangerlooking16:44
pabelangerltgm +316:44
*** ganso has joined #openstack-infra16:44
*** mika has quit IRC16:46
*** shardy has quit IRC16:46
*** esberglu has joined #openstack-infra16:46
jeblairmore like +816:47
openstackgerritMerged openstack-infra/infra-manual master: Add announce-release job to stay in project-config  https://review.openstack.org/51090516:48
*** andreas_s has joined #openstack-infra16:52
*** tesseract has quit IRC16:53
*** derekh has quit IRC16:53
openstackgerritPaul Belanger proposed openstack-infra/project-config master: base-test: ignore_errors for logstash processing  https://review.openstack.org/51094316:54
pabelangerjeblair: mordred: clarkb: fungi: ^proposed fix to ignore errors on logstash workers.16:54
openstackgerritMerged openstack-infra/project-config master: Offline inap-mtl01 region  https://review.openstack.org/51093616:55
pabelangerOther option is to setup ignore_errors directly on tasks in each role16:55
clarkbpabelanger: why do you switch it to include roles at the same time?16:55
clarkbis it to avoid ^16:55
pabelangerclarkb: right16:55
pabelangerI don't know if we can use top level ignore_errors for a play16:55
pabelangerlet me test quickly16:55
*** Swami has joined #openstack-infra16:56
pabelangeroh, apparently you can16:56
pabelangerlet me do that16:57
*** gouthamr has joined #openstack-infra16:57
*** ociuhandu has joined #openstack-infra16:57
jeblairpabelanger: if we use include role, need to check with dmsimard to make sure it's not the variety that will bomb the json plugin16:57
jeblairit's probably worth noting that if we do this, we will never see these errors16:58
*** jascott1 has joined #openstack-infra16:58
jeblairi'm having second thoughts :)16:58
openstackgerritPaul Belanger proposed openstack-infra/project-config master: base-test: ignore_errors for logstash processing  https://review.openstack.org/51094316:58
pabelangerjeblair: clarkb: ^ should work too, at least in local testing16:58
pabelangerjeblair: Ya, thats the downside16:59
*** andreas_s has quit IRC16:59
*** gouthamr has quit IRC17:00
dmsimardpabelanger: would love to troubleshoot the zuul_json issue when we have a chance.. need executor logs17:01
*** gouthamr has joined #openstack-infra17:01
openstackgerritSam Betts proposed openstack-infra/devstack-gate master: Allow EOL branches to use the test-matrix.py script  https://review.openstack.org/51094617:02
*** slaweq has quit IRC17:02
*** MasterOfBugs has joined #openstack-infra17:02
*** inc0 has quit IRC17:02
pabelangerdmsimard: which review?17:03
*** inc0 has joined #openstack-infra17:03
*** lifeless has quit IRC17:03
*** dhinesh has joined #openstack-infra17:03
*** baoli has quit IRC17:04
*** mika has joined #openstack-infra17:04
*** mika has joined #openstack-infra17:04
*** dbecker has quit IRC17:04
*** ralonsoh_ has quit IRC17:04
*** baoli has joined #openstack-infra17:04
*** trown is now known as trown|lunch17:05
*** jpena is now known as jpena|off17:05
*** Apoorva has joined #openstack-infra17:07
*** dhinesh has quit IRC17:08
*** panda is now known as panda|off17:08
dmsimardpabelanger: https://review.openstack.org/#/c/504238/17:09
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095217:10
pabelangerdmsimard: k, I should have some time shortly to look at executors17:10
dmsimardIt might be too far back, hopefully not17:10
*** armaan_ has quit IRC17:12
*** armaan has joined #openstack-infra17:12
*** jkilpatr_ has joined #openstack-infra17:13
*** sambetts is now known as sambetts|afk17:14
*** jkilpatr has quit IRC17:14
*** rwsu has quit IRC17:16
*** armaan has quit IRC17:17
openstackgerritMerged openstack-infra/project-config master: base-test: ignore_errors for logstash processing  https://review.openstack.org/51094317:17
*** ram5391 has joined #openstack-infra17:17
AJaegerfrickler: periodic jobs did not run since they have lowest prio17:19
*** Goneri has joined #openstack-infra17:20
*** lifeless has joined #openstack-infra17:20
AJaegerthingee: we can only merge the contributor guide if it has adoptiong PTL+1 - so talk with pkovar first17:20
AJaegerclarkb: yes, we made the job non-voting so that we can make a judgement call and check that everything is ok.17:21
*** baoli has quit IRC17:23
AJaegerfrickler: sorry, was reading backscroll and see fungi answered already17:23
*** lukebrowning has quit IRC17:24
*** baoli has joined #openstack-infra17:24
*** lukebrowning has joined #openstack-infra17:24
clarkbAJaeger: thingee so I guess we recheck that change to get the post failure sorted and then decide about judgement calling?17:25
mgagnepabelanger: sorry... the plug got pulled earlier than expected...17:26
*** MasterOfBugs has quit IRC17:27
*** rwsu has joined #openstack-infra17:27
*** baoli has quit IRC17:28
AJaegerclarkb: and wait for docs PTL +1 first...17:29
pabelangermgagne: np, thanks for the heads up17:29
AJaegerclarkb: I haven't seen pkovar comment on any of these reviews and we merge new project creation only with +1 from adopting PTL17:30
*** persia has quit IRC17:30
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Add the statusbot to the #openstack-tc IRC channel  https://review.openstack.org/51096417:30
*** rbrndt has quit IRC17:30
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: DNM: testing base-test changes  https://review.openstack.org/51096517:31
*** rwsu has quit IRC17:32
AJaegerclarkb, thingee - I pinged pkovar on #openstack-doc...17:32
clarkbAJaeger: thanks17:33
thingeeAJaeger: oh right, thank you17:33
*** pkovar has joined #openstack-infra17:34
pkovarAJaeger: am here17:34
AJaegerthingee: pkovar left some questions for you;) I asked him to join us here17:34
pkovarjust to confirm the status of the new project, really17:34
*** baoli has joined #openstack-infra17:35
AJaegerpkovar: if it's in docs team, the docs team will own it but might have a subteam for it...17:35
*** baoli has quit IRC17:35
pkovarAJaeger: right, so we could also say the subteam would own it ;)17:36
AJaegerpkovar: in case the subteam has problems or disolves, it would go back to the docs team...17:36
AJaegerpkovar: you can also object and thingee needs to put it elsewhere. Since it looks you haven't discussed this yet: I suggest that thingee proposes on openstack-dev mailing list with tag [docs] the contributor guide and how it would be part of docs team...17:38
pkovarAJaeger: well yes, but i really want to make sure the responsibilities at this point are clearly set17:38
*** nicolasbock_ has quit IRC17:38
pkovarAJaeger: fine with me17:38
*** dtantsur is now known as dtantsur|afk17:38
*** nicolasbock has joined #openstack-infra17:39
pkovaror, if thingee agrees to just go ahead and create a specialty team, that's also fine with me17:39
thingeepkovar: sounds fine to me that the subteam takes responsibility17:39
pkovarthingee: cool, let's comment on the change so i can give a formal +1 then :)17:40
AJaegerthingee, pkovar, could either of you follow up on the mail that thingee sent and say that this will be part of docs team as a speciality team? We should inform rest of docs team. We can reference the discussion, but let's summarize and communicate...17:41
*** priteau has joined #openstack-infra17:41
pkovarthingee: are you ok with leading the subteam? or can you appoint a contact?17:42
*** tosky has quit IRC17:43
AJaegerthingee: could you add a change similiar to https://review.openstack.org/510925  then, please? We'd like to have subteams documented17:43
*** baoli has joined #openstack-infra17:45
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Mask dependent config errors  https://review.openstack.org/51096817:48
jeblairmordred: ^ that's a thing that we batted around a few days ago i think.  i suspect that class of change will occur quite a lot over the next couple of weeks, so if we get that in before tomorrow, we may save a lot of people a lot of confusion.17:49
jeblairAJaeger, fungi, clarkb, pabelanger: ^17:50
AJaegermordred: why is https://review.openstack.org/#/c/510237/3/zuul.d/project.yaml needed?17:50
AJaegerjeblair: yeah, seen it quite often - thanks for working on it!17:51
thingeeAJaeger: I'm ok with leading the subteam. I'll propose the change now.17:51
jeblairalso, i think it would be really good to carefully consider the wording in that message.  i tried to write it as clear and concisely as possible, but if it doesn't help people understand we should change it.  :)17:51
pkovarthanks, thingee17:51
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095217:51
*** cody-somerville has joined #openstack-infra17:52
mordredAJaeger: it is not - that got sucked in accidentally - fixing17:52
*** sshnaidm is now known as sshnaidm|off17:52
pkovarand i'll send a followup email to -dev later if you don't beat me to it :)17:52
pkovargot to go now17:53
*** jcoufal has joined #openstack-infra17:53
*** dhinesh has joined #openstack-infra17:53
mordredAJaeger: oh! wait - I remember now ...17:53
AJaegerjeblair: text is good, one suggestion inline17:53
mordredAJaeger: it is needed - or at least it was for a minute nad will again ...17:53
AJaegermordred: something to mention in commit message?17:54
*** csomerville has quit IRC17:54
mordredAJaeger: the tox-linters job for openstack-zuul-jobs pulls in the contents of zuul-jobs so that it can validate against roles that come from zuul-jobs ... but that patch had a depends-on on a zuul-jobs patch, thus the linters did not have the correct roles available17:54
mordredAJaeger: yah17:54
fungiclarkb: the [nova] in the subject line for that e-mail you just responded to on the infra ml makes me think this is a confused user posting questions about running openstack in the wrong venue. expect a very confused response17:54
mordredAJaeger: actually ...17:55
AJaegermordred: ah, I see.17:55
clarkbfungi: ya could be17:56
*** priteau has quit IRC17:56
clarkbfungi: though they did ask about builds logs, so maybe they just want nova things specifically?17:56
fungiclarkb: maybe. when i first read the message i thought they were asking about indexing nova instance boot logs in elasticsearch for a deployment of openstack17:57
fungibut i can see that without the [nova] in the subject i may have interpreted the rest differently17:57
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job  https://review.openstack.org/51023717:57
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add zuul-jobs to required-projects for tox-linters  https://review.openstack.org/51097117:57
mordredAJaeger: how about I make it its own commit for even better clarity ^^ :)17:58
AJaegermordred: perfect ;)17:58
AJaegermordred: why is zuul now unhappy? ;(17:59
*** baoli has quit IRC17:59
*** sree has quit IRC17:59
*** Apoorva_ has joined #openstack-infra18:00
*** sree has joined #openstack-infra18:00
*** baoli has joined #openstack-infra18:00
*** ram5391 has quit IRC18:01
*** bhavik1 has quit IRC18:03
*** Apoorva has quit IRC18:03
*** tosky has joined #openstack-infra18:04
*** sree has quit IRC18:05
*** trown|lunch is now known as trown18:05
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095218:05
mordredAJaeger: well - because?18:06
mordredAJaeger: oh! I forgot another depends-on there18:07
*** persia has joined #openstack-infra18:07
pkovarAJaeger: thingee approved both changes18:07
mordredAJaeger: https://review.openstack.org/#/c/502185/18:07
mordredalthough that commit message is pretty bad18:08
*** ijw has quit IRC18:08
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job  https://review.openstack.org/51023718:08
*** jtomasek has quit IRC18:09
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add release-openstack-javascript job  https://review.openstack.org/50218518:10
mordredbetter commit message18:10
AJaegermordred, fungi, two quick reviews on removing unused jobs if you have time, please: https://review.openstack.org/510765 and https://review.openstack.org/51068318:11
*** pkovar has quit IRC18:15
*** coolsvap has quit IRC18:15
thingeeAJaeger https://review.openstack.org/#/c/510979/18:15
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add comment explaining gitpython requirement  https://review.openstack.org/51098018:15
thingeeAJaeger: that wiki link is mostly a placeholder right now18:16
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095218:16
*** Swami has quit IRC18:17
openstackgerritPaul Belanger proposed openstack-infra/project-config master: base: ignore_errors for logstash processing  https://review.openstack.org/51098218:17
pabelangerclarkb: fungi: mordred: jeblair: promote up our logstash ignore_errors patch to base^. The commit message has results of testing18:18
openstackgerritAndrey Pavlov proposed openstack-infra/project-config master: ec2-api: move OOO job to experimental  https://review.openstack.org/51098318:18
andreafdmsimard, jeblair, mordred: I managed to make the stage-artifacts role accept a list of files and folders, like in https://review.openstack.org/#/c/509272/8/playbooks/post-tempest.yaml18:18
pabelangerclarkb: also, I am likely to miss most of our infra meeting today. Have to run an errand wit my family.18:19
*** Apoorva_ has quit IRC18:19
*** Apoorva has joined #openstack-infra18:20
AJaegerthingee: thanks. Will review tomorrow, have to call it a day now...18:21
jeblairpabelanger, clarkb, mordred, fungi: i think i want to hold off on 510982 in favor of running with mordred's change for a bit so we can try to catch the error(s)18:21
clarkbpabelanger: ok, I think its largely just going to be catch up on zuulv3 things. If there is nayhting pressing on your side for that you might want to write it down18:21
pabelangerjeblair: wfm18:21
pabelangerI'll mark WIP for now18:21
*** electrofelix has quit IRC18:21
jeblairpabelanger: cool.  thanks for getting it ready -- if we don't see/fix the error before we cutover, we should land that.  maybe we should land it at cutover anyway.  :)18:22
*** rbrndt has joined #openstack-infra18:22
mordredjeblair, pabelanger: is our current working theory that it was a logstash submission issue?18:23
*** baoli has quit IRC18:23
mordredjeblair: yah - I think landing it at or around cutover anyway is maybe not a bad idea18:23
fungiit's a reasonable theory since it's a post_failure which takes place after logs are successfully uploaded, and logstash/subunit2sql tasks are the only other post playbooks involved at that stage18:23
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095218:24
jeblairyeah, it seems like it's either that, or failure uploading some particular file later, or... something we haven't imagined.  in that order of probability.  :)18:25
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095218:25
jeblairi'm going to grab lunch; should be back by meeting; can do full zuul restart afterwords18:28
thingee#thanks AJaeger for keeping the Contributor Guide progress going on infra and doc team sides.18:29
openstackstatusthingee: Added thanks to Thanks page18:29
pabelangermordred: ya, I think we get some error after upload-logs role, which are not exposed to us currently18:32
pabelangerbut, just a guess18:32
dansmithI have a job that has been stuck in gate for a long time now, all jobs are finished, one failed, but it won't kick out so I can recheck it (it failed git clone)18:33
dansmithbesides re-pushing a large series to kick it, can anything be done?18:34
dansmithit's 506416,1818:34
pabelangerdansmith: abandon / restore works too18:34
*** baoli has joined #openstack-infra18:34
pabelangerlooking18:34
dansmithpabelanger: will that affect the patches above it in the series/18:34
pabelangerdansmith: it would dequeue them too18:35
dansmithpabelanger: they're not in the gate at the moment, I only care about not having to rebase them18:35
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add git timeout  https://review.openstack.org/50951718:35
pabelangerdansmith: ya, I don't think you'll need to rebase18:36
dansmithpabelanger: okay, so is it waiting for the things in front of it to finish before being kicked out?18:36
dansmithI thought since the ones in front passed the test it failed it would kick out immediately18:37
pabelangerdansmith: ya, it won't get kicked out until 500384 passed. If 500384 fails, it will be rebased and all tests run again18:38
dansmithokay I thought that was only if the job it failed failed in front of it18:38
dansmithbut okay18:38
dansmithsince there are some ahead of it with jobs still queued I guess I'll kick it18:38
fungidansmith: it's assuming that any changes ahead of it in the shared queue might be causing the failure, and would retest it without if one of those subsequently fails18:38
*** esberglu has quit IRC18:39
pabelangeryah, best you could hope for now is a patch because you to fail, to cause your test to be re-run18:39
fungidansmith: which is the reason why it's kept around in a dependent pipeline like the gate (rather than reporting back immediately and dequeuing)18:39
dansmithfungi: yeah, I just thought that was done per-job only.. doesn't make much sense for nova unit tests, but that's obscure18:39
*** esberglu has joined #openstack-infra18:39
dansmithabandon/restore seemed to do the trick, thanks18:40
fungiwe have the chance to make zuul v3 much smarter about such things. v2 didn't really know about which jobs used what other repos18:40
fungisince zuul v2 was only involved in the scheduling side and not the job definitions18:41
dansmithcool18:41
pabelangerhttps://review.openstack.org/95035/ is a long run request to add dequeue to zuul client for admins to be able to run it18:41
pabelangerI need to follow up on it and address the comments18:42
*** baoli has quit IRC18:43
*** eroux has joined #openstack-infra18:43
*** esberglu has quit IRC18:44
mordredpabelanger: wow. that's a 5-digit patch18:47
pabelanger:)18:47
pabelangermordred: maybe we can get SpamapS to push on it again :)18:48
SpamapShaha18:49
SpamapSno18:49
*** boden has joined #openstack-infra18:49
SpamapS;)18:49
SpamapSmaybe I guess18:49
*** esberglu has joined #openstack-infra18:50
clarkbinfra meeting is in ~10 minutes in #openstack-meeting18:50
bodenhowdy, zuul v3 question if someone gets a chance… is there a reason I wouldn’t have any recent zuul v3 results for the vmware-nsx project?  https://review.openstack.org/#/q/project:openstack/vmware-nsx+label:Verified%252Czuul   I was hoping to get a warm and fuzzy for tomorrow switch18:51
mnaserboden the queue is quite backed up because zuulv3 doesnt have a lot of resources at the moment18:51
mnaserboden and zuulv3 is undergone a few restarts as fixes are being tested so its likely that it was lost as well18:52
bodenmnaser: ack, well I guess I’ll have to swap that warm and fuzzy for a finger cross :)18:52
bodenhow far is it backed up, do you know?18:52
clarkbit was more than 30 hours last I looked18:52
bodenoh ok… well it’ll be difficult for me to ensure for tomorrows switch I guess.. we’ll have to reach out if something crops up real-time18:53
*** cody-somerville has quit IRC18:53
mnaserboden a lot of people will be available to help, also i remember reading zuulv3 might be getting restarted shortly18:53
bodenBTW: thanks for all the work on this too.. I’m sure it’s been quite an effort18:53
mnaserso you can slip a recheck quickly after teh restart, aha18:53
bodenmnaser: ack, and thanks18:53
*** cody-somerville has joined #openstack-infra18:55
*** rwsu has joined #openstack-infra19:00
*** andreas_s has joined #openstack-infra19:05
*** slaweq has joined #openstack-infra19:06
*** baoli has joined #openstack-infra19:09
*** boden has left #openstack-infra19:11
*** jkilpatr_ has quit IRC19:12
*** armax has quit IRC19:14
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364519:16
*** rcernin has joined #openstack-infra19:16
*** andreas_s has quit IRC19:18
*** eharney has quit IRC19:26
*** jkilpatr has joined #openstack-infra19:27
*** rbrndt has quit IRC19:29
openstackgerritMerged openstack-infra/zuul-jobs master: Remove integration test playbooks from zuul-jobs  https://review.openstack.org/51086819:30
*** rbrndt has joined #openstack-infra19:30
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove legacy announce-release jobs  https://review.openstack.org/51068319:31
*** andreas_s has joined #openstack-infra19:32
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures  https://review.openstack.org/51095219:35
*** andreas_s has quit IRC19:36
*** andreas_s has joined #openstack-infra19:37
*** eharney has joined #openstack-infra19:41
*** pkovar has joined #openstack-infra19:41
*** thorst has quit IRC19:46
ianwhmm, are we really going to have to add sbin to PATH like https://review.openstack.org/#/c/504743/11/roles/multi-node-bridge/tasks/main.yaml everywhere?  that seems something we should do globally19:46
*** thorst has joined #openstack-infra19:51
*** rkukura_ has joined #openstack-infra19:53
*** andreas_s has quit IRC19:55
*** pkovar has quit IRC19:55
*** thorst has quit IRC19:56
*** rkukura has quit IRC19:56
*** rkukura_ is now known as rkukura19:56
*** e0ne has joined #openstack-infra20:00
*** eroux has quit IRC20:00
clarkbthingee: before I approve https://review.openstack.org/#/c/509943/7 is there an existing repo we need to be importing?20:01
clarkbthingee: it is much easier to do that at project creation time than it is at a later time20:01
*** rbrndt has quit IRC20:02
*** rbrndt has joined #openstack-infra20:02
*** ijw has joined #openstack-infra20:06
ianwclarkb: do you remember if https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n38 was really just "has a routable ipv6 interface"?20:06
ianwcause ansible has a fact for that, that i think avoids a bit of hassle20:06
ianwbut also not sure it wasn't working around some bizzare "looks routable but isn't" type issue20:07
clarkbianw: iirc its just that. Because some clouds (rax vexxhost osic) have proper routable ipv6 and others don't (the rest)20:07
clarkbso as long as the scope isn't local we are fine20:07
*** eroux has joined #openstack-infra20:08
*** thorst has joined #openstack-infra20:08
ianwyep, ansible only puts routable addresses in there, so that's a nice way around that20:08
pabelangerand back20:08
*** andreas_s has joined #openstack-infra20:09
*** andreas_s has quit IRC20:13
thingeeclarkb: there is no existing repo.20:13
clarkbthingee: thank you for confirming, I'll approve the change in a moment20:13
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Mask dependent config errors  https://review.openstack.org/51096820:13
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add comment explaining gitpython requirement  https://review.openstack.org/51098020:13
thingeeclarkb: aside from my local one20:14
thingeecan that be pushed up first?20:14
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add an origin remote for tag-releases for notes  https://review.openstack.org/51100420:14
clarkbthingee: if you don't start with a seed repo during project creation you'll have to push everything but the first commit we add with a .gitreview thorugh code review20:15
clarkbthingee: the alternative is to host a git repo somewhere else with any existing content and we will import that in at project creation time in gerrit20:15
thingeeclarkb: I only have the cookie cutter and another commit on top of that. not worried about losing history20:16
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Google Cal sync error  https://review.openstack.org/51101120:16
clarkbthingee: ok, so you are fine with the change as is and will just push content up when gerrit project exists?20:16
SamYaplethingee: with all the projects ive added, i always go with empty repos personally20:16
thingeeclarkb: yes push is fine.20:17
thingeeSamYaple: thanks20:17
thingeeclarkb: eventually this will need to publish20:17
thingeeI can do a follow up commit tho20:17
openstackgerritMerged openstack-infra/openstackid-resources master: Google Cal sync error  https://review.openstack.org/51101120:18
clarkbthingee: ok it is on its way20:18
SamYaplei typically do voting gates too. that way the first commit must pass the gate. but thats just me20:18
thingeeSamYaple: there was some issue with a translation job that was part of the docs build job. We decided to go a noop approach since we're kind of in a hurry, because we're always in a hurry :)20:19
*** e0ne has quit IRC20:20
SamYapleyea thats compeltely understandable20:20
inc0soo..question, any chances you can give some time between now and full zuulv3 rollout without legacy jobs?20:22
inc0right now it's impossible to work on zuulv3 gates because it's overloaded20:22
inc0and if you roll in it fully, our gates will be broken again until we fix it20:22
clarkbmordred: jeblair pabelanger has http://logs.openstack.org/23/475123/10/check/legacy-tempest-dsvm-neutron-multinode-full/4f8b030/job-output.txt.gz#_2017-10-09_12_02_20_776368 been debugging and/or fixed?20:23
*** slaweq has quit IRC20:23
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf  https://review.openstack.org/51101720:23
inc0we're close but we could use at least some time (few days at most) when we can work on v3 gates without impacting regular gates20:23
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul master: Add statsd to zuul.conf  https://review.openstack.org/51101820:23
*** e0ne has joined #openstack-infra20:23
inc0I think other projects could use window like that as well20:23
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Error on Admin action calendar sync  https://review.openstack.org/51101920:24
* clarkb looks t odebug that problem20:24
pabelangerclarkb: I am not sure, but I can look20:24
*** e0ne has quit IRC20:25
clarkbpabelanger: looks like frickler may have fixed it I8e7240f28e1aa991c637b8aee421a2f5b4eba9e920:25
inc0thing is, if we won't have it, that puts us in code freeze until we figure out zuulv3 gates, I hope I don't need to tell you that it's not great :/20:25
openstackgerritMerged openstack-infra/openstackid-resources master: Error on Admin action calendar sync  https://review.openstack.org/51101920:25
clarkbinc0: I'm not sur ehow that would logistically function/work20:26
clarkbinc0: we are currently down a cloud and alreayd splitting resources and unable to keep up20:26
mordredinc0, clarkb: there's nothing stopping landing a removal of your legacy changes from v3 right now20:26
inc0mordred: we removed it, but our new one is not ready yet20:27
mordredthe main issue there is that if they're removed then you'll have a perioud of time post rollout with none of those jobs in your gate20:27
inc0it's virtually impossible to work on zuulv3 right now20:27
pabelangerclarkb: cool20:27
mordredinc0: oh - so you already removed your legacy jobs? I may be misunderstanding the ask ... reading again20:27
inc0well also when we had legacy v3 jobs they weren't working for us20:28
inc0yeah, we removed it (they were broken anyway)20:28
SamYaplemordred: he is trying to avoid a code freeze, but cant make sure teh v3 gates are working because v3 is slow20:28
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364520:28
SamYapleso v3 is going to land with non-funcational kolla gates and force a code freeze20:28
clarkband things are slow basiclly because ENOTENOUGHCOMPUTE20:28
mordredgotcha. I now understand20:28
pabelangerYah, I think it might have been premature to remove legacy jobs, even thought that's what we want long term20:28
inc0legacy didn't work20:29
clarkbas fungi mentioned earlier today, you could stop pushing code to address that... but otherwise we are down a cloud and so on20:29
mordredwell - the legacy jobs for kolla also didn't work - so time spent fixing the new jobs is time better spent20:29
jeblairwell, they might work at this point20:29
pabelangerby tomorrow, you should be in a better spot20:29
inc0well no20:29
jeblairpart of why we rolled back was because we missed things in the legacy conversions20:29
inc0because tomorrow you roll in all zullv320:29
pabelangerwhat was the issue with legacy?20:29
openstackgerritMerged openstack-infra/project-config master: Adding the Contributor Guide Project  https://review.openstack.org/50994320:30
inc0lots of small issues that we're partially resolving with our zuulv3 work20:30
inc0different dirs and whatnot, don't forget Kolla is not your typical python service20:30
inc0we did quite a bit of OS-level stuff in our gates to make it work20:30
SamYapleyea inc0 the only reason i got my v3 gates done is because i did them on the weekends when all the queues were empty20:30
jeblairyes, but the work we did on legacy jobs was around mimicing v2, so if we got that right, they should work despite their differences.20:31
mordredinc0: I think logistically we just don't have the nodebandwidth to do it - BUT - with v3 working properly iterating on fixes to your v3 jobs should go much quicker - and with us not needing to put out as many fires we should be able to help20:31
inc0and that's ok, zuulv3 makes it whole lot easier, but we could really use time window to make it work20:31
*** andreas_s has joined #openstack-infra20:31
inc0so what I'm asking is to remove legacy jobs from current split20:31
inc0and give us few days like that20:31
SamYapleinc0: what patchset are you working with right now for the v3 gates?20:31
clarkbwe've also intentionally chosen this time of the release cycle knowing that things may/will be affected20:32
inc0https://review.openstack.org/#/c/508759/ this is for ansible20:32
inc0kolla proper build already is green and deploy needs ^20:32
pabelangeragree with jeblair, legacy jobs should have been working or pretty much close to working. But admit, I never seen a log to debug why.  It is good you moved jobs to zuulv3 however20:32
inc0well I can't remember exactly what was wrong, but it didn't work.20:32
*** openstackgerrit has quit IRC20:33
inc0that's why I didn't pause before removing legacy jobs and going all v320:33
mordredinc0: in any case, I hear your concern, but at this point I think we're just gonna have to dive in tomorrow and help you fix them - I've gotten pretty good at fixing jobs quickly ...20:33
pabelangeragree20:33
mnaserfor puppet, i personally just refactored small bits and left the main bash shell still doing the job (called by ansible)20:33
mnaserthat's the best way to fix things quickly20:33
inc0ok, let's do that then20:34
mnaseri think you're taking too big of a bite if you want to rewrite them all with ansible + roles etc20:34
inc0we're pretty close I think20:34
SamYapleinc0: i can get you a well tested gate with docker installed and working to replace a large section of your work with ansible native code. can i add to Jeffrey4ls patch?20:34
toskymnaser: that's what I'd like to do as well (rewrite directly)20:34
mnaserhttps://review.openstack.org/#/c/508296/20:35
thingeeSamYaple: oh and I can it to my .zuul.yml locally. I don't see if it says that gets used over what's in project config https://docs.openstack.org/infra/manual/creators.html#adding-in-repo-zuul-jobs20:35
thingeecan add it*20:35
inc0SamYaple: sure, feel free, although lots of that (for k-ansible at least) is already done by bootstrap servers20:35
mnaseryou'll notice the playbooks are all calling the main bash script20:35
inc0right now we have ansible calling command: ansible-playbook ...;)20:35
jeblairclarkb, fungi, mordred: should i install zuulv3 branch tip with or without my repl backdoor?  (in case we have craziness that needs on-line debugging; or should we trust what we've done so far)20:35
inc0so at some point we'll want it all to be one large ansible run20:36
SamYaplethingee: you can parent jobs from project-config, but i dont think you can *override* jobs20:36
*** sree has joined #openstack-infra20:36
clarkbjeblair: probably best to keep the backdoor for now and remove it once we've proven the install?20:36
mordredjeblair: I say without - let's use the next 18 hours or so to prove out that we're good20:36
mordredha20:36
mordredclarkb and I took opposite sides  :)20:36
jeblairneed tiebreaker!20:36
SamYapleinc0: right, i can get you a full gate setup with docker and ansible, then you can just shell: ansible-playbook to get you most of the way there20:36
toskythat said - does anyone have a clue on this "permission denied" error? http://logs.openstack.org/79/510579/2/check/legacy-sahara-dsvm-scenario-neutron/7874161/job-output.txt.gz#_2017-10-10_17_00_50_20241420:36
inc0SamYaple: let's deal with it later, kolla proper gates seems to be fine20:37
inc0https://review.openstack.org/#/c/508661/20:37
mnasertosky seems weird20:38
toskyand also this error, which looks like my job should require tripleo-ci.git (!) http://logs.openstack.org/31/510631/1/check/legacy-sahara-cli/6d2ff42/logs/devstack-gate-setup-workspace-new.txt20:38
inc0after we're in state when our gates works we'll slowly get rid of shell in favor of ansible20:38
clarkbtosky: first thing that comes to mind is the pip + sudo behavior where sudo pip install will create wheels in the cache of the homedir of the user but sudo doesn't change the homedir20:38
clarkbtosky: so you end up with files owned by root then later try to edit them as regular user and it breaks20:38
mnaser^ that's what i was guessing as well20:38
pabelangertosky: there is a lot of ERROR logging in that review too20:38
thingeeSamYaple: ah ok good to know20:39
pabelangerhttp://logs.openstack.org/79/510579/2/check/legacy-sahara-dsvm-scenario-neutron/7874161/job-output.txt.gz#_2017-10-10_17_00_49_96616520:39
inc0thingee: btw, new Blade Runner - totally awesome20:39
mnaserand tosky correct, i believe it's missing in required-projects20:39
SamYaplethingee: all the jobs live in a global namespace. so you can't same name jobs20:39
toskypabelanger: uhm, I see - I think I know what to fix there20:39
toskybut for the other job, the requirement to tripleo-ci, I'm not sure if it's for sure, if it's a mistake, or it's a red herring20:40
*** sree has quit IRC20:40
pabelangertosky: http://logs.openstack.org/79/510579/2/check/legacy-sahara-dsvm-scenario-neutron/7874161/job-output.txt.gz#_2017-10-10_17_00_33_21311720:40
pabelangerchown zuul:stack20:41
pabelangerno longer using jenkins user20:41
toskypabelanger: right, thanks, that should be easy20:41
*** openstackgerrit has joined #openstack-infra20:41
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Specify duplicate project defn behavior  https://review.openstack.org/51102520:41
pabelangertosky: yay20:41
jeblairthingee: ^ i found that wasn't even mentioned in the zuul docs :)20:41
*** dprince has quit IRC20:42
jeblairi'm going to perform a full shutdown of zuulv3 now, then repartition20:44
pabelangerwfm20:45
*** trown is now known as trown|outtypewww20:48
*** andreas_s has quit IRC20:54
thingeeclarkb: can you please add me to the openstack-contributor-guide-core group https://review.openstack.org/#/admin/projects/openstack/contributor-guide,access20:55
SamYapleinc0: if you recheck quickly you can get the first zuulv3 patch going!20:56
jeblairnot yet; but i'll let you know when i restart it20:56
* SamYaple points out this is how he got most of his jobs fixed20:57
*** wolverineav has quit IRC20:59
*** srobert_ has joined #openstack-infra20:59
toskymnaser: I think that should be the fix for the tripleoci thing; I didn't notice https://review.openstack.org/51103720:59
clarkbthingee: I've added pkovar as well since its technically a docs project21:00
jeblairi've restarted nodepool (zuulv3 is still offline)21:00
jeblair(rather, the v3 nodepool launchers)21:00
toskyuh, no notifications for changes to openstack-zuul-jobs here? Only on #zuul maybe?21:01
jeblairtosky: should be here21:01
*** srobert has quit IRC21:01
toskyoh, ok, zuul v3 is being restarted, maybe that's the reason21:02
jeblairtosky: the bot may have missed it or had an issue with the irc server21:02
thingeeclarkb: thanks. I just added openstack-doc-core as training-guides and other sub teams are setup :)21:03
*** srobert_ has quit IRC21:03
*** gouthamr has quit IRC21:06
*** mat128 has quit IRC21:06
jeblair  Downloading GitPython-2.1.7-py2.py3-none-any.whl (446kB)21:06
*** mat128 has joined #openstack-infra21:07
jeblairclarkb, mordred, fungi: ^ that did not install the fork21:07
jeblairthat was with "sudo pip3 install -U ." in a zuul checkout21:07
* clarkb looks at the requirements file again21:07
jeblairmaybe it's cached somewhere?21:07
*** andreas_s has joined #openstack-infra21:08
clarkbya I wonder if #egg=GitPython means it will grab something out of cache?21:08
*** jamesdenton has quit IRC21:08
clarkbor that it can?21:08
*** Apoorva_ has joined #openstack-infra21:09
*** jcoufal has quit IRC21:10
*** eharney has quit IRC21:11
*** mat128 has quit IRC21:11
*** andreas_s has quit IRC21:12
*** Apoorva has quit IRC21:12
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364521:12
jeblairokay i just did "sudo pip3 -r requirements.txt" which seemed to work21:13
jeblairblack magic21:13
*** esberglu has quit IRC21:14
clarkbjeblair: I reproduced the initial behavior you ran into locally21:14
clarkbis it possible this is a pbr behavior?21:14
clarkbmordred: ^21:14
*** esberglu has joined #openstack-infra21:14
*** claudiub|2 has joined #openstack-infra21:15
clarkbjeblair: reading pbr source I think it likely that pbr is at fault21:16
clarkbFor the requirements list, we need to inject only the portion after egg= so that distutils knows the package it's looking for <- from pbr comments in source21:16
clarkbI think because setuptools doesn't grok those lines only pip does21:16
clarkbso you have to install requirements with pip21:16
fungiokay, back and catching up. looks like i missed a bunch of discussion in the past hour-ish21:17
jeblairwhen i followed that up with another 'pip3 install -U .' it installed the wheel again21:17
jeblairi think it's likely that both this and the github3.py thing aren't actually working for us in prod21:17
dtroyerI am seeing an error in the legacy OSC functional jobs with a 'cp -dRl' command that appears to come from zuul-cloner-shim.py: http://logs.openstack.org/89/359489/3/check/legacy-osc-dsvm-functional-tips/0f9592d/job-output.txt.gz#_2017-10-09_20_51_19_67384421:18
dtroyerlooks like cross-filesystem link failure: cp: cannot create hard link21:18
*** rcernin has quit IRC21:19
jeblairzuulv3 has restarted21:19
jeblairinc0: ^21:19
ianwdtroyer: hmm, i wondered if that was ever going to go across fs's21:19
ianw/tmp/python-openstackclient-tox_install-1NfuS51 ... so it's a tmpfs?21:19
*** armaan has joined #openstack-infra21:20
dtroyerthat's my guess21:20
dtroyerthere appears to be a check for that in zuul-cloner-shim.py21:20
clarkbianw: dtroyer I wrote a fix for that21:20
jeblairyeah, it landed a week ago :/21:20
clarkbwhich I thought was working for tripleo at least21:20
mordredclarkb: looking ...21:21
dtroyerthis run was last night21:21
dtroyersomething I need to add to our job?21:21
*** thorst has quit IRC21:21
*** armaan has quit IRC21:21
*** ldnunes has quit IRC21:21
*** armaan has joined #openstack-infra21:22
fungijeblair: clarkb: mordred: if we restart now without the repl support and then discover we need it, that's yet another restart possibly in the midst of people trying to do rushed migration work... whereas if we start with it then we can kick the restart down the road to when we're comfortable it won't be needed?21:24
clarkbdtroyer: no it should be used already, its possible my fix is buggy or not able to address all situations21:24
*** jgriffith has joined #openstack-infra21:24
clarkbI just added an issue to the issues with job's list for broken d-g/devstack caching21:24
pabelangermordred: jeblair: ze01 doesn't look healthy. I see 3 zuul-executor processes again, 4 if you count defunct21:24
clarkbI'll add dtroyers hardlink thing to the list too21:25
jeblairfungi, clarkb, mordred: i ran with the repl21:25
pabelangerze02 also21:25
* mordred supports our repl overlords21:25
jeblairpabelanger: hrm, i'll try that again21:26
*** Apoorva_ has quit IRC21:26
pabelangerand ze0421:26
*** anupn has joined #openstack-infra21:26
*** Apoorva has joined #openstack-infra21:26
*** yamahata has joined #openstack-infra21:27
pabelangerze08 seems to have 1 zuul-executor process, other defunt21:27
jeblairpabelanger: yeah, i'm restarting them now21:27
anupnHi all, I am seeing "no module named keystonemiddleware" inside my gate although i see keystone-auth1 and python-keystoneclient has been installed21:27
pabelangerdefunct*21:27
pabelangerkk21:27
anupnhere is the log http://logs.openstack.org/52/509652/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-src-ubuntu-xenial/6fbca9c/logs/screen-s-proxy.txt.gz#_Oct_10_18_55_29_49301121:28
pabelangerjeblair: I think ze08 has a hung git process.21:28
pabelangermaybe that is why we get into this state21:28
anupnThe patch i am working is for testing py3 inside ironic-python-agent but I see unrelated error of no module named keystonemiddleware, can someone point out what might be going wrong21:29
fungipabelanger: yeah, my expectation was that the patches to time out hung git operations would address the wedged executors21:29
clarkbianw: dtroyer on ubuntu I didn't think /tmp was a tmpfs21:29
*** dhinesh has quit IRC21:29
clarkbianw: dtroyer reading my code I suppose there is a race in it if /proc/mounts changes during the running of z-c21:30
clarkbsince I only read the mounts list the one time21:30
pabelangerfungi: I'm not sure ze08 has the lastest patch for git timeout, but it looks like if we have hung git process, then stop. We don't properly shutdown a zuul-executor process. Then, if we start again, we might end up with 3 zuul-executor processes.21:30
pabelangerbut, just speculation currently21:31
fungipabelanger: right, i thought jeblair's current round of restarts was to also make sure we run the executors with the git timeout fixes21:31
jeblairokay, all executors re-restarted21:33
*** edmondsw has quit IRC21:33
fungianupn: are you installing the keystonemiddleware package too?21:33
clarkbfungi: I'm looking at how to fix the devstack-gate caching situation and wondering if maybe a symlink from /home/zuul/cache/ to /home/jenkins/cache would work? I guess then it comes down to permissions and whether or not everything dealing with that cache is going to follow symlinks21:33
anupnHi fungi, i see that installed here http://logs.openstack.org/52/509652/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-src-ubuntu-xenial/6fbca9c/logs/pip3-freeze.txt.gz21:34
ianwclarkb: didn't we fix that :/21:34
ianwclarkb: https://review.openstack.org/#/c/509116/ is what i'm thinking of21:35
*** edmondsw_ has joined #openstack-infra21:35
clarkbianw: that only affects the jenkins user, but reading that I think I know how to fix it for the zuul user now too21:35
fungianupn: "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py"21:35
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements  https://review.openstack.org/50364521:35
fungianupn: note the "python2.7" there21:35
fungianupn: if memory serves, swift-proxy doesn't support python3 yet21:35
ianwclarkb / dtroyer : yes ... how did the (presumed) tmpfs get in there?21:36
anupnfungi, oh yeah i was just checking inside pip2-freeze right now21:36
anupnfungi, and i dont see keystonemiddleware there21:36
fungianupn: last time i looked, ay least, i think devstack was still running swift-proxy under python2.7 when you asked to use python321:36
pabelangerJust noticed requirements-check is failing: http://logs.openstack.org/94/484594/94/check/requirements-check/34c4822/job-output.txt.gz#_2017-10-10_21_26_19_00374021:37
pabelangeranybody working on that?21:37
* pabelanger checks etherpad21:37
anupnfungi, so does that mean swift-proxy always run under python2.721:37
jeblairpabelanger: yes, currently line 60/6121:38
pabelangeryay: https://review.openstack.org/51056521:38
clarkbanupn: fungi ya swift doesn't support python321:38
fungianupn: i think so? i mean, if they don't support python3 yet then it doesn't make a ton of sense to run it under python321:38
fungiso you need to make sure its dependencies are installed where python2.7 can find them21:38
anupnclarkb, fungi: ah21:38
anupnfungi, so this problem comes when it tries to run everything under python321:39
anupnand not just ironic21:39
*** edmondsw_ has quit IRC21:39
fungianupn: well, it looks like whatever the source of the problem is, the result is that keystonemiddleware is not being installed globally for python2.7 such that swift-proxy can import it21:40
*** andreas_s has joined #openstack-infra21:40
fungianupn: you may have better luck taking this up with the devstack devs in #openstack-qa21:40
openstackgerritClark Boylan proposed openstack-infra/project-config master: Put /home/zuul/cache in place on test node images  https://review.openstack.org/51105621:40
clarkbianw: ^21:40
anupnfungi, hmm yeah21:41
pabelangerjeblair: thanks, I've +2 mordred patch21:41
jeblairi have to run for a bit, bbl.21:43
*** aeng has joined #openstack-infra21:44
pabelangerclarkb: ianw: should we kick off new image builds for /home/zuul/cache fixes? Or just wait until next round of DIB builds21:46
fungipabelanger: i'd kick off new builds asap21:47
clarkbya having that in place would be nice21:47
fungibetter than waiting until cut-over time tomorrow to find out that devstack jobs all broke21:47
anupnfungi, clarkb: I notice that gate failed when i set USE_PYTHON3=True and passed when set to lowercase true21:47
anupnhttps://review.openstack.org/#/c/509641/2/devstack-vm-gate-wrap.sh@6021:47
anupnfungi, clarkb: the reason i think is because here https://github.com/openstack-dev/devstack/blob/master/inc/python#L58821:48
pabelangerfungi: clarkb: sure, wanted to confirm21:48
anupnif it is set to True devstack tries to run all services with py321:48
anupnand hence as swift do not support py3 it is giving the prob21:48
clarkbanupn: I think we blacklist swift in other jobs that use python321:49
clarkbanupn: you can look at the generic python3 tempest job to see how that is done21:49
toskyI see that some variables have been replaced with something else (for example I was looking for ZUUL_BRANCH https://review.openstack.org/#/c/486246/)21:50
anupnclarkb, sorry do you have a pointer to that handy?21:50
anupnclarkb, not sure exactly where to check that21:50
toskyI'm not sure how it should replaced: just a variable that can be used in ansible conditions? Is there a table which says which variable became what?21:50
pabelangertosky: yes, see https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#var-zuul for more information21:50
clarkbanupn: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n74821:51
pabelangertosky: they will be written to the ansible inventory file21:51
toskyah, there, thanks21:51
anupnclarkb, cool, thanks21:51
toskypabelanger: I was looking only in https://docs.openstack.org/infra/manual/zuulv3.html21:51
openstackgerritMerged openstack-infra/project-config master: Put /home/zuul/cache in place on test node images  https://review.openstack.org/51105621:56
*** ijw has quit IRC21:56
*** andreas_s has quit IRC21:57
*** threestrands has joined #openstack-infra21:58
*** camunoz has quit IRC22:01
*** andreas_s has joined #openstack-infra22:02
*** felipemonteiro_ has quit IRC22:04
fungitosky: the mapping is created here, if it helps you at all: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv322:04
fungithat's a relatively human-readable translation between the two (hurray python?)22:05
toskyfungi: oooh, thanks22:05
fungithat's the compatibility shim we're calling zuul_legacy_vars in the legacy playbooks22:06
*** dhinesh has joined #openstack-infra22:07
*** baoli has quit IRC22:08
*** mat128 has joined #openstack-infra22:08
*** edmondsw has joined #openstack-infra22:08
*** bobh has quit IRC22:11
toskyok, so another question: I have two job in sahara-extras which 1) created some artifacts 2) pushed them to tarballs.o.o - are those going to be the same job, with specialized run: and post: playbooks now?22:12
toskyor better, tasks22:12
*** dhinesh_ has joined #openstack-infra22:12
*** edmondsw has quit IRC22:13
fungiyou can probably look at the native tarball jobs... lemme get you a link22:13
pabelangerYa, publish-artifacts job would be a great example22:15
*** dhinesh has quit IRC22:16
*** andreas_s has quit IRC22:16
*** kgiusti has left #openstack-infra22:17
*** dhajare has quit IRC22:18
fungitosky: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n33722:18
fungitosky: it's calling these: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/python-tarball22:19
*** yamamoto has joined #openstack-infra22:19
toskythanks22:20
*** andreas_s has joined #openstack-infra22:21
jeblairpabelanger, fungi: can you review https://review.openstack.org/511018 when you get a chance?  i'd like to have that in place before some more statsd work i'm doing lands22:21
mordredjeblair: lgtm22:21
jeblair(i'm eager to have a per-executor current jobs gauge)22:22
fungitosky: and parented to this: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/publish/openstack-artifacts.yaml22:23
toskyfungi: just found!22:24
toskyso yes, I would need to merge the two previously different jobs in one, and inherit from publish-openstack-artifacts22:24
fungipresumably22:25
*** andreas_s has quit IRC22:25
fungitosky: i'm not sure what the difference was in your old jobs22:25
mordredfungi: we used to have build then publish - we've moved to having them combined in our current v3 jobs22:25
pabelangerjeblair: +322:25
mordredalthough I'm working on writing up a proposal to change that a little bit :)22:26
fungifor example if one was to test that tarball building works and the other was to build a tarball and upload it, then you could do like we have for the python-tarball jobs where we just omit the post playbook on one22:26
SamYaplewould anyone want to review a bindep patch? https://review.openstack.org/#/c/506502/22:26
SamYaplei hear reviewing Parsley things counts as a serving of vegatables22:26
fungimordred: just so i'm not confused, you mean we used to have "build" and "build and publish"22:27
fungi?22:27
toskyfungi: gate-sahara-extra-artifacts and the next post-sahara-extra-artifacts http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/sahara.yaml#n15922:27
fungitosky: okay, yep then that's what i'm talking about. the gate version just tested that you could build a tarball, the post version built and published a tarball22:28
toskyand if I understand it correctly, the gate-sahara-extra-artifacts will be the pre/run, while post-sahara-extra-artifacts will be the post22:28
toskybut for the same job22:28
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Revert "Revert "Add redirect from status.o.o/zuul to zuulv3.openstack.org""  https://review.openstack.org/51106722:28
mordredtosky: not quite22:29
toskyoh22:29
mordredtosky: so ... sahara-extra-publish is basically a thing that does what the publish-openstack-artifacts base job does - it uploads artifacts to tarballs.o.o/{project-name}22:30
*** chlong has quit IRC22:30
pabelangermordred: I should be online early for tomorrows rollout. For now, I'm preping some patches with topic:zuulv3-rollout22:30
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add zuul-jobs to required-projects for tox-linters  https://review.openstack.org/51097122:30
pabelangergetting patches for nodepool ready now22:30
mordredtosky: you'll want a job, for instance 'sahara-build-extra-artifacts' that runs your tools/build-all-artifacts.sh script22:31
mordredtosky: then you'll want a job, for instance 'sahara-publish-extra-artifacts' that has publish-openstack-artifacts as a base job, runs tools/build-all-artifacts.sh and copies them back to the executor into {{ zuul.executor.work_root }}/artifacts22:32
mordredthat way you can put sahara-build-extra-artifacts into check and gate and sahara-publish-extra-artifacts into post22:33
toskyoh, I see, still two jobs22:33
*** kjackal_ has quit IRC22:33
toskyfor some reason I thought that the post: action was invoked only when the job was in the post queue, but yeah, it's not that22:34
mordredyah - that's phases of the job itself - but the word post sure does show up both places22:35
fungiit's a bit overloaded22:35
openstackgerritMerged openstack-infra/project-config master: Use the shared openstack-tox-pylint jobs  https://review.openstack.org/51063622:35
openstackgerritMerged openstack-infra/project-config master: Switch TripleO scenario upgrade jobs to nv  https://review.openstack.org/51063822:35
toskydefinitely moreclear now, thanks22:35
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Allow all zuul servers to send to statsd  https://review.openstack.org/51106822:35
fungimaybe we should rename our "post" pipeline in zuul to "merged" to disambiguate22:36
jeblairmordred, pabelanger, fungi: ^ one more prep thing22:36
jeblairor post-merge22:36
fungiafter-merge?22:36
openstackgerritMerged openstack-infra/project-config master: Remove legacy jobs from kuryr-k8s  https://review.openstack.org/51076922:36
mordredjeblair: I +2'd - but in case you care there are tabs in that patch22:36
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: Add unbound role  https://review.openstack.org/51072522:36
jeblairapres-merge22:36
SamYaplenot that one22:36
mordredfungi, jeblair: after-shve perhaps?22:36
jeblairi'm imagining all of these ideas on roadside signs22:37
SamYaplepost-merge seems good22:37
mordredwe could call it 'aqua-velva' and confuse everyone who is younger than 40 and not from the US22:37
SamYaplestill has post in the name, explains what it is22:37
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Allow all zuul servers to send to statsd  https://review.openstack.org/51106822:37
jeblairmordred: i think i got all the tabs22:37
fungijeblair: i approved it, but i think it may get kicked back for the mixing of tabs and spaces22:38
fungi511068 i mean22:38
fungioh, you fixed22:38
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Test unbound role  https://review.openstack.org/51073422:38
* mordred has a feeling that somewhere deep in the bowels of something us changing th ename of the post pipeline would wreak crazy kinds of havoc in unexpected places22:38
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Revert "Revert "Shift nodepool quota from v2 to v3""  https://review.openstack.org/51106922:38
SamYaplemordred: lets just change it on the zuul page lol22:39
mordredSamYaple: :) ... if only the zuul page wasn't data-driven22:39
*** sdague has quit IRC22:39
SamYapleindeed22:40
mordredpabelanger: revert revert patches are always my favorite22:40
pabelanger++22:40
*** jgriffith has quit IRC22:40
pabelangerI _think_ that is everything. But we'll know more in the morning22:41
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Switch TripleO scenario upgrade jobs to nv  https://review.openstack.org/51063722:41
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs from Kuryr-libnetwork  https://review.openstack.org/50998622:41
*** mat128 has quit IRC22:41
pabelangerexcited for the morning22:42
pabelangerlike christmas all over again22:42
mordred\p/22:42
mordredinfra-root: I just sent an email to the infra list about interfaces for publishing things - basically the topic we were just discussing with tosky22:43
*** rbrndt has quit IRC22:45
*** rbrndt has joined #openstack-infra22:45
*** rbrndt has quit IRC22:45
*** jgriffith_ has quit IRC22:46
mordredit's not a thing we need to do before the rollout, so feel free to read or ignore as you will between now and tomorrow22:46
*** armax has joined #openstack-infra22:48
*** andreas_s has joined #openstack-infra22:48
openstackgerritLuigi Toscano proposed openstack-infra/openstack-zuul-jobs master: Remove the legacy pylint jobs (replaced)  https://review.openstack.org/51107122:49
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Integration tests: run mirror-info before configure-mirrors  https://review.openstack.org/51088122:49
ianwhas anyone thought of putting something at the top of the console log outputting the hostname/ip address it's running on.  i used to just look at the ip in the link, now it's really hard to tell22:49
pabelangerwe opted to output to add it into zuul-info, which gets upload after the job finished22:50
pabelangermakes it a little harder to get the info and SSH into a running job22:51
*** vhosakot has quit IRC22:51
ianwclarkb / jeblair: want to weigh in on the home for the unbound configuration role?  i think otherwise it's ready -> https://review.openstack.org/#/c/510725/22:52
toskymordred: should I wait to convert my publisher job until that change proposal is discussed and approved?22:52
jeblairianw, clarkb, mordred: i think i lean toward openstack-zuul-jobs because it depends on some pretty openstacky things we're doing in base images.22:54
ianwjeblair: ok, i'll move it.  at first i thought i might port all the package install etc into it ... but it seems better to do in the image.  there's no point having all the setup of the dns cacher fail due to dns issues :)22:55
*** aeng has quit IRC23:01
*** sree has joined #openstack-infra23:01
*** andreas_s has quit IRC23:01
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf  https://review.openstack.org/51101723:01
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor  https://review.openstack.org/51107323:01
*** rhallisey has quit IRC23:02
clarkbjeblair: ianw ya I think even assuming unbound is present is fairly openstacky23:03
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Add unbound role  https://review.openstack.org/51073423:05
ianwclarkb / jeblair : ^ ok, moved23:05
*** sree has quit IRC23:05
*** xarses has quit IRC23:09
*** andreas_s has joined #openstack-infra23:11
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add configure-unbound role to base-test job  https://review.openstack.org/51072823:13
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add configure-unbound to base job  https://review.openstack.org/51107523:13
*** claudiub|2 has quit IRC23:15
*** Apoorva has quit IRC23:16
*** aeng has joined #openstack-infra23:17
*** Apoorva has joined #openstack-infra23:18
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Add unbound role  https://review.openstack.org/51073423:20
*** thorst has joined #openstack-infra23:22
*** hongbin has quit IRC23:22
*** andreas_s has quit IRC23:24
*** thorst has quit IRC23:27
*** jascott1 has quit IRC23:32
clarkbpabelanger: were you going to trigger those image builds?23:35
*** tosky has quit IRC23:35
openstackgerritMerged openstack-infra/tripleo-ci master: Stop using the cached cirros image  https://review.openstack.org/51083923:35
clarkbwait isn't ^ fixed by my image change?23:36
clarkbsshnaidm|off: FYI the image is actually cached at /opt/cache/files regardless of zuulv3 or zuulv223:37
clarkbsshnaidm|off: we then have (or will have) handy symlinks to point to that location from /home/jenkins/cache and /home/zuul/cache23:37
*** mriedem has quit IRC23:38
*** mat128 has joined #openstack-infra23:39
*** gouthamr has joined #openstack-infra23:40
clarkbianw: unbound changes lgtm, I've +2'd the first pair23:40
ianwclarkb: thanks, not sure why 511075 is unhappy23:40
clarkbianw: I think it was because you pushed a new ozj patchset after pushing 07523:41
clarkbI rechecked it23:41
clarkbsame thing happaned with 72823:41
mordredclarkb: if you're in a reviewing mood, https://review.openstack.org/#/c/510686/ and https://review.openstack.org/#/c/502185/ could use a +A23:41
clarkbianw: as for hardlinks stuff, probably want to run the python snippet that determines filesystem on a rackspace xenial node?23:41
ianwyeah, moving onto that now23:42
*** Apoorva_ has joined #openstack-infra23:42
*** Apoorva has quit IRC23:42
*** andreas_s has joined #openstack-infra23:43
mordredclarkb, ianw: I +A'd the unbound role add and the base-test change23:45
*** rlandy has quit IRC23:45
ianwclarkb: was there a reason to not compare stat dev?23:45
clarkbianw: I read that and thought it wouldn't work for some reason but now rereading it am thinking it will23:46
ianwoh i guess two partitions on the same device it still won't work for23:46
*** andreas_s has quit IRC23:47
ianwhmmm23:47
clarkboh maybe that was it, was not sure if device is physical device or fs23:47
fungiit's filesystem23:47
fungihardlinks are just additional labels for the same inode23:47
*** andreas_s has joined #openstack-infra23:47
clarkbfungi: but stat.st_dev is filesystem id not physical device id?23:48
fungiso a hardlink to an inode on a different filesystem doesn't make sense23:48
clarkbya the reason its failing is you can't hardlink across filesystems23:48
ianwfungi: right, but comparing stat's st_dev it's 100% safe, if two partitions are on the same device23:48
fungiit's the id of the partition afaik23:48
ianwisn't i mean23:49
ianwbut it's probably safe enough23:49
fungipartitions are block devices23:49
fungias are logical volumes23:49
*** gildub has joined #openstack-infra23:49
fungiso it's whatever (pseudo)device the filesystem maps onto23:49
ianwi guess it's major:minor and so it will work23:51
fungiyeah, it'll be unique per filesystem, in effect23:51
ianwthis doesn't however, answer the question of how the heck we got a different fs in the mix23:51
clarkbianw: oh!23:53
clarkbianw: I know, its because /tmp is on the root device and /opt is on the ephemeral disk on rax23:53
clarkbso ya maybe the thing to do is rewrite that checker to use stat.st_dev instead23:53
ianwhmm, *that* would make sense23:53
fungithat will be the case, yes23:53
ianwyep, ok, that should work23:53
ianwwill do23:53
*** andreas_s has quit IRC23:56
*** ijw has joined #openstack-infra23:56
*** andreas_s has joined #openstack-infra23:56
mordredclarkb: oh right. the ephemeral disks23:57
clarkbI'm helping to debug a duplicate acccount in Gerrit and think I need to run `UPDATE account_external_ids SET account_id = 21561 WHERE account_id = 27031;` Basically 27031 is the extra account and want to flatten its openid into 21561. Is that valid sql to update the column you are filtering by (I think it is as those are discrete steps of operation)?23:58
*** mancdaz has quit IRC23:58
clarkbI'm not ready to run that just yet as waiting for ocnfirmation that we don't want to use 27031 going forward23:58
clarkbmordred: fungi ^23:58
ianwtextwrap.dedent()!  how did i not know about that23:59
pabelangerclarkb: ya, I can do kick off images now23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!