Wednesday, 2020-05-06

openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox use venv  https://review.opendev.org/72573700:19
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox use venv  https://review.opendev.org/72573700:26
*** larainema has joined #opendev00:42
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip : fix Xenial exported virtualenv command  https://review.opendev.org/72574300:57
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip : fix Xenial exported virtualenv command  https://review.opendev.org/72574301:06
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip : fix Xenial exported virtualenv command  https://review.opendev.org/72574301:09
*** _mlavalle_1 has quit IRC01:10
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip : fix Xenial exported virtualenv command  https://review.opendev.org/72574301:15
openstackgerritMerged zuul/zuul-jobs master: ensure-pip : fix Xenial exported virtualenv command  https://review.opendev.org/72574302:06
openstackgerritIan Wienand proposed openstack/project-config master: Drop pip-and-virtualenv from SUSE 15  https://review.opendev.org/72574902:41
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Remove opensuse-15-plain testing  https://review.opendev.org/72575002:42
openstackgerritIan Wienand proposed openstack/project-config master: Stop building opensuse-15-plain images  https://review.opendev.org/72575102:45
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] focal ubuntu-minimal testing  https://review.opendev.org/72575202:49
fungi#status log unlocked mirror.opensuse afs volume and manually performed vos release to clear stale lock from 2020-04-28 afs01.dfw outage02:55
openstackstatusfungi: finished logging02:55
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: use venv to install  https://review.opendev.org/72573703:01
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] test with plain nodes  https://review.opendev.org/71281903:09
mordredianw: so ... next time you're bored - would you look at https://review.opendev.org/#/c/715717/ ? (it's in merge conflict, and I'm not 100% sure it's right - but I was in there and it seems like we're making a logic error - you touch the nodepool siblings stuff more than I do - take a peek and see if you think I'm right?03:12
ianwhrm looking03:14
openstackgerritMerged opendev/system-config master: Revert "Clear LD_PRELOAD variable on zuul-web containers"  https://review.opendev.org/72573003:36
*** ykarel|away is now known as ykarel03:54
*** openstack has joined #opendev04:21
*** ChanServ sets mode: +o openstack04:21
*** cgoncalves has joined #opendev05:38
*** ysandeep|away is now known as ysandeep05:51
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Remove some temporary files  https://review.opendev.org/72565005:54
*** diablo_rojo has quit IRC05:57
*** dpawlik has joined #opendev06:03
*** ysandeep is now known as ysandeep|away06:12
*** roman_g has joined #opendev06:28
*** ysandeep|away has quit IRC06:29
*** ysandeep has joined #opendev06:38
*** ysandeep is now known as ysandeep|brb06:38
*** ysandeep|brb is now known as ysandeep06:39
*** ralonsoh has joined #opendev06:45
*** lpetrut has joined #opendev07:19
*** tosky has joined #opendev07:30
*** rpittau|afk is now known as rpittau07:31
*** cgoncalves has quit IRC07:34
*** cgoncalves has joined #opendev07:40
*** dtantsur|afk is now known as dtantsur07:40
*** roman_g has quit IRC07:41
*** openstackstatus has quit IRC07:46
*** openstack has joined #opendev07:48
*** ChanServ sets mode: +o openstack07:48
*** citizen_stig has quit IRC08:03
*** ysandeep is now known as ysandeep|lunch08:03
*** dpawlik has quit IRC08:09
*** dpawlik has joined #opendev08:09
*** roman_g has joined #opendev08:28
*** larainema has quit IRC08:30
*** elod is now known as elod_pto08:43
*** ykarel is now known as ykarel|lunch08:45
openstackgerritMerged zuul/zuul-jobs master: Remove some temporary files  https://review.opendev.org/72565009:00
*** ysandeep|lunch is now known as ysandeep09:20
*** ykarel|lunch is now known as ykarel09:27
*** rpittau is now known as rpittau|bbl09:44
*** owalsh has quit IRC09:46
*** owalsh has joined #opendev10:09
*** priteau has joined #opendev11:03
*** ysandeep is now known as ysandeep|brb11:16
*** ysandeep|brb is now known as ysandeep11:54
*** hashar has joined #opendev12:11
*** rpittau|bbl is now known as rpittau12:18
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Retry buildx builds  https://review.opendev.org/72584312:23
*** tkajinam has joined #opendev12:34
*** ykarel is now known as ykarel|afk12:38
openstackgerritMerged zuul/zuul-jobs master: Retry buildx builds  https://review.opendev.org/72584312:47
openstackgerritThierry Carrez proposed openstack/project-config master: Add repository for oslo.metrics  https://review.opendev.org/72584712:49
*** ralonsoh has quit IRC13:08
*** ralonsoh has joined #opendev13:11
*** olaph has joined #opendev13:12
*** cgoncalves has quit IRC13:19
fungi#status log unlocked mirror.yum-puppetlabs afs volume and manually performed vos release to clear stale lock from 2020-04-28 afs01.dfw outage13:22
openstackstatusfungi: finished logging13:22
*** cgoncalves has joined #opendev13:25
mordredinfra-root: I'm going to put nodepool hosts into emergency and then land the nodepool patch so that we can update them one at a time13:28
fungimordred: thanks, i'll be around to help13:31
fungioptimal caffeination nearly achieved13:31
*** slittle1 has quit IRC13:37
mordredfungi: I feel like that's an approximation that would be better descibed by a limit13:47
mordredbecause i'm not sure it's possible to ever actually reach that state, but only to approach it closer and closer13:47
fungiasymptotic caffeine ideal13:49
*** ykarel|afk is now known as ykarel13:54
*** slittle1 has joined #opendev13:54
*** slittle1 has quit IRC14:01
*** ysandeep is now known as ysandeep|away14:02
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: WIP configure cache-to and cache-from for buildx  https://review.opendev.org/72586214:02
*** slittle1 has joined #opendev14:03
openstackgerritMerged opendev/system-config master: Run nodepool launchers with ansible and containers  https://review.opendev.org/72052714:21
*** jhesketh has quit IRC14:22
*** lpetrut has quit IRC14:30
mordredwoot14:40
mordredinfra-root: I'm gonna run ansible on nl01 - I'm in a screen in case anyone wants to watch, but I'm fairly sure it'll go fine14:44
*** mlavalle has joined #opendev14:49
fungijumping on now, thanks14:49
fungii guess it's not there yet? or already gone14:50
*** jhesketh has joined #opendev14:51
mordredfungi: I'm in screen14:52
mordredon bridge14:52
mordredfungi: ok - I did chowns on nl01 and did a docker-compose start - and it seems like it's running well14:55
mordredwell - except- we seem to be logging to the docker logs instead of to /var/log/nodepool14:56
mordredbut docker-compose logs -f seems to look decent - except for some quota issues14:57
* mordred is going to wait here and not do any further nodes so people can look at things14:57
*** smcginnis has quit IRC14:58
fungioh, bridge15:00
fungiduh ;)15:00
*** lpetrut has joined #opendev15:00
fungimind if i scroll the buffer?15:01
clarkbmordred: did the logging config not get supplied properly? it co figures the /var/log/nodepool stuff iirc15:04
*** smcginnis has joined #opendev15:04
mordredfungi: please do!15:07
mordredclarkb: we are totally not passing -l /etc/nodepool/launcher-logging.conf in the compose file15:08
*** hashar has quit IRC15:08
fungiokay, and the usermod errors are expected since there are running processes for that user, same as we saw with zuul15:11
fungilgtm15:11
openstackgerritMonty Taylor proposed opendev/system-config master: Configure nodepool launcher to use logging config  https://review.opendev.org/72588915:17
mordredclarkb, fungi: ^^15:17
fungiso the command parameter defaults to just `nodepool-launcher -c /etc/nodepool/nodepool.yaml` somehow?15:19
fungi(and where was that happening?)15:19
mordredfungi: https://opendev.org/zuul/nodepool/src/branch/master/Dockerfile#L4115:20
mordredin the zuul setup, we have a zuul.conf file which is where we configure which logging config to use, so it wasnt' an issue15:21
mordredfungi: actually - we're logging to docker in nb04 too15:22
fungineat15:22
mordredand we don't write out a logging config there15:23
mordredso - should we align to just using docker logs for this? or also add a builder logging config?15:23
mordred(the dib builds are still logging to /var/log/nodepool/builds)15:23
*** ykarel is now known as ykarel|away15:23
mordredclarkb, corvus : ^^ thoughts?15:23
clarkbmordred: I think the special logging is more important for the builder than the launcher as it split logs out by image15:24
clarkbwe then expose those per image build logs publicly15:24
mordredyeah - and that's working on the builder15:26
mordredshould we just keep logging the service logs to docker for both?15:26
fungii agree separating out builder logs for publication is more important15:26
mordredif so - I can just stop writing out a logging config file15:26
fungibut if we can publish launcher logs safely too, then why wouldn't we?15:26
clarkbmordred: if we do that I think we should consider doing it for all the services we run too (at least as much as we can, haproxy will only log to syslog iirc)15:27
clarkbfor me the big thing is consistency and not trying to figure out where logs are on different hosts/services15:27
mordredclarkb: yeah - I think we've got a hybrid mix of logging currently15:27
mordredbut - I think for zuul/nodepol that sounds liek a good reason to do logging config files15:28
mordredsince we're doig them for zuul15:28
mordredthat way at least we're consistent within that service15:28
clarkb++15:30
clarkbinfra-root I've pulled the latest zuul-web image down which is built without jemalloc. The docker compose file for zuul-web no longer updates LD_PRELOAD. I'd like to restart the zuul-web service now. Let me know if I should hold off on that15:30
mordredclarkb: go for it15:31
clarkbdone15:32
clarkbthis should serve as a last sanity check for the jemalloc cleanup addressing memory pressure15:32
corvusmordred: i'd be more comfortable logging to disk everywhere15:32
*** roman_g has quit IRC15:33
*** roman_g has joined #opendev15:34
openstackgerritMonty Taylor proposed opendev/system-config master: Configure nodepool to use logging config  https://review.opendev.org/72588915:37
mordredcorvus, clarkb: ^^ that should get us logging to disk on builder and launcher15:37
clarkbmordred: how does https://review.opendev.org/#/c/725889/2/playbooks/roles/nodepool-builder/files/logging.conf produce the per image logs? or is nodepool doing that internally? I thought at one time we generated a very large logging.conf with all our images listed15:38
mordredclarkb: I think nodepool is doing that internally15:39
clarkbmordred: also found a file path bug on the latest ps15:39
mordredclarkb: that's the logging config file from puppet15:39
clarkb(its noted inline)15:39
clarkbmordred: gotcha15:40
openstackgerritMonty Taylor proposed opendev/system-config master: Configure nodepool to use logging config  https://review.opendev.org/72588915:40
clarkbmordred: double check ps3 too. I think builder and logging are transposed in the two locations they are used15:41
mordredawesome15:41
clarkbbuilder-logging.conf vs logging-builder.conf15:41
openstackgerritMonty Taylor proposed opendev/system-config master: Configure nodepool to use logging config  https://review.opendev.org/72588915:42
mordredclarkb: yup. thanks15:42
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Split the arch building and pushing separately  https://review.opendev.org/72590515:44
*** panda|pto has quit IRC16:03
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567816:06
*** panda has joined #opendev16:06
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567816:10
*** dtantsur is now known as dtantsur|afk16:11
mordredfungi, corvus : if you get a sec, could you +A https://review.opendev.org/#/c/725889/ so we can roll forward with nodepool?16:11
mordredfungi: thanks!16:14
openstackgerritMerged zuul/zuul-jobs master: Split the arch building and pushing separately  https://review.opendev.org/72590516:14
mordredonoes.16:14
fungiwot?16:15
mordredfungi: https://zuul.opendev.org/t/openstack/build/476e48ecdf6f4c63a37f9ce9a91ec9fa16:17
mordredfungi: file (/var/log/nodepool) is absent, cannot continue16:17
mordredwhat?16:17
mordredoh. HAHAHAHAHA16:18
openstackgerritMonty Taylor proposed opendev/system-config master: Configure nodepool to use logging config  https://review.opendev.org/72588916:18
mordredfungi, clarkb : please to enjoy the silly mistake - and to also be thankful that we have testing16:18
fungioh, indeed, a /var/log/nodepool file would have been less useful16:20
clarkbdirectories and files are just inodesin the end right? why is ext4 so picky :P16:20
mordredclarkb: we should switch to running hurd16:21
clarkbrelated: you can't hardlink directories, but ufs has/had an internals debugging tool that did allow you to effectively hardlink directories by constructing the inode contents correctly16:22
*** rpittau is now known as rpittau|afk16:24
*** priteau has quit IRC16:25
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567816:29
Open10K8SHey infra team!16:37
Open10K8SI am facing POST_FAILURE in zuul gating16:37
AJaegerdo you have a link to a failure, Open10K8S ?16:38
Open10K8Ssure16:38
Open10K8Shttps://zuul.opendev.org/t/vexxhost/status16:39
*** hrw has quit IRC16:39
Open10K8Sthere are 2 projects16:39
Open10K8Shttps://review.opendev.org/#/c/723087/16:39
Open10K8Shttps://review.opendev.org/#/c/725923/16:39
AJaegerLinks just to "finger://ze09.openstack.org/9a693d2cf30e4ba1ba649d0df2e3e291" in one case ;(16:40
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567816:40
AJaegerOpen10K8S: best to recheck and follow the log file to see what's happening16:40
Open10K8SAJaeger: ok, gotcha16:41
fungior wait until one of those reports and see if you get anything useful from the build result page for it16:41
Open10K8Sfungi: ok, anyway, could you share some infor related to this failure or your idea?16:41
Open10K8Sfungi: i mean the root cause :)16:42
AJaegerOpen10K8S: Is that a job using container?16:42
Open10K8Syeah, exactly k8s16:42
AJaegeropenstack-operator:linters:tox does not use container, does it?16:43
smcginnisLot's of CANCELED and a POST_FAILURE on this one as well: https://review.opendev.org/#/c/722715/16:43
Open10K8Saha, yes, I only mentioned the whole pipeline16:43
AJaegerinfra-root, 722715 looks serious ^16:44
smcginnisSeeing quite a few post_failures looking at the check queue.16:47
mnaserPOST_FAILURE seems to occur after log upload16:49
mnaserhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e04/722715/1/gate/openstack-tox-lower-constraints/e0438b4/job-output.txt this POST_FAILURE'd and not really anything helpful16:49
fungi2020-05-06 16:41:57,490 INFO zuul.ExecutorClient: [e: 55d61c7159914bd1a5117b757ecf5db3] [build: 02e48b46d36b4622aae99c3df5063525] Cancel build <Build 02e48b46d36b4622aae99c3df5063525 of openstack-tox-docs voting:True on <Worker ze01.openstack.org>> for job openstack-tox-docs16:49
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567816:50
corvusfungi, AJaeger, smcginnis: well, it looks like someone configured fail-fast on octavia: https://opendev.org/openstack/octavia/src/branch/master/zuul.d/projects.yaml#L6116:51
corvusso, i mean, zuul did exactly what you asked it to?16:51
corvusi did not think we were planning on using that in openstack16:52
fungiyeah, that looks to be the case for the cancelled states16:52
johnsomI did16:52
smcginnisAh, OK. I haven't done enough with octavia to have seen that one before.16:52
fungiso it's just the openstack-tox-lower-constraints post_failure which triggered all those16:53
johnsomThose jobs can run well over an hour, so since there is no point letting them run if one fails I enabled that.16:53
smcginnisSo looks like just a transient failure that then caused the rest of the CANCELED ones.16:53
smcginnisMakes sense.16:53
fungiand that looks like it could be a rogue vm in inap16:53
fungisomething at least caused the executor to think the ssh host key for the job node changed when it tried to collect some data from it16:54
smcginnisWould that be what's impacting all of those POST_FAILURES?16:54
fungiif the other post_failure results all correlate to similar ip addresses in inap, then that's the most likely explanation16:54
smcginnisHere's a release job failure: https://zuul.opendev.org/t/openstack/build/11cee1a24fc64f048459c7eb732f549716:55
smcginnisNo link to results, so I'm not sure.16:55
fungii'll see if we have something useful in executor logs on that one16:56
clarkbmnaser: note we can't tell if the issue is after log upload as we upload before the upload completes (if that makes sense)16:56
clarkbmnaser: from a logging perspective there is a race there that we can't solve without time travel16:57
clarkbwhat we can do though is check the executor logs16:58
clarkbas those don't have the upload race with time moving in one direction16:58
fungii see "Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Service Unavailable (HTTP 503)" in a traceback when communicating with ovh (swift presumably)17:00
fungiso maybe at least some of these post_failure results are an ovh swift issue?17:01
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567817:01
fungiyeah, the exception was raised when running upload-logs-swift17:02
clarkbfungi: we forced v3 there iirc because they were dropping v217:02
clarkbfungi: possible they don't like the force of v3 anymoer?17:03
fungiooh, maybe that was taking effect today17:03
fungiand yeah, i find a few of those in the executor-debug.log on ze01, but not prior to 16:32:3717:04
fungii'll check other executors17:04
clarkbin mnasers example we have: 2020-05-06 16:40:49,425 DEBUG zuul.AnsibleJob.output: [e: 55d61c7159914bd1a5117b757ecf5db3] [build: e0438b4528fa4c559a09b504025543e7] Ansible output: b'[WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin' which is a different issue during fetch-subunit-output17:04
clarkbthats an actual issue in zuul's callback plugins I think. Unfortunately thats as much info as we seem to get17:04
corvusclarkb: i don't think that's a fatal error?17:07
fungiokay, so every executor has logged multiple tracebacks for ovh swift like that, the earliest was 15:00:05 on ze1117:08
clarkbcorvus: ya maybe. Its confusing what we get in the output it says failed: 1 with no obvious failures then later 2020-05-06 16:40:50.196207 | POST-RUN END RESULT_NORMAL: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/unittests/post.yaml@master so maybe it is reporting it as a failure but not actually failing?17:08
clarkbfungi: on bridge I can run catalog list against BHS1 and GRA1 and we don't seem to force an identity version so thats my best guess still17:09
fungishould we temporarily disable ovh swift in zuul, or try that as a fix?17:09
clarkboh we don't set it for our ovh secret either from what I can see17:10
corvusclarkb: the error for e0438b4528fa4c559a09b504025543e7 on ze01 is WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!17:11
fungialso possible this was transient... checking to see when the last errors were logged17:11
fungicorvus: for that one, was it also in inap?17:11
fungiovh swift error does not seem to have abated on its own, btw, ze02 saw it as recently as 17:11:2817:12
corvusfungi: yes e0438b4528fa4c559a09b504025543e7 on ze01 was on ubuntu-bionic-inap-mtl01-001640650417:12
fungicorvus: so that makes two post failures we've seen for a host key error in inap, https://zuul.opendev.org/t/openstack/build/e0438b4528fa4c559a09b504025543e7 was another17:13
clarkbcorvus: I guess that doesn't show up if you grep the uuid? I don't get that in my grep output17:13
fungicorvus: oh, wait, that's the same one i already looked at earlier17:13
fungiso we only have one case of that failure17:14
clarkblooks like we pass in opendev/base-jobs/zuul.d/secrets.yaml:opendev_cloud_ovh_bhs directly as the cloud config to shade/openstacksdk17:16
clarkbthats a profile, auth dict with name, passwd, and project name, then a region name17:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567817:16
clarkbthe project name value should force an implied v3 identification17:16
clarkbthats basically the same config that works on bridge17:17
clarkbthis makes me wonder if it is a "we need to update openstacksdk to have new profile for ovh" problem17:17
corvusclarkb: it does show up in a grep for the uuid; log line starts with 2020-05-06 16:41:43,430 DEBUG zuul.AnsibleJob.output: [e: 55d61c7159914bd1a5117b757ecf5db3] [build: e0438b4528fa4c559a09b504025543e7] Ansible output17:18
fungiopenstacksdk 0.46.0 is what's on the executors, ftr17:18
clarkbze01 openstacksdk==0.41.0 bridge is ^ ya that17:18
clarkbfungi: remember zuul executors run ansible in venvs17:19
clarkbits the venv version that matters here I think17:19
fungioh, is this ansible calling openstacksdk?17:19
clarkbfungi: yes17:19
fungi/usr/lib/zuul/ansible/2.8/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py17:20
fungiindeed17:20
corvusopenstacksdk==0.41.017:20
corvusis what i see on the 2.8 venv on ze0117:20
fungiopenstacksdk 0.41.0 according to /usr/lib/zuul/ansible/2.8/bin/pip17:20
fungiyep17:20
fungii guess we just also have openstacksdk installed globally17:21
fungis/globally/in the system context/17:21
clarkbthe ovh vendor profile hasn't changed meaningfully17:22
clarkbits possible its a bug/change in openstacksdk itself then17:22
clarkbcorvus: re that REMOTE HOST I guess I was blind or mixing up my greps on ze01 and of https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e04/722715/1/gate/openstack-tox-lower-constraints/e0438b4/job-output.txt any idea why it doesn't show up in the uploaded log?17:23
clarkbmaybe I'm still blind and it is there too17:23
openstackgerritMerged opendev/system-config master: Configure nodepool to use logging config  https://review.opendev.org/72588917:23
corvusclarkb: i don't think all the ansible warnings go in to the console log17:24
fungii feel like we ought to disable uploads to ovh swift while we keep troubleshooting keystone/sdk17:24
fungii can push up that change unless there are objections17:24
clarkbfungi: I've yet to reproduce outside of zuul though17:24
clarkbso we may be turning off our ability to debug this easily (we can use base-test though)17:24
clarkband ya that seems like a reasonable next step. maybe switch base-test to ovh only too and we can start iterating there?17:25
fungiare additional job failures going to make this easier to debug? we already have dozens (maybe hundreds)17:25
clarkbfor reproduction I've installed openstacksdk==0.41.0 in a venv on bridge and am using the clouds.yaml there (whcih appears equivalent to my eye)17:25
fungii'll do the base-test tweak in the same change, sure17:25
clarkbfungi: its not additional failures, but being able to cahgne something and then check if that cahnges behavior since we don't have a reproducer right now17:26
*** sshnaidm is now known as sshnaidm|afk17:26
fungii can't remember, is this all inherited from opendev/base-jobs now or do i need to make a separate change for each tenant's config repo?17:27
clarkbfungi: it should all be in opendev/base-jobs17:27
fungithanks, that's quicker at least17:27
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567817:27
clarkbhttps://auth.cloud.ovh.net/ produces what I expect from my home network connection17:31
openstackgerritJeremy Stanley proposed opendev/base-jobs master: Temporarily disable OVH Swift uploads  https://review.opendev.org/72594317:32
fungiinfra-root: ^17:32
clarkbfungi: ah ok I see now you noted we get an HTTP 50317:32
clarkbthat likely explains why we can't reproduce17:32
clarkbits a server side error17:32
fungiwe might want to submit that in gerrit to reduce delay, especially since it's likely to post_failure on the problem it's trying to stop17:32
clarkbfungi: before we land that can we check really quickly if we still observe the 503s?17:32
fungichecking17:33
fungimost recent occurrence was 17:11:39 on ze0417:35
fungithere's every chance that this is only happening for some fraction of api calls, and so could require lots of tries to reproduce17:35
clarkblooks like they happened in clusters about every 20 minutes or so from 15:40 to 17:1117:36
fungialso maybe it was finally fixed 25 minutes ago17:36
corvusclarkb, fungi: todays post failures: http://paste.openstack.org/show/793212/17:36
fungiwell, i also saw it as early as 14:0017:36
fungier, 15:0017:36
clarkband this is executor to ovh regionless url so won't be region specific17:37
corvusi think the cluster around 1630 is what got people's attention.  it may have settled back down to normal levels17:37
corvus(we can't tell whether those post failures are all log upload failures)17:38
clarkbhttp://travaux.ovh.net/?do=details&id=44436 possibly related17:38
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567817:38
clarkbhowever I don't think the timestamps line up properly for us in that ticket17:39
clarkbcorvus: fungi ya maybe we can check the list again in 10 minutes or so and see if it has grown new cases?17:40
clarkbif it has then remove ovh from the rotation if not then its looking happier?17:40
corvusyeah, i'm inclined to think it's abated17:41
fungiclarkb: corvus: yeah, could have been a temporary glitch while they were maintenancing17:42
clarkbthat said I think we may want to consider updating openstacksdk periodically in those virtualenvs. If we get to running in containers I think that will happen automatically17:42
clarkbopenstacksdk gets updates to handle changes in real world cloud deployments so being able to pull those in seems like a good thing17:43
clarkb(and maybe the answer is just containers)17:43
corvuscontainers on executors are still blocked by afs17:43
clarkbze07 and ze12 have just ercordred occurences at 17:4417:45
clarkbbut only one each17:45
fungiso it's infrequent, but frequent enough to be disruptive17:46
clarkbya there has been talk of having that role retry but the way we feed it random inputs makes that difficult17:47
clarkbessentially we'd we retrying against the same cloud which in many cases is the wrong thing (though in this case might be the correct thing)17:48
clarkbor correct enough in this case anyway since the 503 seems infrequent17:48
*** hashar has joined #opendev17:58
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567818:05
*** ralonsoh has quit IRC18:19
*** lpetrut has quit IRC18:22
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567818:23
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567818:44
clarkbmordred: is nl01 happy now? should we proceed with the others?18:58
mordredclarkb: let me restart it and make sure it's logging to the right place18:59
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567819:03
openstackgerritMonty Taylor proposed opendev/system-config master: Pass -f to nodepool to run in foreground  https://review.opendev.org/72596719:07
mordredclarkb, fungi, corvus : ^^19:08
mordredwe almost got the last patch right19:08
mordredclarkb: I applied that by hand on nl01 and it's happy19:08
mordredclarkb: so once we land that patch ^^ I believe we can proceed with the others19:08
mordred(and nl01 - bugs not withstanding, went quickly - so we shoudl be able to knock this out)19:09
clarkbmordred: I don't understand the relatonship between logging config and foreground vs not19:09
mordredclarkb: we are overriding the command line in the compose file19:09
mordredthe commmand line in the dockerfile that we're overriding has a -f19:09
clarkbgotcha19:09
mordredsorry - I englished bad19:10
clarkband because its docker it doesn't like proper daemon19:10
mordredwell - that -- and also when we daemonize we want to write a pidfile19:10
mordredand we want to write it in a location that doesn't exist in the container19:10
mordredso it flat won't start19:10
*** DSpider has joined #opendev19:13
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567819:22
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567819:34
mordredclarkb: feel like +Aing https://review.opendev.org/#/c/722483/ and https://review.opendev.org/#/c/725611/ ?19:35
clarkbmordred: done19:37
clarkbmordred:  Iguess once the container nodepool rollout is done we can replace all the builders and launchers with bionic (or even focal?) hosts?19:38
mordredclarkb: yup19:39
mordredclarkb: should be a piece of cake19:39
mordredclarkb: speaking of: https://review.opendev.org/#/c/723528/ fixes base-server for focal19:40
mordredclarkb: so once that's in - we can basically do a rolling replacement of ze* nl* nb* zm* with focal19:40
clarkbmordred: for some reason I thought we had a 5 node limit on our jobs?19:42
clarkb(that adds a 6th node)19:42
mordredclarkb: we apparently do not19:46
fungii thought it was 7, but i really don't have much evidence on which to base that vague recollection19:47
corvus10: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L319:49
corvussince https://opendev.org/openstack/project-config/commit/58c9ca0e9a910e8bf101f2b4a95abbf7aac8f0ce19:50
clarkbmordred: noted a couple small cleanups on that change I think we should do to avoid confusion19:50
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567819:50
corvuslooks like we're maintaining our "looking one year ahead" track record :)19:51
mordredclarkb, corvus: should we keep installing jemalloc on the executors?19:53
mordredespecially on focal19:53
clarkbmordred: I've had a bit of a think on that. My hunch is that the jemalloc on Kubic (is that what our base docker images are?) is buggy19:54
clarkbmordred: because python is going to use the same number of mallocs and frees if running with jemalloc or not19:54
mordredclarkb: s/kubic/buster/19:54
clarkbmordred: its likely that newer jemalloc on ubuntu would suffer similar issues if sharing a similar jemalloc version19:55
mordredyeah - focal is also jemalloc219:55
corvusyeah, given that things are looking better without it so far (under different circumstances, but it's the best we got), i'd say lets try focal without jemalloc19:55
mordredso I think based on our container experience we should avoid jemalloc on focal19:55
mordred++19:55
openstackgerritMonty Taylor proposed opendev/system-config master: Test zuul-executor on focal  https://review.opendev.org/72352820:00
mordredcorvus, clarkb: how do those changes look?20:00
mordredI realized we also weren't actually, you know, using jemalloc1 on not-focal even though we were installing it20:01
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567820:02
openstackgerritMonty Taylor proposed opendev/system-config master: Test zuul-executor on focal  https://review.opendev.org/72352820:04
mordredclarkb: I mean - we _were_ using jemalloc on the existing executors - but only be accident since there was a defaults file there already20:04
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567820:10
fungiinfra-root: just as a heads up, we're still seeing keystone 503 errors out of ovh as recently as 20:02:3020:11
clarkbfungi: I did +2 the change back when it was pushed so that we can land it if necessary20:12
fungimake that 20:08:1220:13
*** diablo_rojo has joined #opendev20:13
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567820:18
openstackgerritMerged opendev/system-config master: Pass -f to nodepool to run in foreground  https://review.opendev.org/72596720:19
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567820:32
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567820:43
*** hashar has quit IRC20:49
fungiinfra-root: update from conversation with an ovh engineer (rledisez) in #openstack-infra just now, this does seem to be a problem on their end and is now being looked into since clarkb brought it to their attention20:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role  https://review.opendev.org/72567820:53
slittle1Not long ago we created a new repo  starlingx/kernel.git20:56
fungii remember20:57
slittle1I do the branch management for starlingx, but I don't seem to be able to push a branch into starlingx/kernel.git20:57
slittle1a new branch that is20:57
slittle1I'm ondering if we missed something20:58
slittle1ondering ?   make that 'wondering'20:58
fungilooks like you're in the starlingx-release group which has create-reference permission for refs/heads/* in that repo20:59
fungislittle1: if you go to https://review.opendev.org/#/admin/projects/starlingx/kernel,branches is the create branch button available20:59
fungi?21:00
slittle1yes21:00
fungiyou should be able to plug a branch name and then a ref into the fields there and create a new branch21:01
fungithis version of gerrit isn't great about equating that to pushing a new branch via git protocol, unfortunately, if that's what you were trying21:01
slittle1I normally work on large numbers of gits at the same time via bash scripts21:01
fungii'm not sure if that will improve in the near future when we upgrade, but it might21:01
fungithere is a rest api which can be used to create branches21:02
fungithe ssh "command-line" gerrit api also has a create-branch command21:02
slittle1so  'git push gerrit <new_branch>'  isn't going to be working for any of our gits ?21:03
slittle1ok, can you point me to a reference for that ?21:03
funginot if the branch doesn't exist yet. branch creation in gerrit is a separate step, at least in gerrit 2.1321:03
fungiyeah, getting you the reference docs for both rest and ssh options now21:03
fungihttps://review.opendev.org/Documentation/rest-api-projects.html#create-branch21:04
fungihttps://review.opendev.org/Documentation/cmd-create-branch.html21:04
fungislittle1: ^ those are scriptable alternatives to the webui "create branch" form21:05
clarkbthe openstack release team has scripts to automate branch creation which may be helpful to21:05
fungislittle1: if you're configured to push via https then the rest api will use the same credentials, or if you push to gerrit via ssh on 29418/tcp then the ssh command-line api uses the same ssh key21:06
mordredclarkb: nl01 seems good. I'm going to move on to nl02 if you concur21:09
slittle1ok, one snag21:10
clarkbmordred: give me a few minutes and I can check in the host21:10
mordredclarkb: cool thanks. I've put the disable file on bridge fwiw21:11
slittle1I'm not branching from an existing revision21:11
fungislittle1: in git there's no such thing21:11
fungiall branches share some history, even if it's the base commit21:11
slittle1not so sure that's tru21:12
mordredfungi: yeah - I agree with slittle1  - this is how we stich different repos together21:12
fungislittle1: there are tricks you can use to graft a branch into a repository where it shares no history with other branches, but that requires rewriting the branch, i think21:12
mordredthey wind up with two completely separate history parents21:12
slittle1exactly what I'm trying to do21:13
* fungi checks on that process21:13
fungibeen a while since i've had to do that21:13
slittle1There was a subdirectory that should have been included in in kernel.git21:14
slittle1I'm tryinh to transfer that directory while preserving it's history21:14
slittle1I had hoped to deliver it to a side branch21:14
slittle1then put the merge commit for normal review21:15
mordredI'll see what fungi finds - I think what you might need to do is create a branch from some ref (doens't matter) - then force-push to that branch (which will overwrite it with your non-parent-sharing new import)21:15
clarkbfwiw the 0 * 40 sha is the null parent21:15
clarkbso fungi is right but there is a hack21:16
mordredand then once you have done that, the refs in it will all exist in gerrit, so when you do the merge commit, it will only be the merge commit that gets reviewed21:16
clarkbto make that true for the first commit21:16
mordredah - cool21:16
fungislittle1: so... just to confirm, you have a bunch of commits already in a branch which aren't reviews in gerrit, and *now* you want to push them into the repository, completely bypassing the code review system21:16
slittle1strictly speaking, they were reviewed in the old project.21:17
fungioh, these are being copied from a different gerrit project?21:18
slittle1don't want to subject the community to dozens of re-reviews21:18
slittle1was hoping for a single review in the merge commit21:18
fungiwell, either way, it's going to require an admin to push --force the commits since they're not being pushed for review21:18
clarkbmordred: I seea running nodepool-launcher, /usr/local/bin/nodepool is not container installed but running list with it showed me a recently used node. Grepping /var/log/nodepool/launcher-debug.log for that nodeid shows me the node was created after the launcher process started21:19
openstackgerritMonty Taylor proposed opendev/system-config master: Test zuul-executor on focal  https://review.opendev.org/72352821:19
clarkbmordred: thats all looking great to me on nl01. I think the only thing I've noticed is we may need to cleanup the global install at some point21:19
clarkbmordred: ++ to nl0221:19
fungislittle1: if you can get the branch you want pushed in published somewhere public i can pull from, i'll add it to the repo in gerrit with whatever branch name you need21:19
mordredclarkb: yes to global install (although that'll get sorted when we re-install on focal)21:20
slittle1ok, so I need to push what I have up to github and ask one of you to do it ?21:20
slittle1the force push that is21:20
fungislittle1: can be github, or really anywhere i can clone from21:21
fungii frequently just push via ssh to a webserver i have access to, that also works21:21
fungijust needs to be reachable long enough for me to clone21:21
mordredclarkb: playbooks/service-nodepool-manual.yaml in /home/zuul/src/opendev.org/opendev/system-config has the playbook I'm using21:21
mordredclarkb: and I am running in a screen21:22
clarkbmordred: k. I'm not on the screen now but let me know if I should be21:24
mordredclarkb: nope. going smoothly. nl02 is done21:28
mordredmoving on to nl0321:28
slittle1ready   https://github.com/slittle1/starlingx-kernel-additions.git    branch=move-mellanox-with-history21:28
fungislittle1: thanks, working on it now21:29
mordrednl03 is done - moving on to 0421:33
fungislittle1: you want the branch in gerrit to be called move-mellanox-with-history as well?21:33
slittle1curiosity:  'fungi' as in 'fun guy'   :)21:34
slittle1yes please21:34
fungislittle1: fungi in my case is an obscure reference to a series of sonnets by an early 20th century horror and science fiction author named howard phillips lovecraft21:35
slittle1cool21:35
* corvus can confirm fungi is fun nonetheless21:36
fungii try ;)21:36
*** DSpider has quit IRC21:36
corvus"fungi is fun in a lovecraftian sort of way"21:37
mordred#status log finished rolling out nodepool ansible change - all launchers now running in docker21:39
openstackstatusmordred: finished logging21:39
mordredinfra-root: ^^21:39
mordredI have removed nodpeool from the emergency file and removed he ansible lock on bridge21:40
clarkbmordred: \o/21:40
fungicorvus: that's likely closer to the truth21:41
fungislittle1: okay, this is the output from pushing (looks like --force was unneeded): http://paste.openstack.org/show/793221/21:42
fungislittle1: see if the history here looks right to you: https://opendev.org/starlingx/kernel/commits/branch/move-mellanox-with-history21:43
slittle1yes, that looks correct21:43
fungiit still would have needed an admin for the impersonate committer permissions21:43
fungislittle1: cool, don't hesitate to let us know if you need anything else21:44
slittle1thanks21:44
fungiany time!21:45
*** smcginnis has quit IRC21:46
clarkbI'm overdue for a bike ride and the sun just popped out. Going to afk for a bit21:46
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Tag the images pulled back into docker for changes  https://review.opendev.org/72600721:47
ianwinfra-root: if i could get one more eye on https://review.opendev.org/#/c/725749/ i think we're ready to drop pip-and-virtualenv from suse at least.  i can monitor it, but we're at a point we can roll-back by just re-adding the element and rebuilding now21:51
ianwalso https://review.opendev.org/725737 ; this modifies ensure-tox to install in a venv instead of --user and is required to run testinfra on our plain nodes, because we run it as a different user21:52
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support  https://review.opendev.org/72030221:53
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Tag the images pulled back into docker for changes  https://review.opendev.org/72600721:54
mordredianw: have fun!21:55
mordredianw: isn't that zuul-jobs patch going to break on xenial for people who are not using opendev nodes?21:56
ianwmordred: i don't think so ... {{ ensure_pip_virtualenv_command }} should aways be a working thing that you can put into virtualenv_command, anywhere21:57
openstackgerritMonty Taylor proposed opendev/system-config master: Add focal to system-config base job  https://review.opendev.org/72567621:59
mordredianw: ok - I just thought there was something about old pip on xenial that's broken or something22:00
mordredbut - I guess if that was what's on the node befoore, this is no different22:00
ianwmordred: it's pip 8 which xenial ships that has problems falling back to pypi with our mirrors; but for any external users (if there are any) who aren't setting up mirroring there won't be any issues22:03
ianwthat was what https://review.opendev.org/#/c/724788/ works around22:03
clarkbmordred: so I dont forget. Did we disable the system units for nodepool on nl01-04?22:04
*** smcginnis has joined #opendev22:05
ianwas soon as i drop pip-and-virtualenv for xenial, we can consistently be using python3 -m venv to install things.  i.e. we can be out of the virtualenv game -- if jobs want it they can manage it themselves (we give them ensure-virtualenv, but it's not required)22:06
openstackgerritMerged openstack/project-config master: Drop pip-and-virtualenv from SUSE 15  https://review.opendev.org/72574922:07
mordredclarkb: no - let me make a patch do that. I think we need to do it on zuul hosts too right?22:08
mordredclarkb: and - it looks like we were never installing units - we were installing sysvinit scripts and letting compat take care of it22:08
clarkbmordred: ya sorry `systemctl disable foo` basically that works for units and compat22:11
openstackgerritMonty Taylor proposed opendev/system-config master: Remove old init scripts and services for zuul/nodepool  https://review.opendev.org/72601122:15
openstackgerritMonty Taylor proposed opendev/system-config master: Revert "Remove old init scripts and services for zuul/nodepool"  https://review.opendev.org/72601222:15
mordredclarkb: there's two - one to land and run and disable/remove things - then a second to remove that from ansible os we don't do it a billion times22:15
corvusmordred: is it correct that we can remove nodepool-base-legacy role now?22:18
mordredcorvus: we still have three hosts on puppet22:19
mordredcorvus: we should be clear to shift them over to ansible/docker now22:19
corvusmordred: ack22:19
mordredianw: any reason to not just update nb01 and nb02 in place?22:20
* mordred is pretty sure we should boot a second builder on arm running the new arm container to verify that22:20
corvusmordred: hostnames for nb01/02?22:21
mordredor - we could also combine it with fresh new servers and roll out nb01.opendev.org and nb02.opendev.org22:21
corvusya that22:21
mordredyeah - so maybe that's the cleanest thing to do22:21
ianwmordred: also config things changed a bit, so i think it would leave a bit of a mess22:21
mordredyeah - I think for the zuul hosts that we did in place doing a rolling replacement is going to be nice22:21
mordredinfra-root: we can now land these: https://review.opendev.org/#/q/topic:infra-prod-zuul+status:open which should open the door to clarkb's reorg of zuul.yaml22:22
ianwi can work on bringing them up now, if we're ready22:22
mordredianw: I think we are, yes22:23
mordredianw: also - well - one sec22:23
ianwwe have nb04 running, so really we could just create 05 and shutdown 01/02 and be in the same capacity22:23
mordredianw: ++22:23
mordredsounds great to me22:24
ianwok, i'll start on that22:24
mordredI think the only thing we have any questions on wrt proceeding is 03 - and we need to publish the docker image for us to test that22:25
corvuswhy wouldn't we call them 01 and 02?22:25
corvusthey're under different domains22:25
openstackgerritMerged zuul/zuul-jobs master: Tag the images pulled back into docker for changes  https://review.opendev.org/72600722:25
mordredgood point22:25
ianwcorvus: when i did try that things exploded :)22:26
corvusdid that not get fixed?22:26
fungii want to say after that we merged a change to start using the fqdn for identifiers in the metadata22:27
fungiinstead of just the short hostname22:27
corvusfungi, ianw: or maybe this was the fix?  https://review.opendev.org/71305722:28
ianwyeah i feel like i commented on a change to confirm that, looking22:28
ianwyeah that's the one i'm thinking of22:29
corvusso maybe we should go with 01/02 and place our bets on us having actually fixed that :)22:29
ianwi can try nb01.opendev.org, and if something doesn't work we'll know what to look at22:29
mordred++ I think that's a great plan22:31
ianware we at the point to try it as a focal image?22:31
mordredianw: this needs to land - but I think it's ready to: https://review.opendev.org/#/c/723528/22:32
mordredianw: probably worth pushing up an patch to add an nb01.opendev.org to the nodepool test running on focal just to make sure22:33
mordredianw: or - we could just stick to bionic on the builders since we know it's working22:34
fungi#status log deleted openstack/rally branches stable/0.10 (67759651f129704242d346a2c045413fcdea912d) and stable/0.12 (99f13ca7972d7f64b84204c49f1ab91da6d6cb6b) per https://review.opendev.org/721687 (stable/0.9 left in place for now as it still has open changes)22:34
mordredand leave focal for later22:34
openstackstatusfungi: finished logging22:34
mordredcorvus: what do you think?22:34
corvusmordred: that change is 99% to the executors22:34
corvusmordred: i think trying focal on the builders is fine, but we should separate that out22:34
mordred++22:35
mordredso let's roll out new builders on bionic to get rid of puppet22:35
mordredand unblock the zk tls work22:35
corvussounds good22:35
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support  https://review.opendev.org/72030222:38
corvusokay, throwing spaghetti at the wall and seeing what sticks :)22:38
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: add Zookeeper TLS support  https://review.opendev.org/72030222:45
mordredcorvus: mmm. spaghetti22:52
*** tosky has quit IRC22:56
openstackgerritIan Wienand proposed opendev/zone-opendev.org master: Add nb01/nb02.opendev.org  https://review.opendev.org/72601822:59
openstackgerritMonty Taylor proposed opendev/system-config master: Test zuul-executor on focal  https://review.opendev.org/72352823:00
mordredianw: woot23:03
openstackgerritIan Wienand proposed opendev/system-config master: Add nb01/nb02 opendev servers  https://review.opendev.org/72602123:11
ianwwe probably have to reboot graphite as i think it is old enough to have a firewall rule from the last time we tried nb01.opendev.org23:11
mordredcorvus: hrm. there still may be a flaw in our wondrous multi-arch work23:13
mordredbut it'll have to be a tomorrow thing23:13
clarkbmordred: the init script stack is +2's all around from me23:38
clarkbmordred: and looks like I've already reviewed the other stack you linked23:39
clarkbmaybe tomorrow we'll be able to land those and the reorg change?23:39
ianwclarkb: if you have second for https://review.opendev.org/#/c/725737/, i might be able to get a arm64 nb container test up23:44
ianwit has to run on a full arm64 stack, though, see https://review.opendev.org/#/c/724439/and https://review.opendev.org/#/c/724435/123:45
clarkbianw: looking23:45
clarkbianw: hrm I worry about that move breaking jobs if they hardcode the old path to tox23:46
ianwthey should be using {{ tox_executable }} though?23:47
clarkbianw: yes, but it wouldn't surprise me if they did something like /home/zuul/.local/pip/bin/tox (I don't know where --user actually stashes it)23:47
clarkbwe can check codesearch for that path and if it shows no results is probably reasonably safe and people are using the ansible var instead23:48
clarkb/home/zuul/.local/bin/tox23:48
clarkbianw: https://opendev.org/x/tobiko/src/branch/master/roles/tobiko-ensure-tox/tasks/tox.yaml#L24 thats the only case I find in codesearch23:50
fungihttp://codesearch.openstack.org/?q=local%2Fbin%2Ftox&i=nope&files=&repos=23:50
fungiyeah, tobiko is the only one i found23:50
fungiand zuul-jobs obviously23:50
ianwarrgghhh23:51
ianwoh, hang on, that's not using our role?23:51
ianwhttps://opendev.org/x/tobiko/src/branch/master/roles/tobiko-ensure-tox/tasks/tox.yaml#L323:51
fungiconvenient!23:51
clarkbianw: oh thats a good point they do their own install at hte top weird23:52
ianwit looks like nb01.openstack.org is borked ATM anyway ... a stuck process it seems23:52
clarkbso ya maybe it is safe afterall (fwiw I+2'd but didn't approve thinking we could fix tobiko then approve zuul-jobs but maybe we can just land it now)23:52
clarkbianw: is it safe to land https://review.opendev.org/#/c/724439/1 or do we want to wait for the tox fixing first?23:54
ianwclarkb: it's safe to land, as it doesn't vote23:54
ianwi think it's that the focal builds bail on the xenial hosts and then leave behind lockfiles.23:54
ianwat this point, i'm just going to push on with the replacement nodes23:55
ianwhttps://review.opendev.org/#/c/726021/ to add the new nodes just reported back, clarkb/fungi maybe you could take a look23:56
clarkbuhm is it safe to reuse the names?23:56
clarkbI feel like we fixed it sufficiently that it would be now but I also don't want to repeat that deletion thing all over again23:56
ianwwe discussed it @http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-05-06.log.html#t2020-05-06T22:29:1323:57
clarkbthanks23:57
openstackgerritMerged opendev/zone-opendev.org master: Add nb01/nb02.opendev.org  https://review.opendev.org/72601823:58
ianwgraphite is the only thing i can see that is holding on to an old firewall rule for the old nb01, i'll reboot that quickly once the new adress resolves ^23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!