| ianw | ok back to nb02 | 00:04 |
|---|---|---|
| ianw | i wonder if all the nested mounts etc means docker pull is unreliable | 00:04 |
| ianw | seems to me "docker-untar / /var/lib/docker/overlay2/c4ccbf94c05603eb4e317f89117879b686031bbe666b77d5a172bf2a7b055126/diff" is what is stuck | 00:05 |
| clarkb | its a different fs thugh | 00:06 |
| clarkb | or maybe not | 00:07 |
| *** mlavalle has quit IRC | 00:08 | |
| ianw | something is up with this host | 00:10 |
| ianw | 2021-04-08 07:13:13.896 | (0001 / 1391) | 00:10 |
| ianw | 2021-04-08 14:54:41.895 | Cloning from ceilometer cache and applying ref * | 00:10 |
| clarkb | wow | 00:11 |
| ianw | it spent 7 hours in the git cloning loop | 00:11 |
| ianw | before something killed it | 00:11 |
| ianw | that untar is going, but *really* slowly | 00:11 |
| ianw | dmesg has a bunch of dracut output ... | 00:13 |
| ianw | it's not "/opt", i just cleaned that up and it went quickly | 00:24 |
| ianw | i'm just going to reboot it, i don't have much else insight | 00:25 |
| ianw | it's back, "docker-compose pull" got new images quickly and it's building | 00:28 |
| ianw | #status log reboot nb02.opendev.org after mystery slowdown, appears to be root disk related | 00:28 |
| openstackstatus | ianw: finished logging | 00:28 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Handle zuul-summary-results as .jar / per-project config https://review.opendev.org/c/opendev/system-config/+/778116 | 01:22 |
| *** hamalq has quit IRC | 01:32 | |
| *** spotz has quit IRC | 01:36 | |
| openstackgerrit | Merged opendev/system-config master: zuul-summary-status : handle SKIPPED and ERROR jobs https://review.opendev.org/c/opendev/system-config/+/777298 | 02:09 |
| openstackgerrit | Merged opendev/system-config master: Set MaxConnectionsPerChild 8192 for Gitea backends https://review.opendev.org/c/opendev/system-config/+/785226 | 02:13 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Handle zuul-summary-results as .jar / per-project config https://review.opendev.org/c/opendev/system-config/+/778116 | 02:50 |
| *** ykarel has joined #opendev | 03:48 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: review02: pin ipv6 configuration https://review.opendev.org/c/opendev/system-config/+/785556 | 04:51 |
| *** marios has joined #opendev | 05:07 | |
| *** ysandeep|away is now known as ysandeep | 06:10 | |
| *** slaweq has joined #opendev | 06:11 | |
| *** amoralej|off is now known as amoralej | 06:16 | |
| *** sboyron has joined #opendev | 06:21 | |
| *** eolivare has joined #opendev | 06:27 | |
| *** fressi has joined #opendev | 06:56 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Add Debian Bullseye Zuul job https://review.opendev.org/c/openstack/diskimage-builder/+/783790 | 07:18 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Add Debian Bullseye Zuul job https://review.opendev.org/c/openstack/diskimage-builder/+/783790 | 07:20 |
| *** gothicserpent has quit IRC | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:33 | |
| *** andrewbonney has joined #opendev | 07:38 | |
| *** ChanServ has quit IRC | 07:58 | |
| *** andrewbonney has quit IRC | 07:58 | |
| *** ykarel has quit IRC | 07:58 | |
| *** paladox has quit IRC | 07:58 | |
| *** lourot has quit IRC | 07:58 | |
| *** JayF has quit IRC | 07:59 | |
| *** iurygregory has quit IRC | 07:59 | |
| *** mugsie has quit IRC | 07:59 | |
| *** cloudnull has quit IRC | 07:59 | |
| *** fressi has quit IRC | 07:59 | |
| *** sboyron has quit IRC | 07:59 | |
| *** Jeffrey4l has quit IRC | 07:59 | |
| *** ianw has quit IRC | 07:59 | |
| *** owalsh has quit IRC | 07:59 | |
| *** hberaud has quit IRC | 07:59 | |
| *** _dvd has quit IRC | 07:59 | |
| *** priteau has quit IRC | 07:59 | |
| *** odyssey4me has quit IRC | 07:59 | |
| *** gnuoy has quit IRC | 07:59 | |
| *** dtantsur|afk has quit IRC | 07:59 | |
| *** tkajinam has quit IRC | 07:59 | |
| *** mordred has quit IRC | 07:59 | |
| *** tristanC has quit IRC | 07:59 | |
| *** mkowalski has quit IRC | 07:59 | |
| *** parallax has quit IRC | 07:59 | |
| *** auristor has quit IRC | 07:59 | |
| *** donnyd has quit IRC | 07:59 | |
| *** gmann has quit IRC | 07:59 | |
| *** osmanlicilegi has quit IRC | 07:59 | |
| *** CeeMac has quit IRC | 07:59 | |
| *** jaicaa has quit IRC | 07:59 | |
| *** noonedeadpunk has quit IRC | 07:59 | |
| *** jrosser has quit IRC | 07:59 | |
| *** mwhahaha has quit IRC | 07:59 | |
| *** erbarr has quit IRC | 07:59 | |
| *** jroll has quit IRC | 07:59 | |
| *** persia has quit IRC | 07:59 | |
| *** fdegir has quit IRC | 07:59 | |
| *** Eighth_Doctor has quit IRC | 07:59 | |
| *** jmorgan has quit IRC | 07:59 | |
| *** seongsoocho has quit IRC | 07:59 | |
| *** bbezak has quit IRC | 07:59 | |
| *** knikolla has quit IRC | 07:59 | |
| *** clayg has quit IRC | 07:59 | |
| *** brinzhang_ has quit IRC | 07:59 | |
| *** avass has quit IRC | 07:59 | |
| *** logan- has quit IRC | 07:59 | |
| *** calcmandan has quit IRC | 07:59 | |
| *** dhellmann has quit IRC | 07:59 | |
| *** redrobot has quit IRC | 07:59 | |
| *** zbr has quit IRC | 07:59 | |
| *** fbo has quit IRC | 07:59 | |
| *** dardelean has quit IRC | 07:59 | |
| *** kevinz has quit IRC | 07:59 | |
| *** cgoncalves has quit IRC | 07:59 | |
| *** Tengu has quit IRC | 07:59 | |
| *** zigo has quit IRC | 07:59 | |
| *** jpena|off has quit IRC | 07:59 | |
| *** rpittau has quit IRC | 07:59 | |
| *** mgagne has quit IRC | 07:59 | |
| *** TheJulia has quit IRC | 07:59 | |
| *** clarkb has quit IRC | 07:59 | |
| *** mnasiadka has quit IRC | 07:59 | |
| *** guilhermesp has quit IRC | 07:59 | |
| *** tbarron has quit IRC | 07:59 | |
| *** felixedel has quit IRC | 07:59 | |
| *** fungi has quit IRC | 07:59 | |
| *** walshh_ has quit IRC | 07:59 | |
| *** mattmceuen has quit IRC | 07:59 | |
| *** SotK has quit IRC | 07:59 | |
| *** johnsom has quit IRC | 07:59 | |
| *** mnaser has quit IRC | 07:59 | |
| *** melwitt has quit IRC | 07:59 | |
| *** Alex_Gaynor has quit IRC | 07:59 | |
| *** openstackgerrit has quit IRC | 07:59 | |
| *** dirk has quit IRC | 07:59 | |
| *** ysandeep has quit IRC | 07:59 | |
| *** mgoddard has quit IRC | 07:59 | |
| *** frickler has quit IRC | 07:59 | |
| *** icey has quit IRC | 07:59 | |
| *** amoralej has quit IRC | 07:59 | |
| *** stephenfin has quit IRC | 07:59 | |
| *** gibi has quit IRC | 07:59 | |
| *** mtreinish has quit IRC | 07:59 | |
| *** amotoki has quit IRC | 07:59 | |
| *** ozzzo has quit IRC | 07:59 | |
| *** guillaumec has quit IRC | 07:59 | |
| *** tobiash has quit IRC | 07:59 | |
| *** otherwiseguy has quit IRC | 07:59 | |
| *** arxcruz has quit IRC | 07:59 | |
| *** chandankumar has quit IRC | 07:59 | |
| *** jhesketh has quit IRC | 07:59 | |
| *** corvus has quit IRC | 07:59 | |
| *** eolivare has quit IRC | 07:59 | |
| *** artom has quit IRC | 07:59 | |
| *** stevebaker has quit IRC | 07:59 | |
| *** bodgix has quit IRC | 07:59 | |
| *** prometheanfire has quit IRC | 07:59 | |
| *** toomer has quit IRC | 07:59 | |
| *** yoctozepto has quit IRC | 07:59 | |
| *** mrunge has quit IRC | 07:59 | |
| *** ttx has quit IRC | 07:59 | |
| *** tinwood has quit IRC | 07:59 | |
| *** swest has quit IRC | 07:59 | |
| *** wendar has quit IRC | 07:59 | |
| *** diablo_rojo_phon has quit IRC | 07:59 | |
| *** gouthamr has quit IRC | 07:59 | |
| *** rm_work has quit IRC | 07:59 | |
| *** ildikov has quit IRC | 07:59 | |
| *** akahat has quit IRC | 07:59 | |
| *** d34dh0r53 has quit IRC | 07:59 | |
| *** amorin has quit IRC | 07:59 | |
| *** ralonsoh has quit IRC | 07:59 | |
| *** lbragstad has quit IRC | 07:59 | |
| *** dmellado has quit IRC | 07:59 | |
| *** dpawlik has quit IRC | 07:59 | |
| *** elod has quit IRC | 07:59 | |
| *** Dmitrii-Sh has quit IRC | 07:59 | |
| *** smcginnis has quit IRC | 07:59 | |
| *** hrw has quit IRC | 07:59 | |
| *** bhagyashris has quit IRC | 07:59 | |
| *** jonher has quit IRC | 07:59 | |
| *** chrome0 has quit IRC | 07:59 | |
| *** tkajinam has joined #opendev | 08:04 | |
| *** owalsh has joined #opendev | 08:04 | |
| *** ianw has joined #opendev | 08:04 | |
| *** Jeffrey4l has joined #opendev | 08:04 | |
| *** sboyron has joined #opendev | 08:04 | |
| *** fressi has joined #opendev | 08:04 | |
| *** ChanServ has joined #opendev | 08:04 | |
| *** verne.freenode.net sets mode: +o ChanServ | 08:04 | |
| *** iurygregory has joined #opendev | 08:04 | |
| *** mugsie has joined #opendev | 08:04 | |
| *** cloudnull has joined #opendev | 08:04 | |
| *** jonher has joined #opendev | 08:04 | |
| *** chrome0 has joined #opendev | 08:04 | |
| *** tristanC has joined #opendev | 08:05 | |
| *** mkowalski has joined #opendev | 08:05 | |
| *** parallax has joined #opendev | 08:05 | |
| *** akahat has joined #opendev | 08:05 | |
| *** d34dh0r53 has joined #opendev | 08:05 | |
| *** amorin has joined #opendev | 08:05 | |
| *** ralonsoh has joined #opendev | 08:05 | |
| *** lbragstad has joined #opendev | 08:05 | |
| *** dmellado has joined #opendev | 08:05 | |
| *** dpawlik has joined #opendev | 08:05 | |
| *** hrw has joined #opendev | 08:05 | |
| *** elod has joined #opendev | 08:05 | |
| *** Dmitrii-Sh has joined #opendev | 08:05 | |
| *** smcginnis has joined #opendev | 08:05 | |
| *** bhagyashris has joined #opendev | 08:05 | |
| *** otherwiseguy has joined #opendev | 08:05 | |
| *** arxcruz has joined #opendev | 08:05 | |
| *** chandankumar has joined #opendev | 08:05 | |
| *** jhesketh has joined #opendev | 08:05 | |
| *** corvus has joined #opendev | 08:05 | |
| *** andrewbonney has joined #opendev | 08:05 | |
| *** ykarel has joined #opendev | 08:05 | |
| *** paladox has joined #opendev | 08:05 | |
| *** lourot has joined #opendev | 08:05 | |
| *** JayF has joined #opendev | 08:05 | |
| *** tinwood has joined #opendev | 08:05 | |
| *** wendar has joined #opendev | 08:05 | |
| *** diablo_rojo_phon has joined #opendev | 08:05 | |
| *** ttx has joined #opendev | 08:05 | |
| *** ildikov has joined #opendev | 08:05 | |
| *** gouthamr has joined #opendev | 08:05 | |
| *** swest has joined #opendev | 08:05 | |
| *** rm_work has joined #opendev | 08:05 | |
| *** fdegir has joined #opendev | 08:06 | |
| *** persia has joined #opendev | 08:06 | |
| *** erbarr has joined #opendev | 08:06 | |
| *** mwhahaha has joined #opendev | 08:06 | |
| *** jroll has joined #opendev | 08:06 | |
| *** jrosser has joined #opendev | 08:06 | |
| *** noonedeadpunk has joined #opendev | 08:06 | |
| *** jaicaa has joined #opendev | 08:06 | |
| *** CeeMac has joined #opendev | 08:06 | |
| *** osmanlicilegi has joined #opendev | 08:06 | |
| *** donnyd has joined #opendev | 08:06 | |
| *** gmann has joined #opendev | 08:06 | |
| *** auristor has joined #opendev | 08:06 | |
| *** clayg has joined #opendev | 08:06 | |
| *** knikolla has joined #opendev | 08:06 | |
| *** seongsoocho has joined #opendev | 08:06 | |
| *** bbezak has joined #opendev | 08:06 | |
| *** jmorgan has joined #opendev | 08:06 | |
| *** tobiash has joined #opendev | 08:06 | |
| *** guillaumec has joined #opendev | 08:06 | |
| *** amotoki has joined #opendev | 08:06 | |
| *** mtreinish has joined #opendev | 08:06 | |
| *** gibi has joined #opendev | 08:06 | |
| *** ozzzo has joined #opendev | 08:06 | |
| *** stephenfin has joined #opendev | 08:06 | |
| *** mgoddard has joined #opendev | 08:06 | |
| *** amoralej has joined #opendev | 08:06 | |
| *** frickler has joined #opendev | 08:06 | |
| *** icey has joined #opendev | 08:06 | |
| *** brinzhang_ has joined #opendev | 08:06 | |
| *** avass has joined #opendev | 08:06 | |
| *** logan- has joined #opendev | 08:06 | |
| *** calcmandan has joined #opendev | 08:06 | |
| *** dhellmann has joined #opendev | 08:06 | |
| *** redrobot has joined #opendev | 08:06 | |
| *** zbr has joined #opendev | 08:06 | |
| *** fbo has joined #opendev | 08:06 | |
| *** kevinz has joined #opendev | 08:06 | |
| *** dardelean has joined #opendev | 08:06 | |
| *** zigo has joined #opendev | 08:06 | |
| *** cgoncalves has joined #opendev | 08:06 | |
| *** Tengu has joined #opendev | 08:06 | |
| *** jpena has joined #opendev | 08:06 | |
| *** mgagne has joined #opendev | 08:06 | |
| *** rpittau has joined #opendev | 08:06 | |
| *** tbarron has joined #opendev | 08:06 | |
| *** TheJulia has joined #opendev | 08:06 | |
| *** mnasiadka has joined #opendev | 08:06 | |
| *** clarkb has joined #opendev | 08:06 | |
| *** guilhermesp has joined #opendev | 08:06 | |
| *** felixedel has joined #opendev | 08:06 | |
| *** fungi has joined #opendev | 08:06 | |
| *** mattmceuen has joined #opendev | 08:06 | |
| *** walshh_ has joined #opendev | 08:06 | |
| *** johnsom has joined #opendev | 08:06 | |
| *** mnaser has joined #opendev | 08:06 | |
| *** SotK has joined #opendev | 08:06 | |
| *** melwitt has joined #opendev | 08:06 | |
| *** Alex_Gaynor has joined #opendev | 08:06 | |
| *** openstackgerrit has joined #opendev | 08:06 | |
| *** dirk has joined #opendev | 08:06 | |
| *** ysandeep has joined #opendev | 08:06 | |
| *** hberaud has joined #opendev | 08:06 | |
| *** priteau has joined #opendev | 08:06 | |
| *** _dvd has joined #opendev | 08:06 | |
| *** odyssey4me has joined #opendev | 08:06 | |
| *** gnuoy has joined #opendev | 08:06 | |
| *** dtantsur|afk has joined #opendev | 08:06 | |
| *** jrosser has quit IRC | 08:07 | |
| *** mwhahaha has quit IRC | 08:07 | |
| *** parallax has quit IRC | 08:08 | |
| *** gouthamr has quit IRC | 08:08 | |
| *** mnasiadka has quit IRC | 08:08 | |
| *** mnasiadka has joined #opendev | 08:09 | |
| *** mwhahaha has joined #opendev | 08:09 | |
| *** gouthamr has joined #opendev | 08:09 | |
| *** tosky has joined #opendev | 08:09 | |
| *** jrosser has joined #opendev | 08:10 | |
| *** lpetrut has joined #opendev | 08:11 | |
| *** parallax has joined #opendev | 08:13 | |
| *** mnasiadka has quit IRC | 08:20 | |
| *** mnasiadka has joined #opendev | 08:20 | |
| *** gouthamr has quit IRC | 08:20 | |
| *** gouthamr has joined #opendev | 08:20 | |
| *** mwhahaha has quit IRC | 08:20 | |
| *** mwhahaha has joined #opendev | 08:20 | |
| *** jrosser has quit IRC | 08:20 | |
| *** jrosser has joined #opendev | 08:20 | |
| *** parallax has quit IRC | 08:20 | |
| *** parallax has joined #opendev | 08:20 | |
| *** Eighth_Doctor has joined #opendev | 08:38 | |
| *** eolivare has joined #opendev | 08:49 | |
| *** artom has joined #opendev | 08:49 | |
| *** stevebaker has joined #opendev | 08:49 | |
| *** bodgix has joined #opendev | 08:49 | |
| *** prometheanfire has joined #opendev | 08:49 | |
| *** toomer has joined #opendev | 08:49 | |
| *** yoctozepto has joined #opendev | 08:49 | |
| *** mrunge has joined #opendev | 08:49 | |
| *** yoctozepto has quit IRC | 08:50 | |
| *** yoctozepto has joined #opendev | 08:50 | |
| *** yoctozepto has quit IRC | 08:53 | |
| *** mordred has joined #opendev | 08:53 | |
| *** yoctozepto has joined #opendev | 08:53 | |
| *** dtantsur|afk is now known as dtantsur | 09:05 | |
| *** dtantsur is now known as dtantsur|afk | 09:29 | |
| *** ysandeep is now known as ysandeep|afk | 09:40 | |
| *** ykarel has quit IRC | 10:44 | |
| *** ykarel has joined #opendev | 10:45 | |
| *** ykarel has quit IRC | 10:47 | |
| *** ykarel has joined #opendev | 10:48 | |
| *** brinzhang_ has quit IRC | 10:51 | |
| *** ykarel_ has joined #opendev | 10:56 | |
| *** ykarel has quit IRC | 10:59 | |
| *** ykarel_ is now known as ykarel | 11:17 | |
| *** fressi has left #opendev | 11:30 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** fressi has joined #opendev | 11:44 | |
| *** ysandeep|afk is now known as ysandeep | 11:47 | |
| *** fressi has quit IRC | 12:17 | |
| *** whoami-rajat_ has joined #opendev | 12:17 | |
| *** fressi has joined #opendev | 12:18 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** whoami-rajat_ is now known as whoami-rajat | 12:39 | |
| *** lpetrut has quit IRC | 12:53 | |
| *** amoralej is now known as amoralej|lunch | 12:58 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels https://review.opendev.org/c/openstack/project-config/+/783613 | 13:04 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs https://review.opendev.org/c/openstack/project-config/+/783633 | 13:04 |
| *** ysandeep is now known as ysandeep|away | 13:16 | |
| *** rpittau is now known as rpittau|bbl | 13:29 | |
| *** roman_g has joined #opendev | 13:32 | |
| *** ysandeep|away is now known as ysandeep | 13:35 | |
| *** amoralej|lunch is now known as amoralej | 13:42 | |
| *** mlavalle has joined #opendev | 13:46 | |
| *** fressi has left #opendev | 14:15 | |
| *** ysandeep is now known as ysandeep|away | 14:27 | |
| *** gothicserpent has joined #opendev | 14:46 | |
| *** d34dh0r53 has quit IRC | 15:04 | |
| *** ykarel is now known as ykarel|away | 15:07 | |
| fungi | #status log Restarted the nodepool-launcer container on nl03.opendev.org in order to free some indefinitely locked node requests | 15:10 |
| openstackstatus | fungi: finished logging | 15:10 |
| *** d34dh0r53 has joined #opendev | 15:12 | |
| *** ykarel|away has quit IRC | 15:13 | |
| corvus | i'm restarting zuul now | 15:15 |
| corvus | #status log restarted zuul at commit 9c3fce2820fb46aa39dbf89984386420fd7a7f70 | 15:16 |
| openstackstatus | corvus: finished logging | 15:16 |
| fungi | thanks! | 15:17 |
| corvus | re-enqueueing | 15:23 |
| corvus | 2021-04-09 15:21:42,216 ERROR zuul.zk.SemaphoreHandler: Releasing leaked semaphore /zuul/semaphores/openstack/translations held by 39aedef4181a4c749936d98bd02b1a3b-upstream-translation-update | 15:24 |
| corvus | i believe that is expected and good! | 15:24 |
| fungi | excellent | 15:24 |
| clarkb | nice | 15:25 |
| clarkb | I'm getting settled in to start the account external id cleanups | 15:25 |
| fungi | #status log Deleted server instance "test" (created 2020-10-23) from nodepool tenant in linaro-us | 15:25 |
| openstackstatus | fungi: finished logging | 15:25 |
| corvus | there are two leaked semaphore releases, and they match the last two acquired semaphores in the log (which were not released before i killed it) | 15:25 |
| fungi | i also removed the lingering xxxlarge ready node in linaro-us a few minutes ago | 15:25 |
| corvus | re-enqueue complete | 15:32 |
| clarkb | I'm double checking my input list to external id cleanups against the list of retirements I did last week. Then will likely do these in batches just to keep things to a reasonable size | 15:32 |
| *** amoralej is now known as amoralej|off | 15:33 | |
| fungi | i'm watching the stream console stream from a centos-8-arm64 job and it's doing jobbity-jobstuffs | 15:39 |
| *** ysandeep|away is now known as ysandeep | 15:50 | |
| clarkb | there were two inactive accounds that showed up in the audit that weren't in the retirement I did. Looking at them they are both retireable and able to be claened up. I have goine ahead and retired them (they were already inactive so this just removes preferred email to prevent it from missing an external id later) | 15:53 |
| clarkb | I think I'm ready to do my first batch based on what I retired last week. Any objections? | 15:58 |
| fungi | no objection from me | 15:59 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Drop Debian PPA from openafs-client role https://review.opendev.org/c/opendev/system-config/+/785675 | 16:07 |
| clarkb | one day I'll remember to tell python to do unbuffered io the first time I run this script after a while | 16:10 |
| fungi | i think you can set that in the script? | 16:10 |
| fungi | as opposed to having to use the cli option | 16:10 |
| clarkb | oh good idea /me looks | 16:10 |
| fungi | but i don't recall how, so maybe i dreamt it | 16:10 |
| clarkb | the internet suggests replacing sys.stdount or flushing every call | 16:11 |
| fungi | ahh | 16:13 |
| clarkb | anyway I've decided to do 4 batches | 16:13 |
| clarkb | the first one is running now with unbuffered io and had me confused initially :) | 16:13 |
| fungi | yeah, so not as trivial as just setting some value in the script anyway | 16:14 |
| clarkb | ah cool it just finished | 16:14 |
| clarkb | the output looks the way I expect it so I will do the next batch. When all 4 are done I'll copy the logs up | 16:14 |
| clarkb | (that was actually quicker than I expected too, fridays ftw!) | 16:15 |
| clarkb | I'll do a consistency check when done and ensure we didn't create preferred email address lack external id issues (shouldn't as I checked we had already retired all the accounts first) and see what our new count of external id conflicts is | 16:19 |
| clarkb | logs are up now. | 16:29 |
| clarkb | I'm working on the consistency check next | 16:29 |
| fungi | i'm about to get started on adding the debian bullseye wheel volumes | 16:29 |
| clarkb | when does bullseye release? | 16:30 |
| clarkb | I was looking last night (briefly and not very in depth) to find out and couldn't see anything | 16:31 |
| fungi | "when it's ready" | 16:31 |
| fungi | debian doesn't really do hard timing on release schedules | 16:32 |
| fungi | there are metrics like number of release critical bugs open and such | 16:32 |
| fungi | though it's getting close to time for a target date to release, i think | 16:32 |
| fungi | explained at https://www.debian.org/releases/bullseye/ | 16:34 |
| *** marios is now known as marios|out | 16:35 | |
| clarkb | and then when that happens you have ~1 year to upgrade buster to bullseye? | 16:36 |
| clarkb | (that was the rough info I was looking for) | 16:36 |
| fungi | yeah, basically, except there's also lts after that these days | 16:37 |
| fungi | oldstable gets normal support for a year, then lts kicks in | 16:37 |
| fungi | https://wiki.debian.org/LTS | 16:37 |
| fungi | the lts team is predicting to take over buster maintenance in july of next year | 16:38 |
| clarkb | now down to 334 external id conflicts from 545 at last state (and original was 600 something) | 16:38 |
| fungi | progress! | 16:39 |
| clarkb | the json for that is in the usual location too | 16:39 |
| clarkb | there were no other issues | 16:40 |
| clarkb | (its good to keep double checking that as removing external ids could orphan a preferred email address if we don't do the retirement first or get something related wrong) | 16:40 |
| clarkb | I'll rerun the audit script in a bit too | 16:42 |
| clarkb | 643 was the original count of conflicts | 16:42 |
| clarkb | progress indeed | 16:43 |
| *** jpena is now known as jpena|off | 16:58 | |
| clarkb | fungi: are we ready to tell Alex_Gaynor that things are happy again? | 17:08 |
| clarkb | Alex_Gaynor: I believe that centos 8 on arm64 is happy again, but fungi can confirm | 17:09 |
| hrw | bullseye is in hard freeze which for me means 'ready to use for most situations' as there will be small amount of changes only | 17:12 |
| clarkb | hrw: ya I'm not in a huge rush to use it, more wondering what the timeline on my buster installs looks like | 17:13 |
| *** marios|out has quit IRC | 17:13 | |
| hrw | clarkb: I moved some of my desktop systems, server will wait for May/June probably and some day of time to make sure all works | 17:14 |
| hrw | but that's system was ubuntu 13.04 - 13.10 - 14.04 - 16.04 - Debian 10 so things were complicated | 17:14 |
| fungi | clarkb: yeah, i was going to wait until i at least saw some centos builds succeed, but things do seem to be running again. we're *very* backed up though since all the changes waiting on centos nodes previously got completely reenqueued during the zuul restart. we'll likely need the weekend to clear that backlog | 17:15 |
| hrw | fungi: if that help: I am fine with all kolla jobs queued >12h being killed | 17:15 |
| fungi | hrw: it might, but at the moment i'm waiting to see if some of the centos-based ones succeed at least | 17:16 |
| Alex_Gaynor | clarkb: thanks! https://github.com/pyca/cryptography/pull/5961 let's see how it goes | 17:16 |
| hrw | as some of patches got merged in meantime | 17:16 |
| fungi | hrw: well, those check builds got their branch bases recreated during the reenqueue at ~15:30 utc, so are running merged with fairly current states | 17:17 |
| hrw | ok | 17:18 |
| JayF | Alex_Gaynor: o/ didn't know you were still openstacking, hope you are well (we worked together in rax sf) | 17:19 |
| *** ysandeep is now known as ysandeep|away | 17:19 | |
| clarkb | JayF: we're helping the pyca group do aarch64 builds of some libs not really openstack, but helps openstack as cryptography and friends underpin much of python including openstack | 17:19 |
| hrw | https://zuul.openstack.org/builds?job_name=kolla-build-centos8s-source-aarch64 lists set of centos-stream-arm64 jobs from today. marked as failure but all worked fine and gave proper output | 17:20 |
| JayF | Well cryptography is everywhere now :D I'm glad we're helpin' em out | 17:20 |
| Alex_Gaynor | JayF: 👋 I'm not actually doing OpenStack stuff :-) I still do python crypto stuff and the opendev folks are very generously helping us out with arm64 CI (I assume because openstack cares about arm64 :D) | 17:20 |
| hrw | ah. cryptography and that rust use... | 17:20 |
| clarkb | hrw: we're trying to help make that rust use less painful for everyone running aarch64 by providing systems to build wheels that go on pypi :) | 17:22 |
| fungi | yeah, having official aarch64 wheels of cryptography helps arm-based tests for openstack and our other projects go much faster since they don't need to recompile from sdist every time | 17:22 |
| clarkb | but ya more generally its a time savings when installing python (including openstack) on aarch64 since cryptography builds from source can sink a bit of time | 17:23 |
| hrw | yep | 17:23 |
| fungi | and these days everything depends (at least indirectly) on cryptography because Alex_Gaynor and other contributors are doing such a great job with it | 17:23 |
| hrw | yep | 17:25 |
| hrw | Alex_Gaynor: would you consider building and uploading also versions used by previous openstack releases? | 17:26 |
| Alex_Gaynor | We build and release our wheels as a part of our release process, because we only do wheels for platforms we're actively testing, we don't generally upload wheels for past releases. | 17:28 |
| hrw | ok, always worth asking ;D | 17:28 |
| fungi | we're still able to build and cache them in our wheel cache in afs anyway | 17:30 |
| fungi | they just won't be available directly from pypi | 17:31 |
| hrw | yep | 17:31 |
| hrw | and linaro CI has own cache too | 17:31 |
| Alex_Gaynor | clarkb: good news, centos8 works. bad news our unrelated windows CI got itself broken | 17:32 |
| fungi | ouch | 17:32 |
| fungi | not good when your jobs bsod ;) | 17:33 |
| *** andrewbonney has quit IRC | 17:34 | |
| clarkb | Alex_Gaynor: great! at least on the half we can help with :) | 17:36 |
| * hrw off | 17:37 | |
| fungi | argh, mirror.wheel.bullseyex64 is >22 bytes long | 17:39 |
| fungi | we need to shave off two bytes... how about mirror.wheel.bullsex64 and mirror.wheel.bullsea64? | 17:40 |
| hrw | bullsa64 bullsx64 to not have 'sex' in url | 17:40 |
| fungi | or we could switch to something like mirror.wheel.deb11x64 | 17:40 |
| hrw | even better | 17:40 |
| clarkb | fungi: using the release number makes sense since those will stay short until they hit 100 :) | 17:40 |
| clarkb | unless debian pulls a suse | 17:41 |
| clarkb | though suse went from 42 to 15 so even then we'll probably be ok :) | 17:41 |
| fungi | or we could switch to something like actually we can fit mirror.wheel.deb011x64 it we're really worried about supporting the next few hundred years of debian | 17:41 |
| fungi | but yeah, i'll go with deb11x64 and deb11a64 | 17:42 |
| * TheJulia looks at the history and blinks a few times | 17:42 | |
| fungi | TheJulia: yep, sorry, i should have read that more closely before suggesting it :/ | 17:43 |
| fungi | i really just trimmed the minimum number of bytes off "bullseye" but... | 17:43 |
| JayF | look, don't have a cow, it was a mistake | 17:49 |
| JayF | :-D | 17:50 |
| TheJulia | oh yeah, totally get that :) | 17:54 |
| TheJulia | just... "surprising" to see ;) | 17:55 |
| *** roman_g has quit IRC | 18:17 | |
| *** roman_g has joined #opendev | 18:17 | |
| *** eolivare has quit IRC | 18:19 | |
| *** sboyron has quit IRC | 18:22 | |
| fungi | i'll admit, i was also surprised to have typed it | 18:23 |
| *** stewie925 has joined #opendev | 18:39 | |
| *** spotz has joined #opendev | 18:41 | |
| fungi | #status log Mounted new AFS volumes mirror.wheel.deb11a64 at mirror/wheel/debian-11-aarch64 and mirror.wheel.deb11x64 at mirror/wheel/debian-11-x86_64 with our standard ACLs and base quotas | 18:44 |
| openstackstatus | fungi: finished logging | 18:44 |
| clarkb | I've got an audit of the remaining conflicts running now | 18:50 |
| fungi | good news, i see a kolla-build-centos8-source-aarch64 which succeeded | 18:55 |
| fungi | kevinz: when you have time (sorry i know it's probably your weekend now, not urgent at all) we have 8 server instances in linaro-us which are stuck in either BUILD or ERROR state for several days now and would appreciate if you could clean them up so we can use the rest of that quota: a25e6f3b-5695-4729-b9c7-bb6160d7dd55 3ec7f816-52e5-480d-828a-767ebae7cf84 6accab6d-e1bd-4998-85eb-d7764d1ee0f5 | 18:58 |
| fungi | a82e3b41-b0f7-4310-a89f-2a84eef8f8b0 15c38603-41c2-484d-8637-62941b871cbe 449f31dc-8207-420a-a182-da7c58d962bd 402bcb9d-f40f-4b71-81e9-3aba5b4b432c 74485f01-22e3-4df5-bc23-d408af99cd6f | 18:58 |
| *** slaweq has quit IRC | 19:26 | |
| clarkb | alright the audit completed and I've stashed it in the usual location. Includes the raw yaml which can be queried as well as some automated categorization in a separate file | 19:36 |
| clarkb | I'm going to stop doing user account stuff now. /me finds something else | 19:36 |
| *** ralonsoh has quit IRC | 19:39 | |
| clarkb | https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo that sounds familiar | 19:41 |
| clarkb | tuning back ssh threads is something we can consider | 19:41 |
| clarkb | if we want to do that, the 3.2.8 upgrade might be a good time for it | 19:42 |
| fungi | huh, yeah, i wonder what the effect is though if you have an insufficient number of ssh threads for the requested operations | 19:49 |
| clarkb | I think they queue up then as timeouts happen clients will eb told to go away | 19:50 |
| clarkb | also you can set threads for bot groups separate from interactive users | 19:50 |
| clarkb | so we could prevent zuul from being starved (though it uses a bunch of http now) | 19:50 |
| fungi | yeah, maybe it'll also just encourage users to switch to https instead of ssh for typical client interactions too | 19:51 |
| *** roman_g has joined #opendev | 19:56 | |
| *** stewie925 has quit IRC | 20:00 | |
| *** whoami-rajat has quit IRC | 20:00 | |
| openstackgerrit | Clark Boylan proposed opendev/git-review master: Add option for disabling thin pushes https://review.opendev.org/c/opendev/git-review/+/785723 | 20:19 |
| clarkb | fungi: ^ something like that to make pushes when the pack issue arises might be good | 20:19 |
| clarkb | for some reason I think I recall that the dest value can't have an underscore in it? but I may be wrong about that | 20:20 |
| fungi | i think it's that gerrit maps underscores to spaces? | 20:33 |
| fungi | at least i recall going through a bunch of different options for getting spaces in topics working at push, and underscores were involved somehow, but ultimately nothing completely worked | 20:34 |
| clarkb | I read the argparse docs and I think it is that by default if I didn't set dest it would be no_thin bceause the -- prefix is removed and the - in the middle is replaced with a _ | 20:34 |
| clarkb | it should work as is and while redundant is consistent with some of the other options there | 20:34 |
| clarkb | I also haven't tested this change yet. | 20:35 |
| fungi | shouldn't be hard to add a functional test for it | 20:35 |
| clarkb | ya I expect just copy a push test and add the no thin? | 20:36 |
| clarkb | it is hard to very that git is doing what we want but we can at least ensure it doesn't flat out break | 20:36 |
| clarkb | I'll start looking into that next | 20:36 |
| *** rpittau|bbl is now known as rpittau | 20:39 | |
| openstackgerrit | Clark Boylan proposed opendev/git-review master: Add option for disabling thin pushes https://review.opendev.org/c/opendev/git-review/+/785723 | 20:50 |
| fungi | right, i don't think we need to test that git does what it says on the tin, that's for git's maintainers to worry about | 20:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels https://review.opendev.org/c/openstack/project-config/+/783613 | 21:25 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs https://review.opendev.org/c/openstack/project-config/+/783633 | 21:27 |
| *** rpittau is now known as rpittau|afk | 22:04 | |
| *** avass has quit IRC | 22:13 | |
| *** roman_g has quit IRC | 22:22 | |
| clarkb | fungi: https://review.opendev.org/c/opendev/git-review/+/785723 that did pass the test I added so I guess its probably good? I will probably try using it a bit next week too | 22:56 |
| fungi | yeah, seems fine to me | 22:57 |
| *** tosky has quit IRC | 23:29 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!