| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: containerfile: automatically search for distro docker files https://review.opendev.org/c/openstack/diskimage-builder/+/790362 | 00:41 |
|---|---|---|
| *** pongboom2 has quit IRC | 00:57 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Install epel-release from URL https://review.opendev.org/c/openstack/diskimage-builder/+/772652 | 01:28 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/700083 | 01:50 |
| *** ysandeep|away is now known as ysandeep|SL | 02:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 02:16 |
| *** hemanth_n has joined #opendev | 02:49 | |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Remove Fedora 32 job https://review.opendev.org/c/opendev/glean/+/790368 | 03:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 03:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 03:38 |
| *** ykarel has joined #opendev | 04:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: containerfile: automatically search for distro docker files https://review.opendev.org/c/openstack/diskimage-builder/+/790362 | 04:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 04:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: dib-lint: match text/x-script.python https://review.opendev.org/c/openstack/diskimage-builder/+/790369 | 04:34 |
| *** ralonsoh has joined #opendev | 05:03 | |
| *** swest has joined #opendev | 05:09 | |
| *** marios has joined #opendev | 05:12 | |
| *** stevebaker has quit IRC | 05:15 | |
| *** stevebaker has joined #opendev | 05:22 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: containerfile: automatically search for distro docker files https://review.opendev.org/c/openstack/diskimage-builder/+/790362 | 05:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 05:36 |
| *** slaweq has joined #opendev | 06:23 | |
| *** sboyron has joined #opendev | 06:46 | |
| *** amoralej|off is now known as amoralej | 06:53 | |
| *** avass has joined #opendev | 07:17 | |
| *** andrewbonney has joined #opendev | 07:19 | |
| *** slaweq has quit IRC | 07:21 | |
| *** slaweq has joined #opendev | 07:23 | |
| *** hashar has joined #opendev | 07:28 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 07:29 |
| *** tosky has joined #opendev | 07:36 | |
| *** felixedel has joined #opendev | 07:43 | |
| *** rpittau|afk is now known as rpittau | 07:48 | |
| *** mgoddard has joined #opendev | 07:52 | |
| *** jpena|off is now known as jpena | 07:55 | |
| *** ykarel is now known as ykarel|lunch | 07:58 | |
| *** kopecmartin has quit IRC | 08:10 | |
| *** fressi has joined #opendev | 08:14 | |
| *** fressi has quit IRC | 08:15 | |
| *** fressi has joined #opendev | 08:20 | |
| *** dpawlik has quit IRC | 08:21 | |
| *** fressi has quit IRC | 08:21 | |
| *** fressi has joined #opendev | 08:22 | |
| *** kopecmartin has joined #opendev | 08:25 | |
| *** fressi has quit IRC | 08:27 | |
| *** dpawlik1 has joined #opendev | 08:28 | |
| *** dtantsur|afk is now known as dtantsur | 08:42 | |
| *** ykarel|lunch has quit IRC | 08:42 | |
| *** ykarel_ has joined #opendev | 08:42 | |
| *** ykarel_ has quit IRC | 08:43 | |
| *** ykarel_ has joined #opendev | 08:43 | |
| *** whoami-rajat has joined #opendev | 08:46 | |
| *** ykarel_ is now known as ykarel | 08:46 | |
| *** hemanth_n has quit IRC | 08:46 | |
| *** hemanth_n has joined #opendev | 08:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:08 | |
| openstackgerrit | Carlos Gonçalves proposed openstack/diskimage-builder master: Auto find greatest Fedora cloud image sub-release https://review.opendev.org/c/openstack/diskimage-builder/+/755992 | 09:14 |
| *** lpetrut has joined #opendev | 09:27 | |
| *** pongboom has joined #opendev | 09:28 | |
| zigo | fungi: ianw: clarkb: Looks like Bullseye is working now, thanks a lot for your help and patience. | 10:21 |
| *** hemanth_n has quit IRC | 10:57 | |
| *** hemanth_n has joined #opendev | 10:58 | |
| *** kopecmartin has quit IRC | 11:01 | |
| *** dpawlik1 has quit IRC | 11:03 | |
| *** kopecmartin has joined #opendev | 11:08 | |
| *** dpawlik0 has joined #opendev | 11:10 | |
| *** jpena is now known as jpena|lunch | 11:26 | |
| *** dpawlik0 is now known as dpawlik | 11:56 | |
| *** hemanth_n has quit IRC | 12:10 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| fungi | zigo: great! there are a lot of moving parts, and it's harder when the distro version isn't released yet | 12:30 |
| *** amoralej is now known as amoralej|lunch | 12:31 | |
| *** lourot has quit IRC | 13:06 | |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| *** lourot has joined #opendev | 13:25 | |
| *** tosky_ has joined #opendev | 13:27 | |
| *** tosky has quit IRC | 13:27 | |
| *** tosky_ is now known as tosky | 13:27 | |
| *** marios is now known as marios|call | 13:28 | |
| *** ykarel_ has joined #opendev | 13:31 | |
| *** ykarel has quit IRC | 13:31 | |
| *** ykarel_ is now known as ykarel | 13:32 | |
| *** vishalmanchanda has joined #opendev | 13:37 | |
| *** marios|call is now known as marios | 13:46 | |
| *** lourot has quit IRC | 13:47 | |
| *** lourot has joined #opendev | 13:49 | |
| *** artom has joined #opendev | 13:51 | |
| *** lpetrut has quit IRC | 14:22 | |
| *** mlavalle has joined #opendev | 15:03 | |
| clarkb | I've realized there is a fun chicken and egg with switching the zuul scheduler. And that is dns | 15:05 |
| clarkb | we need to update the zuul.o.o CNAME from zuul01.openstack.org to zuul02.opendev.org when we get there and that typically relies on zuul :) | 15:05 |
| clarkb | I'm going to start booting the new instance shortly, but thought I'd mention that in case anyone had good ideas for dealing with it | 15:05 |
| clarkb | One thought I had was testing the queue dump and restore (as much as possible) works using the actual host fqdn and not the cname. If that works then we probably don't need to worry about dns too much in the switch itself, then can ensure dns is updated afterwards | 15:06 |
| fungi | i'd say merging the dns change and waiting for it to deploy would be the first step in the upgrade maintenance | 15:06 |
| clarkb | ya that is another option | 15:06 |
| clarkb | when I launch the new zuul I'll use a modified make_swap.sh that applies the ephemeral disk to the zuul log dir | 15:07 |
| fungi | also the queue state backups in /var/lib/zuul/backup/ seem to only cover the openstack tenant | 15:09 |
| clarkb | ok I usually run it manually anyway. Also I suspect I may need to sed the resulting file to change hostnames since dns will be weird :) | 15:19 |
| clarkb | the instance is booting now. I'll double check /var/log/zuul looks good after it boots before I do anything else. Then if that is happy I'll push up a bunch of changes that add it to inventory, update dns, etc etc | 15:20 |
| clarkb | we should be able to safely update dns to add zuul02.opendev.org and add the server to inventory so that it gets configured as a non running zuul scheduler. Then we can work through the steps for the maintenance | 15:21 |
| clarkb | I should probably also ensure that the zuul.o.o (for opendev and openstack) CNAMEs have ttls of 300 | 15:21 |
| *** amoralej is now known as amoralej|off | 15:46 | |
| *** hashar is now known as hasharDinner | 15:47 | |
| clarkb | anyone know if the key under letsencrypt-certs needs to be unique per host | 15:47 |
| clarkb | for example can I have a zuul-opendev-main under zuul01.openstack.org and zuul02.opendev.org? | 15:48 |
| clarkb | I'm beginning to suspect that I can't since that is used to track data onto other hosts | 15:49 |
| clarkb | and it won't know whats what if it is done that wy? | 15:49 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add zuul02 https://review.opendev.org/c/opendev/zone-opendev.org/+/790480 | 15:52 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add zuul02 to inventory https://review.opendev.org/c/opendev/system-config/+/790481 | 15:54 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Swap zuul.opendev.org CNAME to zuul02.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/790482 | 15:57 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Reset zuul.o.o CNAME TTL to default https://review.opendev.org/c/opendev/zone-opendev.org/+/790483 | 15:57 |
| *** ykarel has quit IRC | 15:57 | |
| *** marios is now known as marios|out | 16:00 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Clean up zuul01 from inventory https://review.opendev.org/c/opendev/system-config/+/790484 | 16:01 |
| *** rpittau is now known as rpittau|afk | 16:03 | |
| clarkb | I think the rough plan is something like land https://review.opendev.org/c/opendev/zone-opendev.org/+/790480 and https://review.opendev.org/c/opendev/system-config/+/790481. Then check zuul02 looks the way we expect it to. When we are happy with its state we land https://review.opendev.org/c/opendev/zone-opendev.org/+/790482 then dump queues on zuul01, stop zuul on all servers, manually run name | 16:04 |
| clarkb | servers playbook to update dns, manually run playbook to update gearman server value for all executors and mergers (similar to how I updated zk servers), start zuul again, restore queues | 16:04 |
| clarkb | infra-root if you can look that over that would be great | 16:04 |
| *** marios|out has quit IRC | 16:04 | |
| clarkb | one thing we might consider doing is updating the gearman server value to zuul.opendev.org now then we don't need to change its value when things are off | 16:04 |
| clarkb | but we weren't doing that already so not sure if there is a reason not to | 16:05 |
| clarkb | I wonder, does zuul support connecting to multiple gearman servers too? that might be another option | 16:07 |
| clarkb | then we just have to ensure only one is ever available at a time | 16:07 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Small playbook to update zuul gearman server https://review.opendev.org/c/opendev/system-config/+/790487 | 16:14 |
| clarkb | that playbook should work for doing the config swap. Its a bit weird bceause server= matches a lot of lines but it seems server=zuul01.openstack.org is unique to the gearman ini section | 16:14 |
| clarkb | I can test that manual playbook too against a single executor and a single merger then revert them back again if we want to go that route | 16:17 |
| corvus | clarkb: i don't think zuul supports >1 geard | 16:18 |
| corvus | +connecting to+ | 16:18 |
| clarkb | corvus: ok, in that case https://review.opendev.org/c/opendev/system-config/+/790487 is probably sufficient and we can run that when things are off | 16:18 |
| corvus | ya | 16:19 |
| clarkb | I'm going to test dumping queues using zuul01.openstack.org instead of zuul.opendev.org as the hostname. Then if that works we should be able to update the result to talk to zuul02.opendev.org and not worry about dns | 16:21 |
| clarkb | and then maybe I should put all this in a doc to keep it straight | 16:21 |
| *** lourot has quit IRC | 16:22 | |
| clarkb | and that doesn't work because ssl. We don't put the host's name as an altname in there | 16:22 |
| clarkb | in that case I think I need to change the order of operations above slightly /me moves that into an etherpad | 16:23 |
| fungi | interesting, i thought for le we defaulted to including the inventory fqdn as an altname | 16:23 |
| fungi | oh, this is the zuul ca though, yeah? | 16:23 |
| clarkb | fungi: no this is talking to https not gearman/zk. its based on what we put in the inventory files | 16:24 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/790481 illustrates the LE config | 16:24 |
| clarkb | and we idnd't do the host as an altname previously so that chagne doesn't either | 16:24 |
| fungi | got it, i guess we just did that for other systems | 16:25 |
| clarkb | https://etherpad.opendev.org/p/opendev-zuul-server-swap | 16:25 |
| *** d34dh0r53 has quit IRC | 16:26 | |
| *** ralonsoh has quit IRC | 16:27 | |
| fungi | if ansible isn't putting a le cert on there yet, we won't be able to start apache | 16:28 |
| clarkb | fungi: landing https://review.opendev.org/c/opendev/system-config/+/790481 should configure everything on zuul02 except for starting zuul services. We can double check everything there at that point too | 16:29 |
| clarkb | (reviewers should check my assertion that zuul services won't be started too) | 16:29 |
| fungi | do you want to add a _acme-challenge.zuul02 in 790480 to ease future transitions? | 16:30 |
| *** d34dh0r53 has joined #opendev | 16:30 | |
| clarkb | fungi: I suspect we may have servername updates that need to happen in apache as well. I think we can probably do that as a followup if we want it? | 16:31 |
| fungi | similar to how etherpad is set up | 16:31 |
| fungi | yeah, just didn't know if you wanted to reduce the amount of dns churn for that | 16:31 |
| clarkb | for now i'm mostly just sticking to how the old server was set up | 16:32 |
| fungi | except on a different domain | 16:32 |
| fungi | makes sense | 16:32 |
| clarkb | right | 16:32 |
| fungi | i think the le setup was fiddly because we were still using the old domain for the inventory | 16:33 |
| fungi | now the only thing which will need to survive in the openstack.org domain for it is the whitelabel site cname and accompanying le cname | 16:33 |
| clarkb | yup | 16:34 |
| clarkb | ok https://etherpad.opendev.org/p/opendev-zuul-server-swap should be fairly complete now. Please point out missing steps if you can think of them | 16:43 |
| clarkb | fwiw I'm also not in a super rush to get all this in beacuse I'm out a good chunk of tomorrow. Then depending on how I'm feeling hopeflly can make progress wednesday. If not wednesday then thursday seems likely | 16:50 |
| *** fungi has quit IRC | 16:57 | |
| *** jpena is now known as jpena|off | 16:59 | |
| *** andrewbonney has quit IRC | 17:09 | |
| *** dtantsur is now known as dtantsur|afk | 17:19 | |
| *** fungi has joined #opendev | 17:34 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:17 | |
| *** fungi has quit IRC | 18:54 | |
| *** fungi has joined #opendev | 18:59 | |
| *** rchurch has joined #opendev | 19:03 | |
| *** fungi has quit IRC | 19:20 | |
| *** rchurch has quit IRC | 19:24 | |
| *** fungi has joined #opendev | 19:29 | |
| openstackgerrit | Ade Lee proposed zuul/zuul-jobs master: Add role to enable FIPS on a node https://review.opendev.org/c/zuul/zuul-jobs/+/788778 | 20:03 |
| *** lourot has joined #opendev | 20:15 | |
| *** lourot has quit IRC | 20:19 | |
| *** lourot has joined #opendev | 20:20 | |
| *** vishalmanchanda has quit IRC | 20:27 | |
| *** sboyron has quit IRC | 20:42 | |
| openstackgerrit | Merged opendev/system-config master: Retire Asterisk service https://review.opendev.org/c/opendev/system-config/+/790190 | 21:45 |
| fungi | ianw: ^ should be all clear to delete the server and dns records now | 21:56 |
| ianw | fungi: thanks! will do | 21:56 |
| *** hasharDinner has quit IRC | 22:02 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add infra-prod-service-lists job https://review.opendev.org/c/opendev/system-config/+/790524 | 22:05 |
| clarkb | That is split into a separate chagne so that we can land the parent, manually run playbooks until we are happy with them, then land that change | 22:06 |
| *** iurygregory has quit IRC | 22:07 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Clean up zuul01 from inventory https://review.opendev.org/c/opendev/system-config/+/790484 | 22:07 |
| clarkb | and that cleanup occured to me on my bike ride randomly | 22:08 |
| clarkb | its weird what comes to you when you can clear you head a bit | 22:08 |
| * fungi clears his head by turning upside down and shaking, like an etch-a-sketch | 22:09 | |
| *** iurygregory has joined #opendev | 22:11 | |
| *** hamalq has joined #opendev | 22:42 | |
| clarkb | ianw: if you get a chance can you review the mailman changes: https://review.opendev.org/c/opendev/system-config/+/789622 I'm happy to do approvals when I've got time and everyone is happy with things. Mostly just looking for review | 22:51 |
| clarkb | also the zuul scheduler upgrade plan is up at https://etherpad.opendev.org/p/opendev-zuul-server-swap with links to changes and happy for reviews on those (and I can do approvals there too) | 22:52 |
| ianw | yep, will do | 22:56 |
| ianw | did we decide no meeting tomorrow? | 22:56 |
| *** tosky has quit IRC | 22:57 | |
| clarkb | ya I sent out an email cancelling it already | 22:59 |
| clarkb | that said if people really want to meet I won't stop you :) | 23:00 |
| *** artom has quit IRC | 23:21 | |
| *** artom has joined #opendev | 23:22 | |
| *** artom has quit IRC | 23:24 | |
| *** artom has joined #opendev | 23:24 | |
| clarkb | ianw: I meant to ask have we seen anymore dib nodepool functional jobs have problems with devstack? Or did that reordering for concurrency seem to fix it? (I think we did fix it, but concurrency issues are always fun so don't want to assume so) | 23:28 |
| ianw | clarkb: i haven't seen any more errors | 23:29 |
| ianw | i'm getting close to getting the containerfile stuff in properly. i plan to start with Fedora to shake out issues, then I think we can consider switching other platforms if we like | 23:29 |
| clarkb | nice | 23:31 |
| *** artom has quit IRC | 23:40 | |
| fungi | that'll be awesome once it's working | 23:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-containerfile element https://review.opendev.org/c/openstack/diskimage-builder/+/790365 | 23:46 |
| *** artom has joined #opendev | 23:49 | |
| *** mlavalle has quit IRC | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!