| ianw | i'll think on it and see if i can come up with something; if it's not working or the change is looking ridiculous we can move to phase 2 | 00:01 |
|---|---|---|
| clarkb | phase 2 == meetpad? that works for me. I didn't want to forget about it then it be the weekend again if others wanted to dig into it sooner. I'm happy to think on it over the weekend too | 00:01 |
| ianw | yep or longer threads or whatever works. i feel like it might slot ontop of 807672 fairly logically. but i may be wrong | 00:02 |
| clarkb | ok sounds like a plan then. Will wait and see how it shakes out | 00:03 |
| ianw | also matches my philosophy of nothing getting people more interested than an obviously incorrect change :) | 00:03 |
| ianw | speaking of https://review.opendev.org/c/opendev/system-config/+/807672, i guess zuul has decided it has nothing to do there? | 00:10 |
| Clark[m] | Most of those jobs run post merge so ya maybe syntax checking is all we'll get? | 00:29 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: infra-prod: split out setup steps https://review.opendev.org/c/opendev/base-jobs/+/807807 | 01:37 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Refactor infra-prod jobs for parallel running https://review.opendev.org/c/opendev/system-config/+/807672 | 01:42 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod: clone source once https://review.opendev.org/c/opendev/system-config/+/807808 | 01:42 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: infra-prod: split out setup steps https://review.opendev.org/c/opendev/base-jobs/+/807807 | 01:43 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: infra-prod: split out setup steps https://review.opendev.org/c/opendev/base-jobs/+/807807 | 02:03 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: infra-prod: split out setup steps https://review.opendev.org/c/opendev/base-jobs/+/807807 | 02:37 |
| *** ysandeep|out is now known as ysandeep | 05:54 | |
| *** slaweq_ is now known as slaweq | 07:06 | |
| *** pjakuszew7 is now known as pjakuszew | 07:09 | |
| *** jpena|off is now known as jpena | 07:38 | |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: Make default tox run more strict about interpreter version https://review.opendev.org/c/zuul/zuul-jobs/+/807702 | 08:19 |
| *** elodilles_pto is now known as elodilles | 08:44 | |
| opendevreview | gnuoy proposed openstack/project-config master: Add openstack-loadbalancer charm and interfaces https://review.opendev.org/c/openstack/project-config/+/807838 | 08:48 |
| opendevreview | Merged openstack/project-config master: Update Neutron's Grafana dashboard https://review.opendev.org/c/openstack/project-config/+/805809 | 09:09 |
| opendevreview | Slawek Kaplonski proposed opendev/irc-meetings master: Remove Neutron L3 subteam meeting https://review.opendev.org/c/opendev/irc-meetings/+/807843 | 09:30 |
| opendevreview | gnuoy proposed openstack/project-config master: Add openstack-loadbalancer charm and interfaces https://review.opendev.org/c/openstack/project-config/+/807838 | 09:33 |
| opendevreview | Merged opendev/irc-meetings master: Move Keystone meeting to 1500 UTC https://review.opendev.org/c/opendev/irc-meetings/+/807729 | 09:51 |
| opendevreview | Merged opendev/irc-meetings master: Remove Neutron L3 subteam meeting https://review.opendev.org/c/opendev/irc-meetings/+/807843 | 09:54 |
| opendevreview | Merged opendev/irc-meetings master: Move openstack-helm meeting to #openstack-helm https://review.opendev.org/c/opendev/irc-meetings/+/805094 | 09:55 |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: ensure-tox: avoid using a reliable version of tox https://review.opendev.org/c/zuul/zuul-jobs/+/807867 | 12:52 |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Introduce a BackendInterface https://review.opendev.org/c/opendev/statusbot/+/807871 | 13:14 |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: ensure-tox: avoid using a reliable version of tox https://review.opendev.org/c/zuul/zuul-jobs/+/807867 | 13:25 |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: Fix historical ansible-lint errors https://review.opendev.org/c/zuul/zuul-jobs/+/807877 | 13:43 |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: [DNM] Test upload_logs_s3 role https://review.opendev.org/c/zuul/zuul-jobs/+/807132 | 13:54 |
| *** diablo_rojo is now known as Guest6695 | 14:25 | |
| *** Guest6695 is now known as diablo_rojo | 14:46 | |
| opendevreview | Michal Nasiadka proposed opendev/irc-meetings master: kolla: update chair https://review.opendev.org/c/opendev/irc-meetings/+/807892 | 14:57 |
| *** tosky is now known as Guest6702 | 15:22 | |
| *** tosky_ is now known as tosky | 15:22 | |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Fix upload_logs_s3 role https://review.opendev.org/c/zuul/zuul-jobs/+/807132 | 15:35 |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Fix zuul-jobs-test-upload-logs-s3 job https://review.opendev.org/c/zuul/zuul-jobs/+/807132 | 15:36 |
| clarkb | I forwarded the most recent certcheck warning about the linaro cert to kevinz | 15:44 |
| opendevreview | Merged opendev/irc-meetings master: kolla: update chair https://review.opendev.org/c/opendev/irc-meetings/+/807892 | 15:52 |
| *** marios is now known as marios|out | 15:58 | |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override"" https://review.opendev.org/c/zuul/zuul-jobs/+/776677 | 16:38 |
| opendevreview | Artem Goncharov proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override"" https://review.opendev.org/c/zuul/zuul-jobs/+/776677 | 16:38 |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Introduce a BackendInterface https://review.opendev.org/c/opendev/statusbot/+/807871 | 16:42 |
| *** jpena is now known as jpena|off | 17:09 | |
| *** ysandeep is now known as ysandeep|out | 19:24 | |
| clarkb | I've discovered that you can explicitly subscribe to github releases on repos so I have done that for etherpad and github | 19:26 |
| clarkb | er etherpad and gitea on github | 19:27 |
| fungi | oh neat | 19:29 |
| fungi | i didn't realize that was possible | 19:29 |
| clarkb | its an option in the watch drop down menu | 19:29 |
| clarkb | you select custom and then in custom can select releases | 19:29 |
| fungi | noted | 19:30 |
| clarkb | I'm hoping that helps us stay on top of those tools and get releases pushed out more frequently | 19:32 |
| corvus | the zuul bugfix change has merged; i'd like to restart all of zuul soon. did the gerrit stuff happen already? | 19:34 |
| fungi | the gerrit server migration has not happened yet | 19:34 |
| fungi | mnaser: ^ can we coordinate a gerrit server migration and zuul upgrade/restart soonish? | 19:35 |
| fungi | that way we minimize lost events from gerrit | 19:35 |
| fungi | i guess the order would be shut down gerrit and start server migration, then restart all of zuul onto new code, then start gerrit again? | 19:36 |
| clarkb | fungi: ++ I was just going to say similar | 19:36 |
| clarkb | I can help though lunch should be here shortly | 19:37 |
| fungi | now that config is persisted in zk, we don't have to worry about cat jobs being unable to clone from gerrit, right? | 19:37 |
| corvus | fungi: i think we still get the branch list from gerrit, so best to start zuul after gerrit is up | 19:37 |
| corvus | so: save zuul queue, down gerrit, down zuul, up gerrit, up zuul, restore queue? | 19:38 |
| clarkb | corvus: sounds good to me | 19:41 |
| clarkb | now we just need mnaser to do the migration after down gerrit and before up gerrit | 19:41 |
| corvus | i await a ping :) | 19:41 |
| clarkb | I do have a conflict for about 45 minutes at 22:20 UTC. Hopefully we can be done by then | 19:44 |
| fungi | yep, that sounds like a perfect plan | 19:44 |
| fungi | i'm basically around, just doing yardwork (with frequent breaks because it's hot, muggy and there's no breeze) | 19:44 |
| corvus | being next to the sea with no seabreeze seems like a weather malfunction | 19:45 |
| fungi | it only does this when i want to knock out the yardwork. nature is conspiring to return this to a natural habitat | 19:56 |
| fungi | (which i'm trying to encourage, within the limit allowed by the town's codes, at least until i find time to lobby to change them) | 19:57 |
| fungi | weather like this almost puts me in the mood for civil disobedience though | 19:58 |
| *** timburke__ is now known as timburke | 20:04 | |
| *** elodilles is now known as elodilles_pto | 20:12 | |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Introduce a BackendInterface https://review.opendev.org/c/opendev/statusbot/+/807871 | 20:38 |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Add Etherpad backend https://review.opendev.org/c/opendev/statusbot/+/807946 | 20:38 |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Add use_ssl option https://review.opendev.org/c/opendev/statusbot/+/807947 | 20:38 |
| opendevreview | Tristan Cacqueray proposed opendev/statusbot master: Handle exception for unprivileged commands https://review.opendev.org/c/opendev/statusbot/+/807948 | 20:38 |
| mnaser | clarkb, fungi, corvus: i am doing the 'initial' online migrate now, i will ping when im ready to 'punt' the system over | 20:43 |
| mnaser | from then it should be a brief reboot | 20:43 |
| clarkb | sounds good | 20:43 |
| clarkb | I'll go warn the openstack release team | 20:44 |
| fungi | thanks! i'm on hand to help | 20:44 |
| fungi | just heating up some leftovers | 20:45 |
| clarkb | er I math'd poorly my conflict is at 21:20 for about 45 minutes (have to go get the kids from school) | 20:49 |
| fungi | don't sweat it, i'm here to stop/restart things | 20:50 |
| corvus | i'm also around | 20:54 |
| clarkb | I need to convince the school to give us the schedule in UTC | 20:55 |
| fungi | hah | 20:55 |
| fungi | also convince them to teach utc | 20:55 |
| fungi | the abbreviation alone is amusing, since it was chosen to be neither the english (cut) nor french (tuc) abbreviation, so it's actually nonsense for the sake of fairness | 20:56 |
| corvus | wait is the english cut or uct? | 20:58 |
| clarkb | recently I watched a tom scott video on how officially denmark's official time is up to like .11 seconds off from utc due to an old law they haven't changed that based the time on a the 15 degree meridian rather than utc. But the country happily syncs to utc clocks and ignores the problem :) | 20:58 |
| clarkb | corvus: "Coordinated Univeral Time" iirc | 20:58 |
| corvus | huh. well. hrm. i'd like to propose a third option! :) | 20:59 |
| fungi | corvus: utc is "coordinated universal time" in english and "temps universel coordonné" in french | 20:59 |
| fungi | you clearly never paid attention to the time announcements on the bbc world service shortwave broadcasts after they switched off gmt ;) | 21:00 |
| corvus | no; wcpe did carry the world news which had them back in the gmt days tho :) | 21:01 |
| corvus | i have run the zull_pull playbook after confirming the lastest image build jobs succeeded | 21:02 |
| corvus | wow we really should call it the zull_pull playbook. so much sense. | 21:03 |
| mnaser | clarkb, fungi, corvus: i'm ready if/when you are | 21:03 |
| fungi | thanks, i'll do a quick status notice and then stop gerrit | 21:03 |
| clarkb | corvus: ^ I'll let you drive since I have to pop out soon if you want to grab queues and do the zuul side? | 21:03 |
| corvus | i have saved zuul queues. so fungi has the next call. | 21:03 |
| corvus | clarkb: ack | 21:04 |
| clarkb | soudns good. I'm around if I can help but will let you two drive | 21:04 |
| fungi | #status notice The Gerrit service on review.opendev.org is going offline momentarily for a host migration and zuul upgrade, downtime should be only a few minutes. | 21:04 |
| opendevstatus | fungi: sending notice | 21:04 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org is going offline momentarily for a host migration and zuul upgrade, downtime should be only a few minutes. | 21:04 | |
| fungi | i downed the containers on review.o.o and then powered off the server instance gracefully from the operating system for good measure | 21:05 |
| fungi | mnaser: you should be clear to trigger final migration | 21:05 |
| mnaser | ok, starting | 21:05 |
| fungi | corvus: stop zuul containers at will | 21:05 |
| corvus | stopping zuul | 21:05 |
| mnaser | fungi: i see it pingable again | 21:07 |
| fungi | thanks, checking | 21:08 |
| fungi | gerrit container is starting now | 21:08 |
| fungi | looks like the gerrit service is online again | 21:09 |
| corvus | i'll start zuul | 21:09 |
| fungi | corvus: you should be clear to start zuul services again if they've finished stopping | 21:09 |
| fungi | perfect | 21:09 |
| mnaser | wee, that was smooth (at least for me :p) | 21:09 |
| fungi | thanks for working with us on it! | 21:10 |
| corvus | ++ | 21:10 |
| clarkb | yup thank you for being available to do it when it was good for us | 21:12 |
| corvus | mnaser: just happened to notice this in the logs in case you're unaware: https://zuul.opendev.org/t/vexxhost/config-errors | 21:13 |
| corvus | tenants loaded; re-enqueueing | 21:19 |
| fungi | awesome | 21:21 |
| corvus | re-enqueue complete | 21:25 |
| fungi | thanks corvus! | 21:28 |
| corvus | i don't see any new exceptions in the log | 21:28 |
| fungi | i'll consider that great news | 21:29 |
| corvus | (i do see one harmless one about unlocking nodesets for canceled jobs, but it's a pre-existing bug, and i suspect the next set of patches will address it) | 21:29 |
| corvus | so i think next step is just keep an eye out for unusual behavior. i'll check back in a bit and look for stuck jobs, etc. | 21:29 |
| fungi | will do | 21:35 |
| corvus | #status log restarted all of zuul on commit 04678e25e666c5e97b76e68838a0ce1cf0761144 | 21:52 |
| opendevstatus | corvus: finished logging | 21:52 |
| Clark[m] | I'm taking this as an opportunity for system updates locally. But will follow-up with the openstack release team if that hasn't happened already | 22:04 |
| clarkb | I've notified the openstack release team that the work is odne and things are happy | 22:25 |
| fungi | thanks | 22:30 |
| clarkb | ianw: if you'd like me to switch diskimage-builder-core group ownership in gerrit from infra-ptl to diskimage-builder-core to make it self owned I can do that now | 22:30 |
| clarkb | and I can do it without escalating privs as I'm the current owner | 22:30 |
| clarkb | I'm happy for that group to continue to self organize as it has largely done so for the last couple of years at least | 22:31 |
| ianw | clarkb: i think so, thank you. we'll add stevebaker and TheJulia and possibly contact old members to see if they want to be removed | 22:31 |
| fungi | sounds great to me. the more reviewers there the better | 22:32 |
| clarkb | that is done now | 22:32 |
| ianw | thank you | 22:32 |
| clarkb | looking at the membership I suspect many of those would prefer to be removed since they havne't done openstacky things or even cloudy things in a long time | 22:33 |
| ianw | indeed, it's a trip down memory lane, that list | 22:34 |
| stevebaker | clarkb, ianw: awesome, thanks. dib is a dependency on an unusual collection of projects, this will help a lot | 22:35 |
| ianw | clarkb: i think the idea of having a single job checkout system-config works and slots in fairly easily; https://review.opendev.org/q/topic:%22s-c-prod-parallel%22+(status:open%20OR%20status:merged) is the collection | 22:35 |
| clarkb | ianw: thanks I've been all over the place today. Let me take a look | 22:36 |
| ianw | no rush, it is one you want to read after a coffee :) | 22:36 |
| clarkb | ianw: the first thing I notice is the order of the changes seems backwards to me | 22:36 |
| clarkb | or maybe the end result here gets squashed together? | 22:37 |
| clarkb | and the third change is largely independnet and can land today? I'll start with that one I guess | 22:39 |
| ianw | if it matches what's in my head; we add new jobs to opendev/base-jobs, but keep the existing base job as is by including both playbooks | 22:39 |
| ianw | then system-config modifies itself to run the code-checkout job first, and to parent it's base job to the one that only setups the executor login | 22:40 |
| ianw | and yeah, the other one that does the initial split up can be separate; although note it still keeps the semaphore at this point to keep things running in order just while we confirm it doesn't go crazy | 22:41 |
| clarkb | ++ I think changing the semaphores is the very last thing we want to do | 22:41 |
| clarkb | ianw: left a thought on 8070807 | 22:47 |
| clarkb | er 807807 | 22:47 |
| ianw | hrm, yes, it probably should do that "is this periodic i'll use master bit" | 22:49 |
| ianw | it might be clearer to do that in a follow-on? | 22:49 |
| clarkb | ya thats fine too as long as we land that before we try to use the new jobs | 22:52 |
| clarkb | because that will cause updates in all the periodic jobs that we don't want if we don't clean up the existing stuff for that | 22:52 |
| ianw | clarkb: urgh, now i think about it 807672 needs up update periodic-hourly and periodic, doesn't it ... | 23:00 |
| clarkb | ianw: yup we have to do everything all at once | 23:02 |
| clarkb | this is part of what makes it complicated | 23:02 |
| ianw | i think i'll have to move the dependencies into the job definitions in infra-prod.yaml | 23:03 |
| ianw | that is indeed the more logical place for them. it just makes it quite difficult to see what the relationships are at a glance | 23:04 |
| clarkb | but also I think keeping the semaphores doing one job at a time is a good sanity checking when we do a big cutover to the new job dep listings and all that | 23:08 |
| clarkb | then once we're happy with that state we can update just the semaphores so that it enforces one buildset at a time and some upper cap of jobs in a buildset (to not overwhelm bridge). That will require two semaphores | 23:08 |
| clarkb | one that a parent paused job holds (I called this the anchor job in previous discussion) and then a semaphore with say a limt of 5 to start that each other job holds | 23:09 |
| fungi | yeah, i like the idea of ratcheting up the semaphore allowance gradually in an effort not to swamp the bastion | 23:10 |
| fungi | we've certainly seen it get overwhelmed with ansible processes before | 23:10 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!