| opendevreview | Jeremy Stanley proposed opendev/system-config master: Move Airship and Kata lists to Mailman 3 https://review.opendev.org/c/opendev/system-config/+/895125 | 02:54 |
|---|---|---|
| fungi | that's ^ for the 15:30 maintenance, and i've warmed preliminary rsyncs of both sites to the new server in preparation (will run again just before the outage) | 02:55 |
| fungi | ttls on the existing dns records are already short, so no need to adjust them in advance | 02:58 |
| hashar | fungi: hello, I have been upgrading Wikimedia to tox version 4 and encountered an issue with skipsdist + usedevelop not working anymore | 04:16 |
| hashar | you commented on the upstream task at https://github.com/tox-dev/tox/issues/2730 and I think I might have found a solution (`package=editable` instead) | 04:17 |
| hashar | I found a few more oddities which I have collected at https://phabricator.wikimedia.org/T345695 , but most are listed on the upstream upgrade guide https://tox.wiki/en/latest/upgrading.html | 04:17 |
| hashar | I am unlikely to follow up here, but happy to chat about it if it can helps you migrate :] | 04:18 |
| hashar | (also the current Debian comes with tox 3.28.0 :\ ) | 04:18 |
| *** osmanlicilegi is now known as Guest0 | 04:33 | |
| opendevreview | Mayank Patel proposed openstack/project-config master: Add node-interface-metrics-exporter app to StarlingX https://review.opendev.org/c/openstack/project-config/+/894910 | 06:07 |
| fungi | hashar: i think all of our projects have migrated to tox v4 or switched to nox by now anyway, so it's not a concern at this point, but thanks for the additional info! | 11:43 |
| fungi | what most projects ended up doing is just removing skipsdist from tox.ini | 11:46 |
| fungi | if we end up needing to skip installing for performance reasons, the package=editable setting does look useful | 11:53 |
| hashar | fungi: yeah looks like one can use `package=editable` and for a venv that still requires a full install optin by using `package=sdist` | 11:54 |
| hashar | I should like at nox one day :] | 11:55 |
| fungi | we switched a lot of opendev's projects over to nox, and also zuul's projects are using nox now. most of openstack stayed on tox though | 12:00 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Move Airship and Kata lists to Mailman 3 https://review.opendev.org/c/opendev/system-config/+/895125 | 12:08 |
| fungi | missed cleaning up the airship domain from the old listserver tests | 12:09 |
| *** amoralej is now known as amoralej|lunch | 12:12 | |
| opendevreview | Merged openstack/project-config master: Add node-interface-metrics-exporter app to StarlingX https://review.opendev.org/c/openstack/project-config/+/894910 | 12:29 |
| fungi | i've refreshed the airship and kata rsyncs to the new server, will do it one more time right before i make dns changes | 12:34 |
| fungi | cloudnull: just heard your news! now can you fix image processing times in iad for me? ;) | 12:47 |
| fungi | (also congrats) | 12:47 |
| *** amoralej|lunch is now known as amoralej | 13:06 | |
| fungi | infra-root: https://review.opendev.org/895125 is passing tests, we'll want to merge it early enough so it deploys prior to the start of the 15:30 utc maintenance window | 13:07 |
| * fungi remembered right before bed that we didn't have a change for that yet | 13:07 | |
| Clark[m] | fungi: change lgtm but I'm on a phone until after school drop off so can't really +2 in gerrit | 13:43 |
| fungi | there's no rush | 13:44 |
| fungi | i'm fleshing out details for the maintenance steps near the bottom of https://etherpad.opendev.org/p/mm3migration starting at line 130 | 13:44 |
| fungi | for the previous migrations we temporarily pointed their dns names to the addresses for review02 (just so that message delivery attempts would get a tcp connection refused rather than a socket timeout), but since that actually serves http(s) and has redirects which browsers might cache, i think this time i'll point traffic to insecure-ci-registry as it has neither smtp nor http(s) listening | 13:47 |
| Clark[m] | Wfm | 13:51 |
| *** tosky_ is now known as tosky | 14:06 | |
| fungi | okay, i think i have all the steps outlined with cut-n-paste commands/values, and also an explicit post-import testing checklist | 14:10 |
| fungi | we're at the one hour mark (slightly past) so i'll approve 895125 and refresh the import content one more time | 14:33 |
| fungi | i've staged the list completion notices as postponed draft e-mails so i can send them quickly when we're at the end | 14:44 |
| fungi | i might should have approved that change earlier. zuul is still estimating another hour for it to finish in the gate pipeline (though that's probably a skewed estimate due to the paused registry job) | 14:51 |
| fungi | though maybe not. it thinks system-config-run-gitea is going to take nearly that long on its own | 14:53 |
| *** dviroel__ is now known as dviroel | 14:59 | |
| *** diablo_rojo_phone is now known as Guest96 | 15:00 | |
| clarkb | yes the gitea job is not fast | 15:07 |
| fungi | everything else has succeeded now, just waiting for the gitea job | 15:09 |
| clarkb | I'm here properly now fwiw | 15:12 |
| fungi | cool, we're almost certainly looking at a delayed start at this point anyway, barring miracles | 15:12 |
| fungi | i guess the lengthy tasks in the gitea job involve cloning git repos? | 15:13 |
| fungi | the stream log says it completed the apache2 service restart 10 minutes ago, and since then it's been silent | 15:14 |
| clarkb | fungi: the main time sink is that we create all the gitea projects and then we do a second pass to ensure it is idempotent | 15:14 |
| clarkb | ebcause we've had problems with that in the past | 15:15 |
| fungi | aha, yeah | 15:15 |
| fungi | zuul-stream keeps timing out the socket because these tasks sit silent for so long | 15:15 |
| fungi | at least it should merge before the next hourly batch | 15:21 |
| fungi | at this rate i'm going to need another cup of tea | 15:23 |
| fungi | (green tea, not gitea) | 15:23 |
| fungi | for those just tuning in, the maintenance scheduled for 15:30 utc is on a delayed start, probably won't get underway properly until 16:00 utc due to unanticipated gating job durations | 15:30 |
| fungi | zuul estimates another 16 minutes until change 895125 merges, and then we need to wait for all the deployment jobs which will be triggered from it (or at least wait through the serialized list of them until the lists jobs complete) | 15:32 |
| fungi | at that point we can proceed with the steps at the bottom of https://etherpad.opendev.org/p/mm3migration (line 146 onward) | 15:33 |
| fungi | apparently that was indeed the create repos/orgs task that silently took about 30 minutes to complete | 15:36 |
| fungi | it's finally on to taking screenshots, so shouldn't be too much longer | 15:38 |
| opendevreview | Merged opendev/system-config master: Move Airship and Kata lists to Mailman 3 https://review.opendev.org/c/opendev/system-config/+/895125 | 15:45 |
| fungi | and there it is. in deploy pipeline now | 15:46 |
| fungi | so many deploy jobs... | 15:47 |
| clarkb | probably due to modiying the inventory | 15:47 |
| fungi | yep | 15:47 |
| fungi | in hindsight, i could have split that to a separate change | 15:47 |
| fungi | doesn't apply to future migrations except maybe the final one | 15:47 |
| clarkb | and we can do the server cleanup independent of the migration I think | 15:48 |
| fungi | lists and lists3 deploy jobs are about 2/3 of the way through the series | 15:48 |
| clarkb | re ssh host keys for gerrit we could publish them at https://opendev.org | 15:49 |
| fungi | this is probably going to push our start time out to more like 16:30 | 15:49 |
| fungi | clarkb: yeah, i'm open to alternative locations to stick the host key fingerprints as long as the new user documentation links to it clearly | 15:50 |
| clarkb | we mgiht also be able to edit our gerrit theme plugin to add that content somewhere on gerrit | 15:51 |
| clarkb | some sort of location conditional check and then put it in the footer | 15:51 |
| clarkb | but a simple addition to the gitea landing page is easy and probably effective | 15:51 |
| fungi | clarkb: what do you think about a feature request for the "get started" link? it seems like people are finding that first and then not noticing project-specific instructions | 15:53 |
| fungi | or maybe our getting started guide should have a first step that says to look at any CONTRIBUTING.rst file in the repository you're planning to work on | 15:53 |
| fungi | i suppose a highly visible admonition in our docs is easier than changing gitea's interface | 15:54 |
| clarkb | the problem with that is gitea will tell you to use the wiki | 15:55 |
| clarkb | I highly suspect that wouldn't get far | 15:55 |
| clarkb | or to put it in the project readme which is rendered on the project root page | 15:55 |
| clarkb | thats what projects should do | 15:55 |
| fungi | yeah, i guess our guide could say to look at both the readme and contributing files for the project | 15:56 |
| clarkb | well the readme is just as visible as a project specific getting started link would be | 15:56 |
| clarkb | so not sure you need to do anything extra if you just put the info in the readme. Maybe I'm old school but that is why readmes are called read me :) | 15:57 |
| fungi | i don't know. i mean we have project-specific buttons to take you to the defect tracker or open changes, those are more visible than digging through a wall of prose | 15:57 |
| fungi | the tl;dr problem | 15:58 |
| fungi | i agree people should read. doesn't mean they will | 15:58 |
| clarkb | I guess. But also upstream issue tracker and code review links built in. Unfortauntely "getting started" isn't a project level thing | 15:58 |
| clarkb | we can file a request but I suspect the response will be "this is why we render readmes on the front page and give you a wiki" | 15:59 |
| fungi | right, so for people who click the get started button and end up in our generic guide, we can just remind them to read whatever project-specific instructions might be provided first | 15:59 |
| clarkb | ya | 15:59 |
| fungi | i.e. people who should be starting from the openstack contributor guide instead of the infra manual | 16:00 |
| fungi | and then get confused because openstack has additional setup expectations which don't apply to all projects we host | 16:00 |
| frickler | we can make comments that cross reference both guides at the start of each of them maybe | 16:07 |
| frickler | or possibly also for others like starlingx from the opendev one | 16:08 |
| fungi | we already do to some extent. like the section about the icla links over to the openstack guide, while the openstack guide links to ours where account setup is concerned | 16:08 |
| fungi | about 5 more jobs to complete before it starts on the lists jobs | 16:26 |
| fungi | lists jobs are starting now. once both have completed i'll proceed with the maintenance (no need to wait for the rest of the deploy jobs to finish as they're for wholly unrelated services) | 16:41 |
| fungi | #status notice The lists.airshipit.org and lists.katacontainers.io sites will be offline briefly for migration to a new server | 16:46 |
| opendevstatus | fungi: sending notice | 16:46 |
| -opendevstatus- NOTICE: The lists.airshipit.org and lists.katacontainers.io sites will be offline briefly for migration to a new server | 16:46 | |
| fungi | ip addresses were updated at 16:48 | 16:49 |
| opendevstatus | fungi: finished sending notice | 16:49 |
| fungi | at 16:53 we should be safe to proceed (all prior records had a 5-minute ttl) | 16:49 |
| fungi | lists.airshipit.org and lists.katacontainers.io no longer respond to smtp or http(s) | 16:53 |
| clarkb | fwiw I am following along :) | 16:58 |
| clarkb | just let me know if I can do anything to be useful | 16:59 |
| fungi | airship list imports are in progress now | 16:59 |
| fungi | log is being recorded in /root/import_airship.log on the new server | 16:59 |
| fungi | kata list imports are in progress now. log is being recorded in /root/import_kata.log on the new server | 17:03 |
| fungi | and that's done, fiddling with django next | 17:06 |
| fungi | django has been fiddled | 17:10 |
| fungi | updating dns to point to the new server now | 17:10 |
| fungi | dns updates applied as of 17:14 so we should be safe to proceed at 17:19 | 17:14 |
| fungi | note that as of now we should expect the migration to be done, we're just waiting for dns propagation before we test to be sure everything's working as expected | 17:15 |
| clarkb | yup the etherpad seems to reflect that its checking stuff at this point | 17:15 |
| fungi | once we perform cursory webui checks, then i'll send confirmation messages to lists on both sites and make sure they come through and appear in the archive | 17:16 |
| fungi | dns propagation should be done, we can proceed with tests | 17:19 |
| clarkb | the mm3 web ui stuff for both lgtm at a quick glance | 17:21 |
| fungi | thanks. yeah i checked all the urls in the pad and they go where i expect and contain what i expect | 17:22 |
| fungi | i'll send the completion notifications next | 17:22 |
| fungi | messages sent | 17:24 |
| fungi | and received | 17:24 |
| clarkb | mm3 archives show the sender time in the thread view. Unexpected but cool | 17:25 |
| fungi | links to both archived messages are in the pad now | 17:25 |
| clarkb | I see both in the archive and in my inbox | 17:26 |
| fungi | so once the config deployed, maintenance was ~40 minutes | 17:27 |
| fungi | i'll start writing the notifications for lists.openinfra.dev and lists.starlingx.io once i get a snack | 17:28 |
| fungi | sent them. also drafted a maintenance plan for next week, starts at line 198 in https://etherpad.opendev.org/p/mm3migration | 17:56 |
| fungi | infra-root: ^ when you get a chance, let me know if you have any concerns but it's basically a rip-n-replace of today's maintenance with adjusted server/site/list names | 17:57 |
| fungi | i'll get the corresponding config change put together shortly | 17:58 |
| clarkb | fungi: the plan for today seemed to go well other than unexecpted timing (which isn'ta big deal imo) | 18:19 |
| clarkb | looks like you've given yourself time for that in the new plan | 18:19 |
| clarkb | so ya lgtm | 18:19 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Move OpenInfra and StarlingX lists to Mailman 3 https://review.opendev.org/c/opendev/system-config/+/895205 | 18:28 |
| fungi | and there's the config change for it | 18:28 |
| fungi | okay, other that reviewing that change, we're as prepared as we can be for next week's maintenance, i think | 18:41 |
| opendevreview | Matthias Runge proposed openstack/project-config master: Add python-observabilityclient https://review.opendev.org/c/openstack/project-config/+/894541 | 18:44 |
| clarkb | tobias-urdin: I think you are involved with gnocchi things? In our Zuul config gnocchixyz/gnocchi is being accessed anonymously which hits rate limits when we do a big zuul restart like we did last weekend | 18:48 |
| clarkb | tobias-urdin: I would like to improve this. One way to do that is to have gnocchi install the opendev zuul app in github then we get the app isntall rate limits which are much higher. Additionally if you are willing to help us test things we could use gnocchi as a test case for limiting access to gnocchi for that app as its is largely only used on the openstack side for integration | 18:49 |
| clarkb | testing | 18:49 |
| clarkb | I think by default that app asks for permissions sufficient for managing the github project test more completely. But that shouldn't be required. Unfortuantely I don't personally have a good github setup/project to go testing this myself | 18:49 |
| clarkb | but I thin gnocchi would be a good test case since it is I think actively used in downstream integration testing | 18:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Convert haproxy-statsd to bookworm base image https://review.opendev.org/c/opendev/system-config/+/895215 | 20:37 |
| clarkb | that change should be very straightforward and risk is low if something uenxecpted happens | 20:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!