| opendevreview | Dr. Jens Harbott proposed opendev/puppet-openstack_health master: Retire opendev/puppet-openstack_health: remove repo content https://review.opendev.org/c/opendev/puppet-openstack_health/+/836711 | 06:28 |
|---|---|---|
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add nebulous/monitoring https://review.opendev.org/c/openstack/project-config/+/898088 | 08:11 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 09:13 |
| opendevreview | Martin Magr proposed openstack/project-config master: Complete config for python-observabilityclient https://review.opendev.org/c/openstack/project-config/+/898098 | 09:33 |
| opendevreview | Merged openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 11:59 |
| yoctozepto | oooh, it seems the "c++" broke the deploy | 12:23 |
| yoctozepto | https://review.opendev.org/c/openstack/project-config/+/898023 | 12:23 |
| yoctozepto | I think we should change it to cpp | 12:23 |
| yoctozepto | fungi ^ | 12:24 |
| yoctozepto | what's the procedure now? | 12:24 |
| fungi | yoctozepto: i'll take a look, but yeah maybe we need to limit what punctuation is allowed in repository names | 12:36 |
| Clark[m] | It looks like nothing has been created for it? If we can verify that (I don't trust my lookups this early on the morning) then we may just be able to update the name in a change and proceed from there. If anything got created we will need to do cleanup/renaming | 12:51 |
| fungi | yeah, that's my hope, i'm just trying to get dug out from morning tasks so i can take a closer look | 12:55 |
| fungi | gitea servers rejected the create request with "422 Client Error: Unprocessable Entity for url: https://localhost:3000/api/v1/orgs/nebulous/repos" | 13:03 |
| fungi | so i think it bailed before trying to add anything to gerrit | 13:03 |
| fungi | yeah, the "Create repos on gitea servers" task is the very first thing in the manage-projects playbook, so basically nothing was done, therefore nothing to clean up | 13:05 |
| fungi | yoctozepto: feel free to proceed with a change to swap the project name to one without "+" symbols, i'll start looking at adding a ratchet for the projects list to limit acceptable characters | 13:05 |
| fungi | infra-root: config-core: heads up that "+" in project names won't work (at least for gitea), and also please don't approve any other changes touching acls or the projects list until we get the cleanup patch landed | 13:07 |
| Clark[m] | fungi: and you confirmed it never ran against gerrit? | 13:11 |
| fungi | yes | 13:12 |
| Clark[m] | Excellent should be straightforward to fix then | 13:12 |
| fungi | the playbook stopped at the failed gitea task, which is the very first task | 13:12 |
| frickler | .oO(c-doubleplus-ungood) | 13:13 |
| fungi | we should pass all project names through an orwell filter | 13:13 |
| opendevreview | Merged opendev/system-config master: Move OpenStack lists to Mailman 3 https://review.opendev.org/c/opendev/system-config/+/897998 | 13:14 |
| fungi | early maintenance prep is done once that ^ deploys, so we should be all set for starting on time in a little over 2 hours from now | 13:16 |
| fungi | reminder, maintenance plan is at https://etherpad.opendev.org/p/mm3migration starting at line 285 now | 13:17 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Check proposed project names for problems https://review.opendev.org/c/openstack/project-config/+/898118 | 13:36 |
| fungi | that ^ should fail on the tip of master, and should succeed once rebased onto yoctozepto's cleanup change (once it exists) | 13:36 |
| opendevreview | Merged opendev/puppet-openstack_health master: Retire opendev/puppet-openstack_health: remove repo content https://review.opendev.org/c/opendev/puppet-openstack_health/+/836711 | 13:40 |
| * yoctozepto going to help fix | 13:52 | |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Fix c++ to cpp https://review.opendev.org/c/openstack/project-config/+/898119 | 13:55 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Check proposed project names for problems https://review.opendev.org/c/openstack/project-config/+/898118 | 13:56 |
| fungi | thanks! | 13:58 |
| * fungi is fixing the flake8 complaints for 898118 | 13:59 | |
| yoctozepto | I also reviewed your change | 14:00 |
| yoctozepto | you might want to improve that minor bit there | 14:00 |
| fungi | oh, sure | 14:00 |
| fungi | yoctozepto: why are the continues a problem? we already short-circuit the loop on a different error condition, i'm just adding two more possible ones | 14:01 |
| yoctozepto | fungi: that one was necessary because the other checks (which check the name) could not run sensibly | 14:01 |
| yoctozepto | these simply prevent the other name checks from happening at the same time | 14:02 |
| yoctozepto | or actually this has really affected the logging | 14:03 |
| yoctozepto | as without the name it did not make sense | 14:03 |
| yoctozepto | nonetheless you don't need the continues now | 14:03 |
| yoctozepto | and you get all errors at once | 14:03 |
| yoctozepto | #benefitseverywhere | 14:03 |
| fungi | the check for slashes could still lead to problems for other checks after it i think? | 14:04 |
| fungi | though looks like it does nothing with the repo_group part | 14:05 |
| fungi | and repo_name is only checked against one thing | 14:05 |
| fungi | so i agree, dropping the additional continues is reasonably safe | 14:06 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Check proposed project names for problems https://review.opendev.org/c/openstack/project-config/+/898118 | 14:08 |
| fungi | thanks again! | 14:08 |
| yoctozepto | yw | 14:10 |
| yoctozepto | so let's merge :-) | 14:10 |
| fungi | yep, i just approved the fix | 14:14 |
| Clark[m] | We can also run system-config-run-gitea against projects.yaml updates if we want to check gitea's acceptance of names | 14:22 |
| fungi | yep, but i'm more worried about choices that cause less-immediate breakage | 14:23 |
| fungi | it's good that gitea at least rejected that at the api rather than going ahead and creating something that would have had subtle problems resulting in a future rename maintenance | 14:24 |
| opendevreview | Merged openstack/project-config master: Fix c++ to cpp https://review.opendev.org/c/openstack/project-config/+/898119 | 14:27 |
| fungi | just under an hour until maintenance time | 14:32 |
| fungi | i've added some approximate times for the longer steps in the maintenance plan, based on earlier test import timings and such | 14:40 |
| fungi | the longer steps involve ~3 hours to complete, so that leaves us only an hour of fudge time in the 4-hour window | 14:41 |
| fungi | yoctozepto: "Build succeeded (deploy pipeline). infra-prod-manage-projects [...] SUCCESS in 4m 03s" | 14:43 |
| * clarkb is finally at the proper computer | 15:03 | |
| clarkb | lists.openstack.org's ssl cert expires in 27 days or less. I don't think we care to debug that as we are switching servers | 15:10 |
| fungi | indeed, chuckled at that as well | 15:12 |
| clarkb | https://review.opendev.org/c/openstack/project-config/+/897710 is a quick change to align jeepyb's gerrit builds with the new base image for gerrit | 15:15 |
| frickler | the manage-projects failure on the lists change is harmless I think? not sure why it ran there at all | 15:17 |
| clarkb | fungi: ^ | 15:17 |
| frickler | https://zuul.opendev.org/t/openstack/build/9054af8bb414441dbbfb60dba0dfa6a9 | 15:18 |
| clarkb | frickler: the error is likely the c++ error | 15:18 |
| clarkb | I don't know why it ran on a lists change | 15:18 |
| frickler | yes, just making sure nothing important was skipped | 15:18 |
| clarkb | it should recreate all the things when it ran with the cpp fix. We don't do deltas and instaed operate on the whole set (in part for this raeson it is really difficult to know what the state is across ansible runs) | 15:19 |
| fungi | correct, i checked it, the lists server deployment succeeded, just manage-projects failed and it's not relevant to the maintenance | 15:21 |
| fungi | (also it ran successfully since then) | 15:22 |
| fungi | starting maintenance in 1 minute | 15:28 |
| fungi | #status notice The lists.openstack.org site will be offline over the next few hours for migration to a new server | 15:30 |
| opendevstatus | fungi: sending notice | 15:30 |
| -opendevstatus- NOTICE: The lists.openstack.org site will be offline over the next few hours for migration to a new server | 15:30 | |
| fungi | dns records updated as of 15:31, we can proceed with service shutdown as early as 15:36 | 15:31 |
| clarkb | ack | 15:31 |
| fungi | we're at line 292 now | 15:32 |
| clarkb | good reminder I should open the etherpad | 15:32 |
| opendevstatus | fungi: finished sending notice | 15:32 |
| fungi | the penultimate rsync is underway in a root screen session on lists.openstack.org | 15:33 |
| fungi | i've also got a root screen session open on lists01.opendev.org for running the migration commands once we reach that step | 15:34 |
| fungi | lists.openstack.org is resolving to the new addresses for me | 15:36 |
| fungi | as soon as this penultimate rsync concludes, i'll stop and disable mailman services on the old server | 15:37 |
| opendevreview | Ghanshyam proposed openstack/project-config master: End gate and update acl for retiring js-openstack-lib https://review.opendev.org/c/openstack/project-config/+/898015 | 15:37 |
| fungi | by "new addresses" i mean the temporary addresses | 15:37 |
| clarkb | fungi: I see updated resolution to the temp address as well | 15:38 |
| fungi | for some reason the penultimate rsync is taking waaaay longer than ones i performed in the past hour | 15:42 |
| fungi | done in 13m7.536s | 15:43 |
| fungi | services stopped/disabled and final rsync in progress now | 15:44 |
| fungi | so much variability in these rsyncs | 15:49 |
| fungi | that one was at least only 5m55.615s | 15:50 |
| fungi | moving on | 15:50 |
| fungi | import is in progress now | 15:51 |
| fungi | prior tests on held nodes took nearly 2.5 hours, but this server is probably faster so it could finish sooner than that | 15:51 |
| clarkb | we can hope it is faster anyway :) | 15:52 |
| fungi | regardless, i'll do something else while i keep an eye on the progress | 15:52 |
| fungi | load average on the old server is finally sub-0.1 | 15:53 |
| fungi | up 475 days since last reboot | 15:54 |
| clarkb | I'll be glad to not worry about kernel compression anymore | 15:54 |
| fungi | yes, i'd be lying if i said it kept me up nights, but it will be nice to have one less oddity to care about | 15:55 |
| fungi | for those following along, we're at line 301 in https://etherpad.opendev.org/p/mm3migration now | 15:55 |
| fungi | and will be for... a while | 15:55 |
| fungi | time estimate is probably spot on. at least the archive import for openstack-discuss took 19m51.119s and when i did the initial timings a year ago that step was 18m42.199s | 16:34 |
| fungi | a little more content in the archive, a little more time to finish importing | 16:34 |
| fungi | so probably getting to our final steps around 18:20z | 16:35 |
| fungi | unless something else is way faster | 16:35 |
| clarkb | someone might press the turbo button on the front of the server :) | 16:52 |
| fungi | i'm a little worried though, the archive reindex for openstack-discuss took 43m23.527s just now vs the 6m41.204s i saw in testing | 17:12 |
| clarkb | thats the biggest one though? | 17:13 |
| clarkb | so even if it is slower hopefully its within a manageable time frame | 17:13 |
| fungi | maybe database contention with the other active lists which i didn't account for in testing | 17:13 |
| fungi | yeah, it's by far the largest | 17:13 |
| fungi | clarkb: i lied (or, well, forgot)... the biggest one is openstack-stable-maint because of the previously hundreds (now merely dozens) of periodic job failure notifications it receives every day | 17:48 |
| clarkb | ah | 17:49 |
| fungi | it's currently 22% into importing the mbox for that now | 17:49 |
| clarkb | oof | 17:49 |
| clarkb | does it run them serially or concurrently? Hopefully serially which would mean this particular one is going faster having not run since the start? | 17:51 |
| fungi | serially | 17:52 |
| clarkb | ok so once this one is done we'll hopefully finish up quickly | 17:52 |
| * yoctozepto looking for approvals on https://review.opendev.org/c/openstack/project-config/+/898088 | 17:52 | |
| fungi | looks like it probably takes an hour for the import step of this list and it's maybe 20 minutes in, so probably shouldn't expect it to move onto indexing until 18:30 | 17:52 |
| * yoctozepto sorry for interrupting otherwise | 17:53 | |
| fungi | though there aren't really any other lists after this one (it's going alphabetically by list address), so should still hopefully finish on time | 17:53 |
| fungi | yoctozepto: yep, i was going to go through a few more pending project-config updates shortly | 17:53 |
| clarkb | fungi: I wonder if the speed difference is in the root disk. I think the test nodes had an ephemeral local drive but this one doesn't? | 17:54 |
| clarkb | anyway probably not worth worrying too much about at this point. Just need patience | 17:54 |
| yoctozepto | 😃 | 17:54 |
| fungi | yeah, i wouldn't be surprised at i/o bottlenecks there. and different phases of this process are probably impacted in different ways | 17:55 |
| clarkb | yoctozepto: the repo has a .DS_Store file is that intentional? | 17:56 |
| fungi | macos to the rescue | 17:57 |
| yoctozepto | clarkb: yeah, well, it is like it is | 18:01 |
| fungi | there are 3 more lists after openstack-stable-maint but they're relatively small (openstack-zh, release-announce, release-job-failures) | 18:05 |
| fungi | of those, release-announce is probably going to take the most time | 18:05 |
| fungi | i don't have relative timing data for all the mailing lists though, so hard to say exactly how long to expect | 18:06 |
| opendevreview | Merged opendev/system-config master: Remove ara from source install option https://review.opendev.org/c/opendev/system-config/+/894576 | 18:07 |
| opendevreview | Merged openstack/project-config master: Report Vitrage changes to stable branches in the IRC https://review.opendev.org/c/openstack/project-config/+/894256 | 18:08 |
| clarkb | fungi: how are things looking an hour later? | 18:47 |
| fungi | indexing openstack-stable-maint took 61m58.525s | 19:02 |
| fungi | er, importing i mean | 19:02 |
| fungi | indexing is running now | 19:02 |
| clarkb | and -discuss the import was half the time of the indexing? | 19:02 |
| fungi | hopefully will finish quickly? hard to say | 19:03 |
| fungi | looking | 19:03 |
| clarkb | probably still on track to get done before the day ends, but take longer than we expected? May be worth doing a #status notice (can't send email heh) | 19:03 |
| fungi | openstack-discuss: import21=1m12.552s hyperkitty_import=19m51.119s update_index_one_list=43m23.527s | 19:05 |
| clarkb | fungi: is that 62 minute time for import the two import steps? In which case we should expect about another hour maybe? | 19:06 |
| clarkb | oh wait thats import21 not import 21m | 19:06 |
| clarkb | so ya ~2x | 19:06 |
| fungi | openstack-stable-maint: import21=0m5.389s hyperkitty_import=61m58.525s update_index_one_list=?(in progress) | 19:06 |
| fungi | on a held test node, update_index_one_list took less time than hyperkitty_import, so different resource constraints | 19:07 |
| fungi | openstack-stable-maint: import21=0m5.389s hyperkitty_import=61m58.525s update_index_one_list=29m24.880s | 19:08 |
| fungi | just finished | 19:08 |
| clarkb | nice | 19:08 |
| fungi | it's on to openstack-zh | 19:08 |
| clarkb | I guess you can't really infer from one list to another | 19:08 |
| clarkb | that stable maint list is probably also fairly unique in that the data in it has a lot of overlap and consistent senders etc | 19:08 |
| fungi | right, apples and oranges | 19:10 |
| fungi | it's already on release-announce | 19:10 |
| fungi | one more after this | 19:11 |
| fungi | we may still finish roughly on time | 19:11 |
| fungi | the remaining steps go quickly | 19:11 |
| clarkb | cool | 19:11 |
| fungi | i've prepped the completion e-mail which will serve as the delivery test, so should be able to send it as soon as dns updates propagate when we get to that step | 19:28 |
| fungi | also i'm logged into everything that needs to be logged in for the remaining steps | 19:32 |
| clarkb | I'm here ready to assist should that be necessary | 19:34 |
| fungi | thanks | 19:36 |
| fungi | looks like lists01 has not been added to cacti yet | 19:39 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Remove the old mailing list server https://review.opendev.org/c/opendev/system-config/+/898004 | 19:44 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add Mailman v3 server to Cacti https://review.opendev.org/c/opendev/system-config/+/898138 | 19:44 |
| opendevreview | Merged openstack/project-config master: Remove ara from Zuul config https://review.opendev.org/c/openstack/project-config/+/894577 | 19:48 |
| clarkb | fungi: I left a couple of notes on that old list server cleanup change | 19:51 |
| fungi | thanks! | 19:53 |
| clockworky | hello! | 19:54 |
| opendevreview | Merged openstack/project-config master: JayF volunteering for IRC ops https://review.opendev.org/c/openstack/project-config/+/896162 | 19:55 |
| clockworky | i see that on the main page it says i can host my project on here, but how would i do that? | 19:55 |
| opendevreview | Merged openstack/project-config master: End gate and update acl for retiring js-openstack-lib https://review.opendev.org/c/openstack/project-config/+/898015 | 19:55 |
| opendevreview | Merged openstack/project-config master: Add nebulous/monitoring https://review.opendev.org/c/openstack/project-config/+/898088 | 19:55 |
| clarkb | clockworky: there is a large probably too long document on the process at https://docs.opendev.org/opendev/infra-manual/latest/creators.html because it tries to cover all the details. | 19:56 |
| clarkb | I was just a few seconds too slow... | 19:56 |
| fungi | speaking of slow, archive indexing for release-announce is still going | 20:01 |
| fungi | i think the archive import time depends on the number of messages, while the search indexing time has more to do with the volume of text inside them | 20:03 |
| fungi | maybe release announcements should be made shorter ;) | 20:05 |
| clockwor2y | clarkb: saw your reply, changed irc clients because the one i was using was a bit too slow | 20:06 |
| clockwor2y | thx :) | 20:06 |
| clarkb | clockwor2y: oh good. The short version of it is that you push a change to a specific repo with some details about your project and when that merges things get created for you | 20:07 |
| clarkb | clockwor2y: https://review.opendev.org/c/openstack/project-config/+/898088 is an example of that | 20:07 |
| clarkb | in general projects need to be OSI licensed and you should probably have some understanding of how Gerrit works (it isn't a PR model but a change model) | 20:07 |
| clarkb | just to make sure that Gerrit is appropriate for you | 20:07 |
| clockwor2y | worked with gerrit back when i was messing around with a mediawiki skin in wikimedia, i liked it pretty well, and my project is OSI licensed | 20:11 |
| clarkb | beyond that we encourage people using the systems to at least have some familiarity with how they are hosted. Part of the reason we haven't tried to hard to get away from the project creation process we have. Those of us operating the systems are also its users and if we can grow that user and operator base all the better | 20:13 |
| fungi | 898088 is an incomplete example because it's relying on an already existing acl | 20:13 |
| clarkb | fungi: good point | 20:13 |
| fungi | #status log Archive imports for lists.openstack.org are taking longer than anticipated to complete... revised maintenance conclusion estimate is 21:00 UTC | 20:20 |
| opendevstatus | fungi: finished logging | 20:20 |
| fungi | the search indexing seems to be very i/o bound, top says most of this is iowait | 20:23 |
| fungi | it's spread across multiple cpus, but not using more than 1 in aggregate, which is odd | 20:23 |
| fungi | probably some weird behavior for linux kernel scheduling with i/o-bound single-threaded processes | 20:24 |
| clarkb | and this is the announce list? I wouldnt' ahve expected it to have a ton of content to index | 20:26 |
| fungi | release-announce, yes | 20:26 |
| fungi | i blame doug for scripting all that amusing variation into openstack release announcement messages ;) | 20:26 |
| fungi | also possible the timing differences we're seeing can be chalked up to the "noisy neighbor" phenomenon | 20:28 |
| clarkb | fungi: does it give a completion percentage for indexing? I think you indicated imports do | 20:30 |
| fungi | imports do, indexing does not unfortunately and that's what we're waiting on | 20:31 |
| fungi | and finally, release-job-failures is importing (i guess it's a good thing openstack's release jobs fail very infrequently and the buildset result messages from zuul's smtp reporter are brief) | 20:43 |
| clarkb | ++ | 20:43 |
| clarkb | is this the last list? | 20:43 |
| fungi | i believe so (alphabetically, which seems to be how they're processed) | 20:44 |
| fungi | it's already onto building the search index for it | 20:44 |
| fungi | done! | 20:48 |
| fungi | moving onto django config | 20:48 |
| fungi | dns updates applied as of 20:51 | 20:51 |
| fungi | should be able to start testing at 20:56 | 20:51 |
| * clarkb waits patiently to load https://lists.openstack.org | 20:51 | |
| opendevreview | Merged openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE https://review.opendev.org/c/openstack/diskimage-builder/+/891299 | 20:53 |
| opendevreview | Merged openstack/diskimage-builder master: Stop creating default user for cloud-init https://review.opendev.org/c/openstack/diskimage-builder/+/891322 | 20:53 |
| clarkb | it is loading for me and I can see things in the service-discuss archive that I expect to be there | 20:56 |
| fungi | test! | 20:56 |
| fungi | yeah, all the urls in the pad do as expected. sending completion message to test a list | 20:58 |
| clarkb | I have recieved your test email | 20:59 |
| fungi | as have i | 20:59 |
| clarkb | it is also in the archive | 21:00 |
| fungi | yep, test checklist complete | 21:01 |
| clarkb | anyone know how to play taps for the old server when we shut it down? | 21:01 |
| fungi | we'll have to send out a call for trumpet players | 21:02 |
| fungi | and now i'm overdue to switch to making pizza | 21:02 |
| clarkb | mmm pizza | 21:02 |
| fungi | will check back in a while to see whether deferred messages are trickling in | 21:02 |
| fungi | already one from 20:36:35 utc has been delivered | 21:03 |
| clarkb | yup I see it too | 21:03 |
| clarkb | fungi: should we put lists.o.o in the emergency hosts lists to prevent anything from restarting services there? | 21:14 |
| JayF | lookin' good! | 21:18 |
| JayF | It's very strange that it has you create a username, then you login with the email, but the prompt on the login page is ambiguous ("login", not username or email) | 21:20 |
| Clark[m] | I think it will accept both on login | 21:21 |
| JayF | that does not match my experience, at least if usernames have consistently applied case (I signed up as JayF and logged in as JayF; I guess if it's lower() on signup but not login it could fail though) | 21:22 |
| Clark[m] | I mean you can use username or email to login. I don't know about case handling | 21:23 |
| Clark[m] | I don't remember how I login. It's however keepassXC tells me to | 21:23 |
| JayF | username works but lowercase only | 21:23 |
| Clark[m] | Ah | 21:24 |
| fungi | clarkb: it's already in the emergency hosts list since around 12:30 utc per the maintenance plan, and will stay there until we remove it from the inventory | 22:05 |
| fungi | but also i set the services to disabled after stopping them, just to be extra sure in case of reboots | 22:06 |
| fungi | related, whatever the problem was with ansible scraping our xenial pip ppa earlier in the week, seems to be solved now | 22:44 |
| fungi | glad i put off looking any deeper into that situation | 22:45 |
| clarkb | the best kind of bugfix is the one that happens magically | 22:47 |
| opendevreview | Merged opendev/system-config master: Add Mailman v3 server to Cacti https://review.opendev.org/c/opendev/system-config/+/898138 | 22:53 |
| clockwor2y | exit | 23:19 |
| clockwor2y | exit | 23:20 |
| clockwor2y | oh | 23:20 |
| clockwor2y | forgot to go in command mode | 23:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!