| ianw | that's interesting about "borg compact". when i look at the borg source that comment seems to predate our use https://github.com/borgbackup/borg/commit/a8d52351bb41dbe82c6fef6010465e05cf4dc43e | 03:40 |
|---|---|---|
| ianw | https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2021-01-19.log.html#t2021-01-19T22:26:29 seems like where we discussed that | 03:40 |
| ianw | <clarkb>ianw: I run borg prune --verbose --list --prefix '{hostname}-' --show-rc --keep-daily 7 --keep-weekly 4 --keep-monthly 6 | 03:40 |
| ianw | that looks suspiciously like the command we use. the conclusion i'm reaching is that i've probably never read that man page :) | 03:41 |
| ianw | however empirically we do know that pruning does actually create free space :) | 03:42 |
| tonyb | it does I think it's akin to "git prune" vs "git repack" | 03:52 |
| ianw | yeah, we probably should start running "compact" | 03:58 |
| ianw | it may, however, be a good time to rotate out the backup volumes before doing that ... | 03:58 |
| tonyb | we can look at that. I don't pretend to understand the implications of that. | 04:01 |
| tonyb | I was kinda thinking we try it as a one of on backup02 and see what happens. | 04:02 |
| opendevreview | Roman Kuznecov proposed zuul/zuul-jobs master: DNM: Investigate POST_FAILURE error https://review.opendev.org/c/zuul/zuul-jobs/+/901294 | 10:31 |
| fungi | tonyb: yeah, rotating the backup volumes is fairly straightforward. make and attach new volumes, change configuration to start putting all new backups on those | 13:05 |
| noonedeadpunk | hey! not sure if it's known or not, but are in-gerrit edits broken after fridays maintenance? | 13:10 |
| noonedeadpunk | as files in edit mode never opens | 13:10 |
| fungi | noonedeadpunk: try a force refresh or cache clear, gerrit seems to cache its javascript client-side and it doesn't work across versions of the server | 13:17 |
| fungi | noonedeadpunk: https://groups.google.com/g/repo-discuss/c/DTrYQtY0j1k/m/7riBbIa5BwAJ | 13:18 |
| noonedeadpunk | fungi: ah, yes, I tried to force refresh on the change page, but not on edit window, that's apparently why it didn't help, thanks for the pointer! | 13:20 |
| fungi | yw | 13:31 |
| fungi | clarkb: ^ looks like we have another data point | 13:31 |
| tonyb | fungi: ianw: of that makes sense and explains the volume name on the host | 13:59 |
| fungi | tonyb: it's also convention, i think, that we delete the old-old volume around the same time. possibly before creating the new one in order to free up cinder quota, i don't recall | 14:00 |
| tonyb | good to know | 14:01 |
| tonyb | would anyone object to me sending a brief message to service-announce. with words the effect of "since the update in browser edits may be not work as expected. we're investigating. in the meantime do a full refresh on any page that is missing data. | 14:03 |
| tonyb | and then sending that link to the various discuss lists? | 14:03 |
| fungi | no objection from me. maybe clarkb can weigh in once it's daytime in his neighborhood | 14:04 |
| tonyb | sounds good | 14:04 |
| tonyb | .... that'll give me time to join some of the other lists | 14:05 |
| *** dhill is now known as Guest7608 | 14:28 | |
| fungi | our automated root e-mails from the backup servers about >90% volume usage seem to have ceased after friday's pruning | 14:43 |
| tonyb | Great success | 14:43 |
| fungi | infra-root: https://public-cloud.status-ovhcloud.com/incidents/ssscy5l81mtm | 14:49 |
| tonyb | Oh, we run borg in append-only mode right? "If borg compact command is used on a repo in append-only mode, there will be no warning or error, but no compaction will happen." | 14:49 |
| fungi | looks like we missed the 2023-11-20 08:30-10:30 window for bhs | 14:49 |
| fungi | not sure if we saw any impact from it | 14:49 |
| fungi | there is another at the same time tomorrow affecting gra | 14:50 |
| fungi | it looks like they don't expect any outages/down-time from it, so i guess no need to temporarily remove them from log uploading | 14:51 |
| tonyb | "Customers must follow RFC rules to avoid having disruptions on their requests" .... What are RFC rules? | 14:51 |
| fungi | "This upgrade will add support for TLS 1.3 as well as strict compliance with RFC7540." | 14:51 |
| fungi | i assume they mean make sure your systems comply with ietf rfc 7540 prior to the maintenance time | 14:51 |
| tonyb | Ahh okay that makes sense | 14:52 |
| fungi | disappearing for an hour to run some errands, back soon | 15:23 |
| tonyb | Kk | 15:23 |
| *** dmellado2 is now known as dmellado | 15:58 | |
| clarkb | I have deleted my gerrit upgrade related autoholds | 16:00 |
| tonyb | Okay | 16:01 |
| clarkb | I'll work on changes to update our gerrit image builds next. Though we probably want to wait until after the long holiday weekend to land any of them | 16:01 |
| clarkb | tonyb: re append only I htink that is correct and explains why we don't compact | 16:01 |
| tonyb | clarkb: That makes some sense at least. | 16:02 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Switch jeepyb to building gerrit 3.8 images https://review.opendev.org/c/openstack/project-config/+/901465 | 16:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch gerrit 3.7 image to 3.8 in a couple places we missed https://review.opendev.org/c/opendev/system-config/+/901466 | 16:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup Gerrit 3.7 image jobs and disable Gerrit upgrade job https://review.opendev.org/c/opendev/system-config/+/901467 | 16:11 |
| fungi | according to the boss, i have more errands to run, so disappearing for another hour-ish (seems like nothing's on fire at the moment) | 16:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.9 image builds https://review.opendev.org/c/opendev/system-config/+/901468 | 16:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.8 to 3.9 upgrade testing https://review.opendev.org/c/opendev/system-config/+/901469 | 16:24 |
| clarkb | https://review.opendev.org/c/openstack/project-config/+/901465 and https://review.opendev.org/c/opendev/system-config/+/901466 are small updates that will need to land before we can start testing the rest of the stack | 16:24 |
| clarkb | infra-root for the old hosts in emergency.yaml I'm basically doing git log -p in system-config and grepping for those names. Confirming that last mention of them was their removal from the inventory and in those cases I'll remove them from the emergency file | 16:26 |
| tonyb | clarkb: Sounds good to me. | 16:27 |
| tonyb | clarkb: I was thinking about doing the 2 OVH mirror nodes at the same time (single zone chnage to add both, single inventory addition etc). Is there any reaon this is a bad idea? | 16:29 |
| clarkb | tonyb: no that should be fine | 16:29 |
| tonyb | Thanks for confrming | 16:30 |
| clarkb | after looking at the emergency file I cleaned up the openedge and citynetwork kna1 mirrors, health01, subunit-worker01/02, nb03.opendev.org | 16:30 |
| clarkb | I've left bridge.openstack.org in place since we probably needto dobuel check that node is fully gone to avoid any accidental restarts | 16:30 |
| clarkb | though bridge.openstack.org is out of the inventory | 16:31 |
| tonyb | Back in a about 30-45mins | 16:49 |
| fungi | okay, back for good | 17:52 |
| tonyb | bold claim ;P | 17:52 |
| fungi | well, for neutral at least, not for evil | 17:53 |
| fungi | probably chaotic good, but we'll see where this week goes | 17:53 |
| tonyb | chaotic seems probable | 18:02 |
| clarkb | for the meeting agenda I was going to drop mm3, keep gerrit 3.8 to recap the upgrade and followup changes above, discuss gitea ssh key rotation stuff. Anything else to add drop? | 18:02 |
| tonyb | Do we want to discuss / think out upgrading the rax trove from mysql-5.7? and/or self hosting the DB? That's probably better as a mailinglist discussion | 18:04 |
| fungi | on the mm3 front, we do have a non-urgent need to add ansible management of templates, but i doubt it requires discussion in the meeting | 18:05 |
| clarkb | fungi: ya I think we can just make that a part of the ansible vars for each list and proceed until that doesn't work for some reason. | 18:05 |
| fungi | tonyb: there are possibly multiple rax trove instances still in use, but i'm assuming you mean the one zuul is relying on (though i guess we could inventory and upgrade all of them if there are still any others) | 18:06 |
| clarkb | I think storyboard, zanata, and zuul are all trove still off the top of my head | 18:06 |
| fungi | i think we took sb's database local in order to improve performance, but maybe we never cleaned up the trove side | 18:07 |
| clarkb | ah | 18:07 |
| fungi | yeah, there's a mysqld running on the sb server | 18:08 |
| tonyb | Yeah I was thinking about zuul specifically. | 18:08 |
| clarkb | fungi: tonyb: https://review.opendev.org/c/openstack/project-config/+/901465 and https://review.opendev.org/c/opendev/system-config/+/901466 are two followup changes that should be quick to review and safe to land now (the project-config change is needed before CI jobs will run against the other changes as well) | 18:09 |
| fungi | both of those lgtm, thanks! | 18:11 |
| opendevreview | Merged openstack/project-config master: Switch jeepyb to building gerrit 3.8 images https://review.opendev.org/c/openstack/project-config/+/901465 | 18:24 |
| clarkb | ok meeting agenda updates are in. Let me know if I missed anything | 18:26 |
| fungi | thanks! | 18:40 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Add new Gerrit submodule java-prettify to Zuul https://review.opendev.org/c/openstack/project-config/+/901479 | 18:49 |
| opendevreview | Merged opendev/system-config master: Switch gerrit 3.7 image to 3.8 in a couple places we missed https://review.opendev.org/c/opendev/system-config/+/901466 | 18:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.9 image builds https://review.opendev.org/c/opendev/system-config/+/901468 | 19:03 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.8 to 3.9 upgrade testing https://review.opendev.org/c/opendev/system-config/+/901469 | 19:03 |
| clarkb | I feel like I have to refresh my memory on how we deal with submodules in gerrit each time they change. tldr is plugins we just copy paste into place and don't submodule init them. jgit, polymer-bridges, and now java-prettify we do submodule init but we rm the gerrit repo git remote first so that it does the init against the local filesystem paths | 19:07 |
| fungi | doesn't zuul-jobs have some helpers for dealing with submodules, or did i dream that? | 19:12 |
| clarkb | unsure. I think gerrit is sufficiently weird we wouldn't want a general solution though | 19:15 |
| fungi | fair | 19:21 |
| tonyb | Looking at the OVH mirror nodes, and I see they're consisent with each other but different to the rax nodes. logical volume names are different, as is the size of the volume: https://paste.opendev.org/show/bJxUerJHsxoUb8qaFaFW/ | 19:23 |
| tonyb | While I get that this isn't a problem, as I'm rebuilding them should I a) stick with the same names/sizes or b) standardize on the rax names and sizes ; or c) $somethign else | 19:25 |
| clarkb | tonyb: the reason for the size difference is that the apache cache cleaner daemon couldn't keep up on rax (slower io iirc) so we gave it larger disk for more headroom | 19:25 |
| clarkb | tonyb: Ithink you can use consistent sizes within ovh as that hasn't been a problem there. | 19:26 |
| clarkb | tonyb: for naming I suspect that is a side effect of needing to do this manually :/ I kinda like the more generic volume naming in rax beacuse we might one day choose to swap out apache for some other proxy cache instead | 19:26 |
| tonyb | naming is hard ;P | 19:27 |
| fungi | that's why i treat it as a drinking game | 19:29 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add a helper script for doing the LVM setup on mirror nodes. https://review.opendev.org/c/opendev/system-config/+/901504 | 19:57 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add a helper script for doing the LVM setup on mirror nodes. https://review.opendev.org/c/opendev/system-config/+/901504 | 21:55 |
| clarkb | tonyb: made some notes on ^ | 22:00 |
| tonyb | clarkb: replied | 22:08 |
| clarkb | thanks responded inline | 22:10 |
| tonyb | okay. I'll address that in a bit. | 22:19 |
| tonyb | clarkb: thanks | 22:19 |
| corvus | tonyb: fyi the reasoning for append-only is for forensic analysis and recovery in the case of compromise | 23:49 |
| corvus | fungi: gerrit's zuul has some gerrit-specific helpers, that's probably what you're thinking of. with a little bit of work they could be genericized, but they still only are appropriate for projects using submodule subscription, so there's a lot of caveats about their use. :) and they're not useful for opendev's gerrit because we aren't in that situation with gerrit (we don't have submodule subscriptions for the plugins) | 23:53 |
| corvus | s/not useful for opendev's gerrit/not useful for opendev's jobs-that-build-gerrit-itself/ :) | 23:57 |
| clarkb | fwiw https://review.opendev.org/c/openstack/project-config/+/901479 is the change we need to make the new submodule work in our buidls of gerrit | 23:57 |
| clarkb | since that is in a trusted repo and requires a zuul config bump it isn't self testing and needs to be merged before it will work | 23:58 |
| corvus | i feel like we run into a high number of meta language issues working on zuul and gerrit | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!