| *** ryohayakawa has joined #opendev | 00:03 | |
| fungi | ianw: that sounds like it could we what we saw on review01 | 00:45 |
|---|---|---|
| ianw | fungi: yeah. i'm afraid i think bup is a dead end, i mean we knew that, but python3 just seems too far off | 00:45 |
| ianw | i'm investigating rdiff-backup | 00:46 |
| fungi | ooh, i used to use rdiff-backup ages ago | 00:46 |
| ianw | that has a *long* history, which is good | 00:46 |
| fungi | like, i think i remember using it at least 15 years ago | 00:47 |
| fungi | clarkb: you said you had one you really like too, right? | 00:47 |
| fungi | i'll readily admit, my personal backups are just lvm snapshots and rsync to remote hosts | 00:47 |
| ianw | the only trick with it is that it seems python2 and python3 versions of rdiff-backup are incompatible | 00:48 |
| clarkb | I use borgbackup | 00:57 |
| *** shtepanie has quit IRC | 01:00 | |
| *** xiaolin has joined #opendev | 01:01 | |
| ianw | essentially they all look like the same model of doing stuff over ssh | 01:14 |
| *** ysandeep is now known as ysandeep|afk | 01:23 | |
| *** tkajinam has quit IRC | 01:55 | |
| *** tkajinam has joined #opendev | 01:55 | |
| *** ysandeep|afk is now known as ysandeep | 01:59 | |
| *** ysandeep is now known as ysandeep|afk | 02:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 03:43 |
| *** rh-jelabarre has quit IRC | 03:46 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 04:05 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 04:24 |
| openstackgerrit | Merged opendev/system-config master: Added development/rawhide image https://review.opendev.org/741287 | 04:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 04:42 |
| *** cloudnull has quit IRC | 04:52 | |
| *** cloudnull has joined #opendev | 04:53 | |
| *** ysandeep|afk is now known as ysandeep|rover | 05:00 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 05:39 |
| *** marios has joined #opendev | 05:52 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 06:02 |
| openstackgerrit | Merged openstack/project-config master: Switch dragonflow ACL to retired https://review.opendev.org/741317 | 06:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 06:41 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 07:10 |
| *** zbr has quit IRC | 07:16 | |
| *** ryohayakawa has quit IRC | 07:20 | |
| *** ryohayakawa has joined #opendev | 07:20 | |
| *** bolg has joined #opendev | 07:27 | |
| *** dougsz has joined #opendev | 07:29 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 07:31 |
| *** ysandeep|rover is now known as ysandeep|lunch | 07:38 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| *** tosky has joined #opendev | 08:02 | |
| *** zbr has joined #opendev | 08:02 | |
| *** tosky has quit IRC | 08:16 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 08:17 |
| *** tosky has joined #opendev | 08:33 | |
| *** DSpider has joined #opendev | 08:37 | |
| *** zbr has quit IRC | 08:50 | |
| *** ysandeep|lunch is now known as ysandeep|rover | 09:12 | |
| *** dtantsur|afk is now known as dtantsur | 09:15 | |
| *** hiwkby has joined #opendev | 09:24 | |
| *** dougsz has quit IRC | 09:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 09:25 |
| *** hiwkby has quit IRC | 09:31 | |
| *** dougsz has joined #opendev | 09:38 | |
| *** roman_g has joined #opendev | 09:50 | |
| *** bolg has quit IRC | 09:51 | |
| *** bolg has joined #opendev | 09:56 | |
| *** ryohayakawa has quit IRC | 10:34 | |
| *** rh-jelabarre has joined #opendev | 11:59 | |
| *** bolg has quit IRC | 12:21 | |
| *** fressi has joined #opendev | 12:29 | |
| ShadowJonathan | python 3.9 adds a PendingDeprecationWarning to `lib2to3`, and also says it can possibly be removed in the future | 13:04 |
| fungi | a quick skim of http://codesearch.openstack.org/?q=2to3 indicates basically nobody here is using the 2to3 module | 13:07 |
| fungi | for that matter pretty much all projects have already gone python3-only | 13:08 |
| fungi | and back when compatible code was still a goal most of them were relying on the "six" module anyway | 13:09 |
| fungi | at the moment the bigger issues i'm worried about with deprecations are in pbr, and will hopefully be fixed with https://review.opendev.org/735433 and https://review.opendev.org/735443 or something like those | 13:14 |
| ShadowJonathan | ah yeah, still pointing out, and i was mainly saying that as lib2to3 could be a deep dependency for other conversion libraries | 13:19 |
| *** manfly000 has joined #opendev | 13:24 | |
| fungi | sure. by now i hope everyone's basically done converting and just drop conversion libs | 13:28 |
| *** manfly000 has quit IRC | 14:23 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Switch reprepro mirroring from Puppet to Ansible https://review.opendev.org/735406 | 14:43 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Clean up old mirror-update puppetry https://review.opendev.org/741469 | 14:43 |
| clarkb | corvus: mordred I think the ze servers are still in emergency right? we may want to do the container conversion first then switch to zk tls? Or bundle them together explicitly | 14:43 |
| clarkb | otherwise we'll end up with unupdated ze's and be confused I think | 14:44 |
| corvus | oh :( | 14:45 |
| clarkb | that was a morning shower light bulb | 14:46 |
| *** tkajinam has quit IRC | 14:47 | |
| clarkb | we could also convert ze01 back to non container, update them all to tls then do the container conversion afterwards | 14:48 |
| *** zbr|ruck has quit IRC | 14:55 | |
| *** SotK has quit IRC | 14:55 | |
| *** zbr|ruck has joined #opendev | 14:57 | |
| *** SotK has joined #opendev | 14:57 | |
| *** ysandeep|rover is now known as ysandeep|away | 15:02 | |
| corvus | i can start on the nodepool side; mordred was going to work on ze's starting today i think. i think it would be fine to restart them into zk-tls and containers at the same time | 15:29 |
| *** mlavalle has joined #opendev | 15:30 | |
| *** marios has quit IRC | 15:31 | |
| *** fressi has quit IRC | 15:59 | |
| clarkb | cl566n over in #openstack-unregistered reports https://review.opendev.org/#/c/741333/3 is not running zuul jobs and I can confirm that seems to be the case. Grepping in the zuul scheduelr debug logs for that change number then the associated event ids isn't helping much. It seems we got a ref-updated event when the commit message was updated which we decided we shouldn't do anything with | 16:07 |
| clarkb | the recheck seems to have been ignored completely? | 16:08 |
| clarkb | anyone else have a moment to look at that? I'm going to switch to getting irc reg/identification sorted so we can use a proper channel | 16:08 |
| corvus | sure | 16:08 |
| clarkb | corvus: the parent change also seems to have been ignored | 16:08 |
| clarkb | so maybe something to do with it instead? | 16:08 |
| clarkb | I did confirm that the repo seems to have jobs configured as well: https://opendev.org/x/ranger-agent/src/branch/master/.zuul.yaml#L1-L13 | 16:10 |
| corvus | clarkb: the parent is 740836? that seems to have run zuul jobs | 16:10 |
| corvus | oh, but not enqueued? | 16:11 |
| clarkb | corvus: yes, but about half an hour ago it was approved and should've gone into the gate which it seems to have not done | 16:11 |
| corvus | gotcha | 16:11 |
| corvus | zuul currently has an event backlog | 16:11 |
| corvus | i think it's doing a reconfig | 16:12 |
| corvus | which it just finished | 16:12 |
| corvus | still nothing on those changes though | 16:13 |
| fungi | scheduler cpu utilization and memory pressure look perfectly fine/normal | 16:13 |
| clarkb | is ref-updated the normal event for a new patchset? | 16:14 |
| corvus | no | 16:14 |
| clarkb | that seems to be the event we processed for that change roughly when the commit message was updated | 16:14 |
| clarkb | hrm I wonder if we missed the patchset created event or maybe it doesn't emit one if you edit commit message in the browser or something? | 16:15 |
| clarkb | that doesn't explain why recheck seems to be a noop though | 16:15 |
| corvus | there is a problem with 740823 | 16:15 |
| corvus | https://review.opendev.org/#/c/740823 | 16:15 |
| fungi | i'm getting a permission denied error from gerrit for one of the associated changes too | 16:16 |
| corvus | (found in an traceback by doing a context grep) | 16:16 |
| clarkb | fungi: me too | 16:16 |
| clarkb | that seem to be a child of 740836 but on a separate tree from 741333 | 16:17 |
| clarkb | but I guess if zuul tries to process 740836 via 741333 and then hits the error from 740823 that could short circuit the whole thing? | 16:17 |
| fungi | yeah, the one with [WIP] prefixing the commit message subject line | 16:18 |
| fungi | 740836 | 16:18 |
| corvus | fungi: 740823 is the error | 16:18 |
| corvus | 836 works fine | 16:18 |
| fungi | oh, indeed, gerrit wasn't updating the url when i clicked on it | 16:19 |
| clarkb | ps9 loads | 16:19 |
| clarkb | https://review.opendev.org/#/c/740823/9 | 16:19 |
| clarkb | hitting ps10 and ps11 doesn't seem to produce an error in the gerrit error log | 16:20 |
| clarkb | looks like those patchsets were being published with the web editor | 16:20 |
| clarkb | maybe ps10 and ps11 were generated in a corrupt manner due to a bug there? | 16:20 |
| fungi | i'm looking in the db | 16:20 |
| clarkb | "those patchsets" the recent ones that we can actually load and render info about | 16:21 |
| clarkb | there isn't anything about ps10 or ps11 | 16:21 |
| clarkb | oh wait I'm wrong | 16:21 |
| fungi | 10 and 11 were both created 2020-07-15 19:41:15 with the exact same commit id | 16:21 |
| clarkb | Patch Set 10: Published edit on patch set 9. | 16:21 |
| clarkb | Patch Set 11: Published edit on patch set 9. | 16:21 |
| clarkb | ya so I'm thinking bug in the web editor created twice as many patchsets as we wanted and that made gerrit sad | 16:22 |
| clarkb | to fix this maybe we can push a ps12 using ps9? | 16:22 |
| fungi | in theory, that would probably work | 16:22 |
| corvus | ++ | 16:22 |
| clarkb | ok I'll try to repush ps9 now | 16:22 |
| corvus | if that doesn't work, maybe abandon the change | 16:23 |
| clarkb | heh no new changes /me makes a new change | 16:24 |
| clarkb | change loads in the web ui now | 16:26 |
| corvus | 823 and 836 are both in check | 16:27 |
| clarkb | I rechecked 741333 but don't see it yet | 16:27 |
| corvus | it's there now | 16:27 |
| clarkb | cool I'm updating in -unregistered now | 16:28 |
| *** sshnaidm is now known as sshnaidm|afk | 16:28 | |
| *** dougsz has quit IRC | 16:35 | |
| *** dtantsur is now known as dtantsur|afk | 16:40 | |
| *** ShadowJonathan has quit IRC | 17:06 | |
| *** ShadowJonathan has joined #opendev | 17:06 | |
| *** qchris has quit IRC | 18:09 | |
| *** qchris has joined #opendev | 18:22 | |
| clarkb | fungi: left some thoughts on the identity broker spec | 18:23 |
| fungi | thanks!!! | 18:23 |
| *** dougsz has joined #opendev | 18:24 | |
| *** dougsz has quit IRC | 18:33 | |
| *** dougsz has joined #opendev | 18:33 | |
| *** dougsz has quit IRC | 18:44 | |
| *** roman_g has quit IRC | 18:49 | |
| *** roman_g has joined #opendev | 18:49 | |
| *** roman_g has quit IRC | 18:49 | |
| *** roman_g has joined #opendev | 18:50 | |
| corvus | oh we didn't actually merge the tls zk change; i've approved it now, and when it goes out, i'll restart the nodepool servers | 18:52 |
| *** roman_g has quit IRC | 18:54 | |
| fungi | i've only been half-around, but what's the plan with the executors? | 18:56 |
| fungi | containerize and roll that into the tls switch for them? | 18:57 |
| clarkb | fungi: yup I think roughly if we merge corvus' change then we can stop ze02, run ansible against it which will both apply the cert stuff and switch it to a container | 18:58 |
| clarkb | ze01 is similar process but it will just be the cert stuff and wewill need to restart it | 18:58 |
| clarkb | I wonder if mordreds internets are less available than expected today? but I can help with that after lunch | 18:59 |
| clarkb | if anyone has time for https://review.opendev.org/#/q/status:open+topic:opendev-git-branches I think we can land the system-config and gerritlib changes pretty safely (we've got testing yay) | 19:06 |
| clarkb | that will check off more of the todo items for doing different git branches in gerrit | 19:06 |
| clarkb | and related to that is https://etherpad.opendev.org/p/E6m-M-3fTLwse2RkrDQL which is an announcement for wanting to deprecate and remove our /p/ gerrit git repo mirror | 19:06 |
| clarkb | there are a couple reasons for the /p/ thing one is upstream gerrit is not using that anymore anyway and the other is its one less place to manage git branches | 19:07 |
| clarkb | I'm going to find lunch and will be bakc in about an hour to help with zk tls things | 19:08 |
| fungi | i thought the problem with /p is that it's now used for something completely different in later gerrit releases | 19:09 |
| fungi | and so we have to stop using it for what we're using it for or it will shadow that | 19:09 |
| clarkb | I wasnt sureif it is being repurposed or we've been warned it may be | 19:10 |
| clarkb | mordred: ^ you probably know | 19:11 |
| fungi | i think paladox was the first to mention it to me | 19:11 |
| paladox | yeh, PolyGerrit takes it over for project dashboards | 19:11 |
| fungi | aha! project dashboards, i couldn't remember. thanks paladox! | 19:12 |
| clarkb | cool so itsproperly repurposed. paladox do you know which releasemakes that switch? I can update my message to be specific | 19:12 |
| paladox | https://github.com/wikimedia/puppet/blob/production/modules/gerrit/templates/apache.erb#L118 | 19:12 |
| paladox | clarkb 2.16 | 19:12 |
| fungi | yeah, so we're (probably) going to have to break anyone still cloning from there when we upgrade | 19:13 |
| clarkb | thanks! | 19:13 |
| paladox | fungi we use a rewrite | 19:13 |
| paladox | which sorts that out | 19:13 |
| clarkb | fungi: yup and its advantageous for us to break them earlier anyway | 19:13 |
| clarkb | because maintaining that extra mirror makes dealing with branches harder | 19:13 |
| clarkb | apache logs show its mostly third party ci systems and not a ton of them anyeay | 19:14 |
| fungi | paladox: yeah, you had the luxury that folks weren't cloning from /p but rather from /r/p which was being transparently rewritten to /p i guess? | 19:15 |
| clarkb | I'll update my email to better reflect the situation | 19:15 |
| paladox | fungi i mean you can do /p/ -> / | 19:15 |
| fungi | but that will break project dashboards, right? | 19:16 |
| paladox | no | 19:16 |
| paladox | RewriteRule ^/p/(.+)/info/(.+)$ https://<%= @host %>/$1/info/$2 [L,R=301,NE] | 19:16 |
| paladox | we don't explicitly redirect /p/* we redirect /p/*/info/* | 19:16 |
| clarkb | if wehad significant users I would try that | 19:17 |
| paladox | (which is what git uses) | 19:17 |
| clarkb | but ainceit seems to be minor getting people away from non standard paths would be good | 19:17 |
| fungi | i didn't realize git requests always include /info/ | 19:18 |
| clarkb | also if we drop the redirect we have now it will serve it from gerrit directly I think | 19:19 |
| clarkb | then when we upgrade it will break people. | 19:19 |
| clarkb | So thats an option if we want to not break people now but also stop mirroring with a warning its going away in the future | 19:19 |
| fungi | paladox: this is how we've been mapping those git requests so far: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gerrit/templates/gerrit.vhost.j2#L66-L71 | 19:21 |
| paladox | oh | 19:22 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix GCS log upload https://review.opendev.org/741528 | 19:32 |
| openstackgerrit | Merged opendev/system-config master: Revert "Revert "Add Zookeeper TLS support"" https://review.opendev.org/741335 | 19:43 |
| fungi | now we have to wait for deploy to complete for that ^ i guess | 19:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix GCS log upload https://review.opendev.org/741528 | 19:49 |
| clarkb | fungi: ya and then we can restart everything but executors | 19:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add a test that exercises the GCS Credentials class https://review.opendev.org/741535 | 19:54 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove default tox_envlist: venv https://review.opendev.org/726830 | 20:00 |
| *** shtepanie has joined #opendev | 20:01 | |
| clarkb | I don't see anything in deploy, and nl01 seems to have the new zk config | 20:08 |
| clarkb | I think that means we could restart services now if we want? | 20:08 |
| clarkb | let me know if I can help /me edits email draft | 20:09 |
| clarkb | fungi: https://etherpad.opendev.org/p/E6m-M-3fTLwse2RkrDQL has minor edits now to reflect the info paladox provided. What do you think about explicitly disabling it at the end of the month? I think that way we can stop maintaining the mirror and shake out people who would break early. An alternative is to stop redirecting to our mirror and have gerrit serve it. That way we don't have to bother with the | 20:11 |
| clarkb | mirror but people will break post upgrade. | 20:11 |
| fungi | i'm just about done prepping dinner, so should be able to read through it shortly | 20:12 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove default tox_envlist: venv https://review.opendev.org/726830 | 20:17 |
| corvus | clarkb: i'll start with restarting nb01; slightly less disruptive | 20:19 |
| clarkb | ++ | 20:19 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add a test that exercises the GCS Credentials class https://review.opendev.org/741535 | 20:23 |
| corvus | i restarted nb01; seems happy | 20:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add a test that exercises the GCS Credentials class https://review.opendev.org/741535 | 20:30 |
| clarkb | corvus: it was the laucnhers in particular that had trouble last time right? | 20:37 |
| clarkb | I guess they will be good canaries for the fix | 20:37 |
| corvus | clarkb: yes i think so; i don't think the builders walk the really big part of the tree | 20:40 |
| corvus | clarkb: i'm going to proceed with the rest of the builders now | 20:40 |
| corvus | clarkb: if you want to start on launchers, i think that'd be fine, otherwise i will when i finish builders | 20:40 |
| clarkb | I'll start with the launchers | 20:41 |
| clarkb | I'll be sure to do a pull too to get the new image with the new kazoo | 20:41 |
| corvus | that's not automatic? | 20:42 |
| corvus | (i mean, don't we run pulls in ansible or something?) | 20:42 |
| clarkb | we run pulls hourly | 20:42 |
| clarkb | which now that I think about it should be fine | 20:42 |
| clarkb | and ya it was a nop | 20:42 |
| corvus | cool; nb02 says 23 hours ago, that sounds right | 20:42 |
| clarkb | nl01 is restarted | 20:43 |
| clarkb | I'll watch it for a bit to ensure the read errors don't return before doing 02-04 | 20:43 |
| clarkb | it is happy so far, cleaned up building nodes on start and is building new nodes now | 20:44 |
| clarkb | corvus: 2281 is the new port right? | 20:46 |
| clarkb | (I'm just double checking with netstat that it is using the port we expect) | 20:46 |
| corvus | yep | 20:46 |
| clarkb | no exceptions yet on nl01, I'll give it another minute then proceed | 20:47 |
| corvus | i finished restarting all the builders | 20:47 |
| clarkb | 02 has been restarted now too | 20:48 |
| clarkb | and now 03 and 04 are done | 20:49 |
| clarkb | we have launch errors with limestone but those seem to be in nova apis not zk | 20:50 |
| clarkb | openstack.exceptions.SDKException: Error in creating the server (no further information available) | 20:50 |
| clarkb | my favorite error | 20:50 |
| clarkb | and that is the only exception I'm seeing across the 4 launchers | 20:52 |
| fungi | okay, sorry, was distracted by awesome gumbo, but back and ready to help now | 20:58 |
| clarkb | I think nodepool is done now and my eyeballs doing pattern matchign against scrolled text haven't found anything concerning | 21:00 |
| clarkb | there is a new launcher exception related to failures to delete insteances in ovh | 21:00 |
| clarkb | nothing zk related that I can see | 21:00 |
| fungi | so assuming we're cool with nodepool, zuul mergers next? | 21:00 |
| clarkb | and/or scheduler | 21:01 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add a test that exercises the GCS Credentials class https://review.opendev.org/741535 | 21:08 |
| corvus | yeah, the scheduler is the only one that's actually going to use the connection at this point | 21:10 |
| fungi | er, actually, do the mergers connect to zk currently? | 21:10 |
| corvus | they'll all connect, but just sit idle | 21:10 |
| fungi | yeah, just what i was starting to realize, thanks | 21:10 |
| fungi | they're only acting on signals from gearman | 21:10 |
| corvus | i think we should go ahead and restart the mergers to exercise this first in the least disruptive way | 21:11 |
| corvus | then we can decide how we want to do the rest (rotate the executors into containers, then do a scheduler restart at the end? big bang all at once?) | 21:11 |
| corvus | i'll go ahead and do zm01 now | 21:11 |
| fungi | sounds good, will keep an eye on its logs | 21:12 |
| clarkb | wfm | 21:12 |
| corvus | i'm refreshing my memory on whether there's a least-disruptive way to stop a merger | 21:13 |
| corvus | we should really log job completion in the mergers :/ | 21:15 |
| corvus | oh you know what? i don't think the other nodes actually connect to zk yet | 21:16 |
| corvus | i have restarted zm01, and it did not complain about the config file. that's probably actually all we need to do; i don't think we need to worry about restarting the executors for zk; we can just restart them for containerization with confidence that they aren't going to choke on the new config. | 21:18 |
| corvus | so really it's just the scheduler that's the last remaining important zk bit | 21:18 |
| fungi | makes sense | 21:18 |
| clarkb | sounds good | 21:31 |
| clarkb | mordred: if you're around today I'm able to help with executors for another couple of hours at least so let me know | 21:31 |
| fungi | i probably can too, dinner is done and kitchen cleaned up, so mostly just catching up on stuff now and reading through the /p removal announcement | 21:47 |
| clarkb | unrelated, I've been looking at openstack memory usage because it came up elsewhere I couldn't help myself. I've discovered that its possible bionic era journald has a memory leak | 21:49 |
| clarkb | I've got tests running on focal that I should be able to use to compare | 21:49 |
| clarkb | I doubt it will impact us much but how fun is that, the system logger may use many memories :) | 21:50 |
| fungi | clarkb: proposed announcement lgtm. from an implementation detail standpoint i suppose we'll just tweak apache to 404 on /p/.* requests? | 21:50 |
| clarkb | fungi: yup that was what I was thinking or 403 forbidden? | 21:50 |
| fungi | wfm | 21:50 |
| clarkb | since technically on older gerrit its still valid | 21:50 |
| clarkb | forbidden makes that distinction a bit more clear | 21:50 |
| fungi | i agree with that logic | 21:51 |
| clarkb | cool I think tomorrow will be email day then I'll try to get that out as well as followup on advisory board | 21:51 |
| fungi | awesome | 21:54 |
| *** DSpider has quit IRC | 22:17 | |
| *** xiaolin has quit IRC | 22:24 | |
| *** xiaolin has joined #opendev | 22:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 22:25 |
| clarkb | ianw: I see you've turned off encrpytion (which is encryption at rest not over the wire) to avoid managing that additional secret? Also, have you set it up to do append only backups? I know that corvus is a fan of that and borg does support it. | 22:26 |
| clarkb | ianw: the only other thing I was going to call out before review too closely is that there were upgrade concerns if your borg was too old. I think focal would be well new enough but xenial may not be | 22:27 |
| clarkb | (not sure if you plan to conver everything) | 22:27 |
| ianw | clarkb: yep, so yeah i did some research and borg seemed probably the best way | 22:27 |
| clarkb | but overall I've been really happy with it locally and am happy to review/help if I can. I rely on the encrypted at rest feature as I backup to my brothers house offsite and love that you can fuse mount backups for easy recovery and browsing. I don't do append only though | 22:28 |
| ianw | in terms of versions; yes it seems it is very wise to a) pin the version you use and b) use the same version on both sides, which is why i've taken a pip install approach | 22:28 |
| ianw | i am not really convinced a container is a good idea for this, although i guess possible. a) because it's whole purpose is to walk the base file system and b) you want as little between you and a successful backup as possible | 22:29 |
| clarkb | ya I don't run it in containers locally either | 22:30 |
| ianw | yeah for encryption, my thinking was that we explicitly trust the remote end, i mean as much as we trust the cloud providers to run the unencrypted vm's anyway, so an extra key seems like another hurdle to restores and another thing to lose | 22:32 |
| ianw | but yeah, i'll write up a proper changelog :) i want it to run a test backup in the gate | 22:33 |
| clarkb | as a followon to the journald thing it seems that focal isn't much better which implies something else is going on. I suppose its possible we log so much all at once during openstack testing that we simply cause journald to bloat in memory use | 22:37 |
| clarkb | and if disk io were quicker we'd keep that to a lower level of memory use? | 22:37 |
| clarkb | that makes me more comfortable about our prod servers | 22:37 |
| ianw | fungi: 3 pretty straight forward ones that update some system-config testing bits if you have time : https://review.opendev.org/#/c/740609/6 https://review.opendev.org/#/c/740608/3 https://review.opendev.org/#/c/740605/5 | 22:40 |
| ianw | clarkb: where was the journald thing raised? | 22:41 |
| clarkb | ianw: I was just poking at it because a discussion about openstack memory use came up in the tc channel | 22:42 |
| corvus | the append-only part helps if we need forensics after a security incident; i'd still like that if we can, but i'll concede it's a secondary goal | 22:42 |
| ianw | ahh, yeah a couple of people seem to have popped up lately asking about memory | 22:42 |
| clarkb | and looking at jobs journald uses a fair bit of memory (more than I would have expected) and from there found a bug fixed early 2019 to correct a memory leak | 22:42 |
| clarkb | so wanted to compare bionic to focal results as the memory leak fix should be in focal but possibly not bionic | 22:43 |
| corvus | ianw: are you looking at doing append-only? | 22:43 |
| clarkb | corvus: it should be doable to have append only backups, but we'd need to test that as I have no existing experience with append only and bord | 22:43 |
| clarkb | *borg | 22:43 |
| ianw | corvus: right now i'm just looking at it doing anything at all -- but it has many flags and good documentation about things to tweak. i'll look at append only before final review | 22:44 |
| ianw | there's other flags about having --readonly / and stuff that seem good as well | 22:45 |
| corvus | cool :) | 22:45 |
| fungi | ianw: 739412 needs a rebase for an updated parent | 22:50 |
| ianw | fungi: oh i should have cut that out ... i guess we want to go with the dns records | 22:52 |
| fungi | ahh, yep sshfp rrs worked great in our test | 22:52 |
| *** mlavalle has quit IRC | 22:53 | |
| *** tkajinam has joined #opendev | 23:02 | |
| *** tosky has quit IRC | 23:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] borg backups https://review.opendev.org/741366 | 23:16 |
| *** zbr|ruck has quit IRC | 23:16 | |
| *** SotK has quit IRC | 23:16 | |
| clarkb | ianw: if you get a chance before you call it a week reviews on https://review.opendev.org/#/q/topic:opendev-git-branches would be helpful (I'm trying to push on that a step at a time as I'm able) | 23:21 |
| clarkb | seems we get a fairly consistent set of question around it for which https://etherpad.opendev.org/p/opendev-git-branches is shareable | 23:21 |
| *** zbr|ruck has joined #opendev | 23:23 | |
| *** SotK has joined #opendev | 23:23 | |
| ianw | fungi: i'm not sure i have the patience to get to a situation of dnssec working between my local unifi environment and my work vm attached to the redhat vpn | 23:23 |
| ianw | i think dnssec-trigger is required to be involved | 23:24 |
| ianw | clarkb: not really related but /org/{org}/repos says it's deprecated in the gitea api overview | 23:30 |
| ianw | oh i guess it's become "orgS" | 23:31 |
| clarkb | oh neat we can update that too | 23:32 |
| ianw | https://try.gitea.io/api/swagger#/organization/createOrgRepo | 23:32 |
| clarkb | its fairly well tested too if you wantto push that change | 23:32 |
| ianw | clarkb: for 741277 shouldn't we see the branch being set to "main" for test-repo-2 ~ https://zuul.opendev.org/t/openstack/build/9dfc59a828324b2c9e57ae630e498375/log/job-output.txt#741 ? | 23:39 |
| fungi | ianw: funny you should mention that, i just a few moments ago upgraded glibc on my workstation and now it needs an extra option set in resolv.conf to tell it not to clear ad flags | 23:41 |
| clarkb | ianw: it should only ve in the jeepyb change as that updates the call to set it https://review.opendev.org/#/c/741279/ | 23:41 |
| clarkb | ianw seems I did miss something https://zuul.opendev.org/t/openstack/build/28a42172d75847cebfca4fde77bcaae4/log/job-output.txt#747 | 23:42 |
| clarkb | main is used later though | 23:43 |
| clarkb | I'll look into ehy the gitreview push wasmissed. I grepped for master but maybe its another file | 23:44 |
| openstackgerrit | Merged opendev/system-config master: Copy generated inventory to bridge logs https://review.opendev.org/740605 | 23:45 |
| openstackgerrit | Merged opendev/system-config master: testinfra: silence yaml.load() warnings https://review.opendev.org/740608 | 23:45 |
| openstackgerrit | Merged opendev/system-config master: Fix junit error, add HTML report https://review.opendev.org/740609 | 23:45 |
| fungi | "Starting with glibc 2.31, the DNS stub resolver does not blindly trust the AD (authenticated data) flag, indicating a DNSSEC validation. By default the name servers and the network path to them are treated as untrusted. In this mode, the AD flag is not set in queries, and it is automatically cleared in responses, indicating a lack of DNSSEC validation." | 23:47 |
| fungi | so apparently i now have to include "options trust-ad" in my /etc/resolv.conf | 23:47 |
| clarkb | yup its in the utils file and I must've grepped poorly | 23:48 |
| clarkb | can `git push origin HEAD:refs/heads/master` be safely rewritten as `git push origin HEAD:remotes/origin/HEAD` ? | 23:52 |
| fungi | huh, glibc uses gerrit now? https://gnutoolchain-gerrit.osci.io/r/c/glibc/+/461 | 23:53 |
| clarkb | looks like if we do a remote update first that may work | 23:55 |
| clarkb | oh hrm this git init is used as our permanent cache too | 23:56 |
| clarkb | so we'd need to init it with the right default branch there as well anyway | 23:57 |
| clarkb | I was trying to simplify and not need to know what the default branch is going to be but I Think we hvae to know | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!