| clarkb | the osf/refstack -> openinfra/refstack redirect works. Logos are not all happy here: https://gitea09.opendev.org:3081/explore/organizations and that is beacuse those orgs don't exist in projects.yaml anymore. | 00:01 |
|---|---|---|
| clarkb | I was worried about that sort of thing with the transplanted database, but I suspect fixing that is simpler than doing a fresh db update to address the redirects | 00:01 |
| clarkb | I think I can manually copy the files in /var/gitea/data/gitea/avatars to address that. Trying now | 00:02 |
| fungi | yeah, maybe we should document a step of rsyncing those and/or include them in backups | 00:04 |
| clarkb | ++ | 00:04 |
| clarkb | I'm going to finish cleaning those up and then push up changes for the next steps | 00:04 |
| clarkb | and really we could have a step that creates the old orgs | 00:04 |
| clarkb | as step 0 in sorting out redirect creation directly rather than as a db transplant | 00:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gitea09 as a Gerrit replication target https://review.opendev.org/c/opendev/system-config/+/874174 | 00:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gitea09 to the gitea load balancer https://review.opendev.org/c/opendev/system-config/+/874175 | 00:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Convert gitea99 test node to Jammy https://review.opendev.org/c/opendev/system-config/+/874176 | 00:07 |
| clarkb | I'm going to WIP those first two changes. The first one because I'd like to land it tomorrow when I can monitor the potentially very long sync from gerrit (also I need to update the host vars for it). And the second because we need to be replicating before it is part of the lb pool | 00:08 |
| clarkb | infra-root ^ if you want to poke around gitea09 both on the host and via the web (https://gitea09.opendev.org:3081) that would be appreciated just to look out for any oddities from the db transplant | 00:09 |
| clarkb | Then assuming you're happy with the results please review the above changes which will get it into production | 00:09 |
| clarkb | fungi: fwiw I believe gitea01 does have backups which should include the non git portions of gitea | 00:11 |
| clarkb | but we can/should doulbe check that | 00:11 |
| *** soniya29 is now known as soniya29|ruck | 03:36 | |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 07:57 | |
| *** marios is now known as marios|ruck | 08:23 | |
| *** jpena|off is now known as jpena | 08:36 | |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 08:54 | |
| mnasiadka | frickler: is there a guide how to encrypt secrets? seems we need to update the docker hub secret in Kolla | 09:44 |
| frickler | mnasiadka: this is what is linked from what I posted yesterday https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#using-secrets | 10:18 |
| mnasiadka | ah, thanks, missed that | 10:34 |
| mnasiadka | was mangling with zuul-client | 10:34 |
| fungi | mnasiadka: also the zuu side documentation is at https://zuul-ci.org/docs/zuul/latest/project-config.html#encryption and https://zuul-ci.org/docs/zuul-client/commands.html#encrypt | 12:27 |
| fungi | we need to update that section of the infra-manual to refer to zuul-client | 12:27 |
| fungi | it hasn't been updated since zuul-client existed | 12:28 |
| fungi | but usage is similar to the old encrypt_secret.py script | 12:28 |
| *** artom_ is now known as artom | 12:31 | |
| frickler | interesting issue arising from the update: https://review.opendev.org/c/openstack/kolla/+/874226 | 12:38 |
| frickler | I guess nobody else has needed to change secrets that are defined in branched repos yeet? | 12:38 |
| frickler | s/ee/e/ | 12:38 |
| frickler | or similar | 12:39 |
| frickler | do we have to rename secrets every time we branch? or drop them from stable branches completely? | 12:40 |
| fungi | frickler: i thought zuul was supposed to ignore secrets if they're defined on other branches, but maybe that has changed or regressed | 12:42 |
| opendevreview | Jeremy Stanley proposed opendev/infra-manual master: Update subsection on Using Secrets https://review.opendev.org/c/opendev/infra-manual/+/874244 | 13:03 |
| fungi | frickler: ^ as discussed | 13:03 |
| frickler | fungi: thx, I didn't realize that the old script reference was also obsolete. also I need to test the --infile option, haven't used that one before | 13:10 |
| fungi | the prior example used --infile too, i didn't change that part (though i did test to make sure it works as written there) | 13:10 |
| opendevreview | Jeremy Stanley proposed opendev/infra-manual master: Update subsection on Using Secrets https://review.opendev.org/c/opendev/infra-manual/+/874244 | 13:17 |
| *** dasm|off is now known as dasm | 14:11 | |
| clarkb | I've updated host vars for https://review.opendev.org/c/opendev/system-config/+/874174/ and I think that can be approved as soon as we are comfortable. Note the comment around reloading replication configs (won't be automatic but I think we can use the `gerrit plugin reload replication` command to avoid a gerrit restart | 16:27 |
| clarkb | I even managed to make sure I was talking to the correct ssh port | 16:27 |
| fungi | thanks! | 16:34 |
| opendevreview | Merged opendev/infra-manual master: Update subsection on Using Secrets https://review.opendev.org/c/opendev/infra-manual/+/874244 | 16:40 |
| *** jpena is now known as jpena|off | 17:01 | |
| clarkb | fungi: re https://review.opendev.org/c/opendev/system-config/+/874174/ should we continue to proceed with that today? I feel like its on the borderline between send it and be cautious during openstack feature freeze. In theory landing it won't make any impacts until we reload the replication plugin or restart gerrit | 17:50 |
| clarkb | but I'm also happy to try and find time for that next week between meetings if wewant feature freeze to burn itself out first | 17:50 |
| clarkb | I've got some zuul reviews I should do now anyway | 17:50 |
| fungi | clarkb: today seems like a fine time to push forward there | 17:58 |
| clarkb | ok want to approve the change? | 17:59 |
| fungi | done | 17:59 |
| clarkb | thanks! | 17:59 |
| opendevreview | Merged opendev/system-config master: Add gitea09 as a Gerrit replication target https://review.opendev.org/c/opendev/system-config/+/874174 | 18:09 |
| clarkb | ok that didn't actually enqueue infra-prod-review or whatever the job is | 18:10 |
| * clarkb writes a followup change to fix that that should trigger he job | 18:10 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Trigger infra-prod-service-review when review02 hostvars update https://review.opendev.org/c/opendev/system-config/+/874290 | 18:12 |
| clarkb | fungi: ^ | 18:12 |
| fungi | looking | 18:13 |
| fungi | unrelated, clarkb do you have an opinion on https://review.opendev.org/856828 ? | 18:13 |
| clarkb | fungi: approved. | 18:15 |
| fungi | thanks! i'll take care of the cleanup once it rolls out | 18:15 |
| opendevreview | Merged opendev/system-config master: Redirect openstack-infra specs to opendev docs https://review.opendev.org/c/opendev/system-config/+/856828 | 18:52 |
| opendevreview | Merged opendev/system-config master: Trigger infra-prod-service-review when review02 hostvars update https://review.opendev.org/c/opendev/system-config/+/874290 | 18:52 |
| clarkb | infra-prod-bootstrap-bridge is running in opendev-prod-hourly at the same time as infra-prod-letsencrypt in deploy. I don't think this is an issue but it is unexpected | 19:03 |
| clarkb | infra-prod-bootstrap-bridge doesn't share the semaphore I think because what it does is actually pretty minimal so this should be fine | 19:05 |
| clarkb | I can't tell if my fix job change is actually going to enqueue the job i wnt though :( might need another followup to force it | 19:14 |
| clarkb | ah we don't see the jobs until they are done waiting for the semaphore? | 19:28 |
| * clarkb attempts patience | 19:28 | |
| fungi | yep | 19:48 |
| fungi | i think my specs index change is blocking it | 19:49 |
| clarkb | ya no job ran | 20:08 |
| clarkb | oh! | 20:09 |
| clarkb | the file doesn't exist because we store everything in secret vars. | 20:09 |
| clarkb | So I think some of these public things like host keys could be moved into the public file? Since we use review99 in CI? | 20:10 |
| clarkb | Lets sort that out another time though. /me figures out how to trigger this job | 20:10 |
| fungi | oh, yeah | 20:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit role readme https://review.opendev.org/c/opendev/system-config/+/874303 | 20:14 |
| clarkb | thats a bit of an easy mode so I don't have to trigger the job by hand | 20:15 |
| clarkb | I need to go figure out lunch now too | 20:15 |
| opendevreview | Merged openstack/project-config master: Step 3: Remove project from infra System - Retire xstatic-font-awesome https://review.opendev.org/c/openstack/project-config/+/872835 | 20:43 |
| clarkb | fungi: just a heads up that I'll probably be doing a school run when the change finally lands. But that should be fine as it is typically a few more minutes until the job runs | 21:55 |
| clarkb | fungi: once the job has run I think the next steps are to `gerrit plugin reload replication` and then trigger replication for a small repo to see if all is well. Then trigger it for everything against gitea09 | 21:56 |
| fungi | i'll be around, but probably eating dinner soon | 21:58 |
| opendevreview | Merged opendev/system-config master: Update Gerrit role readme https://review.opendev.org/c/opendev/system-config/+/874303 | 22:36 |
| fungi | deploy completed at 22:42:32 utc | 22:44 |
| Clark[m] | Cool I'm on my way home now | 22:45 |
| clarkb | fungi: ok I'll reload the replication plugin now and ask bindep to replicate | 22:57 |
| clarkb | hrm remote plugin administration is disabled os that didn't work | 22:58 |
| clarkb | ok remoate plugin admin defaults to false and we don't override that. I think we need to do a gerrit restart. | 23:00 |
| clarkb | I didn't expect this. Maybe the thing to do is consider a config change that has the replication plugin automatically reload its config. Then restart to pick that up as well as the gitea09 addition | 23:01 |
| clarkb | this way we don't need to do a bunch of restarts as we roll new giteas in and old out. /me works on that change | 23:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Enable Gerrit replication autoreload https://review.opendev.org/c/opendev/system-config/+/874340 | 23:07 |
| clarkb | I think that may be our best option. But I'm open to ideas. Not sure I want to do a gerrit restart right now during openstack feature freeze | 23:07 |
| clarkb | re ideas: I think we can enable remote plugin admin including plugin installs in order to manually reload the replication plugin (I think this has the same dropping events problem that auto reloads of the config has though) | 23:09 |
| clarkb | I'd rather take autoreloading with that problem than create potential plugin management issues | 23:10 |
| fungi | clarkb: one of the reasons we have that disabled is that it used to wipe the replication queue on reload. no idea if it still does, but it resulted in incomplete mirrors missing objects/commits | 23:20 |
| clarkb | fungi: yup I made note of that in my change. I think if we make `gerrit plugin replication reload` work it would have the same problem | 23:21 |
| clarkb | so I prefer auto to that to avoid opening the whole can of worms for remote admin of all the plugin things | 23:21 |
| clarkb | If we prefer gerrit restarts we can do that too. Mostly thinking we could do this temporarily and perform a full sync after each config reload or something | 23:22 |
| clarkb | then when giteas are all rotated in/out undo the config | 23:22 |
| clarkb | fungi: if we feel more comfortable with restarts that fine too. Feel free to -1 the change. | 23:25 |
| clarkb | I just didn't want to do a restart until we had considered making it so future restrats aren't necessary | 23:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!