| *** snbuback_ is now known as snbuback | 09:22 | |
| *** TheJulia_ is now known as TheJulia | 10:36 | |
| *** gouthamr_ is now known as gouthamr | 10:39 | |
| clarkb | infra-root if any of you are around today prior to the gerrit upgrade it might be good to look over https://review.opendev.org/c/opendev/system-config/+/813072 as landing that and getting it applied is one of the first items on the upgrade todo list. | 17:45 |
|---|---|---|
| clarkb | I was thinking I could approve it around 19:30 UTC to ensure it was done by the time the window started but realized it doesn't have any reviews yet | 17:45 |
| fungi | oh, good call. i'll go over it now, thanks! | 18:00 |
| fungi | i was planning to be around for the 21:00-23:00 window anyway | 18:01 |
| opendevreview | Jeremy Stanley proposed opendev/infra-specs master: Add a specification for Mailman 3 https://review.opendev.org/c/opendev/infra-specs/+/810990 | 18:27 |
| opendevreview | Jeremy Stanley proposed opendev/infra-specs master: Add backups section to template https://review.opendev.org/c/opendev/infra-specs/+/813301 | 18:31 |
| opendevreview | Jeremy Stanley proposed opendev/infra-specs master: Simplify assignees section of template https://review.opendev.org/c/opendev/infra-specs/+/813303 | 18:33 |
| clarkb | fungi: re your comment: yes, that whole stack of changes works through adding 3.4 images, moving the upgrade testing to 3.3 -> 3.4 and then removes the 3.2 images | 19:26 |
| clarkb | Looks like I've got the +2's now. I'll hit +A a bit closer to 20:00 so that it is ready and in place for when the window begins | 19:27 |
| clarkb | ok I have approved the change that will udpate our docker-compose file to reference the 3.3 image instead of the 3.2 image. This shouldn't affect the running gerrit and we will have to walk through the steps in the etherpad in an hour or so to acutally do the upgrade | 19:56 |
| clarkb | feel free to -W or -2 should something come up that questions any of that while it is in the gate | 19:56 |
| ianw | thanks for gettign that started | 20:35 |
| opendevreview | Merged opendev/system-config master: Upgrade Gerrit to 3.3 https://review.opendev.org/c/opendev/system-config/+/813072 | 20:45 |
| fungi | so we just need it to deploy in the next 15 minutes | 20:46 |
| clarkb | ya I realized we could spend half the window just waiting for that to land and deploy figured getting it done early was a good idea | 20:50 |
| clarkb | oh this is unexpected though it is running manage-projects | 20:51 |
| clarkb | I think it may have pulled the new images for us to run manage-projects | 20:53 |
| clarkb | since we do a docker run not a docker exec | 20:53 |
| clarkb | that should be fine and we can pull as planned to ensure we have all the images we need | 20:53 |
| clarkb | hrm maybe not. the manage-project script still references the 3.2 image | 20:54 |
| ianw | yeah i think to avoid this sort of thing we have to manually pull the latest images on gerrit | 20:55 |
| clarkb | I see what happened. The manage-projects task file pulls the gerrit image using the variable we set to set the version | 20:55 |
| clarkb | but it doesn't update its own script file | 20:55 |
| fungi | i guess that's preferable anyway | 20:55 |
| clarkb | I think the "fix" there is to have it also update its script file. But ya I don't think this did anything we didn't want so I'm happy with it | 20:56 |
| fungi | the pull will be a no-op now, but it didn't run with the new image either | 20:56 |
| clarkb | correct | 20:56 |
| ianw | #status notify Gerrit will be unavailable for maintenance for up to a few hours; see http://lists.opendev.org/pipermail/service-announce/2021-October/000024.html | 20:57 |
| opendevstatus | ianw: unknown command | 20:57 |
| clarkb | ianw: it is notice not notify | 20:57 |
| ianw | #status notice Gerrit will be unavailable for maintenance for up to a few hours; see http://lists.opendev.org/pipermail/service-announce/2021-October/000024.html | 20:58 |
| opendevstatus | ianw: sending notice | 20:58 |
| -opendevstatus- NOTICE: Gerrit will be unavailable for maintenance for up to a few hours; see http://lists.opendev.org/pipermail/service-announce/2021-October/000024.html | 20:58 | |
| fungi | Build succeeded (deploy pipeline). | 21:00 |
| fungi | guess we're ready | 21:00 |
| ianw | cool; https://etherpad.opendev.org/p/gerrit-upgrade-3.3 is what is being worked through | 21:00 |
| clarkb | has review02.opendev.org been put in the emergency file yet? | 21:01 |
| fungi | i can do that now | 21:01 |
| clarkb | the hourly jobs ignore it now, but seems like a good thing anyway | 21:02 |
| fungi | and done | 21:02 |
| clarkb | on item 3.5 I think I convinced myself while checking on that that it isn't really necessary | 21:03 |
| clarkb | the cron jobs run at times that shouldn't conflict | 21:03 |
| fungi | i think i may have collided with someone else's edit of the emergency file | 21:03 |
| clarkb | fungi: wasn't me so probably ianw | 21:03 |
| clarkb | I'll let you two sort out things and I'll stay out of the way :) | 21:03 |
| ianw | fungi: heh sorry yes was doing it :) i've started a screen | 21:03 |
| clarkb | I'm in the screen on review is there one on bridge? | 21:04 |
| fungi | i'm attached to the screen session on review now | 21:04 |
| fungi | looks like you commented out the cronjobs | 21:04 |
| ianw | no i didn't do one on bridge | 21:05 |
| clarkb | ok I don't think we need one, but thought I'd ask | 21:05 |
| clarkb | and ya disabling the crons is fine with me | 21:05 |
| ianw | yep; i think it's good to keep this like a generic upgrade template as much as possible | 21:05 |
| fungi | so we're up to pulling images now | 21:08 |
| fungi | a no-op as expected | 21:08 |
| clarkb | we should make a mental note to prune the images on that server | 21:09 |
| clarkb | maybe in a week when we are happy with things we can do that | 21:09 |
| ianw | ok, all looks sane | 21:10 |
| clarkb | yup and mariadb can take a bit of time to start | 21:11 |
| clarkb | ah cool it logs explicitly when it is ready for connections that is nice | 21:11 |
| fungi | very | 21:12 |
| clarkb | those java warnings are expected | 21:12 |
| clarkb | people have complained about them to the gerrit maintainers | 21:12 |
| fungi | that's a fun warning | 21:13 |
| clarkb | and that looks about right, it is a simple migration to change the name of the service user group | 21:13 |
| fungi | illegal access? /me looks around for the cops | 21:13 |
| clarkb | I suspect that may be what is keeping them from goign to java 17 any time soon | 21:13 |
| clarkb | (java 17 is the next lts) | 21:14 |
| fungi | i can get to the webui | 21:14 |
| fungi | pretty snappy | 21:14 |
| clarkb | the log indicates it is ready too | 21:15 |
| ianw | 🎉 No changes need your attention 🎉 -- party time | 21:15 |
| fungi | heh | 21:15 |
| fungi | i'll drink to that | 21:15 |
| clarkb | heh the emoji are literal too | 21:15 |
| clarkb | (this is the attention set stuff I've talked to corvus about in the past) | 21:16 |
| clarkb | basically gerrit tries to keep track of what comments need addressing by code submitters and reviewrs and then calls things to your attention so taht you can address them | 21:16 |
| clarkb | I think it has a lot of potential but we'll have to see how it goes. | 21:16 |
| ianw | https://review.opendev.org/c/openstack/kolla-ansible/+/813306/ has already uploaded something that is in the gate | 21:16 |
| clarkb | wow that was quick | 21:17 |
| clarkb | one thing we should double check is that recheck comments work properly. I can recheck my prometheus spec change | 21:17 |
| clarkb | (because the way gerrit emits those has changed but we do believe zuul should handle them properly already) | 21:17 |
| ianw | a recheck in https://review.opendev.org/c/opendev/system-config/+/812765 is in the gate | 21:18 |
| clarkb | I rechecekd the spec and it is in the check queue so that lgtm | 21:18 |
| ianw | there's always interesting things to find in the error log, but nothing seems fatal | 21:21 |
| clarkb | ianw: the gc log complaints are something I wish I understood better, but also I guess at this point we could drop the gc logging since we don't have gc problems | 21:21 |
| clarkb | that should make it quiet about those logs file rotations | 21:21 |
| ianw | gerrit.config appears to have acquired quotes | 21:25 |
| clarkb | ianw: fungi: that diff looks pretty clean to me. | 21:25 |
| ianw | - reportBugUrl = https://docs.openstack.org/infra/system-config/project.html #contributing | 21:25 |
| ianw | + reportBugUrl = https://docs.openstack.org/infra/system-config/project.html#contributing | 21:25 |
| clarkb | ianw: other way around the new side is the old config | 21:25 |
| ianw | doesn't look right | 21:25 |
| clarkb | and ya I think that may be the only thing that is possibly broken | 21:25 |
| ianw | i think it may think that is a comment | 21:25 |
| clarkb | ++ | 21:25 |
| clarkb | for the email template things I don't think we should manage them unless we decide we need to change their content and we haven't had to do that so far | 21:26 |
| clarkb | anyway I suspect we can clean this up non urgently | 21:26 |
| clarkb | https://docs.openstack.org/infra/system-config/project.html vs https://docs.openstack.org/infra/system-config/project.html#contributing when you click report a bug is probably close enough for now | 21:26 |
| fungi | seems fine to me, yep | 21:26 |
| clarkb | I config https://docs.openstack.org/infra/system-config/project.html is the link when you hover over report a bug at the buttom of a gerrit page | 21:27 |
| fungi | and also we could stand to replace that url anyway | 21:27 |
| clarkb | fungi: ya its stale as is | 21:27 |
| fungi | anything else we want to check before we undo the emergency file and restore the cronjobs? | 21:29 |
| clarkb | once we are happy with things https://review.opendev.org/c/openstack/project-config/+/813081 is a good change to land. https://review.opendev.org/c/opendev/system-config/+/792047/ too though less urgently | 21:29 |
| clarkb | fungi: I'm not seeing anything. A good test is to acutally land something like 813081, but I think that can happen afterwards | 21:30 |
| fungi | clarkb: 813081 drops system-config-build-image-gerrit-master and doesn't mention why, though i expect that's good cleanup | 21:31 |
| clarkb | fungi: mostly because it was a waste and doesn't work half the time | 21:31 |
| fungi | yeah, that was the reason i expected | 21:32 |
| clarkb | I guess I should've called that out more explicitly in the commit message. I didn't want jeepyb changes being held up by flaky gerrit master builds | 21:32 |
| clarkb | and seemed like a good time to make that edit while I was updating the file | 21:32 |
| clarkb | I have detached from the screen so that it can be clsoed saving the log file. I'll let ianw or fungi clean up the mergency file since you two edited it previously :) | 21:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: Update quotes for 3.3 https://review.opendev.org/c/opendev/system-config/+/813310 | 21:34 |
| clarkb | ianw: I wonder if adding quotes to the bug url will produce the same result | 21:35 |
| clarkb | worht a try | 21:35 |
| ianw | we should be able to inspect the config file collected from the job, i guess? | 21:36 |
| clarkb | ianw: oh g ood point | 21:36 |
| clarkb | yes that should be the case | 21:37 |
| ianw | i also cleaned up a few other things in emergency that have gone now (review-test, review-dev01, eavesdrop01.openstack.org) | 21:37 |
| fungi | i've also detached from the screen session | 21:37 |
| clarkb | ianw: do you want to review and hopefully approve https://review.opendev.org/c/openstack/project-config/+/813081 as a sanity check we can land code? | 21:38 |
| clarkb | (that chagne needs to land to reflect the new state of the system anyway) | 21:39 |
| clarkb | and then we can also double check replication for project-config afterwards | 21:39 |
| clarkb | I don't expect replication trouble fwiw, the plugin hasn't really changed between several gerrit versions | 21:40 |
| ianw | i does tell me it's "my turn to take action" | 21:41 |
| clarkb | (though sometimes being super sure of that is difficult with the way they merge up instead of down) | 21:41 |
| clarkb | ya I think to make it work right we need to be more disciplined about marking things as addressed or not | 21:41 |
| clarkb | it may take some time to get the workflow down | 21:41 |
| ianw | on https://review.opendev.org/c/opendev/system-config/+/813310 you just put a +2, which it seems to take as a comment | 21:42 |
| clarkb | the default for top level comments is that they are "resolved" which I thought would prevent it from kicking back to you | 21:44 |
| clarkb | I guess it assumes that its on you if I've said my end is resovled | 21:45 |
| ianw | when you add your vote, down the bottom there's "bring to the attention of" which seems to autofill ... i'm not sure, at least the owner, and maybe other reviewers | 21:47 |
| clarkb | ya its meant to figure it out based on who is commenting and interacting with the system | 21:48 |
| clarkb | I suspect we'll just have to work with it a bit to udnerstand what it wanst then maybe we can write some guidelines for people to use it effectively | 21:48 |
| ianw | for our workflow, it seems like making positive votes with no comments should default to not updating attention sets | 21:49 |
| clarkb | that seems reasonable. I don't know how tunable this is. But all stuff we can provide as feedback upstream at least | 21:49 |
| clarkb | re the gerrit stack of chagnes https://review.opendev.org/c/opendev/system-config/+/813073/ should also be safe to land, but I think we should hold off on https://review.opendev.org/c/opendev/system-config/+/813074/ until we are reasonable happy with 3.3 and don't want to revert (mid week maybe?) | 21:50 |
| ianw | "If a reviewer replies, then the change owner (and uploader) are added to the attention set" | 21:50 |
| ianw | (https://gerrit-review.googlesource.com/Documentation/user-attention-set.html) | 21:50 |
| clarkb | maybe we treat that as "find a second reviewer now if you got one +2?" but we typically dno't operate that way | 21:51 |
| opendevreview | Merged openstack/project-config master: Update jeepyb to trigger Gerrit 3.3 image builds https://review.opendev.org/c/openstack/project-config/+/813081 | 21:52 |
| clarkb | https://opendev.org/openstack/project-config/commit/c61ad2d4674512084eaf8a80fd15ecacf19179bb ^ replicated | 21:52 |
| clarkb | that all appears happy. ianw anything else you want me to stick around to look at? I'll check back in later this evenign either way, but I think I'm running out of things to do without starting my workweek extra early :) | 21:53 |
| ianw | clarkb: nope, many thanks to yourself and fungi for helping out! | 21:53 |
| fungi | thanks for driving this! | 21:53 |
| fungi | i'll try to be on hand earlyish in my day too in case people spot any problems with 3.3 | 21:54 |
| clarkb | ++ its nice to make progress on stuff like this. Assuming it remains happy I'll plan to do the gitea 1.15.4 upgrade tomorrow too | 21:54 |
| fungi | though i expect it'll mostly just be confusion from the new workflows | 21:54 |
| clarkb | I marked https://review.opendev.org/c/opendev/system-config/+/813074/ WIP too just to avoid it landing early | 21:55 |
| clarkb | fungi: ya I suspect that one thing we'll need to do is add third party CI bots to the Service Users group | 21:56 |
| clarkb | oh and possibly zuul too? | 21:56 |
| clarkb | ianw: fungi: I'd be ok if we want to just go ahead and start doing that. I had previously checked if the Service Users group was used for anything and from what I could tell it wasn't | 21:57 |
| clarkb | (well its old name anyway) | 21:57 |
| ianw | that stops them adding the attention set right? | 21:58 |
| clarkb | Maybe double check that and if we're happy with the double checking we add Zuul and third party CI systems to it to avoid confusion with attention sets. Also worst case we can just disable attention sets. | 21:58 |
| clarkb | ianw: ya, it also allows us to pool them into different thread pools but I don't think we're setting up special thread pools for non interactive users | 21:58 |
| clarkb | sshd.batchThreads is the option | 21:59 |
| ianw | i'll look into that | 22:00 |
| clarkb | ya we don't set that value so everything shares the same pool so should be a noop to add those users to Service Users group if I read things correctly. The other thing to check is just that we don't give Service Users any unexpected permissions somewhere (this is what I was saying would be good to double check) | 22:00 |
| clarkb | ianw: thanks! I'm going to pop out now, but will check back in in a few hours | 22:01 |
| clarkb | change.enableAttentionSet is the option to set to False should we decide it just isn't working, but I think it is worth tryting to make work first | 22:02 |
| opendevreview | Merged opendev/system-config master: Start building gerrit 3.4 https://review.opendev.org/c/opendev/system-config/+/792047 | 22:30 |
| ianw | it looks like zuul is adding the uploading user back to the attention set | 23:05 |
| ianw | i'll just get a cup of tea then look into the service users group | 23:06 |
| ianw | https://review.opendev.org/admin/groups/5ebcbd5ffce95bd1d8eb9ac279cc930d5e6fc6f9,members | 23:20 |
| ianw | currently has release and proposal bot, which seems right | 23:20 |
| ianw | i think we should add "Third-Party CI" and "Continuous Integration Tools" | 23:22 |
| ianw | ok, that is done | 23:30 |
| ianw | that seems to have fixed it. "Starting gate jobs" in https://review.opendev.org/c/openstack/devstack/+/803731 didn't modify the attention set | 23:38 |
| fungi | good call | 23:38 |
| fungi | thanks for the fix | 23:38 |
| fungi | hopefully that doesn't some with unexpected side effects | 23:38 |
| ianw | yeah, i'm keeping a close eye to make sure and watching zuul logs, it seems ok | 23:39 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!