| ianw | what's interesting is that if you don't have a submit-requirement; or it seems that a submit-requirement of "is:true" also does this -- the votes come up in the "trigger votes" section | 00:00 |
|---|---|---|
| clarkb | ya I think a submit requirement is anything that must be satisfied to make teh change mergeable and a trigger requirement is anything else | 00:00 |
| ianw | yep, but i think it adds another layer of confusion to the whole thing | 00:01 |
| clarkb | yup especially since the trigger requirements are defined as submit requirements in the acls | 00:01 |
| ianw | right, i've given everything a submit-requirement, even if it's just "is:true", for consistency | 00:02 |
| opendevreview | Merged opendev/system-config master: Add gitea10-12 to our inventory https://review.opendev.org/c/opendev/system-config/+/876202 | 00:04 |
| clarkb | yup just confirmed it will be a bit before that gets to the giteas. I'll keep an eye on it through my evening but with progressively less attention. | 00:05 |
| clarkb | Assuming they deploy as easily as gitea09 this should be fine though | 00:05 |
| opendevreview | Ian Wienand proposed opendev/system-config master: doc/gerrit : update copyCondition https://review.opendev.org/c/opendev/system-config/+/876236 | 00:09 |
| opendevreview | Ian Wienand proposed opendev/system-config master: doc/gerrit : update to submit-requirements https://review.opendev.org/c/opendev/system-config/+/876237 | 00:09 |
| opendevreview | Ian Wienand proposed opendev/system-config master: doc/gerrit : update to submit-requirements https://review.opendev.org/c/opendev/system-config/+/876237 | 00:58 |
| clarkb | ianw: I had a mini panic that I forgot to accept the ssh hostkeys on bridge, but I guess the new ansible host keys content takes care of that because is all fine | 01:42 |
| clarkb | testing it it all works without my previous intervention. Thats awesome | 01:43 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: A new diskimage-builder command for yaml image builds https://review.opendev.org/c/openstack/diskimage-builder/+/876245 | 03:07 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : submit-requirements for deprecated NoOp function https://review.opendev.org/c/openstack/project-config/+/875804 | 03:42 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : add submit requirements to NoBlock labels https://review.opendev.org/c/openstack/project-config/+/875993 | 03:42 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : handle key / values with multiple = https://review.opendev.org/c/openstack/project-config/+/875994 | 03:42 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Update Review-Priority to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875995 | 03:42 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Convert remaining AnyWithBlock to submit requirements https://review.opendev.org/c/openstack/project-config/+/875996 | 03:42 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: Provide deploy-microshift role https://review.opendev.org/c/zuul/zuul-jobs/+/876081 | 07:44 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: Provide deploy-microshift role https://review.opendev.org/c/zuul/zuul-jobs/+/876081 | 07:46 |
| *** odyssey4me is now known as odyssey4me__ | 10:58 | |
| *** odyssey4me__ is now known as odyssey4me | 10:58 | |
| *** jpena|off is now known as jpena | 11:00 | |
| opendevreview | Ebbex proposed openstack/diskimage-builder master: Fix double-keyed json https://review.opendev.org/c/openstack/diskimage-builder/+/876292 | 11:04 |
| *** odyssey4me is now known as odyssey4me__ | 11:06 | |
| *** odyssey4me__ is now known as odyssey4me | 11:06 | |
| *** odyssey4me is now known as odyssey4me__ | 12:02 | |
| bbezak | is there any problems with this host? Server Error (\"Get \"https://mirror.iad3.inmotion.opendev.org:4447/v2/\": dial tcp 173.231.253.126:4447: connect: no route to host\")\\n' | 13:51 |
| bbezak | will recheck my job, but I recall there were issues with it recently? | 13:52 |
| mnasiadka | fungi: is that possible that mirror is down again? ;-) | 14:13 |
| fungi | it just came up in the release meeting as well, i'm checking to see if the server spontaneously died again | 14:14 |
| fungi | OS-EXT-STS:power_state=Shutdown | 14:15 |
| fungi | i'll start it again, but we should probably take the region out of service until someone can look into why that vm keeps getting turned off by nova | 14:15 |
| fungi | #status log Booted mirror.iad3.inmotion via Nova API after it was found in power_state=Shutdown since 13:39:47 UTC | 14:18 |
| opendevstatus | fungi: finished logging | 14:18 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Revert "Revert "Temporarily stop booting nodes in inmotion iad3"" https://review.opendev.org/c/openstack/project-config/+/876365 | 14:23 |
| fungi | infra-root: ^ | 14:23 |
| Clark[m] | I think you can go ahead and self approve that one. Also for everyone else looking at this problem all of our mirrors are externally available which allows you to check if this is consistent and reproducable | 14:25 |
| fungi | with the caveat that the name nodes usually access our mirrors in rackspace by aren't externally reachable, but there are externally-reachable names for them | 14:31 |
| fungi | but this case wasn't in rackspace anyway | 14:31 |
| opendevreview | Merged openstack/project-config master: Revert "Revert "Temporarily stop booting nodes in inmotion iad3"" https://review.opendev.org/c/openstack/project-config/+/876365 | 15:03 |
| clarkb | I'm catching up on email and some paperwork then I'm going to start on gitea brain transplants for gitea10-12. Hoping to get a change up to replicate to them today | 16:48 |
| fungi | thanks! | 16:57 |
| *** jpena is now known as jpena|off | 17:21 | |
| dtantsur | could someone try https://zuul.opendev.org/t/openstack/build/1b381b57ef304017b772da1b91d7839a, is it ridiculously slow for everyone or just me? | 17:21 |
| dtantsur | I mean, it opens, but even selecting text takes many seconds | 17:22 |
| yoctozepto | dtantsur: same here, overloaded DOM | 17:24 |
| dtantsur | yeah. it probably does not like large job outputs. | 17:24 |
| fungi | ideally, redirect some of the noisier but less frequently referenced output to separate log files and collect those instead of having it all in the stdout/stderr stream | 17:30 |
| yoctozepto | fungi: hi! so we would prefer a separate tenant... how do I go about this? my guess is that I should adapt my change (https://review.opendev.org/c/openstack/project-config/+/876054) to include the creation of the zuul config repo and then it should be configured properly, right? | 17:30 |
| yoctozepto | dtantsur, fungi: as a matter of fact, that's what kolla has done and we have improved the speed of loading of these pages | 17:30 |
| yoctozepto | fungi: (continued) and then only create the tenant | 17:31 |
| dtantsur | fungi: that's just a tox unit test job... | 17:31 |
| fungi | yoctozepto: yes, we don't create new tenants very often, so it's possible we may have to break up the tenant creation and project addition changes separate, but testing should tell us one way or another | 17:31 |
| yoctozepto | I see | 17:31 |
| yoctozepto | what testing do you mean? | 17:31 |
| fungi | yoctozepto: the tests we run on proposed changes to our configuration | 17:32 |
| yoctozepto | ah, this validation | 17:32 |
| yoctozepto | but it's optimistic as far as I can tell, right? | 17:32 |
| yoctozepto | it just tries to lint and weed out obvious deviations? | 17:32 |
| yoctozepto | and only after merge we learn the truth | 17:33 |
| * yoctozepto trying to wrap his head around these things to be a better opendev citizen | 17:33 | |
| fungi | well, i think the tenant config additions will run additional deployment test jobs, not positive | 17:33 |
| yoctozepto | ok, then I will try this in one go and then we iterate on that | 17:34 |
| fungi | probably we need to create the repositories in gerrit, then add the zuul tenant and can add the projects to it when we do that | 17:34 |
| fungi | or we could create the zuul tenant with no projects in it, then create the projects, then add them to the tenant... but that's more changes i think | 17:35 |
| fungi | clarkb: ^ do you recall if there's any reason not to create repositories and then create the tenant with the projects added to it at the same time? | 17:35 |
| fungi | basically same as creating repositories and then adding them to an existing tenant, just you're also creating the new tenant in the second change | 17:37 |
| clarkb | give me a few minutes then I can page things in | 17:37 |
| fungi | basically change #1 is to add the projects in gerrit/projects.yaml and their acls in gerrit/acls, then change #2 is to add the tenant and its projects in zuul/main.yaml | 17:39 |
| fungi | then once #1 deploys successfully, #2 can merge | 17:39 |
| yoctozepto | yeah, that was my initial idea | 17:39 |
| yoctozepto | so maybe I will go with this one | 17:40 |
| yoctozepto | I will send a stack of patches soon | 17:40 |
| yoctozepto | just need to handle one other thing | 17:40 |
| fungi | of the projects in change #1, one of them will need to be the new config project which contains your base job, pipeline definititions, nodesets, and so on | 17:40 |
| fungi | bootstrapping a tenant may require we bypass zuul to merge the initial configuration in the config repo, i don't remember, but seems like there's a walkthrough of the steps in the zuul docs | 17:42 |
| fungi | because config projects can't run proposed configuration, and without configuring the tenant you won't be able to merge any changes to the config repo for it | 17:42 |
| fungi | so there's an interim step to add a gate pipeline which just runs the builtin noop job | 17:43 |
| fungi | yeah, https://zuul-ci.org/docs/zuul/latest/tutorials/quick-start.html#configure-a-base-job details bypassing zuul in order to merge the base job change in the tenant's config repo | 17:46 |
| fungi | er, wrong section, it's https://zuul-ci.org/docs/zuul/latest/tutorials/quick-start.html#configure-a-base-job to merge the pipeline configuration | 17:48 |
| fungi | gah, pasted from the wrong buffer | 17:48 |
| fungi | https://zuul-ci.org/docs/zuul/latest/tutorials/quick-start.html#configure-zuul-pipelines | 17:48 |
| clarkb | fungi: I think they are separate to prevent zuul errors | 17:48 |
| fungi | right | 17:49 |
| clarkb | fungi: zuul will clone the repos and you want ot nesure they are all present first | 17:49 |
| clarkb | also lets not land those things today so that I can safely perform gitea brain surgery | 17:49 |
| clarkb | at least let me get that done first which is next on my todo list I think | 17:49 |
| fungi | well, sure, i don't expect to merge them today | 17:49 |
| yoctozepto | gitea brain surgery sounds interesting | 17:52 |
| clarkb | yoctozepto: the easiest way to preserve historical project redirects in gitea is to transplant the database from an old server to a new server | 17:54 |
| clarkb | so thats what I'm doing for the new gitea servers I'm building | 17:54 |
| yoctozepto | I see, sounds tedious tbh | 17:55 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repos https://review.opendev.org/c/openstack/project-config/+/876054 | 17:56 |
| yoctozepto | meh, forgot the ACLs | 17:57 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repos https://review.opendev.org/c/openstack/project-config/+/876054 | 18:00 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the NebulOuS tenant https://review.opendev.org/c/openstack/project-config/+/876414 | 18:09 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repos https://review.opendev.org/c/openstack/project-config/+/876054 | 18:11 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the NebulOuS tenant https://review.opendev.org/c/openstack/project-config/+/876414 | 18:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Replicate Gerrit repos to gitea10-12 https://review.opendev.org/c/opendev/system-config/+/876419 | 18:23 |
| clarkb | infra-root ^ please check gitea10-12 and if they look good I think we can land ^ to start replication | 18:24 |
| yoctozepto | fungi, clarkb: it seems there is no relevant validation to the tenant config - at least in the check pipeline - it passes happily with not-yet-existent repos | 18:32 |
| clarkb | ya its done by zuul when it relaods its config which is why we have been cautious in the past | 18:33 |
| yoctozepto | fingers crossed then | 18:33 |
| clarkb | those errors probably won't break zuul today though but once upon a time zuul was less able to handle errors loading base tenant configs? Either way I think splitting up the merges and making sure things are happy at each step is a good idea | 18:33 |
| yoctozepto | yeah, I agree | 18:34 |
| yoctozepto | better to first create the repos, ensure they are there | 18:34 |
| yoctozepto | then create the tenant | 18:34 |
| yoctozepto | reload as need | 18:34 |
| yoctozepto | ed | 18:34 |
| yoctozepto | check again if it works | 18:34 |
| yoctozepto | and then worry about other things | 18:34 |
| fungi | clarkb: gitea10-12 lgtm | 19:08 |
| clarkb | fungi: thanks. Its probably unlikely that anyone else would get to reviewing it today. Should we go ahead and approve it? | 19:11 |
| fungi | i have done so now | 19:18 |
| clarkb | thanks! | 19:18 |
| clarkb | one thing I was looking for to check the redirects was a redirect due to a project rename and not an org move but I couldn't find any of those | 19:18 |
| fungi | forgot it's friday already (so saturday on the other side of the sphere). where has this week gone | 19:18 |
| clarkb | its been a quick one | 19:18 |
| fungi | project rename... yeah i'm not sure we've done anything besides namespace moves since going from cgit to gitea | 19:19 |
| fungi | clarkb: oh! x/devstack-plugin-topiko -> x/devstack-plugin-tobiko maybe? | 19:22 |
| fungi | it was a typo correction, but presumably reflected in the redirects | 19:22 |
| clarkb | looking | 19:23 |
| fungi | also found openstack/charm-lxd -> openstack/charm-nova-lxd | 19:24 |
| clarkb | yup the topiko one redirects as expected | 19:24 |
| fungi | so there are some, just not many | 19:24 |
| clarkb | https://gitea10.opendev.org:3081/x/devstack-plugin-topiko | 19:24 |
| clarkb | will take you to x/devstack-plugin-tobiko | 19:24 |
| opendevreview | Merged opendev/system-config master: Replicate Gerrit repos to gitea10-12 https://review.opendev.org/c/opendev/system-config/+/876419 | 19:36 |
| clarkb | hrm ^ didn't trigger a gerrit update? | 19:43 |
| clarkb | I see the issue | 19:43 |
| clarkb | we had the same issue with gitea08 and I thought I fixed it but I hadn't... | 19:45 |
| clarkb | anyway change shortyl | 19:45 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix infra-prod-service-review file matchers https://review.opendev.org/c/opendev/system-config/+/876430 | 19:48 |
| clarkb | fungi: ^ | 19:48 |
| fungi | oh right | 20:07 |
| fungi | also infra-prod-base failed, i'm checking logs now | 20:22 |
| fungi | mirror01.regionone.osuosl.opendev.org "Failed to update apt cache: E:Failed to fetch http://ddebs.ubuntu.com/dists/focal-proposed/main/binary-arm64/Packages.xz File has unexpected size (38816 != 38444). Mirror sync in progress?" | 20:23 |
| fungi | a manual apt update doesn't error for me | 20:26 |
| fungi | must have been temporary | 20:26 |
| Clark[m] | Lunch now, but that happened yesterday too. It went away when I tried manually and on the next run | 20:33 |
| fungi | yeah, i remember that | 20:47 |
| yoctozepto | what's the plan to merge the addition of the new project? | 21:18 |
| clarkb | yoctozepto: I think we can probably land things early next week once we've stabilized the new giteas? | 21:20 |
| clarkb | the next step is replicating all of the git repos to them from gerrit | 21:20 |
| fungi | and then usually we get two reviewers to go over the changes so we don't miss anything | 21:20 |
| clarkb | once that is done it should be fine to add new projects | 21:20 |
| yoctozepto | ack, thanks | 21:22 |
| clarkb | fungi: once gerrit replication configs update I'll trigger replication for bindep against the three new servers to ensure general functioning then trigger global replication for the three | 21:34 |
| clarkb | that will liekyl take the better part of the next 24 hours | 21:34 |
| clarkb | but we should be in a good spot on monday to put them behind haproxy and pull out some of the old servers | 21:34 |
| fungi | sounds good | 21:40 |
| clarkb | yoctozepto: the spelling of this word is such that I feel I need to check every occurrence of it to ensure we didn't swap an 'o' and a 'u' anywhere | 21:46 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: Correct boot path to cover FIPS usage cases https://review.opendev.org/c/openstack/diskimage-builder/+/876192 | 22:12 |
| opendevreview | Merged opendev/system-config master: Fix infra-prod-service-review file matchers https://review.opendev.org/c/opendev/system-config/+/876430 | 22:17 |
| clarkb | bindep replication seems to have gone well. I'm going to trigger full replication for those three nodes now | 22:48 |
| clarkb | I was going to shutdown gitea on gitea08 at this point but I'm fairly certain our daily jobs will just turn it back on again | 22:50 |
| clarkb | Instead I'll work on removing it and 05-07 together as the next step in this process | 22:50 |
| fungi | yeah, as long as it's out of the haproxy and replication config it should be fine until we can delete it | 22:52 |
| clarkb | ya I think remove 05-07 from haproxy, then remove 05-07 from gerrit replication, then remove 05-08 from inventory, then delete 05-08 are the next steps | 22:54 |
| clarkb | and they are all waiting on replication to run first | 22:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!