| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : remove deprecated NoOp function https://review.opendev.org/c/openstack/project-config/+/875804 | 00:18 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/project-config master: acls : handle submit requirements in normalise tool https://review.opendev.org/c/openstack/project-config/+/875992 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : convert NoBlock to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875993 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : handle key / values with multiple = https://review.opendev.org/c/openstack/project-config/+/875994 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Update Review-Priority to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875995 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Convert remaining AnyWithBlock to submit requirements https://review.opendev.org/c/openstack/project-config/+/875996 | 00:18 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : disallow "function" https://review.opendev.org/c/openstack/project-config/+/875997 | 00:18 |
| ianw | a bit bigger stack than i thought, but i think that gets us there with submit requirements in the acl bits anyway | 00:19 |
| clarkb | ianw: I think https://review.opendev.org/c/openstack/project-config/+/875804/2/gerrit/acls/opendev/infra-specs.config is flawed because max with block or whatever is the default label function. You need to explicitly set the function to noop when using submit requirements | 00:23 |
| clarkb | ianw: this is the thing I got all confused about when reading the upstream docs | 00:23 |
| clarkb | and noblock == noop so the next change has a similar problem? We should probably standarize on a single predicate there to avoid confusion about potnetial behavior differences | 00:24 |
| ianw | ahh yes i remember this discussion now | 00:26 |
| ianw | we never got an official answer on that did we | 00:26 |
| clarkb | we did not, but we poked around in the code and seemed to reach the conclusion you should explicitly set the deprecated values (ugh_) because the first enum entry is max with block and that would potentially override the submit requirements so yo uneed to set noop/noblock explicitly | 00:27 |
| ianw | https://groups.google.com/g/repo-discuss/c/tZwIp3Hx-wA/m/SV46dMBuAAAJ | 00:27 |
| clarkb | I suspect that in gerrit 3.9 or so we'll end up deleting all the noop entries | 00:27 |
| ianw | that's right, it's paging back in now :) | 00:28 |
| clarkb | and to be clear I personally find this whole thing confusing so if someone ends up with an alternative understanding please don't be afaid to present it :0 | 00:31 |
| clarkb | er :) | 00:31 |
| clarkb | it is entirely possible I'm wrong | 00:31 |
| ianw | i'm putting a 3.7 on hold and will poke to get some definitive answers (that may raise more questions, but anyway :) | 01:00 |
| ianw | remote: ERROR: commit 2c8917f: Cannot delete 'label.Review-Priority.function'. Label functions can only be set to {NO_BLOCK, NO_OP, PATCH_SET_LOCK}. Use submit requirements instead of label functions. | 03:14 |
| ianw | so yeah, we can't delete the function | 03:14 |
| ianw | if you use the ui, it does default to adding MaxWithBlock | 03:34 |
| ianw | an "is:true" submit-requirement seems to still make it a "trigger vote" | 03:37 |
| ianw | https://gerrit-review.googlesource.com/Documentation/config-submit-requirements.html#trigger-votes | 03:37 |
| ianw | i can't see that the submit-requirement description shows up in the UI anywhere | 03:40 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : submit-requirements for deprecated NoOp function https://review.opendev.org/c/openstack/project-config/+/875804 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : add submit requirements to NoBlock labels https://review.opendev.org/c/openstack/project-config/+/875993 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : handle key / values with multiple = https://review.opendev.org/c/openstack/project-config/+/875994 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Update Review-Priority to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875995 | 05:48 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : Convert remaining AnyWithBlock to submit requirements https://review.opendev.org/c/openstack/project-config/+/875996 | 05:48 |
| *** jpena|off is now known as jpena | 08:27 | |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repo https://review.opendev.org/c/openstack/project-config/+/876054 | 10:15 |
| yoctozepto | infra-root: ^ I was not sure if I should go for a new zuul tenant (as its creation is not really documented in the "Project Creator's Guide") and so I have chosen the opendev tenant | 10:17 |
| yoctozepto | please let me know what would be the best approach here | 10:17 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add the main NebulOuS repo https://review.opendev.org/c/openstack/project-config/+/876054 | 10:31 |
| frickler | yoctozepto: some more information in term of the expected scope/size of your project might be helpful in order to decide whether a dedicated tenant would be useful. like if you are talking about the "main" repo, how many do you expect to have in the end? | 10:52 |
| frickler | also, do you plan to only consume resources or also to contribute, like in terms of CI capacity or administration man-power? | 10:53 |
| yoctozepto | frickler: a max of roughly a dozen, let's say 15, if all components decide to split out from the initial monorepo (not likely); we can offer some administration manpower (for one, you will see more of my activity :D I can also get trained on the work behind the scenes to help ourselves and you all) but otherwise I don't believe we have any relevant computing capacity in the budget (but I will check with my supervisors/management) | 11:03 |
| frickler | yoctozepto: o.k., let's wait and see what the others think | 11:18 |
| yoctozepto | ok | 11:20 |
| fungi | yoctozepto: i don't object, just be aware that there will be some overhead on the repo count. at a very minimum you'll need a base jobs repo for the tenant, you might be able to get away with reusing it as your central trusted config repo too but may want to make that separate, and then you'll likely also want a single-branch untrusted repo to hold most of your central job configuration that | 13:04 |
| fungi | doesn't need access to things like secrets | 13:04 |
| fungi | maybe look at the vexxhost or zuul tenants for some smaller examples | 13:07 |
| yoctozepto | fungi: is this documented somewhere? as to what tradeoffs are there with each option? I don't mind running as part of the opendev tenant in general, were just curious what the approach is | 13:07 |
| yoctozepto | I didn't mean to sound like I demand a separate tenant | 13:08 |
| yoctozepto | ack, zuul and vexxhost noted for later | 13:08 |
| fungi | yoctozepto: the main tradeoff is you're taking on management of the aforementioned overhead repos. the closest thing there is to a document describing that repo split is, i think, https://zuul-ci.org/docs/zuul/latest/howtos/pti.html | 13:09 |
| yoctozepto | fungi: I see; though what do I *gain*? :D or do I only get more work to do for no gain? :D | 13:12 |
| yoctozepto | this is what I was pondering upon | 13:12 |
| fungi | yoctozepto: you gain the flexibility to diverge more from the job setup in other tenants, and control of all of that without having to wait for other gatekeepers to approve your patches for it | 13:13 |
| fungi | but as is often said of open source, if it breaks you get to keep the pieces ;) | 13:14 |
| fungi | (in seriousness, i'm happy to help if you run into issues with your tenant configs though, as i expect are other folks) | 13:14 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: Provide deploy-microshift role https://review.opendev.org/c/zuul/zuul-jobs/+/876081 | 13:14 |
| fungi | yoctozepto: note that changes to jobs in trusted config repos, including your base job and anything it runs, can't safely be directly tested pre-merge. we've figured out a workflow for mostly safely making changes to ours, which you might want to follow if you go the route of having your own tenant: https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L5-L23 | 13:17 |
| yoctozepto | fungi: I see, thanks for the clarifications! I will give it some thought later after doing some more reading too; one last question for now - will the separate tenant have access to the same pool of nodes? | 13:17 |
| yoctozepto | ah yeah, I have seen this discussion before | 13:18 |
| opendevreview | Espen Flage-Larsen proposed openstack/diskimage-builder master: Removed erroneous, comment, check for grub.d https://review.opendev.org/c/openstack/diskimage-builder/+/876084 | 13:18 |
| fungi | yoctozepto: yes, our pools are available to all tenants | 13:19 |
| fungi | keep in mind that job configuration can't be inherited between tenants though, which is why you'll see openstack projects included in the projects list for tenants like vexxhost or zuul | 13:19 |
| fungi | if you want to run jobs or consume other configuration from specific repos, you have to explicitly add them to the tenant (even if that tenant is not gating changes for them, it needs to know to include the configuration from them) | 13:20 |
| yoctozepto | oh, yeah, I have seen this pattern while editing the zuul tenant config, seems reasonable enough | 13:22 |
| yoctozepto | I think we won't be needing them for now at least, this is pretty independent | 13:22 |
| fungi | right, if you decided you wanted to run... say... devstack jobs (or worse still, tempest jobs) then you'd need to include openstack's kitchen sink | 13:23 |
| fungi | those sort of interdependency tentacles are one of the main reasons we've struggled to split some repositories out to different tenants | 13:24 |
| yoctozepto | yeah, I can imagine | 13:25 |
| fungi | zuul's composable configuration model is a double-edged sword. it can be really convenient to incorporate configuration from another project, but do that too often without a clear plan and you can end up hamstrung with dependency hell type problems | 13:27 |
| fungi | so like any complex system, design and planning are important | 13:28 |
| yoctozepto | at least it makes us keep our jobs in the long term haha | 13:29 |
| yoctozepto | I agree totally | 13:29 |
| *** tweining[m] is now known as tweining|ghost | 13:34 | |
| priteau | Hello. Just so you know, I've had a job fail with: "ECDSA host key for 172.99.67.80 has changed and you have requested strict checking." | 14:18 |
| priteau | https://zuul.opendev.org/t/openstack/build/5409a583810845deaccab4210458a8fc | 14:18 |
| fungi | priteau: we see that from time to time when nova has lost track of a vm in some cloud provider (thinks it was successfully deleted but remained running for some reason), and then neutron assigns the same ip address to a new server instance we've created, then network connections get randomly sent to the old vm resulting in that error (this time appears to have been in rackspace's iad region). | 14:31 |
| fungi | they usually run periodic cleanup activities in the background to find and terminate such "rogue" virtual machines so the problem ip address usually clears up after a short time | 14:31 |
| fungi | if we see frequent occurrences we'll often correlate the ip addresses from the errors and supply a list to the provider in order to let them know | 14:32 |
| frickler | that the same address that was mentioned a couple of days ago, let me see if I can catch the duplicate host | 14:38 |
| frickler | hmm, seems not to belong to our set of CI nodes currently, at least I cannot login via SSH. iirc contact to rackspace would be via their ticket system? not sure if twice in a week would already fall under "frequent" | 14:43 |
| frickler | here are two more https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%20%22ECDSA%20host%20key%20for%20172.99.67.80%20has%20changed%22'),sort:!()) | 14:46 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: Provide deploy-microshift role https://review.opendev.org/c/zuul/zuul-jobs/+/876081 | 15:48 |
| clarkb | gerrit community meeting was quick, but I got some helpful feedback. | 16:24 |
| clarkb | Re java 17 NasserG didn't think the workaround suggested was problematic but was also apparently surprised it worked with java 17. They also clarified java 11 support isn't going away anytime soon so unless we needed functionality in java 17 (most likely gc features) there isn't any urgency to upgrade | 16:25 |
| clarkb | given ^ I think maybe we shelf java 17 work for now and hope upstream is able to fix the problem and then we won't have to provide workarounds to java command lines | 16:25 |
| clarkb | for the sshd thing it sounds like the config option probably shouldn't be secret and should be documented so I volunteered to work up a change for that. And NasserG said they would try and test ianw's patch in their systems as they had to set the flag to disable things as it created a ton of failures for them | 16:26 |
| clarkb | I'm going to find breakfast but then I'll wip the java 17 change and I think we come back to that when the issues are sorted or when it becomes urgent enough we're willing to apply the workaround | 16:28 |
| fungi | k | 16:31 |
| *** xhku_ is now known as xhku | 16:37 | |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: WIP: save the boot filesystem for fips workload cases https://review.opendev.org/c/openstack/diskimage-builder/+/876192 | 16:44 |
| hashar | clarkb: hi, good to see you ar eattending the Gerrit community meeting :] | 16:47 |
| clarkb | hashar: yup, I'm trying to get more involved with Gerrit. We've been managing to push some bug fixes and changes occasionally and have been keeping our Gerrit install far more up to date | 16:48 |
| hashar | about Gerrit SshChannelNotFoundException, looks like it got talked aobut on the mailling list and someone mentioned that `sshd.enableChannelIdTracking` setting. my summary is https://phabricator.wikimedia.org/T263293#6986975 | 16:49 |
| hashar | apparently I have set it to `false` back in January but I have not revisited the task since then | 16:49 |
| clarkb | hashar: yup that seems to be the common action people have taken. But ianw dug into it and found a bug with the original channel id tracking change. He pushed a fix but we are not sure if that fixes the underlying issue or just to source of the specific exception | 16:49 |
| hashar | ah good ianw!!! | 16:50 |
| clarkb | NasserG said they would try and test ianw's change to see if it fixes the underlying problem | 16:50 |
| clarkb | hashar: https://gerrit-review.googlesource.com/c/gerrit/+/358314/1 | 16:50 |
| *** bhagyashris|ruck is now known as bhagyashris | 16:51 | |
| hashar | nice :] | 16:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove gitea08 from haproxy https://review.opendev.org/c/opendev/system-config/+/876194 | 16:55 |
| clarkb | infra-root ^ thats the next step in gitea things other than restarting gerrit to pick up the replication plugin config changes | 16:55 |
| clarkb | I'm going to go bring the gerrit restart up in openstack-release now to make sure we don't interfere with RCs | 16:55 |
| clarkb | 876194 should be safe though as its just removing a node from haproxy config | 16:56 |
| hashar | clarkb: I have grep in our Gerrit log and apparently the issue no more occurs so I guess sshd.enableChannelIdTracking=false did the trick :] | 16:59 |
| clarkb | ya that disables the functionality entirely so it can't trigger the broken registration of the objects | 17:00 |
| clarkb | but whether or not there is some underlying issue is still not understood I think | 17:00 |
| *** jpena is now known as jpena|off | 17:47 | |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: WIP: save the boot filesystem for fips workload cases https://review.opendev.org/c/openstack/diskimage-builder/+/876192 | 17:51 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add gitea10-12 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/876201 | 18:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gitea10-12 to our inventory https://review.opendev.org/c/opendev/system-config/+/876202 | 18:20 |
| clarkb | infra-root ^ production line gitea deployments. Please double check I didn't copy pasta anything poorly or miss a step | 18:20 |
| clarkb | I went ahead and single core approved the gitea08 haproxy removal | 18:31 |
| clarkb | its a pretty safe chagne since the server is sticking around for a little while longer and only affects the load balancer | 18:32 |
| fungi | wfm, thanks! | 18:35 |
| clarkb | I'm also thinking once 09-12 are up and rnning we may temporarily remove the old servers from haproxy to see how four servers cope | 18:36 |
| clarkb | I know you mentioned wanting to keep 8 but these servers are a fair bit bigger so I'm not sure we need to keep 8. Experiments should help us answer that question | 18:37 |
| clarkb | maybe we need 6 instead of 8 etc | 18:37 |
| clarkb | I was going to add the gitea ssh hostkeys to gerrit at this point too until I remembered I need the port 222 hostkey which doesn't exist yet | 18:38 |
| clarkb | Once 876202 and 876201 land the next hold up is going to be restarting gerrit to pick up replication changes | 18:38 |
| clarkb | the openstack release team indicated after 1900 UTC would be ok for them | 18:39 |
| fungi | yeah, mainly thinking that we're more memory-bound when we get a backend overloaded, so the memory increase while keeping the same cpu count should mean less disruption | 18:39 |
| clarkb | also I intend on removing gitea08 replication when I add 10-12 replication. This reduces the number of replication plugin reloads which should reduce changes for replication tasks getting lost | 18:42 |
| fungi | but then again, usually the offender ends up hitting a single backend, so more backends doesn't really make that much of a difference if the added memory per backend serves as a sufficient cushion | 18:42 |
| clarkb | fungi: exactly | 18:42 |
| clarkb | fungi: ianw: how about ~2100UTC to down up review? I think that has good overlap with my day and the two of you should anything unexpected occur (mostly thinking about the change of bind mounts | 18:44 |
| fungi | i'll be around, sounds fine with me | 18:45 |
| fungi | happy to help with it too | 18:46 |
| clarkb | thanks! | 18:46 |
| opendevreview | Felipe Reyes proposed openstack/project-config master: Add Ironic Dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/876205 | 19:15 |
| opendevreview | Felipe Reyes proposed openstack/project-config master: Add Ironic Dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/876205 | 19:43 |
| opendevreview | Merged opendev/system-config master: Remove gitea08 from haproxy https://review.opendev.org/c/opendev/system-config/+/876194 | 20:03 |
| clarkb | looking on review02 the current opendevorg/gerrit:3.6 image seems to match the image at https://hub.docker.com/layers/opendevorg/gerrit/3.6/images/sha256-c3bd03fcee7751849ea8cdb7045fb4d1118296037824bdc5d277158986b8cf12?context=explore | 20:32 |
| clarkb | 876194's merging has triggered a manage projects job run in deploy. I think we want that to complete before the gerrit restart. I've done some extra double checking on review02 and I think things are the way we want them. Just need to do a down them up | 20:39 |
| clarkb | looks like ansible galaxy changed their content responses and broke our test for proxying it | 20:43 |
| clarkb | do we know if that is being used yet? there is a good chnace it broke any of them too | 20:43 |
| fungi | gah | 20:43 |
| clarkb | (it resulted in a -1 on my gitea10-12 change) | 20:43 |
| clarkb | https://zuul.opendev.org/t/openstack/build/176bfa5354bb48aab0b412aca964c0b2 | 20:43 |
| fungi | Tengu: ^ you were the one working on that, right? | 20:44 |
| clarkb | ok the base job failed for 876194 which means manage-projects isn't running. However the gitea-lb job is running so it didn't care about that failure. I think this means w edon't need to wait on gerrit restarts and that the haproxy config update should apply anyway. The base job failed due to apt index issues with the osuosl mirror | 20:52 |
| clarkb | I suspect that issue will go away on its own once the upstream mirror that node talks to is fully in sync again | 20:52 |
| ianw | i'm around if you want to restart gerrit :) | 20:56 |
| clarkb | yup I'm going to do a docker-compose pull just in case mariadb wants to update | 20:56 |
| clarkb | it did | 20:57 |
| clarkb | anyone else want ot check the gerrit image lines up with what we expect before I down then up? | 20:58 |
| clarkb | the jdk package chagne added 220mb to the image :/ but I think thats worth it should it become useful for debugging | 20:59 |
| clarkb | fungi: ianw: let me know when you're ready for me to do the restart. I think I've done the checks I was going to do | 21:00 |
| ianw | all lines up with "opendevorg/gerrit@sha256:c3bd03fcee7751849ea8cdb7045fb4d1118296037824bdc5d277158986b8cf12 from 875553 so lgtm | 21:02 |
| clarkb | fungi: ^ last call? | 21:03 |
| sanchankun | hi, i'm having troubles setting up horizon zun-ui integration. Horizon apache logs only mention glance not supporting docker image backend. zun-api,compute,ws-proxy,cni;kuryr , work as expected. Any ideas where to look for logs? :-( | 21:03 |
| clarkb | sanchankun: give us a few as we are about to do a gerrit restart to pick up a config change | 21:04 |
| clarkb | ianw: I'll give fungi a few more minutes to tell us to hold up, otherwise I'll proceed | 21:05 |
| clarkb | alright proceeding now. I'll do a docker-compose down, then a docker-compose up -d | 21:08 |
| clarkb | thats done. Web ui loads for me. New version is reported in the page footer. I see content in /home/gerrit2/review_site/data | 21:10 |
| clarkb | this also picked up the brand new gerrit 3.6.4 release | 21:11 |
| clarkb | they don't have release notes up for that yet :/ | 21:13 |
| ianw | the website was lagging on 3.7.1 too | 21:13 |
| fungi | no ned to hold up, i'm ready to help with the restart | 21:13 |
| fungi | s/ned/need/ | 21:13 |
| ianw | looks like both in review https://gerrit-review.googlesource.com/c/homepage/+/361258 | 21:13 |
| clarkb | fungi: its done :) at this point just double checking that things look good would be helpful. Maybe review and/or land https://review.opendev.org/c/opendev/zone-opendev.org/+/876201 as a good way to exercise things | 21:14 |
| clarkb | ianw: ^ fyi | 21:14 |
| clarkb | sanchankun: is this for a CI job? or for a deployment? We help run the CI systems and may be able to help with issues related to CI jobs, but if this is for a deployment you may have better luck with the horizon and zun teams | 21:15 |
| sanchankun | it is for a deployment. allright , i will try to contact them . thanks a lot ! | 21:16 |
| ianw | clarkb: interestingly i had to log back in, did you? | 21:17 |
| clarkb | ianw: I did not | 21:17 |
| ianw | ok, probably just a local timeout. wondered if it was the datadir thing | 21:17 |
| clarkb | I think this has come up in the past and I suspect that the cache for logins is only so long or something | 21:17 |
| clarkb | no as far as I know the data dir is only used by plugins and not core gerrit | 21:18 |
| ianw | clarkb: 876201 lgtm. did those sshfp records spit out OK from the launch node? i feel like i have changes out for that, but maybe not | 21:18 |
| clarkb | ianw: they did not. I just ssh'd in and did ssh-keyscan -D to get them. I had to ssh in because our jammy image is old and won't autoupdate sosreport which leads to unattended upgrades complaints | 21:18 |
| clarkb | so I ssh'd in manually dist-upgraded to fix sosreport, rebooted for good measure, then did ssh-keyscan commands on the host | 21:19 |
| clarkb | I should probably upload a new jammy image to vexxhost before doing the next batch | 21:19 |
| ianw | actually it might have been https://review.opendev.org/c/opendev/system-config/+/868369 | 21:19 |
| ianw | i think we probe for the ssh keys before ssh is up again | 21:20 |
| clarkb | ah | 21:20 |
| ianw | might be worth putting that in before launching more nodes for testing. it *should* work but it's been lots of fussing between old bridge versions that didn't have the tools, and this | 21:21 |
| clarkb | ok I'll take a look. First I need to look at the failing mirror testinfra tests. But should be able to look at that next | 21:22 |
| clarkb | my goal/hope is that I'll be able to add gitea10-12 to the gerrit replication config tomorrow and have the replication run over the weekend | 21:22 |
| clarkb | but I need the add giteas to inventory change to land after the dns change and that change is currently failing | 21:22 |
| clarkb | manually trying to reproduce the mirror failure I end up reproducing what the test is looking for so I don't understand why it would fail. Maybe there was a momentary blip on the ansible galaxy side of things? | 21:34 |
| clarkb | I'll recheck and see if it is consistent I guess | 21:34 |
| fungi | or they have a cdn/lb sometimes returning content from a troubled backend | 21:35 |
| clarkb | ianw: couple of comments on the ssh busy wait loop change | 21:37 |
| clarkb | fungi: https://review.opendev.org/c/opendev/zone-opendev.org/+/876201 adds the new giteas to dns so that they can provision ssl certs when they are added to inventory | 21:38 |
| clarkb | unrelated to everything else I've done today I noticed manila had a change stuck in check for ~9 hours. This appears due to them retrying a 3 hour long job which appears to be retrying because disk is filling and ansible can't find a suitable /tmp or equivalent which is treated as a network connection error. I've pinged the manila time in their irc channel about this | 21:44 |
| clarkb | I suggested they fix the job or move it to experimental to avoid 9 hour delays getting test results back | 21:44 |
| clarkb | gitea08 is much quieter after being removed from haproxy too | 21:50 |
| clarkb | `gerrit show-queue` shows a new task for com.googlesource.gerrit.plugins.replication.AutoReloadRunnable I guess this is what watching for config updates | 21:51 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Ironic program adopting virtualpdu https://review.opendev.org/c/openstack/project-config/+/876231 | 21:55 |
| opendevreview | Merged opendev/zone-opendev.org master: Add gitea10-12 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/876201 | 21:56 |
| JayF | if one of you will let me know if that proj-conf change looks good, I'll add it to the agenda | 22:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gerrit image builds for 3.6.4 and 3.7.1 tags https://review.opendev.org/c/opendev/system-config/+/876233 | 22:07 |
| clarkb | infra-root ^ re gerrit releases that is a bookkeeping chagne for us as I checked the versions we built our new images with and the new tags just retag old commits with one exception and we were already checking out master for that one exception. | 22:07 |
| clarkb | JayF: that looks about right. I think the docs ask that you use a speciifc topic (whihc ou can set in the web ui) | 22:08 |
| JayF | clarkb: is it OK to add to the lsit before the governance patch lands? | 22:09 |
| clarkb | yes | 22:09 |
| JayF | although I would assume it's a fait accompli | 22:09 |
| clarkb | JayF: then generally the way the rename process works is we make sure everyone has the prep work and sign offs done. We schedule a gerrit downtime. During that downtime we stop gerrit. Move things around. We start gerrit and then quickly merge the change you pushed to avoid any creation of the old project again and we're in sync at athat point | 22:12 |
| clarkb | with more than two changes proposed for projects renames I will need to go and rediscover how it is that two manage-project runs in a row don't create problems. I think it is because we always update openstack/project-config from master and don't use the zuul version | 22:13 |
| clarkb | (thats a me thing to worry about not you) | 22:13 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Ironic program adopting virtualpdu https://review.opendev.org/c/openstack/project-config/+/876231 | 22:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch: add a probe for ssh after reboot https://review.opendev.org/c/opendev/system-config/+/868369 | 22:19 |
| clarkb | ianw: did you catch my notes from the gerrit community meeting earlier today? tldr is I think we'll hold off on java 17 since there isn't any urgency upstream or on our end to move to 17 yet and that gives them time to work out the issue I had to workaround. And for the ssh thing NasserG said they would try to test it in their env since they had to set the secret flag to false. I'll | 22:30 |
| clarkb | also try to make a change to document the secret flag | 22:30 |
| ianw | yeah, agree on keeping on 11 for now. 17 definitely feels like maybe a mid-release thing we can update to and roll back easily from | 22:35 |
| clarkb | yup, the main benefits are in the garbage collector(s) and we haven't had issues with that recentl so I think we can hold off | 22:35 |
| clarkb | apparently newer javas have all done improvement to the garbage collection systems and that is a big reason people often upgrade | 22:36 |
| ianw | the flag i have to think about | 22:36 |
| clarkb | ianw: I get the sense that everyone was just setting it to false | 22:36 |
| clarkb | so its good to call attention to it and have someone whose installations were more affected by it check the change you pushed | 22:37 |
| ianw | the fix as merged isn't, iiuc, really doing anything. so then the flag is turning off the not-working workaround | 22:38 |
| clarkb | right, but your change should fix the fix. At least it will expose if the fix is fixing anything rather than exploding | 22:38 |
| ianw | agree, it would be great to test with that. but i'm not sure what the secret flag is doing to get things working | 22:40 |
| clarkb | oh I see. I think maybe it removes the errors from your logs so you stop worrying about it :) | 22:40 |
| ianw | that might be correct ... end users still have the same result, but you just don't know about it from your logs | 22:41 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/876202 has a +1 now. The mirror error must've been transiet | 22:41 |
| fungi | fun | 22:45 |
| ianw | thanks, lgtm; left it for you to +w if you want to watch it? | 22:45 |
| clarkb | thanks ya I'll probably approve it now so that things are ready for friday replication starts | 22:46 |
| clarkb | the only question is whether or not I need to double check the osuosl mirror can update its apt indexes yet | 22:47 |
| clarkb | as I think failed base jobs will break the gitea deployments on those nodes | 22:47 |
| clarkb | it failed to fetch focal-updates previously. Manually running apt-get update on the host was successful I think thats fine | 22:48 |
| clarkb | I've approved it and will keep an eye on it. This step will deploy the empty giteas and is pretty low impact if anything goes wrong | 22:49 |
| clarkb | ianw: where did we end up with the function = NoOp stuff? | 23:57 |
| clarkb | it sounded like your testing indicated we did need to keep it after all? | 23:57 |
| ianw | clarkb: yes, you can't push an acl change without a function = line | 23:58 |
| ianw | so i've changed them all to NoOp in the series now | 23:58 |
| ianw | and implemented submit-requirements where required | 23:58 |
| ianw | what i'm doing now is just getting a diff together and doc updates for All-Projects | 23:58 |
| clarkb | cool, good to have that run down. I'll have to rereview those changes | 23:59 |
| ianw | https://104.130.253.50/c/x/test-project/+/1 | 23:59 |
| clarkb | I've just realized that the gitea deployment is likely to take a number of hours due to the inventory updates. I'll try to check on it later this evening but may just end up digging into the results tomorrow morning | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!