| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Allow the bootloader element with the default block device https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 00:27 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Revert "Use rpm -e instead of dnf for cleaning old kernels" https://review.opendev.org/c/openstack/diskimage-builder/+/827381 | 00:27 |
| *** dviroel is now known as dviroel|out | 00:59 | |
| *** rlandy|ruck is now known as rlandy|out | 01:23 | |
| ianw | time="2022-02-02T01:26:41Z" level=error msg="invalid internal status, try resetting the pause process with \"podman system migrate\": command required for rootless mode with multiple IDs: exec: \" | 01:28 |
| ianw | newuidmap\": executable file not found in $PATH" | 01:28 |
| ianw | hrm, not sure why this happens on the builders but not in the gate | 01:28 |
| *** lajoskatona_ is now known as lajoskatona | 01:34 | |
| ianw | there's a lot of out-of-date packages in the builder image. i wonder why that is | 01:36 |
| ianw | ... i wonder why we have unstable in there? deb http://deb.debian.org/debian unstable main | 01:37 |
| ianw | oh that's right, we need to podman unstable because it uses incompatible calls | 01:40 |
| opendevreview | Merged opendev/system-config master: Stop mirroring centos-8 https://review.opendev.org/c/opendev/system-config/+/827186 | 01:43 |
| ianw | well, installing uidmap/unstable into the builder containers makes it work ... this leaves the question of why it works in the gate sill unresolved | 01:51 |
| fungi | do we save the package version info in the gate jobs? or maybe the package install stdout | 01:58 |
| ianw | yes it's all a bit jumbled up but it is in the image build logs, e.g. https://6598fbd266c429fdaf1f-2c751aa3402a41175598f85942c9c421.ssl.cf2.rackcdn.com/825957/12/check/nodepool-build-image-siblings/01099b7/job-output.txt | 02:02 |
| ianw | WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers | 02:06 |
| ianw | ERRO[0000] invalid internal status, try resetting the pause process with "podman system migrate": command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH | 02:06 |
| ianw | both do not occur in the gate | 02:06 |
| ianw | 2022-02-01 05:08:55.764517 | ubuntu-focal | Setting up podman (3.4.4+ds1-1) ... | 02:06 |
| ianw | $ dpkg --list | grep podman | 02:07 |
| ianw | ii podman 3.4.4+ds1-1 amd64 engine to run OCI-based containers in Pods | 02:07 |
| ianw | we are using the same podman in the gate as deployed on the builders | 02:07 |
| ianw | hrm, we run on a focal node in the gate, but nb02 is bionic | 02:09 |
| fungi | that could be it | 02:22 |
| fungi | maybe we just need to upgrade the builders | 02:22 |
| ianw | i'm thinking in-place upgrade is probably fine. these aren't much more than container runners | 02:23 |
| ianw | sigh, i in-place upgraded nb02 and ... same thing | 03:06 |
| ianw | but still, a useful process. notes | 03:06 |
| ianw | 1) re-run base with --flush-cache | 03:07 |
| ianw | 2) rm /usr/local/bin/pip* as the python upgrade breaks the global pip install. removing these indicates to ensure-pip role to re-install via get-pip.py | 03:07 |
| ianw | 3) ansible-playbook -v --limit nb02.opendev.org ./playbooks/service-nodepool.yaml | 03:08 |
| *** ysandeep|out is now known as ysandeep | 03:33 | |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: WIP/DNM - Test new version of python3 https://review.opendev.org/c/zuul/zuul-jobs/+/827366 | 04:03 |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: WIP/DNM - Test new version of python3 https://review.opendev.org/c/zuul/zuul-jobs/+/827366 | 04:49 |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: WIP/DNM - Test new version of python3 https://review.opendev.org/c/zuul/zuul-jobs/+/827366 | 05:15 |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: WIP/DNM - Test new version of python3 https://review.opendev.org/c/zuul/zuul-jobs/+/827366 | 06:00 |
| ianw | f34 & f35 have built with the equivalent of https://review.opendev.org/c/zuul/nodepool/+/827388 applied. i have no idea why :/ | 06:01 |
| ianw | nb02 has been upgraded to focal in-place. i will do nb01/nb03 tomorrow to ensure things are kept consistent | 06:01 |
| ianw | this way at least our test environment is more equal to production | 06:02 |
| *** marios is now known as marios|ruck | 06:05 | |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Add py36 variant of get-pip.py https://review.opendev.org/c/openstack/project-config/+/827418 | 07:10 |
| *** amoralej|off is now known as amoralej | 07:22 | |
| lourot | clarkb, re: master branch existence check, sure I'll have a look, thanks! | 08:18 |
| amoralej | may i get attention on https://review.opendev.org/c/zuul/zuul-jobs/+/826603 ? | 08:53 |
| opendevreview | Alfredo Moralejo proposed opendev/system-config master: Add CentOS SIGs for CentOS Stream 9 to AFS mirrors https://review.opendev.org/c/opendev/system-config/+/827441 | 09:13 |
| *** ysandeep is now known as ysandeep|lunch | 09:39 | |
| *** ysandeep|lunch is now known as ysandeep | 10:53 | |
| *** rlandy|out is now known as rlandy|ruck | 11:03 | |
| *** dviroel|out is now known as dviroel | 11:15 | |
| *** pojadhav is now known as pojadhav|brb | 12:19 | |
| *** pojadhav|brb is now known as pojadhav | 12:43 | |
| *** amoralej is now known as amoralej|lunch | 12:56 | |
| *** amoralej|lunch is now known as amoralej | 14:01 | |
| *** ysandeep is now known as ysandeep|dinner | 14:43 | |
| *** ysandeep|dinner is now known as ysandeep | 15:24 | |
| clarkb | infra-root I suppose today is probably as good as any to land https://review.opendev.org/c/opendev/system-config/+/826146 and then double check unattended-upgrades are still functional a day later? | 15:41 |
| clarkb | any objection to me approving that now? | 15:41 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Add py36 variant of get-pip.py https://review.opendev.org/c/openstack/project-config/+/827418 | 15:42 |
| fungi | clarkb: i say go for it | 15:43 |
| clarkb | thanks done | 15:45 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove unused puppet modules https://review.opendev.org/c/opendev/system-config/+/827525 | 16:07 |
| clarkb | thats step 0 to retiring repos. I wanted to make sure there weren't any leftovers in system-config that might confuse us. As far as I can tell that is the cleanup we can safely do today | 16:08 |
| *** pojadhav is now known as pojadhav|afk | 16:09 | |
| *** dviroel is now known as dviroel|lunch | 16:09 | |
| opendevreview | Alfredo Moralejo proposed opendev/system-config master: Add CentOS SIGs for CentOS Stream 9 to AFS mirrors https://review.opendev.org/c/opendev/system-config/+/827441 | 16:13 |
| amoralej | frickler, i left a comment in https://review.opendev.org/c/zuul/zuul-jobs/+/826603 wrt failing cs9 jobs, it's kind of chicken-and-egg we need that patch to be merge to get them fixed | 16:21 |
| amoralej | so my expectation is that, once we have updated the configure-mirrors, which is also enabling crb repo we can make them voting | 16:22 |
| clarkb | amoralej: you should be able to stack the changes and flip the change to voting on the child | 16:23 |
| amoralej | the problem is that those jobs are not using the proposed change in configure-mirrors until we merge it, that's how i get it | 16:23 |
| amoralej | clarkb, doesn't those jobs use the last commited? | 16:23 |
| clarkb | they may not pass immediately but you can recheck once the mirror is updated | 16:23 |
| amoralej | they will use roles from parent? | 16:23 |
| clarkb | amoralej: no they should largely be self testing | 16:23 |
| amoralej | but those ensure-pip jobs, i.e. are not taking the change in configure-mirror in the same review | 16:24 |
| amoralej | so it seems only changes in ensure-pip role are tested there? | 16:24 |
| clarkb | currently there is no relationship between them right? | 16:27 |
| clarkb | you have to set that relationship | 16:27 |
| opendevreview | Merged opendev/system-config master: Manage 10periodic and 20auto-upgrades together https://review.opendev.org/c/opendev/system-config/+/826146 | 16:27 |
| opendevreview | Merged openstack/project-config master: Add py36 variant of get-pip.py https://review.opendev.org/c/openstack/project-config/+/827418 | 16:27 |
| opendevreview | Alfredo Moralejo proposed zuul/zuul-jobs master: Make CentOS9 jobs voting https://review.opendev.org/c/zuul/zuul-jobs/+/827529 | 16:28 |
| *** ysandeep is now known as ysandeep|out | 16:28 | |
| fungi | amoralej: have you confirmed the failing job actually uses the configure-mirrors role and not the other one (the name of which escapes me at the moment)? | 16:29 |
| amoralej | i'd say so but lemme double check | 16:30 |
| amoralej | yes, they does, but they report "WARNING: CentOS mirrors are not supported either by this role yet. The execution of the job will continue without setting up cached mirrors." which is what i'm fixing | 16:31 |
| amoralej | i've piled a new review making the jobs voting, let's see if the child gets the change | 16:32 |
| amoralej | nop, it's just using it from master branch apparently | 16:35 |
| amoralej | so, how may i create that relationship for child reviews to use roles from parent? | 16:36 |
| clarkb | amoralej: fungi: the base job that configures the repos will use merged code. But then there should be role specific tests that use the updated code in a safe manner. | 16:36 |
| clarkb | what I was saying is that for the first part you may need to land the changes one at a time in order. But when you do that you can recheck the child change and confirm it is happy | 16:36 |
| amoralej | ok, sounds good to me | 16:37 |
| clarkb | And reviewers can approve the parents based on the test results from the sandboxed role tests if they exist | 16:37 |
| clarkb | But this way reviewers undersatnd the relationship is there | 16:37 |
| clarkb | infra-root https://etherpad.opendev.org/p/opendev-repo-retirements that list may not be complete but I Think it is a really good start. Can you look over the list and mark any that you think we shouldn't retire? Feel free to add additional repos that we should retire | 16:38 |
| *** marios|ruck is now known as marios|out | 16:48 | |
| clarkb | lourot: fungi: re https://review.opendev.org/c/openstack/project-config/+/827307/ it would be good to avoid doing this with a throwaway project too | 16:52 |
| clarkb | Somthing lower priority that won't be affected greatly by working through any issues that pop up is preferable | 16:53 |
| fungi | yeah, part of the problem with trying to find a balance there is that openstack will want its official deliverables to be consistent, so *if* they decide they're going to rename all their master branches to main, then doing it with something that isn't throw-away would be fine. the problem arises that if they do decide they won't be renaming branches in the other repos and whatever it is needs | 16:56 |
| fungi | to have its imported main branch renamed to master... also finding a completely disconnected new deliverable which won't need working integration testing with other projects that use different branch naming could be challenging | 16:56 |
| fungi | or finding something to import which doesn't need to be working for possibly months while we work through the logistical tests | 16:56 |
| clarkb | I agree. I'm just still minorly annoyed that we've got like 3 copies of all of openstack in our hosting system due to the way packaging repos work and would like yo avoid perpetuating that sort of issue if we can | 16:57 |
| fungi | ideally we'd test with the sandbox repo, but it already exists, so deleting it cleanly and reimporting is probably not a valid test | 16:58 |
| fungi | (mainly because i don't know how to truly delete it) | 16:59 |
| clarkb | ya. There is the delete repo plugin but as you say untested and who knows what it leaves behind | 17:00 |
| fungi | i'm definitely open to suggestions. it's just that finding a unicorn which is a project openstack will use eventually so it's not throw-away, but it doesn't need to be usable for the foreseeable future while we test, is tough | 17:02 |
| fungi | i do agree it feels "dirty" to make a repo we know we're just going to retire | 17:03 |
| fungi | from a governance perspective it would probably be fine to just assign it to the tact sig until we're done with it | 17:03 |
| clarkb | fungi: lourot: a compromise might be to use a smalelr dummy repo than one with 1.5k commits? | 17:05 |
| fungi | ahh, i didn't even look at which one was being imported. but yeah, we don't need something massive | 17:06 |
| *** dviroel|lunch is now known as dviroel | 17:12 | |
| *** lbragstad9 is now known as lbragstad | 17:19 | |
| jrosser | could i get some advice on this https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827400 | 17:30 |
| jrosser | i feel like this is a catch-22 somehow | 17:31 |
| fungi | jrosser: insofar as that we'll need to remove the nodeset reference from all branches at the same time? | 17:32 |
| jrosser | if you look at the topic on that patch there are tons of similar patches that i've been able to merge | 17:33 |
| jrosser | but i really don't know why this one fails instantly like that, ironically when it's the very job i'm trying to remove /o\ | 17:34 |
| fungi | yes, but the nodeset definition was taken out early today (or last night depending on where in the world you are) | 17:34 |
| jrosser | others are working where i just remove jobs though, i'm kind of confused | 17:34 |
| fungi | ahh | 17:34 |
| fungi | well, other infra-root folk might have an idea i'm not coming up with, but worst case i can bypass zuul to merge those changes directly via the gerrit api and get things unstuck | 17:35 |
| jrosser | it is getting to that point i think | 17:35 |
| jrosser | i'm kind of tired of this after spending nearly two weeks on it | 17:35 |
| fungi | yeah, we didn't want to have it be so sudden, but the centos mirror operators didn't really give us much choice. by the time we realized they'd ripped all the packages out of the mirrors, we'd already copied that to ours | 17:36 |
| jrosser | indeed | 17:36 |
| clarkb | I think that usually happens beacuse something is overly aggressive about applying configs from one branch to another | 17:37 |
| jrosser | to make merging my job removal patches harder, that mirror removal has also broken all my stable branches due to an unrelated bug | 17:37 |
| jrosser | i have to go do childcare now but i may need some help to progress these | 17:38 |
| fungi | in this case there are two removal changes for openstack-ansible-os_keystone which are failing (victoria and wallaby) so it's presumably something to do with the config in that particular repo | 17:38 |
| jrosser | the comments that zuul has left on https://review.opendev.org/c/openstack/openstack-ansible/+/824570 kind of sums up the horror of this | 17:39 |
| fungi | and yes, to clarkb's point, there might be explicit branch matchers in the configs of those branches? | 17:39 |
| clarkb | side note: ubuntu and debian seem to keep the package repos up for what may as well be forever | 17:42 |
| clarkb | its a bit disappointing that centos-8 gave everyone a month and then sent it. I mean I wanted to clean things up on our side too but even then was willing to work with people to clean it up overtime | 17:42 |
| fungi | jrosser: when you have a moment to put together a list of which changes you think need to bypass testing in order to be able to get the others running clean, i'm happy to help try and get you unstuck. just a word of warning, any time i think it's safe to bypass testing on an untested change, i inevitably end up merging a new bug in the process :/ | 17:44 |
| clarkb | just as a sanity check I've looked at the zuul scheduler log for that change reporting and it doesn't say anything more | 17:45 |
| clarkb | so zuul isn't hiding any info from us. | 17:45 |
| fungi | yeah, so i guess turning on debug in the patch wouldn't help in that case | 17:45 |
| clarkb | I wonder if the required-projects list for the OSA functional jobs forces all of them to be evaluated | 17:46 |
| clarkb | whcih leads to this sort of problem where you get a chicken and egg | 17:46 |
| clarkb | or maybe the issue is just that the entire os_keystone repo's config across all branches is validated and since you can't land a change to all branches at the same time it leads to this | 17:47 |
| clarkb | corvus: ^ are either of those possibilities somethign that you would expect zuul to do ? | 17:47 |
| corvus | aroo? reading | 17:48 |
| fungi | tl;dr is opendev had to remove the centos-8 nodeset, it's being referenced in jobs used in multiple branches of a repository, and seems like we have to simultaneously remove it from multiple branches since the per-branch removal patches are erroring with zuul complaining it can't evaluate the configuration | 17:51 |
| fungi | notably, the configuration on those branches refer to jobs in another repository, so it seems like a deadlock | 17:52 |
| corvus | i see that zuul is saying you can't remove that job definition because it's used in pipelines of other repos | 17:52 |
| corvus | like, you have to remove it from the check+gate pipelines of os_manila before you can delete the job definition | 17:52 |
| fungi | clarkb: maybe terribly idea, but what if we reintroduced the centos-8 nodeset and made it based on the centos-9-stream label? | 17:52 |
| fungi | er, centos-8-stream label i mean | 17:53 |
| clarkb | fungi: that would probably work. Jobs might fail but they are likely going to do that anyway | 17:53 |
| fungi | jobs referencing the label go from being unresolvable configuration to being maybe-broken builds, but unconfiguring those is likely simpler | 17:53 |
| corvus | maybe i'm missing something, but the change you linked isn't complaining about labels.... | 17:54 |
| clarkb | corvus: https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/827400 that change | 17:54 |
| corvus | oh i was looking at 824570 | 17:55 |
| clarkb | the one with os_manila appears to be in process via https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/827515 and I expect that stack is resolvable | 17:55 |
| fungi | yeah, there are possibly multiple sources of deadlocks here too. restoring the nodeset might resolve some and not all | 17:55 |
| corvus | anyone have a link to the nodeset removal change? | 17:55 |
| * fungi gets it | 17:55 | |
| clarkb | corvus: https://review.opendev.org/c/opendev/base-jobs/+/827181 | 17:55 |
| fungi | yeah, beat me to it | 17:56 |
| corvus | okay, good, zuul did report that would be a problem but for a tenant that doesn't vote on that change :) | 17:57 |
| fungi | yes, zuul did the right things. we knew it was going to lead to breakage in the openstack tenant | 17:57 |
| clarkb | yes and I think we knew it would be a problem but then redhat/centos forced our hand by completely breaking the distro | 17:57 |
| clarkb | I think what we're confused about now is why a change that removes the use of that label fails because it complains about hte use of the label | 17:58 |
| corvus | so yeah, i think you should restore the centos-8 nodeset, then unwind from the outside in | 17:58 |
| fungi | i've got the change about ready to push | 17:58 |
| corvus | AH that's the question :) | 17:58 |
| corvus | zuul freezes the old and new configuration :/ | 17:58 |
| corvus | that's how the automatic job delta detection works | 17:59 |
| clarkb | aha | 17:59 |
| fungi | ohhhhhh | 17:59 |
| clarkb | and it is the old side that is failing | 17:59 |
| corvus | so it probably failed on the old side | 17:59 |
| corvus | that could probably be made more robust | 17:59 |
| clarkb | fungi: and if you push a revert up too then we can recheck the revert until it doesn't error from the other tenants | 18:00 |
| corvus | would either just run too many or not enough jobs, but better than failing | 18:00 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Partially revert "Remove centos-8 as it is EOL" https://review.opendev.org/c/opendev/base-jobs/+/827535 | 18:00 |
| fungi | i didn't bring back the arm64 nodeset, as i'm not sure it's involved | 18:00 |
| clarkb | fungi: ya I think we can take it one step at a time +2 from me | 18:01 |
| clarkb | Is anyone interested in reviewing the threaddump that corvus took of gerrit during the git pull slowness? I'm going to try and take a look at it myself today and attach it to the issue we created and will skim it. But I expect it is quite large and a second set of eyes may be a good idea | 18:04 |
| fungi | looking for sensitive data? | 18:05 |
| fungi | someone copied it out of the container fs right? | 18:05 |
| fungi | looks like it's ~clarkb/dump | 18:06 |
| *** amoralej is now known as amoralej|off | 18:08 | |
| fungi | it does contain client ip addresses | 18:09 |
| fungi | only one, and it seems to be on line 4831 | 18:10 |
| clarkb | ya sensitive info like that or secrets etc | 18:11 |
| fungi | i don't see any obvious signes of keys or credential strings or e-mail addresses | 18:11 |
| clarkb | looks like there are usernames | 18:14 |
| clarkb | the N/A where the IP addr is that you found would be a username if it was authenticated. The ssh threads have usernames but no IPs | 18:15 |
| clarkb | I'll work on scrubbing that | 18:15 |
| fungi | oh, yep i see zuul's username for example | 18:16 |
| clarkb | I think we're ok leaving zuul in there. But I'll scrub the other users out | 18:17 |
| clarkb | fungi: I'm working on dump_sanitized in the same dir and have corrected the things we've identified so far | 18:18 |
| fungi | thanks | 18:21 |
| clarkb | 'H2 File Lock Watchdog' has some file paths in them, but I don't think they are sensitive | 18:23 |
| fungi | yeah, that seems fine | 18:25 |
| clarkb | there are replication threads that disclose our replication targets btu that info is already public | 18:29 |
| clarkb | I suspect that SSH-Interactive-Worker-85 may be the sort of info that gerrit would be looking at to investigate this issue. Though I'm not sure I undersatnd it fully (I also don't think it exposes anything it shouldn't) | 18:34 |
| clarkb | ok I've gone over the whole file skimming it and updating things in dump_sanitized. The end result was removing several usernames and one ip address. The diff between the files shows this. fungi maybe you want to look it over and make sure I didn't miss anything in the cleanup? | 18:43 |
| fungi | sure, checking | 18:44 |
| clarkb | I went ahead and approved the partial revert of the centos-8 nodeset removal change | 18:44 |
| fungi | thanks | 18:49 |
| fungi | clarkb: the sanitized dump lgtm | 18:51 |
| fungi | i don't see any more obviously sensitive info in it | 18:51 |
| clarkb | cool I'll give it a few more hours in case anyone else wants to look then can attach it to the issue | 18:51 |
| fungi | sounds great | 18:52 |
| opendevreview | Merged opendev/base-jobs master: Partially revert "Remove centos-8 as it is EOL" https://review.opendev.org/c/opendev/base-jobs/+/827535 | 18:53 |
| *** artom__ is now known as artom | 18:53 | |
| clarkb | I rechecked the change jrosser pointed out and it is gating now | 19:02 |
| fungi | jrosser: ^ may be worth rechecking the changes which complained about the missing nodeset, since it's "sort of" back for now | 19:02 |
| clarkb | Looks like it is running different centos-8 jobs still which may have been part of the problem too | 19:03 |
| jrosser | oh cool, thankyou all for taking a look at that | 19:05 |
| jrosser | i know where those other jobs are from, it's this https://review.opendev.org/c/openstack/openstack-ansible/+/824570 | 19:05 |
| jrosser | the patch to os_keystone was removing jobs defined and used in that repo | 19:06 |
| clarkb | yup but if it can't run the jobs to check/gate the repo it would still fail I think. Hopefully the nodeset with the hacked label for stream will get things moving forward | 19:06 |
| jrosser | but there are project templates defined outside that repo which also carry centos-8 jobs | 19:06 |
| clarkb | I spot checked the unattended upgrades config change and it appears to ahve applied as expected. Same source file into two different files managed by the different involved packages | 19:30 |
| clarkb | lgtm is what I'm saying :) | 19:30 |
| fungi | awesome | 19:31 |
| *** timburke_ is now known as timburke | 20:57 | |
| clarkb | infra-root when do you think we should rip out the gerrit 3.3 images? Has enough time passed? | 21:02 |
| clarkb | I ask beacuse https://review.opendev.org/c/opendev/system-config/+/824039 is impacted by that. Maybe land this chagne first and then do cleanup? | 21:02 |
| ianw | yeah i think we're unlikely to go back now | 21:03 |
| ianw | clarkb: especially since we have a java developer on the team ;) | 21:03 |
| clarkb | >_> | 21:04 |
| clarkb | ianw: if you have time can you check the review02:~clarkb/dump and ~clarkb/dump_sanitized files to make sure we aren't overdisclosing anything if I upload dump_sanitized to the gerrit bug tracker? | 21:05 |
| clarkb | dump is the original and dump_sanitized I've removed some content from | 21:05 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove unused puppet modules https://review.opendev.org/c/opendev/system-config/+/827525 | 21:08 |
| clarkb | I left one of my local notetaking comments in ^ and it was bothering me. :) | 21:08 |
| ianw | yep, one sec | 21:11 |
| admin1 | hi .. is there a nova command to see all ongoing migrations ? | 21:16 |
| clarkb | admin1: we run the software development tools for openstack but don't haev much experience running openstack ourselves. You'll probably have better luck in #openstack (I don't know the answer to your question) | 21:17 |
| admin1 | sorry .. wrong channel :) | 21:19 |
| admin1 | it was meant for #openstack-nova | 21:19 |
| *** dviroel is now known as dviroel|out | 21:22 | |
| ianw | clarkb: thanks for approving through https://review.opendev.org/c/zuul/nodepool/+/827388 | 21:23 |
| clarkb | and thank you for debugging even if we don't fully understand it yet | 21:24 |
| ianw | might have noticed from scrollback, in desperation i in-place upgraded nb02 yesterday. i'll do that for the rest of the builders today to keep things in sync | 21:24 |
| clarkb | ++ | 21:24 |
| ianw | https://grafana.com/blog/2022/02/02/announcing-grafana-incident-smart-incident-management-for-your-teams/ | 21:25 |
| ianw | it is cool | 21:25 |
| ianw | not sure how much more effective it is than irc and an etherpad though | 21:25 |
| clarkb | having the graphs overlaid with actions and checkpoints is nice for tracking changes to the system | 21:26 |
| clarkb | I think that is what they are showing the system does | 21:26 |
| ianw | (this comment will probably age like the famous HN "dropbox is stupid, you can do that with a vps and rsync") | 21:26 |
| * join_subline irc+etherpad masterrace 👑 | 21:36 | |
| fungi | clarkb: i was about to ask about ripping out gerrit 3.3 images yesterday. seems like it's time to me | 21:41 |
| clarkb | excellent. I guess if we can land that reordering of the plugins first thenwe can base changes to remove 3.3 off of that | 21:42 |
| clarkb | for some reason I thought we had conflicted with that reordering somehow but seems like we haven't | 21:42 |
| clarkb | the melody addition must be what I thought would do it | 21:43 |
| opendevreview | Ramil Minishev proposed openstack/diskimage-builder master: Make growvols config path platform independent https://review.opendev.org/c/openstack/diskimage-builder/+/827557 | 21:50 |
| opendevreview | Ramil Minishev proposed openstack/diskimage-builder master: Add grow_gpt to growvols https://review.opendev.org/c/openstack/diskimage-builder/+/827558 | 21:50 |
| opendevreview | Ramil Minishev proposed openstack/diskimage-builder master: Make xfs_growfs execution back-compatible https://review.opendev.org/c/openstack/diskimage-builder/+/827559 | 21:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop building Gerrit 3.3 images https://review.opendev.org/c/opendev/system-config/+/827561 | 21:59 |
| clarkb | Something like that ? | 21:59 |
| fungi | yep | 21:59 |
| opendevreview | Ramil Minishev proposed openstack/diskimage-builder master: Add grow_gpt to growvols https://review.opendev.org/c/openstack/diskimage-builder/+/827558 | 22:00 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Switch jeepyb over to building Gerrit 3.4 images https://review.opendev.org/c/openstack/project-config/+/827562 | 22:03 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop building Gerrit 3.3 images https://review.opendev.org/c/opendev/system-config/+/827561 | 22:03 |
| ianw | i'm going to do those builders now | 22:21 |
| opendevreview | Merged openstack/project-config master: Switch jeepyb over to building Gerrit 3.4 images https://review.opendev.org/c/openstack/project-config/+/827562 | 23:08 |
| ianw | #status log in-place upgraded nb0<1,2,3>.opendev.org to focal to better match production to our gate test environment | 23:09 |
| opendevstatus | ianw: finished logging | 23:09 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch nb01 to focal in testing https://review.opendev.org/c/opendev/system-config/+/827572 | 23:30 |
| clarkb | ianw: ^ updated to match in system-config-run | 23:30 |
| ianw | clarkb: good catch, thanks | 23:34 |
| opendevreview | Merged opendev/system-config master: Better organize gerrit plugins in job defs https://review.opendev.org/c/opendev/system-config/+/824039 | 23:51 |
| opendevreview | Merged opendev/system-config master: Remove unused puppet modules https://review.opendev.org/c/opendev/system-config/+/827525 | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!