| clarkb | does ansible really have no way to do a file lookup against a remote file? | 00:03 |
|---|---|---|
| clarkb | I'd like to generate an ssh key pair and then put the pubkey into the test gitea. But if I do that the key is on the remote and you can't read the pubkey back in to upload with the uri module without first copying that file to the host running ansible? | 00:04 |
| corvus | clarkb: slurp handles that | 00:04 |
| corvus | (it's awkward, but that's the mechanism) | 00:05 |
| clarkb | corvus:thanks. I found a stackoverflow for this and its the same exact use case too (grabbing public keys) | 00:05 |
| clarkb | corvus: it seems weird to have multiple mechanism for this that work in different contexts. But slurp is what I needed thanks | 00:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: Add introduction to the container jobs documentation https://review.opendev.org/c/zuul/zuul-jobs/+/798969 | 00:06 |
| corvus | clarkb: don't forget the b64decode | 00:06 |
| clarkb | ++ | 00:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 00:09 |
| clarkb | corvus: ^ I think I got it right there. Thanks again. fungi ^ that should test that pushing over ssh works | 00:10 |
| clarkb | (or doesn't I suppose that would be a valid result of this. We may want to split the ssh push out into a separate change on top of 1.15.11 but one thing at a time) | 00:10 |
| fungi | maybe only bother to split it out if it fails | 00:11 |
| clarkb | wfm | 00:12 |
| opendevreview | Ian Wienand proposed opendev/lodgeit master: Default to private pastes https://review.opendev.org/c/opendev/lodgeit/+/828201 | 00:25 |
| opendevreview | Ian Wienand proposed opendev/system-config master: lodgeit: switch pastes to be private by default https://review.opendev.org/c/opendev/system-config/+/828202 | 00:46 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 01:08 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh https://review.opendev.org/c/opendev/system-config/+/828203 | 01:08 |
| clarkb | the test push over ssh did end up failing so I've split it in two so that we can ensure it is working on 1.15.11 before digging in too much | 01:08 |
| ianychoi[m] | Hi ianw, hope that you have chance to see translation job failures after https://review.opendev.org/c/zuul/zuul-jobs/+/827588 (from some investigation starting from https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-02-07.log.html#t2022-02-07T14:05:29 ) | 01:12 |
| ianychoi[m] | While an ultimate goal should be to migrate Zanata to another new translation platform, would you help some issues regarding wheels? | 01:13 |
| fungi | ianychoi[m]: as further investigation revealed, the various failures seem to have started at roughly the same time zuul-jobs switched the ensure-sphinx role to using python3 instead of python2 | 01:17 |
| fungi | i expect explains the coincident timing for all of the various failures you found | 01:18 |
| fungi | it's complicated by the fact that those jobs apply openstack/requirements upper-constraints.txt for their corresponding branches, some of which dropped support for ubuntu-bionic a while ago | 01:19 |
| fungi | i expect the lingering support for python 2.7 in constraints is the only reason they were working prior to the ensure-sphinx role change | 01:19 |
| fungi | we need to brainstorm some options, though if zanata-client really won't work with any jdk version available for ubuntu focal, then that's sure to complicate the available solutions | 01:21 |
| ianw | (sorry, just got to finish something, bib) | 01:24 |
| ianw | hrm, sorry i hadn't seen that | 01:48 |
| ianw | perhaps we need to stick it in a continer | 01:54 |
| ianw | container | 01:54 |
| ianw | let me start an etherpad page, as it seems we have a bit to untangle | 01:54 |
| *** rlandy|ruck is now known as rlandy|out | 02:03 | |
| ianw | focal also has java-8 | 02:21 |
| ianw | https://etherpad.opendev.org/p/translation-job-failures-02-2022 | 02:26 |
| *** pojadhav|afk is now known as pojadhav | 02:36 | |
| ianw | ok, i at least have a root cause for what's going on with the wheel generation | 03:14 |
| ianw | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/828213 wheel-cache: update openstack package for 8-stream [NEW] | 03:39 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh https://review.opendev.org/c/opendev/system-config/+/828203 | 03:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 03:55 |
| *** ysandeep|out is now known as ysandeep | 03:57 | |
| mnasiadka_ | fungi: fantastic, thanks | 06:33 |
| opendevreview | ZhouHeng proposed openstack/project-config master: Re-maintenance neutron-fwaas* https://review.opendev.org/c/openstack/project-config/+/825412 | 06:55 |
| *** ysandeep is now known as ysandeep|afk | 07:11 | |
| *** persia is now known as Guest2205 | 07:19 | |
| *** amoralej|off is now known as amoralej | 07:33 | |
| opendevreview | Merged openstack/project-config master: Retire security-specs repo - Step 3 https://review.opendev.org/c/openstack/project-config/+/827178 | 07:35 |
| *** jpena|off is now known as jpena | 08:03 | |
| *** ysandeep|afk is now known as ysandeep | 08:45 | |
| ianw | fungi / ianychoi[m]: it looks to me like a confluence of issues. i'll try to pick things apart, but feel free to add to https://etherpad.opendev.org/p/translation-job-failures-02-2022 and we can see what we can do | 09:07 |
| opendevreview | Merged zuul/zuul-jobs master: Make CentOS9 jobs voting https://review.opendev.org/c/zuul/zuul-jobs/+/827529 | 09:19 |
| opendevreview | Aurelien Lourot proposed openstack/project-config master: Mirror charm-ops-interface-* to GitHub https://review.opendev.org/c/openstack/project-config/+/828166 | 09:33 |
| opendevreview | Riccardo Pittau proposed openstack/diskimage-builder master: [WIP] Fallback to persistent netifs names with systemd https://review.opendev.org/c/openstack/diskimage-builder/+/828266 | 09:59 |
| *** dviroel|out is now known as dviroel|ruck | 10:05 | |
| mnasiadka_ | How often the CentOS mirror sync is running? http://mirror.centos.org/centos/8/storage/x86_64/ceph-octopus/ seems to be missing from OpenDev mirrors. | 10:31 |
| frickler | mnasiadka_: I see https://mirror.iad3.inmotion.opendev.org/centos/8-stream/storage/x86_64/ceph-octopus/ , are you using the correct URL? We don't mirror centos/8 anymore afaict | 10:45 |
| mnasiadka_ | let me see - Status code: 404 for http://mirror-int.dfw.rax.opendev.org/centos/8/storage/x86_64/ceph-nautilus/repodata/repomd.xml | 10:46 |
| mnasiadka_ | hmm, why centos-release-ceph-nautilus package points to 8 | 10:47 |
| frickler | mnasiadka_: IIUC RDO had to make similar adjustments. to answer your original question, mirror jobs should be running every 2 hours and https://mirror.iad3.inmotion.opendev.org/centos/timestamp.txt should tell you when the last successful sync and publish happened | 10:53 |
| mnasiadka_ | frickler: thanks | 10:54 |
| opendevreview | Riccardo Pittau proposed openstack/diskimage-builder master: [WIP] Fallback to persistent netifs names with systemd https://review.opendev.org/c/openstack/diskimage-builder/+/828266 | 11:08 |
| *** rlandy|out is now known as rlandy|ruck | 11:11 | |
| *** ysandeep is now known as ysandeep|mtg | 12:01 | |
| *** pojadhav is now known as pojadhav|afk | 12:33 | |
| opendevreview | Riccardo Pittau proposed openstack/diskimage-builder master: [WIP] Fallback to persistent netifs names with systemd https://review.opendev.org/c/openstack/diskimage-builder/+/828266 | 12:37 |
| *** ysandeep|mtg is now known as ysandeep | 12:55 | |
| *** ysandeep is now known as ysandeep|coffee | 13:04 | |
| *** amoralej is now known as amoralej|lunch | 13:18 | |
| *** pojadhav|afk is now known as pojadhav | 13:31 | |
| *** ysandeep|coffee is now known as ysandeep | 13:39 | |
| *** amoralej|lunch is now known as amoralej | 14:22 | |
| opendevreview | Attila Fazekas proposed opendev/elastic-recheck master: Revert "Add query for WARN: failed: route add -net bug 1857151" https://review.opendev.org/c/opendev/elastic-recheck/+/828355 | 14:52 |
| *** akaha|rover is now known as akahat|dinner | 14:55 | |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:11 | |
| *** ysandeep is now known as ysandeep|dinner | 15:14 | |
| *** pojadhav is now known as pojadhav|dinner | 15:24 | |
| sshnaidm | how can I delete a branch in gerrit repo? | 15:33 |
| sshnaidm | (can I?) | 15:33 |
| fungi | sshnaidm: if it's in an openstack project, the release managers have the necessary permissions to call the branch deletion api method in gerrit's rest api | 15:35 |
| sshnaidm | fungi, I'm not sure what is release manager in repo openstack/ansible-collections-openstack | 15:36 |
| sshnaidm | I can tag there and release.. | 15:36 |
| sshnaidm | but I couldn't delete a branch | 15:36 |
| fungi | sshnaidm: the openstack release team has branch deletion permissions over all repositories in the openstack/ namespace | 15:37 |
| sshnaidm | fungi, ack, so I need to ask on #openstack-release? | 15:37 |
| fungi | that's probably the easiest way to find them, yes | 15:37 |
| sshnaidm | fungi, cool, thanks | 15:37 |
| *** ysandeep|dinner is now known as ysandeep | 15:44 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh https://review.opendev.org/c/opendev/system-config/+/828203 | 15:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 15:55 |
| clarkb | if that doesn't make things clear I'll probably need to hold the nodes | 15:55 |
| *** pojadhav|dinner is now known as pojadhav | 15:58 | |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/project-config master: Give perm to release team to delete branches https://review.opendev.org/c/openstack/project-config/+/828371 | 15:58 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:06 | |
| clarkb | I've approved the pastes are private by default changes | 16:40 |
| fungi | oh! i missed those even got pushed | 16:51 |
| clarkb | fungi: https://review.opendev.org/c/opendev/lodgeit/+/828201 wasthe main one | 16:51 |
| clarkb | then on the system-config side we updated our test job to check it | 16:51 |
| opendevreview | Merged opendev/lodgeit master: Default to private pastes https://review.opendev.org/c/opendev/lodgeit/+/828201 | 17:02 |
| *** jpena is now known as jpena|off | 17:04 | |
| *** akahat|dinner is now known as akahat|rover | 17:05 | |
| clarkb | infra-root: today is EMS billing day. I didn't see the email notifying us of payment so logged in to check on it. I think everything is ok, but one weird thing is when you login it sends you straight to the setup a new home server wizard and asks for your name. I set that before I realized what it was asking for and I think it changed the string from what it was set to before. I've | 17:13 |
| clarkb | since reset it back to what it should've been before. I don't expect any issues but heads up that we should double check both this month and next month are handled without trouble. | 17:13 |
| clarkb | I guess beware of that logging into the management system you can click on the drop down in the top right to get to the account and existing homeserver bypassing the setup thing. I think their login links are all geared towards people signing up for new service ... | 17:14 |
| fungi | interesting, thanks for the heads up | 17:15 |
| *** sshnaidm is now known as sshnaidm|afk | 17:18 | |
| *** ysandeep is now known as ysandeep|out | 17:21 | |
| clarkb | nice my gitea ssh testing change finally passes. Once I get data from the child change I can decide if I need to clean up the extra debugging output that is likely to get in the way of typical failure debugging | 17:31 |
| clarkb | apparently the trick was to be very specific about which key to use to auth | 17:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh https://review.opendev.org/c/opendev/system-config/+/828203 | 17:37 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 17:37 |
| fungi | also that pr you linked makes me glad i don't maintain any go-based software. having to worry about runtime linker differences between root and rootless containers... | 17:37 |
| clarkb | infra-root the parent change in that stack should be good to go whenever we are happy with it | 17:37 |
| clarkb | the child change may deserve more vetting via a held node based on what careful review of the changelog reveals | 17:38 |
| clarkb | but ssh pushing does seem to pass so I must've gotten the container image updates related to that at least semi correct | 17:38 |
| *** amoralej is now known as amoralej|off | 17:39 | |
| clarkb | infra-root and looks like today's billing notice came through and looks good. I guess we just watch out for next month but I expect it is fine | 17:41 |
| clarkb | luca mentioned on the mailing list that a gerrit 3.5.1 should happen soon to fix a number of issues. Just calling that out as we ramp up 3.5 investigations | 19:01 |
| clarkb | I'm going to send that service coordinator email now (or as soon as I can get it written) | 20:00 |
| fungi | thanks! | 20:02 |
| ianw | fungi: ok, always trust the todo list :) | 20:05 |
| fungi | heh | 20:05 |
| ianw | backup01.ord.rax.opendev.org:/opt/bup-202007 is now mounted | 20:06 |
| fungi | though i went poking on the server out of curiosity and found a /var/lib/redis/dump.rdb.2020-08-21.upgrade | 20:06 |
| ianw | so indeed i had *not* cleaned that up, and cleaning it up is still a todo item :) | 20:06 |
| ianw | now, how to extract anything from bup is an open question | 20:07 |
| fungi | it may be easier to use the manual db dump i found since it's from roughly the same timeframe as that bup volume? | 20:07 |
| fungi | but that still leaves me with being unsure how to use that dump short of booting a new server | 20:08 |
| ianw | the only thing is didn't you need a 2019 version? | 20:08 |
| ianw | that far back might be in the bup history | 20:08 |
| fungi | well, "2019 q3" is when he said he thinks he last checked that it was intact, no idea how soon after that it was broken | 20:09 |
| clarkb | ok email sent. Sorry for missing that previously. And now lunch | 20:10 |
| fungi | i suppose i could have rackspace live snapshot that server, boot a copy of it at a new ip address, and then try to import the 2020-08-21 dump into its redis. that might be fairly low-effort | 20:11 |
| fungi | and low-risk | 20:12 |
| clarkb | ya that would be one approach to the deploy a new ethercalc to look at the contents of the redis db | 20:13 |
| clarkb | its possible the redis db is human readable too. Though I doubt it | 20:13 |
| fungi | file says "/var/lib/redis/dump.rdb.2020-08-21.upgrade: data" so probably not, no | 20:13 |
| ianw | i don't think i realised ethercalc was not containerised yet | 20:14 |
| fungi | yeah, it's one of those remaining xenial servers i mentioned | 20:15 |
| ianw | https://hub.docker.com/r/audreyt/ethercalc looks maintained | 20:16 |
| ianw | theoretically, that seems like it would just read the db dump | 20:17 |
| *** prometheanfire is now known as Guest0 | 20:56 | |
| *** osmanlicilegi is now known as Guest1 | 20:56 | |
| opendevreview | Merged openstack/diskimage-builder master: Add new container element - Rocky Linux https://review.opendev.org/c/openstack/diskimage-builder/+/825957 | 21:24 |
| *** Guest0 is now known as prometheanfire | 21:56 | |
| clarkb | Gerrit java 17 work has begun. Sounds like they have identified a number of things they need to address. This is good news beacuse java 17 supposedly performs quite well | 22:00 |
| fungi | oh, very cool! | 22:01 |
| *** dviroel|ruck is now known as dviroel|ruck|afk | 22:02 | |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 test node https://review.opendev.org/c/openstack/project-config/+/828435 | 22:57 |
| opendevreview | Neil Hanlon proposed opendev/base-jobs master: Add rockylinux nodeset https://review.opendev.org/c/opendev/base-jobs/+/828436 | 22:58 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 test node https://review.opendev.org/c/openstack/project-config/+/828435 | 22:58 |
| clarkb | ianw: NeilHanlon ^ I left a comment on that but what I realized is I think we need a dib release | 23:02 |
| clarkb | so that nodepool has access to the updated rocky image building stuff | 23:02 |
| ianw | yes we do need a release for that, and a bump in nodepool requirements.txt to pull in that release | 23:03 |
| NeilHanlon | ah, i see thank clarkb - i was looking at https://opendev.org/openstack/project-config/commit/dce378a6b46ebcbd4a88a0306518d1d69917524f for inspiration | 23:04 |
| ianw | although the package maps will work, i think there's quite a few points in project-config/nodepool/elements that are matching on "centos" specificially | 23:04 |
| clarkb | NeilHanlon: adding the bits that were removed from that commit should get you most of the way there. The dib release is probably the biggest not straightforward thing, but also those updates tend to not be too bad and ianw has been driving them periodically for us for other random features | 23:05 |
| clarkb | ianw: oh that is a good point too. Is there a way to match centos or rocky other than if centos or rocky being checked directly? | 23:05 |
| ianw | i don't think so, unfortunately. i think it will mostly look like openeuler as a separate match | 23:06 |
| clarkb | got it | 23:06 |
| ianw | i don't exactly know what openeuler is based on, but it may not be very far from what rocky is doing | 23:06 |
| clarkb | ya I think it is also a rhel derivative so openeuler is probably a good reference as far as bits to toggle | 23:07 |
| clarkb | ianw: maybe land the fedora-35 locale fix and then do a release? Not sure what else has landed though | 23:07 |
| ianw | yeah, in #openstack-dib stevebaker[m] asked me to look at some things too. i'll do that today and i think we'll be in shape to release | 23:08 |
| clarkb | nice | 23:08 |
| NeilHanlon | are the nodepool/nlXX.opendev.org.yaml files auto-generated or do they need to be touched up manually? | 23:09 |
| ianw | NeilHanlon: they are all under manual control | 23:09 |
| NeilHanlon | gotcha, thanks | 23:09 |
| opendevreview | Merged opendev/system-config master: lodgeit: switch pastes to be private by default https://review.opendev.org/c/opendev/system-config/+/828202 | 23:18 |
| clarkb | ianw: re ^ I think we may have to wait for the daily periodic job for lodgeit or manually pull and restart. I'm not in a hurry but thought I'd call it out | 23:18 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 23:20 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 23:24 | |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 23:25 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: centos-8 : fail if pointing to centos-8-stream image type https://review.opendev.org/c/opendev/base-jobs/+/828437 | 23:28 |
| opendevreview | Neil Hanlon proposed opendev/base-jobs master: Add rockylinux nodeset https://review.opendev.org/c/opendev/base-jobs/+/828436 | 23:28 |
| ianw | clrkb/fungi: ^ that's where I ended up after our discussion in the meeting about failing the centos-8 node type | 23:28 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 23:28 |
| fungi | ianw: oh, that's a nice lightweight solution | 23:29 |
| fungi | however, testing it will be key | 23:30 |
| fungi | a typo in there will go unnoticed until it merges, and could break every single job we run until reverted (bypassing testing, because it would break its own revert too) | 23:31 |
| ianw | yeah, if you agree on the general vibe, i think we can merge it to base-test and try | 23:31 |
| clarkb | I guess that will work as long as no one else is setting nodesets that way | 23:32 |
| clarkb | I'm not opposed to test it | 23:32 |
| ianw | the only corner case i can think of is if someone sets centos-8 manually as centos-8-stream | 23:32 |
| ianw | heh, what clarkb said :) i codesearched and couldn't find anything like that | 23:32 |
| fungi | i too wondered about that, but suspected it was a fairly unlikely configuration | 23:35 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: base: fail centos-8 if pointing to centos-8-stream image type https://review.opendev.org/c/opendev/base-jobs/+/828437 | 23:38 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: base-test: sync with base/pre.yaml https://review.opendev.org/c/opendev/base-jobs/+/828439 | 23:38 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: base-test: fail centos-8 if pointing to centos-8-stream image type https://review.opendev.org/c/opendev/base-jobs/+/828440 | 23:38 |
| ianw | fungi: if you could check https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/828219 -- i'll try to look at the other translate failures soon and see if they sync with that being the problem | 23:47 |
| fungi | sure thing | 23:55 |
| fungi | ianw: were the errors coming from that venv? the failures i looked at were installing doc/source/requirements.txt into the sphinx venv | 23:57 |
| fungi | er, doc/requirements.txt i mean | 23:59 |
| ianw | entirely possible i misread it | 23:59 |
| ianw | https://etherpad.opendev.org/p/translation-job-failures-02-2022 is my notes | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!