| ianw | hrm | 00:02 |
|---|---|---|
| puck | Ideally I'd just change the email address in review.opendev.org to match my current work email address. | 00:03 |
| ianw | we probably need to add the openid as an externalaccount | 00:09 |
| ianw | i'm just trying to see if we have any prior history of this. we might have to loop in clarkb, but he is currently on ~ UTC time | 00:18 |
| *** rlandy|bbl is now known as rlandy | 00:19 | |
| ianw | ahh, ok, i can see a new account created actually now | 00:20 |
| ianw | puck: it's created a login for you now with an email @catalystcloud.nz | 00:29 |
| puck | Yeah, I created that account earlier today before I remembered I already had an identity. | 00:29 |
| ianw | ah, and you can log in with that ok? | 00:30 |
| puck | Yup | 00:30 |
| ianw | what we want to do then is make 9950 inactive, then you can add the catalyst.net.nz account to this new account | 00:31 |
| puck | Okay, that sounds ideal | 00:31 |
| ianw | puck: ok, that is done, try adding the catalyst.net.nz email and it should allow you | 00:36 |
| puck | Error 422 (Unprocessable Entity): Identity 'mailto:puck@catalyst.net.nz' in use by another account | 00:37 |
| puck | Can you change the other one to be something like puck+old@catalyst.net.nz ? | 00:38 |
| *** rlandy is now known as rlandy|out | 00:41 | |
| ianw | puck: try again now | 00:44 |
| puck | Woohoo, done! Thank you | 00:48 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit docs: cleanup and use shell-session https://review.opendev.org/c/opendev/system-config/+/845066 | 00:56 |
| ianw | clarkb: ^ i'll sync with you at another point | 00:56 |
| ianw | but the gist is that there was an external-id for 9950 "identity":"mailto:...@catalyst.net.nz" which I removed via the api call | 00:58 |
| ianw | it doesn't seem like the duplicate instructions, where we disable the original account, take into account having to remove this external-id? seems like we'd always have to do that to allow the email to be re-used? | 00:58 |
| Clark[m] | ianw: if you look into system-config/tools/gerrit-something are three scripts. One to collect info on those collisions, another to retire an account (disable it and remove the preferred email) and finally one to delete external ids from accounts. Typically you do need to retire and account then delete external ids in that order to fix these issues | 03:10 |
| Clark[m] | The reason for that is often the preferred email is also the conflicting email address and if you don't remove the preferred email first you create a new db error state for that account where the preferred email has no matching external id | 03:11 |
| ianw | thanks; we might just want to take a look at the duplicates instructions in the admin doc page, i do have a memory of all this now you mention those scripts with the retirement work you've done | 03:47 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] test f36 builds https://review.opendev.org/c/openstack/diskimage-builder/+/844912 | 03:59 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] test f36 builds https://review.opendev.org/c/openstack/diskimage-builder/+/844912 | 04:49 |
| *** Tengu_ is now known as Tengu | 05:56 | |
| *** ysandeep|out is now known as ysandeep | 06:15 | |
| frickler | ianw: clarkb: fungi: could you please put https://review.opendev.org/c/openstack/project-config/+/844422 onto your review list? yet another attempt at making the requirements proposal bot work again | 06:32 |
| *** ykarel__ is now known as ykarel | 06:48 | |
| opendevreview | Merged openstack/project-config master: propose-updates: Run ensure-python for python 3.9 https://review.opendev.org/c/openstack/project-config/+/844422 | 07:02 |
| frickler | ianw: now the centos8 wheel builds started failing too. not sure if there actually might be a relation to your dib issue? new kernel or such? https://zuul.opendev.org/t/openstack/builds?job_name=publish-wheel-cache-centos-8-stream&project=openstack/requirements | 07:26 |
| *** jpena|off is now known as jpena | 07:30 | |
| ianw | that looks like openafs isn't installed, that generally means that the kernel the images are running is out of date with the upstream headers | 07:31 |
| ianw | which means we have old images | 07:31 |
| ianw | which, yes might all tie into centos issues :/ | 07:31 |
| frickler | oh, yes, https://nb01.opendev.org/centos-9-stream-0000005770.log | 07:33 |
| frickler | /opt is full on nb01+02. maybe more symptom than root cause | 07:35 |
| frickler | sadly due to the way nodepool recycels logs it will likely not be possible to look at the original failures though | 07:36 |
| frickler | wow, seems all our images are two weeks old? guess I should move "make a job to check image freshness" up on my todo so at least we can notice | 07:39 |
| Clark[m] | With that timing jt may have been an openstacksdk 0.99.0 problem? That has been capped now so fixing thr disk issue may be sufficient | 07:41 |
| frickler | maybe, I'll give it a go now | 07:48 |
| ianw | https://bugzilla.redhat.com/show_bug.cgi?id=2094683 is a initial bug for the selinux issue | 07:50 |
| ianw | it will need more investigation than i have time for right now | 07:51 |
| opendevreview | Dr. Jens Harbott proposed openstack/diskimage-builder master: Drop tumbleweed job https://review.opendev.org/c/openstack/diskimage-builder/+/845097 | 07:52 |
| *** ysandeep is now known as ysandeep|lunch | 07:58 | |
| frickler | wow, that cleanup sure does take its time. anything else to rm other than /opt/dib_tmp/dib_* ? | 07:59 |
| Clark[m] | That's the main thing. You can also look for leaked intermediate vhd files in /opt/nodepool_dib iirc. Those files are only used in converting from raw to vhd and once that is done they are unneeded. They have a .intermediate file suffix iirc | 08:03 |
| opendevreview | Dr. Jens Harbott proposed opendev/glean master: Drop opensuse jobs https://review.opendev.org/c/opendev/glean/+/845104 | 08:17 |
| *** anbanerj is now known as frenzy_friday | 08:30 | |
| *** ysandeep|lunch is now known as ysandeep | 08:44 | |
| frickler | o.k., c8s built successfully, now waiting to see if the uploads are working too | 10:01 |
| frickler | #status log cleanup up /opt on nb01+2 and rebooted them | 10:31 |
| opendevstatus | frickler: finished logging | 10:31 |
| frickler | nb01 might need some deeper cleanup at some time http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68032&rra_id=all | 10:32 |
| *** rlandy|out is now known as rlandy | 10:33 | |
| frickler | iiuc nodepool also keeps around the two most recent copies of each image on each builder? which might be twice as much as we actually need | 10:33 |
| frickler | maybe have a fixed allocation of images to builders instead? possibly failover if one is down? though that very likely would be a new feature for nodepool? | 10:34 |
| Clark[m] | frickler: two total may be two on one or one and one | 10:39 |
| frickler | Clark[m]: but if nb01 has built two images, then nb02 builds the next, should nb01 actually notice and delete its older copy? | 10:44 |
| frickler | because from my casual look that doesn't seem to happen | 10:44 |
| Clark[m] | Yes it should. There may be some issue preventing that or maybe they leaked? | 11:05 |
| frickler | they didn't look too old at least. I'll try to trace some in more detail. maybe best once we're back to two days of stable image builds | 11:17 |
| *** dviroel|afk is now known as dviroel | 11:19 | |
| frickler | 170M of setfiles warnings. I think we should disable c9 image builds until ianw can make progress. and possibly also wheel builds for it | 12:10 |
| frickler | #status log paused centos-9-stream image builds due to https://bugzilla.redhat.com/show_bug.cgi?id=2094683 | 12:19 |
| opendevstatus | frickler: finished logging | 12:19 |
| Clark[m] | ++ to pausing. Thank you for handling that | 12:22 |
| *** ysandeep is now known as ysandeep|out | 14:31 | |
| *** dviroel is now known as dviroel|lunch | 14:47 | |
| frickler | is there a way to actually see which images are paused? | 15:36 |
| *** marios is now known as marios|out | 15:47 | |
| Clark[m] | Maybe with dib-image-list? | 15:52 |
| Clark[m] | I'm not sure | 15:52 |
| *** dviroel|lunch is now known as dviroel | 16:05 | |
| fbo[m] | Hi, is there an issue with the opendev AFS mirror ? The centos-stream 9 mirror seems out of sync. | 16:37 |
| fbo[m] | I don't see libselinux-3.4-2 on https://mirror.dfw.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ but is available since yesterday on https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages | 16:40 |
| frickler | fbo[m]: seems our upstream mirror is lagging, then: https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ | 16:50 |
| frickler | every call of the nodepool cli now seems to log this https://paste.opendev.org/show/bJq6aL1eUJ4XzcylOR8N/ | 17:18 |
| rlandy | fbo[m]: thanks - following here | 17:38 |
| *** jpena is now known as jpena|off | 17:49 | |
| rlandy | frickler: is it possible to force update the mirror? | 17:59 |
| Clark[m] | We sync from Facebook iirc. You should check if our upstream is updated. If not a force won't help anything | 18:08 |
| dasm|ruck | Clark[m]: it's there: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/?C=M;O=A libselinux-utils-3.4-2.el9.x86_64.rpm 2022-06-07 12:12 183K | 18:27 |
| dasm|ruck | but it's missing everywhere else, starting with dfw.rax, then vexx | 18:27 |
| dasm|ruck | Clark[m]: fb mirror is lagging: http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ | 18:36 |
| dasm|ruck | even tho it shows this too: http://mirror.facebook.net/centos-stream/timestamp.txt | 18:36 |
| dasm|ruck | rcastillo is the one who pointed that out | 18:37 |
| dasm|ruck | it seems to be fetching from here https://composes.stream.centos.org/production/CentOS-Stream-9-20220606.0/compose/BaseOS/x86_64/os/Packages/?C=N;O=D | 18:41 |
| opendevreview | Gleb Aronsky proposed openstack/project-config master: Add KubeVirt app to StarlingX https://review.opendev.org/c/openstack/project-config/+/845021 | 18:45 |
| dasm|ruck | does anyone know how te reach out to fb to update their mirrors? | 19:00 |
| rlandy | looks like the current update is in the morning - we may get an update tomorrow? | 19:04 |
| dasm|ruck | let's have a hope for that | 19:05 |
| *** dviroel is now known as dviroel|afk | 20:27 | |
| *** dasm|ruck is now known as dasm|ruck|afk | 21:05 | |
| ianw | frickler: yeah, thanks for pausing that, i realised it was probably in a death spiral doing that late last night | 22:02 |
| ianw | thankfully, though, i've woken up to reports that this is fixed by an upstream commit already in https://bugzilla.redhat.com/show_bug.cgi?id=2094683 | 22:04 |
| ianw | i was getting a little bit worried it might have been something where generically it wouldn't run on !9-stream kernels or something else particularly difficult to work-around | 22:04 |
| ianw | turns out it is a generic problem of running setfiles in chroots without /proc | 22:05 |
| ianw | ahh, i see we're interested in the same 3.4-2 libselinux package anyway discussed above | 22:15 |
| ianw | i just tried from the regular mirrors and didn't get that new package. i'm not really sure but i don't think it is rolled out yet | 22:15 |
| ianw | https://kojihub.stream.centos.org/koji/buildinfo?buildID=21551 has "candidate" and "pending" flags | 22:20 |
| rlandy | ianw: hi - we see libselinux-3.4-2.el9.x86_64.rpm https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/ if that helps | 22:30 |
| *** rlandy is now known as rlandy|bbl | 22:42 | |
| ianw | yeah, i'm not sure what the roll-out process for a package is | 23:05 |
| ianw | odcs seems to be some sort of on-demand-compose-service. i'm not sure what that is | 23:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!