| ianw | "Ansible plugin dir /var/lib/zuul/builds/ee919b7ceebe45c48c3c05d2a5d1e2d3/untrusted/project_0/opendev.org/opendev/system-config/playbooks/filter_plugins found adjacent to playbook /var/lib/zuul/builds/ee919b7ceebe45c48c3c05d2a5d1e2d3/untrusted/project_0/opendev.org/opendev/system-config/playbooks in non-trusted repo." | 00:07 |
|---|---|---|
| ianw | that's a new one... | 00:07 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 00:19 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 00:28 |
| ianw | ok, if i do a "DOCKER_BUILDKIT=1 docker pull docker.io/opendevorg/assets:latest" before building, it seems to work | 00:49 |
| ianw | this does seem to suggest that buildkit has some issue pulling it | 00:49 |
| *** ysandeep|away is now known as ysandeep | 00:59 | |
| ianw | now i can not get it to fail again, even after deleting buildkit directories | 01:06 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] testing if building with buildkit helps pulling image https://review.opendev.org/c/opendev/system-config/+/806832 | 05:30 |
| *** ykarel is now known as ykarel|away | 06:07 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] pause assets build for inspection https://review.opendev.org/c/opendev/system-config/+/806832 | 06:44 |
| *** jpena|off is now known as jpena | 07:03 | |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Correct path for CentOS 9 stream base image https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 07:19 |
| *** yoctozepto1 is now known as yoctozepto | 07:49 | |
| *** arxcruz is now known as arxcruz|training | 08:16 | |
| *** ysandeep is now known as ysandeep|lunch | 08:26 | |
| *** ksambor_ is now known as ksambor | 08:59 | |
| *** ysandeep|lunch is now known as ysandeep | 09:00 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:08 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report https://review.opendev.org/c/opendev/elastic-recheck/+/805638 | 13:03 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report https://review.opendev.org/c/opendev/elastic-recheck/+/805638 | 13:11 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report https://review.opendev.org/c/opendev/elastic-recheck/+/805638 | 13:11 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report https://review.opendev.org/c/opendev/elastic-recheck/+/805638 | 14:21 |
| clarkb | infra-root fyi I have approved https://review.opendev.org/c/opendev/system-config/+/803992 and will check the results of that fail cleanly as expected this evening (that should shift to success once we've redeployed zuul with the new command in the image) | 14:51 |
| clarkb | After breakfast I'm going to start paging in the gerrit account stuff in order to run the external id cleanup script against the last batch | 14:52 |
| fungi | cool. also i've noticed the puppet-else deploys are failing on the logstash server, vcsrepo is failing to update kibana i think, based on what's in the logs on bridge | 14:54 |
| fungi | came to my attention when the openstackid removal (which only should have run fewer things in that job, not more) failed to "deplpy" | 14:55 |
| fungi | "deploy" | 14:55 |
| fungi | but looks like it's failing consistently, i haven't checked the history to see when it started | 14:55 |
| clarkb | might be time to just comment out/delete that bit of code | 14:56 |
| fungi | yeah, maybe | 14:57 |
| opendevreview | Artem Goncharov proposed opendev/base-jobs master: Change quick-download link to construct url properly https://review.opendev.org/c/opendev/base-jobs/+/806941 | 15:00 |
| clarkb | fungi: now that the system-config depends on for https://review.opendev.org/c/openstack/project-config/+/806788 has landed as well as the dns updates for the mirror are we good to land that change? | 15:10 |
| fungi | yup | 15:31 |
| opendevreview | Merged openstack/project-config master: INAP mtl01 region is now owned by iWeb https://review.opendev.org/c/openstack/project-config/+/806788 | 15:44 |
| clarkb | I'm ready to run the external id cleanup for the accounts I retired ~4 weeks ago. I haven't seen any complaints from users about the retirements so should be good to proceed with the external id deletions | 15:54 |
| opendevreview | Merged opendev/system-config master: Accomodate zuul's new key management system https://review.opendev.org/c/opendev/system-config/+/803992 | 15:56 |
| fungi | clarkb: sounds good, thanks for the heads up | 15:57 |
| clarkb | script is done running and the log is in the usual location. I'm going to run the consistency check query against gerrit now | 16:01 |
| *** jpena is now known as jpena|off | 16:01 | |
| *** marios is now known as marios|out | 16:04 | |
| clarkb | 33 conflicting accounts remain. | 16:05 |
| fungi | that's, like, an order of magnitude lower than where you started, right? | 16:06 |
| clarkb | ya we started at 700 something | 16:07 |
| clarkb | the consistency check results are in the normal location too fwiw | 16:07 |
| clarkb | The next step here is to start writing emails to those 33 individuals, make a plan for their accounts and commit that into a All-Users checkout. Then once we've addressed all 33 push that and have gerrit validate it | 16:08 |
| clarkb | oh shoot I just dropped privs then remembered I should rerun the user audit script with this set of 33 to gather up to date info on usage. I'll do that as I dig into writing emails as that will give me info to propose plans | 16:09 |
| clarkb | But now I want to get back to reviewing some zuul/nodepool sos changes | 16:10 |
| clarkb | I keep telling myself if we get past these 33 then fixing screwups in the process becomes really easy :) | 16:13 |
| clarkb | when we had 700 conflicts 4 months ago this was not great, but now we could probably sprint to a less than ideal finish if necessary then fixup afterwards (though still trying to avoid that) | 16:13 |
| *** ysandeep is now known as ysandeep|out | 16:15 | |
| opendevreview | Piotr Parczewski proposed openstack/diskimage-builder master: Fix doc typo https://review.opendev.org/c/openstack/diskimage-builder/+/806954 | 16:19 |
| mazzy | fungi, mordred: so after some thinking I've abandoned the idea to bring in support to Flatcar. Unfortunately the current dib design makes some assumptions that make Flatcar support not possible. I'm not saying it is not possible but this would require a big portion of time to think differently dib and how flatcar would be introduced. thanks anyway | 16:34 |
| mazzy | for your support | 16:34 |
| fungi | mazzy: no worries, is there a summary somewhere of the design disconnect between dib and flatcar? ideas on what changes could be made in dib so that it also covers the flatcar use case? | 16:36 |
| fungi | so some quick mailman restart testing on lists.katacontainers.io indicates that stopping a set of mailman queue runners which had been going for a while (~5 days?) on focal freed up 124mib ram, and starting it back again consumed 93mib | 17:12 |
| clarkb | that seems reasonable for a *5 set on lists.o.o | 17:13 |
| clarkb | not a ton of room but should be sufficient? | 17:13 |
| fungi | that was the used column from free, shared memory and buffers/cache were negligible | 17:13 |
| fungi | and swap is not being consumed at all on that server, making the math fairly simple | 17:13 |
| fungi | the lists.openstack.org server has 4000mib ram, currently using 1346mib active plus 582mib swap | 17:16 |
| fungi | so adding an additional site for lists.openinfra.dev probably shouldn't strain it all that much | 17:17 |
| clarkb | oh that is twice as much as I thought it had | 17:20 |
| clarkb | agreed that should be plenty of space | 17:20 |
| fungi | it'll remain to be seen how much the memory utilization on the server goes up or down after upgrading from xenial to focal however | 17:22 |
| fungi | we'll have graphs we can look at to compare | 17:22 |
| fungi | unfortunately we don't have lists.k.i in cacti so missed an opportunity to get a pre-upgrade utilization baseline for it | 17:23 |
| fungi | i'll be in and out, need to get ahead on some yardwork before the remnants of the hurricane are on top of us | 17:25 |
| clarkb | darn weather, enjoy the time before the rain arrives | 17:26 |
| clarkb | I keep trying to review this zuul chagne then I see the line delta and find something else to do for 5 minutes ... I need to just dig through it | 17:26 |
| mazzy | fungi: no but I can try and prepare them. | 17:30 |
| fungi | mazzy: even just a few sentences could be helpful so the dib devs know what the specific pain points were | 17:35 |
| opendevreview | Piotr Parczewski proposed openstack/diskimage-builder master: Fix doc typo https://review.opendev.org/c/openstack/diskimage-builder/+/806954 | 17:59 |
| clarkb | queues are much emptier than I expected | 19:48 |
| fungi | vacay | 19:50 |
| clarkb | ianw: I've rechceked https://review.opendev.org/c/opendev/system-config/+/805932/3 as zuul-registry should have been updated with the size info fix. but it only updates size when pushing into the registry if I read the code correctly so we have to rerun the image build on that change | 20:23 |
| clarkb | I'm not sure if rechecking the child will cause it to find the previous image build so holding off on rechecking that one until 805932 is done | 20:24 |
| corvus | clarkb: if it depends on it, it should wait | 20:56 |
| corvus | so you should be able to recheck | 20:57 |
| clarkb | corvus: its a normal git tree dag child | 20:57 |
| clarkb | is that == depends on for this? | 20:58 |
| corvus | that counts | 20:58 |
| clarkb | ok rechecked | 20:58 |
| *** stevebaker_ is now known as stevebaker | 21:19 | |
| ianw | 805933 needs an update to stop pausing/failing | 21:44 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 21:45 |
| opendevreview | Michal Arbet proposed openstack/diskimage-builder master: Fix cron not installed in debian https://review.opendev.org/c/openstack/diskimage-builder/+/806990 | 21:49 |
| ianw | yay, it's building | 22:04 |
| ianw | oh, spoke too soon | 22:04 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: buildset-registry: add flag to make job fail https://review.opendev.org/c/opendev/base-jobs/+/806818 | 22:14 |
| ianw | corvus: ^ sorry, switched to fail | 22:18 |
| mazzy | fungi: some words about the dib assumptions that maxe | 22:42 |
| *** dviroel|ruck is now known as dviroel|out | 22:43 | |
| mazzy | hard adding support of flatcar. First to say flatcar does not come out with a package manager. Dib makes as assumption that in order to use caching, python must be installed | 22:44 |
| mazzy | In fact cache-URL and many other elements in an indirect way depends by Python. Now installing anything without a package manager is hard. You need either build from scratch the image or | 22:46 |
| mazzy | try to install for instance python or anything else in the chroot is very hard | 22:48 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 22:53 |
| ianw | yeah, i think dib could be fiddled to produce some output that is a customised flatcar environment. but i can't see it makes much sense as a nodepool node. the whole point that it is *not* a generic image | 23:01 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add reinstall flag to install-packages, use it in bootloader https://review.opendev.org/c/openstack/diskimage-builder/+/804818 | 23:12 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: WIP Add secure boot support to ubuntu. https://review.opendev.org/c/openstack/diskimage-builder/+/806998 | 23:12 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!