| *** rlandy is now known as rlandy|out | 00:08 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: install cri-dockerd https://review.opendev.org/c/zuul/zuul-jobs/+/851093 | 00:19 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: install cri-dockerd https://review.opendev.org/c/zuul/zuul-jobs/+/851093 | 00:30 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: install cri-dockerd; fix networking https://review.opendev.org/c/zuul/zuul-jobs/+/851093 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: test/upload-pypi : install venv https://review.opendev.org/c/zuul/zuul-jobs/+/851089 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: setup.py: override modules to stop auto-detection https://review.opendev.org/c/zuul/zuul-jobs/+/851088 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: fix spaces between filters https://review.opendev.org/c/zuul/zuul-jobs/+/851014 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add names to blocks https://review.opendev.org/c/zuul/zuul-jobs/+/851015 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: rename loop variable https://review.opendev.org/c/zuul/zuul-jobs/+/851016 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/zuul/zuul-jobs/+/851017 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: setup.py: override modules to stop auto-detection https://review.opendev.org/c/zuul/zuul-jobs/+/851088 | 01:14 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: fix spaces between filters https://review.opendev.org/c/zuul/zuul-jobs/+/851014 | 01:14 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add names to blocks https://review.opendev.org/c/zuul/zuul-jobs/+/851015 | 01:14 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: rename loop variable https://review.opendev.org/c/zuul/zuul-jobs/+/851016 | 01:14 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/zuul/zuul-jobs/+/851017 | 01:14 |
| *** ysandeep|out is now known as ysandeep | 01:19 | |
| tonyb | Hi All, I'm to use hound/codesearch to, well search the code, but on various stable branches. I poked around in opendev/system-config but I can't find the hound config to see which refs are indexed. | 02:25 |
| fungi | tonyb: it's only indexing (and from what i understand, only capable of indexing) the master branches of repositories | 02:28 |
| tonyb | fungi: Based on: https://github.com/hound-search/hound/pull/345 I *think* it can index non-default branches but I can't tell if it can do multiple branches. | 02:30 |
| tonyb | fungi: I guess I'll have to cobble something together locally to answer the requirements question(s) I have | 02:31 |
| fungi | tonyb: yeah, last time we looked into it, we could set it to index a different branch than master, but then it wouldn't index master | 02:38 |
| tonyb | I can forsee multiple instances running for codesearch.opendev.org/{main,zed,yoga,wallaby}/...... | 02:39 |
| tonyb | I've very far from serious | 02:39 |
| fungi | yeah, that would be kinda rough | 02:42 |
| tonyb | for now: for d in ~/projects/OIF/openstack/* ; do git -C $d grep --color=never -E '(${pypi_library})' origin/stable/wallaby -- \*requirements.txt 2>/dev/null| sed -e "s/^/$d:/g"; done | 03:09 |
| tonyb | will have to do | 03:10 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] upload-git-mirror : testing https://review.opendev.org/c/zuul/zuul-jobs/+/851147 | 03:26 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] upload-git-mirror : testing https://review.opendev.org/c/zuul/zuul-jobs/+/851147 | 03:59 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] upload-git-mirror : testing https://review.opendev.org/c/zuul/zuul-jobs/+/851147 | 04:23 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] upload-git-mirror : testing https://review.opendev.org/c/zuul/zuul-jobs/+/851147 | 04:29 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: upload-git-mirror: fix ssh key newline https://review.opendev.org/c/zuul/zuul-jobs/+/851147 | 04:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: fix spaces between filters https://review.opendev.org/c/zuul/zuul-jobs/+/851014 | 04:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add names to blocks https://review.opendev.org/c/zuul/zuul-jobs/+/851015 | 04:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: rename loop variable https://review.opendev.org/c/zuul/zuul-jobs/+/851016 | 04:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/zuul/zuul-jobs/+/851017 | 04:42 |
| opendevreview | Ian Wienand proposed openstack/project-config master: proposal updates: fix typo in release match https://review.opendev.org/c/openstack/project-config/+/851156 | 04:50 |
| opendevreview | Merged openstack/project-config master: proposal updates: fix typo in release match https://review.opendev.org/c/openstack/project-config/+/851156 | 05:11 |
| *** ysandeep is now known as ysandeep|afk | 06:08 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: rename loop variable https://review.opendev.org/c/zuul/zuul-jobs/+/851016 | 07:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: fix spaces between filters https://review.opendev.org/c/zuul/zuul-jobs/+/851014 | 07:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: add names to blocks https://review.opendev.org/c/zuul/zuul-jobs/+/851015 | 07:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters: update to ansible-lint 6 https://review.opendev.org/c/zuul/zuul-jobs/+/851017 | 07:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: emit-job-header: noqa on error ignore https://review.opendev.org/c/zuul/zuul-jobs/+/851163 | 07:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: disable progressive mode https://review.opendev.org/c/zuul/zuul-jobs/+/851164 | 07:20 |
| *** jpena|off is now known as jpena | 07:35 | |
| *** ysandeep|afk is now known as ysandeep|lunch | 08:00 | |
| *** undefined is now known as Guest6176 | 08:19 | |
| *** undefined is now known as Guest6178 | 09:23 | |
| hrw | morning | 09:29 |
| hrw | is someone using centos-stream-9 nodes? | 09:29 |
| hrw | as I start to wonder why kolla-ansible CI job OOMs on cs9 while it works on Debian and Ubuntu | 09:33 |
| ianw | hrw: they do get some use; i would say that you might like to look at the dstat service run by devstack as a monitoring tool | 09:38 |
| hrw | ianw: I booted local cs9 node and deployed. rabbitmq container eats 25% of ram.. | 09:38 |
| *** ysandeep|lunch is now known as ysandeep | 10:01 | |
| *** rlandy|out is now known as rlandy | 10:34 | |
| *** dviroel|out is now known as dviroel | 11:35 | |
| jrosser | hrw: fwiw we see OOM all the time on openstack-ansible jobs for centos-8/9-stream where the debian/ubuntu is fine | 11:46 |
| hrw | jrosser: thanks | 11:46 |
| jrosser | it is one of the things we just can't really get much grip on understanding | 11:46 |
| jrosser | we've turned all the threads/processes down to minimum, in fact too much because thats now causing other types of error | 11:47 |
| *** ysandeep is now known as ysandeep|break | 12:19 | |
| fungi | hrw: jrosser: do we maybe set up swap differently between them? | 12:31 |
| fungi | also be aware, in last week's tc meeting there seemed to be consensus that centos stream isn't a stable enough for pre-merge testing in openstack and folks were going to work on dropping the requirement to just daily periodic jobs | 12:32 |
| jrosser | hmm well unfortunatley in a deployment tool a large proportion of time is spent wrangling the differences between distro X and Y, so thats a lot harder | 12:34 |
| jrosser | and looking at the osa dstat setup seems this has bitrotted and needs some tlc | 12:38 |
| hrw | fungi: we (kolla) plan to go rockylinux9 but it is not there yet | 12:39 |
| fungi | jrosser: "this" being swap setup? | 12:40 |
| jrosser | oh no, something has changed with dstat and the way we run it makes an error rather than a useful report | 12:41 |
| fungi | oh, got it | 12:41 |
| fungi | thought you meant dstat was telling you something other than "dstat broke" ;) | 12:42 |
| jrosser | ahhhh if only :) | 12:42 |
| jrosser | what is certainly true is we have struggled with these OOM for quite some time now | 12:43 |
| fungi | yeah, without some profiling to tell you what's eating all your ram or happening to spot some inconsistency like less/no swap on centos nodes, it's hard to guess | 12:44 |
| jrosser | it's revealed a few places we forgot to turn down the threads/processes, fixing those helped some jobs. but it still feels too tight | 12:44 |
| fungi | also this may be another sign of openstack needing to go on a diet. as time goes on, we seem to be making it less memory-efficient rather than more | 12:45 |
| jrosser | part of the trouble with really being aggressive on threads/processes seems to be that a long running request can take longer than the haproxy healthcheck interval, and if that doesnt get serviced then everything comes crashing down | 12:46 |
| *** ysandeep|break is now known as ysandeep | 12:58 | |
| *** rlandy is now known as rlandy|biab | 13:17 | |
| *** rlandy|biab is now known as rlandy | 13:33 | |
| *** dasm|off is now known as dasm | 14:42 | |
| *** ysandeep is now known as ysandeep|out | 14:54 | |
| *** dviroel is now known as dviroel|lunch | 14:57 | |
| *** Guest6178 is now known as rcastillo | 15:11 | |
| clarkb | jrosser: hrw: iirc frickler found newer qemu is very memory inefficient and had to work around that on debian jobs with devsatck | 15:40 |
| clarkb | it allocates a ton of memory even for tiny VMs or something. I would check to see if that is an issue on centos 9 stream and if so look to see what was done on debian to address it | 15:40 |
| clarkb | jrosser: re dstat it is provided by https://pcp.io/ now and pcp relies on a bunch of daemons managed by systemd and half the time the services don't start properly on ubuntu. | 15:50 |
| clarkb | really unfortunate that dstat proper died as it was much simpler | 15:50 |
| *** dviroel|lunch is now known as dviroel | 15:52 | |
| jrosser | there are some zuul roles for this i think, i probably have to look at retrofitting what they do into OSA jobs | 15:52 |
| hrw | clarkb: rabbitmq allocated lot of memory. like 13x more | 15:57 |
| hrw | will look into that later today/tomorrow | 15:57 |
| *** marios is now known as marios|out | 16:04 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set ensure_pip_virtualenv_command in Focal proposal jobs https://review.opendev.org/c/openstack/project-config/+/851239 | 16:10 |
| corvus | i'm looking into the stuck queue (zuul tenant, gate pipeline) | 16:26 |
| *** jpena is now known as jpena|off | 16:32 | |
| opendevreview | Alex Kavanagh proposed openstack/project-config master: Complete charm-cinder-dell-emc-powerstore config https://review.opendev.org/c/openstack/project-config/+/851242 | 16:40 |
| opendevreview | Merged openstack/project-config master: Complete charm-cinder-dell-emc-powerstore config https://review.opendev.org/c/openstack/project-config/+/851242 | 17:00 |
| corvus | i identified the problem and have a fix in progress. i'm going to restart zuul02 which should correct the current issue | 17:27 |
| corvus | #status log restarted zuul02 to resolve conflicting change key error | 17:29 |
| opendevstatus | corvus: finished logging | 17:29 |
| fungi | thanks for finding/fixing! | 17:57 |
| fungi | pip 22.2.1 is out, maybe watch out for new issues around setuptools' vendored copy of distutils: https://pip.pypa.io/en/stable/news/#v22-2-1 | 18:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 18:37 |
| clarkb | fungi: ^ that is a very early mostly outline/sketch of what things will look like. It doesn't actually install any mailman3 or apache stuff yet. But it should run a test and ensure our LE stuff and docker install works on jammy | 18:37 |
| clarkb | and have all the other files and bits we'll be modifying to add in the mailman3 stuff | 18:38 |
| clarkb | I think I'm switching to the laptop after lunch as well so pushing what i have now makes it easy to pull it on the other machine :) | 18:38 |
| fungi | thanks! i'm hacking on the pvmove for the afs and backup servers in preparation for the upcoming block device maintenance, but will take a look once i have these underway | 18:38 |
| clarkb | the parent chagne for that should be mergeable though. | 18:40 |
| fungi | afs01.ord has a root screen session with a `pvmove /dev/xvdb1 /dev/xvdc1` now underway | 18:42 |
| fungi | with a goal of freeing up the afs01.ord.openstack.org/main01 volume for deletion | 18:43 |
| fungi | backup01.ord has two volumes to move off of, so it'll probably take a bit longer | 18:43 |
| fungi | backup01.ord has a root screen session with a `pvmove /dev/xvdb /dev/xvdc1` now underway for the main-202010 vg (and yes, the old one is a raw device while the new one is a partition, slowly getting this server consistent with the others) | 18:52 |
| fungi | all these volumes are 1tb, so pvmove will take a while | 18:52 |
| fungi | clarkb: where's the magic that installs the distro release-specific sources.list files? | 19:09 |
| fungi | oh, found it i think | 19:09 |
| fungi | playbooks/roles/base/repos/tasks/Debian.yaml also covers ubuntu, in contrast to its name | 19:10 |
| fungi | yeah, the "Replace sources.list file" task in there is the magic i was looking for | 19:13 |
| fungi | system-config-run-mirror-arm64 isn't really exercising the ubuntu-jammy-arm64 addition in 851094 but i don't suppose that's especially important for now as this is mainly in service of getting mailman running which we'll do on amd64 anyway | 19:17 |
| Clark[m] | Ya sorry, stepped away to figure out lunch. Ya this is early bootstrapping. There is an arm base job which should exercise jammy though | 19:17 |
| fungi | system-config-run-base-arm64? that wasn't run | 19:19 |
| fungi | oh, wait... | 19:19 |
| fungi | nevermind, i was looking right at it | 19:20 |
| fungi | i forgot there's more than one buildset | 19:20 |
| fungi | pvmove on the afs server and first round on the backup server are nearly done | 20:09 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 20:22 |
| fungi | first old volume detached from backup01.ord and cleaned up, second volume attached and prepped and now `pvmove /dev/xvdf /dev/xvdb1` is underway there (yes another move from a raw disk to a partition) | 20:23 |
| fungi | and only a few percent left on the afs server | 20:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 20:52 |
| *** timburke_ is now known as timburke | 20:58 | |
| opendevreview | Merged openstack/project-config master: Set ensure_pip_virtualenv_command in Focal proposal jobs https://review.opendev.org/c/openstack/project-config/+/851239 | 21:01 |
| *** dasm is now known as dasm|off | 21:02 | |
| fungi | old afs cinder volume has been cleaned up now, just waiting on the second backup volume pvmove to complete | 21:06 |
| opendevreview | Merged zuul/zuul-jobs master: upload-artifactory: fix version comparision https://review.opendev.org/c/zuul/zuul-jobs/+/851087 | 21:22 |
| clarkb | I've got a weird ansible templating issue in the mailman3 change. I have a lists99.opendev.org.yaml.j2 template file for the host_vars. But that needs to be sent through raw without substituting variables and we need to preserve the existing raw block there. It seems we can't do nested raw blocks because it ends at the first endraw? | 21:24 |
| clarkb | I guess I could modify the setup base playbook to copy that file directly without interpolating vars | 21:24 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 21:31 |
| clarkb | that seems extra verbose but I can't figure out how to make jinja2 happy and we don't actually need to substitute any values there | 21:31 |
| fungi | yeah, that's unfortunately probably the simplest hackaround | 21:32 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: pull cri-dockerd systemd from tag https://review.opendev.org/c/zuul/zuul-jobs/+/851263 | 21:41 |
| corvus | if you want to take a break and see something cool, load up https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a5a/851107/2/check/zuul-build-dashboard-opendev/a5a80be/npm/html/ then click on openstack projects, then opendev/system-config then enter "master" into the branch selection at the bottom of the page | 21:48 |
| corvus | (that will totally be deep-linkable when it lands, but you have to go through the steps manually since it's a preview build) | 21:48 |
| corvus | (i'm kind of thinking the arrows should go the other way, ie, show the dependency relationship rather than the job flow. i think that will make it easier to read) | 21:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 21:58 |
| clarkb | corvus: scrolling works oddly on that graphic (it does zoom rather than shift up and down). But I agree pretty neat | 22:00 |
| corvus | clarkb: with a scroll wheel? | 22:00 |
| clarkb | corvus: two finger scrolling on a touchpad so ya should be the same as a scroll wheel I think | 22:00 |
| clarkb | also ++ to changing arrow direction | 22:01 |
| corvus | neat. i've been zooming with double-click and shift-double-click. left-click should pan it. we might be able to prevent it from acting on the scroll wheel/touchpad | 22:01 |
| corvus | (i guess it depends on what people expect; i don't have a scroll wheel/touchpad so i have no expectation :) | 22:02 |
| clarkb | I think scrollwheel should scroll the page and keep zoom level the same. But ya I can see an argument for either method | 22:02 |
| *** rlandy is now known as rlandy|bbl | 22:06 | |
| corvus | we can see at a glance that nothing is using 3.8 -- are we keeping those images around for uwsgi? | 22:10 |
| clarkb | ya I think mostly because at one time other people were using them and zuul only recently got off of 3.8 | 22:11 |
| clarkb | we likely can clean up 3.8 at this point | 22:11 |
| ianw | very cool. yeah the scroll did get me in that when it comes up the mouse is over the image and scrolling zooms it in and out | 22:20 |
| ianw | so it's like the page scrolls down until it hits the image, then starts zooming the image | 22:21 |
| ianw | i guess with something like google maps, it's a "full page" app or whatever so doesn't deal with scrollbars with zooming | 22:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 22:24 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Ensure /etc/apt/sources.list.d exists https://review.opendev.org/c/opendev/system-config/+/851266 | 22:24 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: pull cri-dockerd systemd from tag https://review.opendev.org/c/zuul/zuul-jobs/+/851263 | 22:26 |
| *** dviroel is now known as dviroel|afk | 22:36 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 23:20 |
| clarkb | ok I think that might actually pass now so all the framework stuff is in place. I'll pick up on adding in service bits tomorrow | 23:20 |
| corvus | any preference between these 3 styles? forw-RL https://imgur.com/Ii3unWe.png forw-LR https://imgur.com/n30UqR1.png back-LR https://imgur.com/tYSEfGK.png | 23:24 |
| corvus | (those all have the arrows in the dependency direction -- so the content is the same, just different layout) | 23:25 |
| corvus | (i think i like back-LR which is the same layout as the current change except the arrows are on the other ends of the edge lines) | 23:26 |
| clarkb | I don't like the second one. Don't really have a preference between the first and third. I think that is my read from left to right bias | 23:26 |
| corvus | yeah, i think i like back-lr (third) because you sort of read time from left to right on that, then go backwards to follow the dependency arrows. | 23:27 |
| ianw | 3 seems most "right" to me also | 23:28 |
| ianw | i could see an argument for the arrows going the other way to | 23:29 |
| ianw | on 3 | 23:29 |
| corvus | kk done | 23:31 |
| corvus | i think it's the dashed line that clinches it for me with the arrows. i was explaining what the arrows and dashed line meant to someone and said "they represent hard or soft dependencies. but the dependencies go the other way." | 23:33 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: upload-git-mirror: no_log around key writing https://review.opendev.org/c/zuul/zuul-jobs/+/851269 | 23:33 |
| fungi | second backup01.ord volume has been swapped out and cleaned up. i'll close out the maintenance ticket now that all the volumes they warned us would be disrupted no longer exist | 23:50 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-kubernetes: install cri-dockerd; fix networking https://review.opendev.org/c/zuul/zuul-jobs/+/851093 | 23:53 |
| opendevreview | Merged zuul/zuul-jobs master: test/upload-pypi : install venv https://review.opendev.org/c/zuul/zuul-jobs/+/851089 | 23:53 |
| opendevreview | Merged zuul/zuul-jobs master: setup.py: override modules to stop auto-detection https://review.opendev.org/c/zuul/zuul-jobs/+/851088 | 23:53 |
| opendevreview | Merged zuul/zuul-jobs master: upload-git-mirror: fix ssh key newline https://review.opendev.org/c/zuul/zuul-jobs/+/851147 | 23:53 |
| opendevreview | Merged zuul/zuul-jobs master: linters: rename loop variable https://review.opendev.org/c/zuul/zuul-jobs/+/851016 | 23:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
