| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove Trusty testing https://review.opendev.org/722168 | 00:00 |
|---|---|---|
| mnaser | mordred: looks like have one last thing and we're good | 00:01 |
| ianw | clarkb: ^^ if you wouldn't mind a quick look for some other eyes to see if you see anything else that might fall under that ... i'd like to unbreak the gate :) | 00:01 |
| mordred | mnaser: haha. also - why is it still not doing the right thing in the uwsgi patch? :( | 00:15 |
| mnaser | mordred: i think it is? the phail is happening now because we did -f | 00:15 |
| mnaser | mordred: teh current failure is - [91mERROR: You must give at least one requirement to install (maybe you meant "pip install /output/packages.txt"?) | 00:16 |
| mordred | oh it is? | 00:16 |
| mnaser | oh wait | 00:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 00:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 00:16 |
| mnaser | 3.7 does the right thing | 00:16 |
| mnaser | but 3.8 doesnt | 00:16 |
| mnaser | 3.8 failed with [0m[91mpython: can't open file 'setup.py': [Errno 2] No such file or directory | 00:16 |
| mordred | oh - interesting | 00:17 |
| mordred | mnaser: that's super weird - I don't see what's wrong :( | 00:19 |
| mnaser | mordred: yeah same.. i am looking at requires and they all seem right? | 00:19 |
| mordred | yeah - they look right to me | 00:19 |
| mnaser | OH | 00:20 |
| mnaser | mordred: PYTHON_VERSION=3.7 | 00:20 |
| mnaser | in the 3.8 job | 00:20 |
| mordred | hahaha | 00:20 |
| mnaser | so its doing FROM blah:3.7 | 00:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 00:20 |
| mordred | that's it | 00:20 |
| mordred | yup | 00:20 |
| mordred | but it's not requiring 3.7 | 00:20 |
| mnaser | build an image with uwsgi inside of it they said | 00:20 |
| mordred | *BRAIN MELTS* | 00:20 |
| mnaser | it'll be easy they said | 00:20 |
| mordred | hahahaha | 00:20 |
| mnaser | if anything, this should be further value for having an image dedicated for this :p | 00:21 |
| mnaser | mordred: the only other problem is that i cant test it for lodgeit bc different tenants :( | 00:21 |
| mnaser | lodgeit was moved to opendev | 00:21 |
| mordred | mnaser: oh yeah | 00:22 |
| mnaser | i mean the change is up | 00:22 |
| mnaser | but i dont think we can use/test it | 00:22 |
| mordred | mnaser: I'm getting us closer to being able to move all of the opendev repos into opendev | 00:22 |
| mordred | but yeah - that's gonna be a few minutes | 00:22 |
| mnaser | mordred: yeah it'd be nice to get consesus on the usage of uwsgi-base image, if not, then i can just create one locally based on the work we did | 00:23 |
| mnaser | the parent change is still useful so it makes it easier to build, but yeah | 00:24 |
| mordred | ++ | 00:27 |
| mnaser | mordred: Created wheel for uwsgi: filename=uWSGI-2.0.18-cp38-cp38-linux_x86_64.whl size=535417 sha256=4f4bfe0529382ef80439f7a6a4465a20e668385975199c2acc9761fc407232c0 | 00:42 |
| mnaser | waaah | 00:42 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: strip env vars in chroot calls https://review.opendev.org/721726 | 00:42 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 00:42 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 00:42 |
| mnaser | mordred: thinking out loud why dont we just always install the contents of /output/wheels ? | 00:42 |
| mnaser | cause the python-builder will drop in there whatever we installed anyways | 00:42 |
| mnaser | wait, hm no | 00:43 |
| mnaser | maybe we need to use 'uWSGI' explicity ? | 00:44 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 00:50 |
| mnaser | lets try that, maybe the pip wheel is hash based | 00:50 |
| clarkb | ianw: +2 | 00:51 |
| mnaser | mordred: well, 3.7 works and 3.8 doesn't after using 'uWSGI' explicitly. i'm out of ideas. | 01:09 |
| *** dpanech has quit IRC | 01:29 | |
| clarkb | mnaser: fwiw Im not against a uwsgi image but it might be good to explain why it exists so that when someone wants a cherrpy or a gunicorn or whatever we know where that boundary is | 02:08 |
| clarkb | and also to understand if python-* are deficient in some way | 02:08 |
| clarkb | I've only overpip installed uwsgi so to me it fits into that existing process | 02:09 |
| clarkb | *only ever | 02:09 |
| mnaser | clarkb: yeah I’m not sure how to go about the reasoning other than “it’s popular and works nicely” :p | 02:09 |
| fungi | infra-root: we just got a provider notification they're having trouble with the host ze03 is on | 02:28 |
| fungi | looks like ze03 got ungracefully rebooted ~10 minutes ago | 02:29 |
| *** mordred has quit IRC | 02:43 | |
| *** mordred has joined #opendev | 02:45 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove Trusty testing https://review.opendev.org/722168 | 03:02 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds https://review.opendev.org/717177 | 03:09 |
| *** ykarel|away is now known as ykarel | 05:02 | |
| *** roman_g has quit IRC | 05:30 | |
| *** DSpider has joined #opendev | 05:36 | |
| *** sgw has quit IRC | 05:55 | |
| *** dpawlik has joined #opendev | 06:01 | |
| *** ysandeep is now known as ysandeep|afk | 06:28 | |
| *** jaicaa has quit IRC | 06:29 | |
| *** jaicaa has joined #opendev | 06:32 | |
| *** diablo_rojo has quit IRC | 06:53 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 07:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 07:02 |
| *** ysandeep|afk is now known as ysandeep | 07:10 | |
| *** tosky has joined #opendev | 07:24 | |
| *** rpittau|afk is now known as rpittau | 07:39 | |
| *** ralonsoh has joined #opendev | 07:44 | |
| *** ykarel is now known as ykarel|lunch | 08:03 | |
| *** ysandeep is now known as ysandeep|lunch | 08:44 | |
| *** sshnaidm|afk has quit IRC | 08:55 | |
| *** ykarel|lunch is now known as ykarel | 08:56 | |
| *** ysandeep|lunch is now known as ysandeep | 09:33 | |
| *** ykarel is now known as ykarel|afk | 09:47 | |
| *** sshnaidm has joined #opendev | 10:14 | |
| *** ykarel|afk is now known as ykarel | 10:21 | |
| *** rpittau is now known as rpittau|bbl | 10:21 | |
| *** roman_g has joined #opendev | 10:37 | |
| *** rpittau|bbl is now known as rpittau | 12:02 | |
| *** ykarel is now known as ykarel|afk | 12:22 | |
| corvus | fungi, mordred: maybe we should land https://review.opendev.org/722134 + | 12:40 |
| mordred | done | 12:42 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 12:42 |
| mordred | fungi, frickler: could I get one of you to approve https://review.opendev.org/#/c/711057/ ? searching for the venv that has it installed on bridge is driving me batty | 12:42 |
| mordred | corvus: I'm grabbing a bionic test vm so I can verify the build steps (you know, assuming things might be slightly different than on my laptop | 12:44 |
| mordred | corvus: however, in good news I can report that we don't need to install buildx - the docker we install from upstream is new enough as long as we set an env flag | 12:44 |
| corvus | mordred: oh interesting | 12:44 |
| frickler | mordred: I was assuming you wanted to drop the /root/.config bindmount on 711057. if you don't, I can still approve it | 12:45 |
| mordred | corvus: DOCKER_CLI_EXPERIMENTAL=enabled | 12:45 |
| mordred | frickler: oh - crap, yes | 12:45 |
| mordred | frickler: let me do that real quick, my bad | 12:45 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use openstackclient from container https://review.opendev.org/711057 | 12:46 |
| mordred | corvus: as long as docker >=19.0.3 that should work | 12:46 |
| corvus | mordred: bionic may have the problem that the binfmt entries don't have the 'F' flag; see https://nexus.eddiesinentropy.net/2020/01/12/Building-Multi-architecture-Docker-Images-With-Buildx/#Problem-QEMU-Not-Registered-With-F-Flag | 12:48 |
| mordred | blerg. I was worried about that | 12:48 |
| mordred | corvus: so maybe I should go ahead and make a focal image | 12:48 |
| corvus | mordred: so we may need to either fix that, or use the 'docker image based installation' https://nexus.eddiesinentropy.net/2020/01/12/Building-Multi-architecture-Docker-Images-With-Buildx/#Docker-Image-Based-Installation | 12:48 |
| corvus | mordred: or a focal image | 12:48 |
| mordred | corvus: oh - so - the linux instructiosn I was following already suggested using docker/binfmt | 12:49 |
| mordred | corvus: let me try those on bionic and see how it goes | 12:50 |
| corvus | mordred: i think that would be fine for this use case (a throwaway build node) | 12:50 |
| corvus | (it's lame for a desktop use case) | 12:51 |
| * mordred thinks a focal base would be nice anyway - but trying to reduce the task load | 12:51 | |
| mordred | yeah | 12:51 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 12:51 |
| openstackgerrit | Merged openstack/project-config master: Revert "Move Ubuntu builds away from nb04" https://review.opendev.org/722134 | 12:53 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role https://review.opendev.org/715028 | 12:53 |
| mordred | corvus: autohold is now the only way to grab a nodepool node isn't it? | 12:54 |
| corvus | mordred: yep | 12:54 |
| mordred | darn. I was going to cheat. :) | 12:54 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve linters execution https://review.opendev.org/722307 | 12:59 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve linters execution https://review.opendev.org/722307 | 13:01 |
| mordred | corvus: fwiw - launch_node is giving us warnings about using things that will be removed in future paramiko releases | 13:01 |
| mordred | /usr/local/lib/python3.6/dist-packages/paramiko/kex_ecdh_nist.py:92: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point | 13:01 |
| mordred | corvus: oh - wait - that's paramiko throwing cryptography warnings | 13:02 |
| mordred | nevermind | 13:02 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions jobs https://review.opendev.org/722309 | 13:02 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Switch remaining tests to fedora-31 https://review.opendev.org/722310 | 13:06 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 13:11 |
| mnaser | mordred: ^ yay finally i think i figured it out, the reason 3.8 didnt work and 3.7 did is because we used the wrong base image later | 13:12 |
| mnaser | so it obivously wasnt in the cache | 13:12 |
| mnaser | so that might/should be the alst thing | 13:12 |
| corvus | mnaser, mordred: are we going to continue the pattern of adding images like that to system-config? | 13:12 |
| corvus | it seems like a weird place to me | 13:13 |
| mnaser | corvus: i think it might make sense for opendev/images or something like that at some point | 13:14 |
| corvus | (and yes, the existing python-base and python-builder images also seem weird to me, but they're at least within my knowlede domain to review; uwsgi is not) | 13:14 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job https://review.opendev.org/722309 | 13:14 |
| mnaser | corvus: given that we can document them too, tbh, i/we consume them (both python-builder and python-base) and that can be helpful for us to have some docs too | 13:14 |
| mnaser | or at least somewhere to have docs for usage eventually | 13:14 |
| corvus | do we want to create a single library of generally useful images? or do we want to enable, say, the openstack project to make uwsgi images that it uses? | 13:15 |
| mnaser | corvus: i guess that's up for discussion. an example of uwsgi being useful for opendev is for lodgeit | 13:16 |
| mnaser | but unfortunately they are in different tenants so https://review.opendev.org/#/c/722149/ is failing | 13:16 |
| mnaser | so i think in the case of uwsgi, it _might_ make sense for some of the python-based services that opendev runs | 13:19 |
| corvus | i think lodgeit may be the only one? | 13:19 |
| corvus | and the way we currently run it is "/usr/bin/python /srv/lodgeit/openstack/manage.py runserver -h 127.0.0.1 -p 5000" | 13:20 |
| mordred | corvus: yah - but in the docker image we're installing uwsgi and running it through that | 13:21 |
| corvus | right, i'm just saying that's all completely new to me | 13:21 |
| mordred | nod | 13:21 |
| mordred | we're using apache mod_wsgi for storyboard - although as we shift that to containers it might make more sense to use uwsgi in the storyboard container and run apache separately with proxypass | 13:22 |
| corvus | i haven't grown the ability to review uwsgi changes. and, tbh, i don't really understand why you chose to do that instead of just "CMD ["python /srv...."]" | 13:22 |
| corvus | mordred: maybe? apparently apache with mod_wsgi can be super scalable.... | 13:22 |
| mordred | corvus: yeah - maybe we just want to make an apache image there | 13:23 |
| corvus | in my mind, openstack/uwsgi-image is better than opendev/images is better than opendev/system-config. but i can see the arguments for opendev providing useful images for all of its tenants to collaborate on. but if that's the goal, maybe we should be intentional about it | 13:23 |
| mordred | corvus: yeah- that's honestly my motivation for working with mnaser on this one | 13:24 |
| *** ykarel|afk is now known as ykarel | 13:24 | |
| mordred | it's close enough to the python-base image that it feels like sharing lifecycle events would be advantageous | 13:24 |
| mordred | but I can totally understand the reluctance to expand scope too much | 13:24 |
| corvus | i'm really reluctant to expand system-config | 13:24 |
| mordred | perhaps python-base, python-builder and friends should go in to a separate repo? | 13:24 |
| mordred | that's just for making useful base images? | 13:25 |
| corvus | yeah, it sounds like you, mnaser and i may all be able to agree on that one | 13:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job https://review.opendev.org/722309 | 13:26 |
| corvus | mordred, mnaser: i could also see adding test jobs to that repo that exercised the images more (eg, ran a uwsgi app) | 13:27 |
| mnaser | corvus: i mean it "works for me locally" right now in our testing but its hard to do speculative jobs because of different tenants | 13:27 |
| corvus | mnaser: yep, which is why if we're going to start hosting images like that, we should probably have good tests | 13:28 |
| corvus | in the same tenant :) | 13:29 |
| mnaser | corvus: do you feel like we can at least land the parent change in the meantime that allows us to arbitrarily install things | 13:29 |
| mnaser | that's tested in the follow up change | 13:29 |
| mordred | yeah - that way you can at least make a local uwsgi-base image easily while we figure out the larger story here | 13:30 |
| mnaser | sigh great | 13:30 |
| mnaser | github returning 500s for the uwsgi-base test | 13:30 |
| mordred | \o/ | 13:30 |
| corvus | no objections | 13:30 |
| mnaser | at least the image builds now.. | 13:31 |
| mnaser | but the puppet tests failed because github returning 500s (unrelated to change) | 13:31 |
| mordred | corvus: while booting the bionic test node, I have used nb04's nodepool container to build a focal-minimal image to upload to rax so that we can boot focal opendev servers | 13:32 |
| mnaser | which means https://review.opendev.org/#/c/722133/7 can land | 13:32 |
| corvus | mordred: neat :) | 13:32 |
| mordred | mnaser: I'm getting closer to having all of those puppet tests not need to run on _every_ patch | 13:32 |
| mnaser | mordred: all good :) i think there's been huge progress the past few weeks | 13:33 |
| mordred | ++ - I'm excited about it - even though my brain is bleeding | 13:33 |
| mnaser | wow | 13:33 |
| mnaser | the puppet change double failed | 13:33 |
| mnaser | 500s talking to github | 13:33 |
| mnaser | oh nevermind, it failed in post because it tried to collect a directory that wasnt crated yet | 13:34 |
| mnaser | "Copy puppet-apply test output to log server" could probably use ignore_errors: yes -- but yea | 13:34 |
| mordred | corvus: WOW - neat thing ... I'm building the nodepool image as a test with buildx | 13:38 |
| *** ysandeep is now known as ysandeep|away | 13:38 | |
| mordred | corvus: we obviously have not pushed python-base or python-builder arm images | 13:38 |
| mordred | corvus: but - since neither of them actually install binary depends - they work as overlays on the multi-arch base images | 13:38 |
| mordred | corvus: so we do not actually have to make multi-arch images of python-base and python-builder | 13:39 |
| corvus | mordred: oh, you can just FROM them in an arm build and it works? | 13:39 |
| mordred | yup | 13:39 |
| corvus | huh | 13:39 |
| mordred | because I guess the reference python which is a multi-arch manfiest, so when it goes to fetch those layers, it fetches the platform specific ones | 13:40 |
| corvus | mordred: so if they *did* install something arch dependent, i assume the user would just find out when they ran something in the image? | 13:40 |
| mordred | yeah | 13:40 |
| corvus | good ux | 13:40 |
| mordred | corvus: real4m11.129s | 13:42 |
| mordred | corvus: that was from a buildx build of nodepool | 13:42 |
| corvus | mordred: native arch? or arm? | 13:42 |
| mordred | both | 13:42 |
| mordred | corvus: so - it's possible that my laptop actually ISN'T a good indication here | 13:42 |
| corvus | wow, that's way faster than expected | 13:42 |
| mordred | corvus: yeah - that makes me want to think about just always building multi-arch in our jobs | 13:44 |
| *** sgw has joined #opendev | 13:54 | |
| mordred | corvus: ok - I think I have figured out the sequence of magical incantations | 13:54 |
| corvus | mordred: docker docker docker? | 13:55 |
| mordred | corvus: you left out a docker | 13:56 |
| mordred | corvus: http://paste.openstack.org/show/792603/ | 13:56 |
| mordred | corvus: sigh. not quite. | 13:58 |
| mordred | corvus: ok - so - the thing we can't do is export into the local docker image list | 14:00 |
| *** sshnaidm has quit IRC | 14:01 | |
| mordred | corvus: after building, we can export to an oci tarball - or we can have the buildx command push to a registry - oh - let me try one other thing | 14:02 |
| *** sshnaidm has joined #opendev | 14:02 | |
| corvus | that's about the most bonkers thing i've heard | 14:02 |
| mordred | corvus: I mean - we can export to an oci image tarball and I'm pretty sure then use skopeo to do our push needs | 14:03 |
| corvus | mordred: why driver=docker-container instead of driver=docker ? | 14:03 |
| mordred | well - that was what I thought I needed to do to get --load to work | 14:03 |
| mordred | but it didn't work | 14:03 |
| mordred | so I think that's a red herring | 14:04 |
| mordred | it doesn't seem to make much difference | 14:04 |
| corvus | mordred: so maybe driver=docker (and try that with and without load?) | 14:05 |
| mordred | corvus: yeah - driver=docker implies load by default | 14:06 |
| fungi | mordred: are we ready to approve 720202 now? | 14:06 |
| mordred | corvus: and it doesn't work - actually documented in the readme that it doesn't work for multi-arch images yet | 14:06 |
| mordred | corvus: basically - there isn't a path, yet, to make a multi-arch image and have it show up in docker images yet | 14:06 |
| corvus | everything about docker is an afterthought | 14:07 |
| mordred | (from buildx) | 14:07 |
| mordred | yup | 14:07 |
| mordred | pushing to a registry works | 14:07 |
| mordred | (and then you can get it loaded in to local docker with a pull) | 14:07 |
| mordred | and exporting to an oci tarball works - which we can then manipulate with skopeo | 14:07 |
| mordred | I'm inclined to do exporting to an oci tarball | 14:08 |
| mordred | and then skopeo - so we've got maximum control | 14:08 |
| corvus | mordred: ok, well the build-docker-image pushes to the buildset registry | 14:08 |
| mordred | yeah | 14:08 |
| corvus | mordred: i'd do that instead of skopeo | 14:08 |
| corvus | mordred: currently there is no skopeo usage in build-docker-image, so you'd be adding that | 14:08 |
| mordred | good point | 14:08 |
| corvus | mordred: instead, i'd do the push then pull approach | 14:08 |
| corvus | since we need to push anyway | 14:09 |
| mordred | what about the multi-tagging | 14:09 |
| mordred | we currently run docker tag in a loop before the push | 14:09 |
| corvus | oh right | 14:09 |
| corvus | mordred: well, how does docker recommend you do that? | 14:09 |
| mordred | jeez. maybe we push, then pull, then tag in loop, then push again? | 14:09 |
| mordred | they don't recommend how I do that at all :) | 14:09 |
| corvus | i mean, how do you push multi-arch? | 14:10 |
| mnaser | i think you have to generate a local manifest | 14:10 |
| mnaser | and do a special type of push for multiarch | 14:10 |
| mordred | corvus: oh - pushing a multi-arch is just buiold --push | 14:10 |
| mordred | mnaser: no - we're using buildx at the moment which is handling all of that for us | 14:10 |
| corvus | mordred: does it take multiple tag arguments? | 14:11 |
| mordred | it does not | 14:11 |
| corvus | mordred: so maybe 'build -t {{tag}} --push' in a loop over tags? | 14:12 |
| corvus | the second builds should be noop | 14:12 |
| mnaser | there is docker tag | 14:12 |
| mnaser | docker tag foo/bar foo/bar2 | 14:12 |
| corvus | mnaser: the image isn't in the local image list | 14:12 |
| corvus | mnaser: yes, that's how the existing system works :) | 14:12 |
| mnaser | oh i see, sorry, i'm not familiar with the whole buildx infra | 14:13 |
| corvus | neither am i | 14:13 |
| mordred | mnaser: yeah - it's a whole set of new madness | 14:13 |
| mordred | corvus: that's worth a try | 14:13 |
| fungi | #status log restarted all mailman sites on lists.openstack.org following oom events around 12:35-12:45 utc | 14:13 |
| openstackstatus | fungi: finished logging | 14:13 |
| mnaser | corvus: would it be ok to land https://review.opendev.org/#/c/722133/7 given https://review.opendev.org/#/c/713953/22 at least demonstrates it works? | 14:13 |
| mnaser | sorry to nag but its useful for us and its been hard to use that without it landing due to container speculative jobs not being possible cross-tenant | 14:14 |
| corvus | mnaser: wfm | 14:16 |
| fungi | mordred: can i go ahead and approve the gitea 1.11.4/go-git v5 upgrade (720202) now, or should i wait? | 14:17 |
| mordred | fungi: I think it's a fine idea to approve | 14:17 |
| mnaser | moving forward, i'll try to find time to create opendev/images and move things there | 14:18 |
| mnaser | it seems like a better home (eventually) | 14:19 |
| mnaser | but a lot of moving parts right now so maybe not the best time to decouple it right now | 14:19 |
| mordred | mnaser: I can help drive that once my head isn't in this - and yeah, I agree re timing - but also that it's potentially a good idea | 14:20 |
| corvus | mnaser: ++ thanks! | 14:21 |
| mnaser | another reason is that it could use some more examples and basic documentation, as me and mordred found out yesterday | 14:22 |
| mordred | corvus: jina question for you ... | 14:28 |
| mordred | if I have a list of dicts, and I want to find out if any of the dicts have a non-empty list in one of the value | 14:29 |
| mordred | corvus: is there a good jinja filter way to do that? | 14:29 |
| mordred | basically "has_arch=False ; for arch in arches: if arch.get('arch', []): has_arch=true" | 14:30 |
| corvus | mordred: i want to say we used something like that in the siblings role | 14:30 |
| mordred | corvus: my thinking is to trigger the buildx logic on someone specifying that they want multi-arch in any of the images | 14:31 |
| mordred | corvus: we made a python module in siblings :) | 14:31 |
| corvus | mordred: no i'm thinking of "{{ zuul.projects.values() | selectattr('required') | list }}" | 14:32 |
| openstackgerrit | Colleen Murphy proposed openstack/project-config master: Add core group for python-keystoneclient https://review.opendev.org/722333 | 14:33 |
| corvus | mordred: arches | selectattr('arch', 'iterable') | 14:33 |
| corvus | mordred: or maybe for this case: arches | selectattr('arch', 'defined') | 14:35 |
| mordred | corvus: ah - yeah - and then arches should be empty if none of them are defined yeah? | 14:35 |
| * mordred tests real quick | 14:35 | |
| openstackgerrit | Merged opendev/system-config master: Use openstackclient from container https://review.opendev.org/711057 | 14:36 |
| corvus | mordred: well, the result will be empty, but yeah. | 14:36 |
| mordred | corvus: woot - works. thanks! | 14:40 |
| corvus | \o/ np | 14:40 |
| openstackgerrit | Dmitry Tantsur proposed openstack/project-config master: Cache cirros UEC images https://review.opendev.org/722338 | 14:44 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: WIP Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 14:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Switch remaining tests to fedora-31 https://review.opendev.org/722310 | 14:46 |
| mordred | corvus: ^^ that's kind of what I'm thinking for overall shape, look ok? | 14:47 |
| mordred | corvus: thing I havne't tried yet though is pushing to something that isn't dockerhub | 14:48 |
| corvus | mordred: generally lgtm, yeah | 14:49 |
| mordred | corvus: oh - https://github.com/docker/buildx#buildx-bake-options-target | 14:50 |
| mordred | corvus: looks like we can pass a json file and specify a list of tags | 14:51 |
| corvus | that looks easy to construct | 14:53 |
| openstackgerrit | Merged zuul/zuul-jobs master: Improve linters execution https://review.opendev.org/722307 | 14:55 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role https://review.opendev.org/715028 | 15:00 |
| *** ykarel is now known as ykarel|away | 15:02 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Update ensure-javascript-packages README https://review.opendev.org/722354 | 15:19 |
| *** sshnaidm has quit IRC | 15:22 | |
| clarkb | fungi: fwiw I'm here now to help watch gitea things as well looks like it hasn't merged yet? | 15:29 |
| clarkb | corvus: mordred mnaser we're removing git from assemble, but isn't that needed for pbr? We don't git clone the repo into the container we just bind mount it in iirc so we don't necessarily have git in that context | 15:32 |
| fungi | clarkb: yeah, it's still running the gauntlet for now | 15:33 |
| clarkb | oh I get it now PACKAGES can't be local pacakges | 15:35 |
| clarkb | because we do isntall git if doing local packages | 15:35 |
| clarkb | mnaser: for the uwsgi image how do you install keystone (or whatever) into that? | 15:36 |
| openstackgerrit | Merged opendev/system-config master: Upgrade to gitea 1.11.4 https://review.opendev.org/720202 | 15:37 |
| openstackgerrit | Merged openstack/project-config master: Cache cirros UEC images https://review.opendev.org/722338 | 15:37 |
| clarkb | mnaser: you use python-builder to stage keystone and its deps, then use uwsgi-base instead of python-base to produce the production image? | 15:38 |
| mnaser | clarkb: i have an example for you :) one sec! | 15:38 |
| mnaser | clarkb: https://review.opendev.org/#/c/713975/ | 15:38 |
| mnaser | this is how i did it for keystone and it worked nicely locally | 15:39 |
| clarkb | thanks | 15:39 |
| mnaser | (it builds but what i meant by worked nicely locally as in docker run from that launched keystone) | 15:39 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 15:42 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role https://review.opendev.org/721584 | 15:43 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 15:48 |
| *** mlavalle has joined #opendev | 15:49 | |
| *** sshnaidm has joined #opendev | 15:55 | |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Add ansible role for managing Luna SA HSM https://review.opendev.org/721349 | 15:59 |
| *** diablo_rojo has joined #opendev | 15:59 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Organize zuul jobs in zuul.d/ dir https://review.opendev.org/722394 | 16:00 |
| clarkb | mordred: ^ I don't know if you were already starting that but I think maybe we should try and land something like that soon | 16:00 |
| clarkb | infra-root ^ any feedback on how the jobs have been organized there is much appreciated. I tried to make it obvious where to look for a job if you see it failing or not running | 16:01 |
| clarkb | I based it on the uwsgi stack to avoid conflicting with that | 16:02 |
| clarkb | that change conflicts with like a billion things fwiw | 16:06 |
| clarkb | so we should land it when we are all happy to rebase | 16:06 |
| openstackgerrit | Merged opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 16:07 |
| openstackgerrit | Merged opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 16:14 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: bindep: Add missing virtualenv and fixed repo install https://review.opendev.org/693637 | 16:15 |
| clarkb | logstash doesn't show a bunch of apt issues in the last 6 hours | 16:17 |
| mordred | clarkb: I hadn't started it - I was gonna wait until we'd landed the zuul and nodepool ansible changes | 16:17 |
| mordred | clarkb: but I agree - I think it's super important | 16:18 |
| mordred | clarkb: oh - one thing I was thinking was to take a page from zuul-tests.d | 16:21 |
| mordred | clarkb: and put the project pipeline defintions in each file too | 16:22 |
| mordred | clarkb: so split this: https://review.opendev.org/#/c/722394/1/zuul.d/project.yaml as well | 16:22 |
| clarkb | mordred: maybe that should be a followon? its easier to reason about this as its largely just moving content without changing it | 16:22 |
| mordred | sure | 16:22 |
| mordred | just saying - I think that should be the ultimate shape - so that it's really clear "these are the gitea docker jobs and when they run" | 16:23 |
| clarkb | I wonder if we want to combine docker image, system-config-run, and infra-prd for ^ | 16:23 |
| clarkb | and to them by service rather than lifecycle stage | 16:24 |
| clarkb | I think that could be a good improvement too | 16:24 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 16:24 |
| clarkb | but probably a followon to avoid too much shuffling in one go | 16:24 |
| mordred | clarkb: hrm. maybe? | 16:24 |
| mordred | that's not a bad idea - although I wonder if it'll be hard to think about the dependency trees for infra-prod if we do that | 16:24 |
| corvus | yeah, i think it's worth looking at, but i don't think it's a slam dunk | 16:25 |
| clarkb | mordred: thats a good point re infra-prod | 16:26 |
| clarkb | oh i that change going to run all the jobs too because I moved them around :) | 16:27 |
| clarkb | I didn't think about that before I pushed it | 16:27 |
| mordred | maybe abandon it for now then - we can circle back to it once we've got fewer s-c patches in flight? | 16:28 |
| clarkb | mordred: well its all those s-c changes that drove me to write it :/ | 16:28 |
| clarkb | it is getting more and more difficult to review things. But if we want to avoid rebasing the stuff in flight thats fine | 16:29 |
| clarkb | is there a rough point in git time where we'd be happy to start rebasing? | 16:29 |
| clarkb | you mentioned zuul/nodepool. What about the puppet splitting with eavesdrop and friends? | 16:32 |
| clarkb | looks like its jus about done running jobs anyway so I'll WIP it instaed of abandoning | 16:34 |
| mordred | ah - cool | 16:38 |
| mordred | clarkb: yeah - I think zuul and nodepool and the current eavesdrop/codesearch patches | 16:38 |
| mordred | that'll get us up to date on all of the big changes - and I'll hold off on writing more puppet split changes until we've done a reorg | 16:38 |
| *** njohnston has joined #opendev | 16:39 | |
| *** rpittau is now known as rpittau|afk | 16:49 | |
| AJaeger | mordred: what about adding new jobs already in the "correct" files when you write them? Or moving .zuul.yaml to .zuul.d/project.yaml and then start adding new files? and in the end splitting up? That would decouple your changes... | 16:53 |
| AJaeger | but you can merge first what we have as well;) | 16:54 |
| clarkb | fwiw school time is starting in a minute but then rereviewing mordreds s-c changes is high on the list | 16:55 |
| clarkb | also I think ianw had nodepool changes that will help make progress on the zuul/nodepool side | 16:55 |
| clarkb | corvus: mordred ^ I haven't looked at those yet but also on my list and you may know more about them | 16:56 |
| clarkb | looks like gitea upgrade happened uneventfully | 17:04 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Stop using mysqlclient ssl flag https://review.opendev.org/722405 | 17:11 |
| clarkb | mordred: fungi ^ noticed that looking at more cronspam | 17:11 |
| clarkb | I haven't tested it yet | 17:11 |
| fungi | yeah, so far things looks sane for gitea servers | 17:12 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: WIP: Add tripleo-ci-shared group https://review.opendev.org/722411 | 17:57 |
| zbr | clarkb: i need bit of assistance creating a new gerrit group | 18:05 |
| zbr | i was not able to find any config for effectively adding the group | 18:06 |
| fungi | zbr: groups are auto-created when referred to in an acl | 18:06 |
| zbr | but who becomes the first owner? | 18:07 |
| *** ralonsoh has quit IRC | 18:07 | |
| clarkb | we have to seed the first member with an appropriate individual | 18:07 |
| fungi | a gerrit admin adds someone on request, usually either the ptl/infra liaison of the project, or the person who authored the change which added the acl | 18:08 |
| fungi | zbr: basically this https://docs.opendev.org/opendev/infra-manual/latest/creators.html#update-the-gerrit-group-members | 18:11 |
| zbr | thanks. i am updating the change now to follow all guidelines. | 18:12 |
| yoctozepto | if I wanted to check why centos-8 infra image is so bulky, where would I want to look? | 18:18 |
| clarkb | yoctozepto: the git repos are the bulk of it | 18:18 |
| clarkb | yoctozepto: but you can download the image from https://nb01.openstack.org/images or https://nb02.openstack.org/images and mount it locally and poke around | 18:19 |
| clarkb | in the past we also cached some fairly large vm images within the VM image but now we are down to cirros and etcd tarballs so that has a much smaller impact these days | 18:20 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add tripleo-ci-shared-core group https://review.opendev.org/722411 | 18:23 |
| yoctozepto | clarkb: thanks, I am worried kolla-ansible is tested against very specific environment, we see system-wide pip-installed packages there | 18:23 |
| clarkb | yoctozepto: yes, there is currently work in progress to stop doing that | 18:24 |
| clarkb | yoctozepto: https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html | 18:24 |
| clarkb | we are currently in the build test plain images, run jobs on them and make the jobs work | 18:24 |
| yoctozepto | oh, cool | 18:24 |
| clarkb | system wide pip packages don't make the images large | 18:25 |
| yoctozepto | but only centos 8 seems affected | 18:25 |
| yoctozepto | is that valid? | 18:25 |
| clarkb | yoctozepto: no all of them are that way except for -plain as per the spec | 18:25 |
| clarkb | yoctozepto: however centos doesn't use /usr/local like debian | 18:25 |
| yoctozepto | hmm, I guess something made centos 8 have even more pip packages | 18:25 |
| clarkb | so its often more noticeable | 18:25 |
| yoctozepto | oh, could be that! | 18:25 |
| clarkb | but again this is barely a blip in the image size | 18:25 |
| clarkb | we are talking a few MB vs many GB of other things | 18:26 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add tripleo-ci-shared-core group https://review.opendev.org/722411 | 18:26 |
| clarkb | the reasons to get away from that aren't image size but conflicts with how people expect to interact with the images | 18:27 |
| yoctozepto | and can I already use some -plain? | 18:27 |
| clarkb | yoctozepto: at your own risk | 18:27 |
| yoctozepto | clarkb: agreed and very much what we just hit | 18:27 |
| clarkb | I don't think we'll treat it as an emergency if those break right now | 18:27 |
| clarkb | since we know we are in the test and figure it out stage | 18:27 |
| yoctozepto | ack, sensible | 18:27 |
| clarkb | and they are likely to go away at some point too | 18:27 |
| clarkb | because we'll replace the existing image labels with them | 18:28 |
| yoctozepto | will not that make hell gates open? ;p | 18:28 |
| clarkb | the goal is not to | 18:28 |
| clarkb | we are updating the jobs to make the old and new cases work | 18:28 |
| clarkb | that is why we built -plain images | 18:29 |
| yoctozepto | ok, I think it starts to stick with me | 18:29 |
| clarkb | so we can run the same jobs on both bionic and bionic-plain, validate that they both function as expected without user intervention | 18:29 |
| yoctozepto | thanks, clarkb :-) any eta on it? | 18:29 |
| clarkb | then once we're satisfied we've done enough test coverage of that on our side we'll probably suggest others do testing if they are concerned and set a flag day to switch | 18:29 |
| fungi | but as clarkb points out, the majority of the image content is whatever we're caching in the images, not really the installed software | 18:29 |
| yoctozepto | yeah, I think I saw plain being tested by Ian | 18:29 |
| clarkb | no I don't think we can put an eta on it becuse there are still a lot of unknowns about how jobs will interact | 18:30 |
| yoctozepto | fair point | 18:30 |
| clarkb | the infra specific stuff all seems fine though. Things like glean | 18:30 |
| clarkb | it looks promising | 18:30 |
| yoctozepto | clarkb, fungi: sorry for confusion, I meant bulkiness in terms of those extra system-wide pip packages, not raw size | 18:31 |
| yoctozepto | should have made that one sentence, oh well | 18:31 |
| fungi | centos-8-0000069746.qcow2 is 8.5G, ubuntu-bionic-0000104857.qcow2 is 9.2G, debian-buster-0000134995.qcow2 is 8.9G | 18:31 |
| fungi | so yeah, for actual images sizes, centos is smaller | 18:31 |
| yoctozepto | yeah, I think clarkb correctly pointed out the /usr/local case could be the culprit that we see effects of | 18:31 |
| fungi | well, there's one other problem in centos/fedora land | 18:32 |
| yoctozepto | because ubuntu seems quite pristine from our pov there | 18:32 |
| fungi | and that's they've written a bunch of their required system tools in python, unlike debian/ubuntu | 18:32 |
| fungi | so there are going to be a slew of preinstalled python libraries required by centos/fedora just because they're expected to be part of the base operating system | 18:33 |
| yoctozepto | sure, that's no problem, we don't rely on that fact nor depend on it | 18:34 |
| yoctozepto | it was just surprising to find out pydocker available system-wide ;p | 18:34 |
| fungi | yeah, but it does mean that there will be a bunch of extra python libraries showing up installed | 18:34 |
| clarkb | yoctozepto: I'm not sure what would pull pydocker in. Maybe tox? | 18:34 |
| yoctozepto | true, but it's under control and resembles real life scenarios | 18:34 |
| fungi | pydocker is probably part of centos | 18:34 |
| clarkb | we install pip, tox, glean globally off the top of my head | 18:35 |
| yoctozepto | no idea, let me dig a bit more | 18:35 |
| clarkb | then a smaller number of things in venvs | 18:35 |
| clarkb | *small number of things | 18:35 |
| yoctozepto | I doubt c8 would install it considering it does not natively support docker in the first place hmm | 18:35 |
| fungi | podman may rely on pydocker, and centos/fedora replace docker with podman but call it docker, right? | 18:35 |
| yoctozepto | that would be odd | 18:35 |
| yoctozepto | but who knows | 18:35 |
| yoctozepto | checking that | 18:36 |
| clarkb | https://nb01.openstack.org/centos-8-0000069746.log doesn't show anything for pydocker or python-docker | 18:36 |
| fungi | yeah, so it must be getting installed when the job runs | 18:36 |
| fungi | if you've got an example build result, we can look in the logs to see when it gets installed and why | 18:37 |
| yoctozepto | thanks, that would be cool | 18:37 |
| yoctozepto | here is the relevant change | 18:37 |
| yoctozepto | https://review.opendev.org/721075 | 18:37 |
| yoctozepto | centos 8 should fail as others did | 18:37 |
| yoctozepto | the job is broken by design | 18:38 |
| yoctozepto | I checked rpm packages but nothing relevant installed | 18:38 |
| yoctozepto | similarly jmespath should not be present | 18:38 |
| clarkb | yoctozepto: fungi I think its gonna be ansible installing it | 18:40 |
| clarkb | that job uses a nested ansible and I bet money ansible pulls it in because modules want to talk to docker containers via python in ansible | 18:40 |
| clarkb | one beers worth :) | 18:40 |
| yoctozepto | clarkb: it never does, we have to install it | 18:42 |
| yoctozepto | for this reason ubuntu and debian fail now | 18:42 |
| clarkb | right wouldn't that result in it being present though? | 18:43 |
| clarkb | fwiw it seems that the job I'm looking at is failing because python docker is NOT installed | 18:43 |
| yoctozepto | you are probably looking at ubuntu/debian then | 18:44 |
| clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e86/721075/1/check/kolla-ansible-centos8-source-cells/e869658/primary/ara-report/ara-html/result/bc1a0c57-7b40-4d6a-b8ad-bd49638ae498/ that says centos-8 | 18:44 |
| fungi | yeah, i think what he's saying is you need to look at the logs for https://zuul.opendev.org/t/openstack/build/26c3a24ac4cb4c6e8d63144169c3d49e and see why/how pydocker is getting installed that makes that build actually work | 18:44 |
| yoctozepto | ah, multinode ones | 18:44 |
| yoctozepto | yeah, I'm talkin singlenode | 18:44 |
| clarkb | yoctozepto: can you link to the exact job then? | 18:45 |
| yoctozepto | clarkb: https://zuul.opendev.org/t/openstack/build/26c3a24ac4cb4c6e8d63144169c3d49e | 18:45 |
| yoctozepto | vs https://zuul.opendev.org/t/openstack/build/8326fbd192f84fed8104564e5a997cd5 | 18:45 |
| clarkb | that job succeeded | 18:45 |
| yoctozepto | it did but should not | 18:45 |
| yoctozepto | that's the trick | 18:46 |
| clarkb | ok so the ubuntu behavior of Failing is what we want | 18:46 |
| yoctozepto | but your observation is correct that multinode somehow do not have py docker on secondary nodes | 18:46 |
| yoctozepto | I missed that | 18:46 |
| clarkb | yoctozepto: that implies it isn't an image issue | 18:47 |
| yoctozepto | yeah, that one is not | 18:47 |
| yoctozepto | but jmespath looks like it | 18:47 |
| fungi | i'm not having much luck finding a log which records where package installation is occurring | 18:47 |
| yoctozepto | https://review.opendev.org/720752 | 18:47 |
| *** sgw has quit IRC | 18:48 | |
| yoctozepto | this is where ubu/deb fail because of lack of jmespath, yet c8 found it | 18:48 |
| yoctozepto | but it looks indeed like the /usr/local thingy | 18:48 |
| clarkb | fungi: ya me either. There doesn't seem to be a clear linear path. The job-output log tells me deploy.sh fails but I can't find a deploy.sh log | 18:49 |
| yoctozepto | so the jmespath mystery looks resolved | 18:49 |
| yoctozepto | and py docker is on me to trace | 18:49 |
| yoctozepto | ah, I can help with that | 18:49 |
| yoctozepto | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_26c/721075/1/check/kolla-ansible-centos8-source/26c3a24/primary/logs/ansible/deploy | 18:49 |
| fungi | the dnf and yum logs collected don't seem to provide much | 18:50 |
| yoctozepto | or just browse ara https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_26c/721075/1/check/kolla-ansible-centos8-source/26c3a24/primary/ara-report/ | 18:50 |
| clarkb | yoctozepto: except that this is likely preara problem? | 18:50 |
| clarkb | yoctozepto: my theory is its being pulled in by the ansible install | 18:50 |
| clarkb | and ara can't log that | 18:50 |
| yoctozepto | preara is job-output in main dir | 18:51 |
| yoctozepto | then everything is ara :-) | 18:51 |
| clarkb | yoctozepto: but that doesn't log what deploy.sh is doing? | 18:51 |
| yoctozepto | correct, it is split into the log I mentioned | 18:51 |
| clarkb | but ara isn't a shell script | 18:52 |
| yoctozepto | do note ubu/deb fail at prechecks so deploy never even happens | 18:52 |
| yoctozepto | yeah, it is not | 18:53 |
| yoctozepto | but deploy.sh is just calling ansible | 18:53 |
| clarkb | yoctozepto: where does reconfigure.sh log? | 18:54 |
| yoctozepto | where deploy but in files named reconfigure* | 18:55 |
| clarkb | I guess thats after the expected failure point | 18:55 |
| yoctozepto | it's the last step though | 18:55 |
| yoctozepto | yeah | 18:55 |
| yoctozepto | the earliest is bootstrap | 18:55 |
| yoctozepto | but it treats all hosts equally so it should patch up other nodes too | 18:55 |
| yoctozepto | let me verify one thing | 18:56 |
| yoctozepto | ok, verified, it got pulled via requirements.txt | 18:56 |
| yoctozepto | so docker is resolved as well | 18:56 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle https://review.opendev.org/720160 | 18:56 |
| yoctozepto | and jmespath is due to not /usr/local | 18:56 |
| yoctozepto | and python see all extra packages | 18:57 |
| yoctozepto | makes sense | 18:57 |
| yoctozepto | all my worries have gone away | 18:57 |
| clarkb | yoctozepto: can you point to that log since I'm curious to see where that ends up? | 18:57 |
| clarkb | I've basically been trying to find where ansible is installed | 18:57 |
| clarkb | and working my way backwards from deploy.sh failing | 18:57 |
| clarkb | ah there it is | 18:58 |
| clarkb | install kolla-ansible and dependencies | 18:58 |
| clarkb | and that is where docker is pulled in | 18:58 |
| yoctozepto | https://f5ae4a32b02585f5baea-19e17e5db2abd0e280dd7dd93f01d60e.ssl.cf5.rackcdn.com/721075/1/check/kolla-ansible-ubuntu-source/8326fbd/job-output.json | 18:58 |
| yoctozepto | look for install kolla... | 18:59 |
| yoctozepto | it installs the deps | 18:59 |
| clarkb | https://zuul.opendev.org/t/openstack/build/26c3a24ac4cb4c6e8d63144169c3d49e/console#2/1/12/primary | 18:59 |
| yoctozepto | now then why ubuntu no longer see them but centos8 still sees them | 18:59 |
| yoctozepto | oh my brain hurts | 18:59 |
| clarkb | yoctozepto: python path probably | 18:59 |
| yoctozepto | yeah, that would align with jmespath | 19:00 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add tripleo-ci-shared-core group https://review.opendev.org/722411 | 19:00 |
| yoctozepto | the breaking patch just creates virtualenv with system-wide packages in | 19:00 |
| yoctozepto | so based on default paths it could break it seems | 19:00 |
| yoctozepto | that's a very valuable lesson | 19:01 |
| yoctozepto | ok, I updated that change with summary comment so I have something to start off; the erratic behaviour actually reminds me of users having trouble with package resolution on ubuntu | 19:06 |
| yoctozepto | thanks again clarkb and fungi, best service around here | 19:06 |
| clarkb | mordred: I'm digging into the puppet stack (I've got more of that paged in than zuul and nodepool right now) and https://review.opendev.org/#/c/721102/9 should be ready to go (I haven't reviewed it yet since others have) | 19:10 |
| fungi | yoctozepto: to be fair, it's the *only* service around here ;) | 19:10 |
| clarkb | mordred: https://review.opendev.org/#/c/721098/28 has comments from corvus and I'm reviewing that one myself now | 19:11 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 19:20 |
| mordred | corvus, clarkb: ^^ I think that's teh whole story with multi-arch | 19:20 |
| yoctozepto | fungi: shh! let me express my praise for your work and hints with no undermining! | 19:21 |
| mordred | clarkb: cool - let me fix those real quick | 19:21 |
| fungi | yoctozepto: in that case, thanks! | 19:22 |
| clarkb | mordred: if you give me about 5 minutes I think I'll be done reviewing it myself | 19:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:24 |
| corvus | mordred, clarkb, ianw: nb04 has built images, and i can docker exec into it, so that's looking good | 19:24 |
| mordred | corvus: oh - poop. well, there it is - but I'll check your reviews as soon as you're done | 19:24 |
| mordred | corvus: I also used it to build a focal image earlier | 19:24 |
| corvus | mordred: yep, just adding additional confirming data :) | 19:25 |
| mordred | ++ | 19:25 |
| mordred | corvus: /opt/nodepool_dib/ubuntu-focal.* on nb04 | 19:25 |
| mordred | next step is uploading to rax-dfw | 19:25 |
| mordred | and then we should be able to boot production focal-based images | 19:26 |
| clarkb | mordred: ok comments left | 19:31 |
| clarkb | I think the install_only thing may be the only thing that needs changing? | 19:31 |
| openstackgerrit | Oleksandr Kozachenko proposed opendev/system-config master: Remove packages.txt after assemble https://review.opendev.org/722479 | 19:31 |
| clarkb | mordred: and I was hoping for a little more clarfication on the site.pp and eavesdrop.pp split | 19:32 |
| clarkb | I don't think its necessary but also doesn't hurt anything | 19:32 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 19:34 |
| mordred | clarkb: fixed and responded | 19:34 |
| mordred | clarkb: mainly making smaller .pp files so that we can do better job triggering | 19:35 |
| clarkb | got it its for the zuul file matcher more than anything else | 19:36 |
| mordred | yeah | 19:37 |
| mordred | cause you're right - we could totally just do the playbook | 19:37 |
| mordred | clarkb: for that matter, we could remove the host matchers from the smaller .pp files | 19:37 |
| mordred | but I figured keeping it that way would be an easier review | 19:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 19:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 19:38 |
| clarkb | mordred: ya | 19:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 19:39 |
| mordred | clarkb: I rebased the stack - it's mostly reviewed | 19:39 |
| clarkb | mordred: on the parent codesearch change you assert that openstack_project::server does nothing. It seems to configure the apt puppet module as well as install openafs client | 19:39 |
| clarkb | mordred: for codesearch those two things are likely not necessary but we may have to be more careful about that removal on other servers | 19:39 |
| mordred | clarkb: we install openafs-client with ansible now | 19:40 |
| clarkb | mordred: but only on ansible hosts | 19:40 |
| clarkb | aiui | 19:40 |
| mordred | yeah - but we're doing playbooks | 19:40 |
| clarkb | so mirror.foo.bar.openstack.org which is puppet still wouldn't | 19:40 |
| mordred | so - my thinking was to remove server and add openafs-client as needed | 19:40 |
| mordred | as we add the playbook | 19:41 |
| clarkb | (though all new mirrors are being built as mirror.*.opendev.org with ansible) | 19:41 |
| clarkb | gotcha | 19:41 |
| clarkb | so we could still have puppet manage old mirrors but run playbook for afs pre puppet | 19:41 |
| mordred | yeah | 19:41 |
| mordred | the nice thing here is that we can actually piecewise move chunks of stuff between teh two since we have service-specific playbooks | 19:41 |
| mordred | (like moving accessbot to docker but leaving the others for now in eavesdrop) | 19:42 |
| yoctozepto | infra-core: one last thing before /me going to bed - could I get 2nd +2 and approval on https://review.opendev.org/717603 ? | 19:43 |
| clarkb | mordred: minor nit on https://review.opendev.org/#/c/721629/9 that can be a followup (or probably no one will notice if we don't fix it either) | 19:43 |
| yoctozepto | clarkb: thanks :-) | 19:47 |
| mordred | clarkb: :) | 19:49 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 20:02 |
| clarkb | fungi: thinking out loud here, should we maybe start a dstat run capturing memory usage on a frequent basis then maybe rotate that weekly? | 20:03 |
| clarkb | fungi: hoping that eventually it will lead us to the OOM culprit on lists | 20:03 |
| clarkb | (sorry I meant to say run dstat on lists) | 20:03 |
| clarkb | maybe to start just run it in a root shell? | 20:03 |
| fungi | maybe... i mean the oom dumps already list the processes consuming memory. the highest vmem consumers are all apache processes | 20:06 |
| clarkb | fungi: isn't it resident memory that matters more though | 20:07 |
| clarkb | fungi: also the other thing we can do is add more swap | 20:07 |
| fungi | and yeah, cacti doesn't give us much useful info, snmpd stops responding during these events | 20:08 |
| clarkb | dstat will give us a much better pciture of total system resource usage | 20:08 |
| fungi | though there's a bit of a lead-up in swap utilization hours before | 20:08 |
| clarkb | to try and help identify bottlenecks (we could be using more memory beacuse some activity is being slow on cpu or disk) | 20:09 |
| fungi | which is strange, fiven that most of the active memory is consumed by buffers and cache but a ton of data is getting paged out? | 20:09 |
| clarkb | fungi: I think linux will put caches into swap too under some circumstances | 20:09 |
| clarkb | bceause they can be retrived from swap more quickly than disk | 20:09 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=219&rra_id=all vs http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=221&rra_id=all | 20:10 |
| clarkb | *regular fs disk | 20:10 |
| clarkb | oh wow huh | 20:10 |
| clarkb | fungi: swappiness on that host is 60 which is default | 20:11 |
| clarkb | I know we turn it down in devstack-gate/devstack | 20:11 |
| openstackgerrit | Merged opendev/system-config master: Set AllowEncodedSlashes NoDecode on 8080 revproxy https://review.opendev.org/717603 | 20:12 |
| fungi | the sustained burst of cpu utilization also sort of coincides with when the swapping began | 20:12 |
| clarkb | fungi: maybe we should turn down swappiness and add another 2GB of swap? | 20:12 |
| clarkb | and then see if we can flatten that memory use out more? | 20:12 |
| fungi | looks like that's a nightly event | 20:12 |
| clarkb | fungi: bup maybe | 20:12 |
| fungi | (the cpu bursts i mean) | 20:12 |
| fungi | yeah, could be | 20:13 |
| fungi | it's almost all iowait | 20:13 |
| clarkb | fungi: what is odd is memory use went down after the host upgrade | 20:13 |
| fungi | and there's a ton of corresponding reads from xvda | 20:13 |
| clarkb | fungi: bup would certainly explain that. Other things it could be robot indexers of the web? | 20:14 |
| clarkb | (I don't think we should turn off indexing of the archives, but if that is a cause understanding it would be good) | 20:14 |
| fungi | so what i can gather from cacti is that there was a bunch of reading off xvda which gobbled up memory buffers (primarily) much of which wound up getting paged to swap | 20:14 |
| fungi | but yeah, maybe dstat to the rescue here, we probably want to start by identifying what's consuming so much memory earlier in that event | 20:16 |
| fungi | and what's also hammering the rootfs so hard | 20:16 |
| clarkb | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L48-L63 fwiw on swappiness | 20:16 |
| *** sgw has joined #opendev | 20:16 | |
| clarkb | the note about mysql might be relevant here | 20:16 |
| clarkb | perhaps apache and/or bup do similar | 20:17 |
| clarkb | ++ to dstat, gather some more data then we acn decide if adjusting swappiness, swap size, or something else makes sense | 20:17 |
| clarkb | I've got to sort out lunch and a bike ride right now. But can help set that up in a root screen later | 20:18 |
| fungi | looking at the swap usage graph for longer timeframes though, today's spike may be an anomaly | 20:18 |
| clarkb | fungi: devstack's dstat service might contain helpful info for logging to disk | 20:18 |
| fungi | the cpu spikes are consistent daily, but not the spikes in memory utilization | 20:18 |
| openstackgerrit | Oleksandr Kozachenko proposed opendev/system-config master: Remove output after install from bindep https://review.opendev.org/722479 | 20:23 |
| *** DSpider has quit IRC | 20:26 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Include octavia files when updating branch constraints https://review.opendev.org/722491 | 20:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add tests for multiarch build https://review.opendev.org/722496 | 20:53 |
| *** dpawlik has quit IRC | 21:09 | |
| fungi | #status log running `dstat -tcmndrylpg --tcp --output dstat-csv.log` in a root screen session on lists.o.o to diagnose recurring oom issue | 21:10 |
| openstackstatus | fungi: finished logging | 21:10 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 21:33 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Fix the checking helm_values_file definition https://review.opendev.org/722516 | 21:35 |
| ianw | clarkb yoctozepto: centos-8 is slightly different in that it only has packaged pip installed -- we do not overwrite it with pip installed versions | 21:47 |
| ianw | however, those packages are pre-installed, as noted, and we are moving to stop doing that | 21:48 |
| ianw | speaking of clarkb, maybe mordred : https://review.opendev.org/#/c/721726/ is the only non-trivial change required to get our suite of platforms building under our container functional tests | 21:49 |
| ianw | clarkb: perhaps you could look over https://review.opendev.org/#/q/status:open+project:zuul/nodepool+branch:master+topic:builder-container too which mordred has already looked at | 21:49 |
| ianw | Functional container tests: update to CentOS 8 (https://review.opendev.org/721509) can't work until 721726 is merged | 21:50 |
| ianw | which will fix the release test job there | 21:50 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nb03 : update to arm64 to inheritance, drop pip-and-virtualenv https://review.opendev.org/720641 | 21:56 |
| ianw | clarkb / mordred: ^ as mentioned in #zuul | 21:56 |
| mordred | ianw: cool. did you see the arm container building stuff? | 22:06 |
| ianw | mordred: yes sort of ... is it all in zuul-jobs or am i missing other bits to look at? | 22:07 |
| mordred | ianw: all in zuul-jobs | 22:07 |
| mordred | ianw: https://review.opendev.org/#/c/722483/ will use it - but the role is in a trusted job, so we have to land it first | 22:08 |
| mordred | ianw: +2 on the p-c change | 22:11 |
| ianw | mordred: is the eventual idea for buildx to be the only path? | 22:13 |
| ianw | mordred: minor comment on 722339 ... generally lgtm; seems like something we should start and iterate on | 22:17 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 22:18 |
| mordred | ianw: well - don't know that we know that ... it's extra moving parts that we don't really need for non-multi-arch | 22:19 |
| mordred | ianw: but ... maybe we'll be happy with multiarch builds enough that we do make it the only path | 22:19 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Stop installing python2 pip venv on focal https://review.opendev.org/722537 | 22:20 |
| mordred | ianw: for now, definitely just keeping it to be conditional while we poke at it (it's very new and experimental from upstream docker) | 22:20 |
| mordred | ianw: ^^ look! your favorite element! | 22:21 |
| mordred | ianw: oh - that's a good question re: buildset_registry | 22:21 |
| mordred | I was mostly looking at the existing checking for if buildset_registry is defined in the role | 22:22 |
| mordred | but I think that can be removed - as well as the assert | 22:22 |
| ianw | johnsom / mordred: thanks, dropped a comment but i'd prefer we put it an error out for this element on focal. it's what we're doing for f31 and tumbleweed in https://review.opendev.org/721763. we don't want to support it | 22:22 |
| mordred | ianw: oh - wait - I know | 22:22 |
| mordred | ianw: buildset_registry might not be defined because that lookup might fail | 22:23 |
| johnsom | Ah, I was looking in the DIB channel and didn't see this chat. | 22:23 |
| johnsom | That is also my least favorite element | 22:23 |
| ianw | mordred: oh, right i see now, thanks | 22:24 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 22:25 |
| johnsom | ianw What is the alternate element? or can we just drop it and use package-installs? | 22:25 |
| mordred | johnsom: oh it's so much more complicated than that :) | 22:25 |
| johnsom | lol, I would guess | 22:25 |
| ianw | yeah, for infra, but for your case, i would say yes just install python3-pip as a package | 22:26 |
| mordred | oh. yeah. sorry - I was thinking about the opendev images | 22:26 |
| ianw | unless you need something very specific | 22:26 |
| * mordred forgets there are other dib users :) | 22:26 | |
| johnsom | Happy to give the service project perspective. grin | 22:26 |
| mordred | ianw: oh - also - I built a focal image but haven't uploaded it anywhere yet | 22:27 |
| ianw | and, if there's any problems with that version of pip, report them upstream :) | 22:27 |
| ianw | mordred: using dib? | 22:27 |
| mordred | ianw: yah | 22:28 |
| fungi | ianw: or to your distro probably | 22:28 |
| mordred | ianw: /opt/nodepoolb_dib/ubuntu-focal.* on nb04 - plan is to upload to rax so we can boot control plane nodes on focal. I'll upload it tomorrow, didn't get to it today | 22:28 |
| mordred | ianw: but it worked like a charm! | 22:28 |
| * fungi notes you should generally refrain from reporting bugs in distro-packaged software to upstream software maintainers unless you are the package maintainer for that distro | 22:28 | |
| ianw | mordred: interesting, it just worked? | 22:28 |
| mordred | yup | 22:28 |
| ianw | fungi: sorry, yeah that's what i meant, the distro. basically, anywhere but to me! :) | 22:29 |
| mordred | ianw: I did this: docker-compose exec nodepool-builder bash /opt/nodepool_dib/make-focal.sh | 22:29 |
| fungi | ;) | 22:29 |
| ianw | (i jest, but that is the point that we don't have infra custom things you have to debug) | 22:29 |
| johnsom | ianw I will give it a go with the drop the element approach and let you know how it goes. | 22:30 |
| * fungi is working on opendev engagement metrics... notes that over the past 30 days 11241 gerrit changes have been touched/altered/updated/commented on in some way | 22:30 | |
| ianw | mordred: well, that's interesting, thanks ... i'll work on some gate jobs | 22:30 |
| mordred | ianw: also - I think it means that this: https://review.opendev.org/#/c/720719/ should work | 22:31 |
| mordred | ianw: although maybe I should add them back to nb04 | 22:32 |
| mordred | oh - actually, I should ONLY add them to nb04 | 22:32 |
| mordred | because we need newer builder to build focal | 22:32 |
| ianw | mordred: yeah it would be good to verify with gate tests. i feel like the debootstrap fix should work. but similarly, i'd like to start this element *without* pip-and-virtualenv | 22:33 |
| ianw | i.e. no need for -plain images; we should just start our testing/bringup on nodes without it | 22:34 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Start building focal images https://review.opendev.org/720719 | 22:35 |
| mordred | ianw: kk. I'll update the config for it | 22:35 |
| mordred | ianw: also - I agree re: gate tests | 22:35 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Start building focal images https://review.opendev.org/720719 | 22:37 |
| mordred | ianw: thereyago | 22:37 |
| ianw | i just need to re-orient to paedology mode in my classroom ... i.e. dining room table. back in a sec | 22:38 |
| clarkb | ianw: catching up on reviews now and starting with the ones you linked | 22:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 22:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 22:46 |
| ianw | mordred / clarkb: ^^ the only thing with using the container tests is i'm not sure how to do an equivalent of the release test | 22:47 |
| ianw | the nodepool release test installs nodepool from source, then dib etc from release, but i can't reuse that job in dib where i'd want nodepool from release and dib from source | 22:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 22:48 |
| clarkb | ianw: minor but important thing on https://review.opendev.org/#/c/720641/2 | 22:49 |
| ianw | oops thanks | 22:50 |
| ianw | you americans and your 'z's throw me off :) | 22:50 |
| clarkb | ianw: we simplified english, ya'll should get on board (unfortuantely it didn't really help english still crazy) | 22:51 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nb03 : update to arm64 to inheritance, drop pip-and-virtualenv https://review.opendev.org/720641 | 22:53 |
| clarkb | johnsom: ianw I think octavia could probably use python3 venv module? | 22:57 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 22:57 |
| clarkb | which is where we are giogn to try and push things on the job nodes for bootstrappign tools too | 22:58 |
| *** tosky has quit IRC | 22:58 | |
| ianw | mordred: ^^ that will test glean | 22:58 |
| johnsom | clarkb Oh, yeah, I remember thinking about that. You can just do "python -m venv" right and you don't need the python3-virtualenv package installed? Is that correct. | 22:59 |
| clarkb | johnsom: you need to do python3 -m venv on debuntu and you need the python3-venv package but ya | 22:59 |
| clarkb | the nice thing about it is its stdlib so nothing from pip sould try and install over it | 22:59 |
| clarkb | which helps to avoid conflicts down the road | 23:00 |
| johnsom | ianw FYI, the image built switching to just handling them in package-installs, so works-for-me. Just fired up a job to see if the image is actually functional. | 23:00 |
| clarkb | ianw: I've reviewed the changes you linked to earlier (at the start of your day) is there a tldr for subsequent things to review? | 23:00 |
| ianw | clarkb: umm, so many things in progress! :) | 23:02 |
| clarkb | ianw: I know thats why I get easily lost :) | 23:02 |
| ianw | clarkb: if you could also poke at https://review.opendev.org/#/c/721763/ ... i would like to get that into the next dib release | 23:02 |
| ianw | that will, i believe, ensure opensuse builds in both -plain and !plain formats in the gate | 23:03 |
| ianw | and with that release of dib, i'll be pretty confident that our container builders can build all our image types | 23:04 |
| ianw | https://review.opendev.org/#/c/721511/ will confirm this too | 23:04 |
| ianw | after that, we should have -plain nodes and i can get back to more focus on the pip-and-virtualenv removal | 23:06 |
| clarkb | ianw: whats with https://review.opendev.org/#/c/721763/10/diskimage_builder/elements/install-bin/pre-install.d/01-install-bin ? I think the difference there is we don't filter for executable files. Were we writing scripts that needed to be chowned? | 23:08 |
| ianw | hrmmm ... that's come in via the revert ... | 23:09 |
| ianw | https://review.opendev.org/#/c/716437/ doesn't mention why it's there ... | 23:10 |
| ianw | it either does something, or was accidentally left in ... hard to tell :) | 23:11 |
| clarkb | ianw: I guess we can add it back in if necessary? I think the only functional difference is filtering executables | 23:11 |
| clarkb | and if we've got non executable files in /bin maybe we should fix that? | 23:11 |
| ianw | yeah, my preference would be to add it back with a changelog if there is an issue to be aware of | 23:12 |
| clarkb | ok change is approved | 23:12 |
| clarkb | you can -W if I did that too quick :) | 23:13 |
| clarkb | mordred: fwiw I didn't approve https://review.opendev.org/#/c/721102/9 because it needs watching and there were a few things going on earlier today. Its late in the day now, but I can help watch that go in tomorrow morning if you'll be around | 23:17 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Update functional tests https://review.opendev.org/722548 | 23:17 |
| openstackgerrit | Ian Wienand proposed opendev/glean master: [wip] drop release test https://review.opendev.org/722549 | 23:17 |
| clarkb | then once that is in I think we can land eavesdrop after assuming it gets another round of review from a second reviewer | 23:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 23:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 23:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 23:19 |
| mordred | clarkb: ++ I agree | 23:27 |
| mordred | clarkb: I think we can land a bunch of that tomorrow actually | 23:27 |
| mordred | clarkb: should be able to do nodepool launchers and zuul - I think they're both ready to go | 23:27 |
| mordred | clarkb: oh - I take that back - nodepool is - but I need to update zuul to not do compose up like we did for nodepool | 23:27 |
| clarkb | I'm updating my tumbleweed install right now, its removing a bunch of python2 as part of that. Unexpected removal due to this: bzr | 23:38 |
| clarkb | mordred: ^ I don't know if you bzr anything at all anymore but that could be important as python2 dies | 23:38 |
| clarkb | basically now is a good time to bzr to git I guess | 23:38 |
| ianw | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e50/721511/12/check/tripleo-buildimage-overcloud-full-centos-7-train/e50f174/job-output.txt | 23:42 |
| ianw | 2020-04-23 23:28:47.039228 | primary | "msg": "failed to create temporary content file: [Errno 104] Connection reset by peer" | 23:42 |
| ianw | it's unclear if that's trying to get to an infra server ... | 23:43 |
| clarkb | ianw: it says "Download tripleo source image" I doubt we are hosting that but it could be | 23:43 |
| clarkb | lets see if codesearch will say more | 23:44 |
| clarkb | ianw: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/oooci-build-images/defaults/main.yaml#L1 that is the default which isn't us | 23:44 |
| clarkb | but could be set in CI to somethign else I guess | 23:44 |
| ianw | hrm i'm guessing that's likely it, sorry just thought it might be our git on the other end | 23:45 |
| ianw | sigh, and now https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c69/721779/6/check/nodepool-build-image-siblings/c69115b/job-output.txt | 23:47 |
| ianw | Could not connect to prod.debian.map.fastly.net:80 (151.101.24.204), connection timed out Could not connect to deb.debian.org:80 (151.101.26.133), connection timed out | 23:47 |
| clarkb | that ran in sjc1 so not an ipv6 only cloud problem | 23:47 |
| clarkb | *vexxhost sjc1 | 23:47 |
| ianw | yeah the other was openedge ... maybe i'm just unlucky | 23:48 |
| ianw | http://zuul.openstack.org/stream/027fae83c27948389e3259feb8cc9447?logfile=console.log will be the one that tries to boot focal | 23:51 |
| openstackgerrit | Merged openstack/diskimage-builder master: pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install https://review.opendev.org/721763 | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!