Saturday, 2020-03-14

ianwis there a particular rush if we get fedora images back in rotation now just using this container?00:01
fungiit would be nice to eliminate the devstack backdoor, independent of the fedora image build challenge00:02
clarkbright that00:02
ianwclarkb: see my comments, i'd appreciate if we could keep track of these things we say we'll do00:08
clarkbianw: ya I can do that, however I'm not sure we can commit to tracking them in devstack?00:09
clarkbunless the list is static I suppose00:09
clarkb(I would have to think about that more)00:09
ianwi'm happy with a static list, but i would like it to be in devstack.  there's chagnes out there for cirros 0.5.0 atm for example that should then add it to the cache in devstack00:10
clarkbright, I'm just not sure what the best way is to handle that for multiarch (multiple files maybe)00:10
clarkbbut we can add tasks for it now then sort out implementation later00:10
ianwyep, that's all i want, so we have a list of things to do that's !irc :)00:11
ianwif we get a fedora image somewhere, gates should be unbroken?00:12
clarkbya00:12
ianwi think i saw a fix for rax tls go in, limestone won't work without the path update which i can look at later, but we should get something up00:13
clarkbianw: I updated that story with two tasks and some notes. Does that seem to sufficiently capture the need here?00:14
clarkbif so want to approve the change? I can keep an eye on it after dinner and see if it breaks any builds00:14
ianwclarkb: yes, thanks.  if you'd like to approve go ahead, i'm pushing up against commitments here00:15
clarkb(I put the one task under project-config since it was already there but it could just as easily be a devstack task)00:15
ianwjust as long as there is a task :)00:16
clarkbyup there is one now00:16
ianwi think the way forward here is to try switching the gate container tests to podman and investigate the mount issues, see what happens there00:16
ianwif that works out, bring up a new nb04 with that -- to avoid any potential of name conflicts00:16
ianwif it's not working out, we can fall back to converting it to docker in production, which we know works now00:17
ianwin the mean time, leave nb01.opendev.org emergency and just doing what it's doing00:17
clarkbya I can help with the rebuild into nb04 on monday if we are ready at that point (podman/docker sorted out I guess)00:18
ianw2020-03-14 00:24:15.360 | Build completed successfully00:24
ianwsee it's that easy!00:24
ianwthat container is still not happy talking to rax00:25
openstackgerritMerged openstack/project-config master: Statically cache devstack images and packages  https://review.opendev.org/71308100:25
*** openstack has joined #opendev00:37
*** ChanServ sets mode: +o openstack00:37
clarkbianw: different web servers in different regions?00:43
clarkbthat should be sufficient to get nodepool testing happy though and we can land things and make updates monday00:43
ianwok, i agree we have a steady state and a path forward to work on00:51
ianwi'm going to head off now, thanks everyone for fighting the fire i started!00:52
clarkbin the end I think we discovered multiple fires :)00:52
clarkbwhatever builds after | gentoo-17-0-systemd-0000137550  | gentoo-17-0-systemd  | nb01        | qcow2,raw,vhd | building | 00:00:32:16 | on nb01 should use the new static list of caching things01:00
clarkbI believe | centos-7-0000123969             | centos-7             | nb02        | qcow2,raw,vhd | ready    | 00:00:41:44 | was built with the new stuff02:29
clarkbI'm trying to confirm from log files now02:29
clarkblooks like it did what I expected02:31
clarkband cache-devstack-repos doesn't show up in the log as expected02:32
clarkbI think this is working as expected02:32
clarkband with that I'll try to weekend niow02:32
*** SotK has quit IRC03:28
*** SotK has joined #opendev03:29
*** openstackstatus has quit IRC03:30
*** DSpider has joined #opendev08:06
openstackgerritMerged openstack/project-config master: add vexxhost/openstack-operator  https://review.opendev.org/71308015:42
mordredianw, clarkb: you konw - we're running gitea in docker - we could also make the call to just keep this with docker and convert gerrit to docker instead of podman15:50
mordredthere's things I like about using podman - like the ability to run the containers from a non-root account - but we're not really making use of those - it may just not be worth it for the time being to add podman to the mix15:51
openstackgerritMonty Taylor proposed opendev/system-config master: Switch back to docker for gerrit and nodepool-builder  https://review.opendev.org/71310116:09
mordredianw, clarkb: there's a patch to switch nb to docker - and also to do the same for gerrit16:09
clarkbya I think I could still go either way. podman is likely to be more viable long term but may not quite be there yet16:28
mordredyeah - or may just be more work/investigation from us than we want to focus on just now16:43
mordredclarkb: it's possible the "podman in prod / don't use docker" story is more "use cri-o behind k8s" instead of "use podman-compose" - and I think we're definitely not to that yet :)16:44
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: wait-for-pods: Wait for all pods to become Ready  https://review.opendev.org/71310718:15
*** DSpider has quit IRC22:00
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-docker: add option to use buildset registry  https://review.opendev.org/71311522:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!