| *** paladox has quit IRC | 00:13 | |
| *** paladox has joined #opendev | 00:15 | |
| *** JayF has quit IRC | 00:33 | |
| *** mlavalle has quit IRC | 02:06 | |
| *** _mlavalle_1 has joined #opendev | 02:07 | |
| *** ysandeep|away is now known as ysandeep | 03:19 | |
| *** hamalq has quit IRC | 04:01 | |
| *** ykarel has joined #opendev | 05:05 | |
| *** ysandeep is now known as ysandeep|afk | 05:08 | |
| *** whoami-rajat__ has joined #opendev | 05:25 | |
| *** marios has joined #opendev | 06:46 | |
| *** ralonsoh has joined #opendev | 07:03 | |
| *** dirk has quit IRC | 07:09 | |
| *** dirk has joined #opendev | 07:10 | |
| *** fressi has joined #opendev | 07:27 | |
| *** noonedeadpunk has joined #opendev | 07:39 | |
| *** ysandeep|afk is now known as ysandeep | 07:42 | |
| *** fressi has quit IRC | 07:46 | |
| *** eolivare has joined #opendev | 07:46 | |
| *** fressi has joined #opendev | 07:58 | |
| *** hashar has joined #opendev | 08:07 | |
| *** fressi has quit IRC | 08:21 | |
| *** andrewbonney has joined #opendev | 08:24 | |
| *** rpittau|afk is now known as rpittau | 08:32 | |
| *** fressi has joined #opendev | 08:35 | |
| *** ykarel is now known as ykarel|lunch | 08:39 | |
| *** tkajinam has quit IRC | 08:55 | |
| *** tosky has joined #opendev | 08:56 | |
| *** fressi has quit IRC | 09:12 | |
| *** fressi has joined #opendev | 09:17 | |
| *** ysandeep is now known as ysandeep|lunch | 09:22 | |
| *** hashar has quit IRC | 09:28 | |
| *** hashar has joined #opendev | 09:28 | |
| *** brinzhang has quit IRC | 09:38 | |
| *** ykarel|lunch is now known as ykarel | 09:54 | |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Replace Fedora 31 functional tests with Fedora 33 https://review.opendev.org/c/openstack/diskimage-builder/+/768092 | 09:56 |
|---|---|---|
| *** DSpider has joined #opendev | 09:57 | |
| *** dtantsur|afk is now known as dtantsur | 09:57 | |
| *** whoami-rajat__ has quit IRC | 10:13 | |
| *** ttx has quit IRC | 10:16 | |
| *** ttx has joined #opendev | 10:17 | |
| *** hashar is now known as hasharLunch | 10:43 | |
| *** ysandeep|lunch is now known as ysandeep | 11:32 | |
| *** hasharLunch is now known as hashar | 12:32 | |
| *** SotK has quit IRC | 12:52 | |
| *** SotK has joined #opendev | 12:52 | |
| *** whoami-rajat___ has joined #opendev | 13:18 | |
| *** stzsch has quit IRC | 13:38 | |
| *** _mlavalle_1 has quit IRC | 13:58 | |
| *** mlavalle has joined #opendev | 13:58 | |
| *** dtantsur is now known as dtantsur|brb | 14:00 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 14:08 |
| danpawlik | ianw_pto: hey. I'm trying to reproduce error for dib-nodepool-functional-openstack-fedora-33-src CI job in https://review.opendev.org/c/openstack/diskimage-builder/+/768092 and normally all is working correctly. Maybe you will see something strange | 14:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 14:15 |
| *** rpittau is now known as rpittau|afk | 14:46 | |
| *** ysandeep is now known as ysandeep|away | 14:49 | |
| fungi | danpawlik: i can hopefully take a look after i finish checking over our afs volume replicas for any more fallout from yesterday | 14:53 |
| *** fressi has quit IRC | 14:54 | |
| danpawlik | ack fungi, thank you | 14:54 |
| *** fressi has joined #opendev | 14:54 | |
| *** fressi has left #opendev | 14:55 | |
| fungi | tarballs site volume has been releasing for the past 8 minutes, should be better shortly i think | 15:08 |
| fungi | #status log killed a hung vos examine process from yesterday's afs issues which was preventing static site volume releases from proceeding | 15:21 |
| openstackstatus | fungi: finished logging | 15:21 |
| fungi | danpawlik: https://zuul.opendev.org/t/openstack/build/ae7382cda6124f79b1431afbff8abd5b/log/nodepool/builds/test-image-0000000001.log#839-855 | 15:26 |
| danpawlik | fungi: indeed, but why it happens so. Normally in f33 is rpm is in 4.16 | 15:37 |
| danpawlik | fungi: but on buildroot which is bionic is 4.14.1 | 15:38 |
| danpawlik | fungi: but I'm not so familiar with DIB project | 15:39 |
| fungi | danpawlik: seems like maybe we need a newer rpm package for ubuntu... even focal is still carrying rpm 4.14.1: https://packages.ubuntu.com/rpm | 15:41 |
| fungi | presumably because that's the newest in debian as well: https://packages.debian.org/rpm | 15:42 |
| fungi | danpawlik: https://bugs.debian.org/979062 | 15:43 |
| openstack | Debian bug 979062 in src:rpm "please update to rpm 4.16.x" [Important,Open] | 15:43 |
| *** dtantsur|brb is now known as dtantsur | 15:45 | |
| danpawlik | thanks fungi++ | 15:46 |
| danpawlik | fungi: I was thinking that Im doing something wrong | 15:46 |
| danpawlik | fungi: could you comment in the PS or should I do :>? | 15:47 |
| fungi | danpawlik: this may be the impetus to go ahead with mordred's previous suggestions of switching to bootstrapping images from existing docker images instead of bootstrapping into a completely empty chroot | 15:51 |
| fungi | presumably there's a f33 x86_64 docker image which contains the correct version of rpm already | 15:52 |
| clarkb | fungi: re the vos examine that was on the side where we do vos release? and not a lock on the server side we needed to break? | 15:57 |
| fungi | clarkb: correct | 15:57 |
| clarkb | fungi: danpawlik aiui ianw is still working on the chroot bootstrap from docker images | 15:57 |
| clarkb | not sure what is needed next there though | 15:57 |
| fungi | the cronjob on mirror-update.o.o to run the static release script every 5 minutes and it does a vos examine before releasing each volume, that was indefinitely waiting for a response from afs01.dfw from back when the server was hung yesterday. the replicas weren't locked and seemed intact | 15:59 |
| fungi | once i killed the vos examine, the next run of the cronjob seems to have worked normally (albeit took a while because of the backlog of writes) | 15:59 |
| clarkb | got it | 16:00 |
| mordred | fungi: I think using docker images as the source for the base chroot for dib would get us a long way | 16:09 |
| mordred | it's a non-zero amount of work still because of some of dib's assumptions about how root elements work and what they provide - but it's also still doable | 16:09 |
| clarkb | yup I think simple testing of it is owrking now | 16:09 |
| mordred | awesome | 16:11 |
| *** jpena has joined #opendev | 16:15 | |
| *** hashar has quit IRC | 16:25 | |
| *** ykarel_ has joined #opendev | 16:26 | |
| *** ykarel has quit IRC | 16:29 | |
| *** whoami-rajat___ is now known as whoami-rajat__ | 16:39 | |
| *** marios is now known as marios|out | 16:44 | |
| *** marios|out has quit IRC | 16:53 | |
| prometheanfire | fungi: clarkb can we cut a release of glean? | 16:54 |
| fungi | prometheanfire: seems timely, sure | 16:55 |
| clarkb | ya I'm not opposed other than it is friday. But we can trigger a nodepool image build early today once that publishes if we need to | 16:56 |
| prometheanfire | cool | 16:56 |
| clarkb | fungi: looks like we have 2 bug fixes so would be 1.18.2 ? Do you want to push that tag or should I go get my keys | 16:57 |
| fungi | i can in a few minutes, sure | 16:58 |
| fungi | 1.18.2 seems reasonable given the merged commits since 1.18.1 | 16:58 |
| prometheanfire | ack | 17:00 |
| *** jpena is now known as jpena|off | 17:02 | |
| *** ykarel_ is now known as ykarel | 17:21 | |
| fungi | glean 1.18.2 is up on pypi | 17:39 |
| *** sboyron has quit IRC | 17:39 | |
| fungi | #status log released glean 1.18.2 | 17:39 |
| openstackstatus | fungi: finished logging | 17:40 |
| fungi | prometheanfire: clarkb: ^ | 17:40 |
| *** sboyron has joined #opendev | 17:41 | |
| ttx | fungi, clarkb: do you know what the current state of our ARM support is? | 17:42 |
| ttx | Do we have ARM hosts in nodepool at the moment? | 17:42 |
| *** ykarel has quit IRC | 17:42 | |
| clarkb | ttx: we do, they are arm64 vms running on top of a cloud supplied by linaro | 17:44 |
| ttx | ok cool. thanks! | 17:44 |
| clarkb | ttx: opendev runs some testing on it for the stuff we deploy into the arm64 cloud. Kolla is also doing testing on it iirc. And we do python crptography wheel builds for arm64 | 17:44 |
| *** JayF has joined #opendev | 17:45 | |
| clarkb | there isn't huge capcaity and it is provided by a single cloud so all the caveats of outages and such apply, but it does work | 17:45 |
| ttx | clarkb: was there attempts to get capacity from vexxhost ARM cloud, for redundancy? Or is it completely different | 17:46 |
| clarkb | I don't think we have tried. We've largely relied on folks that grok arm and its requirements to help drive it (linaro really) | 17:47 |
| ttx | ok, makes sense | 17:47 |
| clarkb | fungi: prometheanfire: I see new glean on pypi | 17:55 |
| clarkb | I was trying to infer if any images are due for a rebuild soon off of https://nb01.opendev.org/images but even sorting by last modified doesn't make that very clear. | 17:58 |
| clarkb | aha a debian buster run is happening now | 17:59 |
| clarkb | if you go to / you see the build logs and the sort there does help | 17:59 |
| clarkb | https://nb01.opendev.org/debian-buster-0000007267.log that image should use the new glean | 17:59 |
| *** andrewbonney has quit IRC | 18:01 | |
| fungi | ttx: notable about the linaro cloud is that the entire deployment of openstack there is running on arm hardware, not just compute hypervisor hosts | 18:11 |
| *** eolivare has quit IRC | 18:14 | |
| *** ralonsoh has quit IRC | 18:17 | |
| clarkb | that buster build looks successful but isn't complete yet (it is hashing the resulting artifacts) | 18:32 |
| clarkb | once hte image is built then it needs to be uploaded. Then we can manually boot one or see if nodepool has successfully booted buster | 18:32 |
| *** dtantsur is now known as dtantsur|afk | 18:43 | |
| clarkb | and the build completed successfully a few minuts ago so ya in a few hours I guess we double check glean still works on buster | 18:43 |
| clarkb | 021-01-08 18:18:00.277 | Successfully installed glean-1.18.2 <- ocnfirms it used the new glean too | 18:43 |
| *** whoami-rajat__ has quit IRC | 18:58 | |
| *** hamalq has joined #opendev | 19:10 | |
| *** hamalq has quit IRC | 19:33 | |
| *** hamalq has joined #opendev | 19:33 | |
| *** stzsch has joined #opendev | 20:31 | |
| *** sboyron has quit IRC | 20:36 | |
| openstackgerrit | lotorev vitaly proposed zuul/zuul-jobs master: Clarity tox_environment accepts dictionary not list https://review.opendev.org/c/zuul/zuul-jobs/+/769433 | 20:43 |
| openstackgerrit | lotorev vitaly proposed zuul/zuul-jobs master: Document Python siblings handling for tox role https://review.opendev.org/c/zuul/zuul-jobs/+/768823 | 20:43 |
| *** diablo_rojo__ has joined #opendev | 20:59 | |
| *** slaweq has quit IRC | 21:13 | |
| prometheanfire | clarkb: fungi thanks :D | 21:13 |
| clarkb | prometheanfire: if you can confirm it is working as expected for you too that would be great. I'm about to actually ssh into nodes we're running that shoul be updated | 21:14 |
| prometheanfire | ya, I'm going to build images | 21:15 |
| *** cloudnull has quit IRC | 21:18 | |
| clarkb | debian-buster 0022454355 was booted in limestone with the new glean and seems to work | 21:19 |
| clarkb | I checked that by running /usr/glean/bin/pip freeze to get the glean version and being able to ssh into it implies that basic functionality is working | 21:19 |
| clarkb | one thing I'm not able to easily check is the /etc/hosts contents bceause it is running an openstack ansible job whihc changes the hostname and stuff in /etc/hosts | 21:19 |
| *** beep1560 has joined #opendev | 21:19 | |
| clarkb | and nodepool just cleaned it up | 21:20 |
| clarkb | I don't think any other images have updated since glean release happened. But buster working is probably a good enough canary | 21:21 |
| *** hashar has joined #opendev | 21:24 | |
| *** beep1560 has quit IRC | 21:25 | |
| *** cloudnull has joined #opendev | 21:28 | |
| *** hashar has quit IRC | 21:29 | |
| fungi | agreed | 22:04 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!