| *** hrw has quit IRC | 00:10 | |
| *** hrw has joined #opendev | 00:12 | |
| *** CeeMac has quit IRC | 00:12 | |
| *** hrw has quit IRC | 00:46 | |
| *** hrw has joined #opendev | 00:47 | |
| *** pongboom has quit IRC | 00:49 | |
| openstackgerrit | Merged opendev/glean master: Fix Gentoo "is" comparisons https://review.opendev.org/c/opendev/glean/+/788126 | 00:58 |
|---|---|---|
| kevinz | ianw: Sorry yesterday I do not check the irc. Is there anything I can help? | 01:30 |
| ianw | kevinz: we were wondering if it would be possible (maybe even better?) to upload .raw images to linaro, instead of qcow2? | 01:31 |
| kevinz | ianw: Yes, I think it should be fine | 01:31 |
| ianw | kevinz: ok, we can propose a change to switch it; should be a bit friendlier on the builder as it won't have to also do qcow2 | 01:33 |
| kevinz | ianw: OK, cool | 01:35 |
| ianw | kevinz: other than that, i did a bit of ping testing. there still seems to be the odd packet loss | 01:35 |
| ianw | I was just doing with "ping -D -O -q -<4 / 6> zk02.openstack.org | 01:36 |
| ianw | both ipv6 and ipv4 dropped some packets; not many, but i guess enough to be annoying to long-running things like image uploads and zk connections | 01:36 |
| kevinz | ianw: Could I ping zk02.openstack.org from our router? I need to check if there are some issue with infra netowrk | 01:37 |
| ianw | kevinz: sure. actually clarkb is in the process of changing them to zk04, zk05, zk06 as part of upgrades | 01:38 |
| ianw | but i think any of them exhibit the same issue | 01:40 |
| ianw | 2021-04-28 00:03:04,652 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 01:40 |
| ianw | 2021-04-28 00:26:30,509 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 01:40 |
| ianw | 2021-04-28 00:58:03,900 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 01:40 |
| ianw | 2021-04-28 01:04:29,465 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 01:40 |
| ianw | 2021-04-28 01:25:56,801 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 01:40 |
| ianw | that's the dropped connections from midnight | 01:41 |
| kevinz | OK | 01:44 |
| kevinz | ianw: how long period can you observe packet loss? | 01:45 |
| ianw | i guess i left it running for 3851 packet, i think it does one a second, so about an hour | 01:47 |
| *** hamalq has quit IRC | 01:52 | |
| *** hamalq has joined #opendev | 01:52 | |
| *** priteau has quit IRC | 02:03 | |
| kevinz | ianw: OK, I'm trying on our router | 02:15 |
| *** redrobot has quit IRC | 02:45 | |
| *** hemanth_n has joined #opendev | 02:49 | |
| openstackgerrit | Merged opendev/glean master: Update hacking and fix pep8 violations https://review.opendev.org/c/opendev/glean/+/782296 | 02:54 |
| *** ykarel has joined #opendev | 03:41 | |
| *** redrobot has joined #opendev | 03:50 | |
| *** vishalmanchanda has joined #opendev | 04:40 | |
| *** mkowalski has quit IRC | 04:42 | |
| *** mkowalski has joined #opendev | 04:42 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove dib-nodepool-functional-openstack-ubuntu-bionic https://review.opendev.org/c/openstack/diskimage-builder/+/788404 | 05:06 |
| *** ykarel has quit IRC | 05:09 | |
| *** marios has joined #opendev | 05:20 | |
| *** ralonsoh has joined #opendev | 05:27 | |
| *** sboyron_ has joined #opendev | 05:57 | |
| *** slaweq has joined #opendev | 06:00 | |
| *** sboyron has joined #opendev | 06:01 | |
| *** sboyron_ has quit IRC | 06:01 | |
| *** lpetrut has joined #opendev | 06:16 | |
| *** ysandeep|away is now known as ysandeep | 06:17 | |
| *** eolivare has joined #opendev | 06:25 | |
| *** ykarel has joined #opendev | 06:56 | |
| *** amoralej|off is now known as amoralej | 06:58 | |
| *** hashar has joined #opendev | 07:00 | |
| *** fressi has joined #opendev | 07:02 | |
| *** hamalq has quit IRC | 07:03 | |
| *** ykarel is now known as ykarel|pto | 07:13 | |
| *** andrewbonney has joined #opendev | 07:15 | |
| *** ykarel|pto has quit IRC | 07:21 | |
| *** jpena|off is now known as jpena | 07:21 | |
| *** rpittau|afk is now known as rpittau | 07:36 | |
| *** hamalq has joined #opendev | 07:39 | |
| *** ysandeep is now known as ysandeep|lunch | 08:02 | |
| *** hamalq has quit IRC | 08:02 | |
| *** hamalq has joined #opendev | 08:09 | |
| *** brinzhang0 has joined #opendev | 08:39 | |
| *** mordred has quit IRC | 08:41 | |
| *** brinzhang_ has quit IRC | 08:42 | |
| *** mordred has joined #opendev | 08:50 | |
| kevinz | ianw: checking from our router, 2.5 hours without any packet loss for IPV4. Will check the network issue from our internal cluster | 08:50 |
| *** mkowalski has quit IRC | 09:09 | |
| *** mkowalski has joined #opendev | 09:11 | |
| *** dhellmann has quit IRC | 09:17 | |
| *** dhellmann has joined #opendev | 09:17 | |
| *** priteau has joined #opendev | 09:22 | |
| *** ysandeep|lunch is now known as ysandeep | 09:26 | |
| *** hashar has quit IRC | 09:35 | |
| *** hamalq has quit IRC | 09:44 | |
| *** tosky has joined #opendev | 09:45 | |
| *** vishalmanchanda has quit IRC | 10:00 | |
| *** vishalmanchanda has joined #opendev | 10:11 | |
| *** whoami-rajat_ has joined #opendev | 10:22 | |
| *** dtantsur is now known as dtantsur|brb | 10:32 | |
| *** brinzhang_ has joined #opendev | 10:59 | |
| *** brinzhang0 has quit IRC | 11:02 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** dtantsur|brb is now known as dtantsur | 11:41 | |
| fungi | ianw: clarkb (when you wake up): this is an interesting failure condition: https://zuul.opendev.org/t/openstack/build/f5858c54e9e2473baf9634a26ee8242e | 12:02 |
| fungi | "Ensure Zookeeper not running" fails on a dib-nodepool-functional-openstack-centos-8-stream-src job | 12:02 |
| *** ysandeep is now known as ysandeep|afk | 12:12 | |
| *** jpena|lunch is now known as jpena | 12:19 | |
| *** amoralej is now known as amoralej|lunch | 12:22 | |
| *** whoami-rajat_ is now known as whoami-rajat | 12:32 | |
| *** hemanth_n has quit IRC | 12:32 | |
| openstackgerrit | daniel.pawlik proposed opendev/lodgeit master: Fixed issue related to new Pillow lib https://review.opendev.org/c/opendev/lodgeit/+/787734 | 12:40 |
| dpawlik | fungi, clarkb: hey, could you check later please ^^ | 12:40 |
| *** lpetrut has quit IRC | 12:44 | |
| *** ysandeep|afk is now known as ysandeep | 12:56 | |
| *** iurygregory has quit IRC | 12:59 | |
| fungi | dpawlik: frickler left you some comments | 13:09 |
| dpawlik | ack, thanks fungi and frickler | 13:12 |
| *** amoralej|lunch is now known as amoralej | 13:15 | |
| *** toomer has quit IRC | 13:19 | |
| *** toomer has joined #opendev | 13:19 | |
| *** fdegir has quit IRC | 13:23 | |
| *** fdegir has joined #opendev | 13:24 | |
| openstackgerrit | daniel.pawlik proposed opendev/lodgeit master: Fix issue related to new Pillow lib; change hash string encode https://review.opendev.org/c/opendev/lodgeit/+/787734 | 13:28 |
| *** iurygregory has joined #opendev | 13:35 | |
| *** mkowalski has quit IRC | 13:40 | |
| *** mkowalski has joined #opendev | 13:41 | |
| *** hrw has quit IRC | 13:52 | |
| *** zoharm has joined #opendev | 14:14 | |
| *** fressi has quit IRC | 14:18 | |
| openstackgerrit | daniel.pawlik proposed opendev/lodgeit master: Fix issue related to new Pillow lib; change hash string encode https://review.opendev.org/c/opendev/lodgeit/+/787734 | 14:39 |
| clarkb | fungi: I'll take a look. Making some tea now | 14:48 |
| clarkb | fungi: I plan to try and catch up on the openstack logging stuff too (can move that into the dib side if no one else has) | 14:48 |
| *** zoharm has quit IRC | 14:59 | |
| *** hashar has joined #opendev | 15:03 | |
| frickler | mnaser: do you have any update on the v6 situation for vexxhost yet? | 15:12 |
| clarkb | fungi: I suspect that `pkill -f zookeeper` returns >0 if zookeeper is not running and that is why we ignore errors on that task | 15:18 |
| clarkb | fungi: ianw: the actual error appears to be 'manifest for zuul/nodepool-launcher:siblings not found: manifest unknown: manifest unknown' | 15:19 |
| clarkb | which implies the parent job that does image builds didn't work? | 15:19 |
| *** mlavalle has joined #opendev | 15:22 | |
| clarkb | corvus: looks like the intermediate registry is having socket/ssl problems? `sudo docker logs --since 2021-04-28T00:00:00 registry-docker_registry_1` I'm not quite sure how to parse what that is telling us yet | 15:25 |
| clarkb | but it seems to be causing jobs like the dib functional jobs to timeout | 15:25 |
| clarkb | seems like maybe we aren't catching some exception that we should be and that caused the whole thing to crash? however the actual process and the container seem to be running | 15:26 |
| clarkb | (I'm inclined to restart it but wanted to give you a chance to look at the error first in case we want to do further debugging) | 15:27 |
| clarkb | fwiw the tracebacks don't seem to bubble up into our application, they stop within cherrypy so not clear if we should be handling that ourselves or if this is a cherrypy bug that needs fixing | 15:32 |
| *** d34dh0r53 has quit IRC | 15:48 | |
| *** ysandeep is now known as ysandeep|away | 15:56 | |
| clarkb | ok I'm going to go ahead and restart it now. I recordred the tracebacks in my homedir on that host | 15:57 |
| openstackgerrit | Merged opendev/lodgeit master: Fix issue related to new Pillow lib; change hash string encode https://review.opendev.org/c/opendev/lodgeit/+/787734 | 15:58 |
| *** d34dh0r53 has joined #opendev | 15:59 | |
| *** marios is now known as marios|out | 16:13 | |
| *** ysandeep|away is now known as ysandeep | 16:14 | |
| *** jpena is now known as jpena|off | 16:18 | |
| yoctozepto | I love it how gerrit says it's running dirty | 16:18 |
| clarkb | because we do our own builds (there is probably a way to fix that) | 16:19 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| *** d34dh0r53 has quit IRC | 16:22 | |
| *** marios|out has quit IRC | 16:30 | |
| *** d34dh0r53 has joined #opendev | 16:30 | |
| corvus | clarkb: interesting; i'm unfamiliar with those errors, and they don't mean anything to m.... | 16:37 |
| clarkb | corvus: that was the end of the log too so I'm pretty sure it was functionally fatal but the process kept going | 16:38 |
| clarkb | I suspect a bug in cherrypy | 16:38 |
| clarkb | maybe we should rebuild the registry and pick up newer cherrypy if available? | 16:39 |
| *** rpittau is now known as rpittau|afk | 16:39 | |
| openstackgerrit | Ghanshyam proposed opendev/irc-meetings master: Update TC office hours time for Xena cycle https://review.opendev.org/c/opendev/irc-meetings/+/788552 | 16:39 |
| corvus | clarkb: maybe; i just happened to be writing 2 changes to the registry; we could merge those :) | 16:39 |
| clarkb | ++ I can take a look shortly | 16:40 |
| *** amoralej is now known as amoralej|off | 16:42 | |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: debian-minimal: Set bullseye version https://review.opendev.org/c/openstack/diskimage-builder/+/787665 | 16:48 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: Collect openstack logs https://review.opendev.org/c/openstack/diskimage-builder/+/788553 | 16:48 |
| clarkb | fungi: ianw ^ that moves the openstack log collection into dib whcih should make that mergeable | 16:48 |
| *** hamalq has joined #opendev | 17:04 | |
| *** hamalq has quit IRC | 17:05 | |
| *** hamalq has joined #opendev | 17:06 | |
| openstackgerrit | Jeremy Stanley proposed openstack/diskimage-builder master: debian-minimal: bullseye: /updates -> -security https://review.opendev.org/c/openstack/diskimage-builder/+/787454 | 17:07 |
| fungi | rackspace opened tickets to let us know that logstash-worker06.openstack.org and lists.katacontainers.io were both on hardware which experienced a failure | 17:11 |
| fungi | lists.k.i seems to be up and running the various python processes mailman expects | 17:12 |
| *** ysandeep is now known as ysandeep|away | 17:14 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (4) https://review.opendev.org/c/openstack/project-config/+/788555 | 17:22 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (5) https://review.opendev.org/c/openstack/project-config/+/788556 | 17:28 |
| *** lbragstad has quit IRC | 17:37 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (6) https://review.opendev.org/c/openstack/project-config/+/788557 | 17:37 |
| *** lbragstad has joined #opendev | 17:38 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (7) https://review.opendev.org/c/openstack/project-config/+/788558 | 17:42 |
| *** gothicserpent has quit IRC | 17:43 | |
| *** eolivare has quit IRC | 17:44 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (8) https://review.opendev.org/c/openstack/project-config/+/788561 | 17:49 |
| *** gothicserpent has joined #opendev | 17:49 | |
| openstackgerrit | Kevin Putnam proposed openstack/project-config master: Adds r/stx.5.0 to the list of branches to promote for starlingx docs. https://review.opendev.org/c/openstack/project-config/+/788562 | 17:52 |
| *** ralonsoh has quit IRC | 17:53 | |
| *** andrewbonney has quit IRC | 18:00 | |
| openstackgerrit | Radosław Piliszek proposed opendev/gerritbot master: Add branch to all the remaining event messages https://review.opendev.org/c/opendev/gerritbot/+/788565 | 18:18 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (9) https://review.opendev.org/c/openstack/project-config/+/788567 | 18:26 |
| *** ifest has joined #opendev | 18:38 | |
| openstackgerrit | Radosław Piliszek proposed opendev/gerritbot master: Add branch to all the remaining event messages https://review.opendev.org/c/opendev/gerritbot/+/788565 | 18:39 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (10) https://review.opendev.org/c/openstack/project-config/+/788569 | 18:46 |
| yoctozepto | infra-root: how costly would it be to make gerritbot run with the fix above? ^ :-) | 18:51 |
| fungi | yoctozepto: do you mean in beers, or bitcoin? ;) | 18:52 |
| clarkb | yoctozepto: I'm not sure I agree with the premise. Many repos are single branch | 18:52 |
| yoctozepto | fungi: I mean in using your precious time | 18:53 |
| yoctozepto | clarkb: other messages show branch | 18:53 |
| clarkb | the biggest cost is you'll be losing message bytes in an already constrained system | 18:53 |
| * yoctozepto needs to rethink the way he builds his questions | 18:53 | |
| clarkb | yoctozepto: ya I'm not saying I'm against it. Just want to point out that the branch isn't necessarily useful even if we emit it | 18:53 |
| fungi | right, as it stands the messages are already a bit longish and in many cases may exceed the max message length for irc and get truncated/rejected/other breakage | 18:54 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (11) https://review.opendev.org/c/openstack/project-config/+/788571 | 18:54 |
| clarkb | and the biggest "cost" is irc messages are limited to 512 bytes including the message prefix stuff so you'd be cutting into that | 18:54 |
| yoctozepto | 512 bytes is a lot, long commit titles are bad anyway | 18:55 |
| openstackgerrit | Kevin Putnam proposed openstack/project-config master: Adds r/stx.5.0 to the list of branches to promote for starlingx docs. https://review.opendev.org/c/openstack/project-config/+/788562 | 18:56 |
| clarkb | yup | 18:56 |
| clarkb | my hunch is that this was simply overlooked and not intentional | 18:56 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (12) https://review.opendev.org/c/openstack/project-config/+/788572 | 18:56 |
| yoctozepto | clarkb: I checked git history and it was no branch at first | 18:57 |
| fungi | 512 bytes aren't really a *lot* when you consider that includes the channel name, bot nick, and whatever fields are being included in the message, along with multi-byte utf-8 encoding of some chanracters... at a minimum the change subject is what should wind up truncated to compensate, but right now it's not the last field (the change url is) so that's not trivial | 18:57 |
| yoctozepto | then it was added to the other messages | 18:57 |
| clarkb | fungi: ya but the vast majority seem to come in well under 512 | 18:58 |
| clarkb | just based on what gerritbot does emit | 18:58 |
| fungi | we might also be able to save some bytes by converting to "old-style" gerrit urls (stripping the project name) | 18:58 |
| yoctozepto | oh, that would be nice | 18:58 |
| yoctozepto | as the project name is already embedded | 18:59 |
| yoctozepto | we can also likely change the order so that the message goes last | 18:59 |
| yoctozepto | but that's optional | 18:59 |
| fungi | the counterargument there is that old-style gerrit urls are no longer guaranteed to be unique, i think? but in practice they still seem to be (for now anyway) | 19:00 |
| clarkb | fungi: they are unique, there is a change number counter somewhere | 19:00 |
| clarkb | I think in all-projects | 19:00 |
| *** hashar has quit IRC | 19:01 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (13) https://review.opendev.org/c/openstack/project-config/+/788573 | 19:02 |
| *** ifest has quit IRC | 19:03 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (14) https://review.opendev.org/c/openstack/project-config/+/788574 | 19:04 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (15) https://review.opendev.org/c/openstack/project-config/+/788577 | 19:08 |
| clarkb | success! https://zuul.opendev.org/t/openstack/build/c7c698c95f2b4590ba95f14a6c97ffc0/log/nodepool/openstack/screen-n-api.txt?severity=3#1803 and by that I mean it failed | 19:11 |
| clarkb | https://zuul.opendev.org/t/openstack/build/c7c698c95f2b4590ba95f14a6c97ffc0/log/nodepool/openstack/screen-keystone.txt?severity=0#8308 keystone doesn't seem to provide much more info though | 19:12 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (16) https://review.opendev.org/c/openstack/project-config/+/788578 | 19:13 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (17) https://review.opendev.org/c/openstack/project-config/+/788580 | 19:17 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (18) https://review.opendev.org/c/openstack/project-config/+/788581 | 19:19 |
| openstackgerrit | Merged openstack/project-config master: Adds r/stx.5.0 to the list of branches to promote for starlingx docs. https://review.opendev.org/c/openstack/project-config/+/788562 | 19:21 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (2) https://review.opendev.org/c/openstack/project-config/+/786739 | 19:22 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (19) https://review.opendev.org/c/openstack/project-config/+/788584 | 19:27 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (3) https://review.opendev.org/c/openstack/project-config/+/786740 | 19:28 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (4) https://review.opendev.org/c/openstack/project-config/+/788555 | 19:29 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (17) https://review.opendev.org/c/openstack/project-config/+/788580 | 19:33 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (16) https://review.opendev.org/c/openstack/project-config/+/788578 | 19:33 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (18) https://review.opendev.org/c/openstack/project-config/+/788581 | 19:33 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (5) https://review.opendev.org/c/openstack/project-config/+/788556 | 19:40 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (6) https://review.opendev.org/c/openstack/project-config/+/788557 | 19:40 |
| *** vishalmanchanda has quit IRC | 19:40 | |
| *** sboyron has quit IRC | 19:41 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (7) https://review.opendev.org/c/openstack/project-config/+/788558 | 19:44 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (8) https://review.opendev.org/c/openstack/project-config/+/788561 | 19:44 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (9) https://review.opendev.org/c/openstack/project-config/+/788567 | 19:44 |
| clarkb | ianw: fungi: to tl;dr the dib functional job failure it seems that the domain is not being supplied with the request. However, we're sourcing the devstack openrc which does set the domain env vars. I've asked about this in #openstack-qa but no responses yet. I think it is possible that this may be a nova issue as it seems to be happening via nova's validate_networks() call and other requests prior | 19:47 |
| clarkb | to the server create work fine using the same credentials | 19:48 |
| clarkb | no clear cause yet though | 19:48 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (10) https://review.opendev.org/c/openstack/project-config/+/788569 | 19:51 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (11) https://review.opendev.org/c/openstack/project-config/+/788571 | 19:51 |
| clarkb | hrm maybe it is happening after validate_networks | 19:52 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (12) https://review.opendev.org/c/openstack/project-config/+/788572 | 19:52 |
| clarkb | _refresh_neutron_extensions_cache() is in the traceback | 19:52 |
| clarkb | I wonder a stale cache / timing issue is why it doesn't fail consistently | 19:52 |
| *** lbragstad_ has joined #opendev | 20:01 | |
| *** lbragstad has quit IRC | 20:03 | |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (13) https://review.opendev.org/c/openstack/project-config/+/788573 | 20:06 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (14) https://review.opendev.org/c/openstack/project-config/+/788574 | 20:06 |
| openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (15) https://review.opendev.org/c/openstack/project-config/+/788577 | 20:07 |
| fungi | clarkb: openstacksdk regression maybe? | 20:50 |
| fungi | the cliaas work hasn't merged yet, right? | 20:51 |
| *** whoami-rajat has quit IRC | 20:52 | |
| *** brinzhang0 has joined #opendev | 20:59 | |
| *** brinzhang_ has quit IRC | 21:02 | |
| *** mkowalski has quit IRC | 21:11 | |
| *** mkowalski has joined #opendev | 21:11 | |
| openstackgerrit | Kevin Putnam proposed openstack/project-config master: Adds docs_branch_path value needed for promoting release branches. https://review.opendev.org/c/openstack/project-config/+/788593 | 21:18 |
| *** slaweq has quit IRC | 21:25 | |
| ianw | clarkb: o/ ... just catching up. so we don't have issues with the intermediate registry any more? i feel like one of my jobs hit that | 21:33 |
| fungi | ianw: in theory the restart has cleared that problem | 21:39 |
| ianw | fungi: i got a response overnight on the aarch64 glibc question in #centos-stream: "<carlwgeorge> ianw: it will be released soon" | 21:46 |
| ianw | so make of that what you will | 21:46 |
| clarkb | fungi: ianw correct looks like the registry restart fixed it | 21:58 |
| clarkb | as far as I could tell it had crashed (tracebacks in my homedir on the host) | 21:58 |
| clarkb | but the process was still running so docker didn't restart it | 21:59 |
| clarkb | ok it has been about 24 hours since the zk cluster upgrade. I haven't seen any indication we will need to rollback | 22:00 |
| clarkb | infra-root ^ ya'll good with me cleaning up the 3 old hosts now? | 22:00 |
| ianw | ++ | 22:03 |
| corvus | ++ | 22:05 |
| clarkb | #status log Deleted zk01-zk03.openstack.org as they have been replaced with zk04-06.opendev.org | 22:07 |
| openstackstatus | clarkb: finished logging | 22:07 |
| clarkb | ianw: fwiw I think https://review.opendev.org/c/openstack/diskimage-builder/+/788553/1 is working proeprly if we can just get it to land now :) | 22:14 |
| clarkb | I'll abandon the nodepool change that did similar | 22:14 |
| ianw | haha yeah, i want to take more of a look in a little after school drop off | 22:16 |
| ianw | i guess we could put a speculative hold on all functional jobs and run it through a few times, hopefully catching one | 22:16 |
| clarkb | ya that may be the next thing to do. Then if the issue is persistent we could hand it off to nova if necessary | 22:17 |
| *** tosky has quit IRC | 22:48 | |
| corvus | ianw: re your q in #zuul | 23:21 |
| ianw | Variant <Job dib-nodepool-functional-openstack-centos-8-stream-src branches: {MatchAny:{ImpliedBranchMatcher:master}} source: openstack/diskimage-builder/.zuul.d/jobs.yaml@master#167> did not match <Change 0x7ffa649ab340 openstack/openstacksdk 788420,1> | 23:21 |
| corvus | 2021-04-28 21:45:33,181 DEBUG zuul.layout: [e: 75d0d82962bd45cc9a168a39c1144cee] Variant <Job dib-nodepool-functional-openstack-centos-8-stream-src branches: {MatchAny:{ImpliedBranchMatcher:master}} source: openstack/diskimage-builder/.zuul.d/jobs.yaml@master#167> did not match <Change 0x7ffa649ab340 openstack/openstacksdk 788420,1> | 23:21 |
| ianw | i think it must be a branch matcher | 23:21 |
| corvus | ianw: also 2021-04-28 21:45:33,181 DEBUG zuul.layout: [e: 75d0d82962bd45cc9a168a39c1144cee] No matching parents for job dib-nodepool-functional-openstack-centos-8-stream-src and change <Change 0x7ffa649ab340 openstack/openstacksdk 788420,1> | 23:22 |
| corvus | ianw: and that change is on stable/victoria, so likely some branch mismatching going on | 23:23 |
| ianw | yeah, the job hierarchy is long | 23:23 |
| ianw | dib-nodepool-functional-openstack-centos-8-stream-src -> dib-nodepool-functional-src-base -> nodepool-functional-container-openstack-siblings-base | 23:24 |
| ianw | -> nodepool-functional-container-openstack-base | 23:26 |
| corvus | ianw: dib is a multi-branch job | 23:28 |
| corvus | ianw: which is why it's getting an implied branch matcher attached to it | 23:28 |
| clarkb | because the feature/v2 branch exists? I think that may have served its usefulness and we can tag it and then delete it? | 23:29 |
| corvus | ianw: assuming you don't want to remove the feature/v2 branch, you may need to set a pragma to tell zuul not to do that | 23:29 |
| corvus | ianw: https://zuul-ci.org/docs/zuul/reference/pragma_def.html#attr-pragma.implied-branch-matchers | 23:29 |
| corvus | clarkb: ya | 23:29 |
| corvus | clarkb: delete would be my recommendation, but you know me, i love deleting branches ;) | 23:29 |
| clarkb | ha, in this case I'm fairly certain the v2 work completed a while back | 23:29 |
| ianw | ahh. yeah v2 was a branch point where we could backport image building changes but keep python2 compat, if we needed such a thing | 23:30 |
| ianw | i'd agree that ship has sailed | 23:30 |
| ianw | now i think about it, this job is going to clone devstack master anyway @ https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-devstack/tasks/main.yaml#L9 | 23:37 |
| clarkb | ianw: ya that was one of the things i checked when lookingat the nova 500 errors. Has me wondering if we tried current stable if we'd get better results (granted current stable just released so unlikely) | 23:39 |
| ianw | so we'd be testing master nodepool with stable/* openstacksdk against a master devstack | 23:39 |
| ianw | thinking that through; i don't think there's any guarantee that master nodepool stays compatible with stable/* openstacksdk. it could easily use new features | 23:42 |
| clarkb | ianw: we have historically been forced to not do that though because production clouds tend to lag significantly | 23:43 |
| clarkb | oh you mean for openstacksdk specifically? ya I suppose that is possible | 23:43 |
| corvus | master sdk should work with old clouds | 23:43 |
| clarkb | corvus: yup I misread | 23:43 |
| ianw | i don't think we'd notice in the siblings image build, because we a) install nodepool b) loop around our listed siblings and install them. we're not checking in b) that the version of openstacksdk we're installing from the zuul chekcout satisfies nodepool requirements | 23:43 |
| ianw | yeah i'm trying to think through what this testing is providing on the openstacksdk side | 23:45 |
| ianw | https://opendev.org/zuul/nodepool/src/branch/master/playbooks/nodepool-functional-openstack/pre.yaml#L18 | 23:51 |
| ianw | non containers does the same thing as well. installs nodepool in a venv, then loops through the required projects to install them one by one | 23:52 |
| ianw | so if nodepool sets openstacksdk>=X , then the openstacksdk uses this test and installs stable/foo that's a breakage | 23:53 |
| clarkb | is the idea to use nodepools less trivial set of tasks as a reasonable sanity check for sdk? | 23:54 |
| clarkb | I think the sdk does have a test suite that tries to do a number of actions against a real cloud | 23:55 |
| ianw | yeah, i should say i'm thinking through openstacksdk's usage on stable branches, not master | 23:59 |
| *** mlavalle has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!