| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : remove deprecated copy conditions https://review.opendev.org/c/openstack/project-config/+/867931 | 03:46 |
|---|---|---|
| *** akahat is now known as akahat|ruck | 05:21 | |
| *** akekane is now known as abhishekk | 05:36 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : remove deprecated copy conditions https://review.opendev.org/c/openstack/project-config/+/867931 | 05:37 |
| ykarel | johnsom, octavia haven't switched yet to jammy https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/861369 | 06:33 |
| ykarel | and the results doesn't look good in that WIP patch | 06:33 |
| *** blarnath is now known as d34dh0r53 | 06:52 | |
| johnsom | ykarel: yet more reason not to use those nodes | 07:01 |
| ykarel | johnsom, yes | 07:05 |
| *** ralonsoh__ is now known as ralonsoh | 07:35 | |
| *** jpena|off is now known as jpena | 08:40 | |
| *** akahat|ruck is now known as akahat|ruck|lunch | 09:08 | |
| *** anbanerj is now known as frenzy_friday | 09:13 | |
| *** bhagyashris is now known as bhagyashris|sick | 09:26 | |
| *** akahat|ruck|lunch is now known as akahat|ruck | 10:17 | |
| *** kopecmartin is now known as kopecmartin|sick | 10:45 | |
| *** rlandy|out is now known as rlandy | 11:11 | |
| *** dviroel|rover|afk is now known as dviroel|rover | 11:16 | |
| *** dviroel|rover is now known as dviroel | 12:20 | |
| *** jpena is now known as jpena|off | 12:44 | |
| *** dasm|off is now known as dasm | 13:36 | |
| *** jpena|off is now known as jpena | 14:06 | |
| *** akekane is now known as abhishekk | 14:11 | |
| *** njohnston_ is now known as njohnston | 14:18 | |
| *** chkumar|ruck is now known as chandankumar | 14:31 | |
| *** dviroel is now known as dviroel|lunch | 15:05 | |
| *** jpena is now known as jpena|off | 15:41 | |
| *** dviroel|lunch is now known as dviroel | 15:58 | |
| ykarel | fungi, hi please drop autoholds for neutron-tempest-plugin-ovn-1 and neutron-tempest-plugin-ovn-2 https://zuul.openstack.org/autoholds | 16:01 |
| ykarel | can rehold if mnaser requires those | 16:01 |
| mnaser | ykarel: what’s the issue? | 16:04 |
| ykarel | mnaser, hi since we switched to jammy nodes recently we hit https://bugs.launchpad.net/neutron/+bug/1999249 | 16:05 |
| ykarel | approx 40% times it fails on nested virt nodes from vexxhost-ca-ymq-1 | 16:06 |
| mnaser | ah, fun | 16:06 |
| mnaser | hm | 16:06 |
| mnaser | i use nested virt quite often, but not on jammy in that region with no issues | 16:07 |
| mnaser | i guess this happened with jammy | 16:07 |
| ykarel | mnaser, yes focal was working great, it started with jammy | 16:08 |
| mnaser | i love nested virt | 16:09 |
| mnaser | :) | 16:09 |
| ykarel | mnaser, if you want to check the affected vm it's 199.19.213.61 and 199.19.213.167, currently on hold | 16:11 |
| mnaser | ykarel: we'll put in something on the list now to verify and see whats up | 16:11 |
| mnaser | i know we've fought something ike this in the past | 16:11 |
| ykarel | mnaser, Thanks much | 16:12 |
| mnaser | do we log hostId's by any chance | 16:12 |
| ykarel | i think that can be get from nodepool side | 16:13 |
| ykarel | for those too vms server id | 16:13 |
| ykarel | it looks like the uuid for 199.19.213.61 is 1982206a-6620-454f-9983-465db3651cf8, while the uuid for 199.19.213.167 is 5200d42a-e41f-4fc4-a5f8-3d8a48a71349 | 16:13 |
| mnaser | ok maybe we can use that | 16:13 |
| mnaser | i know we've got different os's across our fleet | 16:13 |
| mnaser | so many it depends on what you're hitting | 16:13 |
| ykarel | mnaser, ok hostId is there in logs | 16:14 |
| ykarel | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d98/867609/1/check/neutron-tempest-plugin-ovn-1/d9848b1/zuul-info/inventory.yaml | 16:14 |
| fungi | yeah, we log host ids for this exact reason ;) | 16:18 |
| clarkb | once we even used the host id to identify a bad cpu in a provider | 16:58 |
| opendevreview | Merged openstack/project-config master: Remove iweb nodepool configs https://review.opendev.org/c/openstack/project-config/+/867264 | 17:27 |
| ykarel | mnaser, https://bugs.launchpad.net/neutron/+bug/1999249/comments/3 filtered some good/bad host_ids | 17:31 |
| * ykarel away | 17:33 | |
| *** rlandy is now known as rlandy|brb | 18:13 | |
| *** rlandy|brb is now known as rlandy | 18:41 | |
| guilhermesp_____ | hey guys, in regards to the nested virt... i did some cross-checking here based on https://bugs.launchpad.net/neutron/+bug/1999249/comments/3 to map the hypervisors which you reported as failures | 19:43 |
| fungi | any clues? | 19:44 |
| guilhermesp_____ | i have checked that all the failing ones are going to hypervisors running bionic, which we know it doenst work with nested virt | 19:44 |
| guilhermesp_____ | and we were converting the computes to focal, so all the success hostsid as on focal hypervisors | 19:44 |
| guilhermesp_____ | so the idea here is keep moving forward with the hypervisor upgrades from bionic to focal | 19:44 |
| guilhermesp_____ | however thats not so fast to be done, it involve us emptying the hv's and re-imaging them to focal | 19:45 |
| guilhermesp_____ | i can prioritize that though, let me chat with mnaser internally for us to coordinate and move forward with that as fast as we can | 19:47 |
| clarkb | on our side we warn people that these problems may occur. I think it is ok if you don't prioritize it for this reason | 19:56 |
| clarkb | but I won't object either :) | 19:56 |
| guilhermesp_____ | i mean, yeah i need the make the most of the opportunity to keep going :P | 20:06 |
| guilhermesp_____ | but i believe as i go the failing ones will become less frequent | 20:07 |
| fungi | that's great info, thanks guilhermesp_____! | 20:21 |
| fungi | johnsom: ^ probably of interest to you | 20:22 |
| fungi | ykarel vanished a few hours ago but will certainly return at some point | 20:22 |
| *** dviroel is now known as dviroel|out | 20:30 | |
| *** dasm is now known as dasm|off | 21:54 | |
| *** osmanlicilegi is now known as Guest0 | 22:33 | |
| *** ChanServ changes topic to "Discussion of OpenStack's Developer tooling and CI jobs | Discussion around infrastructure services now in #opendev" | 22:48 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!