| ccstone | Thanks mriedem, we were able to correct the transport_url in the DB using that and that's what finally made the error stop. | 00:00 |
|---|---|---|
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova.compute.rpcapi.ComputeAPI.router a singleton https://review.opendev.org/649197 | 00:01 |
| mriedem | ccstone: "using that" => nova-manage cell_v2 update_cell? | 00:01 |
| *** cooper6581 has joined #openstack-nova | 00:05 | |
| *** dave-mccowan has quit IRC | 00:05 | |
| *** lbragstad has quit IRC | 00:05 | |
| cooper6581 | mriedem: (with ccstone and eandersson being trained). You are right that it should have just returned the URL as-is, but I'm assuming CONF.transport_url ended up not being None | 00:08 |
| cooper6581 | https://ghosthub.corp.blizzard.net/openstack/nova/blob/import/rocky/nova/objects/cell_mapping.py#L145 | 00:08 |
| cooper6581 | oops, wrong link | 00:08 |
| ccstone | mriedem: Yeah - we had updated nova.conf with a new rabbit password and were still getting the "unmatched '{'" error using nova-manage. Updating the transport_url in the DB via update_cell and the link you provided was what ultimately fixed it for us. Seems like simple_cell_setup allowed us to provide a bad transport_url without URL encoding. | 00:08 |
| mriedem | ok | 00:09 |
| cooper6581 | https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145 | 00:09 |
| mriedem | i guess if there is anything bug worthy to follow up with this to help someone else from falling in the same trap that would be good to report, maybe it could just be docs | 00:11 |
| *** slaweq has joined #openstack-nova | 00:11 | |
| *** gibi has quit IRC | 00:16 | |
| *** gibi has joined #openstack-nova | 00:18 | |
| mriedem | ha this actually ran tests https://review.opendev.org/#/c/655222/ | 00:18 |
| mriedem | was sure that would explode in devstack setup | 00:18 |
| mriedem | dansmith: ^ | 00:19 |
| mriedem | looks like the only thing that shit the bed was cold migration testing since tempest doesn't have a toggle for that | 00:19 |
| *** dikonoor has joined #openstack-nova | 00:21 | |
| *** slaweq has quit IRC | 00:24 | |
| *** gyee has quit IRC | 00:24 | |
| *** lbragstad has joined #openstack-nova | 00:24 | |
| mriedem | i see why it's failing there, i'm saying resize to same host is ok b/c i only have 1 compute per cell, but you can't cold migrate to the same host with the libvirt driver, that will be fixed with cross-cell resize ftw | 00:32 |
| *** itlinux has joined #openstack-nova | 00:34 | |
| *** yedongcan has joined #openstack-nova | 00:38 | |
| *** igordc has quit IRC | 00:40 | |
| openstackgerrit | Merged openstack/os-vif master: Refactor functional base test classes https://review.opendev.org/643101 | 00:46 |
| *** itlinux has quit IRC | 00:58 | |
| *** brinzhang has joined #openstack-nova | 00:58 | |
| *** gibi has quit IRC | 00:58 | |
| *** whoami-rajat has joined #openstack-nova | 01:02 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add nova-multi-cell job https://review.opendev.org/655222 | 01:03 |
| *** ricolin has joined #openstack-nova | 01:05 | |
| mriedem | efried: just fyi i've removed the cross-cell-resize series from the runway slot (time expired) and added https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | 01:09 |
| mriedem | boxiang: ^ | 01:09 |
| mriedem | https://etherpad.openstack.org/p/nova-runways-train | 01:09 |
| *** mriedem has quit IRC | 01:11 | |
| *** slaweq has joined #openstack-nova | 01:13 | |
| *** gibi has joined #openstack-nova | 01:16 | |
| *** alex_xu has quit IRC | 01:17 | |
| *** slaweq has quit IRC | 01:24 | |
| *** tetsuro has joined #openstack-nova | 01:25 | |
| *** markvoelker has quit IRC | 01:34 | |
| *** cooper6581 has quit IRC | 01:45 | |
| *** tetsuro has quit IRC | 01:51 | |
| melwitt | lyarwood, mriedem: ceph dashboard fix attempt https://review.opendev.org/655591 | 01:52 |
| *** tetsuro has joined #openstack-nova | 02:06 | |
| *** slaweq has joined #openstack-nova | 02:11 | |
| *** tetsuro has quit IRC | 02:18 | |
| *** tetsuro has joined #openstack-nova | 02:23 | |
| *** slaweq has quit IRC | 02:24 | |
| *** dikonoor has quit IRC | 02:25 | |
| *** alex_xu has joined #openstack-nova | 02:29 | |
| *** itlinux has joined #openstack-nova | 02:32 | |
| *** lbragstad has quit IRC | 02:36 | |
| *** JamesBenson has joined #openstack-nova | 02:41 | |
| *** alex_xu has quit IRC | 02:45 | |
| *** ricolin has quit IRC | 02:46 | |
| *** hongbin has joined #openstack-nova | 02:51 | |
| *** smcginnis has quit IRC | 03:06 | |
| openstackgerrit | Merged openstack/nova stable/stein: Fix {min|max}_version in ironic Adapter setup https://review.opendev.org/655429 | 03:08 |
| openstackgerrit | Merged openstack/nova stable/rocky: Fix regression in glance client call https://review.opendev.org/655167 | 03:08 |
| *** ivve has quit IRC | 03:10 | |
| *** dikonoor has joined #openstack-nova | 03:11 | |
| *** hongbin has quit IRC | 03:12 | |
| *** slaweq has joined #openstack-nova | 03:16 | |
| *** slaweq has quit IRC | 03:24 | |
| *** brinzhang has quit IRC | 03:25 | |
| *** brinzhang has joined #openstack-nova | 03:26 | |
| *** markvoelker has joined #openstack-nova | 03:35 | |
| *** psachin has joined #openstack-nova | 03:52 | |
| *** udesale has joined #openstack-nova | 03:55 | |
| *** ricolin has joined #openstack-nova | 04:10 | |
| *** pcaruana has joined #openstack-nova | 04:11 | |
| *** slaweq has joined #openstack-nova | 04:13 | |
| *** JamesBenson has quit IRC | 04:18 | |
| *** ileixe has quit IRC | 04:22 | |
| *** slaweq has quit IRC | 04:24 | |
| *** ivve has joined #openstack-nova | 04:25 | |
| *** ileixe has joined #openstack-nova | 04:25 | |
| *** ileixe has quit IRC | 04:26 | |
| *** ileixe has joined #openstack-nova | 04:27 | |
| *** pcaruana has quit IRC | 04:43 | |
| *** tetsuro has quit IRC | 04:51 | |
| *** ivve has quit IRC | 05:06 | |
| *** kukacz has quit IRC | 05:06 | |
| *** kukacz has joined #openstack-nova | 05:08 | |
| *** slaweq has joined #openstack-nova | 05:11 | |
| *** ivve has joined #openstack-nova | 05:18 | |
| *** slaweq has quit IRC | 05:21 | |
| *** Luzi has joined #openstack-nova | 05:27 | |
| *** slaweq has joined #openstack-nova | 05:27 | |
| *** itlinux has quit IRC | 05:30 | |
| *** slaweq has quit IRC | 05:41 | |
| *** jiaopengju_2 has joined #openstack-nova | 05:49 | |
| *** jiaopengju_1 has quit IRC | 05:52 | |
| *** slaweq has joined #openstack-nova | 05:56 | |
| *** slaweq has quit IRC | 06:02 | |
| *** igordc has joined #openstack-nova | 06:08 | |
| *** phasespace has quit IRC | 06:09 | |
| *** shilpasd has joined #openstack-nova | 06:10 | |
| *** slaweq has joined #openstack-nova | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** JamesBenson has joined #openstack-nova | 06:18 | |
| *** pcaruana has joined #openstack-nova | 06:20 | |
| *** JamesBenson has quit IRC | 06:22 | |
| *** jiaopengju_1 has joined #openstack-nova | 06:27 | |
| *** jiaopengju_2 has quit IRC | 06:30 | |
| *** Luzi has quit IRC | 06:33 | |
| *** Luzi has joined #openstack-nova | 06:34 | |
| *** slaweq has joined #openstack-nova | 06:44 | |
| *** luksky has joined #openstack-nova | 06:46 | |
| *** phasespace has joined #openstack-nova | 06:48 | |
| *** jiaopengju_1 has quit IRC | 06:49 | |
| *** jiaopengju_1 has joined #openstack-nova | 06:49 | |
| *** phasespace has quit IRC | 06:52 | |
| *** igordc has quit IRC | 06:53 | |
| *** ricolin has quit IRC | 06:53 | |
| *** evrardjp has quit IRC | 06:55 | |
| *** evrardjp has joined #openstack-nova | 06:55 | |
| *** evrardjp has quit IRC | 06:58 | |
| *** evrardjp has joined #openstack-nova | 06:58 | |
| *** ccamacho has joined #openstack-nova | 07:08 | |
| *** ccamacho has quit IRC | 07:09 | |
| *** ccamacho has joined #openstack-nova | 07:10 | |
| *** dpawlik has quit IRC | 07:14 | |
| *** zbr has joined #openstack-nova | 07:25 | |
| *** ricolin has joined #openstack-nova | 07:28 | |
| *** slaweq has quit IRC | 07:31 | |
| *** slaweq has joined #openstack-nova | 07:32 | |
| *** slaweq has quit IRC | 07:34 | |
| *** slaweq has joined #openstack-nova | 07:34 | |
| *** slaweq has quit IRC | 07:35 | |
| *** slaweq has joined #openstack-nova | 07:36 | |
| *** slaweq has quit IRC | 07:36 | |
| *** dpawlik has joined #openstack-nova | 07:36 | |
| *** slaweq has joined #openstack-nova | 07:37 | |
| *** slaweq has quit IRC | 07:38 | |
| *** slaweq has joined #openstack-nova | 07:38 | |
| *** ttsiouts has joined #openstack-nova | 07:42 | |
| kashyap | efried: Hehe, I learnt a new expression, "one-armed-wallpaperer" (or "one-armed paper hanger", I see). | 07:44 |
| *** rpittau|afk is now known as rpittau | 07:58 | |
| *** phasespace has joined #openstack-nova | 08:02 | |
| *** tetsuro has joined #openstack-nova | 08:05 | |
| *** ralonsoh has joined #openstack-nova | 08:05 | |
| *** tssurya has joined #openstack-nova | 08:05 | |
| *** ttsiouts_ has joined #openstack-nova | 08:05 | |
| *** tetsuro has quit IRC | 08:06 | |
| *** ttsiouts has quit IRC | 08:07 | |
| *** derekh has joined #openstack-nova | 08:14 | |
| *** zbr is now known as zbr|rover | 08:18 | |
| *** ttsiouts has joined #openstack-nova | 08:19 | |
| *** ttsiouts_ has quit IRC | 08:20 | |
| kashyap | efried: sean-k-mooney: Just looked at at the video models thing: yes, it makes sense to add the me, FWIW. However, I'd appreciate a one-line summary of each of the three models being added in the BP | 08:23 |
| kashyap | efried: sean-k-mooney E.g. "The 'none' video model type disables the automatic addition of a video device to domains with 'graphics' specified in their XML. This can be useful with GPU mediated devices which can serve as the only rendering devices within the guest." | 08:23 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655332 | 08:23 |
| *** zigo has quit IRC | 08:27 | |
| *** gmann has quit IRC | 08:28 | |
| *** ttsiouts_ has joined #openstack-nova | 08:32 | |
| *** ttsiouts has quit IRC | 08:35 | |
| *** ttsiouts has joined #openstack-nova | 08:49 | |
| *** ttsiouts_ has quit IRC | 08:50 | |
| *** tkajinam has quit IRC | 08:54 | |
| *** rha has joined #openstack-nova | 08:57 | |
| *** rcernin has quit IRC | 08:58 | |
| gibi | efried, mriedem: regarding the length of the PTG slot :1330-1415: Let's plan the next steps of the bandwidth support feature (gibi) | 09:00 |
| gibi | efried, mriedem: I think we can speed things up, so we can be done in ~15-20 mins | 09:01 |
| *** tobias-urdin has quit IRC | 09:01 | |
| *** tobias-urdin has joined #openstack-nova | 09:17 | |
| *** dims has quit IRC | 09:20 | |
| *** luksky has quit IRC | 09:21 | |
| aspiers | check jobs seem pretty flakey at the moment | 09:25 |
| aspiers | it's not normal to need 2 rechecks, right? | 09:25 |
| *** dims has joined #openstack-nova | 09:26 | |
| *** tobias-urdin has quit IRC | 09:27 | |
| aspiers | kashyap: any idea? https://review.opendev.org/#/c/655268/2 keeps getting V-1 for no good reason AFAICS | 09:30 |
| kashyap | aspiers: Sometimes is completely "normal" to require more than 2 rechecks :-( | 09:31 |
| aspiers | :-( | 09:31 |
| aspiers | This is an awesome technique for dealing with troublesome CI https://cfp.all-systems-go.io/ASG2018/talk/207/ | 09:31 |
| aspiers | I think until OpenStack has something like that, flakey gates will always remain common | 09:31 |
| kashyap | I don't have any bright ideas (besides "recheck and pray to the JuJu at the bottom of the sea"). I've experienced the same in the past. | 09:31 |
| aspiers | Using ML means the "flakes" can be clustered automatically (unsupervised) which then means it's crystal clear which flakes cause the most failures and require the most human attention to fix | 09:32 |
| aspiers | That way we could get to the long tail of weird corner cases quicker | 09:33 |
| aspiers | e.g. one flake might cause 30% of failures, and another 0.3% | 09:33 |
| *** dims has quit IRC | 09:33 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| aspiers | but until you gather stats on that, it's hard to know where to put effort | 09:33 |
| *** mdbooth has quit IRC | 09:34 | |
| *** dims has joined #openstack-nova | 09:34 | |
| aspiers | elastic recheck aims for that of course, but it requires manual maintenance | 09:34 |
| kashyap | aspiers: Yeah, suggest it on the 'openstack-discuss' list, if you have a minute to write the email | 09:35 |
| *** tobias-urdin has joined #openstack-nova | 09:40 | |
| *** zbr|rover has quit IRC | 09:42 | |
| *** mdbooth has joined #openstack-nova | 09:43 | |
| *** zbr has joined #openstack-nova | 09:44 | |
| *** jbernard has quit IRC | 09:48 | |
| *** dklyle has quit IRC | 09:50 | |
| *** dklyle has joined #openstack-nova | 09:50 | |
| *** luksky has joined #openstack-nova | 09:59 | |
| aspiers | kashyap: might be better to find mtreinish and the infra folks in Denver next week first | 10:03 |
| aspiers | I've already reached out to the cockpit project | 10:03 |
| aspiers | no reply yet | 10:03 |
| *** jbernard has joined #openstack-nova | 10:04 | |
| aspiers | kashyap: when you get a moment, reviews on https://review.opendev.org/#/c/655268/ and https://review.opendev.org/#/c/633855/ would be much appreciated ;-) | 10:04 |
| kashyap | aspiers: Will do. Currently doing something I promised efried that I'll address (in my own spec, though) today. Trying to keep the word. :D | 10:05 |
| aspiers | sure, good luck with it :) | 10:05 |
| *** ccamacho has quit IRC | 10:11 | |
| *** mvkr has joined #openstack-nova | 10:13 | |
| *** ttsiouts_ has joined #openstack-nova | 10:24 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.opendev.org/555081 | 10:26 |
| *** ttsiouts has quit IRC | 10:29 | |
| *** ttsiouts_ has quit IRC | 10:29 | |
| *** ttsiouts has joined #openstack-nova | 10:31 | |
| openstackgerrit | Adam Spiers proposed openstack/os-resource-classes master: Add MEM_ENCRYPTION_CONTEXT resource class https://review.opendev.org/655665 | 10:33 |
| *** lajoskatona has joined #openstack-nova | 10:37 | |
| *** tbachman has quit IRC | 10:43 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.opendev.org/555081 | 10:46 |
| stephenfin | sean-k-mooney: You're mostly happy with ^ at this point, I assume? | 10:46 |
| sean-k-mooney | ya mostly | 10:47 |
| stephenfin | I've added more words and fixed the typos to assuage efried's concerns (I hope) | 10:47 |
| sean-k-mooney | i was going to suggest using a different threath the HW_CPU_HYPERTHREADING | 10:47 |
| sean-k-mooney | but its already in os traits | 10:47 |
| sean-k-mooney | e.g. HW_CPU_SMT | 10:47 |
| stephenfin | sean-k-mooney: We could just _not_ use that? | 10:48 |
| sean-k-mooney | then we have another dead trait in plamcent | 10:48 |
| stephenfin | Or even remove it if no-one else is using it yet | 10:48 |
| sean-k-mooney | *os-traits | 10:48 |
| sean-k-mooney | stephenfin: we cant remove traits from os-traits | 10:48 |
| sean-k-mooney | its an add only model | 10:49 |
| stephenfin | if no one is using it anywhere though? | 10:49 |
| sean-k-mooney | to simplfy db updates | 10:49 |
| stephenfin | and never has | 10:49 |
| stephenfin | Oh, there's a DB involved | 10:49 |
| sean-k-mooney | even then | 10:49 |
| sean-k-mooney | ya | 10:49 |
| aspiers | stephenfin: you're repeating an earlier discussion ;-) | 10:49 |
| aspiers | this is now mentioned in http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html | 10:49 |
| aspiers | I asked the exact same question a few days ago | 10:49 |
| sean-k-mooney | yep | 10:50 |
| stephenfin | Fair. I'd also be ok with a dead trait, tbh, but I also don't care that much | 10:50 |
| sean-k-mooney | im fine with hte hyperthreaing one | 10:50 |
| aspiers | I think dead traits can be hidden to the point that they're only visible in the os-traits code where hardly anyone looks | 10:50 |
| sean-k-mooney | people use it as a generic term even if it is a threadmark | 10:50 |
| aspiers | well I suppose they're visible via introspection too | 10:50 |
| stephenfin | fwiw, I tried to use SMT in the documentation e.g. https://docs.openstack.org/nova/pike/admin/cpu-topologies.html | 10:50 |
| sean-k-mooney | yes i know and you gave a talk on the topic | 10:51 |
| stephenfin | indeed :) | 10:51 |
| sean-k-mooney | so ya i was in the middel of reading v27 | 10:51 |
| *** dpawlik has quit IRC | 10:52 | |
| sean-k-mooney | ill skim through v28 | 10:52 |
| stephenfin | Yeah, spotted an issue with my description of that trait, actually :) | 10:52 |
| stephenfin | I must admit, I really don't like Gerrit for doc reviews :-\ | 10:52 |
| sean-k-mooney | why | 10:53 |
| stephenfin | It's too hard to keep track of conversations | 10:53 |
| sean-k-mooney | you jut have to wait a bit and collect the changes | 10:53 |
| aspiers | stephenfin: that's because we're using an ancient Gerrit | 10:53 |
| aspiers | Gerrit 2.16 is awesome | 10:53 |
| stephenfin | aspiers: Oh yeah? Anything specific? | 10:53 |
| sean-k-mooney | we are using 2.13 | 10:53 |
| stephenfin | The polymer UI? | 10:53 |
| aspiers | stephenfin: much more than that | 10:53 |
| aspiers | https://review.gerrithub.io/q/status:open | 10:54 |
| sean-k-mooney | whats so different about that | 10:55 |
| aspiers | https://youtu.be/WsPhoPGUsss?t=9m47s | 10:55 |
| aspiers | https://youtu.be/WsPhoPGUsss?t=19m10s | 10:55 |
| sean-k-mooney | that is just the polomer ui + a theme | 10:55 |
| aspiers | nope | 10:55 |
| stephenfin | aspiers: Oh, nice | 10:56 |
| aspiers | this is just one of many specific improvements | 10:56 |
| aspiers | I can't list them all for you | 10:56 |
| stephenfin | Resolvable comments are indeed one of the things I'd like | 10:56 |
| stephenfin | I like that in gdocs | 10:56 |
| aspiers | also bear in mind those videos are 1.5 years old now | 10:58 |
| stephenfin | yeah, so things will have moved on a lot since then | 10:58 |
| aspiers | OpenStack really REALLY needs a Gerrit upgrade | 10:58 |
| stephenfin | I imagine the infra folks are snowed under with the opendev migration but this sure would be a nice next step | 10:58 |
| sean-k-mooney | aspiers: we used to have some downstream only changes | 10:58 |
| sean-k-mooney | because we need to fix bug that had not merged upstream | 10:59 |
| aspiers | stephenfin: yeah. it will happen | 10:59 |
| stephenfin | big changes in 3.0 too, from the looks of things https://www.gerritcodereview.com/3.0.html | 10:59 |
| sean-k-mooney | im not sure what is left to get upstream at this point | 10:59 |
| *** ttsiouts has quit IRC | 10:59 | |
| aspiers | sean-k-mooney: if a patch to 2.13 hasn't landed by now, it never will | 10:59 |
| sean-k-mooney | aspiers: yes which may mean we never update | 10:59 |
| aspiers | sean-k-mooney: highly doubt that | 10:59 |
| sean-k-mooney | there were one or two that we need to work at our scalse in the past the got push back which stop openstack updating | 11:00 |
| aspiers | clarkb / corvus / fungi started working on it a long time ago | 11:00 |
| sean-k-mooney | but i think most of that has gone upstream now | 11:00 |
| *** ttsiouts has joined #openstack-nova | 11:00 | |
| aspiers | sean-k-mooney: yeah, we're not the biggest Gerrit user | 11:00 |
| aspiers | there are several other huge scale installs | 11:01 |
| aspiers | I met them all at the conference | 11:01 |
| sean-k-mooney | i think we were 6 years ago when i started on openstack | 11:01 |
| aspiers | and the others are all head in terms of HA too | 11:01 |
| aspiers | ahead | 11:01 |
| aspiers | Google is heavily investing in Gerrit | 11:01 |
| aspiers | they have a team of 20 or so IIRC | 11:01 |
| aspiers | stephenfin: 3.0 is more about ditching the dead stuff which everyone will have migrated off of by 2.16 | 11:02 |
| stephenfin | looks like it, yeah | 11:02 |
| stephenfin | I'd imagine we'll be getting 2.16 first, in that case | 11:02 |
| aspiers | for sure | 11:03 |
| *** ttsiouts_ has joined #openstack-nova | 11:05 | |
| *** ttsiouts has quit IRC | 11:06 | |
| openstackgerrit | Adam Spiers proposed openstack/os-traits master: Update SEV trait docs to avoid misleading people https://review.opendev.org/655671 | 11:12 |
| openstackgerrit | Adam Spiers proposed openstack/os-traits master: Document policy of never removing traits https://review.opendev.org/655673 | 11:19 |
| openstackgerrit | Adam Spiers proposed openstack/os-traits master: Document policy of never removing traits https://review.opendev.org/655673 | 11:23 |
| *** dpawlik has joined #openstack-nova | 11:25 | |
| *** zbr has quit IRC | 11:31 | |
| *** udesale has quit IRC | 11:33 | |
| *** zbr has joined #openstack-nova | 11:34 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/stein: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655678 | 11:36 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec https://review.opendev.org/645814 | 11:36 |
| kashyap | efried: As promised ^^ | 11:36 |
| *** boxiang has quit IRC | 11:37 | |
| *** boxiang has joined #openstack-nova | 11:37 | |
| jaypipes | efried: morning. sorry, just saw your question from yesterday evening. | 11:38 |
| jaypipes | efried: I'll be leaving around 1:30-2pm on the Friday. I'll just make do with side conversations etc at the summit about the various specs I'm concerned about (and continue to add reviews to said specs) | 11:39 |
| sean-k-mooney | jaypipes: i think efried has tried to front load most of the topic that are in your usuall area. not sure they all fit | 11:41 |
| sean-k-mooney | one thing i think will come up in the feeback session is the ptg and summit should over lap again | 11:41 |
| sean-k-mooney | we will see but the room was pretty split at the last ptg on that topic | 11:42 |
| jaypipes | ack | 11:43 |
| *** jiaopengju_1 has quit IRC | 11:44 | |
| *** jiaopengju_1 has joined #openstack-nova | 11:44 | |
| *** dpawlik has quit IRC | 11:52 | |
| *** zigo has joined #openstack-nova | 11:53 | |
| *** panda is now known as panda|lunch | 11:59 | |
| openstackgerrit | Merged openstack/nova stable/queens: Fix regression in glance client call https://review.opendev.org/655186 | 12:05 |
| *** tbachman has joined #openstack-nova | 12:07 | |
| openstackgerrit | Merged openstack/nova stable/queens: libvirt: disconnect volume when encryption fails https://review.opendev.org/653033 | 12:10 |
| *** markvoelker has quit IRC | 12:14 | |
| *** markvoelker has joined #openstack-nova | 12:15 | |
| *** dpawlik has joined #openstack-nova | 12:15 | |
| fungi | aspiers: stephenfin: sean-k-mooney: yes, current plan i think is going straight from 2.13 to 2.16 we just need to re-test the upgrade process on our staging server. the transition from mysql to notedb is going to be the biggest challenge as it will likely require a day-long service outage to complete | 12:23 |
| *** derekh has quit IRC | 12:23 | |
| fungi | we didn't want to conflate that work with the opendev switch for gerrit/git, but this ptg will be the time to make plans | 12:24 |
| sean-k-mooney | is there an infra meetup at the ptg or just people will be about so it will come up | 12:24 |
| fungi | infra is meeting thursday and friday all day (we share a room with qa) | 12:25 |
| *** tosky has joined #openstack-nova | 12:25 | |
| sean-k-mooney | ah ok | 12:25 |
| sean-k-mooney | i kind of want to get a bit more involved with the infra side of things but on the other hand was not sure how to go about that so i was just going to say hi | 12:26 |
| *** vabada has joined #openstack-nova | 12:28 | |
| *** altlogbot_1 has quit IRC | 12:32 | |
| *** ttsiouts has joined #openstack-nova | 12:37 | |
| *** nicolasbock has joined #openstack-nova | 12:38 | |
| *** ttsiouts_ has quit IRC | 12:38 | |
| *** altlogbot_0 has joined #openstack-nova | 12:39 | |
| *** ttsiouts_ has joined #openstack-nova | 12:40 | |
| *** ttsiouts has quit IRC | 12:42 | |
| *** ttsiouts has joined #openstack-nova | 12:43 | |
| *** ttsiouts_ has quit IRC | 12:43 | |
| *** derekh has joined #openstack-nova | 12:44 | |
| *** altlogbot_0 has quit IRC | 12:47 | |
| *** altlogbot_0 has joined #openstack-nova | 12:49 | |
| kashyap | fungi: What is "notedb"? /me looks up | 12:54 |
| kashyap | Okay, it is this: "NoteDb is the next generation of Gerrit storage backend, which replaces the traditional SQL backend ..." | 12:54 |
| kashyap | https://gerrit-review.googlesource.com/Documentation/note-db.html | 12:54 |
| fungi | kashyap: yep, basically we'll be etl'ing gigabytes of data from an rdbms into git repositories | 12:55 |
| fungi | not a speedy process | 12:55 |
| *** udesale has joined #openstack-nova | 12:56 | |
| fungi | at one point the upgrade process was necessary to be performed stepwise first moving some of the data into an h2 database and then moving it from there into git in a subsequent gerrit version, but we hear they've added the ability to migrate directly from sql to notedb now so it may go a lot more quickly, we just need to test | 12:57 |
| *** eharney has quit IRC | 13:00 | |
| stephenfin | fungi: Thanks for the context. That's all very good to hear :) | 13:05 |
| *** mchlumsky has joined #openstack-nova | 13:05 | |
| *** dikonoor has quit IRC | 13:06 | |
| kashyap | fungi: Yeah, noted | 13:07 |
| fungi | also part of the upgrade process involved moving to a new server on a newer ubuntu release so we would have a new enough version of java to work with newer gerrit. we got that done last year, which was a nontrivial undertaking | 13:07 |
| sean-k-mooney | fungi: out of interst shicne the data will now be sotred in git | 13:08 |
| sean-k-mooney | could we use the notedb data in tools like gertty | 13:08 |
| sean-k-mooney | or will that content not be downloadable by others | 13:08 |
| fungi | well, gertty already has access to the relevant data (modulo access controls allowing your account to access it) through gerrit's rest api | 13:09 |
| sean-k-mooney | gerrty effectivly buildis its own sqlite db replicting the gerrit one if that was not required it could be a nice improvemnt | 13:09 |
| fungi | oh, i see what you're asking... i think the data in notedb is aggregated in ways which wouldn't make that useful for per-user data tracking | 13:10 |
| fungi | what it does enable though is distribution of state, an there's a plugin recently contributed upstream to gerrit by the gerrithub folks which does multi-master replication between notedbs | 13:11 |
| *** smcginnis has joined #openstack-nova | 13:11 | |
| fungi | so we could finally have a decent cross-provider high availability solution for it | 13:11 |
| sean-k-mooney | ya that would be cool | 13:11 |
| fungi | i know the wmf folks have just upgraded to 2.16 and are looking at leveraging that plugin soon | 13:12 |
| sean-k-mooney | wmf? | 13:12 |
| sean-k-mooney | wiki media foundaiton? | 13:12 |
| sean-k-mooney | actully looking at https://gitenterprise.me/tag/notedb/ it looks like it will allow you to work offline | 13:15 |
| stephenfin | ffs https://review.opendev.org/651297 | 13:15 |
| *** tetsuro has joined #openstack-nova | 13:15 | |
| * stephenfin rechecks _again_ | 13:15 | |
| stephenfin | there is definitely something buggy with one of our lower-constraint packages to say I keep seeing these StringException errors http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/testr_results.html.gz | 13:16 |
| stephenfin | But who knows what | 13:16 |
| *** tetsuro has quit IRC | 13:18 | |
| *** tetsuro has joined #openstack-nova | 13:19 | |
| efried | stephenfin: That's bug 1825435 | 13:19 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/1825435 | 13:19 |
| sean-k-mooney | did we find where that was intoduced yet | 13:20 |
| *** lajoskatona has quit IRC | 13:20 | |
| efried | it seems almost exclusively isolated to py36, so it hits lower-constraints (which runs on py36) and the py36 UT. | 13:20 |
| efried | sean-k-mooney: No. mriedem has been trying to debug it (https://review.opendev.org/#/c/654468/ and its dep) but we can't get it to reproduce with those patches <fume> | 13:20 |
| sean-k-mooney | apparently it started to show up first in stephenfin cells v1 removal patches but that could also be a coincidnces | 13:20 |
| efried | sean-k-mooney: If you can update the video models bp description as requested by kashyap, I'll approve it. | 13:21 |
| sean-k-mooney | am sure ill go do that now. | 13:21 |
| efried | jaypipes: I was specifically asking if those two topics were ones you wanted to be around for. If so, I can try to shuffle things around. | 13:21 |
| efried | thanks sean-k-mooney | 13:21 |
| *** ricolin has quit IRC | 13:21 | |
| kashyap | sean-k-mooney: Thank you! | 13:21 |
| *** lbragstad has joined #openstack-nova | 13:22 | |
| efried | aspiers: If you're around and want to fix that one typo in https://review.opendev.org/#/c/655673/ I'll fast approve. | 13:22 |
| *** mriedem has joined #openstack-nova | 13:22 | |
| sean-k-mooney | i have it more or less in the relse note https://review.opendev.org/#/c/647733/3/releasenotes/notes/extend-libvirt-video-model-support-d630b99ef5039f51.yaml | 13:22 |
| efried | If you're not around, I'll fix it. | 13:22 |
| *** tetsuro has quit IRC | 13:22 | |
| efried | sean-k-mooney: copy paste, baby | 13:22 |
| efried | kashyap: You okay with the wording there --^ ? | 13:23 |
| * kashyap looks | 13:23 | |
| sean-k-mooney | i need to change it slightly but to make sense in a blueprint | 13:23 |
| sean-k-mooney | mainly the tense but ill use it as teh basis | 13:23 |
| stephenfin | efried: I don't think it is, actually | 13:23 |
| mriedem | lyarwood: seems like an easy bug to fix, but you should look https://bugs.launchpad.net/nova/+bug/1825882 | 13:23 |
| openstack | Launchpad bug 1825882 in OpenStack Compute (nova) "Virsh disk attach errors silently ignored" [Undecided,New] | 13:23 |
| stephenfin | http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/job-output.txt.gz | 13:23 |
| *** altlogbot_0 has quit IRC | 13:23 | |
| kashyap | efried: I'll add a comment there. But the 'none' model should be described. You can copy/paste the phrase I posted earlier | 13:24 |
| stephenfin | It looks like the test never finishes. If you grep for 'test_get_transport_url' the status is inprogress | 13:24 |
| * kashyap adds a comment in the change | 13:24 | |
| stephenfin | though I could be misreading that | 13:24 |
| efried | mriedem: Is that not the same bug? http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/job-output.txt.gz#_2019-04-25_07_44_46_919295 | 13:25 |
| stephenfin | I also missed that you'd already rechecked. Oops | 13:25 |
| efried | no takesie backsies | 13:26 |
| kashyap | sean-k-mooney: efried I added the phrasing in the change. | 13:26 |
| mriedem | efried: yes http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/job-output.txt.gz#_2019-04-25_07_42_59_709756 | 13:26 |
| efried | stephenfin: ^ | 13:27 |
| stephenfin | jaypipes: If you've time, mind glancing over the cpu-resources spec to make sure I haven't cannibalized it too much? https://review.opendev.org/#/c/555081 | 13:27 |
| *** artom has quit IRC | 13:27 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec https://review.opendev.org/645814 | 13:28 |
| openstackgerrit | Eric Fried proposed openstack/os-traits master: Document policy of never removing traits https://review.opendev.org/655673 | 13:28 |
| efried | aspiers: FTFY and fast approved ^ | 13:29 |
| stephenfin | efried, mriedem: I saw that but I didn't see the link to the failing tests. I'd have expected to have seen the name of the failing test reported before stacktrace | 13:29 |
| kashyap | efried: Meanwhile, in your copious free time ^ As best as I can addressed remarks from mriedem, sean-k-mooney et al. If you want to fast-approve. | 13:29 |
| efried | stephenfin: It varies which test fails | 13:29 |
| kashyap | (Also I will add / update the spec as I learn other important details, docs while implenenting.) | 13:30 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662 | 13:30 |
| efried | jeebuz, with that big a delta, I doubt I'll be able to fast approve | 13:30 |
| efried | but thanks for updating. | 13:30 |
| kashyap | efried: Nothing fundamental has changed. I largely added clarifying text | 13:30 |
| kashyap | efried: Or removed unnecessary bit. The main work mentioned in the Work Items is still in tact | 13:31 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Server move operations with ports having resource request https://review.opendev.org/652608 | 13:31 |
| lyarwood | mriedem: ack I'll take a look now | 13:32 |
| mriedem | efried: want to continue the backport train? https://review.opendev.org/#/c/655429/ | 13:33 |
| jaypipes | stephenfin: ack. will do that right now. | 13:33 |
| *** altlogbot_0 has joined #openstack-nova | 13:33 | |
| efried | kashyap: Okay, you've said nova will validate, but I'm still not seeing the part where you say: "...and will cause {spawn to fail | nova-compute to refuse to start}, with a helpful log message indicating which combination of thingies was bad" | 13:33 |
| jaypipes | efried: no, not a huge prio for me. | 13:33 |
| stephenfin | ta | 13:33 |
| efried | jaypipes: ack, thx | 13:33 |
| efried | mriedem: Yes, need to do it manually, will get to it asap. | 13:33 |
| kashyap | efried: Okay, I used the word "validate" too loseely then. Let me add that phrase in. | 13:33 |
| efried | right, I (as the reader) don't know what that means. | 13:34 |
| kashyap | (The word "validate" was supposed to capture that. And I know I will uncover more details as I mull on the implementation.) | 13:34 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655692 | 13:35 |
| gibi | efried: I saw the discussion about the lenght of the bandwith PTG slot | 13:35 |
| gibi | efried: I think we can shrink that to 15-20 minutes | 13:35 |
| efried | gibi: ack, I saw your response, will shrink. | 13:35 |
| efried | thanks. | 13:35 |
| gibi | efried: thanks | 13:36 |
| *** tetsuro has joined #openstack-nova | 13:36 | |
| gibi | efried: I also saw that you prepared the commit for the onboarding session. Is there anything I can still help with? | 13:36 |
| openstackgerrit | Eric Fried proposed openstack/nova stable/rocky: Fix {min|max}_version in ironic Adapter setup https://review.opendev.org/655693 | 13:38 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/queens: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655694 | 13:39 |
| efried | mriedem, melwitt: ^ | 13:39 |
| efried | gibi: I'm not completely sure how I'm going to present that. What I would like to do is have my actual IDE running on the projector. Fallback is showing everything in browser. | 13:40 |
| efried | I also don't know if I can fill 40 minutes with that demo :P | 13:40 |
| efried | if you have suggestions, I'm all ears. | 13:40 |
| efried | In any case, I would love to have you around for moral support. | 13:40 |
| *** eharney has joined #openstack-nova | 13:41 | |
| gibi | efried: If we run out of things and there are no questions then we can pull out the berlin presentation and talk about some part of it | 13:41 |
| gibi | efried: I will definitly in the room to help | 13:41 |
| efried | cool, thank you. | 13:41 |
| gibi | efried: especially the last slide https://docs.google.com/presentation/d/1T6_CZvDf2eFgbtTecIfbrMBkPiX9-yYRXXKakZjlyXs/edit?usp=sharing | 13:42 |
| *** altlogbot_0 has quit IRC | 13:42 | |
| sean-k-mooney | efried: kashyap i think https://blueprints.launchpad.net/nova/+spec/libvirt-video-device-models should now be fine? | 13:42 |
| kashyap | sean-k-mooney: I'd use capitalization correctly, but that's a nit :D. Also there is an unwanted line break for the word "graphic" | 13:44 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Remove IP proxy methods https://review.opendev.org/655695 | 13:44 |
| *** jiaopengju_2 has joined #openstack-nova | 13:44 | |
| *** jiaopengju_2 has quit IRC | 13:44 | |
| sean-k-mooney | updated | 13:45 |
| *** jiaopengju_2 has joined #openstack-nova | 13:45 | |
| sean-k-mooney | the content is there and anyone can edit so ill leave it to other to fix futher if it bothers them | 13:45 |
| *** brinzhang has quit IRC | 13:46 | |
| efried | sean-k-mooney, kashyap: Approved, thanks for the follow through. | 13:46 |
| *** tetsuro has quit IRC | 13:46 | |
| *** jiaopengju_1 has quit IRC | 13:46 | |
| kashyap | efried: Nod | 13:47 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec https://review.opendev.org/645814 | 13:47 |
| kashyap | efried: ^ Added the sentence on error logging. Does it look any better? | 13:47 |
| openstackgerrit | Merged openstack/os-traits master: Document policy of never removing traits https://review.opendev.org/655673 | 13:48 |
| efried | kashyap: Yes, thank you, except for the typo serice | 13:48 |
| sean-k-mooney | by the way i treied to have https://review.opendev.org/#/c/647733/ more or less be mergable in the first revision but i did wonder is there a place in our docs where the values of hw_video_model would/should be documented | 13:48 |
| kashyap | gibi: You were also +2 on the above. If you want to re-{ACK,NACK} in your copious free time | 13:48 |
| kashyap | Damn | 13:48 |
| sean-k-mooney | stephenfin: ^ any idea | 13:48 |
| kashyap | efried: I corrected it, didn't commit it :-( | 13:48 |
| gibi | kashyap: ack, I will try to go back there today | 13:48 |
| mriedem | adrianc: can you or hamdy (or moshe) start a backport for https://review.opendev.org/#/c/649345/ to stable/stein so we have it lined up? | 13:49 |
| *** yedongcan has left #openstack-nova | 13:49 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec https://review.opendev.org/645814 | 13:49 |
| mriedem | adrianc: oh nvm wrong patch | 13:49 |
| kashyap | gibi: Thanks. It's largely clarification text added based on the feedback | 13:49 |
| mriedem | adrianc: oh i guess that was right, i was looking at the neutron spec for some reason | 13:50 |
| efried | mriedem: Do you want a final look at ^ ? It had multiple +2s and I just asked kashyap to add detail on what "validate" means (it means refuse to start nova-compute, with a helpful log) so I was going to fast approve it. | 13:50 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Stop ignoring unknown libvirtError exceptions during volume attach https://review.opendev.org/655696 | 13:50 |
| mriedem | if you and gibi and alex_xu and everyone else are fine with it then no i don't need to spend time re-reviewing | 13:50 |
| efried | ack, thx | 13:51 |
| mriedem | lyarwood: thanks | 13:51 |
| kashyap | mriedem: Your comments were useful. I spent 3+ hours thinking and writing a response to it this morning. Thanks for taking time! | 13:52 |
| stephenfin | sean-k-mooney: Hmm, not sure. We have https://docs.openstack.org/nova/latest/admin/flavors.html but no image equivalent | 13:52 |
| *** dikonoor has joined #openstack-nova | 13:52 | |
| mriedem | stephenfin: it's in the glance docs | 13:53 |
| mriedem | oh, that's not the extra spec one | 13:53 |
| aspiers | efried: nova onboarding? maybe I can help with that | 13:53 |
| sean-k-mooney | stephenfin: ya i quickly looked before and didnt fined one so didnt add any | 13:53 |
| aspiers | efried: as a newbie to nova but not newbie to git/gerrit etc. | 13:54 |
| efried | aspiers: Love it. What did you have in mind? | 13:54 |
| sean-k-mooney | i am updating the glance metadef however to track the new values | 13:54 |
| aspiers | efried: nothing, I only just saw your reference to onboarding in the scrollback ;) | 13:54 |
| efried | aspiers: Good perspective: what were the most important things that helped you onboard into nova, being already familiar with openstack and its toolage? | 13:54 |
| sean-k-mooney | stephenfin: we also have ovos for the image properties to that also track the allowed values. | 13:54 |
| mriedem | stephenfin: sean-k-mooney: if you're looking for image metadata docs they are here https://docs.openstack.org/glance/latest/admin/useful-image-properties.html | 13:54 |
| sean-k-mooney | ah they are in glance | 13:55 |
| mriedem | managing images via the command line would be something i'd expect to find in the glance admin docs | 13:55 |
| efried | aspiers: i.e. what would you tell someone to do if they were in a similar position and looking to get onboarded | 13:55 |
| *** Luzi has quit IRC | 13:55 | |
| aspiers | efried: hrm, I guess "read the docs and ask a lot of questions on IRC" | 13:55 |
| sean-k-mooney | ill update them when i update the image metadefs | 13:55 |
| mriedem | https://docs.openstack.org/glance/latest/user/glanceclient.html | 13:55 |
| efried | aspiers: If you want to give a brief off-the-cuff/anecdotal talk in the session, that would be most welcome. | 13:55 |
| aspiers | efried: that might be possible. when is it? | 13:55 |
| efried | ... | 13:55 |
| sean-k-mooney | actuly i think these might be auto generated form teh metadefs | 13:55 |
| mriedem | stephenfin: https://docs.openstack.org/glance/latest/admin/manage-images.html | 13:56 |
| *** tetsuro has joined #openstack-nova | 13:56 | |
| mriedem | sean-k-mooney: they are not | 13:56 |
| mriedem | the useful-image-properties page is hand-written | 13:56 |
| efried | aspiers: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23620/nova-project-onboarding | 13:56 |
| sean-k-mooney | oh ok ill update them so. thanks for the pointer | 13:56 |
| kashyap | edleafe: On that traits patch, please see if I've answered your question | 13:56 |
| mriedem | lyarwood: if you fix the alignment here https://review.opendev.org/#/c/655696/1/nova/virt/libvirt/driver.py i'm +2 | 13:56 |
| kashyap | edleafe: Ah, you've actually responded; I just needed to refresh it | 13:57 |
| *** altlogbot_1 has joined #openstack-nova | 13:57 | |
| edleafe | kashyap: damn, I'm fast! | 13:57 |
| *** tetsuro has quit IRC | 13:57 | |
| kashyap | Damn, what I thought was a drive-by is becoming a shaving a farm of yaks :D | 13:58 |
| efried | kashyap: I was going to respond on that one too, with a summary. TLDR, what I said in my last response, do that split and "deprecate" the HW_CPU_AMD_SEV in favor of HW_CPU_X86_AMD_SEV | 13:58 |
| kashyap | efried: By "deprecate", you mean just add a note that it's bogus, right? | 13:59 |
| efried | yes | 13:59 |
| sean-k-mooney | efried: just re reviews kashyap cpu selection spec and im fine with the new verions. thanks kashyap | 13:59 |
| kashyap | sean-k-mooney: Thanks! | 13:59 |
| edleafe | Yeah, since the trait won't ever be removed. More of a "we screwed up; don't use this one" note | 14:00 |
| kashyap | sean-k-mooney: We'll get to debate fun details in the implementation :D I have a WIP post patch to introduce the infra, before it gets lost | 14:00 |
| *** hongbin has joined #openstack-nova | 14:00 | |
| efried | kashyap: re edleafe's comment on the NO_SSB thing, I thought that wasn't so much a negative trait as advertising that "this CPU has had the horrible thing removed". | 14:00 |
| *** tetsuro has joined #openstack-nova | 14:00 | |
| efried | iow, absent that trait, you can't tell whether it's there or not | 14:00 |
| efried | so we want that trait so the consumer can be sure the horrible thing is gone. | 14:01 |
| kashyap | Exactly | 14:01 |
| edleafe | efried: if that's the case, then rename it to a positive assertion of a capability | 14:01 |
| kashyap | That's why I wanted to mention it. Let me read Ed's comment | 14:01 |
| efried | edleafe: like SSB_REMOVED? | 14:01 |
| edleafe | sure, something along those lines | 14:01 |
| efried | edleafe: Would prefer to retain symmetry with the CPU traits themselves, though. | 14:01 |
| *** panda|lunch is now known as panda | 14:02 | |
| kashyap | "NO_SSB" == No more SSB vulnerability on this host CPU -- isn't that clear enough? | 14:02 |
| sean-k-mooney | kashyap: i would like to see two thing. 1 on sucessful validation print the cpu configurtion the guest will see as an info long. on failure pritn the requested combination, whant the host supports and what could not be supproted as an error log. | 14:02 |
| efried | with the names from the architecture, that is | 14:02 |
| *** tetsuro has quit IRC | 14:02 | |
| efried | kashyap: IMO it is, and is preferable to renaming it to make it sound more positive | 14:02 |
| efried | just need to get edleafe on board with that | 14:02 |
| kashyap | Okay, I'll go re-read the comments. These flags have some dizzying acronyms | 14:02 |
| sean-k-mooney | kashyap: im not sure we should be tracking vulnerblities as traits | 14:03 |
| kashyap | sean-k-mooney: Huh, we're not | 14:03 |
| edleafe | efried: not having something isn't a capabilty. Being clean of some defect is. | 14:03 |
| stephenfin | sean-k-mooney: What happens if this isn't true? https://review.opendev.org/#/c/629589/27/nova/virt/libvirt/driver.py@8161 | 14:03 |
| efried | edleafe: These are going to be consumed by people who are familiar with the arch and are looking for the equivalent trait, not people who are just casting around and trying to understand a trait based on its name. | 14:03 |
| sean-k-mooney | NO_SSB kind of is | 14:03 |
| kashyap | sean-k-mooney: It's a CPU flag. Please ask before you assume :-) | 14:03 |
| sean-k-mooney | i know what it is | 14:04 |
| sean-k-mooney | i was not assuming | 14:04 |
| kashyap | Well, it's a CPU flag. We need to track it as a "trait", if you want to report that a given Compute host is no more vulnerable to it | 14:04 |
| kashyap | Is something wrong with t? | 14:04 |
| kashyap | s/t/it/ | 14:04 |
| edleafe | kashyap: is it generally referred to as NO_SSB in the relevant documentation (asking because I have no idea) | 14:04 |
| sean-k-mooney | if we have NO_SSB i can do traits:no_SSB=forbiden and then deploy a workload that expliot it to a host that has the vulnerablity | 14:04 |
| kashyap | edleafe: Exactly what efried said. These are for admins who read their documentation (and they will be sufficiently rewarded) | 14:05 |
| efried | edleafe: https://git.qemu.org/?p=qemu.git;a=blob;f=docs/qemu-cpu-models.texi#l313 | 14:05 |
| mriedem | tssurya: fyi in case you haven't seen this https://review.opendev.org/#/c/636947/ | 14:06 |
| *** dpawlik has quit IRC | 14:06 | |
| efried | edleafe: This set of traits is being mapped directly from that document | 14:06 |
| mriedem | bauzas: ^ is a nice little perf improvement when listing AZs in a large cloud | 14:06 |
| edleafe | efried: ok, I understand a bit more. | 14:06 |
| bauzas | mriedem: okay | 14:06 |
| efried | edleafe: sections important_cpu_features_intel_x86 and important_cpu_features_amd_x86 | 14:06 |
| tssurya | mriedem: yeah I have it open | 14:06 |
| bauzas | mriedem: FWIW, I'll be traveling starting tonight | 14:06 |
| tssurya | thanks for doing that | 14:07 |
| edleafe | That doc also defines things such as mutually exclusive traits, which os-traits and placement doesn't enforce | 14:07 |
| sean-k-mooney | efried: edleafe i still think we need to consider the securtiy impact of having NO_SSB as a trait before we add it | 14:07 |
| efried | edleafe: true story, but not really relevant; the driver is going to detect what's on the CPU and report that. | 14:07 |
| mriedem | tssurya: thank avolkov | 14:07 |
| kashyap | edleafe: Only 'amd-no-ssb | 14:07 |
| kashyap | edleafe: Only 'amd-no-ssb' is the mutually exclusive trait. | 14:08 |
| kashyap | Again, what efried said ^ (on driver detecting) | 14:08 |
| edleafe | sean-k-mooney: what would the potential impact be? A CPU reporting it that is actually vulnerable to it? | 14:08 |
| sean-k-mooney | yes | 14:08 |
| tssurya | I tweaked a downstream patch when we opened that bug, never got around to doing it upstream | 14:08 |
| sean-k-mooney | and then a user uploading an image that require a cpu with the vulnerablity | 14:09 |
| efried | edleafe: sean-k-mooney is saying a malicious user could specify NO_SSB as a forbidden trait to deliberately land on a vulnerable host. | 14:09 |
| kashyap | sean-k-mooney: Also what is "forbidden" in this context? | 14:09 |
| efried | kashyap: We have the ability to specify a request with "DON'T land on a host if it has this trait" | 14:09 |
| sean-k-mooney | kashyap: we support requireing traitns or selecting host that do not have a trait | 14:09 |
| kashyap | efried: Ah-ha | 14:09 |
| efried | kashyap: The placement request would look like required=!HW_CPU_X86_AMD_NO_SSB | 14:09 |
| sean-k-mooney | so if we report either the presence or absence of ssbd | 14:09 |
| efried | kashyap: From a flavor extra spec like trait:HW_CPU_X86_AMD_NO_SSB=forbidden IIRC | 14:10 |
| sean-k-mooney | i can exploit that to target a host with the volnerablity with my payload | 14:10 |
| efried | Yeah, I don't know if this is a problem though sean-k-mooney | 14:10 |
| kashyap | Yeah, noted. | 14:10 |
| edleafe | sean-k-mooney: so without that trait, attackers could just spin up random VMs and see if they are on a vulnerable host. Having the trait just makes those hosts easier to find. Is that it? | 14:10 |
| kashyap | efried: Yeah, that's what I was wondering. The positives of having it outweigh some theoretical threat | 14:10 |
| efried | The admin is responsible for putting together the flavors. Why would the admin do that? | 14:10 |
| sean-k-mooney | edleafe: yes | 14:10 |
| sean-k-mooney | i would like the openstack security team to weigh in on this personally | 14:11 |
| efried | there's an email address for that, right? | 14:11 |
| mriedem | efried: gibi: trying to recreate this TestRPC infinite recursion is getting frustrating :) https://review.opendev.org/#/c/654468/ - i'm thinking maybe push another change to oslo.config (or nova?) that sets the max recursion depth and see if that blows things up? | 14:11 |
| sean-k-mooney | efried: users can upload image with traits request | 14:11 |
| sean-k-mooney | so its not admin im worried about | 14:11 |
| kashyap | efried: Yes, very good point on admin's responsibility | 14:12 |
| efried | http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-security | 14:12 |
| gibi | mriedem: I also tried to reproduce it locally without success. Do you assume that if you set the recursion limit to small enough then the stack trace will not be truncated? | 14:12 |
| efried | ugh, do you have to be subscribed to post to that? | 14:12 |
| *** phasespace has quit IRC | 14:12 | |
| kashyap | efried: Not necessarily | 14:13 |
| sean-k-mooney | efried: there is but we can also talk to them in person next week | 14:13 |
| kashyap | efried: Mailman moderator can approve you | 14:13 |
| sean-k-mooney | we will all be in the same place | 14:13 |
| kashyap | sean-k-mooney: In-person, things get too long-winded, and they get lost. | 14:13 |
| efried | sean-k-mooney: Okay, so maybe we need to have an explicit check for such traits and disallow them from image meta? | 14:13 |
| gibi | mriedem: anyhow sounds like a good idea | 14:13 |
| mriedem | gibi: that's the idea | 14:13 |
| *** dikonoor has quit IRC | 14:13 | |
| sean-k-mooney | efried: maybe | 14:13 |
| mriedem | alternatively, and this is probably easier, just change my oslo.config change from logging a traceback whe it hits a recursion of 3 to just raise | 14:13 |
| mriedem | i'll do that first | 14:13 |
| efried | sean-k-mooney, kashyap: So is one of you going to compose an email? Please copy me. (I'm not subscribed to that list.) | 14:14 |
| *** mmethot has quit IRC | 14:14 | |
| stephenfin | ralonsoh: It's a nit, but any chance you can fix this before we merge and backport? https://review.opendev.org/#/c/655332/4/vif_plug_linux_bridge/linux_net.py | 14:14 |
| *** mlavalle has joined #openstack-nova | 14:14 | |
| kashyap | efried: I'm not subscribed either. But will do, if I write. | 14:14 |
| efried | thanks. | 14:14 |
| sean-k-mooney | ok as i said im sure we can grab 10 mins with them next week at the ptg/fourm too | 14:15 |
| kashyap | Also, "exploiting" here is all hand-wavy, and requires some serious competency, and timing to take advantage of such things. | 14:15 |
| sean-k-mooney | efried: and yes we could have a blacklist of traits that cant be in the image | 14:15 |
| sean-k-mooney | kashyap: sure but not allo fo them do | 14:16 |
| efried | sean-k-mooney: Security people will tell you it should be a whitelist :( | 14:16 |
| *** tetsuro has joined #openstack-nova | 14:16 | |
| kashyap | sean-k-mooney: The thing is, I'll forget it in 5 seconds. My memory is worse than a bumblebee's (less than 8 seconds) | 14:16 |
| sean-k-mooney | well the sepc said that too if i recall | 14:16 |
| efried | or a goldfish | 14:16 |
| efried | did we have this conversation? | 14:16 |
| sean-k-mooney | but we never implemented it | 14:16 |
| kashyap | efried: Heh, I guess so | 14:16 |
| efried | and you forgot | 14:16 |
| efried | typical | 14:16 |
| efried | (My memory is pretty unpredictable. Some things I remember forever, some I need hammered in a thousand times and still can't hold onto.) | 14:17 |
| kashyap | efried: Yeah. On the other hand, we (I) should work towards getting the rest of the "uncontroversial" triats in, as that helps admins _today_ | 14:17 |
| aspiers | efried, kashyap: what's the current thinking on how to handle the SEV trait? keep as HW_CPU_AMD_SEV, or move to HW_CPU_X86_AMD_SEV? | 14:18 |
| kashyap | efried: Yeah, I hear ya...something similar here. | 14:18 |
| efried | kashyap, sean-k-mooney: I'm going to say it's fine to add the traits to os-traits regardless; the vulnerability occurs when we expose them via compute. | 14:18 |
| ralonsoh | stephenfin, well it makes sense. Because both parameters should be different, the current implementation is valid. But yes, I'll change it | 14:18 |
| aspiers | efried, kashyap: I have ongoing reviews which kinda rely on a decision on this ... | 14:18 |
| sean-k-mooney | efried: im wondering if we should namespace them | 14:18 |
| efried | aspiers: "deprecate" HW_CPU_AMD_SEV, "add" HW_CPU_X86_AMD_SEV | 14:18 |
| sean-k-mooney | or otherwise mark them up in os-traits | 14:19 |
| aspiers | efried: do we need a traits deprecation mechanism? other than just documenting? | 14:19 |
| efried | sean-k-mooney: Namespace them as HW_CPU_X86_POTENTIAL_SECURITY_VULNERABILITY_NO_SSB?? | 14:19 |
| *** tetsuro_ has joined #openstack-nova | 14:19 | |
| efried | aspiers: It's been discussed before, but we decided not to do it. So just documenting/commenting. | 14:19 |
| aspiers | efried: decided not to do it ever, or just not any time soon? | 14:19 |
| kashyap | aspiers: I saw your reorg comment, still have to look at it. (I'm still a bit unwell, so, slow today as well) | 14:19 |
| aspiers | kashyap: sorry to hear that :-( no pressure | 14:20 |
| efried | when we first split os-traits out of placement (in nova) there was a proposal (from jaypipes iirc) to do it with a more complex architecture that would allow for deprecation and removal. | 14:20 |
| efried | but we decided on the super-simple approach instead. | 14:20 |
| *** JamesBenson has joined #openstack-nova | 14:20 | |
| sean-k-mooney | efried: not quite but maybe have _NO_IMG_ or something | 14:20 |
| efried | aspiers: No plans in the forseeable future. | 14:20 |
| aspiers | efried: just wondering if https://review.opendev.org/#/c/655673/ should have explained that too | 14:20 |
| sean-k-mooney | that or add them to a seperate dict in os-traits | 14:21 |
| efried | sean-k-mooney: Yeah, no. We don't want to try to predict what those would be, and that's not the responsibility of os-traits IMHO. | 14:21 |
| sean-k-mooney | or list | 14:21 |
| kashyap | aspiers: Ah, nice. It's actually merged; efried++ | 14:21 |
| *** tetsuro has quit IRC | 14:21 | |
| efried | aspiers: Sure, could do, looks like mriedem felt the same. It's docs, easy to add more. | 14:21 |
| sean-k-mooney | well we should ask ourselve is this a security issue when adding new traits in general. anway i need to go finsih packing | 14:22 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655332 | 14:22 |
| kashyap | efried: Thanks; good point -- so nothing is blocking from adding the trait. Good | 14:23 |
| jaypipes | efried: you mean os-resource-classes, not os-traits, but yeah. | 14:23 |
| efried | sean-k-mooney: If we do the whitelist thing, then it forces us to think about it for every trait. But in the context of nova, not os-traits. | 14:23 |
| aspiers | efried: Well, if adding a deprecation mechanism to the code isn't completely ruled out, then probably no change is needed | 14:23 |
| efried | jaypipes: oh, was it os-resource-classes, okay. | 14:23 |
| jaypipes | efried: yeah. my proposal was here: https://github.com/jaypipes/os-resource-classes | 14:23 |
| efried | aspiers: FYI ^ | 14:23 |
| sean-k-mooney | efried: we could do it in nova i guess instead | 14:23 |
| aspiers | efried: In general I don't think it makes sense to document "we have no plan to do $X any time soon", since $X could be any number of things :) | 14:23 |
| kashyap | efried: (The "good point" being your "vuln. occurs when we expose them [traits] via compute") | 14:23 |
| jaypipes | efried: with a version history: https://github.com/jaypipes/os-resource-classes/blob/master/os_resource_classes/__init__.py#L21 | 14:23 |
| sean-k-mooney | just have a list of traits in nova that lists trats that cant be in an image or put it in a config | 14:24 |
| jaypipes | we ended up not going with that and instead simply listing constants in a library. | 14:24 |
| efried | aspiers: Yes, that's true, but I think we're talking more about like in the SEV case: "We misnamed this one when we introduced it - please use $Y instead." | 14:24 |
| sean-k-mooney | then as we add usage in nova we can extend the defualt of the config option | 14:24 |
| aspiers | efried: no I was talking specifically about https://docs.openstack.org/os-traits/latest/contributor/index.html | 14:24 |
| aspiers | efried: but I was also about to ask whether I should submit a follow up to https://review.opendev.org/#/c/655671/ to "rename" the SEV trait (deprecating the original) ? | 14:25 |
| aspiers | efried: or whether I should instead amend that review | 14:25 |
| *** tetsuro_ has quit IRC | 14:25 | |
| aspiers | I guess amending makes more sense | 14:25 |
| *** Sundar has joined #openstack-nova | 14:25 | |
| efried | aspiers: I think kashyap is going to do that as part of his patch, because he's going to be splitting x86 into amd and intel | 14:25 |
| efried | It could be done as separate patches, but meh. | 14:25 |
| *** gmann has joined #openstack-nova | 14:26 | |
| aspiers | kashyap: when do you expect your reorg to land? | 14:26 |
| kashyap | aspiers: As soon as I address the feedback :-) We can work it out | 14:27 |
| openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648659 | 14:27 |
| efried | kashyap: CPU with hypervisors spec is merging and bp is approved, striking from PTG agenda, cool? | 14:28 |
| mriedem | btw, with trait names like HW_CPU_X86_POTENTIAL_SECURITY_VULNERABILITY_NO_SSB i look forward to the day when we blast the url limit for GET query params | 14:28 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468 | 14:29 |
| efried | I was being facetious with that. | 14:29 |
| sean-k-mooney | mriedem: im sure we already to if we try to use everything we currently support | 14:29 |
| efried | mriedem: By then we'll surely have json payload for requests. | 14:29 |
| jaypipes | mriedem: coupled with NET_BW_EGR_KILOBIT_PER_SEC, that would be awesome. | 14:29 |
| aspiers | kashyap: just trying to work out the order of work, since a lot of these changes depend on each other | 14:29 |
| jaypipes | used to be even longer than that :) | 14:29 |
| *** tssurya has quit IRC | 14:29 | |
| sean-k-mooney | efried: if we do have a json payload we will also need to use a post not a get | 14:29 |
| efried | you keep saying that :P | 14:30 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't run tempest/devstack jobs on nova/test.py only changes https://review.opendev.org/655121 | 14:30 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468 | 14:30 |
| sean-k-mooney | i do | 14:30 |
| efried | I'm sure you must be right. I don't really care what http verb it is. | 14:30 |
| aspiers | do we have any mechanism for plotting dependency graphs between commits in Gerrit? | 14:30 |
| *** artom has joined #openstack-nova | 14:30 | |
| sean-k-mooney | because i keep seeing peole propose apis with get bodies | 14:30 |
| efried | aspiers: You mean visually? | 14:30 |
| mriedem | efried: gibi: ^ so that series now does 2 things: 1. the nova change sets a low recursion limit (probably won't help) and 2. the oslo.config change doesn't print, it just counts recursion and raises if we call _get() more than 5 times on the same option | 14:31 |
| efried | mriedem: ack | 14:31 |
| kashyap | aspiers: Will address that as quickly as I can. | 14:31 |
| gibi | mriedem: ack, fingers crossed | 14:31 |
| efried | aspiers: And do you mean dependencies based on line conflicts, or based on commits in series and Depends-On? | 14:31 |
| openstackgerrit | Merged openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec https://review.opendev.org/645814 | 14:32 |
| efried | The former is displayed in gerrit, top right corner, "Conflicts With" | 14:32 |
| efried | ^ kashyap \o/ | 14:32 |
| aspiers | efried: commits in series and Depends-On | 14:32 |
| kashyap | efried: In your "copious free time", want to respond to Ed here on "NO_{trait_name}", since your "Traits muscle" is better trained? -- https://review.opendev.org/#/c/655193/2/os_traits/hw/cpu/amd.py@19 | 14:32 |
| aspiers | efried: well, IMNSHO https://github.com/aspiers/git-deps does a nicer job of the former ;-) | 14:33 |
| efried | aspiers: I know of no graphical tool, but that definitely doesn't mean there isn't one. Might ask the infra guys | 14:33 |
| kashyap | efried: Thank you! IIRC, it is "encouraged" to update the spec file as "new information arises" (I know jaypipes likes to use Labowski's phrase here :D)? | 14:33 |
| aspiers | efried: yeah I was thinking of something like the way launchpad shows a dep graph for blueprints | 14:34 |
| efried | kashyap: Yes, definitely. Common practice to propose spec amendments e.g. once the impl lands to make it reflect reality. | 14:34 |
| kashyap | Alright, I've seen a few amendments in the specs repo. | 14:34 |
| aspiers | efried: speaking of which, I noticed one of the SEV work items might need changing but I'm not sure | 14:34 |
| mriedem | gibi: i see you have https://blueprints.launchpad.net/nova/+spec/support-server-move-operations-with-ports-having-resource-request which your move spec is tracked against, and you also have https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request | 14:34 |
| efried | kashyap: I can do that (respond to edleafe in the review) or you could just point to the eavesdrop of the conversation we had above. | 14:34 |
| mriedem | gibi: does the former supersede the latter? | 14:35 |
| kashyap | efried: Okay, I'll do the latter; it'll save you time. | 14:35 |
| efried | kashyap: thanks | 14:35 |
| mriedem | gibi: or is the latter the reason for ptg discussion? | 14:35 |
| gibi | mriedem: https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request is a collection of further enhancements for bandwidth | 14:35 |
| aspiers | efried: work item #7 http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html | 14:35 |
| efried | mriedem: As I understand it, enhance-support is an overarching longer-term bp that excludes support-server-move | 14:35 |
| mriedem | ok | 14:35 |
| gibi | mriedem: on the ptg I would like to get decision about 1) the microversion for move 2) the priority of the items in https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request | 14:36 |
| aspiers | efried: should it be checking the trait or the resource class? I guess only the trait is SEV-specific and so is the required XML, so maybe this is still correct | 14:36 |
| mriedem | btw, we have a working multi-cell ci job https://review.opendev.org/#/c/655222/ | 14:36 |
| efried | aspiers: I would check the allocation. Remember, the request won't necessarily include the trait. | 14:36 |
| efried | aspiers: So yeah, that should be amended at some point. | 14:37 |
| dansmith | is this a known thing? http://logs.openstack.org/37/655137/2/check/openstack-tox-lower-constraints/6363387/testr_results.html.gz | 14:37 |
| efried | You may want to open a change set with that, but leave it -W to collect such deltas as they arise. | 14:37 |
| dansmith | testrpc fail | 14:37 |
| kashyap | aspiers: BTW, to quickly answer on the SEV trait -- I'd suggest to keep it as-is: HW_CPU_AMD_SEV (in amd.py) | 14:37 |
| dansmith | on a lower constraints job no less :) | 14:37 |
| aspiers | efried: I was thinking the request filter could add not only resources:MEM_ENCRYPTION_CONTEXT=1 but also traits:HW_CPU_X86_AMD_SEV=required | 14:37 |
| efried | dansmith: yes, that's bug... | 14:37 |
| kashyap | aspiers: Because we'll be consolidating all AMD-related stuff in one place, as it should be. | 14:38 |
| efried | dansmith: bug 1825435 | 14:38 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/1825435 | 14:38 |
| dansmith | ack | 14:38 |
| aspiers | kashyap: so not under HW_CPU_X86_AMD? | 14:38 |
| aspiers | isn't that the opposite of what efried wrote 20 mins ago? | 14:38 |
| kashyap | aspiers: Wait, disregard me. I'll re-read the chat and slow-respond later. | 14:39 |
| aspiers | kashyap: OK :) | 14:39 |
| efried | aspiers: We talked about this. The request filter shouldn't do that. The admin (or whoever authors the flavor/image-meta) should request SEV via the trait if they want SEV specifically. Otherwise you could theoretically land anywhere that supports mem enc. | 14:39 |
| aspiers | efried: but at some point in the workflow the code needs figure out that SEV is the technology which will be used to implement it. Where should that happen? | 14:40 |
| aspiers | I guess your point is that doing it in a request filter is too early | 14:40 |
| sean-k-mooney | aspiers: in the virt dirver | 14:40 |
| efried | aspiers: Right; at spawn time, libvirt will use the same mechanism it used to decide to expose that trait in the first place. | 14:40 |
| efried | aspiers: ...or possibly just look for the trait itself as already exposed, not sure if you have access to the provider tree in spawn | 14:41 |
| sean-k-mooney | placment will land the vm on a host that support mem encryption and then the virt driver can figure out which tech implementes it | 14:41 |
| efried | ^ | 14:41 |
| aspiers | yeah makes sense | 14:41 |
| mriedem | dansmith: yes | 14:42 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-multi-cell job https://review.opendev.org/655222 | 14:42 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Enable n-novnc in nova-multi-cell job https://review.opendev.org/655711 | 14:42 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Stop ignoring unknown libvirtError exceptions during volume attach https://review.opendev.org/655696 | 14:42 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Always disconnect volumes after libvirtError exceptions https://review.opendev.org/655712 | 14:42 |
| mriedem | http://status.openstack.org/elastic-recheck/#1825435 | 14:42 |
| aspiers | efried, sean-k-mooney: so other than updating the commit message and maybe the trait name, I think https://review.opendev.org/#/c/633855/ can remain the same | 14:42 |
| mriedem | http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005392.html | 14:43 |
| dansmith | mriedem: I didn't get an e-r report on it | 14:43 |
| mriedem | dansmith: e-r bot has been off for weeks as far as i know | 14:43 |
| openstackgerrit | Stephen Finucane proposed openstack/os-vif stable/stein: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655678 | 14:43 |
| aspiers | efried, sean-k-mooney: and I'll add a separate review which adds the new config option and inventory of the new resource class from it | 14:43 |
| mriedem | dansmith: i have a couple of debug patches up to try and recreate to figure out what's causing the recursion stack dump | 14:43 |
| dansmith | mriedem: consider me shamed | 14:43 |
| *** mvkr has quit IRC | 14:43 | |
| mriedem | dansmith: https://review.opendev.org/#/q/topic:bug/1825435+status:open | 14:43 |
| mriedem | if you have ideas on that it'd be cool because so far my recreate attempts are failing | 14:44 |
| mriedem | it seems to have started somewhere along the lines of the cells v1 removal series | 14:44 |
| mriedem | but looking through those changes i don't see anything obvious | 14:44 |
| mriedem | it's in both py36 (upper-constraints) and lower-constraints so it's not some specific oslo.config version either | 14:44 |
| efried | aspiers: I haven't reviewed that patch in forever, sorry. | 14:45 |
| mriedem | i also didn't see any python3.6 changes recently in the bionic UCA changelog | 14:45 |
| dansmith | mriedem: ack, I heard about the recursion thing, but this is a single output message (not a very long stack trace) so I didn't connect the two | 14:45 |
| aspiers | efried: it's drastically simpler now | 14:45 |
| efried | aspiers: I probably won't get a chance to review patches-that-aren't-specs until after the PTG dust settles. | 14:45 |
| mriedem | dansmith: yeah the TestRPC ... inprogress seems to be a result of the stack dump earlier in the logs | 14:46 |
| aspiers | efried: because kashyap persuaded me to split out most of it into https://review.opendev.org/#/c/655268/ | 14:46 |
| mriedem | the former doesn't always show up when this hits, but the latter does | 14:46 |
| efried | aspiers: sight unseen, split is good. | 14:46 |
| aspiers | efried: ok | 14:46 |
| kashyap | Yeah, that split is useful -- because that split-up patch, which introduces getDomainCapabilities API, can be used by many other things | 14:47 |
| kashyap | So it can be merged as a separate piece once we iron out any remaining wrinkles. | 14:47 |
| kashyap | Thanks for doing the split, aspiers. | 14:48 |
| * kashyap --> "low tea" | 14:48 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Enable n-novnc in nova-multi-cell job https://review.opendev.org/655711 | 14:50 |
| *** jiaopengju_2 has quit IRC | 14:50 | |
| *** jiaopengju_2 has joined #openstack-nova | 14:50 | |
| mriedem | dansmith: btw https://review.opendev.org/#/c/655222/ is passing which is pretty exciting | 14:50 |
| dansmith | mriedem: that's cool | 14:51 |
| dansmith | mriedem: is that intended to be voting? | 14:52 |
| mriedem | once that lands, i can throw a patch at the end of my cross-cell-resize series to enable resize and cold migrate across the cells | 14:52 |
| mriedem | dansmith: yes | 14:52 |
| dansmith | are you ready for it to go in? | 14:52 |
| mriedem | we could recheck a few times if we care and then turn it on next week | 14:53 |
| mriedem | i can also send a heads up to the ML | 14:53 |
| openstackgerrit | garyk proposed openstack/nova master: VMware: populate datastore refs at init https://review.opendev.org/574688 | 14:53 |
| mriedem | if we want, we can land it as non-voting check only for now | 14:53 |
| mriedem | but i'd really like it to be voting asap since we should have had this several releases ago | 14:53 |
| mriedem | i'll post to the ML first | 14:54 |
| openstackgerrit | garyk proposed openstack/nova master: Improve metadata performance https://review.opendev.org/615435 | 14:55 |
| *** ttsiouts has quit IRC | 14:56 | |
| *** ttsiouts has joined #openstack-nova | 14:57 | |
| *** mmethot has joined #openstack-nova | 14:57 | |
| efried | mriedem: Do we start -2ing pike patches now, unless they're security or something? | 14:59 |
| mriedem | no | 14:59 |
| efried | (and by "we" I mean stable folk) | 14:59 |
| mriedem | that's pre-em process | 15:00 |
| mriedem | because there was an eol | 15:00 |
| openstackgerrit | Adam Spiers proposed openstack/nova-specs master: Update SEV work item to new approach based on MEM_ENCRYPTION_CONTEXT https://review.opendev.org/655717 | 15:00 |
| mriedem | with em we've done away with eol unless we can't manage the branch anymore (or don't want to) | 15:00 |
| aspiers | efried, sean-k-mooney: ^^^ minor tweak based on discussion just now | 15:00 |
| efried | so what does em actually mean? | 15:00 |
| *** _erlon_ has joined #openstack-nova | 15:00 | |
| efried | how is it different from just continuing to backport changes? | 15:00 |
| *** ttsiouts has quit IRC | 15:01 | |
| mriedem | efried: https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.html | 15:01 |
| mriedem | or https://superuser.openstack.org/articles/extended-maintenance-openstack/ if you like bloggy | 15:02 |
| sean-k-mooney | efried: extended mainatince | 15:02 |
| *** ivve has quit IRC | 15:03 | |
| efried | aspiers: not quite, commented. | 15:03 |
| efried | mriedem: ugh, I'm totally not going to read that much text right now. I'll satisfy my curiosity later. | 15:03 |
| mriedem | mission accomplished | 15:04 |
| aspiers | efried: OK thanks. I was thinking it could rely on the translation from hw:mem_encryption to resource class in the request filter, but checking allocations dict also sounds fine | 15:04 |
| efried | aspiers: Remember that translation isn't being persisted, so the flavor spawn gets to look at won't have it. | 15:05 |
| aspiers | efried: ohhhh, you mean that only happens in the scheduler, and the resulting RequestSpec doesn't get passed through to the nova-compute service? | 15:06 |
| efried | mriedem: I've updated the nova mtg agenda | 15:06 |
| efried | aspiers: Yes, iiuc. I could be missing something though | 15:06 |
| aspiers | efried: that makes sense - this stuff isn't obvious to me yet since I'm still very unfamiliar with the nova codebase, so thanks for the pointers | 15:07 |
| efried | aspiers: But IMO looking at the allocation is a cleaner solution regardless. | 15:07 |
| aspiers | efried: yup, sounds fine to me | 15:07 |
| mtreinish | aspiers: unfortunately I won't be at Denver. But I save that talk to watch later, at least based on the abstract it looks like they had a similar idea to what andreaf and I were starting to do in CIML (at a high level) | 15:08 |
| efried | aspiers: If you wanted to use hw:mem_encryption=true, you would have to check both the flavor and the image. So yeah, use the allocation. | 15:08 |
| mtreinish | but they approched it with different ML techniques (and ones that might be better suited for the problem) | 15:08 |
| aspiers | mtreinish: cool! very interested to hear your take on the talk. BTW I noticed earlier that elastic recheck / crm114 work seems to have made some v nice progress since last time I looked | 15:09 |
| aspiers | efried: changing the spec tweak now | 15:09 |
| aspiers | mtreinish: sorry to miss you this time though. we'll have to sort out those stestr issues online instead ;-) | 15:10 |
| mtreinish | aspiers: yeah it's too bad. Luckily for you though masayukig will be there and he can help sort through those in person :P | 15:11 |
| aspiers | mtreinish: true :) | 15:12 |
| mtreinish | aspiers: hmm, I wasn't aware anyone had done anything with the crm114 stuff in a while. But I hadn't been watching it super closely | 15:12 |
| aspiers | mtreinish: well it's a *long* time since I last more than glanced at http://status.openstack.org/elastic-recheck/ | 15:13 |
| *** itlinux has joined #openstack-nova | 15:13 | |
| aspiers | mtreinish: maybe I just never noticed some of the nice stuff before | 15:14 |
| *** itlinux has quit IRC | 15:14 | |
| *** igordc has joined #openstack-nova | 15:19 | |
| bauzas | efried: I wasn't originally thinking to discuss about NUMA affinity for vGPUs in the PTG https://review.opendev.org/#/c/650963/ but given gibi provided some good concerns, I wondered when we could have time for eventually discussing it at the PTG | 15:20 |
| bauzas | this isn't technically a placement-related spec, but I'd love to see it discussed then | 15:21 |
| openstackgerrit | Adam Spiers proposed openstack/nova-specs master: Update SEV work item to new approach based on MEM_ENCRYPTION_CONTEXT https://review.opendev.org/655717 | 15:21 |
| *** luksky has quit IRC | 15:22 | |
| bauzas | efried: so I proposed it for the x-proj session between nova/placement if you are okay with | 15:22 |
| *** itlinux has joined #openstack-nova | 15:25 | |
| *** lpetrut has joined #openstack-nova | 15:25 | |
| efried | bauzas: Yup, wfm. cdent is going to be organizing that chunk of time. | 15:25 |
| bauzas | k | 15:25 |
| dansmith | mriedem: what's your feeling on adding (i.e. requiring) tests for the virt drivers I'm adding image type flags for? | 15:26 |
| dansmith | if there's nothing dynamic, then it's just a test to assert that two dicts match | 15:26 |
| mriedem | thus not a very useful test, i don't have strong feelings about that | 15:27 |
| mriedem | i'd say make sure it's mentioned in the commit message though... | 15:27 |
| dansmith | okay | 15:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: handle port allocation during migration https://review.opendev.org/655112 | 15:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: func test for migrate server with ports having resource request https://review.opendev.org/655113 | 15:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations https://review.opendev.org/655114 | 15:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls https://review.opendev.org/655721 | 15:28 |
| efried | kashyap: secure boot topic, okay if I move it to Thursday 1400? | 15:29 |
| jaypipes | stephenfin: +2 | 15:30 |
| kashyap | efried: Thursday is the first day, right? | 15:30 |
| stephenfin | jaypipes: \o/ | 15:30 |
| efried | yes | 15:30 |
| stephenfin | jaypipes: Thanks for taking the time to look through that | 15:30 |
| kashyap | efried: I might need to be elsewhere. But we've discussed it also in Dublin, and people agreed it is something we want. | 15:30 |
| kashyap | efried: So even if I miss it, it should be OK? | 15:30 |
| efried | kashyap: If Saturday is better, I'll leave it there. Just had an opening. | 15:31 |
| jaypipes | stephenfin: np, that *you* for having the energy to keep it going. | 15:31 |
| efried | kashyap: But even better, if it's something we don't need to discuss, I can just strike it. | 15:31 |
| kashyap | efried: You know what, maybe just put it on Thurs 14:00. I'll work it out to be there | 15:31 |
| efried | okay | 15:32 |
| stephenfin | jaypipes: Also, fwiw, I'd love to deprecate those options and would like to do so at some point in the future. Just need to figure out how to deprecate these things that are essentially unversioned APIs | 15:32 |
| stephenfin | Baby steps | 15:32 |
| kashyap | efried: It should be a quick thing. No elaborate discussion needed | 15:32 |
| * kashyap bbiab | 15:32 | |
| mriedem | stephenfin: do you know what might have changed in our tox setup such that if you specify a target that doesn't exist, it used to fail fast but now it doesn't | 15:34 |
| mriedem | i.e. tox -e foo | 15:34 |
| mriedem | pretty sure i've fat-fingered some target to see it pass when the actual thing would have failed a test | 15:35 |
| mriedem | yeah heh i ran tox -e fast7 | 15:36 |
| mriedem | and tox -e api-samles | 15:36 |
| stephenfin | mriedem: Hmm | 15:36 |
| stephenfin | mriedem: I've spotted that and I don't know, unfortunately. | 15:36 |
| stephenfin | $ tox -e fast7 | 15:36 |
| stephenfin | ERROR: unknown environment 'fast7' | 15:36 |
| stephenfin | ^ I still see that on my machine (TM) | 15:36 |
| mriedem | which version of tox do you have? | 15:36 |
| stephenfin | $ tox --version | 15:37 |
| stephenfin | 3.4.0 | 15:37 |
| *** liuyulong has quit IRC | 15:37 | |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Add new "supports_amd_sev" capability to libvirt driver https://review.opendev.org/638680 | 15:37 |
| mriedem | $ tox --version | 15:40 |
| mriedem | 3.7.0 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc | 15:40 |
| stephenfin | mriedem: https://tox.readthedocs.io/en/latest/changelog.html#v3-8-0-2019-03-27 | 15:40 |
| stephenfin | "Fix missing error for tox -e unknown when tox.ini declares envlist. - by @medmunds #1160" | 15:40 |
| stephenfin | Looks like a bug in 3.7.0 | 15:40 |
| mriedem | ah https://github.com/tox-dev/tox/issues/1160 | 15:40 |
| mriedem | Successfully installed tox-3.8.0 | 15:41 |
| mriedem | osboxes@osboxes:~/git/nova$ tox -e foo | 15:41 |
| mriedem | ERROR: unknown environment 'foo' | 15:41 |
| mriedem | hot diggity damn | 15:41 |
| mriedem | thanks stephenfin | 15:41 |
| stephenfin | no problemo | 15:41 |
| *** efried has quit IRC | 15:43 | |
| *** efried has joined #openstack-nova | 15:43 | |
| efried | wow | 15:44 |
| *** dave-mccowan has joined #openstack-nova | 15:44 | |
| efried | note to self: Ctrl+Alt+C does *not* open the calculator | 15:44 |
| melwitt | mriedem: grafana shenanigans proposed here https://review.opendev.org/655591 wondering if I should add stable branch panels to it too | 15:44 |
| aspiers | efried: lol | 15:45 |
| mriedem | melwitt: ah it was the per-project thing that was busted, maybe that's what i talked about with corvus before | 15:46 |
| mriedem | thanks for fixing that | 15:46 |
| *** lajoskatona has joined #openstack-nova | 15:47 | |
| *** boxiang has quit IRC | 15:48 | |
| *** dave-mccowan has quit IRC | 15:49 | |
| *** boxiang has joined #openstack-nova | 15:49 | |
| lajoskatona | mriedem: Hi, regardig https://review.opendev.org/640600 & https://review.opendev.org/640601 (tempest schema validation for nova v2.70 & 2.71) if you need help to push them I am happy to help you out | 15:50 |
| mriedem | lyarwood: fyi https://graphite.opendev.org/render/?width=1231&height=593&_salt=1556207646.49&from=-8weeks&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.master.job.tempest-full-py3.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.master.job.devstack-plugin-ceph-tempest.FAILURE | 15:54 |
| mriedem | you can get the graph from graphite | 15:54 |
| mriedem | lajoskatona: go ahead, i'm not working on those | 15:55 |
| mriedem | melwitt: nice catch on the tempest-full-py3 vs devstack-plugin-ceph-tempest (not py3) in the nova case, we should probably change nova to run against the devstack-plugin-ceph-tempest-py3 job instead | 15:55 |
| lajoskatona | mriedem: ok, do I need any special thing to test in devstack? | 15:55 |
| melwitt | mriedem: yeah, I was wondering that too | 15:56 |
| mriedem | melwitt: that will throw the graph data off, but we could switch now, make your graph change depend on that and then revisit in a couple of weeks to compare stability | 15:56 |
| mriedem | or just do that in a follow up, whatever | 15:57 |
| mriedem | lajoskatona: i'm not sure i understand the question | 15:57 |
| * melwitt nods | 15:57 | |
| *** weshay|rover is now known as weshay | 15:59 | |
| lajoskatona | mriedem: never mind, I try to to push, and come back if I have questions :-) | 16:00 |
| mriedem | lajoskatona: ack and thanks | 16:00 |
| lajoskatona | mriedem: :-) | 16:01 |
| *** lajoskatona has quit IRC | 16:01 | |
| *** boxiang has quit IRC | 16:05 | |
| *** rpittau is now known as rpittau|afk | 16:05 | |
| *** boxiang has joined #openstack-nova | 16:05 | |
| *** wwriverrat has quit IRC | 16:10 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add ironic driver image type capabilities https://review.opendev.org/655729 | 16:11 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add vmware driver image type capabilities https://review.opendev.org/655730 | 16:11 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add xenapi driver image type capabilities https://review.opendev.org/655731 | 16:11 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add zvm driver image type capabilities https://review.opendev.org/655732 | 16:11 |
| dansmith | I dunno who to tag on the zvm one for review | 16:12 |
| dansmith | but I'm pretty sure they only support raw | 16:12 |
| melwitt | mriedem: I dunno if you saw, they did hit the eventlet monkey patching issue in ubuntu, last comment https://review.opendev.org/647310 | 16:13 |
| *** dklyle has quit IRC | 16:15 | |
| *** dklyle has joined #openstack-nova | 16:15 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Always pass HostAPI to get_availability_zones https://review.opendev.org/655583 | 16:16 |
| *** pcaruana has quit IRC | 16:19 | |
| *** igordc has quit IRC | 16:21 | |
| *** igordc has joined #openstack-nova | 16:23 | |
| *** cooper6581 has joined #openstack-nova | 16:26 | |
| *** tbachman has quit IRC | 16:27 | |
| *** ivve has joined #openstack-nova | 16:30 | |
| *** wwriverrat has joined #openstack-nova | 16:30 | |
| *** Sundar has quit IRC | 16:30 | |
| mriedem | melwitt: yeah i saw | 16:35 |
| mriedem | coreycb didn't +1 it though... | 16:35 |
| mriedem | dansmith: i suppose add jichenjc | 16:35 |
| *** zbr is now known as zbr|rover | 16:37 | |
| melwitt | I guess I took his comment on the review to be in support. but yeah, coreycb if you could +1 https://review.opendev.org/647310 if you are in support of backporting it | 16:38 |
| *** alex_xu has joined #openstack-nova | 16:40 | |
| *** pcaruana has joined #openstack-nova | 16:40 | |
| imacdonn | somewhat of a tangent, I suppose ... but I think we need to figure out what to do about the eventlet monkey_patching vs. WSGI issue ... it's blocking me from upgrading to Stein | 16:40 |
| melwitt | afaik there's not a clear thing we should do to address the problem, or did I miss it? eventlet monkey patching is not something I understand well | 16:42 |
| imacdonn | same here | 16:44 |
| *** gyee has joined #openstack-nova | 16:44 | |
| imacdonn | the nova docs rather strongly suggest moving to WSGI, but maybe my only option is to go back to running nova-api the old way :/ | 16:45 |
| imacdonn | alternative interim may be to hack the monkey-patching out, but I don't understand the implications of that well enough ... | 16:46 |
| dansmith | yeah I don't think you can just remove it | 16:46 |
| imacdonn | I don't have it in my Rocky deployments, though | 16:47 |
| dansmith | don't have what? the problem or the monkeypatching? | 16:47 |
| imacdonn | both ;) | 16:47 |
| dansmith | hrm, I'm trying to think about what the api service does that might continue to work without being patched | 16:48 |
| dansmith | if we use all the regular primitives, then it might just work with regular threading, but I expect there are some subtle behavior assumptions | 16:48 |
| imacdonn | it was added here: https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae | 16:48 |
| dansmith | locks we don't hold | 16:48 |
| melwitt | I have a patch series that is going on the route to removing eventlet dependency for nova-api https://review.opendev.org/650172 but it needs a follow up patch that makes the executor type configurable (or outright changes it away from green threads) | 16:48 |
| dansmith | imacdonn: right, but it was added because we didn't realize that we weren't patching when the wsgi stuff was split out from service originally, IIRC | 16:49 |
| melwitt | I'm not sure if there are other reasons aside from scatter-gather that eventlet is a dependency in nova-api | 16:49 |
| imacdonn | dansmith: right, and presumably someone decided that it was, infact, needed, with WSGI | 16:50 |
| *** udesale has quit IRC | 16:50 | |
| imacdonn | dansmith: there's mention of multiple cells (which I don't have (other than cell0)) ... but dunno if there are other cases where it's "needed" | 16:50 |
| dansmith | melwitt: that'd be the big one, but any other spawn we do in some layer that can be called from api and other services | 16:50 |
| dansmith | imacdonn: right but even with cell0 you're affected by that | 16:51 |
| coreycb | melwitt: mriedem: added my +1 | 16:51 |
| *** igordc has quit IRC | 16:52 | |
| dansmith | I've been a bit disconnected.. is the problem that seems to be introduced the amq heartbeat thing? | 16:52 |
| *** ivve has quit IRC | 16:52 | |
| mriedem | imacdonn: i thought https://review.opendev.org/#/c/647310/ was in response to https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae in a way, but you said that fix doesn't help you right? | 16:53 |
| imacdonn | mriedem: that just changes where monkey-patching gets inserted ... I have my problem either way | 16:54 |
| mriedem | i guess you could try exporting OS_NOVA_DISABLE_EVENTLET_PATCHING=1 but that's likely not something you want to run with | 16:57 |
| imacdonn | yeah | 16:59 |
| mriedem | dansmith: there are only 4 places we use the long_rpc_timeout heartbeat stuff and they'd all be calls from either conductor or compute neither of which are wsgi so i'm not sure how that would be involved, unless it was the moving of the monkey patch location in https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae | 16:59 |
| dansmith | mriedem: I think the long rpc stuff all uses regular threading primitives so it should work regardless of if you're patched or not right? | 17:00 |
| mriedem | idk, i've never looked at the oslo.messaging implementation | 17:00 |
| dansmith | mriedem: do you have reason to believe the the long rpc stuff is related, or are you just looking for related things? | 17:00 |
| mriedem | imacdonn: have you tried putting melwitt's series https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged) into rocky and see if that fixes your issue? | 17:01 |
| dansmith | mriedem: that patch doesn't actually change anything yet, IIUC | 17:01 |
| mriedem | dansmith: just in response to "(11:52:08 AM) dansmith: I've been a bit disconnected.. is the problem that seems to be introduced the amq heartbeat thing?" | 17:01 |
| dansmith | it just wraps use of the eventlet stuff so we can do it differentyly | 17:01 |
| melwitt | that's still using eventlet, you'd have to change the futurist executor to the native one | 17:01 |
| dansmith | yeah ^ | 17:01 |
| dansmith | mriedem: I think you're confusing heartbeats no? | 17:02 |
| dansmith | mriedem: the heartbeat that long_rpc added is layer 7, unrelated to AMQ heartbeats, which I thought was the subject of that thread | 17:02 |
| mriedem | oh i guess i am then | 17:03 |
| mriedem | i'll step out of this mess (this is why i'm not involved) | 17:03 |
| dansmith | mriedem: yeah, re-reading, they're getting a connection timeout because AMQP heartbeats aren't being sent (a different thing) | 17:04 |
| dansmith | the theory is that eventlet patching prevents some thread from running that sends those | 17:04 |
| dansmith | totes unrelated to long_rpc | 17:04 |
| imacdonn | yes, that's how it appears | 17:04 |
| *** derekh has quit IRC | 17:08 | |
| *** Sundar has joined #openstack-nova | 17:10 | |
| openstackgerrit | melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486 | 17:13 |
| *** dtantsur is now known as dtantsur|afk | 17:14 | |
| *** ricolin has joined #openstack-nova | 17:20 | |
| *** ralonsoh has quit IRC | 17:20 | |
| openstackgerrit | Adam Spiers proposed openstack/os-traits master: Update SEV trait docs to avoid misleading people https://review.opendev.org/655671 | 17:21 |
| *** luksky has joined #openstack-nova | 17:27 | |
| *** psachin has quit IRC | 17:28 | |
| *** lpetrut has quit IRC | 17:34 | |
| *** igordc has joined #openstack-nova | 17:35 | |
| dansmith | melwitt: the Future object makes it sound like cancel() might not always be supported.. have you confirmed? | 17:38 |
| *** itlinux has quit IRC | 17:39 | |
| melwitt | dansmith: yeah, was just writing that on the review. I think it only does something for eventlet GreenThreadPoolExecutor. been looking around in the source to confirm because it doesn't seem like the docs are explicit about which types support cancel() | 17:39 |
| dansmith | melwitt: "If the call is currently being executed and cannot be cancelled then the method will return" | 17:39 |
| dansmith | I think since it's really just a wrapper around a queue, it's just going to cancel if the thing hasn't been dequeued yet | 17:40 |
| *** itlinux has joined #openstack-nova | 17:40 | |
| dansmith | might not even kill a thread in process, I dunno | 17:40 |
| melwitt | yeah, cancel() returns True/False and for native threads I expect it just always returns False | 17:40 |
| melwitt | right | 17:40 |
| melwitt | yeah, I don't see futurist doing anything different for cancel() other than passing it through to the primitives | 17:42 |
| *** jangutter has quit IRC | 17:42 | |
| dansmith | yeah | 17:43 |
| melwitt | so it would be a matter of what does the Future primitive do for cancel() for the different types of executors | 17:43 |
| melwitt | not really seeing it documented on https://docs.python.org/3/library/concurrent.futures.html#future-objects | 17:43 |
| dansmith | melwitt: just to be clear, when I say the primitives, I'm talking about the threading library not the concurrent stuff | 17:43 |
| melwitt | oh, ok. my bad | 17:43 |
| dansmith | the latter is the high-level implementation, which may very well not be in py2 | 17:43 |
| *** igordc has quit IRC | 17:43 | |
| dansmith | concurrent (and c.futures) just use the low-level primitives in threading, I imagine | 17:44 |
| melwitt | yeah, it's not but it's all included in a separate futures package containing backports https://pypi.org/project/futures/ | 17:44 |
| dansmith | which is why I said what I did | 17:44 |
| dansmith | aye | 17:44 |
| melwitt | yeah, I imagine the same, using same low-level primitives underneath | 17:44 |
| *** igordc has joined #openstack-nova | 17:45 | |
| *** itlinux has quit IRC | 17:46 | |
| *** itlinux has joined #openstack-nova | 17:47 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Run revert resize tests in nova-live-migration https://review.opendev.org/653498 | 17:51 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert "Wait for network-vif-plugged on resize revert" https://review.opendev.org/639396 | 17:51 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert resize: wait for external events in compute manager https://review.opendev.org/644881 | 17:51 |
| openstackgerrit | Merged openstack/os-vif master: Remove IP proxy methods https://review.opendev.org/655695 | 17:51 |
| *** nicolasbock has quit IRC | 17:53 | |
| *** tbachman has joined #openstack-nova | 17:55 | |
| mriedem | the cancel() thing came up in kevin's changes for canceling queued live migrations, | 17:57 |
| mriedem | there is probably something in the compute manager code around that | 17:57 |
| *** itlinux has quit IRC | 17:58 | |
| mriedem | oh right cancel() just returns false if it's already executing and you can't stop it | 17:58 |
| melwitt | yeah, so far I'm not seeing in the futurist source that it event does a green thread kill() when cancel() is called with the GreenThreadPoolExecutor | 17:59 |
| melwitt | *even | 17:59 |
| melwitt | I can't tell that cancel() ever does anything | 17:59 |
| *** igordc has quit IRC | 18:00 | |
| *** itlinux has joined #openstack-nova | 18:00 | |
| *** nicolasbock has joined #openstack-nova | 18:02 | |
| dansmith | ...because it's just a wrapper around a work queue :) | 18:03 |
| *** itlinux has quit IRC | 18:04 | |
| *** igordc has joined #openstack-nova | 18:06 | |
| *** tbachman has quit IRC | 18:07 | |
| melwitt | here's what cancel() does in concurrent.futures https://github.com/python/cpython/blob/d7befad328ad1a6d1f812be2bf154c1cd1e01fbc/Lib/concurrent/futures/_base.py#L352 | 18:17 |
| dansmith | "I'm guessing that this cancel is actually a condition variable" | 18:18 |
| dansmith | some dude said ^ | 18:18 |
| *** lpetrut has joined #openstack-nova | 18:18 | |
| *** tosky has quit IRC | 18:20 | |
| *** lpetrut has quit IRC | 18:23 | |
| melwitt | dansmith: you.... win! | 18:31 |
| melwitt | I like futures, this is fun | 18:36 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells https://review.opendev.org/650172 | 18:40 |
| *** itlinux has joined #openstack-nova | 18:43 | |
| cooper6581 | mriedem: Quick follow up for the { in the password for the transport_url yesterday during our upgrade. I don't know enough if this is a bug or a documentation issue, but I wrote a quick test that reproduces the issue - https://gist.github.com/cooper6581/467f982a0a44494ca32d0c2f755112bc | 18:44 |
| efried | gibi: FYI I copied the berlin onboarding deck and updated it for denver/train https://docs.google.com/presentation/d/1W81DsZGG_bqns2JEdSKwqPCLdAQlR5gTtss-kiGT7Pk/edit?usp=sharing | 18:45 |
| cooper6581 | Specifically, I don't know if we should have ever gotten into a state where transport_url was set, and the config had a non templated url. Maybe it could have been an error on our side during the upgrade v0v | 18:45 |
| melwitt | lyarwood, mriedem: it's alive... http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 (and not tracking tempest-full as nicely as we would hope) | 18:46 |
| mriedem | cooper6581: hmm, i'm not sure, that would require some brain power for sorting out the templated stuff which i don't have loaded up right now, dansmith might know off hand though | 18:47 |
| mriedem | i have to re-learn how the templating code each time i look at it b/c i'm slow | 18:47 |
| mriedem | *code works | 18:47 |
| mriedem | melwitt: that's just 6 hours though | 18:48 |
| melwitt | I'm gonna propose a new change to make the graphs cover more time (make look more like http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1) and add more projects and stable branches | 18:48 |
| mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-30d&to=now | 18:48 |
| mriedem | melwitt: you can toggle that in the dashboard | 18:48 |
| melwitt | oh cool, I didn't know that | 18:48 |
| mriedem | top right | 18:48 |
| melwitt | oh I see it now | 18:48 |
| melwitt | niiiiiice | 18:48 |
| mriedem | looks like the neutron one is default 7 days | 18:48 |
| mriedem | i'm more interested in why the nova graph doesn't go beyond .... oh | 18:49 |
| mriedem | opendev rename | 18:49 |
| mriedem | the graph won't go beyond 4/20 | 18:49 |
| mriedem | when infra smoked a bowl and renamed everything | 18:49 |
| melwitt | yeah, bummer | 18:49 |
| melwitt | haha | 18:49 |
| mriedem | but the 7 day trend seems normalish | 18:50 |
| mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-7d&to=now | 18:50 |
| melwitt | the cinder graph is whack | 18:50 |
| mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=3&from=now-7d&to=now | 18:50 |
| melwitt | yeah 7 day looks reasonable | 18:50 |
| mriedem | yeah... | 18:50 |
| mriedem | not sure what's going on there | 18:50 |
| mriedem | maybe ask eharney if he knows? | 18:50 |
| mriedem | i don't really know why these would be different between the projects | 18:51 |
| dansmith | cooper6581: not sure I understand the test.. "...if the base URLs are set" -- what does that mean? | 18:51 |
| melwitt | yeah, me neither, that's what I'm stuck on | 18:51 |
| *** itlinux has quit IRC | 18:52 | |
| cooper6581 | I just copied the comment from the test test_non_formatted_url_with_no_base :p - I'm assuming what is referred to as base URL ends up being CONF.transport_url (I'm totally keyboard dog right now though) https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145 | 18:53 |
| cooper6581 | this is the test a copied https://github.com/openstack/nova/blob/ca6c32f279cf62915a11b32339cbac8128a8656e/nova/tests/unit/objects/test_cell_mapping.py#L252 | 18:53 |
| dansmith | cooper6581: those assertions are just to handle a case where you have templated a url in the database but don't have anything to parse from the config to format those things.. I'm not sure what that has to do with your case | 18:55 |
| *** itlinux has joined #openstack-nova | 18:58 | |
| *** ivve has joined #openstack-nova | 18:58 | |
| cooper6581 | Ahh, I see. I understand better now. Thanks! | 19:00 |
| melwitt | mriedem: hm, I'm seeing the snapshot tests failing in cinder land http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/testr_results.html.gz which looks like our glance issue we just fixed on stable recently on the surface | 19:02 |
| melwitt | http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_16_18_32_53_335460 | 19:02 |
| melwitt | it's the same thing, but how if it's been fixed on master for a long time | 19:03 |
| melwitt | oh that's stable/rocky | 19:03 |
| melwitt | bad example on my part | 19:04 |
| dansmith | cooper6581: would it be better (than what we have) if we catch ValueError and explicitly log "yo dawg, this looks like a template because it has templating characters in it, but it failed to work with python's format() so you might want to fix it" ? | 19:06 |
| melwitt | I looked at a couple of failures on master and they looked random | 19:06 |
| dansmith | cooper6581: we could also treat a ValueError as "probably not a template", but then someone trying to use templates loses the checking until they realize that nova is trying to connect to rabbit as "worker-{user" | 19:07 |
| *** lpetrut has joined #openstack-nova | 19:14 | |
| *** igordc has quit IRC | 19:17 | |
| *** eharney has quit IRC | 19:25 | |
| *** rchurch_ has quit IRC | 19:28 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.opendev.org/651947 | 19:35 |
| *** itlinux has quit IRC | 19:35 | |
| *** itlinux has joined #openstack-nova | 19:37 | |
| *** rchurch has joined #openstack-nova | 19:41 | |
| *** samueldmq has joined #openstack-nova | 19:44 | |
| *** igordc has joined #openstack-nova | 19:49 | |
| *** boxiang has quit IRC | 19:55 | |
| *** boxiang has joined #openstack-nova | 19:56 | |
| efried | melwitt: I made a grokkable tinyurl and put it on the master. lmk if you hate it. | 19:57 |
| cooper6581 | dansmith: I still need to double check with ccstone, but I think the code is totally fine. I think the problem was when we stepped to newton, that logic wasn't in there, so it got inserted into the DB | 20:01 |
| mriedem | weeee https://bugs.launchpad.net/nova/+bug/1826382 | 20:03 |
| openstack | Launchpad bug 1826382 in OpenStack Compute (nova) "Updates to placement api fail if placement endpoint changes" [Undecided,Triaged] - Assigned to Liam Young (gnuoy) | 20:03 |
| mriedem | we're too smart for our own good | 20:03 |
| mriedem | jaypipes: ^ thoughts on how to handle and auto-detect/correct that? | 20:03 |
| mriedem | tl;dr if don't configure nova to talk to placement with a specific endpoint url, we look it up via the service catalog and ksa; if the endpoint changes and we're using a stale client adapter, all requests fail (http to https) | 20:04 |
| mriedem | you have to restart all the computes to fix it | 20:04 |
| efried | mriedem: There was some attempt to address that via safe_connect | 20:05 |
| mriedem | that probably gets a bit confusing though b/c we raise a nova-specific exception, | 20:06 |
| efried | mriedem: One thing we could do is rebuild the adapter on SIGHUP | 20:06 |
| mriedem | in this case ResourceProviderRetrievalFailed | 20:06 |
| mriedem | heh, i mentioned that in the bug, | 20:06 |
| mriedem | but if sighup were fixed | 20:06 |
| efried | mriedem: Well, there was *one* path through @safe_connect that swallowed the error and rebuilt the adapter and retried. But I don't think it was the right one. | 20:06 |
| mriedem | still, let's say you have 1000 computes and you change the placement endpoint, do you want to sighup 1k computes? maybe that's normal | 20:06 |
| *** mgariepy has quit IRC | 20:07 | |
| mriedem | right that's the EndpointNotFound case | 20:07 |
| efried | yeah, why tf are you changing the endpoint | 20:07 |
| mriedem | ksa doesn't raise an exception here, we just get a 400 response back | 20:07 |
| efried | you deserve to have to sighup 1000 computes | 20:07 |
| efried | whoah | 20:07 |
| mriedem | true, i can't really decide if this is low priority or what | 20:07 |
| *** Sundar has quit IRC | 20:07 | |
| efried | Because the endpoint is there, just no longer accepting http? Is that what you meant by (http to https)? | 20:07 |
| artom | Yeah, I'd have thunk if you change the endpoint you put a redirect in place first | 20:08 |
| *** mchlumsky has quit IRC | 20:08 | |
| mriedem | "In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started." | 20:08 |
| efried | because if the old endpoint is gone gone, that should be a connect failure, which ksa converts. | 20:08 |
| efried | Yeah, that makes... some sense I guess? | 20:08 |
| efried | but yeah, a redirect would be nice | 20:08 |
| *** mvkr has joined #openstack-nova | 20:09 | |
| *** _erlon_ has quit IRC | 20:09 | |
| artom | I don't really think it's reasonable to expect Nova to pull the endpoint list form the catalog for every single request. | 20:09 |
| efried | mriedem: btw, I got ultra screwed on the school runs, won't be back until even later than originally anticipated. | 20:09 |
| efried | artom: ++ | 20:09 |
| artom | What do we do for Neutron and Cinder? | 20:10 |
| mriedem | artom: i don't think anyone is suggesting that | 20:10 |
| efried | mriedem: actually, the fact that we get a 400 from the API is the kicker here. | 20:10 |
| mriedem | right | 20:10 |
| efried | If it were any kind of 500, I would say we might should do something about it. | 20:10 |
| mriedem | we could put a decorator on the get/put/post/delete methods to sniff the response for a 400 and if "Reason: You're speaking plain HTTP to an SSL-enabled server port." is in the text, but that's super hacky | 20:10 |
| efried | but the fact that there's still something at that endpoint, just ain't working anymore, is not our fault. | 20:10 |
| efried | yeah, that's bs | 20:10 |
| jroll | I've never seen an https web server return a response when you try to talk http to it | 20:11 |
| artom | Wait, they switched to HTTPS *on the same port*? | 20:11 |
| artom | Is that a thing? | 20:11 |
| jroll | the client just fails to negotiate a tls connection | 20:11 |
| fungi | i *think* you can do starttls that way, but it's more common to see it on the assigned plaintext port equivalent | 20:11 |
| efried | artom: Suspect portless therefore using defaults (80/443) | 20:11 |
| efried | gotta run | 20:12 |
| fungi | also more common for other protocols than http | 20:12 |
| jroll | huh, looking at the bug and apparently it's a thing, wtf | 20:12 |
| artom | fungi, ah, true | 20:12 |
| fungi | e.g., you can initiate a tls negotiation blind on the smtps port, or on the smtp port you can issue a starttls command (if its advertised on ehlo) and then start tls negotiation | 20:13 |
| fungi | https://en.wikipedia.org/wiki/Opportunistic_TLS | 20:14 |
| fungi | leads to https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header | 20:14 |
| melwitt | efried: tinyurl for what now? | 20:15 |
| fungi | so possible, and has a standard, but... not common in the wild | 20:15 |
| artom | fungi, yeah, though in this case it's almost inverted: the server responds with a 400 because the request *isn't* HTTPS | 20:18 |
| *** eharney has joined #openstack-nova | 20:18 | |
| *** itlinux has quit IRC | 20:18 | |
| * artom has to run (bike, to be exact) as well | 20:18 | |
| fungi | oh, so protocol downgrade? that's generally just considered a security vulnerability | 20:18 |
| *** itlinux has joined #openstack-nova | 20:19 | |
| fungi | ahh, or client trying to make plaintext connection on tls port? i think the phrase for that is "broken configuration" | 20:19 |
| jroll | fungi: right, I'm surprised that apache actually responds to that, rather than the tls negotiation failing | 20:22 |
| *** dave-mccowan has joined #openstack-nova | 20:23 | |
| *** artom has quit IRC | 20:24 | |
| fungi | i'd have come up with a more humorous term, but my comedy circuits are currently overridden by the fact that my mower just kicked up a clam shell, and i'm certain there's a joke in there somewhere if i can just shuck it out | 20:26 |
| mriedem | i see what you did there | 20:26 |
| *** panda is now known as panda|off | 20:30 | |
| *** cooper6581 has quit IRC | 20:33 | |
| *** igordc has quit IRC | 20:36 | |
| *** cooper6581 has joined #openstack-nova | 20:38 | |
| melwitt | mriedem: I was just reading into the ML thread and looking at your results here https://review.opendev.org/654468 and wondered, don't you need LIBS_FROM_GIT=oslo.config to avoid it using the installed version from pypi? I see it checking out the commit you Depends-On in the job output but then see oslo.config installed later? http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_27 | 20:38 |
| melwitt | 0275 | 20:38 |
| melwitt | http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_270275 | 20:38 |
| mriedem | i was under the impression zuulv3 handled that for us | 20:41 |
| mriedem | but maybe that's only if it's in required_projects in zuul.yaml | 20:42 |
| mriedem | pip freeze is showing a tag rather than a git hash though http://logs.openstack.org/68/654468/6/check/openstack-tox-py36/6c5ead4/tox/py36-4.log | 20:43 |
| mriedem | oslo.config==6.8.1 | 20:43 |
| melwitt | yeah that was my concern | 20:44 |
| mriedem | not sure what this is about "cannot set the recursion limit to 10 at the recursion depth 55: the limit is too low" | 20:45 |
| mriedem | the python docs don't say anything about a lower bound | 20:45 |
| *** eharney has quit IRC | 20:46 | |
| melwitt | where are you seeing that? | 20:46 |
| mriedem | http://logs.openstack.org/68/654468/6/check/openstack-tox-py36/6c5ead4/testr_results.html.gz | 20:46 |
| melwitt | oh different job | 20:47 |
| *** pcaruana has quit IRC | 20:48 | |
| mriedem | yeah the bug only hits on py36 for some reason | 20:48 |
| fungi | mriedem: melwitt: yes, a repository will only be checked out alongside if it's in the required-projects list | 20:49 |
| mriedem | ack, ok, that's easily fixable in this hack patch | 20:49 |
| *** ricolin has quit IRC | 20:49 | |
| fungi | otherwise depends-on means "don't merge until this other change merges" | 20:49 |
| fungi | (which also applies to a required project, but an rp will also be tested in conjunction with your change) | 20:50 |
| *** takashin has joined #openstack-nova | 20:50 | |
| melwitt | mriedem: I guess the doc does say "Changed in version 3.5.1: A RecursionError exception is now raised if the new limit is too low at the current recursion depth." https://docs.python.org/3/library/sys.html#sys.setrecursionlimit | 20:51 |
| melwitt | so it has to be at least 55? | 20:51 |
| melwitt | based on the error message it put | 20:51 |
| mriedem | hmm | 20:52 |
| mriedem | b'RecursionError: cannot set the recursion limit to 56 at the recursion depth 55: the limit is too low' | 20:52 |
| *** ttsiouts has joined #openstack-nova | 20:52 | |
| melwitt | lol, what are we supposed to | 20:52 |
| melwitt | do | 20:52 |
| melwitt | I guess that means at that moment, the depth is 55 | 20:53 |
| mriedem | yeah | 20:54 |
| mriedem | i bumped to 100 and then it passed | 20:54 |
| melwitt | so the limit has to be some headroom above that to allow for more | 20:54 |
| melwitt | ok, cool | 20:54 |
| mriedem | it was 1000 in my local py3.6 | 20:54 |
| mriedem | i mean the default or whatever | 20:54 |
| mriedem | which is a bionic node | 20:54 |
| melwitt | ok, I see | 20:54 |
| *** cooper6581 has quit IRC | 20:59 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468 | 20:59 |
| mriedem | fungi: melwitt: ^ hopefully that does it | 20:59 |
| mriedem | nova meeting | 21:00 |
| melwitt | good call excluding all the other jobs too | 21:00 |
| *** ttsiouts_ has joined #openstack-nova | 21:00 | |
| *** gmann is now known as gmann_afk | 21:01 | |
| *** ttsiouts has quit IRC | 21:02 | |
| *** itlinux has quit IRC | 21:03 | |
| *** ivve has quit IRC | 21:11 | |
| *** alex_xu has quit IRC | 21:12 | |
| *** sean-k-mooney has quit IRC | 21:19 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Move legacy-grenade-dsvm-neutron-multinode-live-migration in-tree https://review.opendev.org/640207 | 21:27 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove deprecated 'default_flavor' config option https://review.opendev.org/645476 | 21:28 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.opendev.org/637716 | 21:29 |
| *** itlinux has joined #openstack-nova | 21:31 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Fix nova-grenade-live-migration run book for opendev migration https://review.opendev.org/655800 | 21:31 |
| efried | mriedem: thanks for running that. | 21:38 |
| mriedem | np | 21:38 |
| *** eharney has joined #openstack-nova | 21:44 | |
| *** tbachman has joined #openstack-nova | 21:46 | |
| efried | melwitt: tinyurl pointing to the charts for the project update | 21:47 |
| efried | melwitt: since there's all those links on 'em | 21:47 |
| melwitt | oh, the slides. yeah, that's fine | 21:48 |
| mriedem | dansmith: i think the multi-cell job has yielded it's first victim | 21:49 |
| dansmith | in a good way? | 21:50 |
| mriedem | if latent bugs are a good thing | 21:50 |
| mriedem | apparently in this addFloatingIP code, we fetch the instance from the cell at the top and pass down the targeted context for that instance, | 21:51 |
| mriedem | and if the floating IP is associated with another instance we fetch that https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2390 | 21:51 |
| mriedem | but if that instance is in another cell, bingo bango you've got yourself a 404 mister | 21:51 |
| mriedem | shits the bed right here https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400 | 21:52 |
| *** itlinux has quit IRC | 21:52 | |
| *** JamesBenson has quit IRC | 21:52 | |
| dansmith | mriedem: pretty sure you're over your bingo bango quota for the week | 21:52 |
| mriedem | umm, actually one of the cells was down and now i have no quota limit | 21:53 |
| mriedem | you get to experience that kind of dry wit and more in person next week | 21:53 |
| dansmith | looking forward to the "and more" there | 21:54 |
| * dansmith packs his pepper spray | 21:54 | |
| mriedem | you'll have to pay for the full riderman experience | 21:54 |
| dansmith | so...many...nws...things....to...say | 21:55 |
| *** slaweq has quit IRC | 21:57 | |
| efried | mriedem, melwitt: As soon as I can figure out how to fixture the sdk (see https://storyboard.openstack.org/#!/story/2005475) we could actually do clouds.yaml immediately. | 21:57 |
| efried | Beyond that, yes, the long-term benefit is cutting the python-*client packages out of the picture, removing that tech debt. That relationship is brittle and causes us to either fear changing it or break things when we do. | 21:57 |
| *** ccamacho has joined #openstack-nova | 21:59 | |
| efried | ...immediately. Because the sdk proxies are Adapter subclasses, so we can just swap them seamlessly for the ksa adapter's we're hacking up today. Then cut the actual client libs out at our leisure. | 21:59 |
| mriedem | but then we'll have (1) old style config for cinder (2) normalized ksa config for all the other non-sdk clients we use and (3) sdk clouds.yaml thing | 21:59 |
| efried | deprecation period for oslo.config stuff obviously. | 21:59 |
| mriedem | seems like a configuration mess until everyting is cut over | 21:59 |
| efried | And operators already have clouds.yaml for the most part, so they'll have very little to do. | 22:00 |
| efried | (by "already" I mean, if they're running stein+) | 22:00 |
| mriedem | "That relationship is brittle and causes us to either fear changing it or break things when we do." as the recent ironicclient stuff can tell i suppose, | 22:01 |
| efried | exactly | 22:01 |
| dansmith | efried: which is like nobody right? | 22:01 |
| mriedem | but i imagine we'll hit a new class of weird bugs once we start using the sdk | 22:01 |
| efried | though I didn't want to bring that up | 22:01 |
| efried | because I broke it by introducing the ksa config... | 22:01 |
| mriedem | i'm not sure how much damage we get from glance/neutron/cinder client | 22:01 |
| mriedem | what's being used for nova/cyborg in that series? just ksa? | 22:02 |
| *** imacdonn has quit IRC | 22:02 | |
| mriedem | or the sdk? | 22:02 |
| efried | dansmith: Right, this is setting up for a future time when people are actually using the releases we're working on and we've got new jobs and don't care anymore. | 22:02 |
| efried | I've been pushing Sundar to use sdk | 22:02 |
| *** imacdonn has joined #openstack-nova | 22:02 | |
| efried | I gave him the placement code for the cyborg side of his demo and set it up to use sdk | 22:03 |
| mriedem | so if we did the sdk thing, cyborg integration could be the guinea pig, and then i'd think cinder since we haven't ksa'd our cinder stuff yet | 22:03 |
| efried | I was going to do placement first, because it would be a transparent swap under the covers - we don't have a placementclient, so we would just set up SchedulerReportClient._client as the Connection.placement proxy (which is an Adapter subclass) instead of the raw Adapter. | 22:04 |
| efried | get/put/etc primitives come along for free. | 22:04 |
| efried | I PoC'd this already and it works seamlessly in tempest. It just blows up all the unit and functional tests until I figure out fixtures (see above) | 22:04 |
| mriedem | ok | 22:05 |
| efried | ...and then ironicclient, because we have lots of support for that from the ironic team, and dtantsur|afk is a sdk core who can help to some extent | 22:05 |
| efried | and I have a guy coming on board (hi dustinc) who's trying to be involved in ironic who has started looking at it. | 22:06 |
| efried | but yeah, could look into tackling cinder early, I see the benefit since it got left behind with ksa. | 22:07 |
| efried | never could figure that f'in thing out. | 22:07 |
| *** ttsiouts_ has quit IRC | 22:07 | |
| *** ttsiouts has joined #openstack-nova | 22:07 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| *** ttsiouts has quit IRC | 22:12 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Correct signature on fake_get_availability_zones https://review.opendev.org/655806 | 22:15 |
| *** alex_xu has joined #openstack-nova | 22:19 | |
| mriedem | dansmith: so i'm going to rev this multi-cell job patch to disable the failing test here until there is a bug fix - while doing that, what do you think about just making the job non-voting in the check queue for a bit until we're happy with it through the ptg? | 22:21 |
| dansmith | mriedem: I'm good with whatever, but it seems like non-voting to collect data is what we would normally do | 22:21 |
| *** lpetrut has quit IRC | 22:26 | |
| mriedem | ack | 22:28 |
| mriedem | done | 22:28 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-multi-cell job https://review.opendev.org/655222 | 22:28 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Enable n-novnc in nova-multi-cell job https://review.opendev.org/655711 | 22:28 |
| *** slaweq has quit IRC | 22:37 | |
| *** whoami-rajat has quit IRC | 22:51 | |
| *** cooper6581 has joined #openstack-nova | 22:52 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells https://review.opendev.org/650172 | 22:52 |
| *** cooper6581 has quit IRC | 22:56 | |
| *** luksky has quit IRC | 22:58 | |
| *** tkajinam has joined #openstack-nova | 23:01 | |
| *** rcernin has joined #openstack-nova | 23:03 | |
| efried | o/ see y'all tomorrow | 23:06 |
| *** slaweq has joined #openstack-nova | 23:16 | |
| *** slaweq has quit IRC | 23:24 | |
| *** artom has joined #openstack-nova | 23:39 | |
| artom | 23:41 | |
| rm_work | so, is it possible to run nova unit tests with tox on OSX? anyone manage this? or is it just impossible (seems that pyinotify is not compatible with osx so the env build just fails) | 23:56 |
| *** gyee has quit IRC | 23:57 | |
| *** hongbin has quit IRC | 23:57 | |
| *** cooper6581 has joined #openstack-nova | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!