| *** slaweq has joined #openstack-nova | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** mtreinish has joined #openstack-nova | 00:18 | |
| melwitt | sean-k-mooney: fyi I've proposed a stable/stein release of os-vif. I didn't find one already proposed https://review.opendev.org/681358 | 00:20 |
|---|---|---|
| *** TxGirlGeek has joined #openstack-nova | 00:24 | |
| *** tetsuro has joined #openstack-nova | 00:26 | |
| *** threestrands has joined #openstack-nova | 00:27 | |
| sean-k-mooney | release or branch. there is a a stable branch but i dont think we have done a stabel release in a while | 00:31 |
| sean-k-mooney | melwitt: in either case a release proably should be done as i dont think we have release since we fixed the cve with the linux bridge flooding | 00:33 |
| alex_xu | sean-k-mooney: so we are still good at current upgrade plan for pcpu? I saw there a looong discussion on that | 00:33 |
| sean-k-mooney | alex_xu: am i think dan wants to make the config outhion not required | 00:33 |
| alex_xu | sean-k-mooney: ok, i see now. | 00:34 |
| sean-k-mooney | by trying to claim with PCPUs and if that fails fall back to claiming with VCPUs and rely on the numa toplogy filter to validate it | 00:34 |
| melwitt | sean-k-mooney: release, for stable branch. and correct, we've not had a release of the stable/stein branch since the cve fix | 00:34 |
| sean-k-mooney | melwitt: ya so i proably should have proposed one when that merged. thanks for doing it | 00:35 |
| alex_xu | sean-k-mooney: thanks, that is interesting... | 00:35 |
| alex_xu | how can I know whether I need to stop fallback after upgrade | 00:35 |
| melwitt | sean-k-mooney: np. I just wanted to make sure I didn't duplicate a release proposal in case I hadn't found it in my gerrit search | 00:35 |
| sean-k-mooney | i think the release are ment to be on a topic branch with the project name but i dont see one looking quickly either | 00:37 |
| sean-k-mooney | ya looking at all the open patches agains the release repo yours is the only os-vif one i see | 00:38 |
| sean-k-mooney | alex_xu: basicailly how it would work is it would ould fallback if the first placmeent query with PCPUs failed then it would try with VCPUs | 00:39 |
| sean-k-mooney | the numa toplogy filter could tell if the compute node was an train node or a pre train node based on the field in the host numa toplogy object | 00:40 |
| sean-k-mooney | so on train nodes all either there ill be no pinned cpus or the will all be used | 00:40 |
| melwitt | sean-k-mooney: lol, looks like my beloved auto topic from git-review removed my good branch name | 00:40 |
| sean-k-mooney | for a stine node we would usee the old logic | 00:40 |
| sean-k-mooney | melwitt: i think that is why it was removed | 00:41 |
| melwitt | really??? | 00:41 |
| melwitt | :P | 00:41 |
| melwitt | I still like it most of the time | 00:41 |
| sean-k-mooney | i think. that if you name you r branches bug/whatever and do git review it will keep it | 00:41 |
| *** eharney has quit IRC | 00:42 | |
| sean-k-mooney | at least i have still been relying on haveing the correct local brnach name | 00:42 |
| alex_xu | sean-k-mooney: ah... | 00:42 |
| *** gyee has quit IRC | 00:42 | |
| sean-k-mooney | alex_xu: im still not 100% sure there are no issue with that but it sound like it might work | 00:42 |
| sean-k-mooney | i think stephen is going to try and code it up tommorow/today wehn he is online in 8 hours | 00:43 |
| alex_xu | yea, I'm trying to build up how that works in my mind also | 00:43 |
| alex_xu | anyway I need moving to office, then continue to look at, see if anything I can help | 00:44 |
| alex_xu | sean-k-mooney: thanks | 00:44 |
| sean-k-mooney | i think it should work in theroy. my concern would be the use of the limit parmater on the second query but in principal i dont imedialy see why it would not work | 00:45 |
| sean-k-mooney | i had assumed somthing like a fallback mechaniums woudl be deamed too complicated. but i guess its for just one cycle | 00:46 |
| sean-k-mooney | well proably more with our tenency not to delete things right away | 00:47 |
| *** TxGirlGeek has quit IRC | 00:48 | |
| alex_xu | yea, but that fallback should be existing in the whole U release | 00:49 |
| *** tetsuro has quit IRC | 00:54 | |
| openstackgerrit | weibin proposed openstack/nova master: Add support for using ceph RBD ereasure code https://review.opendev.org/681188 | 00:56 |
| *** tetsuro has joined #openstack-nova | 01:02 | |
| sean-k-mooney | alex_xu: ya we might have to wait till V to drop it but in V it could be removed | 01:04 |
| sean-k-mooney | we want to support people moving from stine to train without needed to modify anything but train to ussuri was when we were thingin of chaning the default to in the config and removing the confiv in V | 01:06 |
| sean-k-mooney | in anycase before you upgrade form U to V we want people to have moved to useing the new way of tracking | 01:06 |
| *** mriedem has joined #openstack-nova | 01:06 | |
| sean-k-mooney | mriedem: the numa job passed by the way https://review.opendev.org/#/c/680739/ result are here https://zuul.opendev.org/t/openstack/build/2ad3676591e440b6a76d30e966833a49 | 01:10 |
| mriedem | 2am in ireland, go to bed | 01:12 |
| mriedem | but thanks :) | 01:12 |
| sean-k-mooney | i will soon im just finishing playing a game to unwind. irc is on a different monitor | 01:14 |
| *** tetsuro has quit IRC | 01:15 | |
| *** Tianhao_Hu has joined #openstack-nova | 01:20 | |
| *** Tianhao_Hu has quit IRC | 01:20 | |
| alex_xu | ls | 01:23 |
| alex_xu | oops | 01:23 |
| *** spatel has joined #openstack-nova | 01:24 | |
| *** liuyulong has joined #openstack-nova | 01:25 | |
| *** tkajinam has quit IRC | 01:25 | |
| *** lbragstad has quit IRC | 01:26 | |
| *** tkajinam has joined #openstack-nova | 01:26 | |
| *** lbragstad has joined #openstack-nova | 01:26 | |
| *** tetsuro has joined #openstack-nova | 01:28 | |
| *** Tianhao_Hu has joined #openstack-nova | 01:34 | |
| *** Tianhao_Hu has left #openstack-nova | 01:34 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 01:35 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 01:35 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 01:35 |
| *** tetsuro has quit IRC | 01:40 | |
| *** tetsuro has joined #openstack-nova | 01:40 | |
| *** nicolasbock has quit IRC | 01:44 | |
| *** nicolasbock has joined #openstack-nova | 01:45 | |
| *** liuyulong has quit IRC | 01:55 | |
| *** nicolasbock has quit IRC | 01:58 | |
| brinzhang | mriedem:https://review.opendev.org/#/c/679413/4/nova/objects/migration.py@172 | 02:02 |
| brinzhang | in self._context are not contain user_id and project_id | 02:02 |
| brinzhang | mriedem:IMO, when the operator post the request, the user_id (self._context._user_id) need to record, isn't it? | 02:04 |
| *** slaweq has joined #openstack-nova | 02:11 | |
| *** FlorianFa has quit IRC | 02:15 | |
| *** slaweq has quit IRC | 02:15 | |
| *** FlorianFa has joined #openstack-nova | 02:35 | |
| *** tetsuro has quit IRC | 02:47 | |
| *** mriedem has quit IRC | 02:48 | |
| *** tetsuro has joined #openstack-nova | 03:00 | |
| *** spatel has quit IRC | 03:03 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add operator user_id/project_id to the migrations https://review.opendev.org/679413 | 03:04 |
| *** tetsuro has quit IRC | 03:05 | |
| openstackgerrit | Merged openstack/nova master: Fixing broken links https://review.opendev.org/681206 | 03:09 |
| *** ccamacho has quit IRC | 03:11 | |
| *** PrinzElvis has quit IRC | 03:39 | |
| *** larainema has joined #openstack-nova | 03:39 | |
| *** knikolla has quit IRC | 03:42 | |
| *** ildikov has quit IRC | 03:42 | |
| *** csatari has quit IRC | 03:42 | |
| *** pas-ha has quit IRC | 03:44 | |
| *** ildikov has joined #openstack-nova | 03:44 | |
| *** knikolla has joined #openstack-nova | 03:45 | |
| *** csatari has joined #openstack-nova | 03:45 | |
| *** pas-ha has joined #openstack-nova | 03:45 | |
| *** samc-bbc has quit IRC | 03:45 | |
| *** PrinzElvis has joined #openstack-nova | 03:45 | |
| *** samc-bbc has joined #openstack-nova | 03:47 | |
| *** udesale has joined #openstack-nova | 04:00 | |
| *** etp has joined #openstack-nova | 04:12 | |
| *** ash2307 has quit IRC | 04:25 | |
| *** ash2307 has joined #openstack-nova | 04:26 | |
| *** dave-mccowan has quit IRC | 04:29 | |
| *** pcaruana has joined #openstack-nova | 04:42 | |
| openstackgerrit | Merged openstack/nova master: Fix the race in confirm resize func test https://review.opendev.org/681238 | 04:49 |
| *** brault has joined #openstack-nova | 05:00 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: Counting both of VCPU and PCPU as core quota https://review.opendev.org/681374 | 05:02 |
| alex_xu | stephenfin: efried sean-k-mooney ^ hope this is what you need | 05:02 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Counting both of VCPU and PCPU as core quota https://review.opendev.org/681374 | 05:04 |
| *** brault has quit IRC | 05:05 | |
| *** ratailor has joined #openstack-nova | 05:06 | |
| *** Luzi has joined #openstack-nova | 05:07 | |
| *** slaweq has joined #openstack-nova | 05:11 | |
| *** pcaruana has quit IRC | 05:12 | |
| *** slaweq has quit IRC | 05:16 | |
| *** ash2307 has quit IRC | 05:18 | |
| *** ash2307 has joined #openstack-nova | 05:20 | |
| *** rcernin has quit IRC | 05:22 | |
| *** redrobot has quit IRC | 05:25 | |
| *** damien_r has joined #openstack-nova | 05:31 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: Counting both of VCPU and PCPU as core quota https://review.opendev.org/681374 | 05:34 |
| *** damien_r has quit IRC | 05:36 | |
| *** rcernin has joined #openstack-nova | 05:38 | |
| alex_xu | stephenfin: https://review.opendev.org/#/c/681374/ according this http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-10.log.html#t2019-09-10T20:04:42, in case you don't know | 05:43 |
| *** TxGirlGeek has joined #openstack-nova | 05:45 | |
| *** brinzhang_ has joined #openstack-nova | 05:50 | |
| *** rcernin has quit IRC | 05:51 | |
| *** brinzhang has quit IRC | 05:54 | |
| *** rpittau|afk is now known as rpittau | 06:01 | |
| *** TxGirlGeek has quit IRC | 06:03 | |
| *** Tianhao_Hu has joined #openstack-nova | 06:05 | |
| *** takashin has joined #openstack-nova | 06:05 | |
| *** rcernin has joined #openstack-nova | 06:09 | |
| *** slaweq has joined #openstack-nova | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** ash2307 has quit IRC | 06:18 | |
| *** luksky has joined #openstack-nova | 06:19 | |
| *** slaweq has joined #openstack-nova | 06:21 | |
| *** pcaruana has joined #openstack-nova | 06:21 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: DNM: Try to fallback to PCPU request when VCPU failed https://review.opendev.org/681383 | 06:25 |
| *** tetsuro has joined #openstack-nova | 06:26 | |
| *** etp has quit IRC | 06:32 | |
| *** etp has joined #openstack-nova | 06:33 | |
| *** ash2307 has joined #openstack-nova | 06:35 | |
| *** ricolin has joined #openstack-nova | 06:38 | |
| *** ricolin has quit IRC | 06:39 | |
| *** ociuhandu has joined #openstack-nova | 06:42 | |
| *** tetsuro_ has joined #openstack-nova | 06:46 | |
| *** damien_r has joined #openstack-nova | 06:46 | |
| *** ociuhandu has quit IRC | 06:46 | |
| *** tetsuro has quit IRC | 06:48 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:51 | |
| *** lpetrut has joined #openstack-nova | 06:52 | |
| *** brault has joined #openstack-nova | 06:57 | |
| *** ociuhandu has joined #openstack-nova | 07:04 | |
| *** ociuhandu has quit IRC | 07:04 | |
| *** ociuhandu has joined #openstack-nova | 07:07 | |
| *** awalende has joined #openstack-nova | 07:08 | |
| *** ociuhandu has quit IRC | 07:08 | |
| *** trident has quit IRC | 07:08 | |
| *** ivve has joined #openstack-nova | 07:09 | |
| *** tetsuro_ has quit IRC | 07:09 | |
| *** luksky has quit IRC | 07:10 | |
| *** tesseract has joined #openstack-nova | 07:11 | |
| *** trident has joined #openstack-nova | 07:17 | |
| *** ralonsoh has joined #openstack-nova | 07:19 | |
| *** tetsuro has joined #openstack-nova | 07:21 | |
| *** trident has quit IRC | 07:22 | |
| *** luksky has joined #openstack-nova | 07:23 | |
| *** ociuhandu has joined #openstack-nova | 07:24 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: DNM: Try to fallback to PCPU request when VCPU failed https://review.opendev.org/681383 | 07:29 |
| *** trident has joined #openstack-nova | 07:31 | |
| alex_xu | stephenfin: efried sean-k-mooney ^ also tested the dansmith idea | 07:31 |
| *** ccamacho has joined #openstack-nova | 07:38 | |
| *** rcernin has quit IRC | 07:38 | |
| *** tetsuro has quit IRC | 07:45 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _revert_allocation nested allocation aware https://review.opendev.org/676138 | 07:47 |
| gibi | bauzas: I needed to recheck anyhow and later part of the series needs a rebase to pick up the race fix. Could you re +W ^^ | 07:49 |
| gibi | ? | 07:49 |
| bauzas | gibi: sure thing | 07:49 |
| gibi | thanks | 07:49 |
| bauzas | and morning | 07:49 |
| gibi | bauzas: yeah, good morning | 07:49 |
| bauzas | gibi: done | 07:50 |
| alex_xu | you guys wakeup early today, or you guy just quiet before~ | 07:51 |
| bauzas | nah it's 9:51am | 07:51 |
| bauzas | but in general I do a lot of internal stuff on mornings :( | 07:51 |
| *** luksky has quit IRC | 07:51 | |
| alex_xu | i see now~ | 07:51 |
| gibi | alex_xu: I wake up earlier today but not due to work. | 07:52 |
| alex_xu | nice | 07:53 |
| gibi | I hit the gym early as I have to flight to Ireland today (internal meeting) | 07:53 |
| gibi | what a better time to have an internal conf than on FF day :) | 07:54 |
| alex_xu | hah | 07:54 |
| alex_xu | or review on the flight :) | 07:54 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | 07:55 |
| gibi | yeah, I'm planning to work whole day regardless of the flights | 07:55 |
| gibi | bauzas: rebase on https://review.opendev.org/676140 as well | 07:55 |
| gibi | bauzas: could you please re+w? | 07:56 |
| bauzas | gibi: done | 07:56 |
| gibi | thanks a lot | 07:56 |
| gibi | bauzas: the next in the series will change to avoid re-introducing the race. So that will take a bit of time to propose | 07:56 |
| *** luksky has joined #openstack-nova | 07:57 | |
| bauzas | gibi: ok ping me then when you're done | 07:59 |
| *** threestrands has quit IRC | 07:59 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 08:01 |
| gibi | bauzas: sure | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 08:01 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 08:01 |
| *** tssurya has joined #openstack-nova | 08:02 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 08:05 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 08:05 |
| efried | o/ nova | 08:09 |
| efried | alex_xu, luyao: Is that ---^ ready for another look? | 08:09 |
| alex_xu | efried: good morning...so early | 08:10 |
| efried | alex_xu: I thought it might be a good idea for me to get up early and overlap more with you & stephenfin | 08:10 |
| alex_xu | efried: hah | 08:10 |
| efried | just today (and maybe tomorrow), not permanently. | 08:10 |
| efried | but yeah, it's closer to "all-nighter with a nap" than "get up early" :P | 08:11 |
| alex_xu | efried: no response from luyao, I guess she just get a break after submit patch :) | 08:12 |
| efried | alex_xu: I saw you worked the quota solution - thank you for that. | 08:12 |
| efried | how's the CI thing coming? Does Rui have it set to pull latest yet? | 08:12 |
| alex_xu | efried: np, one more interesting thing https://review.opendev.org/681383 | 08:12 |
| *** ociuhandu has quit IRC | 08:12 | |
| alex_xu | efried: no, I talk with Dolpher that depends on CI team, they are in US | 08:13 |
| alex_xu | efried: we will know after few hours i guess | 08:13 |
| efried | argh! If only we had known, we were looking at this yesterday during the day. | 08:13 |
| *** ratailor_ has joined #openstack-nova | 08:14 | |
| *** ociuhandu has joined #openstack-nova | 08:15 | |
| efried | alex_xu: Did https://review.opendev.org/#/c/678455/27 wind up sane? | 08:15 |
| alex_xu | efried: let me check now | 08:16 |
| efried | If so, perhaps we can ask gibi/bauzas to have a look at it and give the other +2 (expecting the first from stephenfin) | 08:16 |
| *** ratailor has quit IRC | 08:16 | |
| bauzas | efried: which one ? | 08:16 |
| efried | bauzas: That one piece of vpmem we talked about yesterday - the part that does the libvirt dom xml business. | 08:17 |
| efried | link above | 08:17 |
| efried | bauzas: but maybe let alex_xu vet it first to make sure it's worth your time. | 08:17 |
| bauzas | efried: okay, honestly, it's prio #3 for me | 08:17 |
| efried | that should work out nicely :) | 08:17 |
| bauzas | but I can try to look while I wait for stephenfin and gibi | 08:17 |
| openstackgerrit | Alexandra Settle proposed openstack/nova stable/stein: Fixing broken links https://review.opendev.org/681401 | 08:17 |
| alex_xu | efried: bauzas I guess the only one I want to check is this https://review.opendev.org/#/c/678455/27/nova/virt/hardware.py@2065, I don't know why she can't just assign a cpu_policy and cpu_thread_plicy directly instead of using the flavor and image_metadata | 08:18 |
| *** derekh has joined #openstack-nova | 08:20 | |
| *** luksky has quit IRC | 08:21 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Filter migrations by user_id/project_id https://review.opendev.org/674243 | 08:21 |
| *** tssurya has quit IRC | 08:22 | |
| luyao | efried: code ready to be reviewed, welcome :) | 08:23 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Filter migrations by user_id/project_id https://review.opendev.org/674243 | 08:23 |
| efried | luyao: on it now | 08:23 |
| alex_xu | luyao: only thing I have https://review.opendev.org/#/c/678455/27/nova/virt/hardware.py@2072 | 08:25 |
| luyao | alex_xu: Okay, I'll test it first. | 08:26 |
| *** tkajinam has quit IRC | 08:27 | |
| alex_xu | luyao: thanks a lot | 08:27 |
| *** jawad_axd has joined #openstack-nova | 08:29 | |
| stephenfin | alex_xu: Nicely done :) Looking | 08:29 |
| alex_xu | stephenfin: \o/ | 08:29 |
| *** takashin has left #openstack-nova | 08:30 | |
| luyao | alex_xu: works well | 08:30 |
| alex_xu | luyao: \o/ | 08:31 |
| openstackgerrit | Merged openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | 08:34 |
| openstackgerrit | Merged openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158 | 08:34 |
| openstackgerrit | Merged openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 08:34 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | 08:34 |
| alex_xu | stephenfin: https://review.opendev.org/#/c/671800/30/nova/virt/hardware.py@2018, I'm not sure we should take crea of numa_usages_from_instance_numa in ResoourceTracker, since the RT only process the instance below to this host, if this host is upgraded, then it will host_cell.pcpuset. If this host is old, then it doesn't know about any pcpuset. But I can help you to test that | 08:35 |
| *** tssurya has joined #openstack-nova | 08:36 | |
| *** avolkov has joined #openstack-nova | 08:37 | |
| gibi | bauzas: the next patch is up: https://review.opendev.org/676972 | 08:37 |
| bauzas | gibi: ack, looking | 08:37 |
| gibi | thanks | 08:37 |
| * stephenfin looks | 08:49 | |
| stephenfin | alex_xu: Hmm, that's a good point actually | 08:50 |
| stephenfin | I keep forgetting the resource tracker runs on the host | 08:50 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | 08:53 |
| alex_xu | stephenfin: yea, we only take care of host_cell, not the instance cell, it should be ok for any migration also | 08:53 |
| aspiers | stephenfin: I just removed SEV from the runway since it merged \o/ | 08:53 |
| aspiers | thanks everyone for all your help | 08:53 |
| *** priteau has joined #openstack-nova | 08:54 | |
| alex_xu | aspiers: congrats, destory the laptop and begin to vacation | 08:54 |
| aspiers | alex_xu: exactly! X-D ;-) | 08:54 |
| aspiers | I'm taking vacation next week | 08:55 |
| alex_xu | aspiers: enjoy :) | 08:55 |
| aspiers | thanks :) | 08:55 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 08:55 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | 08:55 |
| *** ttsiouts has joined #openstack-nova | 09:00 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | 09:00 |
| *** shilpasd has joined #openstack-nova | 09:01 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extract pf$N literals as constants from func test https://review.opendev.org/680991 | 09:03 |
| stephenfin | aspiers: \o/ | 09:04 |
| stephenfin | efried: You're mad, btw | 09:04 |
| stephenfin | (the getting up early thing :D) | 09:04 |
| *** lpetrut has quit IRC | 09:04 | |
| * gibi going to the airport, back online in ~45 mins | 09:05 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve dest service level func tests https://review.opendev.org/680998 | 09:05 |
| efried | fly safe gibi | 09:05 |
| efried | \o/ aspiers! | 09:06 |
| alex_xu | stephenfin: sean-k-mooney efried found an intersting case for this https://review.opendev.org/#/c/681383/2 | 09:07 |
| alex_xu | but I guess in the end, it will become why choice fallback to PCPU or fallback to VCPU. | 09:08 |
| aspiers | efried: \o/ indeed, can't thank you enough for all your help! it's been a long road (basically a whole year IIRC) | 09:08 |
| stephenfin | alex_xu: I just left comments on that | 09:08 |
| stephenfin | tl;dr: I don't think we should be doing the service version check | 09:09 |
| efried | aspiers: sadly, your experience was very smooth and quick compared to most of a similar size. | 09:09 |
| aspiers | haha wow really :) | 09:09 |
| efried | ask anyone | 09:09 |
| stephenfin | aspiers: yup, he's not lying | 09:09 |
| aspiers | I honestly didn't think it would be anywhere near as hard to complete, or as long | 09:09 |
| aspiers | classic case of engineering underestimation | 09:10 |
| alex_xu | stephenfin: except we define a rule, the operator must use cpu_dedicated_set for pinning host after upgrade to T | 09:10 |
| stephenfin | cpu-resources is on its...fourth cycle? | 09:10 |
| efried | it really shouldn't be, aspiers | 09:10 |
| aspiers | stephenfin: :-o | 09:10 |
| efried | ...as hard/long to complete | 09:10 |
| *** rcernin has joined #openstack-nova | 09:10 | |
| stephenfin | alex_xu: I've been told we can't. We'll break upgrades. You can't require nova.conf changes simply to upgrade to a new major version | 09:10 |
| aspiers | efried: what are the lessons learnt? other than we should all chip in to help fix CI more :) | 09:11 |
| efried | aspiers: We keep learning the same lessons over and over again but not reacting | 09:11 |
| stephenfin | aspiers: bug people and be as responsive as possible, pretty much | 09:11 |
| alex_xu | stephenfin: emm...I guess because we have no way to force that, the upgrade-check can't check all the hosts' configuration file | 09:12 |
| aspiers | stephenfin: pretty sure I did a lot of bugging - sorry about that ;-) | 09:12 |
| stephenfin | alex_xu: Indeed. Tbh, I don't think it's an issue anyway | 09:13 |
| efried | aspiers: I see the main problems being 1) we commit to too much stuff per release, and 2) we *somehow* need more consistent core engagement throughout the cycle (and it would be nice if less "bugging" was required to achieve that) | 09:13 |
| stephenfin | The NUMATopologyFilter will save us | 09:13 |
| alex_xu | stephenfin: what do you mean "you can't require nova.conf changes simply to ugprade to a new major version"? | 09:14 |
| efried | fwiw my plan for ussuri is to be strict about 1. This was the first release where I had an employer with real business commitments based on the assumption that "approved spec" actually means a thing is expected to get done. (And that's the cycle I had to run for PTL, shoot-self-in-foot) | 09:14 |
| aspiers | efried: yeah. It feels like more resources are required for reviewing in general and the cores are being pulled in too many directions at once | 09:15 |
| efried | ...and therefore the first cycle I really felt the pain of stuff slipping | 09:15 |
| aspiers | ouch | 09:15 |
| stephenfin | alex_xu: I mean if you have Stein-based deployment that's not emitting deprecation warnings, you should be able to upgrade it to Train without changing nova.conf at all and it'll just work | 09:15 |
| aspiers | I can relate - I felt a lot of pressure to land SEV in Train (original plan with AMD was Stein), although it turns out a lot of that pressure was just from me | 09:16 |
| stephenfin | It might start issuing new warnings saying "you will need to do X before you upgrade to U" but that's it | 09:16 |
| kashyap | aspiers: Nice on SEV completion, that's one fo the first things I was looking if it was all merged this morning. | 09:16 |
| stephenfin | efried: Right? ^ | 09:16 |
| aspiers | kashyap: thanks a lot for all your help! | 09:16 |
| efried | stephenfin: that is my understanding, yes | 09:17 |
| stephenfin | Sweet | 09:17 |
| alex_xu | stephenfin: emm...right, we have rule about the configuration file should works with same as before | 09:17 |
| stephenfin | alex_xu: So yeah, tl;dr: we can insist '[compute] cpu_dedicated_set' is configured before U | 09:18 |
| stephenfin | and add an upgrade check to ensure that | 09:18 |
| stephenfin | but not T | 09:18 |
| kashyap | aspiers: NP; glad to be useful. Now I can bug you for Secure Boot stuff -- turning out to be more work than I anticipated :/ | 09:18 |
| stephenfin | because we gave them no warning and no migration path in Stein | 09:18 |
| stephenfin | alex_xu: However, yeah, I don't think this is in an issue. Consider this | 09:18 |
| aspiers | kashyap: bug soon as I'm on vacation next week ;-) | 09:19 |
| stephenfin | Stein-based compute node: reports VCPU, uses NUMATopology.cpuset to determine what cores can be pinned to (with the NUMATopologyFilter simply copying 'cpuset' to 'pcpuset' to keep that passing) | 09:20 |
| kashyap | aspiers: Heh, noted; well-deserved | 09:20 |
| stephenfin | Train-based compute node without new config options: reports VCPU, uses NUMATopology.pcpuset (p!) to determine cores can be pinned to but this == NUMATopology.cpuset (so no need for NUMATopologyFilter to copy 'cpuset' to 'pcpuset') | 09:21 |
| alex_xu | yup | 09:21 |
| stephenfin | Train-base compute node with new config options: reports PCPU, uses NUMATopology.pcpuset (p!) to determine cores can be pinned to and this is unique (again, no need for NUMATopologyFilter to copy 'cpuset' to 'pcpuset') | 09:22 |
| stephenfin | sooo | 09:22 |
| stephenfin | from a placement perspective, things have changed | 09:22 |
| luyao | efried: do you need a unittest for this? https://review.opendev.org/#/c/678450/11/nova/compute/manager.py@2224 | 09:23 |
| stephenfin | but from a NUMATopologyFilter (nova-scheduler) and nova-compute perspective, all three will work perfectly fine together | 09:23 |
| alex_xu | yup, agree | 09:23 |
| efried | luyao: was there a unit test for the exception case before? If so, it should be retained. If not, it wouldn't hurt to add one, but I'm not sure it's necessary. stephenfin, what do you think? | 09:24 |
| luyao | efried: there was one before | 09:24 |
| stephenfin | alex_xu: the _only_ thing that can hurt us is Train-based compute nodes that have no CPU configuration or have new config options but only for shared CPUs ('[compute] cpu_shared_set') | 09:24 |
| efried | luyao: okay, so if it needs to be adjusted based on the code having been moved, do that, but don't remove the unit test. | 09:25 |
| stephenfin | In that case, the second request to placement will return these hosts since they're reporting VCPU. However, they're _really_ reporting VCPU and don't have any PCPU inventory | 09:25 |
| alex_xu | stephenfin: why | 09:25 |
| stephenfin | but, like I said, the NUMATopologyFilter will protect us from that | 09:25 |
| stephenfin | since NUMATopology.pcpuset will be set to None and therefore it will fail to pin | 09:26 |
| *** zhubx has joined #openstack-nova | 09:26 | |
| luyao | efried: okay | 09:26 |
| alex_xu | stephenfin: ok | 09:26 |
| stephenfin | and obviously if NUMATopologyFilter is disabled for some reason, the instance will fail to schedule when it lands on the host | 09:26 |
| alex_xu | stephenfin: that make senses I think | 09:27 |
| alex_xu | stephenfin: have you see the resize case I wrote at https://review.opendev.org/#/c/681383/2 | 09:27 |
| stephenfin | alex_xu: Phew, glad you understood. It's really hard to put this stuff into words without waffling or going down ratholes :-D | 09:27 |
| stephenfin | Looking | 09:27 |
| alex_xu | stephenfin: hah | 09:27 |
| stephenfin | alex_xu: Could we call that out as a known issue? | 09:28 |
| *** boxiang has quit IRC | 09:29 | |
| alex_xu | stephenfin: maybe, I'm thinking whether we request PCPU, and fallback to VCPU, can make that better. | 09:29 |
| alex_xu | but I feel tired, my brain works very slow... | 09:29 |
| *** dtantsur|afk is now known as dtantsur | 09:30 | |
| stephenfin | Yeah, I guess we'd need to implement the retry logic at a higher level | 09:30 |
| alex_xu | stephenfin: ah, I guess that doesn't work. The problem is placement always can get you same host allocation candidates, but the request is reject by the scheduler filters | 09:30 |
| stephenfin | Yup. So if we wanted to solve that, we'd have to wrap both the call to placement and the call to the scheduler filters | 09:31 |
| stephenfin | and the attempt to schedule to the guest too, in case the user doesn't have NUMATopologyFilter enabled | 09:31 |
| stephenfin | that sounds like a lot of effort for corner case that will only happen while an operator is in the process of upgrading their configuration | 09:31 |
| *** ratailor_ has quit IRC | 09:32 | |
| *** ratailor_ has joined #openstack-nova | 09:32 | |
| alex_xu | I feel I just tired | 09:33 |
| stephenfin | alex_xu: If it gets too late for you, I can take over https://review.opendev.org/#/c/681383/ | 09:33 |
| alex_xu | stephenfin: sorry, I need to take a rest little bit. you can free to take over those two patches. I probably online after one or two hours. | 09:34 |
| stephenfin | Sounds good (y) | 09:34 |
| efried | luyao: btw, here's what I did to that provider tree unit test when I was messing around with it locally: http://paste.openstack.org/show/775098/ | 09:35 |
| efried | that last assertTrue fails with copy.copy but succeeds with deepcopy. | 09:35 |
| luyao | efried: get it, thanks. | 09:38 |
| efried | luyao: note that https://review.opendev.org/#/c/678453/26 is in merge conflict (probably aspiers' fault :P ) so you'll need to rebase again. | 09:39 |
| luyao | efried: haha :D | 09:40 |
| *** rcernin has quit IRC | 09:41 | |
| *** ociuhandu has quit IRC | 09:43 | |
| *** brinzhang_ has quit IRC | 09:45 | |
| *** brinzhang_ has joined #openstack-nova | 09:46 | |
| *** brinzhang_ has quit IRC | 09:46 | |
| luyao | efried: I check the unittests for get_allocations, it tests the function that we move the get_allocations_xx out of | 09:49 |
| *** ociuhandu has joined #openstack-nova | 09:49 | |
| luyao | efried: so I have to remove it | 09:49 |
| efried | luyao: then I say just leave that part of the patch as is | 09:50 |
| efried | luyao: stephenfin and I have already reviewed it and accepted it this way, so it's not worth the trouble of changing. | 09:50 |
| *** Tianhao_Hu has quit IRC | 09:52 | |
| luyao | efried: Okay | 09:52 |
| *** ociuhandu has quit IRC | 09:54 | |
| luyao | efried: sorry, confirm again, need try/except or drop? | 09:54 |
| efried | luyao: Sorry, this has gotten confusing. | 09:56 |
| efried | Remove the try/except. I.e. keep this part of the patch as it appears in PS22 https://review.opendev.org/#/c/678450/22/nova/compute/manager.py@2215 | 09:56 |
| luyao | efried: Okay, thanks. | 09:57 |
| efried | luyao: Okay, I've finished re-reviewing the series (except for https://review.opendev.org/#/c/678455/, which I'm still not going to touch) | 09:58 |
| efried | stephenfin: Do you need my re-look at any of the cpu-resources patches at this point? | 09:59 |
| stephenfin | efried: In about a half hour, yeah | 09:59 |
| stephenfin | I think the quota problem is solved | 09:59 |
| efried | sweet | 09:59 |
| stephenfin | So it's just replacing the scheduler conf option with retry logic for placement left | 09:59 |
| stephenfin | almost done with tests | 10:00 |
| *** cdent has joined #openstack-nova | 10:02 | |
| efried | luyao: note response on the defaultdict thing -- I still don't think you need it https://review.opendev.org/#/c/678449/21/nova/compute/provider_tree.py@70 | 10:03 |
| luyao | efried: you're right. :) | 10:06 |
| efried | phew. I like being right | 10:07 |
| bauzas | stephenfin: I wonder something | 10:08 |
| bauzas | stephenfin: if we ask for an o.vo field not existing, can we just do something like 'if field not in my_object:' ? | 10:08 |
| bauzas | IIRC, we need to use a specific o.vo method | 10:09 |
| bauzas | unless the field is set to None | 10:09 |
| stephenfin | bauzas: You can do that, yeah | 10:09 |
| stephenfin | For some time now | 10:09 |
| stephenfin | you used to need to use is_attr_set or something like that | 10:09 |
| bauzas | ok, I had concerns about the virt.hardware module in https://review.opendev.org/#/c/671800/ | 10:09 |
| stephenfin | but __contains__ was implemented many moons ago | 10:09 |
| stephenfin | anywhere is particular? | 10:09 |
| *** tssurya has quit IRC | 10:10 | |
| *** tbachman has quit IRC | 10:12 | |
| bauzas | nah it's okay then | 10:17 |
| bauzas | my o.vo skills are a bit rusty | 10:18 |
| bauzas | but I could have tested it | 10:18 |
| sean-k-mooney | efried: after 6 year of working on nova the main lesson i have learns is unlesss you have 2 cores from different compaines lined up to review the think you want to do before you start it will take 2 cycles to complete | 10:19 |
| sean-k-mooney | the first cycle you get it working and people beging to become familar with and the second you finall land the thing | 10:19 |
| *** brinzhang has joined #openstack-nova | 10:19 | |
| efried | sean-k-mooney: I would love to be able to commit cores to review blueprints before/as they are approved. I just a) doubt cores would be willing to commit, and b) don't want to do that much PM type work | 10:20 |
| sean-k-mooney | its partly a social thing. you need to explain why this thing you care about is imporant so that people will be interested enogh to review and demonstrate it works and is sane. | 10:21 |
| sean-k-mooney | the non craze telco things i have worked on are less hard to enabel but im normlally in craze telco land | 10:22 |
| efried | It sucks that "be interested" is a criterion | 10:22 |
| *** ttsiouts has quit IRC | 10:23 | |
| sean-k-mooney | well it only is in terms of making it higher on list of other things you have to review | 10:23 |
| *** ttsiouts has joined #openstack-nova | 10:23 | |
| sean-k-mooney | there are only a limit number of sme fore each system and they are overloaded frequently | 10:24 |
| *** maciejjozefczyk has quit IRC | 10:24 | |
| *** maciejjozefczyk has joined #openstack-nova | 10:25 | |
| sean-k-mooney | if we had more subsystem maintainer or more reviewres it would help but we work with what we got. | 10:25 |
| sean-k-mooney | we still get a lot done each cycle | 10:25 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: DNM: Try to fallback to PCPU request when VCPU failed https://review.opendev.org/681383 | 10:25 |
| sean-k-mooney | the latency can just be a bit high sometimes | 10:25 |
| stephenfin | efried, bauzas: Can you sanity check before I waste more time fixing up tests ^ | 10:25 |
| stephenfin | the approach in general, that is | 10:26 |
| bauzas | stephenfin: I left a comment on https://review.opendev.org/#/c/671800/ | 10:26 |
| bauzas | stephenfin: cool, I'll do it a bit later (needs to refuel my stomach) | 10:28 |
| *** maciejjozefczyk has quit IRC | 10:28 | |
| *** ttsiouts has quit IRC | 10:28 | |
| *** maciejjozefczyk has joined #openstack-nova | 10:29 | |
| efried | stephenfin: sorry, looking at this patch for the first time, and... is this really the right approach? If we bounce with PCPUs, try VCPUs?? If they *have* cut over, and we're just *actually* out of PCPUs in the cloud, they're going to get *not* desired behavior, nah? | 10:32 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 10:33 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448 | 10:33 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 10:33 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 10:33 |
| *** HagunKim has quit IRC | 10:39 | |
| * efried caffeinates | 10:44 | |
| *** belmoreira has joined #openstack-nova | 10:45 | |
| *** cdent has quit IRC | 10:50 | |
| *** ttsiouts has joined #openstack-nova | 11:03 | |
| sean-k-mooney | efried: the numa toplogy filter will handel the out of PCPU case when they have swapped over to using cpu_dedicted_set | 11:04 |
| * bauzas is back | 11:04 | |
| efried | eh? | 11:05 |
| sean-k-mooney | the numa toplogy filter will still run | 11:05 |
| sean-k-mooney | it can tell the difference between a pre train host and a train host with non pCPUs free | 11:06 |
| efried | If I'm out of PCPU resource per placement, this code will try the alloc request again with VCPUs. All of that happens before we get to the filter, yah? | 11:06 |
| sean-k-mooney | yep | 11:06 |
| efried | so... that's bad | 11:06 |
| sean-k-mooney | and the filter will reject all host that are train but have not pcpus left when we are given vcpu allocations | 11:06 |
| efried | that's not the scenario I'm concerned with. | 11:07 |
| sean-k-mooney | what is? | 11:07 |
| *** ttsiouts has quit IRC | 11:07 | |
| *** shilpasd has quit IRC | 11:07 | |
| efried | Let's say I'm fully upgraded, with my confs properly cut over to cpu_{shared|dedicated}_set. | 11:07 |
| sean-k-mooney | yep | 11:07 |
| efried | this means my cloud has both PCPU and VCPU resources scattered around it. | 11:08 |
| sean-k-mooney | yep | 11:08 |
| efried | sometimes on the same host, maybe sometimes not, whatever. | 11:08 |
| sean-k-mooney | sure | 11:08 |
| efried | I request a dedicated VM. | 11:08 |
| efried | But I'm genuinely out of PCPU resources in my cloud. | 11:08 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681445 | 11:08 |
| sean-k-mooney | yep then we fall back | 11:08 |
| efried | This code will bounce in GET /a_c and then *retry* with VCPUs instead. | 11:08 |
| efried | So I'll wind up with my instance on VCPUs | 11:08 |
| sean-k-mooney | no | 11:08 |
| stephenfin | efried: nope | 11:09 |
| stephenfin | scroll up and look at the conversation between alex_xu and I about two hours ago | 11:09 |
| sean-k-mooney | you get allocation from placmenet that are for VCPUs | 11:09 |
| sean-k-mooney | but then the numa toplogy filter will reject all the hosts | 11:09 |
| sean-k-mooney | stephenfin: do you want to explain why | 11:09 |
| efried | ...because the host knows the VCPUs aren't dedicated. | 11:09 |
| efried | ? | 11:09 |
| stephenfin | efried: Yeah | 11:10 |
| efried | which it doesn't know pre-Train, but in Train it does | 11:10 |
| stephenfin | Placement will try for PCPU, not get it, and then try for VCPU | 11:10 |
| efried | now, this is the version of the filter that runs on the compute host, not the scheduler, right? | 11:10 |
| stephenfin | For Stein hosts, that's fine | 11:10 |
| sean-k-mooney | efried: not its in the schduler | 11:10 |
| efried | oh | 11:10 |
| stephenfin | For Train hosts, the NUMATopologyFilter will then reject the host | 11:10 |
| sean-k-mooney | stepehn modifled the host numa toplogy object | 11:10 |
| stephenfin | and if that's disabled, it'll late fail on the compute node | 11:11 |
| efried | ...so in a partially-upgraded scenario, how does the sched know the diff between stein & train hosts? | 11:11 |
| stephenfin | NUMATopology.pcpuset | 11:11 |
| efried | ack | 11:11 |
| efried | it all comes together | 11:11 |
| stephenfin | For Stein nodes, that isn't set | 11:11 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 11:11 |
| donnyd | sean-k-mooney: I made some adjustments yesterday and I am hoping that fixed the issue we are hainvg getting the numa job to run | 11:11 |
| alex_xu | actually scheduler needn't know the diff of stein and trait | 11:11 |
| stephenfin | For Train nodes with vcpu_pin_set, it's set to the same value as NUMATopology.cpuset | 11:12 |
| sean-k-mooney | i only kicked off one run last night and it passed | 11:12 |
| *** ttsiouts has joined #openstack-nova | 11:12 | |
| stephenfin | For Train nodes with cpu_dedicated_set, it's set to those cores | 11:12 |
| sean-k-mooney | donnyd: there is a new version of the numa migration scripts so i can kick it off again to test tehm | 11:12 |
| stephenfin | For Train nodes with _only_ cpu_shared_set, it's set to None | 11:12 |
| donnyd | ok | 11:12 |
| sean-k-mooney | alex_xu: we want the filter to know the difference so it can do the right thing in a mixed case | 11:13 |
| stephenfin | We check to see if it's unset (meaning this is Stein compute node) and set it to the same value as NUMATopology.cpuset manually in the scheduler | 11:13 |
| donnyd | lmk if there are more issues, I would like to get them ironed out so the label is actually usable | 11:13 |
| stephenfin | However, if it's set to None, we do nothing, which means we'll fail to pin | 11:13 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 11:13 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 11:13 |
| stephenfin | My functional tests prove that out. See here | 11:13 |
| alex_xu | sean-k-mooney: the numa filter doesn't know. if stephenfin is going to update the patch, we will ensure we have value in host.pcpuset and host.cpuset. Then you asking pcpu, we get it from host.pcpuset, if you asking vcpu, then get it from host.cpuset | 11:14 |
| sean-k-mooney | donnyd: sure. i vexhost and limestone might also be willing to provide the lables so we could maybe make this a voting job in the future | 11:14 |
| stephenfin | https://review.opendev.org/#/c/681445/1/nova/tests/functional/libvirt/test_numa_servers.py@249 | 11:14 |
| alex_xu | ensure we have value in host.pcpuset and host.cpuset whatever it is stein node or train node. | 11:15 |
| stephenfin | It's now the NUMATopologyFilter failing the request due to lack of PCPUs, not placement | 11:15 |
| * stephenfin goes to roll that back into alex_xu's patch | 11:15 | |
| stephenfin | TIL 'git commit --amend --fixup=???' resets the author commit message (losing the change-ID). The more you know | 11:15 |
| efried | stephenfin, sean-k-mooney: I re-commented. Good to move forward. Thanks for patiently explaining :) | 11:17 |
| sean-k-mooney | we still need the numa toplogy filter for a release or two anyway since plament know nothing about numa yet so it will be enabled if you are using this feature anyway | 11:18 |
| efried | yah | 11:18 |
| efried | I was going to complain about that aspect, but yah. | 11:18 |
| sean-k-mooney | once we fix placment of tis john snow syntrom re numa that is also proably the releas we want to drop this fallback and the numa toplogy filter | 11:19 |
| sean-k-mooney | im guessing that will be V | 11:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383 | 11:19 |
| *** maciejjozefczyk has quit IRC | 11:19 | |
| sean-k-mooney | assuming we make good progress in Ussuri on using the new placment features | 11:20 |
| sean-k-mooney | that might be a bit aggressive but i dont think a short time line will be possible | 11:20 |
| *** maciejjozefczyk has joined #openstack-nova | 11:21 | |
| sean-k-mooney | we will likely want at least 1 release where the numa toplogy filter is optional | 11:21 |
| sean-k-mooney | which hopefully could be ussuri? | 11:21 |
| *** gibi is now known as gibi_fly | 11:22 | |
| sean-k-mooney | we woudl need to get mempages,cpus,vgpus and pci devices reported in placment under numa nodes first | 11:22 |
| *** cdent has joined #openstack-nova | 11:24 | |
| stephenfin | efried: If that reworded comment looks good to you, I'll go ahead and squash that commit back | 11:26 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383 | 11:26 |
| stephenfin | in there ^, that is | 11:26 |
| * stephenfin goes for lunch | 11:26 | |
| bauzas | dang | 11:28 |
| bauzas | stephenfin: got questions for you https://review.opendev.org/#/c/675571/21 | 11:28 |
| *** udesale has quit IRC | 11:31 | |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738 | 11:34 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with old lable https://review.opendev.org/680739 | 11:34 |
| sean-k-mooney | artom: you patches just went into merge conflict | 11:37 |
| sean-k-mooney | artom: looks like its only the functional tests | 11:37 |
| sean-k-mooney | is it ok to rebase them | 11:38 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 11:38 |
| sean-k-mooney | i want to test the different nodepool lables | 11:38 |
| *** nicolasbock has joined #openstack-nova | 11:39 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 11:39 |
| artom | sean-k-mooney, I know - I don't want to do it just yet | 11:41 |
| sean-k-mooney | its such a dumb confilct | 11:41 |
| artom | sean-k-mooney, make it Depends-on on the patch below it | 11:41 |
| sean-k-mooney | <<<<<< HEAD | 11:41 |
| sean-k-mooney | import fixtures | 11:41 |
| sean-k-mooney | ======= | 11:41 |
| artom | I'd rather make mriedem's and dansmith's job easier at this point | 11:41 |
| sean-k-mooney | import itertools | 11:41 |
| sean-k-mooney | >>>>>>> Functional tests for NUMA live migration | 11:41 |
| sean-k-mooney | ya ok i can do that | 11:41 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 11:42 |
| sean-k-mooney | both fixure and itertools are need by the way and its in alphabetical order so we jsut need to remove the conflict markers. | 11:42 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 11:43 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 11:46 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 11:47 |
| * bauzas disappears for 1.5h | 11:48 | |
| efried | stephenfin: commented. Sorry for being dense here. | 11:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 11:51 |
| *** brault has quit IRC | 11:54 | |
| *** luksky has joined #openstack-nova | 11:55 | |
| *** brault has joined #openstack-nova | 11:55 | |
| *** brault has quit IRC | 11:55 | |
| *** brault has joined #openstack-nova | 11:56 | |
| *** tbachman has joined #openstack-nova | 11:57 | |
| *** spsurya has joined #openstack-nova | 11:58 | |
| luyao | sean-k-mooney: I gave some wrong infomation about vpmem yestoday, so sorry to misleading you. I clarified at here https://review.opendev.org/#/c/678455/24/nova/virt/libvirt/config.py@3180 | 11:58 |
| sean-k-mooney | ok so we dont need to subtract and it should be fine to just have 2KB lables since they are just markers | 12:00 |
| sean-k-mooney | to denote the start and end of the namespace right? | 12:01 |
| luyao | sean-k-mooney: yes, I think so | 12:02 |
| *** dave-mccowan has joined #openstack-nova | 12:05 | |
| sean-k-mooney | cool | 12:05 |
| efried | luyao: Any update on the vpmem CI? | 12:07 |
| *** lpetrut has joined #openstack-nova | 12:13 | |
| yaawang | Hi, could anyone review this patch, it's part of vCPU models selection. https://review.opendev.org/#/c/670298 | 12:13 |
| *** ociuhandu has joined #openstack-nova | 12:13 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** derekh has quit IRC | 12:16 | |
| *** ociuhandu has joined #openstack-nova | 12:16 | |
| efried | yaawang: I've been staying clear of that one because Alex is representing. bauzas or gibi_fly, might you have room for that ^ | 12:16 |
| *** ociuhandu has quit IRC | 12:16 | |
| efried | kashyap: Could you please clarify your opinion on https://review.opendev.org/#/c/678453/27/nova/virt/libvirt/driver.py@447 -- should we use InternalError to be consistent, or go rogue and use InvalidConfig? | 12:17 |
| efried | stephenfin: also ^ when you get back | 12:19 |
| luyao | efried: I know Dolpher and Rui is trying to apply the comunity patches then trigger the tests. Besides, I'm not very clear about that. | 12:19 |
| efried | luyao: okay, thanks. | 12:20 |
| efried | http://52.27.155.124/55/678455/28/check/pmem-tempest-plugin-filtered/7336479/job-output.txt.gz @ 2019-09-11 12:14:29.162146 looks like we're still getting PS13 | 12:22 |
| efried | that completed less than 10 minutes ago. | 12:22 |
| *** awalende has quit IRC | 12:22 | |
| *** awalende has joined #openstack-nova | 12:23 | |
| sean-k-mooney | luyao: efried how is the intel ci running? if its using zuul v3 i dont really see how it difficult to remote the presetp that is hardcoding the patch. | 12:25 |
| sean-k-mooney | it would not be too hard to make it skip it the nova repo did not have teh changeid of the pmem patches in the git log either. | 12:25 |
| efried | sean-k-mooney: agreed, I don't think it's hard, I just think the owners aren't up yet. | 12:25 |
| sean-k-mooney | the repos will be set up before your main playbook runs | 12:25 |
| kashyap | efried: Looking | 12:26 |
| sean-k-mooney | ok so there are no public repos with the job definitions? | 12:26 |
| kashyap | efried: I'd say InternalError for now, for consistency's sake. | 12:26 |
| sean-k-mooney | its managed internally? | 12:26 |
| efried | kashyap: ack, thanks. luyao ---^ | 12:26 |
| efried | sean-k-mooney: I have no idea. | 12:26 |
| sean-k-mooney | i mean that fine but if there is we could porbaly fix it | 12:26 |
| kashyap | efried: And I agree with your point there, in a future commit, it can be changed | 12:26 |
| efried | sean-k-mooney: https://github.com/intel/Intel-OpenStack-CI-jobs/ | 12:27 |
| *** awalende has quit IRC | 12:27 | |
| luyao | kashyap: okay, I'll change it cake to InternalError. :) | 12:27 |
| sean-k-mooney | ok its not mnaged via gerrit so we cant use depens on to test job changes | 12:28 |
| sean-k-mooney | unless they are using gerrithub | 12:28 |
| sean-k-mooney | that woudl be a no | 12:29 |
| kashyap | luyao: Thanks | 12:29 |
| *** etp has quit IRC | 12:31 | |
| luyao | efried, sean-k-mooney: Rui told me just now, them have a patch, but they have no permission to merge it into CI , waiting for matt to merge, then we can appy the latest patches and do the tests | 12:32 |
| efried | luyao: Matt who? Merge where? | 12:32 |
| sean-k-mooney | ok i was just looking at the repo to see if i could write one | 12:32 |
| sean-k-mooney | i did not see it in that repo | 12:32 |
| luyao | efried: Merge ti CI-jobs I think? I don't know about CI | 12:33 |
| sean-k-mooney | any way we coudl get efried or alex_xu added to the people that can merge to it if its going to run against nova | 12:33 |
| *** awalende has joined #openstack-nova | 12:34 | |
| luyao | efried: Matt is helping setup CI, I saw he is the intel ci job repo owner | 12:34 |
| efried | Matt who? (Last name?) | 12:35 |
| sean-k-mooney | efried: https://github.com/matt-welch | 12:35 |
| efried | okay. | 12:35 |
| efried | sean-k-mooney: how does one find out who's got merge perms on a github repo? | 12:35 |
| sean-k-mooney | i dont see the hardcoding in that repo | 12:36 |
| sean-k-mooney | no idea buyt that is the mat that has merged things | 12:36 |
| luyao | efried: Matt Welch | 12:36 |
| sean-k-mooney | you can see it in the org | 12:36 |
| sean-k-mooney | but im not a memebr of the intel org anymore | 12:36 |
| efried | looks like he's the only one who has merged things. | 12:36 |
| efried | ugh, and he's west coast, which probably means he won't be up for another several hours. | 12:37 |
| sean-k-mooney | im not sure if this hardcoding is in that repo or in the tempest plugin repo | 12:37 |
| *** ociuhandu has joined #openstack-nova | 12:38 | |
| efried | sean-k-mooney: https://github.com/intel/Intel-OpenStack-CI-jobs/blob/master/roles/upgrade-libvirt-qemu/tasks/main.yaml#L105-L108 | 12:38 |
| efried | seems like a weird place for it, but whatevs. | 12:38 |
| sean-k-mooney | ah that is why i have not found it it does not have pmem in the name | 12:38 |
| *** awalende has quit IRC | 12:38 | |
| sean-k-mooney | well the task has | 12:39 |
| sean-k-mooney | ya there are btter ways to do that | 12:40 |
| sean-k-mooney | oh | 12:40 |
| sean-k-mooney | i wonder if we can use a pull request in the depens on | 12:40 |
| sean-k-mooney | that should work if they have zuul set up correctlyly | 12:40 |
| efried | brb | 12:41 |
| *** jmlowe has quit IRC | 12:42 | |
| stephenfin | efried: sean-k-mooney or mriedem are you people RE: InternalError or InvalidConfig. It was the former's comment but the latter agreed | 12:42 |
| stephenfin | *your | 12:43 |
| *** maciejjozefczyk has quit IRC | 12:43 | |
| efried | stephenfin: Sorry, I'm failing to parse that | 12:43 |
| efried | oh, got it. | 12:44 |
| efried | sean-k-mooney: Could you please clarify your opinion on https://review.opendev.org/#/c/678453/27/nova/virt/libvirt/driver.py@447 -- should we use InternalError to be consistent, or go rogue and use InvalidConfig? | 12:44 |
| sean-k-mooney | well its not an internal error but i dont really care | 12:45 |
| stephenfin | bauzas: replied on https://review.opendev.org/#/c/675571 (tl;dr: I spotted it and removed it - should have been a different patch/called out in the commit message but it's done now) | 12:45 |
| sean-k-mooney | we proably shoudl change them all to invalid config | 12:45 |
| sean-k-mooney | we can do that in U | 12:45 |
| stephenfin | efried: I vote do the new thing and fix the old things later | 12:45 |
| efried | sigh | 12:45 |
| sean-k-mooney | iternal error implies the something in nova broke and it did not. this is user error | 12:45 |
| stephenfin | foolish consistency and all that | 12:46 |
| efried | luyao: Looks like kashyap is outvoted ^ so please just fix the docstring and we'll move on. | 12:47 |
| sean-k-mooney | efried: from a time perspecitve i dont think this is important but i also dont think internal error is correct for config errors | 12:47 |
| sean-k-mooney | so ill be happy with whatever ye choose | 12:47 |
| efried | sean-k-mooney: no argument that InvalidConfig is better. The only argument to use InternalError is consistency. | 12:47 |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: WIP: Build pdf docs https://review.opendev.org/681463 | 12:48 |
| kashyap | Yeah, maybe in this case being 'consistent' isn't worth it. I don't know - whoever has super strong opinion on it (seems like stephenfin has) | 12:49 |
| sean-k-mooney | doing something consitenly wrong does not make it correct | 12:49 |
| efried | no, it just makes it predictable for the operator | 12:49 |
| *** mriedem has joined #openstack-nova | 12:49 | |
| sean-k-mooney | but two days before freatre freeze i dont really want to block on that | 12:49 |
| sean-k-mooney | true | 12:50 |
| kashyap | sean-k-mooney: Sure :-). I do see the good argument for doing the right thing with InternalConfig | 12:50 |
| sean-k-mooney | if i was an operator i would want to see invalide config | 12:50 |
| sean-k-mooney | because then i know it shte config thats broken not nova | 12:50 |
| *** eharney has joined #openstack-nova | 12:51 | |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Update bug link in docs to point to storyboard https://review.opendev.org/681464 | 12:51 |
| openstackgerrit | Chris Dent proposed openstack/os-traits master: WIP: Build pdf docs https://review.opendev.org/681465 | 12:51 |
| efried | So all we have left is https://review.opendev.org/#/c/678455/ and for stephenfin to re+2 the whole pile. | 12:53 |
| * efried bbiab | 12:53 | |
| *** efried is now known as efried_afk | 12:53 | |
| mriedem | might want to keep an eye on this https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c0b/676972/18/check/nova-tox-functional/c0b03d1/testr_results.html.gz | 12:56 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22DestinationDiskExists%3A%20The%20supplied%20disk%20path%20(path)%20already%20exists%2C%20it%20is%20expected%20not%20to%20exist.%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 12:56 |
| *** jmlowe has joined #openstack-nova | 13:01 | |
| *** cdent has quit IRC | 13:01 | |
| *** derekh has joined #openstack-nova | 13:01 | |
| stephenfin | sean-k-mooney: What do you think of https://review.opendev.org/#/c/678455/28/nova/virt/libvirt/driver.py@5468 ? | 13:02 |
| *** ratailor_ has quit IRC | 13:02 | |
| sean-k-mooney | i dont like it but its what we said we would do a the PTG | 13:04 |
| sean-k-mooney | i would prefer to actully give the guest a numa toplogy like we do with hugepages | 13:04 |
| sean-k-mooney | but just not requrie the pmem namespace to be numa affined | 13:04 |
| stephenfin | Me too | 13:04 |
| sean-k-mooney | but i know some people want to supprot non numa guest with pmem | 13:04 |
| *** nweinber_ has joined #openstack-nova | 13:05 | |
| stephenfin | But the spec says this "Persistent memory is by nature NUMA sensitive." | 13:05 |
| sean-k-mooney | yes it is | 13:05 |
| stephenfin | and also "Libvirt domain specification requires each virtual persistent memory to be associated with one guest NUMA node." | 13:05 |
| sean-k-mooney | its litrally memory dimms that happen to not loose ther content on host reboot | 13:05 |
| sean-k-mooney | yes so we are working around a qemu limitation by giving the guest a virtual numa toplogy of 1 numa node | 13:06 |
| stephenfin | hmm, yeah, this should definitely have NUMA affinity enforced | 13:06 |
| sean-k-mooney | then not affinig it to a host numa node to make it act like a floating vm | 13:06 |
| stephenfin | via the NUMATopologyFilter, of course, until we have NUMA in placement | 13:06 |
| sean-k-mooney | thats what i argued but again we chose not too | 13:07 |
| *** tbachman has quit IRC | 13:07 | |
| sean-k-mooney | without reopentin the spec i think we have to just live with this in train | 13:07 |
| sean-k-mooney | and kill it with fire in U | 13:08 |
| stephenfin | idk. I'd like to see how hard it would be to do the NUMA affinity thing | 13:08 |
| stephenfin | It seems easy | 13:08 |
| stephenfin | just another conditional in numa_get_constraints | 13:08 |
| sean-k-mooney | it would be | 13:08 |
| sean-k-mooney | yep | 13:08 |
| sean-k-mooney | the proable with that is it mean that you cant have geust with more cores then fit on one host numa node | 13:09 |
| sean-k-mooney | without have a multi numa guest | 13:09 |
| sean-k-mooney | and that is want people dont want to have to care about | 13:09 |
| stephenfin | that's no different to huge pages and PCI devices | 13:09 |
| sean-k-mooney | pci device does not creat a numa toplogy | 13:09 |
| sean-k-mooney | hugepage does | 13:09 |
| stephenfin | sorry, not PCI devices | 13:09 |
| stephenfin | yeah | 13:09 |
| sean-k-mooney | hugepages and cpu pinning | 13:10 |
| sean-k-mooney | i would prefer not to special case pmem | 13:10 |
| sean-k-mooney | if we were to special case pmem i would prefer to relax hugepages and cpu pinning to be the same | 13:10 |
| stephenfin | So would I. We have way too many special corner cases as-is. We should be reducing that, not increasing it | 13:10 |
| stephenfin | Yeah, that | 13:10 |
| sean-k-mooney | e.g i have always wanted to decouple pinning/hugepages form numa | 13:11 |
| *** ociuhandu has quit IRC | 13:12 | |
| stephenfin | oh, I'm well aware :) | 13:12 |
| sean-k-mooney | so ya that is where i stand. im not going to block it but i consider it tech debth | 13:13 |
| *** ociuhandu has joined #openstack-nova | 13:13 | |
| sean-k-mooney | stephenfin: at least im consitent on the topic :) | 13:14 |
| *** cdent has joined #openstack-nova | 13:14 | |
| sean-k-mooney | stephenfin: alex_xu luyao https://libvirt.org/formatdomain.html#elementsMemory | 13:15 |
| sean-k-mooney | The node subelement configures the guest NUMA node to attach the memory to. The element shall be used only if the guest has NUMA nodes configured. | 13:15 |
| sean-k-mooney | so it looks like not is optional in the target field | 13:16 |
| sean-k-mooney | so lest just not set it | 13:16 |
| sean-k-mooney | and get rid fo the virtual numa node. | 13:16 |
| stephenfin | I see this above in the opening paragraph though: "Some hypervisors may require NUMA configured for the guest." | 13:16 |
| stephenfin | If kvm/qemu one such hypervisor? | 13:17 |
| sean-k-mooney | if the guest has a numa toplogy we could set it to 0 but i think that would be the default | 13:17 |
| sean-k-mooney | no | 13:17 |
| sean-k-mooney | https://github.com/qemu/qemu/blob/master/docs/nvdimm.txt | 13:17 |
| sean-k-mooney | -machine pc,nvdimm | 13:18 |
| sean-k-mooney | -m $RAM_SIZE,slots=$N,maxmem=$MAX_SIZE | 13:18 |
| *** ociuhandu has quit IRC | 13:18 | |
| sean-k-mooney | -object memory-backend-file,id=mem1,share=on,mem-path=$PATH,size=$NVDIMM_SIZE | 13:18 |
| sean-k-mooney | -device nvdimm,id=nvdimm1,memdev=mem1 | 13:18 |
| sean-k-mooney | basic usage create an nvdim device with no numa node | 13:18 |
| sean-k-mooney | efried_afk: alex_xu luyao ^ | 13:18 |
| stephenfin | okay, if that's true I can definitely live with it | 13:19 |
| stephenfin | remind me to go and convert the QEMU docs to rST sometime | 13:19 |
| *** efried_afk is now known as efried | 13:19 | |
| stephenfin | or Markdown or something | 13:19 |
| luyao | sean-k-mooney: the vm can't run if no numa node | 13:19 |
| sean-k-mooney | plain text is fine | 13:19 |
| sean-k-mooney | luyao: have you tested it | 13:19 |
| luyao | sean-k-mooney: yes | 13:19 |
| sean-k-mooney | well qemu said it can | 13:20 |
| sean-k-mooney | what error did you get | 13:20 |
| stephenfin | But rendered HTML is better (for me, anyway) ;-) | 13:20 |
| sean-k-mooney | stephenfin: and much worse for me :) | 13:20 |
| stephenfin | Is dansmith around yet? | 13:20 |
| * stephenfin taps fingers on desk | 13:20 | |
| luyao | sean-k-mooney: I don't remember it clearly, something like you must specify an numa node | 13:22 |
| sean-k-mooney | can you test it again | 13:22 |
| luyao | sean-k-mooney: actually, the libvirt part you mentioned is my updating before | 13:22 |
| sean-k-mooney | ill try a nested env to test it | 13:22 |
| sean-k-mooney | which libvirt part | 13:22 |
| luyao | sean-k-mooney: About the nvdimm config | 13:23 |
| sean-k-mooney | right the libvirt doc say the node is optional in the target | 13:23 |
| sean-k-mooney | so libvirt hsould not require a numa toplogy | 13:24 |
| sean-k-mooney | and the qemu examle does not create a numa toplogy | 13:24 |
| openstackgerrit | Chris Dent proposed openstack/os-traits master: Build pdf docs https://review.opendev.org/681465 | 13:24 |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Build pdf docs https://review.opendev.org/681463 | 13:25 |
| luyao | sean-k-mooney: that node description is for all memory devices, for nvdimm , we do need a numa | 13:26 |
| *** jaosorior has quit IRC | 13:26 | |
| luyao | sean-k-mooney: I'll try to do a test | 13:26 |
| *** ociuhandu has joined #openstack-nova | 13:26 | |
| sean-k-mooney | i think we may only need the node to be set if there is a numa node in the guest | 13:27 |
| sean-k-mooney | if its not present i think we wont need it | 13:27 |
| *** nweinber__ has joined #openstack-nova | 13:27 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 13:27 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 13:27 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 13:27 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 13:27 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 13:27 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 13:27 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 13:27 |
| sean-k-mooney | so we could hard code it ot 0 if there is a numa toplogy and not set it if there is not | 13:27 |
| *** nweinber_ has quit IRC | 13:29 | |
| mriedem | why is it that Intel_Zuul comments can't be hidden with the 3rd party toggle CI button? | 13:30 |
| sean-k-mooney | it does not have CI in the name | 13:30 |
| mriedem | blarg | 13:30 |
| sean-k-mooney | which it should have | 13:30 |
| mriedem | alex_xu: ^ can we get that fixed? | 13:30 |
| *** ociuhandu has quit IRC | 13:31 | |
| *** eharney has quit IRC | 13:32 | |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] pmem pull request test. https://review.opendev.org/681474 | 13:32 |
| *** eharney has joined #openstack-nova | 13:32 | |
| *** gbarros has joined #openstack-nova | 13:33 | |
| sean-k-mooney | efried: alex_xu luyao in theory ^ should cause the intel ci to run with my pull request that removes the hardcoding | 13:33 |
| sean-k-mooney | it also might make the first party ci unhappy but i have turned off all the check test anyway | 13:33 |
| mriedem | artom: you still have a busted test in https://review.opendev.org/#/c/634606/84 but if you want to tack on a follow up patch to the end of the series i wouldn't be opposed to just fixing it that way | 13:34 |
| artom | mriedem, what's the risk in pushing a new PS? | 13:35 |
| mriedem | is dan doing reviews on the other patches later up the stack? | 13:35 |
| mriedem | just wait for me to hit https://review.opendev.org/#/c/640021/50 again quick | 13:35 |
| artom | Ack | 13:36 |
| efried | stephenfin, sean-k-mooney, luyao: here's spec discussion about numa stuff https://review.opendev.org/#/c/601596/16/specs/train/approved/virtual-persistent-memory.rst@270 | 13:37 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] pmem pull request test. https://review.opendev.org/681474 | 13:37 |
| efried | and the one below it | 13:38 |
| sean-k-mooney | efried: yes they have implemented what the spec said but its still problemenatic | 13:39 |
| efried | but you agreed it was okay to be consistent with the other problematic things that do the same? | 13:39 |
| sean-k-mooney | it is not consitent4 | 13:39 |
| stephenfin | yeah, it's not consistent | 13:39 |
| stephenfin | that's the bit I don't like | 13:39 |
| stephenfin | at all at all | 13:39 |
| sean-k-mooney | the other thing actully pin the cores and memoyr to a host numa node | 13:40 |
| *** panda is now known as panda|ruck | 13:40 | |
| efried | oh, so it's bad in a different way than the other things are bad | 13:40 |
| efried | and we'd rather have it be consistently bad. | 13:40 |
| sean-k-mooney | the other things imporve performance | 13:40 |
| efried | but we're agreed it's too hard to have it be good (and inconsistent) until "later". | 13:40 |
| sean-k-mooney | this does not and it make the other code more complex | 13:40 |
| *** ociuhandu has joined #openstack-nova | 13:41 | |
| sean-k-mooney | efried: it hink we can get the same beahvior if we jsut dont set the node elemnt in the xml | 13:41 |
| stephenfin | efried: I'm not sure what you mean | 13:41 |
| sean-k-mooney | and dont create a numa toplogy at all | 13:41 |
| stephenfin | we're saying it should either not have a NUMA topology, if that's possible (which sean-k-mooney thinks it is) | 13:41 |
| stephenfin | and if it has a NUMA topology, it should behave like everything else with NUMA topologies | 13:42 |
| stephenfin | namely, each guest NUMA node is mapped to a unique host NUMA node | 13:43 |
| sean-k-mooney | im going to quickly update my multi numa test and start stacking the latest version of atroms code then manully try to test if a numa toplogy is need for pmem | 13:43 |
| stephenfin | if we want to loosen that constraint in the future, grand, but it should be done across the board (so hugepages too) | 13:43 |
| luyao | sean-k-mooney: http://paste.openstack.org/show/775121/ | 13:44 |
| efried | I don't understand the issue, so I'm just operating at a very abstract level. My understanding: | 13:44 |
| efried | - The way other resources "do it" is bad | 13:44 |
| efried | - The way pmem is doing it in the current PS is bad, but different bad from ^ | 13:44 |
| efried | - Doing it "good" is hard | 13:44 |
| efried | - So do it the same bad as the "other resources" for now | 13:44 |
| efried | - And some time in the future, make all the things do it good. | 13:44 |
| efried | - ...unless we can do it good now, easily. | 13:44 |
| sean-k-mooney | luyao: remove the numa element on line 7-9 | 13:44 |
| *** tbachman has joined #openstack-nova | 13:45 | |
| *** panda|ruck is now known as panda|rover | 13:45 | |
| sean-k-mooney | if you have numa element in the cpu element i think you need node to be se | 13:45 |
| sean-k-mooney | *set | 13:45 |
| sean-k-mooney | if you dont you should not | 13:45 |
| sean-k-mooney | efried: by they i dont think doing it "good" was hard but let not dwell on that | 13:46 |
| stephenfin | luyao: Yeah, you're hitting this https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11525-L11532 | 13:46 |
| sean-k-mooney | efried: also the way the other numa toplogy stuff works is not bad | 13:46 |
| stephenfin | - The way other resources "do it" is bad | 13:46 |
| stephenfin | it's not bad | 13:46 |
| sean-k-mooney | i just wish it was less copled | 13:47 |
| stephenfin | there's a very good reason we do that | 13:47 |
| efried | okay, this reinforces that I should not be involved with this part of the series. | 13:47 |
| * efried backs away slowly | 13:47 | |
| stephenfin | not at all | 13:47 |
| stephenfin | just that we have strong opinions about this stuff ;) | 13:47 |
| sean-k-mooney | efried: this is why we do it https://software.intel.com/en-us/articles/openstack-epa-feature-breakdown-and-analysis#inpage-nav-6-3 | 13:47 |
| efried | sean-k-mooney: ^ meaning that even if there's only one NUMA node, turning NUMA "on" is better perf than leaving it "off"? | 13:48 |
| sean-k-mooney | the numa affintiy on old systems gave you like 7-10% performacne improvement on its own | 13:48 |
| sean-k-mooney | efried: yes because if the host has multiple numa nodes the vm wont use remote memory form the other numa node or float across them | 13:49 |
| sean-k-mooney | if the host has one numa nod its has no effect | 13:49 |
| efried | yahbut... | 13:50 |
| *** redrobot has joined #openstack-nova | 13:50 | |
| efried | what if my VM needs more CPUs than are available in one NUMA node? | 13:50 |
| sean-k-mooney | create a vm with 2 numa nodes | 13:50 |
| efried | no, I mean... | 13:50 |
| sean-k-mooney | it will perfome better | 13:50 |
| efried | IIUC we're doing this thing to create an implicit single-NUMA guest if no NUMA topo was specified? To realize this perf gain. | 13:51 |
| sean-k-mooney | i get the argument that people dont care | 13:51 |
| sean-k-mooney | and i want ot support that | 13:51 |
| sean-k-mooney | but i think we can support that by just no setteing lines 7-9 in http://paste.openstack.org/show/775121/ | 13:51 |
| sean-k-mooney | efried: no we are not | 13:51 |
| luyao | sean-k-mooney: got a error if i remove numa , Total size of memory devices exceeds the total memory size | 13:52 |
| sean-k-mooney | the current code does not have the performace gain | 13:52 |
| stephenfin | luyao: Can you paste? | 13:52 |
| sean-k-mooney | i thikn we still need to reserv dim slots | 13:52 |
| luyao | efried: Rui told me dean trove also has the permission to merge | 13:52 |
| sean-k-mooney | <maxMemory slots='16' unit='KiB'>1524288</maxMemory> | 13:53 |
| sean-k-mooney | maybe not | 13:53 |
| *** rpittau is now known as rpittau|afk | 13:54 | |
| *** jawad_axd has quit IRC | 13:55 | |
| luyao | sean-k-mooney, stephenfin: http://paste.openstack.org/show/775126/ | 13:56 |
| mriedem | artom: https://review.opendev.org/#/c/640021/50 | 13:57 |
| mriedem | dansmith: i think a couple of questions for you to opine on in there as well ^ | 13:57 |
| *** dtroyer has joined #openstack-nova | 13:58 | |
| efried | o/ dtroyer | 13:58 |
| efried | sean-k-mooney, would you please explain to dtroyer what needs to happen in that CI repo to make it right? | 13:58 |
| dtroyer | o/ | 13:58 |
| sean-k-mooney | can you try one last thing can you sett <maxMemory slots="16" unit='GiB'>8</maxMemory> to inclnde the pmem but set <memory unit='KiB'>524288</memory> | 13:59 |
| sean-k-mooney | <currentMemory unit='KiB'>524288</currentMemory> | 13:59 |
| sean-k-mooney | to a lower value | 13:59 |
| *** Luzi has quit IRC | 13:59 | |
| sean-k-mooney | if that does not work then in the sake of time i think we have to live with this for Train | 14:00 |
| artom | mriedem, ack, looking | 14:00 |
| sean-k-mooney | and in Ussuri we need to find a way to converge this somehow | 14:00 |
| stephenfin | luyao: I'm assuming that's because you're setting <maxMemory> and not <memory> | 14:02 |
| sean-k-mooney | yep | 14:02 |
| stephenfin | You were able to omit that before because "In case NUMA is configured for the guest the memory element can be omitted" | 14:02 |
| stephenfin | from https://libvirt.org/formatdomain.html#elementsMemoryAllocation | 14:02 |
| stephenfin | sean-k-mooney: Cool. Thought that's what you were saying but I wasn't sure :) | 14:02 |
| efried | dtroyer: the simple solution is removing this thing https://github.com/intel/Intel-OpenStack-CI-jobs/pull/1/commits/5d4a998a890fef3a2b0a46f73d26aa1deeb3cd8b | 14:02 |
| artom | mriedem, about the qemu/kvm stuff - it's not "lower" that's the problem, is that the code used to check for "kvm", but then we realized that it should be checking for "qemu", and not all of the tests were changed | 14:02 |
| sean-k-mooney | the max memory is the maium memory the geust can boot with or have hotplugged as addtional memeory | 14:03 |
| artom | mriedem, the first one is harmless, but second one actually needs to change, because it was triggering a false positive by passing the hypervisor check before the thing we were actually testing | 14:03 |
| sean-k-mooney | so we shoudl be setting that to guest ram + nvdim size | 14:03 |
| sean-k-mooney | and the guest memory shoudl be set to the ram size in the flavor | 14:03 |
| dtroyer | efried: would getting rid of the hard-coded ps13 work? | 14:04 |
| efried | dtroyer: I think that's what we want long term. For now it would be nice if it only ran the pmem tests against patches that have at least https://review.opendev.org/#/c/679640/ involved -- otherwise it's running on everything and will therefore fail handily. | 14:04 |
| efried | dtroyer: idk, does refs/changes/$gerritnum work without /$psnum ? | 14:04 |
| sean-k-mooney | it would fail until its merged but it would work after that | 14:04 |
| mriedem | artom: ok i guess i can buy that second one | 14:04 |
| mriedem | artom: so might as well leave them | 14:04 |
| dtroyer | yeah, the filter needs cleaning up, its pretty open now | 14:04 |
| sean-k-mooney | dtroyer: i dont know if the ci is using the git driver or github dirver | 14:05 |
| efried | dtroyer: I always use git review -d, never git fetch | 14:05 |
| * alex_xu is back without headache | 14:05 | |
| sean-k-mooney | dtroyer: if its using the github dirver then https://review.opendev.org/#/c/681474/2 should work | 14:05 |
| sean-k-mooney | efried: same | 14:05 |
| efried | sean-k-mooney: yeah, unless the CI env doesn't have git-review installed :P | 14:06 |
| dtroyer | mee too… I approved that PR, we can iterate from there | 14:06 |
| sean-k-mooney | efried: no it will work | 14:06 |
| sean-k-mooney | this shoudl all be done via the zuul cloner | 14:06 |
| sean-k-mooney | e.g. zuul should be preparing all the repos before the job starts | 14:06 |
| sean-k-mooney | and then the jobs should use those prepreapred repos | 14:07 |
| sean-k-mooney | so we shoudl not use git fetach or git review in the scripts | 14:07 |
| sean-k-mooney | dtroyer: oh i actully didnt mean for it to be aproved since i think there was an internal one coming but i guess thats fine too | 14:08 |
| luyao | sean-k-mooney: http://paste.openstack.org/show/775129/ | 14:08 |
| dansmith | mriedem: just got off a call so I'll look at that deprecation patch in a minute | 14:09 |
| *** maciejjozefczyk has joined #openstack-nova | 14:09 | |
| dtroyer | sean-k-mooney: I think Rui is looking at this in the internal env, yes, I'm not sure where he is on that... | 14:09 |
| dansmith | mriedem: considering how long things are taking to get through the gate, are you comfortable going ahead and letting the bottom three patches there go in or do you want to wait? | 14:09 |
| *** nicolasbock has quit IRC | 14:10 | |
| alex_xu | dtroyer: the mroning, Rui and Dolpher talk with me we can remove the hardcode patch checkout and add a filter. Dolpher will do that when he wakeup I think. Dolpher in the US now | 14:10 |
| stephenfin | luyao, sean-k-mooney: so that's triggering this https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 | 14:11 |
| *** nicolasbock has joined #openstack-nova | 14:11 | |
| dtroyer | alex_xu: ok, thanks, I'll check with him | 14:11 |
| *** nicolasbock has quit IRC | 14:11 | |
| alex_xu | dtroyer: thanks | 14:11 |
| luyao | sean-k-mooney: that's it! | 14:12 |
| alex_xu | yea...that's it | 14:13 |
| sean-k-mooney | this really isnt a hotplug but i guess they build pmem on that backend | 14:13 |
| *** nicolasbock has joined #openstack-nova | 14:13 | |
| luyao | stephenfin: thanks | 14:13 |
| stephenfin | so it looks like we need a guest NUMA topology, meaning we're back to "do we have to treat VPMEM different from hugepages/other things that imply a NUMA topology in nova" | 14:13 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ib50b6b02208f5bd2972de8a6f8f685c19745514c https://review.opendev.org/681490 | 14:14 |
| mriedem | dansmith: i've already started +2ing | 14:14 |
| mriedem | just cross referencing logs with the latest passing CI job run | 14:14 |
| stephenfin | dansmith: Morning. Can I get you to weigh in on https://review.opendev.org/#/c/681383 before I merge it back? | 14:15 |
| *** gbarros has quit IRC | 14:15 | |
| dansmith | mriedem: but the bottom one? /me looks | 14:15 |
| stephenfin | (this is the "try for PCPUs, and if that fails try for VCPUs" change) | 14:15 |
| dansmith | mriedem: oh I see okay | 14:15 |
| dansmith | stephenfin: okay, in a bit | 14:15 |
| luyao | stephenfin: yes, we create one guest numa if no numa specified in Nova, if numa is set, just put vpmems on numa 0 | 14:15 |
| stephenfin | ta | 14:15 |
| *** nicolasbock has quit IRC | 14:15 | |
| stephenfin | luyao: Okay, but why don't we do the pinning to a host NUMA node like we do normally? | 14:16 |
| sean-k-mooney | stephenfin: it did in a previous version | 14:16 |
| sean-k-mooney | it was removed to allow vms that are bigger then a singel numa node to work | 14:16 |
| stephenfin | can we not do that? | 14:17 |
| alex_xu | stephenfin: just because in PTG someone said we should support the floating VM with vpmem | 14:17 |
| stephenfin | and fix it later, for huge pages too | 14:17 |
| *** BjoernT has joined #openstack-nova | 14:17 | |
| alex_xu | then that use-case write into the spec, then that is why we support that | 14:17 |
| sean-k-mooney | alex_xu: i can pretty much guranette that this wont be supported with floating instance downstream. | 14:17 |
| sean-k-mooney | it might but we have no customer askign for that as far as im aware | 14:18 |
| *** ricolin has joined #openstack-nova | 14:18 | |
| stephenfin | I don't think that's a valid request, and I don't think we should be catering to it this cycle | 14:18 |
| stephenfin | If we do want to cater to it, we should do the same for hugepages too | 14:18 |
| sean-k-mooney | and cpu pinning | 14:19 |
| stephenfin | and CPU pinning, yeah | 14:19 |
| sean-k-mooney | there was a propal for hyperv to allow this more or less years ago | 14:19 |
| alex_xu | sean-k-mooney: yea, I have to say the initiial proposal doesn't support that also, it added due to the ptg discussion. | 14:19 |
| alex_xu | emm.... | 14:19 |
| alex_xu | I think how much effort we can remove it | 14:19 |
| stephenfin | (apologies, btw, for brining this up so late in the day /o\ This week was the first time I made my way that far up the patch list) | 14:19 |
| mriedem | artom: a couple more nits in https://review.opendev.org/#/c/634606/84 if you're respinning or add them in a follow up | 14:20 |
| alex_xu | do we need to change hw:pmem parse too much | 14:20 |
| artom | http://giphygifs.s3.amazonaws.com/media/rl0FOxdz7CcxO/giphy.gif | 14:20 |
| bauzas | stephenfin: do you want to continue reviewing your series ? | 14:20 |
| sean-k-mooney | alex_xu: i think we dont need to chagne that at all | 14:20 |
| artom | mriedem, yeah, respinning at this point | 14:20 |
| stephenfin | bauzas: Yup, go for it. There's only two changes to be aware of | 14:20 |
| bauzas | or maybe I should look at gibi_fly's one | 14:20 |
| alex_xu | sean-k-mooney: stephenfin how about just like cpu_policy, when the cpu_policy specific, then we create numa_topology for the instance in the api layer | 14:21 |
| sean-k-mooney | alex_xu: the only thing that would change is we create a numa toplogy in the schduler and we ingore pmem devices in the numa toplogy filter | 14:21 |
| sean-k-mooney | alex_xu: that is what we want to do | 14:21 |
| alex_xu | luyao: ^ sorry, is that easy for your | 14:21 |
| sean-k-mooney | we want to add anothger condtional to get_instance_numa_constiratis ? | 14:22 |
| stephenfin | bauzas: Your call :) I would ask that you hit the bottom three though since you hit them already and all I've done if merge back the follow-ups | 14:22 |
| stephenfin | They start here https://review.opendev.org/#/c/671793/ | 14:22 |
| sean-k-mooney | stephenfin hand the sepcific funtion that need 1 more if | 14:22 |
| bauzas | stephenfin: cool | 14:22 |
| *** cdent has quit IRC | 14:22 | |
| stephenfin | sean-k-mooney: um, what? :) | 14:22 |
| alex_xu | sean-k-mooney: stephenfin luyao I guess we just fall into this branch https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1740 | 14:22 |
| stephenfin | oh, as if that needed another if statement | 14:23 |
| sean-k-mooney | stephenfin: what was the name of the function in hardware.py that we need to add the if tooo | 14:23 |
| stephenfin | ohhh | 14:23 |
| mriedem | artom: ack, you'll have to rebase your functional test patch as well | 14:23 |
| stephenfin | numa_get_constraints, iirdc | 14:23 |
| stephenfin | *iirc | 14:23 |
| mriedem | and https://review.opendev.org/#/c/680739/2 i guess? | 14:23 |
| artom | mriedem, yep, I intentionally left it in merge conflict to facilitate checking changes on the lower patches | 14:23 |
| stephenfin | alex_xu: yup, that's what I think | 14:23 |
| alex_xu | stephenfin: cool | 14:23 |
| alex_xu | luyao: so sorry | 14:24 |
| alex_xu | luyao: are you still at office, I can take over that | 14:24 |
| sean-k-mooney | basically we need to update this right https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1684-L1708 | 14:24 |
| artom | mriedem, and I think at this point that's how it'll stay until the things merge? I don't want to have you wade through a rebased PS just to see that I addressed the test/wording feedback | 14:24 |
| *** tbachman has quit IRC | 14:24 | |
| sean-k-mooney | so that if the flavor has pmen we go into that if and call _get_numa_topology_auto | 14:25 |
| sean-k-mooney | which creates the numa toplogy of 1 | 14:25 |
| stephenfin | exactly | 14:25 |
| luyao | alex_xu: sean-k-mooney stephenfin : I prefer current solution first, when we support numa affinity for vpmem, then generate numa in scheduler | 14:25 |
| sean-k-mooney | actully on just make it hit https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1741 | 14:26 |
| sean-k-mooney | either will work | 14:26 |
| mriedem | artom: ? you can address changes on https://review.opendev.org/#/c/634606/84 and https://review.opendev.org/#/c/640021/50 without touching the already approved patches | 14:26 |
| mriedem | just start at https://review.opendev.org/#/c/640021/50, git rebase -i HEAD~2, | 14:26 |
| mriedem | and then git review -R -y | 14:26 |
| artom | mriedem, right, I was saying I'd have to rebase those as well if I want to get the func test out of merge conflict | 14:26 |
| mriedem | oh yeah i care less about that right now | 14:26 |
| artom | So instead of seeing simple one-line changes, you'd see the rebase stuff as well | 14:26 |
| sean-k-mooney | luyao: yes but the current solution is techdebt you know that right and it does nto work the same as other case that create a implcit numa toplogy | 14:27 |
| mriedem | yeah i hate expecting a simple diff from the previous patchset and finding the person rebased on master | 14:27 |
| alex_xu | sean-k-mooney: is there any case that doesn't work | 14:27 |
| alex_xu | I didn't know the previous discussion | 14:27 |
| sean-k-mooney | alex_xu: yes one. you request more cpus or ram then fit on 1 numa node | 14:28 |
| mriedem | nothing like last minute redesigns for vpmems | 14:28 |
| sean-k-mooney | if we leave the code as it is | 14:28 |
| dansmith | artom: mriedem: I replied on the deprecation patch.. I'll take another scan through the last functional patch and try to give my go/no-go which I think is what mriedem is looking for | 14:28 |
| mriedem | i should get a nickel for every time sean-k-mooney says "we'll do x in U" | 14:28 |
| sean-k-mooney | please remvoe any reference to creating an implcit numa toplogy in the docs because it does not act the same | 14:29 |
| alex_xu | sean-k-mooney: I don't get that. there is any limit on the number of cpus and ram in 1 numa node? | 14:29 |
| stephenfin | luyao: Why can't we do NUMA affinity for VPMEM? Just return the needed info in the NUMATopology object and let the NUMATopologyFilter do the work for you? | 14:29 |
| stephenfin | Just like we do for hugepages | 14:29 |
| sean-k-mooney | alex_xu: we do not allow guest to oversubsribe against it self | 14:29 |
| sean-k-mooney | if you have a numa toplogy of 1 then you cant have more cpus or ram then is availabel on that numa node | 14:30 |
| sean-k-mooney | this is check both in the driver and the numa toplogy filter | 14:30 |
| luyao | stephenfin: we can, but the first design not support that, alex_xu had discuss on PTG before | 14:30 |
| alex_xu | sean-k-mooney: the floating VM can | 14:30 |
| sean-k-mooney | only within the host numa cell | 14:31 |
| sean-k-mooney | but yes | 14:31 |
| efried | dansmith: if you can ack the approach for the quota (even if minor tweaks are needed) I can unblock cpu-resources so we have time for the zillion rechecks that will surely be needed. | 14:31 |
| alex_xu | sean-k-mooney: so it still works :) | 14:31 |
| sean-k-mooney | if the host has 16 core and 64GB of ram in the numa cell you cant have a vm wiith 20 core or 96GB of ram | 14:32 |
| alex_xu | stephenfin: yes, we said no NUMA affinity in the first step | 14:32 |
| sean-k-mooney | i think in the sake fo time we jsut have to follow what the spec says | 14:32 |
| alex_xu | stephenfin: I guess the reason the people doesn't want us put anything more in NUMATopology object anymore | 14:33 |
| sean-k-mooney | its sucks we have to special case and we shoudl document this does not work like other implict numa toplogies | 14:33 |
| sean-k-mooney | alex_xu: ya that was why we did not do numa this cycle | 14:33 |
| alex_xu | sean-k-mooney: qemu will refuse to startup that VM? | 14:33 |
| *** pierreprinetti has joined #openstack-nova | 14:33 | |
| sean-k-mooney | alex_xu: no the numa toplogy filter will prevent it | 14:33 |
| sean-k-mooney | nova does not allow that | 14:34 |
| *** priteau has quit IRC | 14:34 | |
| alex_xu | sean-k-mooney: no, the numa topology filter won't stop that, since the instance has no numa_toplogy obj | 14:34 |
| sean-k-mooney | righ with the special case code it wont | 14:34 |
| alex_xu | sean-k-mooney: we only implicit to create one in libvirt driver | 14:34 |
| sean-k-mooney | so that is why it works | 14:34 |
| mloza | Is there an option in nova for instance timeout to ERROR state when a compute node fails? | 14:34 |
| sean-k-mooney | but was awsering your question about what does hw:numa_nodes=1 do | 14:35 |
| sean-k-mooney | all other implcit numa toploges act exactly like hw:numa_nodes=1 | 14:35 |
| alex_xu | yes | 14:35 |
| sean-k-mooney | hw:pmem is not the same as hw:numa_nodes=1 | 14:35 |
| stephenfin | okay, I hate the idea of the implicit NUMA affinity when VPMEM is specified working differently from implicit VPMEM for other things (hugepages, CPU pinning), but I agree it doesn't make sense to work that way if we can't provide VPMEM NUMA affinity | 14:36 |
| sean-k-mooney | and we should not imply it is in any docs, comment or release notes | 14:36 |
| stephenfin | I'm going to have a look to see how hard tacking on proper NUMA affinity is | 14:36 |
| sean-k-mooney | stephenfin: its not hard it has been blocked because its not plamcent native. | 14:37 |
| stephenfin | I don't buy the argument that we shouldn't add more things to the NUMATopology object. NUMA in placement isn't ready so we've to use what we got | 14:37 |
| stephenfin | put politely, that's nonsense :) | 14:37 |
| *** ttsiouts has quit IRC | 14:37 | |
| *** zhubx has quit IRC | 14:37 | |
| *** zhubx has joined #openstack-nova | 14:37 | |
| *** ttsiouts has joined #openstack-nova | 14:37 | |
| sean-k-mooney | its been nonsense for 3 years but that is not going to change. so we should enbrace doing it the plamcnet way. in U to give mriedem a nickel | 14:38 |
| stephenfin | gimme a half hour to check out how tough this would be to do. If it's terrible, I'll hold my tongue | 14:38 |
| alex_xu | stephenfin: yes, it will be very easy to support numa affinity once we have numa in placement | 14:39 |
| sean-k-mooney | its 1 field in the host numa toplogy blob and a tweek to the numa toplogy filter to match on it in the numa_fit_instace_to_host funciton | 14:39 |
| alex_xu | stephenfin: just like vgpu, the first step no numa affinity | 14:39 |
| sean-k-mooney | alex_xu: its very easy to support it without placmeent but not polically viable | 14:39 |
| stephenfin | vGPU didn't imply an implicit guest NUMA topology though, that's the key difference here | 14:39 |
| bauzas | looks like we side-tracked the discussion | 14:40 |
| bauzas | if that's about providing a NUMA support in placement, that's something we'll do in Unicorn | 14:40 |
| bauzas | because we understand the concerns | 14:40 |
| alex_xu | stephenfin: the imply an implicit guest NUMA and NUMA affinity are different thing | 14:41 |
| bauzas | my PMs can pay for a nickel if that's the problem | 14:41 |
| sean-k-mooney | alex_xu: not currently | 14:41 |
| *** zhubx has quit IRC | 14:41 | |
| sean-k-mooney | form a nova point of view | 14:41 |
| stephenfin | alex_xu: only for VPMEM though | 14:41 |
| sean-k-mooney | but they woudl be with vpmem | 14:41 |
| *** zhubx has joined #openstack-nova | 14:41 | |
| sean-k-mooney | so that break operators understand of what we men when we say there is an implcit numa toplogy | 14:42 |
| *** ttsiouts has quit IRC | 14:42 | |
| sean-k-mooney | alex_xu: we can crate a numa toplogy as a impmeenation detail | 14:42 |
| sean-k-mooney | but we should never telll people we are because they will assume it works the same way as the other cases we do that | 14:42 |
| *** belmoreira has quit IRC | 14:43 | |
| *** gbarros has joined #openstack-nova | 14:43 | |
| *** mlavalle has joined #openstack-nova | 14:43 | |
| *** zhubx has quit IRC | 14:43 | |
| *** zhubx has joined #openstack-nova | 14:44 | |
| sean-k-mooney | alex_xu: just to clarify we use teh terms implcit and explcit numa topoyg in downstream docs and in bug reports and customer expect an implict numa toplogy to be the same as and expclti one. that is the only reason i bring up docs/release notes | 14:44 |
| mriedem | here is an idea, defer vpmems since you're still arguing about designs 1 day before FF and just sort it out in Ussuri :) | 14:45 |
| mriedem | rather than defer and promise to fix things later | 14:45 |
| alex_xu | mriedem: come on... | 14:45 |
| sean-k-mooney | as i said lest just follow the spec and use the code as it is | 14:46 |
| alex_xu | mriedem: so that is works a later fix? | 14:46 |
| sean-k-mooney | alex_xu: we would need to change qemu/libvirt to not do what the current code does | 14:47 |
| mriedem | alex_xu: i'm not reviewing the series nor am i following the argument here, but i know when we say "we'll hack it in x and fix it for real in x+1" that x+1 rarely happens if ever (x+5) | 14:47 |
| sean-k-mooney | or require a realy numa toplogy with affity | 14:47 |
| mriedem | bauzas: you know you already have head count to work on numa in placement support for nova in ussuri? | 14:49 |
| bauzas | mriedem: if I don't have other things like internal crisises, then yes | 14:50 |
| bauzas | or escalations if you prefer | 14:50 |
| sean-k-mooney | at this point not having it block so much of our roadmap since we cant do it the non plamcent way i dont see we have any choice but to dedicate headcount to it | 14:51 |
| mriedem | bauzas: so that means no | 14:51 |
| bauzas | sean-k-mooney: yeah and I went out of steam trying to find some other way for vGPU affinity | 14:51 |
| *** panda|rover is now known as panda|ruck | 14:52 | |
| mriedem | let's just not bullshit ourselves here is all i'm saying | 14:52 |
| bauzas | mriedem: nah, it just means I'm telling the truth : we can't continue implementing NUMA related features without having NUMA support in placement | 14:52 |
| mriedem | sure you can! | 14:52 |
| bauzas | and that's something I'll make it clea to who it could be | 14:52 |
| mriedem | stephenfin doesn't see why ont | 14:52 |
| mriedem | *not | 14:52 |
| stephenfin | jaypipes | 14:52 |
| stephenfin | that was the main reason why, in the past :) | 14:52 |
| sean-k-mooney | no not jaypies | 14:52 |
| mriedem | god rest his soul | 14:53 |
| stephenfin | placement will solve <X> | 14:53 |
| stephenfin | RIP | 14:53 |
| sean-k-mooney | we can implemntate we jsut cant get it merged | 14:53 |
| *** cdent has joined #openstack-nova | 14:53 | |
| sean-k-mooney | anyone here if his house/partner/dogs were safe after the storm | 14:53 |
| mriedem | so what's the actual decision that's blocking the pmems stuff at this point? whether numa is implied or not? | 14:53 |
| stephenfin | by specifying VPMEM, you get an implicit NUMA topology but that implicit NUMA topology works differently to every other implicit NUMA topology | 14:54 |
| mriedem | differently how | 14:54 |
| dansmith | that's BS | 14:54 |
| dansmith | don't do that | 14:54 |
| alex_xu | dansmith: you said yes in PTG :) | 14:54 |
| sean-k-mooney | it does not give numa affinity fo the cpus and memory | 14:54 |
| stephenfin | normally, each guest NUMA node is mapped to a unique host NUMA node | 14:55 |
| stephenfin | with this, they'll float | 14:55 |
| sean-k-mooney | so its not the same as hw:numa_nodes=1 | 14:55 |
| dansmith | alex_xu: sean-k-mooney said I said that too, but I don't agree that I did | 14:55 |
| stephenfin | so the guest has a NUMA topology but it's literally there to work around a quirk in libvirt | 14:55 |
| dansmith | alex_xu: I know that because I'm quite sure I didn't say anything positive about vpmems | 14:55 |
| sean-k-mooney | dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea | 14:55 |
| alex_xu | dansmith: haha | 14:55 |
| stephenfin | I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU) | 14:56 |
| stephenfin | and we're also creating yet another special case that we'll have to clean up later | 14:56 |
| mriedem | so with pmem do you have to have an implicit numa topology? | 14:57 |
| stephenfin | yup, see here | 14:57 |
| stephenfin | https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 | 14:57 |
| mriedem | i mean, will libvirt fail to boot the guest w/o numa defined in the xml? | 14:57 |
| stephenfin | yup | 14:57 |
| mriedem | ah ok | 14:57 |
| alex_xu | sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :) | 14:57 |
| mriedem | alex_xu: remove what? qemu code? | 14:57 |
| stephenfin | alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link | 14:57 |
| mriedem | that is likely years before it hits LTS | 14:58 |
| stephenfin | libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology | 14:58 |
| alex_xu | stephenfin: add numa topology explicityly | 14:58 |
| mriedem | ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova, | 14:58 |
| *** TxGirlGeek has joined #openstack-nova | 14:58 | |
| mriedem | right? | 14:58 |
| bauzas | oh gosh | 14:58 |
| stephenfin | mriedem: Right. But | 14:59 |
| sean-k-mooney | mriedem: yes | 14:59 |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | 14:59 |
| stephenfin | the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter | 14:59 |
| dansmith | artom: commented on the top numa functional patch | 14:59 |
| bauzas | ... and all Hatters technically need to bail out for attending some meeting :) | 14:59 |
| dansmith | bauzas: not all of them :) | 14:59 |
| artom | dansmith, ack - I'm still trying to write words that make sense for https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@179 | 15:00 |
| sean-k-mooney | dansmith: can i join your team to not do numa stuff. it seam simpler | 15:00 |
| mriedem | stephenfin: would the numa topo filter need to look at pmems or something? or just the hardware module that the filter calls? | 15:00 |
| alex_xu | stephenfin: emm...no, that isn't the reason | 15:00 |
| bauzas | dansmith: heh, so true | 15:00 |
| artom | It's tricky because... just because we found some old computes in the cell, doesn't mean the source and dest are old | 15:00 |
| stephenfin | to be able to do NUMA affinity properly, we'll need to provide information about the VPMEM devices available in each node as part of the NUMATopology/NUMACell objects | 15:00 |
| dansmith | sean-k-mooney: heh | 15:00 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | 15:00 |
| artom | So I want to word around that | 15:00 |
| dansmith | artom: ack | 15:00 |
| *** TxGirlGeek has quit IRC | 15:00 | |
| mriedem | artom: the code isn't granular to just the source and selected dest - where you're checking we don't have a dest yet in some cases, | 15:01 |
| mriedem | so i'm not sure why that would matter for the reno | 15:01 |
| mriedem | oh this is the log | 15:01 |
| *** spatel has joined #openstack-nova | 15:01 | |
| artom | mriedem, yeah - I don't want to say "WARNING LEGACY BEHAVIOR" unconditionally | 15:01 |
| stephenfin | Yeah, the NUMA fitting code in the hardware module only gets InstanceNUMATopology, NUMATopology, PCIRequest and whatever the host PCI tracking object is | 15:01 |
| mriedem | artom: i think it's fine to say computes aren't new enough | 15:02 |
| mriedem | that's generic enough | 15:02 |
| artom | Yeah, except they *might* be new enough | 15:02 |
| mriedem | that doesn't matter | 15:02 |
| artom | If that specific source and dest are, even if others in the cell aren't | 15:02 |
| mriedem | b/c that's not how the code works | 15:02 |
| alex_xu | stephenfin: ah, you complain the part of explicltly numa but no affinity | 15:02 |
| mriedem | iow i don't see a problem with the paraphrase that dan gave | 15:02 |
| alex_xu | stephenfin: but you are ok with implicity numa? | 15:03 |
| artom | So, if we got passed the service check, it means there are some old computes in the cell | 15:03 |
| alex_xu | to be host, I'm confused what we discussion for now.... | 15:03 |
| mriedem | artom: correct | 15:03 |
| artom | But... the actual source and dest might both be new | 15:03 |
| artom | In which case, assuming the workaround is enabled, NUMA-LM will happen | 15:03 |
| stephenfin | alex_xu: That was the reason we didn't do the implicit NUMA affinity, right? | 15:03 |
| mriedem | sure, but we don't necessarily have a dest at the point of that check | 15:03 |
| mriedem | yes | 15:03 |
| artom | So I don't want to say "you enabled the workaround, things will explode" | 15:04 |
| mriedem | artom: i mentioned this here https://review.opendev.org/#/c/640021/48/nova/conductor/tasks/live_migrate.py@196 | 15:04 |
| *** jmlowe has quit IRC | 15:04 | |
| artom | More like "you enabled the workaround, things will explode if the source and dest are old" | 15:04 |
| artom | (explode == eggagerating) | 15:04 |
| luyao | stephenfin, stephenfin : i'm confused about 'implicit' numa | 15:04 |
| spatel | Help!! - I want to evacuate compute nodes because it has hardware issue, but all running vm using local-disk ( NOT shared disk ) can i do evacuate? | 15:04 |
| alex_xu | stephenfin: no, that isn't. The reason do implicit NUMA topology for instance just due to someone asking that usecase in the PTG | 15:04 |
| mriedem | artom: i don't think that's what dan suggested | 15:04 |
| mriedem | "Computes are too old to do the smart thing, workaround is enabled, doing that" | 15:04 |
| dansmith | spatel: see topic | 15:04 |
| alex_xu | stephenfin: the explicity NUMA topology for instance and without affinity is clear plan in the PTG | 15:05 |
| mriedem | you adjust the warning to say that computes are too old to do the supported method or whatever | 15:05 |
| dansmith | spatel: you might notice that we're a day before FF and there are multiple dev-related discussions going on in here | 15:05 |
| spatel | dansmith: sorry about that, you guys continue. | 15:05 |
| luyao | stephenfin,alex_xu: what is implicit NUMA affinity | 15:05 |
| stephenfin | alex_xu: It's sounding like the implications of that plan were not properly understood though. Certainly I didn't grasp them | 15:05 |
| alex_xu | luyao: I guess that is just stephenfin typing wrong word, actually mean implicity NUMA topology | 15:07 |
| alex_xu | stephenfin: ^ is that right? or you pointed to something I may not get | 15:07 |
| mriedem | artom: i left a suggestion on wording | 15:07 |
| stephenfin | luyao: Implicit NUMA affinity is what we do for hugepages and CPU pinning. If you use either of those features, your guest will have a one-node NUMA topology and it will be pinned to a host NUMA node | 15:07 |
| stephenfin | alex_xu: Yeah, an implicit NUMA topology | 15:07 |
| luyao | stephenfin: I thought implicit Alex said is the guest numa not bind to host numa | 15:08 |
| mriedem | stephenfin: oh i have a solution - drop the numa part but only support vpmems on ppc64 https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 :) | 15:08 |
| mriedem | power to the rescue | 15:08 |
| stephenfin | fight the power | 15:08 |
| luyao | stephenfin, alex_xu: so you are discuss different things ? | 15:08 |
| *** jmlowe has joined #openstack-nova | 15:09 | |
| mriedem | does tonyb have openshift working on power yet? | 15:09 |
| stephenfin | luyao: no, the disagreement is on the lack of pinning | 15:09 |
| artom | mriedem, https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@200 | 15:09 |
| *** TxGirlGeek has joined #openstack-nova | 15:10 | |
| stephenfin | the fact that for hugepages, the single guest NUMA cell is pinned to a host NUMA cell, but for VPMEM, the single guest NUMA cell floats across all host NUMA cells | 15:10 |
| stephenfin | *for hugepages and CPU pinning | 15:10 |
| alex_xu | stephenfin: yes, so you want the pinning, but is it ok without affinity for now? | 15:11 |
| stephenfin | bauzas: Think you could hit these two too? https://review.opendev.org/#/c/681060/ https://review.opendev.org/#/c/681061/ | 15:11 |
| stephenfin | alex_xu: I'll take that as a middle ground, yes | 15:11 |
| bauzas | stephenfin: looking | 15:12 |
| alex_xu | stephenfin: so the only I need is fall into this branch https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1740, right | 15:12 |
| stephenfin | yup | 15:12 |
| alex_xu | cool | 15:12 |
| mriedem | artom: replied | 15:12 |
| mriedem | with anger | 15:13 |
| luyao | stephenfin: Does it make sense to do the numa pinning now without affinity. | 15:13 |
| sean-k-mooney | luyao: yes | 15:13 |
| stephenfin | luyao: It's definitely sub-optimal, I agree, but it's better than breaking our long-standing one guest NUMA node == one host NUMA node policy | 15:13 |
| sean-k-mooney | its not ideal but its not a bad thing | 15:14 |
| artom | mriedem, ok, there's something I genuinely don't get then | 15:14 |
| luyao | sean-k-mooney: Okay | 15:14 |
| stephenfin | you'll get inferior performance but from a user perspective, things behavior as expected | 15:14 |
| artom | Not trying to make you angry on purpose, for serious | 15:14 |
| stephenfin | *behave | 15:14 |
| artom | If the conductor's train, and the source is train, and the dest is train, but there are other stein computes in the cell | 15:14 |
| artom | We'll get past the min sevice version check | 15:15 |
| artom | get past = not return | 15:15 |
| artom | Meaning we check the value of the workaround | 15:15 |
| artom | If the workaround is disabled, that's that | 15:15 |
| artom | If the workaround is enabled, in *this specific case of dest and source being new* | 15:16 |
| artom | We'll get NUMA LM | 15:16 |
| mriedem | artom: even if the source is new and you checked that, the dest - which we might not have yet if the user didn't force the dest from the api and bypass the scheduler - might not be new, and we won't know in that code with the warning log message because it comes before we ask the scheduler for a dest | 15:17 |
| mriedem | so if we get lucky and the source host and selected dest host are new enough, yeah it might be fine | 15:18 |
| mriedem | but, | 15:18 |
| dansmith | mriedem: I don't think that's whathe's saying | 15:18 |
| mriedem | it probably won't be because if you have old computes, | 15:18 |
| mriedem | rpc is going to be pinned and we'll backlevel the new numa migrate_data object stuff | 15:18 |
| dansmith | this ^ is what he's saying | 15:18 |
| dansmith | so he just doesn't want to specifically say that the src or dst is too old.. right? | 15:19 |
| mriedem | we never asked him to say that | 15:19 |
| mriedem | we just said "say something about computes not being upgraded yet" | 15:19 |
| dansmith | I don't think he claimed we did | 15:19 |
| mriedem | who's on first? | 15:19 |
| dansmith | I think he just said he's trying to wordsmith a suitably generic message right? | 15:19 |
| mriedem | i provided one | 15:20 |
| artom | Well there's the logging on L196 as well - it all fits together | 15:20 |
| artom | Or I'm just overcomplicating everything? | 15:20 |
| dansmith | I don't see any logging on L 196 | 15:20 |
| artom | You asked for it :) | 15:21 |
| mriedem | artom: i think dan just meant adjusting the msg object | 15:21 |
| dansmith | you mean the logging that would be on L196 if you updated? | 15:21 |
| mriedem | if the config is enabled, we log a warning | 15:21 |
| mriedem | else we raise, but with the same message | 15:21 |
| dansmith | mriedem: I was suggesting one more log message inside the service version check | 15:22 |
| *** gyee has joined #openstack-nova | 15:23 | |
| mriedem | ok i'm going to move onto reviewing gibi's stuff now | 15:23 |
| artom | Haha | 15:23 |
| dansmith | how about artom just fixes the test issue, adds logs like he thinks he needs, and then we can sort out the log minutia in a later patch if necessary? | 15:23 |
| bauzas | stephenfin: good news, I now start looking at your reshaper change :) | 15:24 |
| mriedem | that's fine | 15:24 |
| stephenfin | huzzah! | 15:24 |
| gibi_fly | mriedem: hit me with any questions | 15:26 |
| artom | dansmith, btw, the compute RPC version is per-cell, right? So, if we're in a cell with mixed stein/train computes, none of the will be able to send 5.3? | 15:27 |
| artom | There's no backporting like what's done for objects - if a node can send 5.3, then all others must be able to receive it? | 15:27 |
| dansmith | artom: not really, but I definitely do not think that level of detail needs to be in there | 15:27 |
| efried | sean-k-mooney: https://review.opendev.org/#/c/681474/2 appears not to have worked -- | 15:27 |
| efried | http://52.27.155.124/74/681474/2/check/pmem-tempest-plugin-filtered/134b243/job-output.txt.gz @ 2019-09-11 14:17:14.914120 is still applying PS13 | 15:27 |
| artom | dansmith, yeah, maybe I'm overthinking it | 15:28 |
| *** macz has joined #openstack-nova | 15:29 | |
| *** ivve has quit IRC | 15:29 | |
| efried | sean-k-mooney, dtroyer: I can't tell by looking at https://github.com/intel/Intel-OpenStack-CI-jobs/pull/1/commits/5d4a998a890fef3a2b0a46f73d26aa1deeb3cd8b -- is this merged? | 15:29 |
| sean-k-mooney | efried in that case they are likely using the git plug not the gitub plugin | 15:29 |
| *** gbarros has quit IRC | 15:30 | |
| sean-k-mooney | efried: the zuul git driver does not support pull request but works with any git repo via a timer | 15:30 |
| sean-k-mooney | the gitub one require the zuul app to be installed in the github org | 15:30 |
| sean-k-mooney | but once that is done pull requeist work | 15:30 |
| sean-k-mooney | i suspect intel it dont allow the zuul app | 15:30 |
| efried | sean-k-mooney, dtroyer: I wanted to just recheck one of the top vpmem patches to get new results | 15:30 |
| efried | but no point unless that pr merged | 15:31 |
| *** igordc has joined #openstack-nova | 15:31 | |
| mriedem | gibi_fly: | 15:32 |
| mriedem | https://review.opendev.org/#/c/676980/19 | 15:32 |
| gibi_fly | mriedem: looking | 15:32 |
| bauzas | I believe I can gibi_fly | 15:33 |
| bauzas | (sorry) | 15:33 |
| mriedem | get out | 15:33 |
| gibi_fly | mriedem: good points. I will fix it up today. | 15:35 |
| *** shilpasd has joined #openstack-nova | 15:35 | |
| sean-k-mooney | efried its not merged by the way | 15:36 |
| gibi_fly | bauzas: I will change to a bus soon so you were on time with that joke :) | 15:36 |
| bauzas | gibi_fly: do you need help for providing a new revision ? | 15:36 |
| sean-k-mooney | efried: the PR woudl be closed if it was merged | 15:36 |
| bauzas | gibi_fly: since you're a bit on-off | 15:36 |
| bauzas | FWIW, I starbucks'd last Friday and it wasn't a great experience | 15:37 |
| gibi_fly | bauzas: thanks. I will manage | 15:37 |
| gibi_fly | bauzas: I will ping you for review :) | 15:37 |
| bauzas | gibi_fly: cool | 15:37 |
| *** eharney has quit IRC | 15:37 | |
| gibi_fly | bauzas: I don't know which net is better, plane or starbucks | 15:37 |
| efried | sean-k-mooney: okay, that's what I thought, but dtroyer said he merged it so I wasn't sure if I just didn't know how to read the thing (which would be totally likely) | 15:38 |
| luyao | stephenfin: could you review rest of the patch series about vpmems? | 15:38 |
| sean-k-mooney | he approved it and i guess that normllay allows a ci to merge it but in this case it might need to be manually submitedd | 15:38 |
| stephenfin | luyao: Already on it :) | 15:38 |
| luyao | stephenfin: Thanks! alex_xu will address the numa stuff. :) | 15:39 |
| bauzas | gibi_fly: I'm just surprised you getting internet connection while on flight | 15:41 |
| bauzas | gibi_fly: I never saw this with BA or Swiss or whatever else | 15:41 |
| alex_xu | luyao: go home, it is too late to be office right now | 15:41 |
| gibi_fly | bauzas: Lufthansa has some paidd service | 15:41 |
| gibi_fly | bauzas: 7 EUR to push patches and read reviews sounded like a bargain | 15:42 |
| luyao | alex_xu: I prepared to fight overnight. :D | 15:43 |
| * bauzas reconsiders Lufthansa then | 15:43 | |
| efried | lufthansa is a lovely airline | 15:43 |
| bauzas | oh gosh it's not | 15:43 |
| *** jaosorior has joined #openstack-nova | 15:44 | |
| efried | but maybe only compared to the shit that's standard in the US | 15:44 |
| bauzas | but that's a sidetrack | 15:44 |
| dtroyer | efried, sean-k-mooney: it is probably my github fail… I hit approve… but I've also been multi-tasking two meetings this whole time... | 15:45 |
| stephenfin | when you've flown Ryanair, anything is a lovely airline | 15:45 |
| melwitt | I find lufthansa's gate boarding "system" a total mess (in that there seems to be no system) but other than that, I've been happy enough with it | 15:45 |
| sean-k-mooney | dtroyer: no worries. you have too also manually submit it if there is not a ci set up to do it | 15:46 |
| efried | aspiers: fyi you can knock a patch out of the gate by rebasing it. I can re+W if you want to do that rather than waiting for the gate to fail. | 15:48 |
| aspiers | efried: ah, nice idea thanks! | 15:49 |
| *** eharney has joined #openstack-nova | 15:50 | |
| bauzas | well, what I can just say about Lufthansa is how they're aggresive with carry-on luggage | 15:50 |
| openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | 15:51 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738 | 15:51 |
| bauzas | every time I flew with them, my carry-on was taken on checked luggage, just because they felt they were running out of space in the cabin | 15:51 |
| *** damien_r has quit IRC | 15:51 | |
| dtroyer | efried, sean-k-mooney: ok, for the record, just talked with Dolpher and his final change is ready to be merged into the Intel-OpenStack-CI-jobs repo, I should get that done after this meeting | 15:51 |
| bauzas | I admit this was in Geneva, but come on, even on August when all people wear light stuff ? | 15:52 |
| efried | dtroyer: nice | 15:52 |
| sean-k-mooney | dtroyer: ok ill close my PR then | 15:52 |
| sean-k-mooney | lets just go with the proper one | 15:52 |
| sean-k-mooney | also i forgot to chnage the rul in the numa tests new patch incoming | 15:53 |
| *** Garyx has quit IRC | 15:53 | |
| *** Garyx has joined #openstack-nova | 15:54 | |
| efried | melwitt: I wonder if you're hitting https://github.com/openstack/keystoneauth/blob/38cd5fc6c39c38a51c11683884caf9696ce5f367/keystoneauth1/exceptions/http.py#L411 which afaict will result in your exception containing bugger-all. | 15:54 |
| openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | 15:56 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738 | 15:56 |
| efried | melwitt: ...which may be possible if the encoding is off or something. | 15:56 |
| *** TxGirlGeek has quit IRC | 15:57 | |
| *** TxGirlGeek has joined #openstack-nova | 15:58 | |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test migration with pinning https://review.opendev.org/679754 | 15:59 |
| melwitt | efried: thanks for the idea | 15:59 |
| *** shilpasd has quit IRC | 16:00 | |
| artom | Stupid sphinx errors line numbers that have nothing to do with anything | 16:01 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test non overlaping vcpu pin sets. https://review.opendev.org/679805 | 16:01 |
| *** spatel has quit IRC | 16:01 | |
| sean-k-mooney | artom: ok all the numa jobs should be using the old nodepool lable now and running agaist the patch before the functionl tests | 16:02 |
| *** spatel has joined #openstack-nova | 16:02 | |
| sean-k-mooney | maybe i missed one | 16:05 |
| mriedem | artom: if it was reno, i use an online sphinx editor to validate those rather than wait to build them locally | 16:06 |
| aspiers | efried: hrm, bit confused - now the failure has disappeared from the gate but there's another job for it still running in the gate's "integrated" queue which has no failures | 16:07 |
| artom | mriedem, thanks for the suggestion, I think I'll do that | 16:07 |
| *** spatel has quit IRC | 16:07 | |
| artom | sean-k-mooney, cheers, thanks! | 16:07 |
| aspiers | efried: might it have failed in a different queue in the gate? which gate queue(s) actually count for the V+2? | 16:07 |
| sean-k-mooney | fixing the last one | 16:07 |
| efried | aspiers: is it possible you were looking at your patch as it existed as part of some other patch's chain? | 16:08 |
| efried | what's the change #? | 16:08 |
| aspiers | efried: 681254 | 16:08 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test non overlaping vcpu pin sets. https://review.opendev.org/679805 | 16:08 |
| aspiers | efried: the failure I saw was at the tip of a chain | 16:08 |
| * aspiers looks in the builds tab | 16:08 | |
| aspiers | weird, can't see it | 16:09 |
| aspiers | the only failure was from the first time around | 16:09 |
| aspiers | i.e. 2019-09-11T05:56:59 which was before my recheck | 16:09 |
| aspiers | maybe I was hallucinating | 16:10 |
| efried | yeah, it looks okay to me | 16:10 |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| sean-k-mooney | donnyd: fyi i have 4 jobs queue 3 using the old lable and 1 using the new one so ill let you know how it goes. hopeful your router tweaks resolved the issue | 16:10 |
| *** jmlowe has quit IRC | 16:13 | |
| sean-k-mooney | stephenfin: efried is there anything i can do to help with either the PCPU in plamcenet seriese or pmem? | 16:14 |
| efried | sean-k-mooney: I'm at a point where I'm willing to proxy your vote on https://review.opendev.org/#/c/678455/ -- once we've agreed whether it needs to be changed or not. | 16:15 |
| efried | I'm not clear whether y'all came to an agreement above? | 16:15 |
| stephenfin | sean-k-mooney: I'm on the vpmem one, so if you fancy looking at either https://review.opendev.org/#/c/674895/ | 16:16 |
| stephenfin | efried: alex_xu is on it | 16:16 |
| mriedem | TestInstanceNotificationSampleWithMultipleCompute.test_multiple_compute_actions just reset the gate again, | 16:16 |
| mriedem | i think we have a regression | 16:16 |
| stephenfin | We agreed it was better to keep the one guest NUMA node to one host NUMA node model, even if we don't do proper affinity for the VPMEM devices | 16:16 |
| efried | and that's because qemu has some checks in place that don't allow us to do the slightly better thing? | 16:17 |
| sean-k-mooney | stephenfin: so like pci passthoug with affinity disabled | 16:17 |
| stephenfin | efried: Yup | 16:17 |
| efried | okay cool. | 16:17 |
| stephenfin | sean-k-mooney: Yeah, good analogy | 16:17 |
| stephenfin | efried: I did have a comment on https://review.opendev.org/#/c/678448/20 though | 16:18 |
| stephenfin | I'm not sure if it's worth respinning for but if we don't, we're stuck with that object name forever | 16:18 |
| sean-k-mooney | whats SlugField | 16:19 |
| sean-k-mooney | oh somting that is alpha numeric or has _ | 16:20 |
| stephenfin | sean-k-mooney: That's the terminology I'm used to from Django. I thought it was more widespread than that, tbh | 16:20 |
| stephenfin | yeah, exactly | 16:20 |
| sean-k-mooney | not really but i have seen you use it before | 16:20 |
| *** luksky has quit IRC | 16:21 | |
| stephenfin | here we go https://en.wikipedia.org/wiki/Clean_URL#Slug | 16:21 |
| efried | stephenfin: I thought slug was with hyphens, not underscores | 16:21 |
| stephenfin | Good point. It is | 16:21 |
| efried | oh, according to that wp article it's either. | 16:21 |
| efried | but also lowercased | 16:22 |
| *** ricolin has quit IRC | 16:22 | |
| efried | and also specific to URLs | 16:22 |
| sean-k-mooney | ya this is specifcly only uppercase | 16:22 |
| efried | I was beaten up about this by jaypipes at some point when I tried to name a method slugify | 16:22 |
| efried | in fact | 16:22 |
| sean-k-mooney | its basically what i know as CONST_CASE | 16:22 |
| efried | I think it was the method now known in os-traits and os-resource-classes as normalize_name :) | 16:22 |
| efried | so no | 16:22 |
| efried | you can't call it a slug | 16:22 |
| efried | because Jay will kick your ass | 16:22 |
| *** tbachman has joined #openstack-nova | 16:22 | |
| efried | IMO it's definitely not worth respinning for this. | 16:23 |
| efried | even if this object winds up single use | 16:23 |
| stephenfin | Fair point. I was on the fence as it was | 16:23 |
| sean-k-mooney | we can alwas do it later | 16:24 |
| sean-k-mooney | if we have a common felid type this can jsut be an alias or inherit | 16:24 |
| stephenfin | very true | 16:24 |
| stephenfin | I switched to +2 anyway | 16:24 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1843615 | 16:24 |
| openstack | Launchpad bug 1843615 in OpenStack Compute (nova) "TestInstanceNotificationSampleWithMultipleCompute.test_multiple_compute_actions intermittently failing since Sept 10, 2019" [Undecided,New] | 16:24 |
| stephenfin | on account of it not being a slug and therefore probably not as reusable as I thought it would be | 16:25 |
| sean-k-mooney | stephenfin: once question does the qutotat code need to be move down in the series | 16:25 |
| stephenfin | sean-k-mooney: Yup, see my comment on same. I'll move it when I squash that fixup patch back | 16:25 |
| stephenfin | except for the functional test. That has to wait til the end, of course | 16:25 |
| sean-k-mooney | ah cool | 16:25 |
| stephenfin | the squash back which will be happening any minute now | 16:26 |
| openstackgerrit | Alex Xu proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 16:29 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 16:29 |
| openstackgerrit | Alex Xu proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 16:29 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 16:29 |
| openstackgerrit | Alex Xu proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 16:29 |
| *** ociuhandu has quit IRC | 16:30 | |
| alex_xu | stephenfin: sean-k-mooney ^ done \o/ | 16:30 |
| alex_xu | really small change actually | 16:31 |
| alex_xu | and tested | 16:31 |
| efried | sean-k-mooney: Can I abandon https://review.opendev.org/#/c/681474/ ? | 16:31 |
| sean-k-mooney | oh yes | 16:31 |
| alex_xu | stephenfin: sean-k-mooney removed the workaround code at https://review.opendev.org/678455, then a super simple check at https://review.opendev.org/678456 | 16:33 |
| sean-k-mooney | alex_xu: i expect the code is simpler overall | 16:34 |
| alex_xu | sean-k-mooney: yea | 16:34 |
| alex_xu | oops, I forget to remove that useless need_pin parameters | 16:35 |
| sean-k-mooney | im not sure get_guest_numa is correct it proablyt should be get_guest_singel_numa | 16:36 |
| *** ralonsoh has quit IRC | 16:37 | |
| efried | these sound like fupables | 16:37 |
| alex_xu | sean-k-mooney: emm? | 16:37 |
| sean-k-mooney | https://review.opendev.org/#/c/678455/30/nova/virt/hardware.py | 16:38 |
| sean-k-mooney | it ignores the flavor and always returns a single numa node | 16:38 |
| sean-k-mooney | so get_guest_numa is missleading | 16:38 |
| alex_xu | sean-k-mooney: sorry, that one should be removed, noone use that method anymore | 16:39 |
| sean-k-mooney | ah ok | 16:39 |
| sean-k-mooney | i was looking for where it was used | 16:39 |
| sean-k-mooney | that explain why i could not find it | 16:39 |
| * bauzas is done for the day unfortunately | 16:40 | |
| bauzas | see you folks tomorrow morning | 16:40 |
| *** lpetrut has quit IRC | 16:40 | |
| * bauzas goes daddy | 16:40 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 16:41 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 16:41 |
| openstackgerrit | Alex Xu proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 16:41 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 16:41 |
| openstackgerrit | Alex Xu proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 16:41 |
| alex_xu | sean-k-mooney: actually, it is all about this little bit https://review.opendev.org/#/c/678456/32/nova/virt/hardware.py@1688 | 16:42 |
| *** ivve has joined #openstack-nova | 16:42 | |
| sean-k-mooney | alex_xu: ya that shoudl cause nodes to be 1 | 16:43 |
| sean-k-mooney | and i think we will take this brance when creating the numa toplgoy right https://review.opendev.org/#/c/678456/32/nova/virt/hardware.py@1742 | 16:44 |
| sean-k-mooney | well we will also call numa_topology = _get_numa_topology_auto(...) | 16:45 |
| *** jaosorior has quit IRC | 16:46 | |
| *** ociuhandu has joined #openstack-nova | 16:46 | |
| sean-k-mooney | ya | 16:47 |
| alex_xu | sean-k-mooney: yes, it is go to the _get_nuam_toplogy_auto | 16:47 |
| sean-k-mooney | the singel numa toplogy is created by https://review.opendev.org/#/c/678456/32/nova/virt/hardware.py@1514 | 16:47 |
| sean-k-mooney | which is exactly what we want | 16:47 |
| alex_xu | yea | 16:47 |
| *** TxGirlGeek has quit IRC | 16:51 | |
| stephenfin | efried: What do you think about https://review.opendev.org/#/c/678455/25/nova/virt/libvirt/driver.py@1299 ? | 16:53 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 16:54 |
| artom | dansmith, mriedem ^^ | 16:54 |
| efried | stephenfin: It is critical that a dirty pmem not be allowed to be assigned to another guest | 16:54 |
| efried | sorry, you're asking whether we should continue cleaning up wait until the end to fail out? | 16:55 |
| *** TxGirlGeek has joined #openstack-nova | 16:55 | |
| stephenfin | yup | 16:55 |
| efried | this is by no means the only way we can fail out of a destroy() early | 16:55 |
| efried | is it? | 16:55 |
| stephenfin | That's what I'm trying to figure out | 16:55 |
| stephenfin | _unplug_vifs is called with ignore_errors=True | 16:56 |
| * artom is reminded he should get food... | 16:56 | |
| efried | looking at L1207-30 | 16:56 |
| efried | anything that's not LibvirtException will cause immediate failure. | 16:56 |
| efried | and even some LibvirtErrorZ | 16:56 |
| efried | so this is far from unprecedented it would seem. | 16:56 |
| stephenfin | good point. If it's broken, it was already broken | 16:57 |
| efried | stephenfin: likewise below when cleaning up bdms | 16:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix race in _test_live_migration_force_complete https://review.opendev.org/681540 | 16:57 |
| mriedem | efried: dansmith: ^ this fixes a gate bug | 16:57 |
| mriedem | which reset us earlier | 16:57 |
| sean-k-mooney | stephenfin: its called where | 16:57 |
| *** tesseract has quit IRC | 16:57 | |
| *** gibi_fly is now known as gibi_bus | 16:58 | |
| sean-k-mooney | oh in distroy | 16:58 |
| stephenfin | https://review.opendev.org/#/c/678455/25/nova/virt/libvirt/driver.py@1196 | 16:58 |
| stephenfin | sean-k-mooney: ^ | 16:58 |
| *** mriedem is now known as gibi_zeppelin | 16:58 | |
| * stephenfin notes gibi_bus is on quite the little roadtrip today | 16:58 | |
| stephenfin | heh | 16:58 |
| *** gibi_zeppelin is now known as gibi_submarine | 16:58 | |
| *** gibi_submarine is now known as mriedem | 16:58 | |
| *** spatel has joined #openstack-nova | 16:59 | |
| gibi_bus | stephenfin: yepp, try to get to Athlone | 16:59 |
| *** derekh has quit IRC | 17:00 | |
| sean-k-mooney | so there is a diffreenc between pmens and vifs | 17:00 |
| efried | mriedem: qualified +2 | 17:00 |
| * artom prepares for gibi_sheep | 17:00 | |
| sean-k-mooney | pmem namespace contain user data which we want to be very careful with ensuring it is earased | 17:00 |
| sean-k-mooney | we shoudl clean up vifs but if noting is plug into it it has little effect | 17:01 |
| sean-k-mooney | ideal neighter would fail | 17:01 |
| sean-k-mooney | but we shoudl make sure we dont skip the rest of the cleanup if they do | 17:01 |
| dansmith | artom: -1 | 17:01 |
| artom | For the sheep joke or the code? | 17:01 |
| dansmith | artom: for the plethora of typos | 17:02 |
| sean-k-mooney | stephenfin: efried so i think we shoudl catch the VPMEMCleanupFailed excetion or put the rest of the cleanup into a finally block | 17:02 |
| dansmith | I assumed you were spending lots of time wordsmithing and checking all that stuff | 17:02 |
| sean-k-mooney | stephenfin: efried or do the pmem cleanup last | 17:02 |
| sean-k-mooney | actullly no we want to always try it | 17:03 |
| *** awalende has joined #openstack-nova | 17:03 | |
| stephenfin | sean-k-mooney: You see what efried said about us already failing hard on other things though? | 17:03 |
| sean-k-mooney | yes and that has bit us over and over again | 17:03 |
| sean-k-mooney | this is one of the places we leak resouces on failed migrations | 17:03 |
| efried | seems like yet another thing we shouldn't start fixing as part of this series, but should do later when we can be more complete | 17:04 |
| sean-k-mooney | i guess | 17:04 |
| artom | dansmith, clearly not enough :( | 17:04 |
| efried | f man, if we try to fix everything ugly we notice or touch or get near, we'll never get anything done anywhere ever. | 17:05 |
| sean-k-mooney | what is the behavior when we raise we go to error then we clean it up later? | 17:05 |
| sean-k-mooney | i assume that woudl jsut fail again | 17:05 |
| dansmith | artom: fix those quick so I can apply my +2 and we can move on | 17:05 |
| mriedem | artom: https://review.opendev.org/#/c/634606/85 | 17:05 |
| mriedem | FUP needed but won't block | 17:05 |
| artom | Ack, thanks | 17:06 |
| *** awalende has quit IRC | 17:07 | |
| sean-k-mooney | efried: ok so we will just leave it raise. | 17:07 |
| efried | ++ | 17:08 |
| *** ociuhandu has quit IRC | 17:08 | |
| sean-k-mooney | efried: its not that i want to fix every ugly thing we see. i just dont want use adding more | 17:08 |
| efried | Yeah, I understand, but in this case it would entail reengineering the whole exception flow. | 17:08 |
| sean-k-mooney | as long as we dont return the namesapce to the pool of alocatable ones then its fine | 17:08 |
| sean-k-mooney | if we did it woudl be a CVE | 17:08 |
| efried | right, IIUC we're careful enough about that. | 17:09 |
| sean-k-mooney | even if you delete the vm | 17:09 |
| efried | I think in all cases where we could have leaked a pmem, we scrub it. | 17:09 |
| * mriedem sets timer for pmem cve | 17:09 | |
| mriedem | now that sean jinxed it | 17:09 |
| sean-k-mooney | if only the hardware did secure erase.... | 17:10 |
| efried | what was that unix util called? | 17:10 |
| efried | shred | 17:10 |
| sean-k-mooney | dban? | 17:10 |
| sean-k-mooney | shred is proably a thing too | 17:10 |
| efried | shred exists on my ubuntu. dban doesn't. | 17:11 |
| mriedem | shred is used in the lvm image backend code | 17:11 |
| artom | dansmith, you made me enable spell check for vim | 17:11 |
| sean-k-mooney | dban is not a utility its a thing you boot https://dban.org/ | 17:11 |
| efried | anyway, I would think a VM wouldn't be able to go looking at a low enough level to recover regular-deleted data. | 17:12 |
| sean-k-mooney | efried: of couse it can | 17:12 |
| efried | thought you needed specialized hardware for that | 17:12 |
| sean-k-mooney | its directly mapped into the guests adress speace | 17:12 |
| sean-k-mooney | it is byte adressable | 17:13 |
| efried | right, but if it's deleted | 17:13 |
| sean-k-mooney | that is why we need to 0 it out | 17:13 |
| efried | isn't that what daxio -z does?? | 17:13 |
| sean-k-mooney | efried: there is no filesystem so to delete it you have to write over it | 17:13 |
| sean-k-mooney | and yes | 17:14 |
| dansmith | artom: no, you did. | 17:14 |
| artom | dansmith, that was a good thing :) | 17:14 |
| dansmith | efried: uh what? | 17:14 |
| efried | sean-k-mooney: yeah, so what I'm saying is, having daxio -z'd the thing, you would need specialized hardware to try to uncover the ghost data | 17:14 |
| mriedem | artom: i've piled on | 17:14 |
| efried | dansmith: what what? | 17:14 |
| sean-k-mooney | efried: you would be surpriesed | 17:15 |
| dansmith | efried: unless you scrub every byte in what you hand to the guest, it can find it | 17:15 |
| efried | scrub meaning overwrite multiple times? | 17:15 |
| sean-k-mooney | efried: yes | 17:15 |
| efried | and... is that not a problem for regular ol ram and disk too? | 17:15 |
| sean-k-mooney | at least with magnetic media there was a bias that a singel over write was not enough | 17:15 |
| efried | do we srsly shred every byte of those between VMs? | 17:16 |
| dansmith | efried: certainly not for ram because it's paged in | 17:16 |
| sean-k-mooney | ram is not persetend so no power no data | 17:16 |
| dansmith | efried: the kernel can zero a page before it lets you read from it if you haven't written to it | 17:16 |
| dansmith | but with media that's not the case | 17:17 |
| dansmith | I dunno what the semantics of the daxio thing are, but in general you have to be careful about that stuff | 17:17 |
| efried | -z, --zero Zero the output device for len size, or the entire device if no length was provided. The output device must be a Device DAX device. | 17:17 |
| dansmith | and there have been exploits where even dropping the map could be circumvented by tricking the thing into granting you access to a region again without it being zeroed | 17:17 |
| efried | I mean, it doesn't say "Really really zero the output device" | 17:18 |
| efried | but still | 17:18 |
| efried | you would think it would be set up to prevent contamination | 17:18 |
| dansmith | efried: and for silicon, it almost never does because it costs money (i.e. wears the media) | 17:18 |
| efried | otherwise what good is it? | 17:18 |
| sean-k-mooney | efried: anyway were are side tracking | 17:18 |
| efried | agreed | 17:18 |
| dansmith | efried: you said "regular deleted data" above, which is what I took exception to | 17:19 |
| efried | ack | 17:19 |
| sean-k-mooney | right its specal hadware that needs special handeling | 17:19 |
| dansmith | but I will say, all the spectre stuff is about sussing out data via sidechannel which is not reading it direclty | 17:19 |
| dansmith | so let's not pretend that's not a thing :) | 17:19 |
| efried | Yeah, of course if you just trash the inode and don't actually zero the content | 17:19 |
| efried | I'm talking about: if you overwrite with zeros, but just once, don't you need something special (that VMs don't have) to read the ghost data? | 17:20 |
| dansmith | not necessarily | 17:20 |
| efried | okay | 17:20 |
| dansmith | you're thinking about low-level hardware detection of things, | 17:20 |
| dansmith | but that's not the only way to get access to that "ghost" data | 17:21 |
| dansmith | that's the whole point of sidechannel attacks | 17:21 |
| alex_xu | efried: do we have something to ignore a host for placement allocation_candidates call now? | 17:22 |
| efried | alex_xu, sean-k-mooney: looks like the CI fixups have merged now https://github.com/intel/Intel-OpenStack-CI-jobs/pull/2 | 17:22 |
| efried | We still have some more minor deltas to do right? | 17:22 |
| efried | alex_xu: you mean something like in_tree=!$uuid ? | 17:22 |
| alex_xu | yes | 17:22 |
| * efried looks | 17:22 | |
| sean-k-mooney | i havent done a full review but the inital change i have looked at seam correct | 17:23 |
| efried | alex_xu: I don't think so. cdent, we haven't done !in_tree yet, right? | 17:23 |
| efried | alex_xu: why? | 17:23 |
| efried | oh | 17:24 |
| alex_xu | efried: for his https://review.opendev.org/#/c/681383/5/nova/scheduler/manager.py@155 | 17:24 |
| alex_xu | if we have that, then we can ignore the src host for the same host resize | 17:24 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 17:24 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 17:24 |
| alex_xu | efried: emm...or we do a manually filter after that | 17:25 |
| sean-k-mooney | all the numa jobs just got killed | 17:25 |
| artom | mriedem, dansmith ^^ | 17:25 |
| efried | alex_xu: Isn't this the famous "doubled allocations" bug? | 17:26 |
| *** luksky has joined #openstack-nova | 17:26 | |
| alex_xu | efried: no, it isn't | 17:26 |
| alex_xu | efried: when a host enable same host resize, and that host is the only host has available PCPU now. | 17:27 |
| sean-k-mooney | alex_xu: cant ever ignore the source host for same host reszie or rebuild | 17:27 |
| *** ociuhandu has joined #openstack-nova | 17:27 | |
| efried | yeah, I would think for same-host resize we would want to restrict to *only* the current host | 17:27 |
| efried | ...having calculated the positive difference from our current allocations | 17:27 |
| alex_xu | efried: we will try to allocate PCPU in placement, and yes, placement tell you the source host has PCPU, but later the scheduler will check ignore host, the source host is in the ignore host. so the scheduling failed. | 17:27 |
| dansmith | efried: you don't know you're doing a same-host resize | 17:28 |
| efried | oh, bummer | 17:28 |
| alex_xu | sean-k-mooney: sorry, I mean the enable same host resize | 17:28 |
| alex_xu | oops | 17:28 |
| alex_xu | disalbe the same host resize | 17:28 |
| sean-k-mooney | same host resize with pcpus is way more involved | 17:28 |
| efried | so what we should really do is two GET /a_c calls | 17:28 |
| sean-k-mooney | well not pcpu numa in general | 17:28 |
| efried | one with in_tree=!$cur_host with the full set of resources | 17:29 |
| efried | one with in_tree=$cur_host with the diff'd set of resources | 17:29 |
| efried | and then combine the results. | 17:29 |
| mriedem | umm, you're not talking about adding that into the series for train now are you? | 17:29 |
| sean-k-mooney | that still has the double allocation proablem | 17:30 |
| alex_xu | dansmith: I know the ignored_host from request spec, so I can check that, if the allocation_candidates return the host in the ignore_hosts, then fallback to VCPU | 17:30 |
| efried | sean-k-mooney: how would it? | 17:30 |
| mriedem | are you essentially trying to say, avoid the same host for a resize if the instance has PCPU resources? | 17:30 |
| dansmith | alex_xu: you want to fall back to vcpu allocation if the only host that gets returned is the one you're on? | 17:30 |
| dansmith | alex_xu: that's....crazy | 17:30 |
| sean-k-mooney | the in_tree=$cur_host call would require double the resouces | 17:30 |
| sean-k-mooney | the current resouce and the resize to resouces | 17:31 |
| dansmith | alex_xu: or are you saying fall back to vcpu allocation but with the current host excluded? | 17:31 |
| sean-k-mooney | anyway the numa toplogy filter requires double the resouces so its not a new thing | 17:31 |
| efried | sean-k-mooney: no, that's why the in_tree=$cur_host is made with the *diff* of the current and new resources (only the positive ones) | 17:31 |
| *** ociuhandu has quit IRC | 17:31 | |
| efried | (this is all theoretical btw) | 17:31 |
| alex_xu | dansmith: fallback to vcpu if the only host get returned is the one you are on. After fallback vcpu, I won't get VCPu availalbe on the source host again | 17:31 |
| sean-k-mooney | efried: i mention this on irc last week | 17:32 |
| sean-k-mooney | in placment channel | 17:32 |
| sean-k-mooney | efried: i have a customer request to fix inplemce rebuild with numa toplogy | 17:32 |
| *** spsurya has quit IRC | 17:32 | |
| efried | sean-k-mooney: Yeah, I thought it sounded familiar. | 17:32 |
| sean-k-mooney | so i or someone on my team needs to actully make this work in U | 17:32 |
| cdent | i confirm that in_tree=! hasn't happened yet (sorry was away from computer for a few minutes doing network management) | 17:32 |
| sean-k-mooney | and backport somthi esle to queens | 17:32 |
| mriedem | sean-k-mooney: there is another nickel | 17:33 |
| *** brault has quit IRC | 17:33 | |
| mriedem | sean-k-mooney: remember when you said you'd make sriov pci device live migration claims part of the actual claims code in U too/ | 17:33 |
| mriedem | ? | 17:33 |
| sean-k-mooney | mriedem: yes but this time we have the customer wanting use to ship them the backported code in october | 17:33 |
| dansmith | U is going to be a big cycle.. HUGE. YUGE. | 17:33 |
| dtroyer | efried, sean-k-mooney, alex_xu: status update: we've re-triggered the pmem job on Intel CI with the changes on PS22, should have some results in about 45 min… | 17:33 |
| mriedem | sean-k-mooney: backported code for PCPU? | 17:33 |
| efried | dansmith: no it's not. | 17:33 |
| mriedem | including all of the upgrade shit that entails? | 17:33 |
| alex_xu | dtroyer: thanks | 17:33 |
| dtroyer | https://review.opendev.org/#/c/679640/22 | 17:33 |
| sean-k-mooney | mriedem: no inplace numa vma rebuild | 17:34 |
| efried | thanks dtroyer | 17:34 |
| mriedem | inplace numa vma rebuild? | 17:34 |
| efried | it appears that 'pleaselookagain' is the magic invocation for pmem CI | 17:34 |
| mriedem | i'm lost - i don't know what that has to do with what we're talking about | 17:34 |
| alex_xu | mriedem: https://review.opendev.org/#/c/681383/5/nova/scheduler/manager.py@155 | 17:34 |
| alex_xu | mriedem: or you can see the PS2 comment I give | 17:34 |
| *** brault has joined #openstack-nova | 17:35 | |
| *** brault has quit IRC | 17:35 | |
| *** brault has joined #openstack-nova | 17:35 | |
| sean-k-mooney | mriedem: https://bugzilla.redhat.com/show_bug.cgi?id=1700412 | 17:35 |
| efried | dansmith: If the PTL has anything to say about it, we're going to accept only blueprints that were already approved or proposed-but-didn't-make-spec-freeze | 17:35 |
| openstack | bugzilla.redhat.com bug 1700412 in openstack-nova "[RFE] Rebuild with a different image fails due to NUMATopologyFilter" [High,New] - Assigned to nova-maint | 17:35 |
| mriedem | "if same-host resize is disabled and the only host available is the host you're on there are resource problems" is not a new issue | 17:35 |
| *** jmlowe has joined #openstack-nova | 17:36 | |
| mriedem | ^ is also a problem for servers in a strict affinity group | 17:36 |
| mriedem | since you can't resize out of that | 17:36 |
| dansmith | efried: good luck with that | 17:36 |
| sean-k-mooney | mriedem: no its not a new proablem at all | 17:36 |
| efried | you can be part of the solution | 17:36 |
| mriedem | i think i tried that strategy back in like ocata | 17:36 |
| dansmith | efried: don't want to | 17:36 |
| alex_xu | mriedem: okay... | 17:37 |
| dansmith | efried: we've approved things I don't think we should do almost every cycle, so...limiting to doing things we've "approved" is counterproductive, IMHO | 17:37 |
| alex_xu | dansmith: yes, you can be :) | 17:37 |
| mriedem | sean-k-mooney: so https://bugs.launchpad.net/nova/+bug/1763766 ? | 17:38 |
| openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | 17:38 |
| artom | mriedem, fwiw, the SRIOV live migration thing can be me | 17:38 |
| alex_xu | dansmith: back to preivous, why I check ignore_host is crazy... | 17:38 |
| sean-k-mooney | mriedem: its related | 17:38 |
| dansmith | alex_xu: what I said is crazy is not what you are actually suggesting I think | 17:39 |
| mriedem | artom: i've resigned that it's never going to happen so was just using it as an example of another "we'll do it in U" thing | 17:39 |
| alex_xu | dansmith: ok...i'm safe | 17:39 |
| sean-k-mooney | for older release i was planing to check for that and allow skiping the numa topology filter if there were no changes in resouce usaage or numa toplogy on rebuild only. | 17:39 |
| artom | mriedem, I mean, I followed through on device tagging attachments, back in the day, didn't I? | 17:39 |
| alex_xu | stephenfin: so...check that https://review.opendev.org/#/c/681383/5/nova/scheduler/manager.py@155, if that is solution for you | 17:39 |
| artom | Though I did drop the ball in placement for osc live migration | 17:39 |
| mriedem | artom: you had a business justification for tagged attachments right? | 17:40 |
| artom | mriedem, - | 17:40 |
| dansmith | artom: you gonna wait for the last three weeks of the U cycle for that one too? | 17:40 |
| artom | Err, 0 | 17:40 |
| mriedem | customers aren't asking that you work on sriov pci device claims architecture | 17:40 |
| sean-k-mooney | mriedem: of corse we will only do dthat if we fix it upstream and block it upstream first | 17:40 |
| artom | It was purely a "let's be good community citizens" thing | 17:40 |
| mriedem | i'm not sure what "placement for osc live migration" is | 17:41 |
| stephenfin | alex_xu: I'm not sure you get you | 17:41 |
| artom | mriedem, err, I meant the osc live migration refactor | 17:41 |
| mriedem | stephenfin: obligatory smiley face please | 17:41 |
| mriedem | artom: i got that | 17:41 |
| mriedem | and the --boot-from-volume sugar | 17:41 |
| artom | I didn't even do the device tagging for it | 17:41 |
| artom | Which we *did* have an ask for | 17:41 |
| mriedem | well the good news is we have a whole big ass etherpad of osc gaps for nova | 17:42 |
| mriedem | https://etherpad.openstack.org/p/compute-api-microversion-gap-in-osc | 17:42 |
| luyao | stephenfin: could I just catch all exceptions, because we will print it later https://review.opendev.org/#/c/678455/25/nova/virt/libvirt/driver.py@1297 | 17:42 |
| mriedem | the bad news (for openstack) is my university mentoring project idea to work on that didn't get people | 17:42 |
| alex_xu | stephenfin: the issue, or the solution? | 17:42 |
| artom | mriedem, huh, that reminds me, we're apparently getting interns - or have signed up to get some, at any rate | 17:43 |
| stephenfin | so if there are no allocation candidates or if allocations_candidates == [current_host], fallback to VCPUs? | 17:43 |
| artom | We were wondering what to set them lose on | 17:43 |
| artom | OSC could be a good idea | 17:43 |
| alex_xu | stephenfin: if allocation_candidates == ignored_hosts, fallback to VCPUs | 17:43 |
| artom | The other idea was test automation stuff with tempest/whitebox, since the learning curbe isn't as steep as nova | 17:43 |
| artom | *curve | 17:43 |
| stephenfin | gotcha | 17:43 |
| stephenfin | I can do that (y) | 17:43 |
| stephenfin | alex_xu: Can I do it as a follow-up though? | 17:44 |
| stephenfin | because that patch is already huge and it's a corner case | 17:44 |
| stephenfin | I'll have it done first thing in the morning | 17:44 |
| alex_xu | stephenfin: sure, or I can help that in my morning | 17:45 |
| stephenfin | sweet | 17:45 |
| *** cdent has quit IRC | 17:45 | |
| stephenfin | I'm just moving your quotas patch further down the series | 17:46 |
| stephenfin | should be done shortly then I'll push it up | 17:46 |
| alex_xu | cool, I thought it should be adjust the order also | 17:46 |
| alex_xu | anyway, see you guys tomorrow | 17:46 |
| stephenfin | o/ | 17:46 |
| efried | stephenfin: did you see luyao's question about https://github.com/intel/Intel-OpenStack-CI-jobs/pull/2 ? | 17:47 |
| efried | s,that link,https://review.opendev.org/#/c/678455/25/nova/virt/libvirt/driver.py@1297, | 17:47 |
| mriedem | https://review.opendev.org/#/q/topic:bp/numa-aware-live-migration+status:open+label:Code-Review=2+project:openstack/nova | 17:47 |
| mriedem | artom: dansmith: sean-k-mooney: ^ | 17:47 |
| mriedem | donnyd: ^ | 17:48 |
| efried | \o/ | 17:48 |
| artom | zomgies | 17:48 |
| mriedem | success bot? | 17:48 |
| efried | ffs wait until they merge | 17:49 |
| mriedem | psh | 17:49 |
| dansmith | mriedem: praise be | 17:49 |
| sean-k-mooney | :) | 17:49 |
| mriedem | gibi_bus: are you going to get https://review.opendev.org/#/c/676980/ updated today or should i work on that? | 17:49 |
| gibi_bus | mriedem: I'm about to finish the rework | 17:50 |
| gibi_bus | mriedem: I will ping you when it is up | 17:50 |
| * gibi_bus has a long day today | 17:50 | |
| mriedem | ok, then i'm going to get lunch | 17:51 |
| gibi_bus | mriedem: have a nice one! | 17:52 |
| * mriedem goes to look for leftovers in the fridge | 17:53 | |
| * gibi_bus has a bit of pastry for dinner | 17:53 | |
| *** dolpher has joined #openstack-nova | 17:55 | |
| efried | o/ dolpher :) | 18:03 |
| * efried feeds face | 18:04 | |
| * donnyd waves to dolpher | 18:07 | |
| dolpher | efried: Hello, seems the filter is not working, no one is triggering the CI job now | 18:07 |
| dolpher | Hi donnyd :) | 18:07 |
| *** Garyx has quit IRC | 18:12 | |
| *** mdbooth has quit IRC | 18:16 | |
| *** Garyx has joined #openstack-nova | 18:16 | |
| donnyd | sean-k-mooney: I didn't hear back on the labels.. but i hope this https://review.opendev.org/#/c/634827/ isn't waiting on jobs for FN | 18:19 |
| donnyd | I submitted a bit more for resources, kinda need it right now | 18:19 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | 18:20 |
| gibi_bus | mriedem: ^^ | 18:20 |
| gibi_bus | bauzas: for tomorrow's coffee ^^ | 18:21 |
| *** panda|ruck is now known as panda|ruck|off | 18:21 | |
| sean-k-mooney | donnyd: no but the jobs got canceld | 18:22 |
| sean-k-mooney | the 4 build i had in flight got killed because of a rebase/fix of the patches | 18:22 |
| sean-k-mooney | i can start them again and see what happens | 18:22 |
| openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | 18:24 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738 | 18:24 |
| sean-k-mooney | donnyd: just rebased those via the gerrit ui that shoudl run with the old and new lable | 18:25 |
| donnyd | sean-k-mooney: ok cool | 18:26 |
| *** maciejjozefczyk has quit IRC | 18:29 | |
| efried | dolpher: Can we kill the filter for now? | 18:39 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 18:39 |
| dolpher | https://github.com/intel/Intel-OpenStack-CI-jobs/pull/4 | 18:40 |
| dolpher | efried: send the pull request to remove the filter, and I'll ping Matt to merge it. | 18:41 |
| efried | okay | 18:41 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | 18:42 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | 18:44 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extract pf$N literals as constants from func test https://review.opendev.org/680991 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 18:52 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 18:52 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve dest service level func tests https://review.opendev.org/680998 | 18:54 |
| *** eharney has quit IRC | 18:55 | |
| eandersson | Having an odd bug - a VM can't be rebuilt with a new image. Pretty sure placement is filtering out all the potential matches. | 18:56 |
| eandersson | Rebuilding it with the same image works fine. | 18:57 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ib50b6b02208f5bd2972de8a6f8f685c19745514c https://review.opendev.org/681490 | 18:57 |
| mriedem | eandersson: which release? | 18:57 |
| eandersson | Running Rocky | 18:57 |
| eandersson | Very latest stable/rocky | 18:57 |
| eandersson | I see that the scheduler gets a single result, but it is failing on NUMA Topology. | 18:58 |
| mriedem | we shouldn't be calling placement for rebuild https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/manager.py#L126 | 18:58 |
| eandersson | > 'NUMATopologyFilter: (start: 1, end: 0)' | 18:58 |
| eandersson | Ah bad assumption | 18:58 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1763766 ? | 18:58 |
| openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | 18:58 |
| mriedem | or https://bugzilla.redhat.com/show_bug.cgi?id=1700412 | 18:59 |
| openstack | bugzilla.redhat.com bug 1700412 in openstack-nova "[RFE] Rebuild with a different image fails due to NUMATopologyFilter" [High,New] - Assigned to nova-maint | 18:59 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | 18:59 |
| eandersson | Yep looks like it | 18:59 |
| dansmith | yeah if the image is what causes it trouble, | 18:59 |
| dansmith | it's likely the new numa topo filtering out the one host you're on | 19:00 |
| mriedem | sean-k-mooney has to fix it and get that backported to queens | 19:00 |
| * dansmith resists making an unhelpful comment about numa as a feature | 19:00 | |
| artom | mriedem, it's not getting backported, but one of the things we discussed was making that RUN_ON_REBUILD flag we added way back when for the rebuild CVE configurable in some way | 19:02 |
| artom | We kinda dreaded coming to upstream with that | 19:02 |
| *** brault has quit IRC | 19:02 | |
| sean-k-mooney | dansmith: for the downstream issue yes its just the numa toploy filter filtering it out beacue tehre is no space for the rebuild | 19:02 |
| dansmith | sean-k-mooney: I'm talking about eandersson's case | 19:03 |
| mriedem | artom: we don't claim on rebuild so if the numa topo changes in the new image we won't claim for it properly, | 19:03 |
| mriedem | so you could just be kicking the error down to compute | 19:03 |
| sean-k-mooney | ill read scollback | 19:03 |
| sean-k-mooney | mriedem: correct | 19:03 |
| artom | mriedem, yeah, it wouldn't be a correct fix | 19:03 |
| sean-k-mooney | we rebuild with the new image but old toplogy | 19:04 |
| artom | More of a candy we throw our customers on, like, 10 to placate them until they an upgrade to whatever release the un-backportable fix lands in | 19:04 |
| mriedem | sean-k-mooney: you mean in some downstream-only patch? | 19:04 |
| sean-k-mooney | mriedem: no updstream if you trun off the numa toplogy filter to get passed it | 19:04 |
| sean-k-mooney | when we rebuild but the toplogy change via the image | 19:05 |
| mriedem | oh | 19:05 |
| sean-k-mooney | we end up with the old numa toplogy and new image | 19:05 |
| sean-k-mooney | so i want to put an upstream check to prevent that | 19:05 |
| mriedem | ok so hear me out, | 19:05 |
| sean-k-mooney | and backport a config option to opt out | 19:05 |
| mriedem | the rebuild api takes new metadata for the server, | 19:05 |
| dansmith | no | 19:06 |
| dansmith | no | 19:06 |
| mriedem | what if we have a skip_numa_topology_filter metadata that the user can pass in?! | 19:06 |
| mriedem | and then the filter checks for that | 19:06 |
| mriedem | and it's great | 19:06 |
| dansmith | noooooooooo | 19:06 |
| mriedem | problem solved | 19:06 |
| * mriedem exits the community with a great idea | 19:06 | |
| mriedem | ok so the image takes random metadata, | 19:06 |
| mriedem | so hear me out | 19:06 |
| mriedem | os_skip_numa_filter_but_only_for_rebuild=true | 19:06 |
| mriedem | we'll fix it in U | 19:07 |
| dansmith | Well, officer, you should have heard what he was suggesting before I shot him... Hear me out... | 19:07 |
| mriedem | that's justifiable homicide in some states | 19:07 |
| dansmith | yeah | 19:07 |
| artom | We fix it? No, U fix it | 19:07 |
| sean-k-mooney | well the custromer has validated that tehy never us the image to change numa toplogy | 19:07 |
| mriedem | get in line with bauzas and the bad jokes | 19:07 |
| sean-k-mooney | so for them that skip on rebuild is valid | 19:08 |
| sean-k-mooney | but i have aded https://bugs.launchpad.net/nova/+bug/1763766 to the downstrema bug | 19:08 |
| openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | 19:08 |
| sean-k-mooney | so what i am going to propose is i implemnt a check for that | 19:08 |
| *** eharney has joined #openstack-nova | 19:08 | |
| sean-k-mooney | and if and only if that passes we skip the numa toplogy filter | 19:08 |
| sean-k-mooney | becasue not numa things changed | 19:08 |
| sean-k-mooney | on rebuild only | 19:09 |
| sean-k-mooney | and like the check we have to prevent numa migratio nwe can backport the saftey check for no resouce change on rebuild | 19:09 |
| sean-k-mooney | and give peole a workaofund config option to disable it if they want to shot them selves in the foot | 19:10 |
| sean-k-mooney | i have aslo said that if we dont fix it upstream im not fixing it downstream only | 19:11 |
| sean-k-mooney | mriedem: dansmith does ^ sound fair. fixing https://bugs.launchpad.net/nova/+bug/1763766 is a prequisite for fixing https://bugs.launchpad.net/nova/+bug/1804502 safely | 19:13 |
| openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow resource consumption changes on image rebuild" [Medium,Triaged] | 19:13 |
| openstack | Launchpad bug 1804502 in OpenStack Compute (nova) "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Undecided,In progress] - Assigned to David Hill (david-hill-ubisoft) | 19:13 |
| eandersson | Would it be possible to add an option to allow rebuilds to re-schedule the VM? | 19:14 |
| eandersson | something like openstack server rebuild <id> --migrate/reschedule/bla | 19:14 |
| sean-k-mooney | like evauate | 19:14 |
| eandersson | I can't reproduce this as well | 19:14 |
| sean-k-mooney | rebuild is ment to be posiblein virt driver that dont support migration | 19:15 |
| sean-k-mooney | there is noting in principal that would disallow a driver form having rebiuld be to a different host like a cold migration but it is not how its currelty impelmeted | 19:16 |
| sean-k-mooney | we would have to change the noop claim to a move claim if the inplcem rebuild did not succesed | 19:16 |
| sean-k-mooney | or expose it at the api like you suggest and make the resuling to a different host user contolerale | 19:17 |
| sean-k-mooney | ingoring numa for a second since image can have tratis we need to check with placmenet in anycase to see it the traits on the new image are valid on the current host | 19:18 |
| sean-k-mooney | eandersson: what is the behavior you would like to see | 19:20 |
| eandersson | Better error message at least :p | 19:21 |
| eandersson | But not sure what the ideal would be. | 19:21 |
| efried | stephenfin, sean-k-mooney, luyao, alex_xu: I'm +2 on everything vpmem except: | 19:21 |
| efried | - https://review.opendev.org/#/c/678455/ where I'll proxy a vote from sean-k-mooney | 19:21 |
| efried | - the bottom where I'll flip from -2 to +2 when we're ready | 19:21 |
| eandersson | Maybe just documentation on how to avoid this on image builds. | 19:21 |
| efried | I'm now going to sleep. | 19:22 |
| eandersson | It would be nice as an admin to have a way to fix it at least. | 19:22 |
| sean-k-mooney | eandersson: well you cant avoid it if the instance has a numa toplogy | 19:22 |
| sean-k-mooney | you can work around it via cold/live migation | 19:22 |
| sean-k-mooney | but that it | 19:22 |
| *** efried is now known as efried_zzz | 19:22 | |
| eandersson | Maybe always force reschedule on rebuild when numa is a thing | 19:23 |
| sean-k-mooney | rebuild are not allowed to change host today | 19:23 |
| sean-k-mooney | so there is no rescudler | 19:23 |
| sean-k-mooney | whats actully happening is the numa toplogy filter is treating the rebuild like a new isntance spawn | 19:24 |
| sean-k-mooney | so you need double the resouces | 19:24 |
| eandersson | oh so that is why some of them worked | 19:24 |
| sean-k-mooney | yep | 19:24 |
| sean-k-mooney | if the instance does not have a numa toplogy or there is enough space it works | 19:24 |
| sean-k-mooney | we can actully now tell if its a rebuild in the filter code but we have not added special handeling for that in the numa toplogy filter | 19:25 |
| sean-k-mooney | the correct thign to do is to check if the numa topldoy woudl change. reject the rebuild if its true or skip the numa toplogy filter if not | 19:26 |
| sean-k-mooney | that is more or less want i want to code up | 19:26 |
| sean-k-mooney | if posible i would like to do the "rebuild chage numa toplogy" check at the api and return an expcti error | 19:28 |
| sean-k-mooney | but i have not figured out all the pices yet | 19:28 |
| *** artom has quit IRC | 19:28 | |
| mriedem | sean-k-mooney: if rebuild + new image + numa sometimes works if the new image numa topo fits on the same host, why wouldn't you just let the scheduler still try to fit it? | 19:31 |
| mriedem | and only outright skip if the image is new + rebuild + none of the numa properties changed | 19:31 |
| sean-k-mooney | mriedem: we dont update the instance numa toplogy in the db | 19:31 |
| sean-k-mooney | because that is dont in the claim | 19:31 |
| mriedem | and we don't claim | 19:31 |
| sean-k-mooney | and we do a noop claim | 19:31 |
| mriedem | yeah ok | 19:31 |
| sean-k-mooney | so if we fixed that too then sure | 19:31 |
| mriedem | so like new image + volume-backed server we just fail fast in the api today | 19:31 |
| sean-k-mooney | thats becasue we dont supprot volume backed rebuild in general? | 19:32 |
| mriedem | no, | 19:32 |
| mriedem | it's because we don't replace the root volume with the new image | 19:32 |
| mriedem | there are approved cinder/nova specs to support all of that in the past and cinder did a bunch of work to plumb a new api for us, but the nova side changes all stalled | 19:33 |
| mriedem | gibi_bus: did you go to bed yet? | 19:33 |
| sean-k-mooney | this liekly will become a spec in anycase. but what i wanted to do was first do the api/filer check to block it. then explore either updating the topology on rebuld and validatin git or otherwise having a more ocmplet fix | 19:34 |
| sean-k-mooney | if it could fix then i would liek to allow it | 19:35 |
| sean-k-mooney | but only if the toploy you ask for in the image is what you actlly end up with | 19:35 |
| sean-k-mooney | which is not what would happen without updating the laim | 19:35 |
| sean-k-mooney | *claim | 19:35 |
| *** artom has joined #openstack-nova | 19:37 | |
| sean-k-mooney | the otherthing with allowing the toplogy to change on a rebuild would be keeping placmenet in sysnc if/when we model numa in placmenet | 19:37 |
| sean-k-mooney | i dont want to complicate numa in plamcent more by having to deal with this at the same time | 19:38 |
| mriedem | i could use some opinions on alternatives i've proposed for some new error handling during prep_resize in gibi's bottom unapproved change https://review.opendev.org/#/c/676980/20/nova/compute/manager.py@4529 which is the only thing holding it up right now | 19:40 |
| mriedem | since i'll probably be the one implementing them today so we can still meet FF for the bw provider migration stuff | 19:40 |
| *** CeeMac has quit IRC | 19:41 | |
| mriedem | sean-k-mooney: yeah i'd focus on whatever you can that is backportable first | 19:42 |
| sean-k-mooney | yep thats the plan. its in our team backlog for U and ill likely volenterr to be the person to fix it so im going to fix what can be backport first then consider how to imporve it later. | 19:43 |
| sean-k-mooney | thanks for the link to the other bug by the way | 19:43 |
| sean-k-mooney | i forgot that was still open | 19:44 |
| openstackgerrit | Merged openstack/nova master: Make _revert_allocation nested allocation aware https://review.opendev.org/676138 | 19:45 |
| sean-k-mooney | im jsut looking at https://review.opendev.org/#/c/676980/20/nova/compute/manager.py@4529 | 19:45 |
| sean-k-mooney | i dont really have context right now but what alternitve were you tinking of | 19:45 |
| *** eharney has quit IRC | 19:45 | |
| mriedem | they are listed below | 19:49 |
| sean-k-mooney | ok ill keep reading | 19:49 |
| mriedem | gibi added code to the prep_resize flow that can result in a BuildAbortException getting raised and in that case we shouldn't try to reschedule | 19:50 |
| mriedem | he added handling for BuildAbortException, | 19:50 |
| sean-k-mooney | the finally block will be run by the way when we catch the BuildAbortException | 19:50 |
| sean-k-mooney | i was just comming that in gerrit | 19:50 |
| mriedem | but was missing some things we do when we don't reschedule from a prep_resize failure, like recording a fault and sending an error notification | 19:50 |
| *** henriqueof has joined #openstack-nova | 19:51 | |
| *** nweinber__ has quit IRC | 19:53 | |
| *** panda|ruck|off has quit IRC | 19:55 | |
| *** gibi_bus is now known as gibi | 19:55 | |
| * gibi reached the hotel | 19:55 | |
| gibi | mriedem: I'm not sleeping yet but I'm already short on brainpower | 19:56 |
| mriedem | gibi: ok if you can look at https://review.opendev.org/#/c/676980/20 before going to bed i can try to update it | 19:56 |
| mriedem | but it's looking non-trivial | 19:56 |
| mriedem | otherwise i'll just move up the series since those changes look simple | 19:57 |
| *** panda has joined #openstack-nova | 19:57 | |
| gibi | now I read your comments. it seems non trivial indeed. Would it be too bad letting the re-schedule happen as a first solution. And optimizing out the unnecessary rechedule later? | 19:59 |
| gibi | mriedem: I cannot implement the functional test and the proper cleanup today. I can do that tomorrow morning. Or If you able to implement it then I can try to review it | 20:01 |
| sean-k-mooney | donnyd: the new lable job passed https://review.opendev.org/#/c/680738/ so im going to call it a day | 20:02 |
| gibi | I will be back in 15 minutes | 20:03 |
| mriedem | gibi: i could let the BuildAbortException handling slide to a follow up but the fact that i can comment out the new code and none of the functional tests fail is kind of a blocker for me | 20:05 |
| gibi | mriedem: ack, that was a surprise for me too. | 20:05 |
| gibi | mriedem: I suggest you move forward with the reviews and I will respin this in the morning with a proper functional test | 20:06 |
| gibi | mriedem: as I run out of time today and I don't want to ask you to troubleshoot those functional tests | 20:07 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 20:07 |
| *** BjoernT has quit IRC | 20:08 | |
| mriedem | gibi: yup sounds like a plan | 20:08 |
| openstackgerrit | Merged openstack/nova master: Nova object changes for forbidden aggregates request filter https://review.opendev.org/671072 | 20:11 |
| *** spatel has quit IRC | 20:18 | |
| gibi | shover thoughts: the computes in those tests are symmetrical. they have the same PF setup. PFs are named the same on host1 and on host2 and we are "updating" the PF name in the PCI request. | 20:21 |
| gibi | when we migrate from host1 to host2 | 20:21 |
| gibi | so I can imagine that without the update the PF names remains the same | 20:22 |
| gibi | and everything works | 20:22 |
| donnyd | sean-k-mooney: woot woot | 20:22 |
| *** mvkr has quit IRC | 20:27 | |
| *** markvoelker has quit IRC | 20:27 | |
| *** mvkr has joined #openstack-nova | 20:32 | |
| *** markvoelker has joined #openstack-nova | 20:36 | |
| *** factor has quit IRC | 20:40 | |
| *** markvoelker has quit IRC | 20:41 | |
| *** factor has joined #openstack-nova | 20:42 | |
| openstackgerrit | Merged openstack/nova master: Improve SEV documentation and other minor tweaks https://review.opendev.org/681254 | 20:43 |
| openstackgerrit | Merged openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 20:43 |
| *** BjoernT has joined #openstack-nova | 20:47 | |
| gibi | I was able to disprove my shover thoughts. | 20:48 |
| gibi | I'm going to sleep now. Talk to you tomorrow | 20:48 |
| gibi | and I disproved my disproval | 20:49 |
| *** ociuhandu has joined #openstack-nova | 20:50 | |
| * gibi kick himself off the net | 20:50 | |
| openstackgerrit | Merged openstack/python-novaclient master: doc: Add support microversions for options https://review.opendev.org/681174 | 20:51 |
| *** BjoernT has quit IRC | 20:51 | |
| *** BjoernT_ has joined #openstack-nova | 20:51 | |
| openstackgerrit | Merged openstack/os-traits master: Build pdf docs https://review.opendev.org/681465 | 20:53 |
| openstackgerrit | Merged openstack/os-traits master: Update README to be a bit more clear https://review.opendev.org/681237 | 20:53 |
| eandersson | thanks for the help sean-k-mooney mriedem | 20:56 |
| eandersson | sorry had to walk away earlier for a meeting | 20:56 |
| *** ociuhandu has quit IRC | 20:57 | |
| aspiers | #success AMD SEV support finally landed in nova after a year https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-kvm.html#amd-sev-secure-encrypted-virtualization | 20:59 |
| openstackstatus | aspiers: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 20:59 |
| mriedem | eandersson: yw, though i'm not sure how helpful it was | 20:59 |
| mriedem | at least you know what the problem is | 21:00 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Counting both of VCPU and PCPU as core quota https://review.opendev.org/681374 | 21:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 21:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Make 'Connection.getHostname' unique https://review.opendev.org/681060 | 21:02 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Mock 'libvirt_utils.file_open' properly https://review.opendev.org/681061 | 21:02 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 21:02 |
| aspiers | #thanks efried, sean-k-mooney, stephenfin, kashyap, bbobrov, and many others for helping land AMD SEV support in Nova! | 21:03 |
| openstackstatus | aspiers: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 21:03 |
| *** ivve has quit IRC | 21:05 | |
| *** ivve has joined #openstack-nova | 21:05 | |
| *** slaweq has quit IRC | 21:10 | |
| *** slaweq has joined #openstack-nova | 21:11 | |
| mriedem | i'm all the way up through the bw migration series and barring some catastrophe i think it'll probably all be +2ed from me tomorrow | 21:15 |
| *** rcernin has joined #openstack-nova | 21:15 | |
| mriedem | ooo the thanks page | 21:15 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Include both VCPU and PCPU in core quota count https://review.opendev.org/681374 | 21:17 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 21:17 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Make 'Connection.getHostname' unique https://review.opendev.org/681060 | 21:17 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Mock 'libvirt_utils.file_open' properly https://review.opendev.org/681061 | 21:17 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 21:17 |
| mriedem | #thanks sean-k-mooney and donnyd for getting numa-aware live migration CI testing going on FortNebula nodes | 21:17 |
| openstackstatus | mriedem: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 21:17 |
| donnyd | np mriedem | 21:17 |
| *** slaweq has quit IRC | 21:18 | |
| donnyd | mriedem: anything else that can be done on the provider end you can think of... lmk and i will do my best | 21:19 |
| *** boxiang has joined #openstack-nova | 21:19 | |
| *** zhubx has quit IRC | 21:20 | |
| mriedem | dansmith: remember this multi-cell floating IP re-associate bug fix https://review.opendev.org/#/c/656594/ ? can we get that re-approved? | 21:26 |
| mriedem | i think the only thing since it was last approved was fixing some words in the test and removing the handling for CantStartEngineError | 21:27 |
| *** slaweq has joined #openstack-nova | 21:27 | |
| *** slaweq has quit IRC | 21:32 | |
| *** zer0c00l has joined #openstack-nova | 21:36 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use multiple attachments in test_list_volume_attachments https://review.opendev.org/681618 | 21:43 |
| donnyd | can someone help me understand how the api-paste.ini works? | 21:44 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use multiple attachments in test_list_volume_attachments https://review.opendev.org/681618 | 21:44 |
| donnyd | for reasons out of my control i need to run an api to include an additional url | 21:44 |
| dansmith | mriedem: jes | 21:44 |
| donnyd | this isn't really for nova... but i am hoping the api-paste works pretty much the same across projects | 21:45 |
| *** takashin has joined #openstack-nova | 21:46 | |
| mriedem | donnyd: the rest all likely copied from nova originally so it should probably work similarly | 21:46 |
| donnyd | so i need to add /image in at the front of the request for glance | 21:47 |
| donnyd | and with the other service i just use apache and that works like a boss | 21:48 |
| mriedem | would that be api-paste.ini or the endpoint url in the service catalog? | 21:48 |
| donnyd | well both | 21:48 |
| mriedem | probably best to ask in #openstack-glance | 21:48 |
| mriedem | oh right glance is yet to support wsgi | 21:48 |
| mriedem | it's all mtreinish's fault | 21:48 |
| donnyd | i did | 21:48 |
| mriedem | maybe #openstack-operators? | 21:49 |
| mriedem | glance room is pretty dead usually | 21:49 |
| donnyd | oh i will give that a swing | 21:49 |
| mriedem | or hunt down rosmaita | 21:49 |
| mriedem | he's in -cinder | 21:49 |
| mriedem | hiding as the cinder ptl | 21:49 |
| donnyd | adding /image in the api-paste works right now | 21:50 |
| donnyd | but the ref links don't come back with the url in them | 21:50 |
| mriedem | ref links from glance or from nova? | 21:51 |
| mriedem | b/c https://docs.openstack.org/nova/latest/configuration/config.html#api.glance_link_prefix :( | 21:51 |
| *** henriqueof1 has joined #openstack-nova | 21:51 | |
| mriedem | i think ^ might only matter if you're using the compute api as a proxy to glance, | 21:52 |
| mriedem | which tools shouldn't be doing anymore | 21:52 |
| *** henriqueof has quit IRC | 21:52 | |
| *** TxGirlGeek has quit IRC | 21:53 | |
| * mriedem has to run | 21:53 | |
| *** mriedem is now known as mriedem_afk | 21:53 | |
| *** hemna has joined #openstack-nova | 21:58 | |
| *** markvoelker has joined #openstack-nova | 21:59 | |
| *** panda has quit IRC | 22:00 | |
| *** panda has joined #openstack-nova | 22:03 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| stephenfin | melwitt: If you're still around, I've addressed your comments on https://review.opendev.org/#/c/681374/ | 22:12 |
| stephenfin | You can find the functional tests in the following patch that turns everything on https://review.opendev.org/#/c/671801/45/nova/tests/functional/libvirt/test_numa_servers.py@241 and https://review.opendev.org/#/c/671801/45/nova/tests/functional/libvirt/test_numa_servers.py@329 | 22:13 |
| melwitt | stephenfin: ack, thanks | 22:14 |
| *** slaweq has quit IRC | 22:16 | |
| stephenfin | and for dansmith, I've got the snazzy "try and try again" scheduler thing here itching for some red pen https://review.opendev.org/#/c/671801 | 22:16 |
| *** mlavalle has quit IRC | 22:17 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add TODO note for mox removal https://review.opendev.org/576758 | 22:25 |
| *** avolkov has quit IRC | 22:31 | |
| *** TxGirlGeek has joined #openstack-nova | 22:32 | |
| *** TxGirlGeek has quit IRC | 22:33 | |
| *** threestrands has joined #openstack-nova | 22:37 | |
| *** luksky has quit IRC | 22:38 | |
| *** TxGirlGeek has joined #openstack-nova | 22:40 | |
| stephenfin | I haven't seen anything stream by from Gerrit in quite some time :-\ | 22:47 |
| stephenfin | Guess that's as good a sign as any to call it | 22:48 |
| sean-k-mooney | this is several hours past when you normally do | 22:48 |
| *** mtreinish has quit IRC | 22:54 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Re-add some useful checks https://review.opendev.org/681626 | 22:54 |
| *** dtruong has quit IRC | 22:55 | |
| *** mtreinish has joined #openstack-nova | 22:55 | |
| *** dtruong has joined #openstack-nova | 22:56 | |
| openstackgerrit | Merged openstack/nova master: vCPU model selection https://review.opendev.org/670298 | 22:59 |
| *** macz has quit IRC | 23:01 | |
| *** tkajinam has joined #openstack-nova | 23:04 | |
| *** hemna has quit IRC | 23:04 | |
| *** slaweq has joined #openstack-nova | 23:11 | |
| openstackgerrit | Merged openstack/nova master: Add compatibility checks for CPU mode and CPU models and extra flags https://review.opendev.org/670299 | 23:15 |
| *** slaweq has quit IRC | 23:15 | |
| openstackgerrit | Merged openstack/nova master: Support reporting multi CPU model traits https://review.opendev.org/670300 | 23:16 |
| *** igordc has quit IRC | 23:18 | |
| *** BjoernT_ has quit IRC | 23:18 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add 'InstanceNUMATopology.cpu_pinning' property https://review.opendev.org/680106 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Use sane indent https://review.opendev.org/680229 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add 'NUMACell.pcpuset' field https://review.opendev.org/680108 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 23:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 23:32 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Include both VCPU and PCPU in core quota count https://review.opendev.org/681374 | 23:32 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 23:32 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Make 'Connection.getHostname' unique https://review.opendev.org/681060 | 23:32 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Mock 'libvirt_utils.file_open' properly https://review.opendev.org/681061 | 23:32 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 23:32 |
| *** igordc has joined #openstack-nova | 23:33 | |
| alex_xu | good morning | 23:35 |
| *** BjoernT has joined #openstack-nova | 23:39 | |
| *** igordc has quit IRC | 23:40 | |
| *** igordc has joined #openstack-nova | 23:42 | |
| *** hemna has joined #openstack-nova | 23:42 | |
| *** BjoernT has quit IRC | 23:48 | |
| *** mriedem_afk has quit IRC | 23:49 | |
| openstackgerrit | Merged openstack/nova master: Fix the incorrect powershell command https://review.opendev.org/679588 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
