| *** felipemonteiro_ has joined #openstack-nova | 00:00 | |
| mnaser | hey | 00:04 |
|---|---|---|
| mnaser | fun bug time | 00:04 |
| *** suresh12 has quit IRC | 00:04 | |
| *** suresh12 has joined #openstack-nova | 00:04 | |
| mnaser | if a user builds an instance that if boot from volume (where nova creates the volume) and nova-compute fails to create the volume, the vm build fails, and it actually counts as a 'consecutive build failures' | 00:05 |
| *** suresh12 has quit IRC | 00:05 | |
| mnaser | so if i'm $bad_user and i try to launch 40 instances but keep hitting quota limits but keep trying again, i will effectively slowly but surely disable every compute node in the cloud | 00:05 |
| *** suresh12 has joined #openstack-nova | 00:05 | |
| mnaser | so while that feature is nice, i think we have to find a way to identify the type of failure it is | 00:06 |
| mnaser | i mean we could disable that feature but that feels dirty | 00:08 |
| *** harlowja has quit IRC | 00:08 | |
| *** suresh12 has quit IRC | 00:11 | |
| *** suresh12 has joined #openstack-nova | 00:11 | |
| *** fragatina has quit IRC | 00:13 | |
| *** fragatina has joined #openstack-nova | 00:14 | |
| *** zhurong has quit IRC | 00:14 | |
| *** mriedem has joined #openstack-nova | 00:14 | |
| mriedem | mnaser: yeah https://bugs.launchpad.net/nova/+bug/1742102 | 00:14 |
| openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [Undecided,Confirmed] - Assigned to jichenjc (jichenjc) | 00:14 |
| mriedem | there is an ops ML thread about similar issues | 00:14 |
| mriedem | i think we likely need to consider a whitelist of acceptable, not threshold inducing exceptions | 00:14 |
| mriedem | the latest comments in there from jichen are actually the exact issue you're describing | 00:15 |
| *** mriedem_afk has quit IRC | 00:16 | |
| mriedem | but we can also be smarter and check quota before trying to create volumes | 00:17 |
| mriedem | like we do for ports | 00:17 |
| mnaser | mriedem: yeah, i think that's better, do we check quota at api layer for ports? | 00:17 |
| mriedem | could do that in conductor so we don't block the API response | 00:17 |
| mriedem | mnaser: yes | 00:17 |
| mriedem | it's the validate_networks call | 00:17 |
| mnaser | that sounds reasonable and backport-able too i think | 00:17 |
| mriedem | based on the number of requested networks and instances, we check quota | 00:17 |
| mnaser | the thing is if i have a quota of 10 volumes, and i launch a 100 instances (one by one), i might still run into that issue i guess | 00:18 |
| mriedem | yeah - either way you'd fail after the 202 you get from the API | 00:18 |
| mriedem | you either fail in conductor or you fail in compute | 00:18 |
| mriedem | sure but we don't disable your compute :) | 00:18 |
| *** fragatina has quit IRC | 00:18 | |
| mnaser | because of the race as conductor still sees you using 0 volumes but as computes start creating volumes | 00:18 |
| mriedem | oh, sure | 00:19 |
| mriedem | we could also just handle volume quota issues in the compute as a whitelist of things to not disable the compute | 00:19 |
| mriedem | multi-part fixes | 00:19 |
| mnaser | i think both out-of-ports and out-of-volumes both seem like reasonable 'skip' failures | 00:19 |
| mriedem | yeah | 00:20 |
| mriedem | the bug above started as a port / fixed ips quota issue | 00:20 |
| mriedem | but also extends to volumes | 00:20 |
| mnaser | i think any api exception should be skipped tbh | 00:20 |
| mnaser | because that means the compute node is fine | 00:20 |
| *** sdague has quit IRC | 00:20 | |
| mnaser | sure it sounds vague but if your cinder is not having a good time then you're slowly disabling all compute nodes | 00:20 |
| mriedem | well, depends on what it is, | 00:20 |
| *** suresh12 has quit IRC | 00:21 | |
| mriedem | if nova.conf is misconfigured on the compute to talk to neutron, that's a different problem | 00:21 |
| mriedem | likely a 403 | 00:21 |
| mnaser | ah yes | 00:21 |
| *** suresh12 has joined #openstack-nova | 00:21 | |
| mriedem | if we get NeutronClientException, then we made a request and it failed but the endpoint, config and token should be ok | 00:21 |
| mriedem | same with CinderClientException | 00:21 |
| mnaser | so we can skip those 2 exceptions because if we get them, it doesn't mean that the compute node has any problems | 00:22 |
| *** r-daneel has quit IRC | 00:22 | |
| *** mdbooth has quit IRC | 00:22 | |
| mriedem | well, if it did turn out that excluding those were too broad, we could narrow it down over time | 00:22 |
| mriedem | vif plug is something that could fail... | 00:23 |
| mriedem | based on bad config of the host | 00:23 |
| mriedem | that's a hard one though | 00:23 |
| mriedem | because we don't get vif plug failures directly | 00:23 |
| *** sree has joined #openstack-nova | 00:23 | |
| mriedem | we get a callback event from neutron that just says plug failed but not why | 00:23 |
| mriedem | but yeah, that's not a NeutronClientException, so ignore me | 00:24 |
| mnaser | i can't imagine a single time that i remember seeing a failed to plug callback | 00:24 |
| mnaser | it either never comes or comes ok :P | 00:24 |
| mnaser | haha | 00:24 |
| mnaser | https://github.com/openstack/python-cinderclient/blob/master/cinderclient/exceptions.py (ClientException) and https://github.com/openstack/python-neutronclient/blob/master/neutronclient/common/exceptions.py (NeutronClientException) | 00:27 |
| mnaser | both seem to be fairly reasonable in terms of things they handle | 00:27 |
| mnaser | thing is NoAuthURLProvided/EndpointNotFound/EndpointTypeNotFound/AmbiguousEndpoints/ConnectionFailed and a bunch of other stuff | 00:28 |
| mnaser | are also under neutronclientexcpetion | 00:28 |
| *** sree has quit IRC | 00:28 | |
| mriedem | we won't hit those anymore, | 00:32 |
| mriedem | because we constructor neutronclient with a ksa session | 00:32 |
| mriedem | so if we fail auth stuff, it will be early with ksa | 00:32 |
| mriedem | *construct | 00:32 |
| mriedem | that's true in at least queens, i'm not sure about pike... | 00:33 |
| mnaser | so that affects the backportable-itiy | 00:35 |
| mnaser | not that i want to stay on pike for much longer | 00:35 |
| mriedem | maybe, would have to dig into it | 00:36 |
| mriedem | trying to fix bug 1679750 atm | 00:36 |
| openstack | bug 1679750 in OpenStack Compute (nova) queens "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,Confirmed] https://launchpad.net/bugs/1679750 | 00:36 |
| mnaser | o | 00:37 |
| mnaser | that's a bad time | 00:37 |
| *** dtruong_ has joined #openstack-nova | 00:42 | |
| *** itlinux has joined #openstack-nova | 00:42 | |
| *** ccamacho has quit IRC | 00:45 | |
| *** dtruong has quit IRC | 00:46 | |
| *** yamamoto has joined #openstack-nova | 00:47 | |
| *** felipemonteiro_ has quit IRC | 00:48 | |
| arvindn05 | mriedem: quick question, i think you answered it once but wanted to reconfirm. "A rebuild is staying on the same host, with optionally a new image, but the flavor stays the same." | 00:48 |
| arvindn05 | https://developer.openstack.org/api-guide/compute/server_concepts.html the documentation does not specify that it stays on the same host...just wanted to confirm if rebuild ALWAYS means staying on same host | 00:49 |
| *** odyssey4me has quit IRC | 00:51 | |
| *** odyssey4me has joined #openstack-nova | 00:51 | |
| *** yamamoto has quit IRC | 00:52 | |
| mriedem | yes it's always the same host | 00:53 |
| *** zhaochao has joined #openstack-nova | 00:54 | |
| mriedem | https://developer.openstack.org/api-guide/compute/server_concepts.html#recover-from-a-failed-compute-host could probably be tightened up | 00:54 |
| *** caisan has joined #openstack-nova | 00:54 | |
| mriedem | where it says, "Evacuate does the same operation as a rebuild. " - that could say, "Evacuate does the same operation as a rebuild except evacuate is on a new host while rebuild is on the same host." | 00:55 |
| mriedem | something like that | 00:55 |
| mriedem | and you can't evacuate with a new image | 00:55 |
| mriedem | arvindn05: when you start talking about evacuate and rebuild, you must read http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | 00:56 |
| arvindn05 | thx for the hint :) | 00:56 |
| *** hoangcx has joined #openstack-nova | 00:57 | |
| arvindn05 | where is the same host enforced? | 00:57 |
| arvindn05 | in code i mean...all that the manager checks is if the hints have the check_type set to rebuild | 00:58 |
| *** mdbooth has joined #openstack-nova | 00:58 | |
| arvindn05 | as per scheduler.utils.request_is_rebuild | 00:59 |
| *** salv-orlando has joined #openstack-nova | 00:59 | |
| arvindn05 | thikning out loud, why would select destination be called in the first place if we dont need to select a host? is it just for some validation? | 01:02 |
| *** phuongnh has joined #openstack-nova | 01:02 | |
| *** salv-orlando has quit IRC | 01:03 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | 01:09 |
| arvindn05 | nvm...we want to run image related filters so thats why we want to call select destination... | 01:13 |
| *** tiendc has joined #openstack-nova | 01:22 | |
| *** namnh has joined #openstack-nova | 01:25 | |
| *** yamahata has quit IRC | 01:25 | |
| Kevin_Zheng | mriedem as for your comments in https://review.openstack.org/#/c/560296/ | 01:26 |
| Kevin_Zheng | I thought that making the order constant in all files make it more easier to read and compare.... | 01:27 |
| mriedem | Kevin_Zheng: i wouldn't mix that into the deduplicate payload change | 01:30 |
| mriedem | if you want to sort the keys globally for those samples, i'd do that once all of the dedup stuff is done | 01:30 |
| Kevin_Zheng | OK, then I will undo them | 01:30 |
| mriedem | arvindn05: for rebuild we set a destination on the request_spec which forces the scheduler to only consider the host that the instance is already one | 01:31 |
| mriedem | arvindn05: here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3158 | 01:32 |
| *** mdbooth has quit IRC | 01:33 | |
| arvindn05 | mriedem: thank you. starting to understand more of the workflow | 01:34 |
| mriedem | arvindn05: np, it only took me about 5 years to understand how rebuild works | 01:35 |
| *** ssurana has quit IRC | 01:37 | |
| arvindn05 | mriedem: lol...i am shifting languages as well so its harder to find workflows | 01:37 |
| *** _ix has joined #openstack-nova | 01:38 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/554920 | 01:38 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete allocations from API if nova-compute is down https://review.openstack.org/560706 | 01:38 |
| arvindn05 | mriedem: i saw def _reset_image_metadata and did not read any code after since in java world seeing a function definition meant i was done with the original function | 01:38 |
| mriedem | inner method | 01:39 |
| mriedem | you can have inline functions in java | 01:39 |
| arvindn05 | inline functions? you mean lambda's? | 01:40 |
| mriedem | no, | 01:40 |
| mriedem | maybe i'm thinking of inline class impls, like inline interfaces | 01:40 |
| mriedem | idk, it's been 7 years since i've written java | 01:40 |
| arvindn05 | yea..anonymous classes you can define them | 01:41 |
| *** _ix_ has joined #openstack-nova | 01:43 | |
| arvindn05 | but cannot define method withhin - quora speaks https://www.quora.com/Can-we-write-a-method-inside-a-method-in-Java | 01:43 |
| *** _ix has quit IRC | 01:44 | |
| arvindn05 | anyway...i learn something new everyday...weird pattern to have methods within methods....havent see it a lot in openstack code either | 01:45 |
| arvindn05 | will keep my eye out from now on | 01:45 |
| *** licanwei has joined #openstack-nova | 01:46 | |
| *** gouthamr has joined #openstack-nova | 01:46 | |
| *** mdbooth has joined #openstack-nova | 01:47 | |
| *** gouthamr has quit IRC | 01:49 | |
| *** gouthamr has joined #openstack-nova | 01:49 | |
| *** QianYu has joined #openstack-nova | 01:50 | |
| *** markvoelker has quit IRC | 01:52 | |
| *** mriedem has quit IRC | 01:54 | |
| *** stakeda has joined #openstack-nova | 01:56 | |
| *** sree has joined #openstack-nova | 01:56 | |
| *** dougshelley66 has quit IRC | 01:59 | |
| *** salv-orlando has joined #openstack-nova | 02:00 | |
| *** rajinir has quit IRC | 02:00 | |
| *** yamahata has joined #openstack-nova | 02:00 | |
| *** QianYu has quit IRC | 02:00 | |
| *** QianYu has joined #openstack-nova | 02:01 | |
| *** salv-orlando has quit IRC | 02:04 | |
| *** sree has quit IRC | 02:05 | |
| *** kmalloc has quit IRC | 02:06 | |
| *** mdbooth has quit IRC | 02:13 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 1 https://review.openstack.org/560296 | 02:16 |
| *** edmondsw has joined #openstack-nova | 02:19 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 1 https://review.openstack.org/560296 | 02:22 |
| *** liuzz has joined #openstack-nova | 02:22 | |
| *** gongysh has joined #openstack-nova | 02:23 | |
| *** liuzz has quit IRC | 02:23 | |
| *** edmondsw has quit IRC | 02:24 | |
| *** liuzz_ has quit IRC | 02:25 | |
| *** fragatin_ has joined #openstack-nova | 02:28 | |
| *** armaan has quit IRC | 02:29 | |
| *** armaan has joined #openstack-nova | 02:29 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 2 https://review.openstack.org/560299 | 02:31 |
| takashin | 02:32 | |
| *** hongbin has joined #openstack-nova | 02:32 | |
| *** gouthamr has quit IRC | 02:33 | |
| *** liuzz has joined #openstack-nova | 02:34 | |
| *** wolverineav has quit IRC | 02:34 | |
| *** wolverineav has joined #openstack-nova | 02:34 | |
| *** janki has joined #openstack-nova | 02:35 | |
| *** gouthamr has joined #openstack-nova | 02:35 | |
| *** wolverineav has quit IRC | 02:40 | |
| *** gouthamr_ has joined #openstack-nova | 02:40 | |
| *** gouthamr has quit IRC | 02:40 | |
| *** liuzz_ has joined #openstack-nova | 02:41 | |
| *** liuzz__ has joined #openstack-nova | 02:41 | |
| *** liuzz has quit IRC | 02:44 | |
| *** wolverineav has joined #openstack-nova | 02:44 | |
| *** liuzz_ has quit IRC | 02:45 | |
| *** wolverineav has quit IRC | 02:51 | |
| *** markvoelker has joined #openstack-nova | 02:53 | |
| *** Spaz-Work has joined #openstack-nova | 02:55 | |
| Spaz-Work | Morning Nova | 02:55 |
| *** suresh12 has quit IRC | 02:56 | |
| openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Handle rebuild of instance with new image https://review.openstack.org/560718 | 02:56 |
| *** Spazmotic has quit IRC | 02:58 | |
| *** salv-orlando has joined #openstack-nova | 03:00 | |
| *** suresh12 has joined #openstack-nova | 03:02 | |
| *** moshele has joined #openstack-nova | 03:03 | |
| *** salv-orlando has quit IRC | 03:05 | |
| *** suresh12 has quit IRC | 03:06 | |
| *** sree has joined #openstack-nova | 03:07 | |
| *** QianYu has quit IRC | 03:08 | |
| *** itlinux has quit IRC | 03:08 | |
| *** imacdonn has quit IRC | 03:08 | |
| *** imacdonn has joined #openstack-nova | 03:09 | |
| *** slaweq has joined #openstack-nova | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| *** germs has quit IRC | 03:16 | |
| *** germs has joined #openstack-nova | 03:17 | |
| *** germs has quit IRC | 03:17 | |
| *** germs has joined #openstack-nova | 03:17 | |
| *** dave-mccowan has quit IRC | 03:18 | |
| *** jchhatbar has joined #openstack-nova | 03:19 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 3 https://review.openstack.org/560727 | 03:21 |
| *** suresh12 has joined #openstack-nova | 03:22 | |
| *** janki has quit IRC | 03:22 | |
| *** markvoelker has quit IRC | 03:26 | |
| *** suresh12 has quit IRC | 03:27 | |
| *** takashin has quit IRC | 03:27 | |
| *** _ix_ has quit IRC | 03:29 | |
| *** bkopilov has quit IRC | 03:29 | |
| *** Tom-Tom has joined #openstack-nova | 03:31 | |
| *** gouthamr_ is now known as gouthamr | 03:31 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560731 | 03:31 |
| *** gyee has quit IRC | 03:33 | |
| *** armaan has quit IRC | 03:34 | |
| *** armaan has joined #openstack-nova | 03:35 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560734 | 03:37 |
| *** armaan has quit IRC | 03:38 | |
| *** armaan has joined #openstack-nova | 03:39 | |
| *** psachin has joined #openstack-nova | 03:40 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 6 https://review.openstack.org/560737 | 03:45 |
| *** amodi has quit IRC | 03:46 | |
| *** moshele has quit IRC | 03:47 | |
| *** armaan has quit IRC | 03:47 | |
| *** gongysh has quit IRC | 03:48 | |
| *** nicolasbock has quit IRC | 03:49 | |
| *** Tom-Tom has quit IRC | 03:51 | |
| *** takashin has joined #openstack-nova | 03:51 | |
| *** Tom-Tom has joined #openstack-nova | 03:51 | |
| *** hiro-kobayashi has joined #openstack-nova | 03:54 | |
| *** Tom-Tom has quit IRC | 03:56 | |
| *** salv-orlando has joined #openstack-nova | 04:01 | |
| *** udesale has joined #openstack-nova | 04:04 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 7 https://review.openstack.org/560748 | 04:04 |
| *** salv-orlando has quit IRC | 04:05 | |
| *** edmondsw has joined #openstack-nova | 04:07 | |
| *** caisan has quit IRC | 04:08 | |
| *** gouthamr has quit IRC | 04:10 | |
| *** edmondsw has quit IRC | 04:12 | |
| *** fragatin_ has quit IRC | 04:15 | |
| *** dustfalling has left #openstack-nova | 04:18 | |
| *** takashin has quit IRC | 04:22 | |
| *** markvoelker has joined #openstack-nova | 04:23 | |
| *** moshele has joined #openstack-nova | 04:25 | |
| *** claudiub has joined #openstack-nova | 04:26 | |
| *** liuzz__ has quit IRC | 04:26 | |
| *** liuzz has joined #openstack-nova | 04:26 | |
| *** markvoelker has quit IRC | 04:27 | |
| *** abhishekk has joined #openstack-nova | 04:30 | |
| *** jichen has joined #openstack-nova | 04:34 | |
| *** markvoelker has joined #openstack-nova | 04:38 | |
| *** hongbin has quit IRC | 04:38 | |
| *** suresh12 has joined #openstack-nova | 04:39 | |
| openstackgerrit | Merged openstack/nova master: Fix race fail in test_resize_with_reschedule_then_live_migrate https://review.openstack.org/560454 | 04:40 |
| *** Zames has joined #openstack-nova | 04:41 | |
| *** amodi has joined #openstack-nova | 04:41 | |
| *** liuzz_ has joined #openstack-nova | 04:42 | |
| *** claudiub|2 has joined #openstack-nova | 04:42 | |
| *** claudiub has quit IRC | 04:45 | |
| *** liuzz has quit IRC | 04:45 | |
| *** hongbin has joined #openstack-nova | 04:46 | |
| *** hongbin has quit IRC | 04:51 | |
| *** Zames has quit IRC | 04:59 | |
| *** sridharg has joined #openstack-nova | 04:59 | |
| *** lpetrut has joined #openstack-nova | 05:02 | |
| *** salv-orlando has joined #openstack-nova | 05:02 | |
| *** yangyapeng has joined #openstack-nova | 05:04 | |
| *** salv-orlando has quit IRC | 05:05 | |
| *** salv-orlando has joined #openstack-nova | 05:05 | |
| *** caisan has joined #openstack-nova | 05:07 | |
| *** links has joined #openstack-nova | 05:09 | |
| openstackgerrit | jichenjc proposed openstack/nova master: remove ec2 in service and cmd https://review.openstack.org/556778 | 05:11 |
| openstackgerrit | jichenjc proposed openstack/nova master: remove ec2 object definitions https://review.openstack.org/557150 | 05:11 |
| openstackgerrit | jichenjc proposed openstack/nova master: remove ec2 db functions https://review.openstack.org/557572 | 05:11 |
| *** gongysh has joined #openstack-nova | 05:11 | |
| *** bkopilov has joined #openstack-nova | 05:17 | |
| *** Tom-Tom has joined #openstack-nova | 05:18 | |
| *** hiro-kobayashi has quit IRC | 05:25 | |
| *** jaosorior has quit IRC | 05:26 | |
| *** hamzy has joined #openstack-nova | 05:27 | |
| *** suresh12 has quit IRC | 05:29 | |
| *** markvoelker has quit IRC | 05:29 | |
| *** evin has joined #openstack-nova | 05:29 | |
| *** markvoelker has joined #openstack-nova | 05:30 | |
| *** ratailor has joined #openstack-nova | 05:30 | |
| *** yangyapeng has quit IRC | 05:31 | |
| *** markvoelker has quit IRC | 05:34 | |
| *** sidx64 has joined #openstack-nova | 05:39 | |
| *** fnordahl has quit IRC | 05:41 | |
| *** fnordahl has joined #openstack-nova | 05:43 | |
| *** trozet has quit IRC | 05:44 | |
| *** trozet has joined #openstack-nova | 05:45 | |
| *** lpetrut has quit IRC | 05:46 | |
| *** yangyapeng has joined #openstack-nova | 05:48 | |
| *** lajoskatona has joined #openstack-nova | 05:48 | |
| *** openstackgerrit has quit IRC | 05:48 | |
| *** dr_gogeta86 has quit IRC | 05:51 | |
| *** yangyapeng has quit IRC | 05:53 | |
| *** jaosorior has joined #openstack-nova | 05:55 | |
| *** lpetrut has joined #openstack-nova | 05:55 | |
| *** edmondsw has joined #openstack-nova | 05:56 | |
| *** udesale_ has joined #openstack-nova | 05:56 | |
| *** AlexeyAbashkin has joined #openstack-nova | 05:58 | |
| *** udesale has quit IRC | 05:59 | |
| *** edmondsw has quit IRC | 06:00 | |
| *** yangyapeng has joined #openstack-nova | 06:01 | |
| *** threestrands has joined #openstack-nova | 06:05 | |
| *** sahid has joined #openstack-nova | 06:06 | |
| *** ssurana has joined #openstack-nova | 06:08 | |
| *** suresh12 has joined #openstack-nova | 06:09 | |
| *** markvoelker has joined #openstack-nova | 06:11 | |
| *** alex_xu has quit IRC | 06:11 | |
| *** suresh12 has quit IRC | 06:13 | |
| *** ccamacho has joined #openstack-nova | 06:16 | |
| *** markvoelker has quit IRC | 06:16 | |
| *** alex_xu has joined #openstack-nova | 06:16 | |
| *** yangyapeng has quit IRC | 06:17 | |
| *** germs has quit IRC | 06:18 | |
| *** germs has joined #openstack-nova | 06:18 | |
| *** germs has quit IRC | 06:18 | |
| *** germs has joined #openstack-nova | 06:18 | |
| *** takashin has joined #openstack-nova | 06:18 | |
| *** openstackgerrit has joined #openstack-nova | 06:20 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Avoid live migrate to same host https://review.openstack.org/542689 | 06:20 |
| *** amodi has quit IRC | 06:20 | |
| *** hoangcx has quit IRC | 06:20 | |
| *** tiendc has quit IRC | 06:20 | |
| *** phuongnh has quit IRC | 06:20 | |
| *** tiendc has joined #openstack-nova | 06:21 | |
| *** phuongnh has joined #openstack-nova | 06:21 | |
| *** hoangcx has joined #openstack-nova | 06:21 | |
| *** AlexeyAbashkin has quit IRC | 06:24 | |
| *** kholkina has joined #openstack-nova | 06:26 | |
| *** andreas_s has joined #openstack-nova | 06:26 | |
| *** sahid has quit IRC | 06:29 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 1 https://review.openstack.org/560296 | 06:30 |
| *** lpetrut has quit IRC | 06:33 | |
| *** salv-orlando has quit IRC | 06:35 | |
| *** AlexeyAbashkin has joined #openstack-nova | 06:35 | |
| *** salv-orlando has joined #openstack-nova | 06:35 | |
| *** armaan has joined #openstack-nova | 06:36 | |
| *** vivsoni has quit IRC | 06:38 | |
| *** salv-orlando has quit IRC | 06:39 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 2 https://review.openstack.org/560299 | 06:41 |
| *** salv-orlando has joined #openstack-nova | 06:42 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | 06:42 |
| *** yangyapeng has joined #openstack-nova | 06:44 | |
| *** AlexeyAbashkin has quit IRC | 06:44 | |
| *** yangyapeng has quit IRC | 06:48 | |
| *** belmoreira has joined #openstack-nova | 06:53 | |
| *** alexchadin has joined #openstack-nova | 06:53 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | 06:54 |
| *** belmoreira has quit IRC | 06:54 | |
| *** armaan_ has joined #openstack-nova | 06:59 | |
| *** avolkov has joined #openstack-nova | 06:59 | |
| *** armaan has quit IRC | 07:00 | |
| *** AlexeyAbashkin has joined #openstack-nova | 07:01 | |
| *** yangyapeng has joined #openstack-nova | 07:02 | |
| *** ekhugen has quit IRC | 07:03 | |
| *** bhagyashris has quit IRC | 07:04 | |
| *** ekhugen has joined #openstack-nova | 07:04 | |
| *** armaan_ has quit IRC | 07:04 | |
| *** bhagyashris has joined #openstack-nova | 07:04 | |
| *** hemna_ has quit IRC | 07:04 | |
| *** yamamoto has joined #openstack-nova | 07:05 | |
| *** yangyapeng has quit IRC | 07:06 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | 07:08 |
| *** armaan has joined #openstack-nova | 07:08 | |
| *** slaweq has joined #openstack-nova | 07:08 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Move update_task_state out of try/except https://review.openstack.org/557152 | 07:08 |
| *** sidx64 has quit IRC | 07:14 | |
| *** _ix has joined #openstack-nova | 07:14 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): Add API to support compute node resource provider update and create https://review.openstack.org/521041 | 07:14 |
| *** markvoelker has joined #openstack-nova | 07:15 | |
| *** ralonsoh has joined #openstack-nova | 07:15 | |
| *** sidx64 has joined #openstack-nova | 07:15 | |
| *** mgoddard has joined #openstack-nova | 07:20 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 3 https://review.openstack.org/560727 | 07:24 |
| *** tiendc has quit IRC | 07:25 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): Add API to support compute node resource provider update and create https://review.openstack.org/521041 | 07:25 |
| *** tiendc has joined #openstack-nova | 07:25 | |
| *** tbachman has quit IRC | 07:27 | |
| *** udesale__ has joined #openstack-nova | 07:28 | |
| *** tbachman has joined #openstack-nova | 07:29 | |
| *** xinliang has quit IRC | 07:30 | |
| *** yangyapeng has joined #openstack-nova | 07:31 | |
| *** udesale_ has quit IRC | 07:31 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** evin has quit IRC | 07:33 | |
| *** xinliang has joined #openstack-nova | 07:35 | |
| *** armaan has quit IRC | 07:35 | |
| *** armaan has joined #openstack-nova | 07:35 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560731 | 07:38 |
| *** tesseract has joined #openstack-nova | 07:39 | |
| *** armaan has quit IRC | 07:40 | |
| *** edmondsw has joined #openstack-nova | 07:44 | |
| *** evin has joined #openstack-nova | 07:44 | |
| *** damien_r has joined #openstack-nova | 07:47 | |
| *** edmondsw has quit IRC | 07:48 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560731 | 07:55 |
| *** lucas-afk is now known as lucasagomes | 07:55 | |
| *** phuongnh has quit IRC | 07:55 | |
| *** hiro-kobayashi has joined #openstack-nova | 07:56 | |
| *** sc has quit IRC | 07:56 | |
| *** frickler has quit IRC | 07:56 | |
| *** phuongnh has joined #openstack-nova | 07:56 | |
| *** gcb has quit IRC | 07:57 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560731 | 07:59 |
| *** armaan has joined #openstack-nova | 08:00 | |
| *** gongysh has quit IRC | 08:01 | |
| *** frickler has joined #openstack-nova | 08:03 | |
| *** adreznec has quit IRC | 08:05 | |
| *** adreznec has joined #openstack-nova | 08:06 | |
| *** mdnadeem has joined #openstack-nova | 08:06 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: libvirt: Block swap volume attempts with encrypted volumes prior to Queens https://review.openstack.org/543569 | 08:09 |
| * lyarwood -> offline until tomorrow \o_ | 08:10 | |
| *** masayukig has quit IRC | 08:12 | |
| *** pcaruana has joined #openstack-nova | 08:12 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_serversV21.py (1) https://review.openstack.org/560821 | 08:12 |
| *** masayukig_ has joined #openstack-nova | 08:13 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_serversV21.py (1) https://review.openstack.org/560821 | 08:14 |
| *** armaan has quit IRC | 08:15 | |
| *** armaan has joined #openstack-nova | 08:15 | |
| *** ktibi has joined #openstack-nova | 08:16 | |
| *** yamamoto has quit IRC | 08:17 | |
| *** armaan has quit IRC | 08:20 | |
| *** evin has quit IRC | 08:21 | |
| *** evin has joined #openstack-nova | 08:22 | |
| *** gongysh has joined #openstack-nova | 08:23 | |
| *** sc has joined #openstack-nova | 08:23 | |
| *** sc is now known as Guest47441 | 08:23 | |
| *** masayukig_ is now known as masayukig | 08:27 | |
| *** AlexeyAbashkin has quit IRC | 08:29 | |
| *** AlexeyAbashkin has joined #openstack-nova | 08:30 | |
| *** salv-orlando has quit IRC | 08:30 | |
| *** salv-orlando has joined #openstack-nova | 08:31 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560731 | 08:32 |
| *** yamamoto has joined #openstack-nova | 08:32 | |
| *** lpetrut has joined #openstack-nova | 08:33 | |
| *** salv-orlando has quit IRC | 08:35 | |
| *** masayukig is now known as masayukig_ | 08:35 | |
| *** yamamoto has quit IRC | 08:36 | |
| *** masayukig_ has quit IRC | 08:40 | |
| *** tssurya has joined #openstack-nova | 08:42 | |
| *** sambetts|afk is now known as sambetts | 08:45 | |
| *** derekh has joined #openstack-nova | 08:47 | |
| *** hiro-kobayashi has quit IRC | 08:51 | |
| *** hiro-kobayashi has joined #openstack-nova | 08:51 | |
| *** linkmark has joined #openstack-nova | 08:53 | |
| *** armaan has joined #openstack-nova | 08:53 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_serversV21.py (2) https://review.openstack.org/560829 | 08:53 |
| *** armaan_ has joined #openstack-nova | 08:54 | |
| *** ttsiouts has quit IRC | 08:54 | |
| *** ttsiouts has joined #openstack-nova | 08:55 | |
| *** armaan has quit IRC | 08:57 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: WIP: complex policy https://review.openstack.org/553776 | 08:58 |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add rules column to instance_group_policy table. https://review.openstack.org/560832 | 08:58 |
| *** _ix has quit IRC | 09:00 | |
| *** sdague has joined #openstack-nova | 09:01 | |
| *** dpawlik has quit IRC | 09:02 | |
| openstackgerrit | jichenjc proposed openstack/nova master: remove ec2 object definitions https://review.openstack.org/557150 | 09:03 |
| openstackgerrit | jichenjc proposed openstack/nova master: remove ec2 db functions https://review.openstack.org/557572 | 09:03 |
| *** dpawlik has joined #openstack-nova | 09:08 | |
| *** ssurana has quit IRC | 09:11 | |
| *** sdague has quit IRC | 09:17 | |
| *** rha has joined #openstack-nova | 09:22 | |
| *** evin has quit IRC | 09:25 | |
| *** do3meli has joined #openstack-nova | 09:26 | |
| *** evin has joined #openstack-nova | 09:28 | |
| *** gaoyan has joined #openstack-nova | 09:31 | |
| *** salv-orlando has joined #openstack-nova | 09:31 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 5 https://review.openstack.org/560734 | 09:33 |
| phuongnh | hi jaypipes, I would like to add one more line in file https://review.openstack.org/#/c/462759/3/os_traits/hw/cpu/x86.py for CPU_FPGA | 09:33 |
| phuongnh | the link to this is: https://en.wikipedia.org/wiki/Field-programmable_gate_array | 09:34 |
| phuongnh | Do I need to propose a blueprint in launchpad? | 09:34 |
| *** sree has quit IRC | 09:36 | |
| *** salv-orlando has quit IRC | 09:37 | |
| *** salv-orlando has joined #openstack-nova | 09:39 | |
| *** yamamoto has joined #openstack-nova | 09:46 | |
| *** alexchadin has quit IRC | 09:47 | |
| *** yamamoto has quit IRC | 09:48 | |
| *** yamamoto has joined #openstack-nova | 09:49 | |
| *** armaan_ has quit IRC | 09:49 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 6 https://review.openstack.org/560737 | 09:50 |
| *** hiro-kobayashi has quit IRC | 09:52 | |
| *** alexchadin has joined #openstack-nova | 09:52 | |
| *** yamamoto has quit IRC | 09:52 | |
| *** Tom-Tom has quit IRC | 09:54 | |
| *** yamamoto has joined #openstack-nova | 09:54 | |
| *** jichen has quit IRC | 09:54 | |
| *** stakeda has quit IRC | 09:57 | |
| *** moshele has quit IRC | 09:58 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** kmalloc has joined #openstack-nova | 10:01 | |
| *** yamamoto has joined #openstack-nova | 10:01 | |
| *** takashin has left #openstack-nova | 10:02 | |
| *** namnh has quit IRC | 10:06 | |
| *** AlexeyAbashkin has quit IRC | 10:10 | |
| *** hoangcx has quit IRC | 10:12 | |
| *** alexchadin has quit IRC | 10:15 | |
| *** sdague has joined #openstack-nova | 10:18 | |
| *** owalsh_afk is now known as owalsh | 10:19 | |
| *** lpetrut has quit IRC | 10:21 | |
| *** threestrands has quit IRC | 10:24 | |
| *** gaoyan has quit IRC | 10:29 | |
| *** elmaciej has joined #openstack-nova | 10:30 | |
| *** dougshelley66 has joined #openstack-nova | 10:34 | |
| *** elmaciej_ has joined #openstack-nova | 10:35 | |
| *** abhishekk has quit IRC | 10:38 | |
| *** sahid has joined #openstack-nova | 10:38 | |
| *** elmaciej has quit IRC | 10:39 | |
| *** elmaciej_ has quit IRC | 10:40 | |
| *** elmaciej has joined #openstack-nova | 10:40 | |
| *** bkopilov has quit IRC | 10:50 | |
| *** elmaciej_ has joined #openstack-nova | 10:51 | |
| *** elmaciej has quit IRC | 10:53 | |
| *** AlexeyAbashkin has joined #openstack-nova | 10:53 | |
| *** phuongnh has quit IRC | 10:54 | |
| *** odyssey4me has quit IRC | 10:55 | |
| *** odyssey4me has joined #openstack-nova | 10:55 | |
| *** tbachman has quit IRC | 10:55 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** licanwei has left #openstack-nova | 10:59 | |
| *** Guest47441 is now known as sc | 11:00 | |
| *** odyssey4me has quit IRC | 11:00 | |
| *** odyssey4me has joined #openstack-nova | 11:00 | |
| *** pchavva has joined #openstack-nova | 11:00 | |
| *** tianhui has quit IRC | 11:02 | |
| *** yamamoto has joined #openstack-nova | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** elmaciej has joined #openstack-nova | 11:04 | |
| openstackgerrit | Merged openstack/nova master: Remove mox in test_virt_drivers.py https://review.openstack.org/559878 | 11:04 |
| *** elmaciej_ has quit IRC | 11:06 | |
| *** gongysh has quit IRC | 11:06 | |
| *** pchavva has quit IRC | 11:07 | |
| *** yamamoto has joined #openstack-nova | 11:08 | |
| *** elmaciej_ has joined #openstack-nova | 11:08 | |
| *** elmaciej has quit IRC | 11:11 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-nova | 11:11 | |
| Shilpa | stephenfin: Hi | 11:12 |
| *** ssurana has joined #openstack-nova | 11:12 | |
| *** cdent has joined #openstack-nova | 11:12 | |
| *** ssurana has quit IRC | 11:17 | |
| *** hshiina|afk has quit IRC | 11:19 | |
| *** edmondsw has joined #openstack-nova | 11:20 | |
| *** sree has joined #openstack-nova | 11:20 | |
| openstackgerrit | Merged openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (1) https://review.openstack.org/558704 | 11:20 |
| *** tianhui has joined #openstack-nova | 11:20 | |
| *** owalsh is now known as owalsh_afk | 11:23 | |
| *** edmondsw has quit IRC | 11:24 | |
| *** elmaciej has joined #openstack-nova | 11:24 | |
| *** sree has quit IRC | 11:24 | |
| *** lpetrut has joined #openstack-nova | 11:25 | |
| stephenfin | Shilpa: o/ | 11:26 |
| *** elmaciej_ has quit IRC | 11:27 | |
| Shilpa | stephenfin: i have added patch set https://review.openstack.org/#/c/550172/2, and added comment there, can you please go through the same. | 11:28 |
| Shilpa | stephenfin: its ok to proceed with noVNC v1.0.0 tag, i have seen issue reported here at https://github.com/novnc/noVNC/issues/1034 | 11:29 |
| *** nicolasbock has joined #openstack-nova | 11:30 | |
| stephenfin | Shilpa: It looks like the CI is failing though? | 11:31 |
| stephenfin | with a valid failure, no less | 11:31 |
| *** lucasagomes is now known as lucas-hungry | 11:31 | |
| Shilpa | stephenfin: i have noted that, but they were failing on earlier patch set too, so just ignored, will check logs | 11:32 |
| *** yamamoto has quit IRC | 11:33 | |
| *** derekh_ has joined #openstack-nova | 11:34 | |
| *** rcernin has quit IRC | 11:35 | |
| *** derekh has quit IRC | 11:36 | |
| *** dougshelley66 has quit IRC | 11:38 | |
| *** elmaciej_ has joined #openstack-nova | 11:40 | |
| *** elmaciej has quit IRC | 11:42 | |
| *** elmaciej has joined #openstack-nova | 11:44 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova-specs master: Amend allow abort live migrations in queued status spec https://review.openstack.org/560872 | 11:46 |
| *** elmaciej_ has quit IRC | 11:47 | |
| *** elmaciej_ has joined #openstack-nova | 11:50 | |
| *** jpena is now known as jpena|lunch | 11:53 | |
| *** amoralej is now known as amoralej|lunch | 11:53 | |
| *** elmaciej has quit IRC | 11:53 | |
| *** caisan has quit IRC | 11:53 | |
| *** dillaman has joined #openstack-nova | 11:56 | |
| *** liverpooler has joined #openstack-nova | 11:56 | |
| bhagyashris | alex_xu: Hi, I just want to know about the spec https://docs.openstack.org/oslo.config/latest/reference/mutable.html which basically give the provision to make the conf parameter as mutable | 11:58 |
| Shilpa | stephenfin: checked logs http://logs.openstack.org/72/550172/2/check/tempest-full/f6945b6/job-output.txt.gz, and observed that 1 test case is failed, and that is tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc[id-c640fdff-8ab4-45a4-a5d8-7e6146cbd0dc] | 11:59 |
| alex_xu | bhagyashris: maybe gcb is good person, but looks like he isn't in this channel | 11:59 |
| bhagyashris | alex_xu: there is one patch submitted in nova https://review.openstack.org/#/c/319203/3 so what are the criteria to make the conf parameter as mutable in any project | 11:59 |
| *** elmaciej has joined #openstack-nova | 12:00 | |
| alex_xu | bhagyashris: ok | 12:00 |
| bhagyashris | alex_xu: as per the operators point of view every parameter should be make it as mutable but why only these two parameters are made it as mutable | 12:01 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | 12:02 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | 12:02 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields consistently https://review.openstack.org/528362 | 12:02 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | 12:02 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | 12:02 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328 | 12:02 |
| *** ralonsoh_ has joined #openstack-nova | 12:02 | |
| bhagyashris | alex_xu: in this patch https://review.openstack.org/#/c/319203/3 | 12:03 |
| *** elmaciej_ has quit IRC | 12:03 | |
| *** ratailor has quit IRC | 12:04 | |
| *** moshele has joined #openstack-nova | 12:04 | |
| alex_xu | bhagyashris: I guess that BP only target to something people just need at that point | 12:05 |
| *** udesale__ has quit IRC | 12:05 | |
| *** ralonsoh has quit IRC | 12:06 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 7 https://review.openstack.org/560748 | 12:07 |
| alex_xu | bhagyashris: I'm not sure all the config option works, but I also didn't see any reject for other options other than that BP | 12:07 |
| *** suresh12 has joined #openstack-nova | 12:10 | |
| bhagyashris | alex_xu: Actually I have few queries in nova only those two parameters are marked as mutable = True in (https://review.openstack.org/#/c/319203/3) so is that means other all the conf parameters are non-mutable. | 12:10 |
| *** dougshelley66 has joined #openstack-nova | 12:11 | |
| openstackgerrit | pippo proposed openstack/nova master: Update eventlet https://review.openstack.org/560876 | 12:11 |
| *** suresh12 has quit IRC | 12:14 | |
| *** elmaciej_ has joined #openstack-nova | 12:14 | |
| *** elmaciej has quit IRC | 12:17 | |
| *** r-daneel has joined #openstack-nova | 12:18 | |
| *** liverpooler has quit IRC | 12:19 | |
| *** liverpooler has joined #openstack-nova | 12:19 | |
| *** elmaciej has joined #openstack-nova | 12:20 | |
| *** eharney has joined #openstack-nova | 12:21 | |
| *** bkopilov has joined #openstack-nova | 12:22 | |
| *** elmaciej__ has joined #openstack-nova | 12:23 | |
| *** elmaciej_ has quit IRC | 12:23 | |
| *** elmaciej has quit IRC | 12:25 | |
| *** AlexeyAbashkin has quit IRC | 12:28 | |
| *** AlexeyAbashkin has joined #openstack-nova | 12:29 | |
| *** yamamoto has joined #openstack-nova | 12:30 | |
| efried | Good UGT morning bhagyashris | 12:32 |
| *** dave-mccowan has joined #openstack-nova | 12:33 | |
| efried | bhagyashris: Are you going to take over https://review.openstack.org/560444 and https://review.openstack.org/560459 ? | 12:33 |
| * efried waves shiny co-author credit | 12:33 | |
| *** yangyapeng has quit IRC | 12:34 | |
| bhagyashris | efried: Good morning :) | 12:34 |
| *** yangyapeng has joined #openstack-nova | 12:35 | |
| *** yamamoto has quit IRC | 12:35 | |
| *** dpawlik has quit IRC | 12:37 | |
| *** tiendc has quit IRC | 12:37 | |
| *** yangyapeng has quit IRC | 12:38 | |
| *** yangyapeng has joined #openstack-nova | 12:39 | |
| *** tbachman has joined #openstack-nova | 12:40 | |
| bhagyashris | efried: ok i will take. | 12:42 |
| efried | bhagyashris: Great. Can you put your name down on the etherpad https://etherpad.openstack.org/p/rocky-nova-priorities-tracking see L115 | 12:43 |
| efried | jaypipes: FYI ^ | 12:43 |
| *** yangyapeng has quit IRC | 12:44 | |
| *** yangyapeng has joined #openstack-nova | 12:44 | |
| bauzas | stephenfin: quick question, if I'm providing a extra spec saying --property hw:numa_nodes=N, does it mean that the instance will use 2 physical NUMA nodes, or just that the instance XML will have 2 nodes ? | 12:47 |
| *** yamamoto has joined #openstack-nova | 12:47 | |
| bauzas | if the latter, are we supporting to not have sharded the NUMA nodes ? | 12:48 |
| efried | sean-k-mooney, cfriesen: ^ | 12:48 |
| bauzas | stephenfin: context is https://review.openstack.org/#/c/552924/7/specs/rocky/approved/numa-topology-with-rps.rst@185 | 12:48 |
| *** yangyapeng has quit IRC | 12:49 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** dklyle has quit IRC | 12:51 | |
| bauzas | efried: as you can see, I wonder if we would still need a specific sharding param | 12:52 |
| bauzas | efried: because when we discussed yesterday, I was only thinking of vGPUs affinited to CPUs, not about vCPUs being anti-affinited by NUMA nodes | 12:52 |
| bauzas | so the latter would need something missing now | 12:53 |
| bauzas | and then, in that case, that would be in my spec : | 12:53 |
| efried | bauzas: Yup; and I totally can't answer that question (whether we need explicit anti-affinity/sharding) | 12:57 |
| bauzas | efried: when looking at the doc, see " FLAVOR-NODES: (integer) The number of host NUMA nodes to restrict execution of instance vCPU threads to. If not specified, the vCPU threads can run on any number of the host NUMA nodes available." | 12:57 |
| efried | But your spec definitely must. | 12:57 |
| bauzas | https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs-numa-topology | 12:57 |
| *** dklyle has joined #openstack-nova | 12:57 | |
| efried | bauzas: Oh, that's interesting. I read that as a *maximum*. | 12:58 |
| efried | In which case, we're covered. | 12:58 |
| efried | sort of | 12:58 |
| efried | We need to split into N numbered request groups | 12:58 |
| *** pchavva has joined #openstack-nova | 12:59 | |
| efried | The only issue is that we can't specify that in a flexible way - it would have to be specific numbers of procs in each group. | 12:59 |
| efried | and the only reasonable way to do that is to divide them evenly. | 12:59 |
| efried | That still means we get *at most* N separate NUMA nodes, but because granular doesn't guarantee separation, we could get anywhere from 1..N | 13:00 |
| bauzas | that said, in https://docs.openstack.org/nova/latest/admin/cpu-topologies.html | 13:00 |
| *** sree has joined #openstack-nova | 13:00 | |
| bauzas | it says " Inadequate per-node resources will result in scheduling failures. Resources that are specific to a node include not only CPUs and memory, but also PCI and SR-IOV resources. It is not possible to use multiple resources from different nodes without requesting a multi-node layout. As such, it may be necessary to ensure PCI or SR-IOV resources are associated with the same NUMA node or force a multi-node layout." | 13:00 |
| *** dpawlik has joined #openstack-nova | 13:00 | |
| efried | That's fair. | 13:01 |
| *** edmondsw has joined #openstack-nova | 13:01 | |
| bauzas | what I understand from the above is that if I'm asking for 2 nodes but then only have 1 node, then NoValidHost | 13:01 |
| *** yamamoto has joined #openstack-nova | 13:01 | |
| bauzas | that's honeslty confusing | 13:01 |
| efried | no, that's not how I read it. | 13:01 |
| bauzas | also given https://bugs.launchpad.net/nova/+bug/1466780 and " There is no correlation required between the NUMA topology exposed in the instance and how the instance is actually pinned on the host. This is by design. See this invalid bug for more information." | 13:01 |
| openstack | Launchpad bug 1466780 in OpenStack Compute (nova) "nova libvirt pinning not reflected in VirtCPUTopology" [Undecided,Invalid] - Assigned to Stephen Finucane (stephenfinucane) | 13:01 |
| *** yamamoto has quit IRC | 13:01 | |
| bauzas | but I take that only for CPU pinning | 13:02 |
| efried | I read it to mean, "if you're restricting to one NUMA node, but no single node has all the resources available, NoValidHost" Which is totally legit. | 13:02 |
| bauzas | let's wait for the others, then | 13:02 |
| kholkina | the spec for user-data update need your review https://review.openstack.org/#/c/547964/ | 13:02 |
| efried | But it brings us to a different issue, which is whether your spec claims to handle device affinity as well as proc/mem | 13:02 |
| bauzas | efried: that said, if we need to shard, what kind of query param are you thinking ? | 13:03 |
| *** lyan has joined #openstack-nova | 13:03 | |
| *** lyan is now known as Guest30936 | 13:03 | |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: move version to string in utils https://review.openstack.org/560455 | 13:03 |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: refactor get_base_config to accept host arg https://review.openstack.org/560456 | 13:03 |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/484997 | 13:03 |
| bauzas | that's the problem with numbered request groups | 13:03 |
| efried | bauzas: For explicitly separating request groups to separate resource providers? I was thinking something like ?separate_providers=resources1,resources2,... | 13:03 |
| efried | ...if placement is where we want to handle it. | 13:03 |
| *** ralonsoh__ has joined #openstack-nova | 13:04 | |
| efried | bauzas: But the other option is to handle it in the NUMATopologyFilter. Placement would give us back all the candidates, which would include the ones that are sharded and the ones that are combined. And the filter would just pick the ones that are appropriately sharded. | 13:04 |
| bauzas | so, ?separate_providers=resources1,resources2&resources1:VCPU=1&resources2:VCPU=1 ? | 13:04 |
| efried | bauzas: yes | 13:04 |
| *** yamamoto has joined #openstack-nova | 13:05 | |
| *** lucas-hungry is now known as lucasagomes | 13:05 | |
| efried | Implementing that will be a bear, but yes. | 13:05 |
| bauzas | if placement folks are accepting that, then I'd prefer to do that by Placement instead of the filter | 13:05 |
| bauzas | because the less we have in the filter, the better it will be | 13:05 |
| efried | I don't see it making Rocky, tbh | 13:05 |
| bauzas | longer term of course | 13:05 |
| bauzas | efried: yeah, I know | 13:05 |
| *** ralonsoh_ has quit IRC | 13:05 | |
| bauzas | efried: what I'd love is some consensus on that spec for Rocky | 13:06 |
| efried | But yes, eventually we recognize we're going to need that functionality in placement. | 13:06 |
| efried | bauzas: which spec? | 13:06 |
| bauzas | but then, earlyj working on Rocky-3 if I'm lucky so we can land Stein-1 | 13:06 |
| bauzas | efried: the NUMA one | 13:06 |
| efried | bauzas: Sure, agreed; but we need to crisp up what exactly you're trying to address, and how. | 13:06 |
| efried | That's not yet clear IMO | 13:07 |
| bauzas | efried: for Rocky, I'm only planning to implement nested RPs for vGPUs and fix the vGPU caveats | 13:07 |
| bauzas | plus that spec | 13:07 |
| bauzas | but implementing that spec for Rocky-3 or later | 13:07 |
| *** jchhatbar is now known as janki | 13:07 | |
| bauzas | unless someone picks the ball | 13:07 |
| *** pchavva has quit IRC | 13:07 | |
| efried | Okay, gotcha. | 13:07 |
| efried | So you're not thinking to land the code related to the NUMA spec until "later". But you want the spec baked by Rocky-3 | 13:08 |
| *** yamamoto has quit IRC | 13:09 | |
| *** dklyle has quit IRC | 13:09 | |
| stephenfin | bauzas: The latter | 13:10 |
| efried | johnthetubaguy_: You around? I approved https://review.openstack.org/#/c/553605/ on the basis that your -1 has now been addressed, but there's still time to pull it out if you disagree. | 13:10 |
| stephenfin | bauzas: typically those guest nodes would be scheduled to different NUMA nodes, but if that's not possible they'll be squeezed onto the same one | 13:10 |
| bauzas | efried: that's correct | 13:11 |
| stephenfin | bauzas: At least that's the case to the best of my recollection | 13:11 |
| *** amodi has joined #openstack-nova | 13:11 | |
| efried | ahjeez. "anti-affinity preferred" is a use case we haven't even thought about yet. | 13:11 |
| bauzas | efried: trying to get a consensus for Rocky-1 or Rocky-2 | 13:11 |
| bauzas | efried: while I'm implementing other things | 13:11 |
| efried | bauzas: roger that | 13:11 |
| bauzas | efried: but once I'm done with the other things, going back to NUMA for implementing | 13:11 |
| bauzas | honestly, if we don't need new Placement version, I think it can be done very quickly | 13:12 |
| bauzas | efried: ^ | 13:12 |
| efried | yes | 13:12 |
| bauzas | efried: it's just about translating specs into Placement queries | 13:12 |
| efried | yes | 13:12 |
| bauzas | so, at least the NUMAFilter would still check the hosts, but it would only check the accepted ones | 13:12 |
| efried | And I actually like the idea of translating the existing numa-related flavor specs into placement queries, rather than asking folks to rewrite their flavors with placement-y syntax. | 13:13 |
| *** mdbooth has joined #openstack-nova | 13:13 | |
| bauzas | and given the default flag value for NUMA will be None, nothing should change actually | 13:13 |
| *** ssurana has joined #openstack-nova | 13:13 | |
| bauzas | unless someone wants to test that magic bullet to restrict hosts passed to the scheduler | 13:13 |
| efried | Because that way we can make placement queries that would be... unreasonable for humans to come up with. | 13:13 |
| bauzas | it's more about an upgrade question for me | 13:14 |
| bauzas | efried: modifying flavors could be a problem for upgrading | 13:14 |
| *** ssurana has quit IRC | 13:14 | |
| *** ssurana has joined #openstack-nova | 13:14 | |
| bauzas | if we support the existing flavors, that's better | 13:14 |
| efried | ++ | 13:14 |
| bauzas | stephenfin: so, to clarify, you mean that we don't explicitely shard between NUMA nodes ? | 13:15 |
| stephenfin | bauzas: We do but it's best effort | 13:15 |
| bauzas | stephenfin: tbc, if you're asking for hw:numa_nodes=2 but you only have one NUMA node (or even a UMA topology), then we accept the host, right?N | 13:16 |
| stephenfin | bauzas: yup | 13:16 |
| bauzas | perfect, efried ^ | 13:16 |
| efried | Cool beans. One down, three to go. | 13:17 |
| bauzas | stephenfin: so, it's more about NUMA "affinity" of multiple resources, rather than NUMA 'anti-affinity' of different CPUs | 13:17 |
| bauzas | still right? | 13:17 |
| bauzas | if so, numbered request groups is the perfect expression | 13:17 |
| stephenfin | It's not really anything to do with NUMA affinity, tbh. It's purely to do with the guest topology | 13:17 |
| bauzas | perfect | 13:17 |
| bauzas | it doesn't guaranttee you'll land all your CPUs on specific NUMA nodes | 13:18 |
| stephenfin | The fact that we don't split a guest NUMA node across a host NUMA node is a performance improvement but not essential by any means | 13:18 |
| bauzas | you could have all your CPUs on the same | 13:18 |
| stephenfin | correct | 13:18 |
| bauzas | excellent, good news | 13:18 |
| bauzas | and tbh, I understand the reasoning | 13:18 |
| stephenfin | so booting an instance with hw:numa_nodes=4 on a dual socket system is a valid thing to do | 13:19 |
| bauzas | you could discover the host topology as an end-user if we were restricting | 13:19 |
| stephenfin | assuming total vCPUs < total host CPUs | 13:19 |
| stephenfin | 'zactly | 13:19 |
| bauzas | just by trying to boot a couple of different topologies | 13:19 |
| stephenfin | There's also zero reason to allow it | 13:19 |
| bauzas | efried: so you agree the numbered request groups feature exactly matches the above ? ^ | 13:19 |
| *** ssurana has quit IRC | 13:20 | |
| *** yangyapeng has joined #openstack-nova | 13:20 | |
| *** edmondsw has quit IRC | 13:20 | |
| sahid | bauzas, stephenfin, if user is asking for hw:numa_node=2 and host does not have at least 2 NUMA node we don't accept the host | 13:20 |
| *** yamamoto has joined #openstack-nova | 13:21 | |
| *** edmondsw has joined #openstack-nova | 13:21 | |
| sahid | or it's a bug and we should fix it | 13:21 |
| stephenfin | sahid: Why? | 13:21 |
| bauzas | sahid: that's very different from what stephenfin says | 13:21 |
| * stephenfin goes to check | 13:21 | |
| bauzas | http://www.quickmeme.com/meme/361gwd | 13:21 |
| *** edmondsw has quit IRC | 13:21 | |
| *** pchavva has joined #openstack-nova | 13:21 | |
| sahid | because it's all performance related | 13:22 |
| sahid | it's all about distance between pci devices, memory channel an cpus | 13:22 |
| *** abhishekk has joined #openstack-nova | 13:23 | |
| sahid | stephenfin: we should have in hardware.py constrainsts something which check that | 13:24 |
| stephenfin | bauzas: Ah, crap. sahid's correct there | 13:24 |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| stephenfin | bauzas, sahid: https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1548-L1553 | 13:24 |
| *** yamamoto has quit IRC | 13:24 | |
| *** yangyapeng has quit IRC | 13:24 | |
| bauzas | sahid: so, like I said, say I'm an evil folk named Eve, I could guess the host topology by booting a couple of test instances having various NUMA guest topologies, right? | 13:25 |
| stephenfin | bauzas: and https://github.com/openstack/nova/blob/master/nova/objects/instance_numa_topology.py#L217-L219 | 13:25 |
| bauzas | I understand the performance reason for affinitizing the guest, but not for anti-affinitizing it | 13:26 |
| stephenfin | I thought that was comparing the length of cell.cpuset, a la https://github.com/openstack/nova/blob/master/nova/objects/instance_numa_topology.py#L82-L83 | 13:26 |
| stephenfin | sahid: Yeah, why do we do that? It seems unnecessary | 13:26 |
| *** andreas_s has quit IRC | 13:27 | |
| stephenfin | Like I said above, I get the reason for not splitting guest NUMA nodes across host NUMA nodes | 13:27 |
| stephenfin | and for placing at least one guest NUMA node's CPUs on the host NUMA node associated with a PCI device | 13:27 |
| bauzas | stephenfin: checking whether the host has the same number of NUMA nodes that the guest is one thing | 13:27 |
| *** links has quit IRC | 13:27 | |
| bauzas | stephenfin: the other thing being that we would restrict that to different nodes | 13:28 |
| sahid | what is the use-cases? I mean why you want to have differente NUMA nodes if the CPU is using a memory channel which is on an different NUMA node? as I said it's all performance related | 13:28 |
| stephenfin | sahid: Right, but I have two NUMA nodes and I boot the instance with four NUMA nodes. Why can't two go on one host node and two on the other? | 13:28 |
| bauzas | sahid: performance is affinity, and I don't disagree with you | 13:29 |
| stephenfin | Sounds like you'd get the same performance as with an instance with two NUMA nodes that's split across the two host nodes | 13:29 |
| sahid | stephenfin: why you want do that? | 13:29 |
| stephenfin | sahid: So I can boot my instance | 13:29 |
| *** jaosorior has quit IRC | 13:29 | |
| bauzas | sahid: in theory, the user doesn't know the host topology | 13:29 |
| bauzas | sahid: he's just booting a flavor | 13:30 |
| bauzas | but what he knows is that he'll get a guest having its own topology | 13:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Rename recreate to evacuate in driver signatures https://review.openstack.org/560900 | 13:30 |
| stephenfin | Yeah, what bauzas said | 13:30 |
| *** psachin has quit IRC | 13:30 | |
| sahid | yes and if he wants 2 numa where don"t want to fake that | 13:30 |
| openstackgerrit | Eric Fried proposed openstack/os-traits master: normalize_name helper https://review.openstack.org/560107 | 13:30 |
| stephenfin | He wants two _guest_ NUMA nodes | 13:30 |
| sahid | yes and if he wants 2 numa nodes for it's guest we don"t want to fake that | 13:30 |
| stephenfin | The entire thing is fake | 13:31 |
| bauzas | sahid: so some evil guy could guess there are no left hosts having 2 numa nodes | 13:31 |
| stephenfin | There aren't actually two guest NUMA nodes. That's just QEMU/KVM mocking it | 13:31 |
| sahid | stephenfin: it's not... the guest memory and vcpus threads are running on a uniq node | 13:32 |
| *** gouthamr has joined #openstack-nova | 13:32 | |
| *** moshele has quit IRC | 13:32 | |
| *** mriedem has joined #openstack-nova | 13:32 | |
| stephenfin | sahid: I get that. I'm not suggesting splitting a guest's NUMA node across a host NUMA node | 13:32 |
| stephenfin | What I'm saying is there's not reason we shouldn't fit N guest NUMA nodes on the same host NUMA node | 13:32 |
| efried | but the other way around is okay | 13:32 |
| *** andreas_s has joined #openstack-nova | 13:32 | |
| stephenfin | efried: Yeah, precisely | 13:33 |
| sahid | stephenfin: it's not what we do | 13:33 |
| bauzas | so the big thing is | 13:33 |
| bauzas | now we know that | 13:33 |
| bauzas | should Placement do that ? | 13:33 |
| bauzas | I don't think so | 13:33 |
| stephenfin | sahid: Indeed, and I think we can change that | 13:33 |
| bauzas | leave that logic to the filter | 13:33 |
| stephenfin | bauzas: No, neither do I | 13:33 |
| bauzas | what Placement would return is just a list of potential targets | 13:34 |
| stephenfin | So long as we can still ensure at least one guest NUMA node is affined with a given host NUMA node | 13:34 |
| bauzas | some can run that workload but on a single node, some can do that better | 13:34 |
| stephenfin | For things like PCI devices and the likes | 13:34 |
| bauzas | so the filter will pick the best one | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: RP list: member_of and resources parameters (v1.3, v1.4) https://review.openstack.org/511183 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: RP delete inventories (v1.5) https://review.openstack.org/514642 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | 13:34 |
| stephenfin | And vSwitches :) | 13:34 |
| Shilpa | stephenfin: Hi, noVNC 1.0.0, i have checked logs http://logs.openstack.org/72/550172/2/check/tempest-full/f6945b6/job-output.txt.gz, and observed that 1 test case is failed, and that is tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc[id-c640fdff-8ab4-45a4-a5d8-7e6146cbd0dc] | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: New dict format of allocations (v1.11, v1.12) https://review.openstack.org/542819 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Transactionally update allocations (v1.13) https://review.openstack.org/546674 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | 13:34 |
| openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Allocation candidates parameter: required (v1.17) https://review.openstack.org/548326 | 13:34 |
| bauzas | oh shit, gerrit dropbomb | 13:34 |
| efried | Placement will give you candidates telling you exactly which NUMA nodes the *actual* resources come from. The virt driver must adhere to that. What it turns around and tells the guest is entirely up to your conscience :) | 13:34 |
| sahid | stephenfin: what do you want to change? | 13:34 |
| Shilpa | stephengin: checked there, URL is correct though 'http://217.182.140.77:6080/vnc.html?path=websockify?token=30848df8-99b2-4a0c-a615-51a5a444660d' | 13:35 |
| bauzas | efried: Placement will give hosts that match the possibility to support a guest having a specific topology | 13:35 |
| stephenfin | Shilpa: Indeed. I thought that was because of the change in the path but your change apparently works with both 1.0 and pre-1.0 | 13:35 |
| stephenfin | Shilpa: So I don't know what's going on | 13:35 |
| efried | bauzas: Maybe. | 13:35 |
| bauzas | efried: but the filter could restrict that list | 13:35 |
| efried | bauzas: Exactly. | 13:35 |
| efried | The filter is allowed to be stricter than placement. That's... kind of its job. | 13:36 |
| bauzas | efried: okay, so I feel numbered request groups can still work | 13:36 |
| stephenfin | sahid: If a user requests N nodes, then nova should place those on host NUMA nodes as necessary | 13:36 |
| *** jpena|lunch is now known as jpena | 13:36 | |
| efried | yes, agree completely. | 13:36 |
| bauzas | provided I'm clear on my spec | 13:36 |
| stephenfin | sahid: So for 4 guest nodes, if it can fit 3 on one host node and 1 on the other, that would be OK | 13:36 |
| bauzas | that, whatever logic I dislike on filter, it's not intended to be in Placement | 13:36 |
| sahid | stephenfin: please don't do that | 13:36 |
| efried | bauzas: Would even work if we wanted to support explicit sharding/anti-affinity - like I said earlier, the filter would be responsible for getting rid of the candidates where placement "incorrectly" clustered separate groups into the same RP. | 13:36 |
| Shilpa | stephenfin: ok will check further, if i will get something, will let you know, thanks | 13:37 |
| bauzas | ++ | 13:37 |
| bauzas | efried: ++ | 13:37 |
| efried | sahid: Please clarify. | 13:37 |
| stephenfin | sahid: But we're not breaking the NUMA affinity with PCI devices or the likes. At least one of those guest nodes would have to be placed on the same host NUMA node as the PCI device | 13:37 |
| sahid | you are going to break the whole aim of numa toplogy | 13:37 |
| *** andreas_s has quit IRC | 13:37 | |
| efried | sahid: You're saying that if we request 4 (host) nodes, we must get 4 (host) nodes? | 13:37 |
| stephenfin | sahid: This is no different to what we do now | 13:37 |
| sahid | yes | 13:37 |
| mdbooth | The multiattach job seems to be unhappy today. Did I miss any traffic about that? | 13:38 |
| efried | sahid: Yeah, bauzas pointed to the flavor docs earlier which strongly implied (to my understanding) that when I say 4, I mean "at most 4" | 13:38 |
| efried | mdbooth: I was waiting for mriedem to arrive so I could pester him about that. | 13:38 |
| stephenfin | sahid: If I have a guest with a PCI device and hw:numa_nodes=2, then only one guest NUMA node would be placed on the same node as the PCI device | 13:38 |
| *** esberglu has joined #openstack-nova | 13:38 | |
| stephenfin | *same host nodes as | 13:38 |
| stephenfin | sahid: Come to think of it, this would improve performance | 13:38 |
| stephenfin | *could | 13:38 |
| efried | mriedem: Here's an example: http://logs.openstack.org/49/560349/1/check/nova-multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR | 13:38 |
| mdbooth | efried: Ok. I won't double-pester him, then :) | 13:38 |
| sahid | stephenfin: again we should not accept the host | 13:39 |
| sahid | we sould never break the guest numa topology | 13:39 |
| efried | mriedem: ...which looks kinda like something wrong in the CI env, not in the code. But I'm pretty ignorant in this realm. | 13:39 |
| stephenfin | sahid: e.g. if we could fit the two guest NUMA nodes on the same host NUMA node as the PCI device, we'd be guaranteeing affinity for all guest CPUs | 13:39 |
| *** artom has quit IRC | 13:39 | |
| mriedem | did you guys try doing a logstash query to find out when that started and if it's just that job? | 13:39 |
| stephenfin | as opposed to half of them | 13:39 |
| mriedem | my guess would be it's related to https://review.openstack.org/#/c/554314/ | 13:39 |
| stephenfin | "stephenfin: again we should not accept the host" what do you mean? | 13:40 |
| efried | mriedem: See, this is why we ask you. | 13:40 |
| sahid | as i have indicated it's all about performance and distance between cpu, memory channel and pci devices | 13:40 |
| mdbooth | efried: Hehe | 13:40 |
| efried | mriedem: You came up with that shit off the top of your head; it would have taken me years to figure out. | 13:40 |
| sahid | you are going to break this affinity | 13:40 |
| mriedem | but i'll be gone some day | 13:40 |
| stephenfin | sahid: I don't think you're understanding what I'm saying | 13:40 |
| sahid | stephenfin: re-reading, sorry | 13:40 |
| * mdbooth will sing Danny Boy | 13:40 | |
| stephenfin | sahid: Let's table this for discussion on the spec. I think bauzas has his answer for now :) | 13:41 |
| mdbooth | Was about to say that looks like a platform error | 13:41 |
| stephenfin | sahid: good talk! :) | 13:41 |
| mriedem | efried: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Connection%20to%20libvirt%20failed%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%3A%20libvirtError%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | 13:41 |
| sahid | stephenfin: sure | 13:41 |
| mriedem | it's definitely due to that change, since it's just that job | 13:41 |
| *** andreas_s has joined #openstack-nova | 13:41 | |
| efried | stephenfin: I don't know if bauzas has his answer. The question was, "if the user asks for 4 nodes, do we gotta get exactly 4 nodes from the host, or is it okay if it's anywhere from 1..4?" | 13:42 |
| efried | stephenfin: Before sahid chimed in, we thought "1..4 is fine" was the answer. | 13:42 |
| efried | stephenfin: Now we don't know anymore. | 13:42 |
| efried | stephenfin: And bauzas definitely needs that answer to do his next spec edit. | 13:42 |
| mriedem | efried: i'm hoping that https://review.openstack.org/#/c/554317/ will handle it | 13:42 |
| stephenfin | efried: Hmm, can we discuss that on the spec so I can write something longer. I'm clearly not doing a good job of explaining here | 13:43 |
| efried | mriedem: But that guy is failing the same way http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/screen-n-cpu.txt.gz?level=ERROR | 13:43 |
| *** yamamoto has joined #openstack-nova | 13:43 | |
| stephenfin | and/or I'm missing what sahid is trying to say | 13:43 |
| stephenfin | efried: bauzas can just choose one and we'll weigh in there | 13:44 |
| efried | fair | 13:44 |
| *** yamamoto has quit IRC | 13:44 | |
| *** burt has quit IRC | 13:45 | |
| *** andreas_s has quit IRC | 13:46 | |
| sahid | i just saying that our current behavior to handle numa topology for guests is right we should not consider ok a guest requesting 2 NUMA nodes fit on one host NUMA node | 13:47 |
| mriedem | efried: i think this is the problem http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/devstacklog.txt.gz#_2018-04-11_18_10_24_388 | 13:47 |
| *** sree has quit IRC | 13:48 | |
| *** sree has joined #openstack-nova | 13:48 | |
| *** yamamoto has joined #openstack-nova | 13:48 | |
| sahid | that because it's all performance related and user are going to place thier applications based on guest topology. if we provide a fake the performance will be degraded | 13:48 |
| *** yamamoto has quit IRC | 13:49 | |
| *** yamamoto has joined #openstack-nova | 13:49 | |
| efried | sahid: Is it ever the case that performance is *improved* by forcing resources to come from *separate* NUMA nodes? | 13:50 |
| efried | ohh, this is the parallelism thing. | 13:50 |
| mriedem | nvm | 13:51 |
| mdbooth | mriedem: If you get a sec could you remove the procedural -2 on https://review.openstack.org/#/c/526346/ ? | 13:51 |
| mriedem | done | 13:51 |
| mdbooth | mriedem: Thanks | 13:51 |
| sahid | efried: it's just that, the vCPUs threads and memory will be placed on the same host numa node, so yes the performance will be ever improved | 13:52 |
| sahid | it's the only reason why numa exist | 13:53 |
| *** sree has quit IRC | 13:53 | |
| efried | sahid: I understand how performance is improved by placing on the *same* node. I'm asking if it's important to be able to explicitly *separate*. | 13:53 |
| *** mlavalle has joined #openstack-nova | 13:54 | |
| *** yangyapeng has joined #openstack-nova | 13:54 | |
| efried | sahid: Because in the case we're talking about, --property hw:numa_nodes=N will spread your resources across *at most* N NUMA nodes. It could end up being anywhere from 1..N. (But it will never be >N) | 13:54 |
| sahid | efried: it depends of the use-case and application. you can request only one. the point is to have cpu/memory on same node | 13:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Make the nova-multiattach job non-voting temporarily https://review.openstack.org/560909 | 13:54 |
| *** yangyapeng has quit IRC | 13:54 | |
| mriedem | efried: stephenfin: ^ | 13:54 |
| efried | mriedem: +2 | 13:55 |
| efried | sahid: So to my understanding, in this case, anything <N is a bonus. | 13:56 |
| *** udesale__ has joined #openstack-nova | 13:56 | |
| stephenfin | sahid: Right, but the CPU and memory will be on the same node. Again, we're not suggesting splitting an individual guest NUMA node | 13:56 |
| efried | sahid: ...unless it's not. Which is what we're trying to ascertain. | 13:56 |
| bauzas | efried: stephenfin: sahid: I think I have my answer tho | 13:56 |
| efried | bauzas: Which is what? | 13:57 |
| bauzas | efried: stephenfin: sahid: in my spec, I'll propose Placement to return all hosts accepting up to N NUMA nodes | 13:57 |
| sahid | but it depends on the application. for example for realtime you don't want them to be separated, same story for zero-drop packets that because of some cpu improvements L2/L3 cache | 13:57 |
| sahid | s/you don"t/you want | 13:57 |
| bauzas | efried: stephenfin: efried: but then, the filter will continue to act as before and only accept hosts that equally match the number of nodes asked by the guest | 13:58 |
| stephenfin | sahid: Say I have a guest with a PCI device. If I boot the instance with two NUMA nodes and place both guest NUMA nodes on the same host NUMA node | 13:58 |
| bauzas | it's just we won't ask the filter to check hosts that don't have *at least* | 13:58 |
| efried | bauzas: ++ but it must not only accept *hosts* - it must filter down *candidates* for that host. | 13:58 |
| stephenfin | sahid: Assuming that host NUMA node is the same one that the PCI device is placed on, what would be wrong with that? | 13:58 |
| bauzas | good correction | 13:58 |
| efried | bauzas: Because you could (and usually will) get multiple candidates for the same host, some of which have N and some of which have <N | 13:59 |
| stephenfin | efried: I do wish someone would write a glossary of placement terminology | 13:59 |
| efried | bauzas: And sometimes you'll *only* get candidates with <N, in which case your filter will have to exclude that host. | 13:59 |
| sahid | stephenfin: i understand what you are saying but it again it depends of the application used | 13:59 |
| * stephenfin goes to figure out how a host != candidate | 13:59 | |
| bauzas | efried: either way, Placement returns candidates to scheduler, but scheduler passes a list of hosts to the filter, right? | 13:59 |
| efried | stephenfin: Ah, by candidate I mean "allocation candidate". | 13:59 |
| stephenfin | sahid: Say for the non-realtime case | 14:00 |
| sahid | or we need an option to say enforce that rule | 14:00 |
| efried | stephenfin: technically the "allocation request" part of the result of GET /allocation_candidates | 14:00 |
| bauzas | efried: say for example Placement returns 2 child RPs, each of them being a NUMA node of host1, it will only pass host1 to the NUMATopoFilter | 14:00 |
| sahid | stephenfin: for realtime use case you want best effort cpu to run on different numa node than the realtime cpus are | 14:01 |
| efried | stephenfin: There will be X candidates coming back from a GET /a_c request, but the total number of providers ("hosts") represented by those candidates can be <=X | 14:01 |
| bauzas | efried: that's not a problem then, right? | 14:01 |
| sahid | so bascially you are going to break realtime use case :) | 14:02 |
| efried | sahid: No | 14:02 |
| *** hemna_ has joined #openstack-nova | 14:02 | |
| *** _ix has joined #openstack-nova | 14:02 | |
| bauzas | efried: if host1 is having 2 nodes, and those 2 nodes are valid candidates for my query where resources1:VCPU=1&resources2:VCPU=1, I'll still pass to NUMATopoFilter only host1 | 14:02 |
| efried | sahid: We're figuring out how to get N nodes when you ask for N nodes. | 14:02 |
| sahid | efried: that seems reasonable | 14:02 |
| jaypipes | efried, bauzas: #openstack-placement this conversation or mriedem is going to blow up. | 14:03 |
| efried | jaypipes: We're talking about how nova is going to use placement | 14:03 |
| *** jackie-truong has joined #openstack-nova | 14:03 | |
| stephenfin | efried: Oh, because the same host can be presented multiple ways? | 14:04 |
| bauzas | jaypipes: the problem is that I'd love to see some NUMA experts chiming in | 14:04 |
| efried | stephenfin: yes, exactly. | 14:04 |
| *** jaosorior has joined #openstack-nova | 14:04 | |
| *** r-daneel has quit IRC | 14:04 | |
| stephenfin | efried: gotcha. Cheers | 14:04 |
| bauzas | efried: again, I don't think there is a problem | 14:04 |
| stephenfin | jaypipes: The damage should be limited to the west (?) coast. I'll be fine ;) | 14:04 |
| bauzas | efried: the fact that placement is returning a list of candidates won't trample what the filter already does | 14:04 |
| bauzas | if the filter restricts, cool | 14:05 |
| bauzas | and again, if we want to make Placement restricting, let's do that *in a separate spec* | 14:05 |
| efried | bauzas: don't think of numa nodes as candidates. Ignoring sharing for now, one allocation candidate (technically allocation request) will include resources from one host. The resources for each allocation request will be spread across one or more providers (numa nodes) in the tree. The filter will have to inspect each allocation request to see if the number of NUMA nodes represented equals the number of hw:numa_nodes in | 14:06 |
| jaypipes | stephenfin: :) | 14:06 |
| efried | bauzas: And remove the *allocation requests* where that's not the case. | 14:06 |
| efried | bauzas: Now you'll still have X allocation requests representing results for Y hosts, where Y<=X | 14:07 |
| bauzas | sec, otp | 14:07 |
| *** lpetrut has quit IRC | 14:08 | |
| efried | stephenfin or bauzas: wanna unblock the gate? https://review.openstack.org/#/c/560909/ | 14:08 |
| stephenfin | efried: sure | 14:08 |
| *** lpetrut has joined #openstack-nova | 14:08 | |
| stephenfin | done | 14:09 |
| *** zhaochao has quit IRC | 14:10 | |
| *** armaan has joined #openstack-nova | 14:12 | |
| *** felipemonteiro_ has joined #openstack-nova | 14:12 | |
| mriedem | i see the problem | 14:12 |
| mriedem | but don't understand it | 14:12 |
| *** evin has quit IRC | 14:12 | |
| mriedem | https://review.openstack.org/#/c/554317/3/playbooks/legacy/nova-multiattach/run.yaml | 14:12 |
| mriedem | removed ENABLE_UBUNTU_CLOUD_ARCHIVE=False right? | 14:12 |
| mriedem | http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/local.conf.txt.gz | 14:12 |
| mriedem | ENABLE_UBUNTU_CLOUD_ARCHIVE=False is in there | 14:12 |
| *** hongbin has joined #openstack-nova | 14:13 | |
| mriedem | that's why it's blowing up | 14:13 |
| mriedem | http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/ara-report/result/ab8397a6-593e-43f5-a593-c2633a0e40de/ | 14:14 |
| mriedem | idk what's up there but it looks like zuul isn't taking that change | 14:14 |
| *** andreykurilin_ has quit IRC | 14:14 | |
| mriedem | mordred: you around for a zuul question? | 14:15 |
| *** ssurana has joined #openstack-nova | 14:16 | |
| mriedem | there was a zuulv3 code deploy last night... | 14:16 |
| * mriedem goes to infra | 14:16 | |
| *** tianhui_ has joined #openstack-nova | 14:16 | |
| *** jackie-truong has quit IRC | 14:18 | |
| *** tianhui has quit IRC | 14:18 | |
| *** ktibi has quit IRC | 14:20 | |
| *** ssurana has quit IRC | 14:21 | |
| mriedem | https://storyboard.openstack.org/#!/story/2001839 for anyone that cares | 14:23 |
| mriedem | the job on master is picking up changes from the playbook in stable/queens | 14:23 |
| *** elmaciej__ has quit IRC | 14:23 | |
| *** suresh12 has joined #openstack-nova | 14:24 | |
| *** dpawlik has quit IRC | 14:24 | |
| *** abhishekk has quit IRC | 14:25 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | 14:27 |
| *** mvk has quit IRC | 14:27 | |
| *** owalsh_afk is now known as owalsh | 14:28 | |
| *** suresh12 has quit IRC | 14:28 | |
| *** caisan has joined #openstack-nova | 14:28 | |
| *** evin has joined #openstack-nova | 14:28 | |
| bauzas | efried: back | 14:29 |
| bauzas | IT person curse | 14:29 |
| bauzas | my brother-in-law called me (while I'm at work!) for some personal issue with windows 10 | 14:29 |
| bauzas | not kidding | 14:29 |
| bauzas | one day, I'll explain to my family that : | 14:30 |
| bauzas | #1 I have a real job even if working at home | 14:30 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | 14:30 |
| bauzas | #2 working on IT doesn't mean I'm expert in windows-ishings | 14:30 |
| dansmith | bauzas: can you help me defrag my hard drive? | 14:30 |
| dansmith | bauzas: also, I think I have a virus | 14:31 |
| dansmith | bauzas: what's your preferred program for video editing on my windows pc? | 14:31 |
| bauzas | dansmith: my brother-in-law has no fucking idea of what 'defrag' means :p | 14:31 |
| dansmith | bauzas: can you recommend a printer for laying ink on dead tree carcass since that's a thing I do | 14:31 |
| dansmith | bauzas: hah | 14:31 |
| dansmith | bauzas: relaying recent questions I have gotten since you're apparently the windows IT expert | 14:32 |
| mriedem | mdbooth: ever seen this before? http://logs.openstack.org/67/560467/1/check/nova-next/9ceb996/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_11_16_19_23_843152 | 14:32 |
| mriedem | swap volume failure | 14:32 |
| *** itlinux has joined #openstack-nova | 14:32 | |
| bauzas | I'm also amazed that people think I can be reached at 4:15pm local time | 14:32 |
| bauzas | anyway | 14:32 |
| bauzas | efried: back to our convo | 14:32 |
| *** moshele has joined #openstack-nova | 14:33 | |
| bauzas | efried: you said the filter will need to introspect the allocation candidates | 14:33 |
| efried | bauzas: ...which are supposed to be opaque, I just remembered :) | 14:33 |
| bauzas | efried: but I thought using nested RPs was implying that we were translating the candidates into hosts | 14:33 |
| efried | bauzas: But yes, there's no way around it. | 14:33 |
| bauzas | or, rather, we were passing the root RPs as candidates | 14:34 |
| bauzas | but with the candidates as part of the answer | 14:34 |
| efried | bauzas: Each allocation request is related to one host. But there may be many allocation requests related to each host. | 14:34 |
| bauzas | I don't exactly remember what Placement returns in the sense of nested RPs | 14:34 |
| efried | bauzas: Nothing yet :) | 14:34 |
| bauzas | ah right then | 14:34 |
| bauzas | so, multiple candidates, each of them being the same | 14:35 |
| *** r-daneel has joined #openstack-nova | 14:35 | |
| *** lajoskatona has quit IRC | 14:35 | |
| bauzas | efried: nothing yet ? man, where is the code that I can review for merging that ? | 14:35 |
| bauzas | efried: I thought the query side of nested RPs was done | 14:35 |
| kashyap | dansmith: I found some info for you | 14:36 |
| kashyap | dansmith: For defragmenting your Windows (presumably) harddrive | 14:36 |
| dansmith | heh | 14:36 |
| dansmith | IT-knowing family members are just proxies to google anyway | 14:36 |
| *** dklyle has joined #openstack-nova | 14:36 | |
| kashyap | dansmith: Without further ado: https://irc.verylegit.link/d_.3u~GKm*e-*7%3Enotice89464-torrent_free.pptx.docm.exe | 14:36 |
| *** udesale__ has quit IRC | 14:36 | |
| bauzas | just to give a sense of what my pain was https://www.youtube.com/watch?v=xptvcKMGSb8 | 14:36 |
| kashyap | bauzas: Click on my link above | 14:36 |
| dansmith | kashyap: oh let me click on that and run it | 14:36 |
| bauzas | dansmith: exactly that | 14:37 |
| *** udesale has joined #openstack-nova | 14:37 | |
| kashyap | dansmith: It is a URL shortner! | 14:37 |
| kashyap | https://verylegit.link/ | 14:37 |
| dansmith | bauzas: give them this: https://xkcd.com/627/ | 14:37 |
| bauzas | dansmith: I have no fucking idea of what windows looks likz | 14:37 |
| *** derekh_ has quit IRC | 14:37 | |
| bauzas | dansmith: b/c the last windows I ran was 7 | 14:37 |
| * kashyap gets back to doing the utterly "unfun" thing that he _must_ finish by tomorrow | 14:37 | |
| dansmith | bauzas: the last one I ran was 2000 :) | 14:37 |
| bauzas | so, 10 is old greek to me | 14:37 |
| mriedem | coincidentally https://adequateman.deadspin.com/should-a-sports-hall-of-fame-have-a-maximum-capacity-1825143616#_ga=2.165512767.1888868685.1523543860-529658380.1521051074 | 14:37 |
| efried | bauzas: The nrp-in-alloc-cands work is here: https://review.openstack.org/#/q/topic:bp/nested-resource-providers-allocation-candidates | 14:37 |
| mriedem | has a section about being the IT guy | 14:37 |
| bauzas | efried: roger this, will sneak peek on those | 14:38 |
| mriedem | starts at "I’m 28 and been married two weeks. " | 14:38 |
| bauzas | what's also sad to me is that my brother-in-law has a daughter of 23 who also studies IT, and she's in couple with a IT PhD student | 14:39 |
| *** Zames has joined #openstack-nova | 14:39 | |
| efried | bauzas: We should clarify our terminology. I've been using "candidate" to mean "one result in the list returned by GET /allocation_candidates". Technically we should be calling that an "allocation request". That's the blob that you would turn around and PUT to /allocations/{consumer_uuid} to create the actual allocation for the instance (consumer). | 14:39 |
| bauzas | so he has 2 good reasons to not call me | 14:39 |
| bauzas | in particular when it's the middle of the afternoon and I'm not on PTO | 14:39 |
| mriedem | dansmith: when you get a chance, channel topic needs updating https://etherpad.openstack.org/p/nova-runways-rocky | 14:40 |
| dansmith | mriedem: I don't think it does yet | 14:40 |
| dansmith | I checked this morning | 14:40 |
| mriedem | runways have changed... | 14:40 |
| dansmith | oh, nm | 14:40 |
| efried | bauzas: Again putting sharing aside, one allocation request is going to be a list of resources from (nested) providers on a single host. So there may be e.g. 10 allocation requests returned, but only three hosts represented (so more than one allocation request per host). | 14:40 |
| dansmith | I had to refresh, it didn't notice | 14:40 |
| *** ChanServ sets mode: +o dansmith | 14:41 | |
| dansmith | hmm, why did the certificate stuff get kicked out? | 14:41 |
| dansmith | that had a few more days | 14:41 |
| dansmith | and the top one is overdue | 14:41 |
| efried | bauzas: In the use case we're talking about, the filter is going to have to look at each of those allocation requests to figure out how many NUMA node providers are in it. | 14:42 |
| *** markvoelker has quit IRC | 14:42 | |
| *** markvoelker has joined #openstack-nova | 14:42 | |
| dansmith | mriedem: ^ | 14:42 |
| efried | bauzas: ...and compare that number against N in hw:numa_nodes=N. If they don't match, the filter removes that allocation request from the list. | 14:42 |
| dansmith | certs were supposed to go until monday | 14:43 |
| mriedem | i didn't move these | 14:43 |
| mriedem | Certificate Validation - https://blueprints.launchpad.net/nova/+spec/nova-validate-certificates [END DATE: 2018-04-16] | 14:43 |
| mriedem | yeah i don't know why that moved | 14:43 |
| dansmith | right | 14:43 |
| dansmith | and the top one | 14:43 |
| dansmith | is supposed to be kicked out | 14:43 |
| dansmith | maybe we remove the top one and put certs back in its place? | 14:43 |
| mriedem | sounds like we need a CA on this one 10-4 | 14:44 |
| efried | bauzas: Having done that, let's say we've filtered our 10 allocation requests down to 4. Of those, one is on host1, three are on host2, and host3 is no longer in the picture (we filtered out all of his allocation requests). | 14:44 |
| mriedem | yeah probably | 14:44 |
| dansmith | mriedem: you gonna do that? | 14:44 |
| mriedem | dansmith: you do it | 14:44 |
| bauzas | efried: wait | 14:44 |
| efried | bauzas: So now we can land on host1 or host2 (but not host3). If we land on host1, we have to use that one allocation request for that guy. If we land on host2, we have a choice of three allocation requests. | 14:44 |
| efried | bauzas: waiting... | 14:44 |
| *** beagles is now known as beagles_brb | 14:45 | |
| bauzas | efried: the fact that we have multiple allocation requests related to one host shouldn't impact filters | 14:45 |
| *** AlexeyAbashkin has quit IRC | 14:45 | |
| dansmith | mriedem: actually melwitt should be around in 15 mins, so let's just wait | 14:45 |
| bauzas | efried: because filters interface is against *host*, not allocation request | 14:45 |
| dansmith | she clearly logged that she removed it yesterday | 14:45 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | 14:45 |
| *** gameon has joined #openstack-nova | 14:46 | |
| efried | bauzas: That's going to need to be rethought, then. | 14:46 |
| bauzas | efried: what needs to rethought ? | 14:47 |
| efried | bauzas: If we can't have NUMATopologyFilter winnow down the list of *allocation requests* then we're back to the drawing board. | 14:47 |
| *** Zames has quit IRC | 14:47 | |
| jaypipes | mriedem, dansmith: your review needed on https://review.openstack.org/#/c/556873/ pls (nested allocation candidates spec) | 14:47 |
| bauzas | efried: I considered Placement as the way to winnow down the list of hosts we were checking | 14:47 |
| *** markvoelker has quit IRC | 14:47 | |
| bauzas | efried: so that's still a net win | 14:48 |
| gameon | Hello all - I am trying to configure live migration between hosts, I have a Broadwell 56 core host and a 32 core Haswell server. Both single sockets. I have set cpu_mode=custom and cpu_model to various things, kvm64, haswell, SandyBridge - but still I get an error about CPU incompatibility when attempting to live migration from 32 to 56 core hosts. Is there any way of fixing this? I thought the configuration of custom mode woul | 14:48 |
| mriedem | gameon: please see channel topic | 14:48 |
| efried | bauzas: Because in our scenario above, we've filtered down to host1 and host2, great, let's say we pick host1 -we can't just pick *any* allocation request that relates to host1. Because some of those still have allocation requests where <#numa node RPs> != hw:numa_nodes from extra specs. | 14:48 |
| gameon | mriedem: Sorry I missed that. | 14:48 |
| bauzas | efried: where is the spec describing the query side of nested RPs ? | 14:48 |
| bauzas | efried: probably worth hangouting you know | 14:49 |
| efried | bauzas: https://review.openstack.org/#/c/556873/ | 14:49 |
| efried | bauzas: But I'm not sure that's going to help you much. | 14:50 |
| bauzas | efried: well, I'm not in need of anything | 14:50 |
| bauzas | efried: here, I'm just saying "let's use Placement to winnow the list of hosts" | 14:50 |
| bauzas | ah snap | 14:50 |
| bauzas | it will work for NUMA | 14:50 |
| bauzas | but not for VGPU | 14:51 |
| bauzas | oh wait | 14:51 |
| efried | separate use case, let's focus on one at a time. | 14:51 |
| bauzas | it will | 14:51 |
| bauzas | no no | 14:51 |
| bauzas | sec | 14:51 |
| efried | no, it won't. But separate use case. | 14:51 |
| bauzas | trying to wrap around my heard | 14:51 |
| bauzas | head | 14:51 |
| bauzas | so | 14:51 |
| bauzas | we said we're going to get a list of allocation requests | 14:51 |
| *** rajinir has joined #openstack-nova | 14:51 | |
| bauzas | then, each filter will go thru the list of corresponding hosts and do the checks they want - which are unrelated to what Placement checked | 14:52 |
| bauzas | at the end, we'll figure out that, say host345, host346 are valid | 14:52 |
| bauzas | then, we'll look back at the allocation candidates | 14:52 |
| bauzas | and claim against one of the allocation requests corresponding to those hosts | 14:53 |
| bauzas | what filtering will do is just reducing that list of allocation requests to the ones that are related to hosts that match filtering | 14:53 |
| efried | Right, but since the filter only filtered *hosts*, there can still be allocation requests we can't use. | 14:54 |
| efried | Placement will winnow the list of hosts so far. The filter can winnow the list of hosts further. But - assuming the filter *only* returns a list of hosts - what's going to be left after that is still going to include allocation requests we can't use. And now there's nobody left to filter those out. | 14:54 |
| *** kholkina has quit IRC | 14:54 | |
| efried | bauzas: So what I was saying we could reimagine was the role of filters in this flow. | 14:54 |
| bauzas | efried: why should we filter more ? | 14:54 |
| sahid | gameon: you should try with cpu_mode=none | 14:54 |
| efried | bauzas: Instead of being just focused on hosts, a filter is allowed to filter allocation requests too. | 14:55 |
| bauzas | efried: filters give you a subset of allocation requests that are supported | 14:55 |
| *** Spaz-Work has quit IRC | 14:55 | |
| bauzas | efried: please no. | 14:55 |
| efried | bauzas: That's not what I understood from what you said above. | 14:55 |
| sahid | so no check are done by in libvirt layer but it's possible that qemu failed to start the -incoming process on destination | 14:55 |
| efried | bauzas: What I understood you to say was that filters give you a subset of *hosts*. | 14:55 |
| sahid | at least you should see the log o QEMU which will indicated which cpu feature is not supported by destination | 14:56 |
| bauzas | they give you a subset of hosts, which turns nova into knowing which allocation requests are valid accordingly | 14:56 |
| *** gouthamr has quit IRC | 14:56 | |
| efried | bauzas: Exactly | 14:56 |
| bauzas | so, we're cool | 14:56 |
| *** artom has joined #openstack-nova | 14:56 | |
| bauzas | in the example of NUMA | 14:56 |
| efried | bauzas: No. The problem is that there are still invalid allocation requests for the remaining hosts | 14:56 |
| bauzas | oh right, because the NUMA filter did crazy things | 14:56 |
| bauzas | so it said "that NUMA node, I'll take it" | 14:57 |
| bauzas | but it doesn't really do that you know | 14:57 |
| bauzas | it just consider the host valid | 14:57 |
| gameon | sahid: setting it to none makes no difference, I get the same error. I can't see which feature set isn't supported. I am beginning to think it's to do with the number of cores on the hosts. Even kvm64 doesn't work for live migration from 56 to 32 cores (it's a 1 core VM..) | 14:57 |
| *** READ10 has joined #openstack-nova | 14:57 | |
| bauzas | efried: the NUMA attachment is a late bind on compute | 14:57 |
| bauzas | efried: IIUC the construct | 14:58 |
| *** mvk has joined #openstack-nova | 14:58 | |
| sahid | gameon: what qemu is saying? /var/log/libvirt/qemu/instance-xxx.log | 14:58 |
| *** artom_ has joined #openstack-nova | 14:58 | |
| *** Spaz-Work has joined #openstack-nova | 14:58 | |
| efried | bauzas: Example: flavor says hw:numa_nodes=2. So we ask for resources1=VCPU:1&resources2=VCPU:1. We get back the following from GET /allocation_candidates: | 14:58 |
| efried | [ [ host1_NUMA0: { VCPU: 1 }, host1_NUMA1: { VCPU: 1 } ], | 14:58 |
| efried | [ host1_NUMA0: { VCPU: 2 } ], | 14:58 |
| efried | [ host2_NUMA1: { VCPU: 2 } ] | 14:58 |
| efried | ] | 14:58 |
| *** hamzy has quit IRC | 14:58 | |
| *** ralonsoh__ has quit IRC | 14:59 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | 14:59 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560930 | 14:59 |
| sahid | gameon: an other point, after updating nova.conf you should restart the service and also create a new guest | 14:59 |
| efried | bauzas: NUMATopologyFilter looks through that, sees that host1 has a valid allocation request - the first one - because it represents two NUMA nodes and we want to adhere to hw:numa_nodes=2. | 14:59 |
| efried | bauzas: So it chucks out host2, and returns "here is your valid list of hosts: [host1]" | 14:59 |
| *** DinaBelova has quit IRC | 15:00 | |
| *** aignatov has quit IRC | 15:00 | |
| efried | bauzas: Next step in the flow says, "Oh, cool, host1 is fine; let's go see what allocation requests are available for host1" | 15:00 |
| efried | bauzas: ...and it picks the second one, which is wrong. | 15:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560931 | 15:00 |
| bauzas | efried: so that's a good reason to port that logic to Placement then | 15:01 |
| bauzas | so we can get rid of that crazy filter which makes assumptions that I disagree | 15:01 |
| efried | bauzas: Or make a filter more fine-grained so that it actually returns the allocation requests instead of just the host name. | 15:01 |
| bauzas | efried: the filters return a boolean | 15:01 |
| gameon | sahid: Thanks for this, I am restarting the service and also the guest which starts it with the updated CPU feature. I have found ' host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]' so I guess that needs to be investigated | 15:02 |
| bauzas | efried: what you ask is a post-placement-filtering | 15:02 |
| *** artom has quit IRC | 15:02 | |
| bauzas | efried: not a scheduler filter adaptation | 15:02 |
| bauzas | efried: wrt what dansmith implemented | 15:02 |
| *** r-daneel has quit IRC | 15:02 | |
| bauzas | whatever the name was, I don't recall exactly | 15:02 |
| *** gouthamr has joined #openstack-nova | 15:02 | |
| efried | bauzas: I admit I don't understand the sequence of events. But yes, I was originally assuming filters are (or can be) applied *after* GET /allocation_candidates. Is that wrong? | 15:02 |
| bauzas | efried: you're still right | 15:03 |
| efried | bauzas: I was also assuming there was nothing stopping us from allowing filters to cull allocation requests. | 15:03 |
| bauzas | efried: the filtering logic is based on iterating over hosts | 15:03 |
| efried | bauzas: But what's the input to the filter? | 15:03 |
| bauzas | efried: sec, giving you code | 15:03 |
| efried | bauzas: Does the filter get to see the result of GET /allocation_candidates? | 15:03 |
| bauzas | as of now, not | 15:04 |
| bauzas | efried: that's the filter interface https://github.com/openstack/nova/blob/master/nova/scheduler/filters/__init__.py#L46 | 15:04 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: RT: replace _instance_in_resize_state with _is_trackable_migration https://review.openstack.org/560467 | 15:04 |
| *** itlinux has quit IRC | 15:05 | |
| bauzas | efried: and that's were we iterate over the list of hosts https://github.com/openstack/nova/blob/master/nova/filters.py#L43-L44 | 15:05 |
| efried | bauzas: What's in filter_properties? | 15:05 |
| *** Eran_Kuris has quit IRC | 15:05 | |
| bauzas | efried: it's the RequestSpec object | 15:06 |
| bauzas | eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L34 | 15:06 |
| efried | bauzas: And does RequestSpec contain the GET /allocation_candidates results? | 15:06 |
| bauzas | we could do that | 15:06 |
| bauzas | but that's not yet the case | 15:06 |
| bauzas | so we could pass the list of requests per host to the filter | 15:07 |
| bauzas | up to the filter to read that info | 15:07 |
| *** burt has joined #openstack-nova | 15:07 | |
| *** AlexeyAbashkin has joined #openstack-nova | 15:07 | |
| bauzas | or we could do a post-filtering-logic, exactly like we have now a pre-filtering-logic | 15:07 |
| bauzas | well | 15:07 |
| bauzas | pre-placement rather | 15:08 |
| sahid | gameon: a possible solution for you now is to set a model that is supported by both of your hosts | 15:08 |
| gameon | sahid: actually that was a red herring, there is no warning in the qemu log when I try to live migrat - svm is for AMD anyway. I have tried setting the most basic model, kvm64 but still it doesnt work.. | 15:08 |
| *** germs has quit IRC | 15:08 | |
| efried | bauzas: post-filtering logic, at the level of allocation requests, is what is needed here, unless we're going to implement it in placement. | 15:09 |
| *** germs has joined #openstack-nova | 15:09 | |
| *** germs has quit IRC | 15:09 | |
| *** germs has joined #openstack-nova | 15:09 | |
| *** ccamacho has quit IRC | 15:09 | |
| efried | bauzas: Sorry, not post-filtering necessarily | 15:09 |
| bauzas | efried: okay, I think I have enough in mind to write in a spec | 15:10 |
| efried | bauzas: Filtering of the result of GET /allocation_candidates. Wherever/however that happens to be. | 15:10 |
| bauzas | efried: so we could chime in that about the possible implementation options we have | 15:10 |
| efried | bauzas: Yes. I see us at some point needing to do filtering of GET /allocation_candidates results. This would be a good use case for setting that up. | 15:11 |
| efried | bauzas: By the way, I assume a weigher has the same issue. It just weighs hosts, not allocation requests. | 15:11 |
| *** AlexeyAbashkin has quit IRC | 15:12 | |
| efried | bauzas: I think that needs to be re-imagined in the same way. | 15:12 |
| sahid | gameon: if you set a specific model you have to configure cpu_mode=custom, that is what you did? | 15:13 |
| gameon | sahid: Yeah I did do that, no dice :( | 15:14 |
| sahid | gameon: that with master? | 15:14 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | 15:14 |
| gameon | sahid: only on hypervisors, not the scheduler or api node | 15:14 |
| sahid | gameon: i mean you are using master branch of nova? | 15:15 |
| gameon | Pike | 15:15 |
| *** DinaBelova has joined #openstack-nova | 15:15 | |
| *** jaosorior has quit IRC | 15:15 | |
| *** aignatov has joined #openstack-nova | 15:15 | |
| bauzas | efried: I don't like the wording "reimagined" for filters and weighers | 15:16 |
| bauzas | efried: given it's one of the most custom pieces we have in Nova, with a ton of operators having their own filters, changing that isn't trivial | 15:17 |
| bauzas | efried: but adding some extra field to the RequestSpec object seems fine with me | 15:17 |
| efried | bauzas: yes, fair enough; it may have to be a new/different kind of filter. | 15:17 |
| bauzas | so that filters can opt-in and check if necessary | 15:17 |
| efried | bauzas: Because filtering at the host level is simply not going to be enough long-term. | 15:17 |
| efried | bauzas: It doesn't have to be customizable or opt-in for the use case we're talking about. | 15:18 |
| *** yangyapeng has joined #openstack-nova | 15:18 | |
| efried | bauzas: Other than in the sense they're customizing/opting-in based on what they put in their flavor | 15:18 |
| jaypipes | bauzas: Besides "check my switch to see if it's on before scheduling to this node", what are the custom filters/weighers you have seen from operators? | 15:18 |
| bauzas | efried: the problem is really because of backwards compatibility you know | 15:18 |
| bauzas | efried: for the VGPU usecase, I don't care about that because scheduler will pick one allocation for me | 15:19 |
| efried | bauzas: I contend that this is the same. | 15:19 |
| *** vladikr_ has joined #openstack-nova | 15:19 | |
| bauzas | I don't have to make sure to mimic any pre-existing logic | 15:19 |
| melwitt | dansmith, mriedem: ah, dammit. I messed up thinking the cert one was supposed to end on 2018-04-11. sorry | 15:19 |
| *** beagles_brb is now known as beagles | 15:19 | |
| efried | bauzas: The scheduler has to pick one allocation request based on the numa topology in the flavor. | 15:19 |
| *** damien_r has quit IRC | 15:21 | |
| gameon | sahid: OK - setting them to kvm64 has seemed to have a different result. Maybe I didn't restart the service. Now I have 'Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor' | 15:21 |
| *** hamzy has joined #openstack-nova | 15:21 | |
| bauzas | efried: for VGPUs ? | 15:22 |
| *** vladikr has quit IRC | 15:22 | |
| efried | bauzas: For NUMA | 15:22 |
| sahid | gameon: ok in same time i was trying to find a patch which address an issue with compare CPU but it seems that it's already on Pike https://review.openstack.org/#/c/53746/ | 15:22 |
| bauzas | efried: I'm confused | 15:23 |
| efried | bauzas: So maybe "filter" is the wrong word for it. The scheduler is going to need to pick from among the allocation requests returned by GET /a_c. It's gotta employ *some* kind of logic to do that. | 15:23 |
| sahid | gameon: I can't help you for the issue with apparmar perhaps you could try #virt in OFTC | 15:23 |
| bauzas | efried: I think we agreed something was necessary for keeping the existing behaviour | 15:23 |
| gameon | sahid: Thank you for pointing me in the right direction, much appriciated | 15:23 |
| bauzas | efried: because of some assumption from the filter | 15:24 |
| bauzas | efried: that's usecase #1 in my spec | 15:24 |
| efried | bauzas: In the case we're talking about, the NUMATopologyFilter may (or may not) have already filtered down to a certain subset of hosts. Now the scheduler knows it can ignore allocation requests related to those hosts. But it still has to employ additional logic to ignore allocation requests that don't have the right number of numa nodes in them. | 15:24 |
| bauzas | efried: for usecase #2 (which also matches the SR-IOV usecase), we don't care | 15:24 |
| bauzas | efried: right, I don't disagree with that | 15:24 |
| efried | bauzas: That being the case, I'm not sure NUMATopologyFilter is actually doing us any good here. | 15:25 |
| *** sree has joined #openstack-nova | 15:25 | |
| bauzas | that's the stephenfin vs. sahid point | 15:25 |
| efried | bauzas: Because the scheduler is having to do the same logic anyway. The difference is that NTF had to throw away a lot of its work to just say "host is valid or not". | 15:25 |
| bauzas | if we want to keep existing behaviour, we need something to force allocation requests to be disregarded by the filter | 15:25 |
| efried | bauzas: Or the scheduler. And eff the filter. | 15:26 |
| *** moshele has quit IRC | 15:26 | |
| *** jcosmao has joined #openstack-nova | 15:26 | |
| cfriesen | jaypipes: currently we have a "does this compute node have at least as good a CPU model as we asked for" filter | 15:27 |
| *** jackie-truong has joined #openstack-nova | 15:29 | |
| cfriesen | jaypipes: we also have a "does this node have access to the physical networks required by the instance" filter | 15:29 |
| melwitt | jackie-truong: hey, if you had noticed earlier that your blueprint got removed from the runway, that was an accident, sorry. I put it back. end date is EOD on April 16 | 15:29 |
| *** sree has quit IRC | 15:30 | |
| jackie-truong | melwitt: No problem! I added a documentation patch to the list, too | 15:30 |
| *** cdent has quit IRC | 15:30 | |
| melwitt | k, cool | 15:30 |
| *** markvoelker has joined #openstack-nova | 15:31 | |
| cfriesen | efried: the filter is part of the scheduler | 15:31 |
| *** AlexeyAbashkin has joined #openstack-nova | 15:32 | |
| *** andreas_s has joined #openstack-nova | 15:32 | |
| cfriesen | efried: or are you using different terminology than I'm used to? | 15:32 |
| efried | cfriesen: But IIUC, filters like NUMATopologyFilter are opt-in, and have a predefined interface (so folks can create custom ones), and only let you say "this host is good or bad". | 15:32 |
| *** gyee has joined #openstack-nova | 15:32 | |
| *** cdent has joined #openstack-nova | 15:32 | |
| cfriesen | efried: to me the scheduler is placement+filters+weighers | 15:32 |
| efried | cfriesen: I'm saying what we need here to make the world sane is a piece of code that "filters" allocation requests out of the result of GET /allocation_candidates. | 15:33 |
| *** felipemonteiro__ has joined #openstack-nova | 15:33 | |
| efried | cfriesen: In order for that code to be in a "filter" (as described above), we would have to extend/reinvent that predefined interface. | 15:33 |
| efried | cfriesen: But if we do that filtering outside of that interface, we're not restricted. | 15:34 |
| efried | cfriesen: Somewhere in the scheduler there is a piece of code that looks at the list of allocation requests we get back from placement, and picks one. | 15:34 |
| efried | (or three) | 15:34 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Don't persist RequestSpec.retry https://review.openstack.org/560143 | 15:35 |
| efried | cfriesen: We need to get our dirty hands into that algorithm in order to satisfy this use case - unless we can find a way for the filtering to be done 100% in placement. | 15:35 |
| *** dansmith changes topic to "Current runways: add-zvm-driver-rocky / nova-validate-certificates / placement-forbidden-traits -- This channel is for Nova development. For support of Nova deployments, please use #openstack." | 15:37 | |
| *** ChanServ sets mode: -o dansmith | 15:37 | |
| *** felipemonteiro_ has quit IRC | 15:37 | |
| edleafe | efried: I really don't like any approach that doesn't treat allocation_candidates as opaque | 15:37 |
| efried | edleafe: Brace yourself. It's going to happen eventually. If not for this, then for something. | 15:38 |
| cfriesen | efried: looks like SchedulerManager.select_destinations() calls out to placement, then passes the returned information down to FilterScheduler.select_destinations() | 15:38 |
| efried | edleafe: Because we can't expect to implement *all* weighing on the placement side. | 15:38 |
| efried | cfriesen: Is a "destination" a host in that context? | 15:39 |
| efried | (trying not to look at code; have a pile of reviews I'm already in the middle of) | 15:39 |
| cfriesen | efried: yes. Looks like when we call down to the actual filters (via _get_sorted_hosts() in this case, we don't pass the allocation candidates) | 15:39 |
| edleafe | efried: it doesn't have to happen | 15:39 |
| *** markvoelker_ has joined #openstack-nova | 15:40 | |
| edleafe | I really oppose making the internal structure of an a-c part of the API contract | 15:40 |
| *** artom_ is now known as artom | 15:43 | |
| efried | edleafe: /me predicts that that will be the path of least resistance when the alternative is implementing every conceivable filtering/weighing algorithm natively in placement. | 15:43 |
| cfriesen | edleafe: I can see what efried is saying though...placement is returning allocation candidates, not hosts. Something in the rest of the code needs to understand how to map that allocation candidate to the actual resources being used. | 15:44 |
| *** markvoelker has quit IRC | 15:44 | |
| efried | cfriesen: That's slightly different. The format of the *allocation* is part of the contract, and the virt driver (or whatever) can introspect it to map to actual resources. | 15:44 |
| gameon | sahid: Thanks again for your help - I have resolved the issue. It turns out Broadwell doesn't have all Haswell features and vice versa. So I've used IvyBridge | 15:44 |
| efried | cfriesen: Although it amounts to the same thing, really. | 15:45 |
| cfriesen | gameon: what was in haswell that's not in broadwell? | 15:45 |
| efried | cfriesen: So yeah, edleafe there's your counterargument. | 15:45 |
| sahid | gameon: cool :) | 15:46 |
| edleafe | efried: I would much prefer modifying the provider info than the a-c | 15:47 |
| efried | edleafe: But our problem isn't picking a provider. It's picking an allocation request. | 15:47 |
| gameon | cfriesen: this was generated by running virsh capabilities - I the features reported by 'cat /proc/cpuinfo' are the same... http://paste.openstack.org/show/719064/ | 15:48 |
| efried | edleafe: ...from among possibly many for a given provider. | 15:48 |
| efried | edleafe: Besides, we're not talking about modifying anything about the format of the response. | 15:48 |
| gameon | cfriesen: happy to hear alternative solutions to setting model to IvyBridge there's a better way of allowing LM between these set of hosts | 15:48 |
| efried | edleafe: But we are talking about introspecting that payload. Which nothing in the doc implies we should avoid... | 15:49 |
| edleafe | efried: but it also doesn't guarantee that the payload structure will never change | 15:49 |
| *** sidx64 has quit IRC | 15:49 | |
| *** edmondsw has joined #openstack-nova | 15:50 | |
| edleafe | "Introspect at your own risk" :) | 15:50 |
| cfriesen | gameon: according to my libvirt cpu model definitions Broadwell should be a superset of Haswell. | 15:50 |
| bauzas | edleafe: just a bit of explanation | 15:50 |
| cfriesen | gameon: and you might want to use the "noTSX" versions | 15:50 |
| bauzas | edleafe: we discussed since 3 hours with efried about the possible implementation for NUMA | 15:51 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Don't persist RequestSpec.retry https://review.openstack.org/560146 | 15:51 |
| bauzas | edleafe: the problem is that the NUMA filter makes some assumptions that are not supported by the numbered request groups feature | 15:51 |
| gameon | cfriesen: I'll give it a go, let's see. So set it to Broadwell or Haswell do you think? | 15:51 |
| bauzas | edleafe: the one in particular is that it checks whether the host has the exact same topology than the asked guest one | 15:51 |
| bauzas | edleafe: which seems weird to me, but meh | 15:52 |
| cfriesen | gameon: interesting that it doesn't report your Haswell as a Haswell. | 15:52 |
| bauzas | edleafe: accordingly, that means that placement will return some allocation requests that are valid from his point of view, but not from the filter's PoV | 15:52 |
| bauzas | edleafe: hence the need of some post-filtering that would only assure the same behaviour | 15:53 |
| bauzas | but | 15:53 |
| gameon | cfriesen: Yes I noticed this, reported as Nehalem which is a bit puzzling... | 15:53 |
| bauzas | there is a but | 15:53 |
| cfriesen | gameon: According to what you've got there I'd use Nehalem, unless you need the newer features. | 15:53 |
| *** sahid has quit IRC | 15:53 | |
| *** yamahata has quit IRC | 15:53 | |
| *** edmondsw has quit IRC | 15:53 | |
| bauzas | edleafe: one other option could be to consider that placement should return the exact same candidates as the ones that the filter would agree | 15:53 |
| bauzas | edleafe: in that case, it would mean some new query param for Placement | 15:54 |
| bauzas | thoughts ? | 15:54 |
| gameon | cfriesen: Nehalem works :) | 15:55 |
| *** sree has joined #openstack-nova | 15:55 | |
| efried | edleafe: The way the doc is written, it *is* guaranteeing that the format won't change (without a new microversion). | 15:56 |
| efried | which is as it should be. | 15:56 |
| edleafe | efried: about to run off to a meeting | 15:57 |
| efried | I've never understood the reasoning behind that payload needing to be opaque, btw. | 15:57 |
| edleafe | efried: all it was supposed to be was something you could send back to allocate/claim | 15:57 |
| cfriesen | bauzas: it seems to me that we have two options. give the filters access to the allocation candidates so they can rule out ones they don't like, or have enough flexibility in placement that we can ensure we never get back invalid candidates. | 15:57 |
| edleafe | it could have been a hash, or a uuid, or... | 15:58 |
| efried | edleafe: And you're supposed to pick one based on... what? | 15:58 |
| *** edmondsw has joined #openstack-nova | 15:58 | |
| cfriesen | edleafe: for that to work we need enough flexibility in what we can request from placement to ensure that all the candidates are valid | 15:58 |
| *** Nel1x has joined #openstack-nova | 15:58 | |
| edleafe | now we have nested, which brings in multiple a-cs per rp | 15:58 |
| melwitt | efried: when you get a chance, wanna write some notes on how the update-provider-tree runway review went at L118? https://etherpad.openstack.org/p/nova-runways-rocky | 15:58 |
| efried | melwitt: ack | 15:58 |
| edleafe | efried: you picked a host, and found the matching a-c | 15:58 |
| efried | edleafe: There were several a-cs for that host. | 15:59 |
| *** l4yerffej has quit IRC | 15:59 | |
| efried | edleafe: How did I pick one? | 15:59 |
| edleafe | now there isn't a 1:1 host:a-c relationship with nesting | 15:59 |
| cfriesen | efried: if they were all actually valid it wouldn't matter | 15:59 |
| efried | cfriesen: Yup, that's the big IF. | 15:59 |
| efried | cfriesen: For this use case, we either implement new logic in placement, or they're *not* all valid and we can't just pick one at random. | 16:00 |
| *** felipemonteiro__ has quit IRC | 16:00 | |
| cfriesen | efried: yes, agreed | 16:00 |
| efried | edleafe: There was never a 1:1 host:a-c relationship. | 16:00 |
| *** efried has quit IRC | 16:03 | |
| *** efried has joined #openstack-nova | 16:04 | |
| *** do3meli has quit IRC | 16:04 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Don't persist RequestSpec.retry https://review.openstack.org/560955 | 16:05 |
| bauzas | efried: just for the context, they're not all valid because of some specific implementation details of the filter that I don't particularly like | 16:06 |
| *** caisan has quit IRC | 16:06 | |
| efried | melwitt: done | 16:06 |
| bauzas | adding more debt to either placement or the filters looks terrible to me | 16:06 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Don't persist RequestSpec.retry https://review.openstack.org/560167 | 16:07 |
| melwitt | efried: thanks | 16:07 |
| *** l4yerffej has joined #openstack-nova | 16:07 | |
| bauzas | efried: from a placement perspective, we can ask for a query that would *shard* resources between children, but that's the only trade-off I'd make | 16:07 |
| bauzas | efried: if we implement such thing, then we wouldn't need to pass the candidates down to the filters | 16:08 |
| efried | bauzas: It's starting to sound like that might be the "easier" option | 16:08 |
| bauzas | not the easier | 16:08 |
| efried | bauzas: And as I've said, we know we're going to want that logic in placement eventually regardless. | 16:08 |
| bauzas | the less debtful | 16:08 |
| efried | bauzas: So this might as well be the motivation. | 16:08 |
| efried | Yeah | 16:08 |
| bauzas | ok, I'll amend my spec accordingly | 16:09 |
| edleafe | efried: sorry, in the API-SIG meeting | 16:09 |
| edleafe | efried: https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L154-L160 | 16:09 |
| edleafe | That creates one a-c per rp | 16:09 |
| melwitt | cdent: the placement-forbidden-traits blueprint has been added to a review runway. please ack if the next two weeks work for you for quick iteration on review | 16:10 |
| edleafe | we always just grab the first one of that list | 16:10 |
| *** lucasagomes is now known as lucas-afk | 16:11 | |
| cdent | melwitt: thanks, it does | 16:11 |
| melwitt | k, great | 16:11 |
| efried | edleafe: It creates a *list* of allocation requests per rp_uuid | 16:11 |
| efried | edleafe: ...by introspecting the payload, by the way :P | 16:11 |
| *** gameon has quit IRC | 16:13 | |
| mriedem | lyarwood: for https://review.openstack.org/#/c/543569/ - do we want a security reno for https://bugs.launchpad.net/nova/+bug/1739593 and CVE-2017-18191? | 16:13 |
| openstack | Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)" [Undecided,Incomplete] | 16:13 |
| mriedem | i see the ossa isn't published | 16:13 |
| *** itlinux has joined #openstack-nova | 16:13 | |
| melwitt | jichen: the z/VM driver series has been added to a review runway. I know you have been active on the patches already but please let us know if there are any problems with the next two weeks for quick iteration on review | 16:14 |
| edleafe | efried: this is the part that I was referring to: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L212-L217 | 16:14 |
| edleafe | "information in the provider summaries" | 16:14 |
| *** armaan has quit IRC | 16:15 | |
| *** armaan has joined #openstack-nova | 16:15 | |
| *** fragatina has joined #openstack-nova | 16:17 | |
| *** fragatina has quit IRC | 16:17 | |
| efried | edleafe: Yeah, I get that we can do *some* weighing/filtering based on the provider summaries; but that still only gets us down to the list of allocation requests for a given host. It doesn't help us pick among those. | 16:17 |
| *** fragatina has joined #openstack-nova | 16:18 | |
| *** ssurana has joined #openstack-nova | 16:18 | |
| *** ssurana has quit IRC | 16:19 | |
| *** ssurana has joined #openstack-nova | 16:19 | |
| mriedem | lyarwood: sounds like the ossa is blocked until the stable/ocata patch is up | 16:20 |
| mriedem | i'm +1 on the stable/pike change now if you want to start on the stable/ocata backport | 16:20 |
| *** ccamacho has joined #openstack-nova | 16:22 | |
| *** ssurana has quit IRC | 16:23 | |
| *** ccamacho has quit IRC | 16:24 | |
| *** ccamacho has joined #openstack-nova | 16:24 | |
| *** yamahata has joined #openstack-nova | 16:26 | |
| openstackgerrit | Brianna Poulos proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 16:27 |
| cfriesen | mriedem: lyarwood: I've got https://review.openstack.org/#/c/560690/ up for the stable/pike backport, but there's a complication in that Pike treats the encryption stuff a bit differently. Wondering how you want to handle it. I wrote it up in the review. | 16:28 |
| mriedem | cfriesen: heh, see https://review.openstack.org/#/c/543569/ | 16:29 |
| mriedem | cfriesen: comments inline | 16:32 |
| *** udesale has quit IRC | 16:32 | |
| openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | 16:32 |
| *** tssurya has quit IRC | 16:34 | |
| mriedem | cfriesen: oh yeah reading https://review.openstack.org/#/c/460243/ i see the problem kind of, | 16:34 |
| mriedem | that was really a frankenstein of a patch, and should have been split up | 16:34 |
| *** sdague has quit IRC | 16:35 | |
| mriedem | https://review.openstack.org/#/c/460243/16/nova/virt/libvirt/driver.py@1453 specifically | 16:36 |
| mriedem | if that's its own bug, we'd have to backport separately before your change, but would need to talk to lyarwood | 16:36 |
| mriedem | i don't know if that applies before the changes to _disconnect_volume though | 16:36 |
| mriedem | if not, don't worry about it | 16:36 |
| *** gouthamr has quit IRC | 16:38 | |
| cfriesen | mriedem: If we wanted to backport "proper" handling of encrypted volumes in the error case it could be done in a separate patch, I don't think the ordering really matters. For now I'll rework the backport to go off the stable/queens one. | 16:38 |
| *** tbachman has quit IRC | 16:39 | |
| mriedem | i think the encrypted volume stuff only needed to change because of the behavior change in _disconnect_volume for luks native encryption | 16:39 |
| mriedem | which we're not going to backport | 16:39 |
| *** mdbooth has quit IRC | 16:39 | |
| cfriesen | mriedem: well...with my fix if we hit exception.DeviceNotFound we'll continue on without every calling encryptor.detach_volume() | 16:41 |
| *** tbachman has joined #openstack-nova | 16:42 | |
| cfriesen | mriedem: so I was wondering if we should move the call to ncryptor.detach_volume() down right above the call to self._disconnect_volume(), but I didn't know enough about that code to know if that was okay. | 16:44 |
| cfriesen | it *seems* analogous to what happens in the newer code, but I could be missing something | 16:44 |
| *** moshele has joined #openstack-nova | 16:44 | |
| mriedem | i defer to lyarwood | 16:45 |
| *** yamamoto has quit IRC | 16:47 | |
| *** fragatina has quit IRC | 16:47 | |
| *** andreas_s has quit IRC | 16:49 | |
| *** jmccarthy1 has joined #openstack-nova | 16:49 | |
| *** jmccarthy1 has left #openstack-nova | 16:49 | |
| openstackgerrit | Chris Friesen proposed openstack/nova stable/pike: libvirt: disconnect volume from host during detach https://review.openstack.org/560690 | 16:50 |
| melwitt | cfriesen: see the earlier version of the patch from before the encryption-related refactor https://review.openstack.org/#/c/515008/9/nova/virt/libvirt/driver.py | 16:50 |
| *** udesale has joined #openstack-nova | 16:51 | |
| *** dpawlik has joined #openstack-nova | 16:52 | |
| efried | melwitt: Care to have a look at https://review.openstack.org/#/c/553475/ ? Then we can put update_provider_tree to bed. | 16:53 |
| efried | melwitt: Should be an easy on. | 16:53 |
| efried | one | 16:53 |
| melwitt | sure | 16:54 |
| cfriesen | melwitt: perfect, that's exactly what I was thinking about doing | 16:55 |
| *** dpawlik has quit IRC | 16:56 | |
| *** dtruong_ has quit IRC | 16:56 | |
| *** udesale has quit IRC | 16:56 | |
| *** dtruong has joined #openstack-nova | 16:57 | |
| *** fragatina has joined #openstack-nova | 16:57 | |
| *** arvindn05_ has joined #openstack-nova | 16:57 | |
| *** udesale has joined #openstack-nova | 16:58 | |
| *** udesale has quit IRC | 16:58 | |
| *** mdnadeem has quit IRC | 16:58 | |
| *** yamamoto has joined #openstack-nova | 16:58 | |
| *** ssurana has joined #openstack-nova | 17:00 | |
| *** elmaciej has joined #openstack-nova | 17:00 | |
| *** fragatina has quit IRC | 17:02 | |
| *** yamamoto has quit IRC | 17:03 | |
| *** dklyle has quit IRC | 17:07 | |
| efried | mikal: What's your feel on deferring the requirements issue out of the zvm driver series? | 17:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Numbered request groups use different providers https://review.openstack.org/560974 | 17:09 |
| *** sdague has joined #openstack-nova | 17:09 | |
| *** janki has quit IRC | 17:09 | |
| mriedem | efried: like we did in the powervm series? :) | 17:10 |
| efried | mriedem: Sure. I.e. nobody cared enough to pursue it, so it dropped. That's as it should be, if nobody cares enough to pursue it. | 17:10 |
| mriedem | i was also going to mention in that ML thread, btw, that if we did get pedantic about requirements, os-brick would also fall into that camp since only the libvirt and hyperv driver use it | 17:11 |
| efried | mriedem: The ML thread is making it clearer with every note that this is a bigger issue than we can/should expect to solve in the zvm driver series. | 17:11 |
| openstackgerrit | Chris Friesen proposed openstack/nova stable/pike: libvirt: disconnect volume from host during detach https://review.openstack.org/560690 | 17:13 |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | 17:13 |
| mriedem | i haven't read the latest | 17:13 |
| mriedem | i know what we'll do, | 17:14 |
| mriedem | i'll run for TC, | 17:14 |
| mriedem | and then push through that all projects must define optional requirements in [extras] | 17:14 |
| *** gouthamr has joined #openstack-nova | 17:14 | |
| mriedem | as a community wide goal | 17:14 |
| dansmith | mriedem: os-brick isn't really environment specific as much though | 17:14 |
| dansmith | well, maybe that's not true, I guess it runs linux commands | 17:14 |
| mriedem | dansmith: this zvm lib dep isn't conditional on arch right? | 17:15 |
| dansmith | but, it seems less confusingly installed than my linux machine with powervm and zvm stuff both installed | 17:15 |
| dansmith | mriedem: arch or platform? | 17:15 |
| dansmith | mriedem: I assume you run nova in linux land on z, so not platform | 17:15 |
| dansmith | and maybe not even on an s390x if it's like an HMC | 17:16 |
| mriedem | i assume this zvm driver runs on a linux host, and then calls REST APIs to some zvm hypervisor | 17:16 |
| dansmith | yeah | 17:16 |
| mriedem | like powervm | 17:16 |
| dansmith | linux for sure, but might even be on x86 | 17:16 |
| mriedem | yeah totes | 17:16 |
| *** fragatina has joined #openstack-nova | 17:16 | |
| mriedem | doesn't need to be linux on s390x | 17:16 |
| mriedem | that would be dumb | 17:16 |
| dansmith | yeah, the arch thing isn't the concern as much as it's a lib for a hypervisor I don't need | 17:16 |
| mriedem | sure, but the point is, we are all over the board | 17:17 |
| dansmith | brick is kinda the same-ish, although it's not as weird I think | 17:17 |
| dansmith | yes, definitely | 17:17 |
| mriedem | os-xenapi is also in requirements.txt | 17:17 |
| mriedem | taskflow is also only used by powervm but in requiments.txt | 17:17 |
| *** jpena is now known as jpena|off | 17:17 | |
| dansmith | I think that for people who get government audits for every line of installed code, it'd be a harder sell than os-brick being there but not used | 17:17 |
| dansmith | but just a guess | 17:17 |
| mriedem | you know what i miss? COOs | 17:18 |
| dansmith | I'd be happy using this as an opportunity to get right with the loahd on here | 17:18 |
| dansmith | mriedem: really? that's funny, I don't miss them at all | 17:18 |
| mriedem | i was being sarcastic | 17:18 |
| dansmith | WAT | 17:18 |
| dansmith | I had no idea | 17:18 |
| mriedem | but your audit comment got me reminiscing | 17:18 |
| dansmith | I know :) | 17:18 |
| mriedem | and now this https://www.youtube.com/watch?v=CZ_3G4xqSDQ | 17:18 |
| mriedem | looking at those guys reminds me i need to schedule a haircut | 17:19 |
| *** hamzy_ has joined #openstack-nova | 17:20 | |
| mriedem | alright wtf was i doing now | 17:21 |
| *** hamzy has quit IRC | 17:22 | |
| mriedem | efried: i have replied for great posterity | 17:24 |
| efried | mriedem: thanks | 17:24 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Wait for network-vif-plugged before starting live migration https://review.openstack.org/558001 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add "bind_ports_to_host" neutron API method https://review.openstack.org/523604 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add "delete_port_binding" network API method https://review.openstack.org/552170 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add "activate_port_binding" neutron API method https://review.openstack.org/555947 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete port bindings in setup_networks_on_host if teardown=True https://review.openstack.org/556333 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement migrate_instance_start method for neutron https://review.openstack.org/556334 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add VIFMigrateData object for live migration https://review.openstack.org/515423 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: compute: use port binding extended API during live migration https://review.openstack.org/551371 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Port binding based on events during live migration https://review.openstack.org/434870 | 17:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537 | 17:25 |
| *** moshele has quit IRC | 17:27 | |
| mriedem | arvindn05: if you're just asking questions in a patch ( https://review.openstack.org/#/c/546357/ ) you don't need to -1 | 17:27 |
| *** suresh12 has joined #openstack-nova | 17:27 | |
| *** suresh12 has quit IRC | 17:28 | |
| *** andreas_s has joined #openstack-nova | 17:29 | |
| *** itlinux has quit IRC | 17:30 | |
| *** corvus is now known as jeblair | 17:31 | |
| *** ssurana has quit IRC | 17:31 | |
| *** jeblair is now known as corvus | 17:31 | |
| *** ssurana has joined #openstack-nova | 17:31 | |
| *** andreas_s has quit IRC | 17:34 | |
| *** ssurana has quit IRC | 17:35 | |
| *** Nel1x has quit IRC | 17:35 | |
| *** Nel1x has joined #openstack-nova | 17:39 | |
| *** wolverineav has joined #openstack-nova | 17:44 | |
| *** lpetrut has quit IRC | 17:44 | |
| *** amoralej is now known as amoralej|off | 17:45 | |
| arvindn05_ | mriedem: thanks...will keep in mind. | 17:47 |
| *** sambetts is now known as sambetts|afk | 17:48 | |
| mriedem | arvindn05_: i put some comments in your spec amendment, thanks for starting that https://review.openstack.org/#/c/560718/1 | 17:48 |
| mriedem | jaypipes: dansmith: efried: ^ that's going to require some placement love | 17:49 |
| dansmith | ugh | 17:49 |
| mriedem | f yeah, rebuild | 17:49 |
| *** eharney_ has joined #openstack-nova | 17:51 | |
| *** eharney has quit IRC | 17:51 | |
| *** moshele has joined #openstack-nova | 17:51 | |
| *** eharney_ is now known as eharney | 17:51 | |
| mriedem | if we can request resources=VCPU:0 and in_tree=<compute node uuid> then i think we'd be ok | 17:52 |
| efried | mriedem: I think jaypipes wanted to deal with that by using the GET /resource_providers API instead of GET /a_c. (I disagreed; I think ?in_tree in GET /a_c is a fine idea) | 17:53 |
| *** amodi has quit IRC | 17:53 | |
| dansmith | but asking for zero resources seems weird | 17:53 |
| mriedem | actually if we added in_tree to GET /allocation_candidates, just make resources optional in the same microversion | 17:53 |
| *** itlinux has joined #openstack-nova | 17:53 | |
| mriedem | so GET /allocation_candidates?in_tree=1234&required=CUSTOM_NEW_IMAGE_TRAIT | 17:54 |
| *** tesseract has quit IRC | 17:55 | |
| dansmith | that also seems a little weird to me | 17:55 |
| mriedem | because we aren't going to actually allocate anything? | 17:55 |
| *** dougshelley66 has quit IRC | 17:56 | |
| dansmith | well, not that we aren't going to allocate anything really | 17:56 |
| dansmith | just that we're asking a weird question there | 17:56 |
| mriedem | ok, well, some options | 17:56 |
| mriedem | this doesn't seem like an impossibly hard issue though | 17:57 |
| *** jackie-truong has quit IRC | 17:57 | |
| dansmith | not to validate it no, just weird to do it that way is all | 18:00 |
| *** suresh12 has joined #openstack-nova | 18:00 | |
| *** suresh12 has quit IRC | 18:01 | |
| *** suresh12_ has joined #openstack-nova | 18:01 | |
| *** gjayavelu has joined #openstack-nova | 18:01 | |
| *** yamamoto has joined #openstack-nova | 18:04 | |
| *** dklyle has joined #openstack-nova | 18:05 | |
| *** wolverineav has quit IRC | 18:06 | |
| *** wolverin_ has joined #openstack-nova | 18:06 | |
| *** pcaruana has quit IRC | 18:09 | |
| *** wolverineav has joined #openstack-nova | 18:10 | |
| *** wolverin_ has quit IRC | 18:10 | |
| *** sidx64 has joined #openstack-nova | 18:10 | |
| dansmith | mriedem: why can't we just GET /rp/$uuid and look at the traits? | 18:13 |
| *** sidx64_ has joined #openstack-nova | 18:13 | |
| *** sidx64 has quit IRC | 18:15 | |
| *** ssurana has joined #openstack-nova | 18:15 | |
| *** felipemonteiro has joined #openstack-nova | 18:17 | |
| *** sree has quit IRC | 18:18 | |
| *** felipemonteiro_ has joined #openstack-nova | 18:18 | |
| *** sree has joined #openstack-nova | 18:18 | |
| *** openstackgerrit has quit IRC | 18:19 | |
| *** itlinux has quit IRC | 18:20 | |
| *** mvk has quit IRC | 18:20 | |
| *** breton_ is now known as breton | 18:20 | |
| *** felipemonteiro has quit IRC | 18:22 | |
| *** itlinux has joined #openstack-nova | 18:23 | |
| mriedem | we could | 18:23 |
| *** moshele has quit IRC | 18:23 | |
| *** damien_r has joined #openstack-nova | 18:23 | |
| mriedem | in fact, | 18:23 |
| arvindn05_ | mriedem: that was the suggestion in the bp | 18:23 |
| arvindn05_ | :) | 18:23 |
| mriedem | GET /resource_providers?in_tree=<node uuid>&required=<traits> | 18:23 |
| *** huanxie has quit IRC | 18:23 | |
| mriedem | arvindn05_: if you mean, "For the above issue, the scheduler can request traits of current host and try | 18:24 |
| mriedem | to match those traits with the traits specifid in the image." | 18:24 |
| mriedem | that wasn't clear to me as a proposed solution at all | 18:24 |
| mriedem | i might have gotten hung up on the 'current host' thing, thinking about RPs | 18:24 |
| mriedem | arvindn05_: if you can clarify that specifically talking about https://developer.openstack.org/api-ref/placement/#list-resource-provider-traits then it's probably fine | 18:25 |
| arvindn05_ | sorry if it was unclear...i will add that we will call /resource_providers/{uuid}/traits | 18:26 |
| mriedem | ok that makes more sense for the alternative now | 18:27 |
| arvindn05_ | i am looking at the comments now...will add the above to make it clearer | 18:27 |
| arvindn05_ | i thoguht about the /resource_providers/{uuid}/traits solution after we discussed the issue yesterday to minimize changes | 18:29 |
| arvindn05_ | mriedem: my original proposal to modify the API to accept no resources is now the alternate suggestion | 18:30 |
| mriedem | yeah now it makes more sense to me | 18:31 |
| *** links has joined #openstack-nova | 18:31 | |
| mriedem | just need to clarify | 18:31 |
| mriedem | i don't exactly know where we'd do this validation...in a new filter? | 18:31 |
| *** arvindn05 has quit IRC | 18:31 | |
| mriedem | or maybe in the ImagePropertiesFilter if we know we're doing a rebuild? | 18:32 |
| *** sree has quit IRC | 18:32 | |
| dansmith | or just in rebuild code in general | 18:35 |
| dansmith | or, I guess we need to know if they had imagepropfilter enabled in that case? | 18:36 |
| dansmith | bleh | 18:36 |
| *** felipemonteiro_ has quit IRC | 18:37 | |
| *** openstackgerrit has joined #openstack-nova | 18:37 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300 | 18:37 |
| *** lpetrut has joined #openstack-nova | 18:37 | |
| *** amodi has joined #openstack-nova | 18:40 | |
| *** damien_r has quit IRC | 18:42 | |
| *** itlinux has quit IRC | 18:42 | |
| *** Nel1x has quit IRC | 18:42 | |
| *** damien_r has joined #openstack-nova | 18:43 | |
| openstackgerrit | Merged openstack/nova master: Make the nova-multiattach job non-voting temporarily https://review.openstack.org/560909 | 18:44 |
| efried | ^ time to queue up the rechecks | 18:45 |
| *** arvindn05 has joined #openstack-nova | 18:46 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 18:47 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | 18:47 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | 18:47 |
| *** mgoddard has quit IRC | 18:47 | |
| arvindn05_ | dansmith: mriedem: my assumption was we'd do the check in select destination before the filters https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L135 | 18:48 |
| arvindn05_ | an else case where in case of rebuild we made an call to /resource_providers/{uuid}/traits to get the traits and match them | 18:49 |
| mriedem | idk, today we only restrict a rebuild with a new image based on the enabled filteres, as dansmith mentions | 18:51 |
| mriedem | so i kind of think we should stick with that behavior | 18:51 |
| mriedem | i think we'd have everything we need within the ImagePropertiesFilter itself to understand if we need to do this check | 18:52 |
| mriedem | it should all be in the reqspec | 18:52 |
| mriedem | and the HostState has the compute node uuid to get the resource provider | 18:52 |
| mriedem | rest api calls from a filter isn't awesome, but if we just do this conditionally when we know we're doing a rebuild, it's the same rest api call, just in a different place | 18:53 |
| dansmith | yeah if you have that filter disabled, no fair rejecting a rebuild based on stuff in there | 18:53 |
| arvindn05_ | not sure i understand...if the image traits have been updated, shouldnt the rebuild take that into account? | 18:54 |
| *** r-daneel has joined #openstack-nova | 18:54 | |
| *** sree has joined #openstack-nova | 18:54 | |
| arvindn05_ | regardless of whether the imagefilter properties filter is enabled | 18:54 |
| arvindn05_ | if the image changed completely with new traits, and image filter was never enabled, then the traits will be ignored | 18:55 |
| mriedem | if you don't care about scheduling based on image properties, then why should we require it? | 18:55 |
| mriedem | if you do care about scheduling based on image properties, then the filter will be enabled | 18:56 |
| *** Nel1x has joined #openstack-nova | 18:56 | |
| mriedem | it's also a default filter, which i don't think anyone disables, so it's probably fine | 18:56 |
| arvindn05_ | the image properties and traits are parallel concepts though | 18:57 |
| mriedem | umm | 18:58 |
| mriedem | we get the traits via the image properties | 18:58 |
| arvindn05_ | i dont need to enable imageproperties filter to use the traits filtering mechanissm | 18:58 |
| mriedem | i guess you could argue that we do traits-based filtering in placement via flavor extra specs regardless of any flavor extra spec specific filters being enabled | 18:58 |
| arvindn05_ | if you completely remove image prop filter...the traits filtering still works since it goes through placement | 18:58 |
| arvindn05_ | yup | 18:59 |
| *** sree has quit IRC | 18:59 | |
| mriedem | idk, i don't want to have to make this decision | 18:59 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: RP list: member_of and resources parameters (v1.3, v1.4) https://review.openstack.org/511183 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: RP delete inventories (v1.5) https://review.openstack.org/514642 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: New dict format of allocations (v1.11, v1.12) https://review.openstack.org/542819 | 19:00 |
| *** sree has joined #openstack-nova | 19:00 | |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Transactionally update allocations (v1.13) https://review.openstack.org/546674 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | 19:00 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Allocation candidates parameter: required (v1.17) https://review.openstack.org/548326 | 19:00 |
| mriedem | arvindn05_: i'm sure bauzas would love to think about this | 19:00 |
| arvindn05_ | will wait for their comments....will update the spec to make it clear based on your comments for now...thanks mriedem | 19:01 |
| *** amodi has quit IRC | 19:02 | |
| *** eharney has quit IRC | 19:03 | |
| *** salv-orlando has quit IRC | 19:04 | |
| arvindn05_ | mriedem: btw are you convinced with the argument that image properties filter and using traits on images are independent of each other? | 19:04 |
| *** salv-orlando has joined #openstack-nova | 19:04 | |
| *** sree has quit IRC | 19:05 | |
| *** AlexeyAbashkin has quit IRC | 19:06 | |
| *** damien_r has quit IRC | 19:07 | |
| mriedem | arvindn05_: no | 19:08 |
| mriedem | but, | 19:08 |
| mriedem | i also don't feel strongly about it | 19:08 |
| arvindn05_ | that's half the battle won :) | 19:08 |
| mriedem | ha, now you're getting it | 19:09 |
| *** salv-orlando has quit IRC | 19:09 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip placement on rebuild in same host https://review.openstack.org/546357 | 19:09 |
| arvindn05_ | but ignoring the rebuild issue, like i mentioned if tommorow someone removed the image properties filter, based on the original BP, there would be no regression of the feature | 19:09 |
| mriedem | hongbin: when you get a minute, can you propose backports for https://review.openstack.org/#/c/546357/ to queens and pike? | 19:10 |
| hongbin | mriedem: sure | 19:10 |
| mriedem | thanks | 19:10 |
| mriedem | you'll have to do it from command line | 19:10 |
| hongbin | i see, sure, no problem | 19:11 |
| openstackgerrit | Merged openstack/nova master: uncap eventlet in nova https://review.openstack.org/560420 | 19:12 |
| openstackgerrit | Merged openstack/nova master: xenapi: Support live migration in pooled multi-nodes environment https://review.openstack.org/489451 | 19:12 |
| *** felipemonteiro has joined #openstack-nova | 19:13 | |
| *** felipemonteiro_ has joined #openstack-nova | 19:14 | |
| openstackgerrit | Merged openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (2) https://review.openstack.org/558993 | 19:15 |
| *** sree has joined #openstack-nova | 19:15 | |
| openstackgerrit | Merged openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (3) https://review.openstack.org/559258 | 19:15 |
| openstackgerrit | Hongbin Lu proposed openstack/nova stable/queens: Skip placement on rebuild in same host https://review.openstack.org/561014 | 19:16 |
| *** felipemonteiro has quit IRC | 19:17 | |
| *** sree has quit IRC | 19:20 | |
| *** gouthamr has quit IRC | 19:20 | |
| *** sree has joined #openstack-nova | 19:21 | |
| *** amodi has joined #openstack-nova | 19:21 | |
| openstackgerrit | Hongbin Lu proposed openstack/nova stable/pike: Skip placement on rebuild in same host https://review.openstack.org/561015 | 19:22 |
| *** sree has quit IRC | 19:25 | |
| openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Handle rebuild of instance with new image https://review.openstack.org/560718 | 19:26 |
| *** sree has joined #openstack-nova | 19:26 | |
| *** sdeath has joined #openstack-nova | 19:30 | |
| *** gouthamr has joined #openstack-nova | 19:30 | |
| *** andreas_s has joined #openstack-nova | 19:30 | |
| *** sree has quit IRC | 19:31 | |
| *** amodi has quit IRC | 19:32 | |
| *** wolverin_ has joined #openstack-nova | 19:34 | |
| *** wolverineav has quit IRC | 19:34 | |
| openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Handle rebuild of instance with new image https://review.openstack.org/560718 | 19:35 |
| *** andreas_s has quit IRC | 19:35 | |
| *** pchavva1 has joined #openstack-nova | 19:36 | |
| *** sree has joined #openstack-nova | 19:38 | |
| cfriesen | melwitt: for the detach_volume() change, are you suggesting a try/except block around the call to encryptor.detach_volume()? | 19:43 |
| *** sree has quit IRC | 19:43 | |
| melwitt | cfriesen: yes. because if you get there on a second attempt and you already ran it in the past, os-brick will raise "unknown device" because it can't find the attached device | 19:43 |
| *** sridharg has quit IRC | 19:43 | |
| cfriesen | makes sense, will do | 19:44 |
| melwitt | we backported the os-brick change back to pike but we can't bump requirements.txt on stable | 19:44 |
| melwitt | so it's not guaranteed that someone running stable/pike will have the os-brick version that will ignore exit code 4 (unknown) for you | 19:45 |
| *** links has quit IRC | 19:46 | |
| *** amodi has joined #openstack-nova | 19:47 | |
| melwitt | s/you/them/ | 19:48 |
| *** linkmark has quit IRC | 19:49 | |
| *** gouthamr has quit IRC | 19:50 | |
| *** gouthamr has joined #openstack-nova | 19:51 | |
| *** pchavva1 has quit IRC | 19:51 | |
| *** pchavva has quit IRC | 19:51 | |
| *** pchavva has joined #openstack-nova | 19:51 | |
| *** moshele has joined #openstack-nova | 19:54 | |
| *** jackie-truong has joined #openstack-nova | 19:54 | |
| *** suresh12_ has quit IRC | 19:55 | |
| *** suresh12 has joined #openstack-nova | 19:56 | |
| *** mvk has joined #openstack-nova | 19:56 | |
| *** suresh12 has quit IRC | 19:57 | |
| *** suresh12 has joined #openstack-nova | 19:57 | |
| *** moshele has quit IRC | 19:58 | |
| *** artom has quit IRC | 19:59 | |
| mriedem | i found another cells upcall | 20:00 |
| mriedem | https://github.com/openstack/nova/blob/c531b7905f5a9f8a5bfa355a2047d04032cbf847/nova/virt/xenapi/host.py#L86 | 20:01 |
| mriedem | however, it's for that completely broken and deprecated anyway host maintenance / power action stuff | 20:01 |
| *** damien_r has joined #openstack-nova | 20:02 | |
| *** felipemonteiro_ has quit IRC | 20:03 | |
| *** felipemonteiro_ has joined #openstack-nova | 20:03 | |
| *** oanson has quit IRC | 20:05 | |
| *** salv-orlando has joined #openstack-nova | 20:05 | |
| *** damien_r has quit IRC | 20:06 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | 20:08 |
| cfriesen | melwitt: I don't have the actual "target" to report in the error message. do you know how to get it, or can I just use "disk_dev" or something? | 20:09 |
| *** salv-orlando has quit IRC | 20:09 | |
| *** salv-orlando has joined #openstack-nova | 20:09 | |
| melwitt | cfriesen: yeah, disk_dev is the equivalent | 20:10 |
| melwitt | this version of the patch was almost right, it just also needed to reraise if not exit code 4 https://review.openstack.org/#/c/515008/3/nova/virt/libvirt/driver.py | 20:11 |
| cfriesen | ah, okay | 20:11 |
| melwitt | because we still want to fail if a legit detach error happened. but we'll want to ignore "unknown" only | 20:11 |
| *** oanson has joined #openstack-nova | 20:12 | |
| *** sree has joined #openstack-nova | 20:15 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: test live_migration_wait_for_vif_plug=True https://review.openstack.org/558006 | 20:17 |
| *** itlinux has joined #openstack-nova | 20:17 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: test live_migration_wait_for_vif_plug=True https://review.openstack.org/558006 | 20:17 |
| *** sree has quit IRC | 20:19 | |
| *** yamamoto has quit IRC | 20:21 | |
| *** yamamoto has joined #openstack-nova | 20:22 | |
| *** markvoelker_ has quit IRC | 20:23 | |
| *** elmaciej has quit IRC | 20:23 | |
| *** archit has joined #openstack-nova | 20:33 | |
| *** sidx64_ has quit IRC | 20:33 | |
| *** amodi has quit IRC | 20:33 | |
| *** archit is now known as amodi | 20:33 | |
| *** wolverin_ has quit IRC | 20:34 | |
| *** wolverineav has joined #openstack-nova | 20:34 | |
| *** tbachman has quit IRC | 20:38 | |
| *** sree has joined #openstack-nova | 20:38 | |
| openstackgerrit | Chris Friesen proposed openstack/nova stable/pike: libvirt: disconnect volume from host during detach https://review.openstack.org/560690 | 20:39 |
| *** wolverineav has quit IRC | 20:39 | |
| *** elmaciej has joined #openstack-nova | 20:42 | |
| *** pchavva has quit IRC | 20:42 | |
| mriedem | melwitt: i'll be skipping the meeting today for fun times, left some links to gate bugs in the meeting agenda this morning, but the multiattach one is already semi resolved | 20:42 |
| *** sree has quit IRC | 20:43 | |
| *** mriedem is now known as mriedem_funtime | 20:43 | |
| melwitt | mriedem: we shall miss you. okay, I'll take a look at those bugs | 20:43 |
| *** tbachman has joined #openstack-nova | 20:46 | |
| *** wolverineav has joined #openstack-nova | 20:46 | |
| *** tbachman has quit IRC | 20:47 | |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | 20:47 |
| *** salv-orlando has quit IRC | 20:47 | |
| *** wolverineav has quit IRC | 20:47 | |
| *** salv-orlando has joined #openstack-nova | 20:48 | |
| *** wolverineav has joined #openstack-nova | 20:48 | |
| *** takashin has joined #openstack-nova | 20:49 | |
| *** hamzy_ has quit IRC | 20:49 | |
| melwitt | nova meeting in 10 minutes | 20:50 |
| *** eharney has joined #openstack-nova | 21:00 | |
| *** itlinux has quit IRC | 21:01 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300 | 21:02 |
| *** gouthamr has quit IRC | 21:06 | |
| *** slaweq has quit IRC | 21:07 | |
| *** slaweq has joined #openstack-nova | 21:08 | |
| *** evin has quit IRC | 21:11 | |
| *** slaweq has quit IRC | 21:13 | |
| *** Nel1x has quit IRC | 21:14 | |
| *** gouthamr has joined #openstack-nova | 21:15 | |
| *** Nel1x has joined #openstack-nova | 21:15 | |
| *** Nel1x has quit IRC | 21:16 | |
| *** Nel1x has joined #openstack-nova | 21:16 | |
| *** dpawlik has joined #openstack-nova | 21:25 | |
| *** openstackstatus has quit IRC | 21:27 | |
| *** openstack has joined #openstack-nova | 21:29 | |
| *** ChanServ sets mode: +o openstack | 21:29 | |
| *** dpawlik has quit IRC | 21:30 | |
| *** yangyape_ has joined #openstack-nova | 21:31 | |
| *** yangyapeng has quit IRC | 21:31 | |
| *** sree has joined #openstack-nova | 21:32 | |
| *** lpetrut has quit IRC | 21:35 | |
| *** sambetts|afk has quit IRC | 21:36 | |
| *** sree has quit IRC | 21:36 | |
| *** sambetts_ has joined #openstack-nova | 21:38 | |
| *** sree has joined #openstack-nova | 21:38 | |
| *** Nel1x has quit IRC | 21:39 | |
| *** Nel1x has joined #openstack-nova | 21:39 | |
| *** burt has quit IRC | 21:42 | |
| *** sree has quit IRC | 21:43 | |
| *** jackie-truong has quit IRC | 21:46 | |
| *** tbachman has joined #openstack-nova | 21:51 | |
| cfriesen | melwitt: I've got a theory for https://bugs.launchpad.net/nova/+bug/1763181 but I guess mriedem and bauzas are away | 21:52 |
| openstack | Launchpad bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,Confirmed] | 21:52 |
| *** gouthamr has quit IRC | 21:54 | |
| *** tbachman has quit IRC | 21:55 | |
| *** sree has joined #openstack-nova | 21:55 | |
| *** esberglu has quit IRC | 21:58 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242 | 21:59 |
| openstackgerrit | Michael Still proposed openstack/nova master: Sync xenapi and libvirt on what flags to pass e2fsck. https://review.openstack.org/554078 | 21:59 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi partition copies to privsep. https://review.openstack.org/553605 | 21:59 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move image conversion to privsep. https://review.openstack.org/554437 | 21:59 |
| openstackgerrit | Michael Still proposed openstack/nova master: We don't need utils.trycmd any more. https://review.openstack.org/554439 | 21:59 |
| openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | 21:59 |
| *** sree has quit IRC | 22:00 | |
| *** tbachman has joined #openstack-nova | 22:03 | |
| *** eharney has quit IRC | 22:03 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: add lower-constraints job https://review.openstack.org/555961 | 22:05 |
| openstackgerrit | Eric Fried proposed openstack/nova master: add lower-constraints job https://review.openstack.org/555961 | 22:05 |
| melwitt | cfriesen: noyce, thanks for the comment on the bug | 22:07 |
| *** felipemonteiro__ has joined #openstack-nova | 22:08 | |
| *** felipemonteiro_ has quit IRC | 22:12 | |
| cfriesen | melwitt: there's another variant that can affect migrations, where we'd want to look at the migration records and include any hosts that are the destination or source of a migration for instances in the group. | 22:13 |
| cfriesen | the joys of race conditions | 22:14 |
| melwitt | guh | 22:15 |
| *** felipemonteiro__ has quit IRC | 22:15 | |
| *** felipemonteiro__ has joined #openstack-nova | 22:16 | |
| *** lbragstad has quit IRC | 22:16 | |
| *** salv-orlando has quit IRC | 22:18 | |
| *** liverpooler has quit IRC | 22:18 | |
| *** salv-orlando has joined #openstack-nova | 22:19 | |
| *** salv-orlando has quit IRC | 22:23 | |
| *** harlowja has joined #openstack-nova | 22:24 | |
| *** rcernin has joined #openstack-nova | 22:30 | |
| *** arvindn05_ has quit IRC | 22:32 | |
| *** arvindn0_ has joined #openstack-nova | 22:33 | |
| *** Sukhdev has joined #openstack-nova | 22:35 | |
| *** linkmark has joined #openstack-nova | 22:39 | |
| *** wolverineav has quit IRC | 22:44 | |
| *** wolverineav has joined #openstack-nova | 22:44 | |
| *** sree has joined #openstack-nova | 22:50 | |
| *** yikun has quit IRC | 22:51 | |
| *** yikun has joined #openstack-nova | 22:52 | |
| *** tbachman has quit IRC | 22:52 | |
| *** sree has quit IRC | 22:54 | |
| *** suresh12 has quit IRC | 22:55 | |
| *** hongbin has quit IRC | 22:58 | |
| *** andreas_s has joined #openstack-nova | 23:03 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_serversV21.py (1) https://review.openstack.org/560821 | 23:05 |
| *** _ix has quit IRC | 23:06 | |
| *** andreas_s has quit IRC | 23:09 | |
| *** swamireddy has quit IRC | 23:14 | |
| *** Guest30936 has quit IRC | 23:14 | |
| mriedem_funtime | new unit test race fail http://logs.openstack.org/78/554378/3/check/openstack-tox-py27/54b38c0/testr_results.html.gz | 23:20 |
| *** mriedem_funtime is now known as mriedem | 23:20 | |
| *** sdague has quit IRC | 23:21 | |
| *** gouthamr has joined #openstack-nova | 23:21 | |
| *** cdent has quit IRC | 23:21 | |
| mriedem | https://bugs.launchpad.net/nova/+bug/1763535 | 23:22 |
| openstack | Launchpad bug 1763535 in OpenStack Compute (nova) "nova.tests.unit.virt.xenapi.test_vm_utils.StreamDiskTestCase.test_non_ami intermittently fails" [High,Confirmed] | 23:22 |
| *** sdeath has quit IRC | 23:22 | |
| mriedem | takashin: ^ you should probably look at that | 23:28 |
| *** Nel1x has quit IRC | 23:30 | |
| *** itlinux has joined #openstack-nova | 23:35 | |
| mriedem | melwitt: https://review.openstack.org/#/q/I78dead482e2ed8262745cb08c9f4ab71035adb33 are what fix the multiattach job | 23:36 |
| mriedem | https://review.openstack.org/#/c/554317/ on top has to depend on the queens backport, | 23:36 |
| mriedem | otherwise zuul pulls the queens job def which has the bad branch specifier in it and borks everything up | 23:37 |
| mriedem | so once those patches to remove the in-tree branch specifier are merged, we can make the multiattach job voting and gating again | 23:37 |
| *** mriedem is now known as mriedem_afk | 23:38 | |
| *** felipemonteiro_ has joined #openstack-nova | 23:39 | |
| -openstackstatus- NOTICE: The Etherpad service at https://etherpad.openstack.org/ is being restarted to pick up the latest release version; browsers should see only a brief ~1min blip before reconnecting automatically to active pads | 23:40 | |
| *** elmaciej has quit IRC | 23:40 | |
| *** felipemonteiro__ has quit IRC | 23:43 | |
| *** mlavalle has quit IRC | 23:45 | |
| *** amodi has quit IRC | 23:47 | |
| *** arvindn0_ has quit IRC | 23:51 | |
| openstackgerrit | Merged openstack/nova master: Remove :return from update_provider_tree docstring https://review.openstack.org/560442 | 23:51 |
| *** claudiub|2 has quit IRC | 23:53 | |
| *** didelspk has joined #openstack-nova | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!