| *** takashin has joined #openstack-nova | 00:00 | |
| *** mlavalle has quit IRC | 00:04 | |
| *** macza has quit IRC | 00:04 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle missing marker during online data migration https://review.openstack.org/605164 | 00:04 |
|---|---|---|
| melwitt | takashin: hi, could you please check the reply to your -1 on this change? https://review.openstack.org/582147 | 00:05 |
| *** mvkr has quit IRC | 00:05 | |
| *** mriedem has quit IRC | 00:05 | |
| takashin | melwitt: I will check it. | 00:05 |
| melwitt | thank you | 00:05 |
| takashin | melwitt: I would like to know other cores' opinions about https://review.openstack.org/#/c/582147/ . | 00:07 |
| melwitt | takashin: ack. will try to look later | 00:07 |
| *** slaweq has joined #openstack-nova | 00:11 | |
| openstackgerrit | Merged openstack/nova stable/pike: unquiesce instance after quiesce failure https://review.openstack.org/581454 | 00:14 |
| *** slaweq has quit IRC | 00:15 | |
| *** moshele has quit IRC | 00:19 | |
| *** tetsuro has joined #openstack-nova | 00:21 | |
| *** dpawlik has joined #openstack-nova | 00:45 | |
| *** dpawlik has quit IRC | 00:50 | |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | 00:53 |
| *** hoonetorg has quit IRC | 00:56 | |
| *** dpawlik has joined #openstack-nova | 01:01 | |
| *** mhen has quit IRC | 01:05 | |
| *** dpawlik has quit IRC | 01:05 | |
| *** mhen has joined #openstack-nova | 01:07 | |
| *** hoonetorg has joined #openstack-nova | 01:10 | |
| *** efried has joined #openstack-nova | 01:15 | |
| *** mrsoul has quit IRC | 01:22 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:23 | |
| *** hongbin has joined #openstack-nova | 01:23 | |
| *** med_ has joined #openstack-nova | 01:27 | |
| *** owalsh_ has joined #openstack-nova | 01:29 | |
| *** brinzhang has joined #openstack-nova | 01:32 | |
| *** owalsh has quit IRC | 01:33 | |
| *** threestrands has quit IRC | 01:33 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add compute version 36 to support ``volume_type`` https://review.openstack.org/579360 | 01:34 |
| *** tiendc has joined #openstack-nova | 01:39 | |
| *** Dinesh_Bhor has quit IRC | 01:45 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:48 | |
| *** hoonetorg has quit IRC | 01:49 | |
| *** Dinesh_Bhor has quit IRC | 01:53 | |
| *** hoonetorg has joined #openstack-nova | 02:01 | |
| *** bhagyashris has joined #openstack-nova | 02:05 | |
| *** efried has quit IRC | 02:10 | |
| *** efried1 has joined #openstack-nova | 02:10 | |
| *** efried1 is now known as efried | 02:12 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088 | 02:29 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088 | 02:32 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088 | 02:33 |
| openstackgerrit | Merged openstack/nova stable/pike: Update RequestSpec.flavor on resize_revert https://review.openstack.org/605879 | 02:51 |
| *** Dinesh_Bhor has joined #openstack-nova | 03:00 | |
| *** med_ has quit IRC | 03:01 | |
| *** dpawlik has joined #openstack-nova | 03:02 | |
| *** dpawlik has quit IRC | 03:06 | |
| *** dave-mccowan has quit IRC | 03:07 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: [WIP]Remove extensions loading framework from wsgi.py https://review.openstack.org/607092 | 03:12 |
| *** hongbin has quit IRC | 03:15 | |
| *** cfriesen has quit IRC | 03:15 | |
| *** dpawlik has joined #openstack-nova | 03:17 | |
| *** med_ has joined #openstack-nova | 03:21 | |
| openstackgerrit | Merged openstack/nova stable/rocky: nova-manage - fix online_data_migrations counts https://review.openstack.org/605828 | 03:21 |
| openstackgerrit | Merged openstack/nova stable/pike: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604449 | 03:22 |
| *** dpawlik has quit IRC | 03:22 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge image_size extension response into image view builder https://review.openstack.org/606845 | 03:24 |
| *** mvkr has joined #openstack-nova | 03:29 | |
| *** med_ has quit IRC | 03:38 | |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | 03:49 |
| *** Dinesh_Bhor has quit IRC | 03:54 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add compute API version for when a ``volume_type`` is requested https://review.openstack.org/605573 | 04:08 |
| openstackgerrit | Merged openstack/nova stable/pike: Use ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa in tree https://review.openstack.org/581444 | 04:24 |
| *** hshiina has joined #openstack-nova | 04:26 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:34 | |
| *** pcaruana has joined #openstack-nova | 04:36 | |
| *** pcaruana has quit IRC | 04:43 | |
| *** dpawlik has joined #openstack-nova | 05:18 | |
| *** dpawlik has quit IRC | 05:23 | |
| *** dpawlik has joined #openstack-nova | 05:34 | |
| *** dpawlik has quit IRC | 05:39 | |
| *** Dinesh_Bhor has quit IRC | 05:48 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:54 | |
| *** awaugama has quit IRC | 05:56 | |
| *** Dinesh_Bhor has quit IRC | 05:58 | |
| *** dpawlik has joined #openstack-nova | 06:00 | |
| *** Luzi has joined #openstack-nova | 06:02 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:03 | |
| *** sapd1 has joined #openstack-nova | 06:08 | |
| *** slaweq has joined #openstack-nova | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:21 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260 | 06:22 |
| *** hoonetorg has quit IRC | 06:36 | |
| *** dims has quit IRC | 06:38 | |
| *** dims has joined #openstack-nova | 06:44 | |
| *** dims has quit IRC | 06:48 | |
| *** dims has joined #openstack-nova | 06:51 | |
| *** tetsuro has quit IRC | 06:53 | |
| *** Dinesh_Bhor has quit IRC | 06:55 | |
| *** rcernin has quit IRC | 07:01 | |
| *** pcaruana has joined #openstack-nova | 07:01 | |
| *** oanson has joined #openstack-nova | 07:01 | |
| *** awaugama has joined #openstack-nova | 07:01 | |
| *** moshele has joined #openstack-nova | 07:01 | |
| *** dpawlik has quit IRC | 07:04 | |
| *** moshele has quit IRC | 07:04 | |
| *** slaweq has joined #openstack-nova | 07:07 | |
| openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Live migration of instances https://review.openstack.org/270116 | 07:11 |
| *** helenafm has joined #openstack-nova | 07:11 | |
| *** aloga has quit IRC | 07:15 | |
| *** aloga has joined #openstack-nova | 07:15 | |
| *** moshele has joined #openstack-nova | 07:23 | |
| *** moshele has left #openstack-nova | 07:23 | |
| *** dpawlik has joined #openstack-nova | 07:24 | |
| *** Dinesh_Bhor has joined #openstack-nova | 07:31 | |
| *** sahid has joined #openstack-nova | 07:33 | |
| *** psachin has joined #openstack-nova | 07:35 | |
| *** tetsuro has joined #openstack-nova | 07:38 | |
| *** slaweq has quit IRC | 07:39 | |
| *** ralonsoh has joined #openstack-nova | 07:40 | |
| *** slaweq has joined #openstack-nova | 07:49 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** priteau has joined #openstack-nova | 07:51 | |
| bauzas | good morning nova | 07:54 |
| * kashyap waves | 07:58 | |
| gryf | morning guys | 08:05 |
| *** owalsh_ is now known as owalsh | 08:06 | |
| bhagyashris | Hi all, I am facing one issue on current master while live migration of the instance , I am trying to live migrate the instance but it's not migrating that instance on given destination node and that instance remains on source node with the status as active, checked n-cpu, n-condunctor logs but didn't get any error at first time. | 08:11 |
| *** alexchadin has joined #openstack-nova | 08:11 | |
| bhagyashris | And if I again try to live migrate that instance then it's giveing me Migration pre-check error: Binding failed for port <port-id> please check neutron logs for more information. (HTTP 400 | 08:12 |
| bhagyashris | Can anyone please help me | 08:12 |
| *** hshiina has quit IRC | 08:19 | |
| *** Luzi has quit IRC | 08:28 | |
| *** derekh has joined #openstack-nova | 08:37 | |
| takashin | Nova cores, could you review https://review.openstack.org/#/c/606947/ ? It is a fix for a gate failure in python-novaclient. | 08:39 |
| *** Luzi has joined #openstack-nova | 08:40 | |
| *** alexchadin has quit IRC | 08:45 | |
| gibi | bhagyashris: what do you see in the neutron log about the bind failure? | 08:48 |
| *** ttsiouts has joined #openstack-nova | 08:51 | |
| gibi | bhagyashris: have you specified target host for the migration ? Have you use force=True in the migration request? | 08:52 |
| bhagyashris | gibi: this is log I get at neutron server side http://paste.openstack.org/show/731235/ | 08:54 |
| *** mhen has quit IRC | 08:55 | |
| bhagyashris | gibi: yes I have specified target host in request but not force=True | 08:55 |
| *** cdent has joined #openstack-nova | 08:56 | |
| *** mhen has joined #openstack-nova | 08:56 | |
| *** Dinesh_Bhor has quit IRC | 08:57 | |
| openstackgerrit | Merged openstack/nova master: Fix neutron-tempest-linuxbridge irrelevant-files https://review.openstack.org/606616 | 08:57 |
| openstackgerrit | Merged openstack/nova master: Add hide server address tests in test_serversV21.py https://review.openstack.org/605435 | 08:57 |
| openstackgerrit | Merged openstack/nova stable/ocata: [Stable Only] Remove soft-deleted instances from quota_usages https://review.openstack.org/570653 | 08:57 |
| openstackgerrit | Merged openstack/nova master: Follow up for Ib6f95c22ffd3ea235b60db4da32094d49c2efa2a https://review.openstack.org/604743 | 08:57 |
| gibi | bhagyashris: if force is False then nova checks the host with the scheduler. did you looked at the scheduler log for the first migration attempt? | 08:57 |
| openstackgerrit | Merged openstack/nova master: Raise error on timeout in wait_for_versioned_notifications https://review.openstack.org/604859 | 08:57 |
| *** alexchadin has joined #openstack-nova | 08:59 | |
| *** ttsiouts has quit IRC | 09:00 | |
| bhagyashris | gibi: checked the n-sch log but there I didn't get any logs | 09:01 |
| gibi | bhagyashris: it is soo strange that nothing in the nova-conductor log | 09:01 |
| *** takashin has left #openstack-nova | 09:02 | |
| bhagyashris | gibi: n-cond or n-sch? | 09:02 |
| gibi | nova-cond | 09:02 |
| *** ttsiouts has joined #openstack-nova | 09:02 | |
| bhagyashris | At n-cond I got the logs | 09:02 |
| gibi | bhagyashris: could you post the n-cond log? | 09:02 |
| bhagyashris | gibi: #link: http://paste.openstack.org/show/731237/ | 09:06 |
| gibi | bhagyashris: are you sure that force wasn't set to True in the live migration request? the last log line in http://paste.openstack.org/show/731237/ shows that claim_resources_on_destination was called to make the allocation for the instance on the dest host but that function only called for force live migrate and force evacuate | 09:16 |
| bhagyashris | gibi: I am sure I havent set force to true | 09:17 |
| *** slaweq has quit IRC | 09:19 | |
| bhagyashris | gibi: openstack server migrate <instance-id> --live <destination-host> --block-migration this command I used for live migration | 09:19 |
| *** slaweq has joined #openstack-nova | 09:21 | |
| gibi | bhagyashris: as you did not specify any api microverion openstack client used the minimum and in there specifying a host means force by default | 09:26 |
| gibi | bhagyashris: https://developer.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action | 09:26 |
| gibi | bhagyashris: this explains why you don't see anything in the nova-sch log | 09:26 |
| bhagyashris | gibi: ohh ok | 09:30 |
| *** hamdyk has joined #openstack-nova | 09:31 | |
| *** hamzy has quit IRC | 09:32 | |
| *** hamzy has joined #openstack-nova | 09:32 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| gibi | bhagyashris: could you check what is the status of the first live migration attempt with nova migration-list command? | 09:33 |
| bhagyashris | gibi: it's showing me the status as error | 09:34 |
| gibi | bhagyashris: and nothing relevan on the source compute or dest compute n-cpu log? | 09:35 |
| bhagyashris | gibi: at destination compute it' check destination check data is LibvirtLiveMigrateData | 09:38 |
| gibi | bhagyashris: I'm clueless how that migration failed without leaving any trace of information anywhere what was the reason fo the failure. I think when it failed it failed to clean up some port binding stuff in Neutron and that causes the second migration attempt to fail at port binding | 09:43 |
| *** Dinesh_Bhor has joined #openstack-nova | 09:43 | |
| bhagyashris | gibi: yeah... but at very first why it's not migrating that instance on destination node with any failure :( | 09:45 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add microversion 2.67 to support volume_type https://review.openstack.org/606398 | 09:52 |
| *** brinzhang has quit IRC | 09:57 | |
| *** mvkr has quit IRC | 09:57 | |
| *** psachin has quit IRC | 09:59 | |
| *** ttsiouts has quit IRC | 10:01 | |
| *** ttsiouts has joined #openstack-nova | 10:03 | |
| *** bhagyashris has quit IRC | 10:06 | |
| *** dpawlik has quit IRC | 10:16 | |
| *** dpawlik has joined #openstack-nova | 10:17 | |
| openstackgerrit | Sam Morrison proposed openstack/python-novaclient master: Fix up userdata argument to rebuild. https://review.openstack.org/605341 | 10:18 |
| *** mvkr has joined #openstack-nova | 10:26 | |
| *** Dinesh_Bhor has quit IRC | 10:27 | |
| *** Dinesh_Bhor has joined #openstack-nova | 10:29 | |
| *** Dinesh_Bhor has quit IRC | 10:31 | |
| openstackgerrit | Merged openstack/nova stable/pike: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600113 | 10:32 |
| *** hamzy has quit IRC | 10:33 | |
| openstackgerrit | Merged openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | 10:34 |
| *** hamzy has joined #openstack-nova | 10:34 | |
| *** tbachman has quit IRC | 10:42 | |
| sean-k-mooney | gibi: that livemigration error bhagyashris had is bizarr i have never seen neutron trigger FlushError i would have liked to see the n-cpu log on the source and dest node i suspect that there was a libvirt/qemu error that resulted in the first failure but not sure what would cause the second. | 10:42 |
| gibi | sean-k-mooney: bhagyashris stated that there was no relevant log in the nova-cpu services :/ | 10:46 |
| gibi | sean-k-mooney: but I share your feeling about the weiredness of the fault | 10:46 |
| *** cdent has quit IRC | 10:48 | |
| sean-k-mooney | ya i would still be interested in seeing the libvirt qemu instance log to confirm there were no qemu/libvirt errors but it is strange. im wonder if the neutron issue could have something to do with multiple port bindings. e.g. we know 1 or 2 bindings now works but did anyone ever test with 3. with out a way to reproduce however its really just guess work on my part | 10:49 |
| *** sapd1 has quit IRC | 10:49 | |
| *** rpittau has joined #openstack-nova | 10:50 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436 | 10:51 |
| *** dave-mccowan has joined #openstack-nova | 11:02 | |
| *** ttsiouts has quit IRC | 11:03 | |
| *** ttsiouts has joined #openstack-nova | 11:03 | |
| *** ttsiouts has quit IRC | 11:08 | |
| *** ttsiouts has joined #openstack-nova | 11:13 | |
| *** med_ has joined #openstack-nova | 11:13 | |
| *** mvkr has quit IRC | 11:16 | |
| *** jaosorior has joined #openstack-nova | 11:16 | |
| *** mvkr has joined #openstack-nova | 11:17 | |
| *** tetsuro has quit IRC | 11:18 | |
| *** alexchadin has quit IRC | 11:21 | |
| *** cdent has joined #openstack-nova | 11:23 | |
| *** ttsiouts has quit IRC | 11:25 | |
| *** hamzy has quit IRC | 11:32 | |
| *** hamzy has joined #openstack-nova | 11:32 | |
| *** jpena is now known as jpena|lunch | 11:47 | |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update the contributor guide https://review.openstack.org/606926 | 12:03 |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update the CLI reference https://review.openstack.org/606871 | 12:04 |
| *** tbachman has joined #openstack-nova | 12:04 | |
| *** tbachman_ has joined #openstack-nova | 12:09 | |
| *** tbachman has quit IRC | 12:09 | |
| *** tbachman_ is now known as tbachman | 12:09 | |
| *** ttsiouts has joined #openstack-nova | 12:10 | |
| *** beagle is now known as beagles | 12:16 | |
| *** _hemna has quit IRC | 12:27 | |
| *** Luzi has quit IRC | 12:29 | |
| *** _pewp_ has quit IRC | 12:30 | |
| *** _pewp_ has joined #openstack-nova | 12:31 | |
| *** markvoelker has joined #openstack-nova | 12:32 | |
| *** Bhujay has joined #openstack-nova | 12:36 | |
| *** jpena|lunch is now known as jpena | 12:42 | |
| *** Luzi has joined #openstack-nova | 12:43 | |
| gibi | sean-k-mooney: same here, without bhagyashris this remains a mystery | 12:44 |
| *** mriedem has joined #openstack-nova | 12:46 | |
| *** tiendc has quit IRC | 12:48 | |
| *** tssurya has joined #openstack-nova | 12:53 | |
| *** hamdyk has quit IRC | 12:54 | |
| *** pcaruana has quit IRC | 12:57 | |
| *** Luzi has quit IRC | 13:06 | |
| *** mdbooth has joined #openstack-nova | 13:06 | |
| mdbooth | gibi efried: I just got hit by my own patch which landed earlier: http://logs.openstack.org/36/605436/4/check/nova-tox-functional-py35/66bf4c1/job-output.txt.gz | 13:07 |
| mdbooth | Just convincing myself that this is genuinely uncovering yet another race | 13:07 |
| *** awaugama has quit IRC | 13:09 | |
| efried | mdbooth: This is good. Very good. | 13:09 |
| mdbooth | efried: Ok, I can reproduce it non-deterministically locally... 'Hopefully' that means it's actually a new race rather than a bug in my other patch :) | 13:10 |
| * mdbooth dons his deerstalker | 13:10 | |
| efried | s/new/latent/ <== good | 13:10 |
| mdbooth | efried: Indeed, an important clarification ;) | 13:11 |
| mdbooth | Ah... actually in this case it's because my patch invalidated some assumptions in the testcase | 13:14 |
| mdbooth | So.. still good | 13:14 |
| mdbooth | 'my patch' being my other patch | 13:15 |
| *** Luzi has joined #openstack-nova | 13:23 | |
| *** ttsiouts has quit IRC | 13:23 | |
| *** pcaruana has joined #openstack-nova | 13:27 | |
| efried | mdbooth: ah, in fact https://review.openstack.org/#/c/605436/ apparently does not closes-bug #1763181 | 13:31 |
| openstack | bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9) | 13:31 |
| efried | but your other patch has made it manifest differently. | 13:31 |
| efried | which may, please gods, make it figure-outable. | 13:31 |
| mdbooth | efried: Oh, I didn't remove that? | 13:32 |
| mdbooth | efried: I have a different patch which actually fixes that | 13:32 |
| efried | okay, I left a reminder with a -1 :) | 13:32 |
| mdbooth | On, no | 13:32 |
| mdbooth | efried: No, actually it really does fix it | 13:33 |
| *** ttsiouts has joined #openstack-nova | 13:33 | |
| efried | But but but | 13:33 |
| efried | the same bug is showing up in the test results | 13:33 |
| mdbooth | efried: That's a bug in the patch :) | 13:33 |
| mdbooth | It's a different bug | 13:33 |
| efried | now I'm thoroughly confused. | 13:33 |
| efried | you're saying that https://review.openstack.org/#/c/605436/ fixes bug #1763181, but there's a *different* race in parallel_evacuate? | 13:34 |
| openstack | bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9) | 13:34 |
| mdbooth | So the functional test used to assume that stuff was deterministically executed in order X | 13:34 |
| mdbooth | My fix for the actual bug broke that assumption in the test | 13:34 |
| mdbooth | I'm just fixing it | 13:34 |
| mdbooth | So it does fix the issue, but it causes the same test to fail differently with a new bug that I just introduced ;) | 13:35 |
| efried | Nice. So... we'll ultimately see those patches in a series together? | 13:35 |
| mdbooth | No, I'll just fix the bug and resubmit in a minute. | 13:35 |
| efried | in https://review.openstack.org/#/c/605436/ ? | 13:36 |
| mdbooth | efried: Yep | 13:36 |
| efried | okay | 13:37 |
| efried | I have clarified my comment | 13:37 |
| efried | fwiw | 13:37 |
| mdbooth | efried: It's a bug in my new patch. | 13:37 |
| *** cdent has quit IRC | 13:37 | |
| mdbooth | It's a bug in my patch to fix the bug. | 13:37 |
| efried | I think I understand now. I'll stop bugging you and let you fix the bug in the fix. | 13:37 |
| mdbooth | efried: ;) | 13:38 |
| *** wznoinsk has joined #openstack-nova | 13:38 | |
| mdbooth | efried: I only highlighted it originally, because I being super-alert to the possibility of having to revert my other patch. | 13:38 |
| efried | but if I remember, I'll try to get several successful rechecks before approving. | 13:38 |
| mdbooth | efried: That will take weeks at the moment ;) | 13:39 |
| mdbooth | But sure | 13:39 |
| efried | okay, "successful" meaning that specific test passes. | 13:39 |
| efried | the runs are only taking a couple of hours. | 13:40 |
| *** cdent has joined #openstack-nova | 13:47 | |
| *** cfriesen has joined #openstack-nova | 13:52 | |
| *** awaugama has joined #openstack-nova | 13:52 | |
| *** awaugama has quit IRC | 13:52 | |
| *** awaugama has joined #openstack-nova | 13:53 | |
| sean-k-mooney | melwitt mriedem so i finally got time to start working on sriov livemigration and i can confirm that we "broke" macvtap based sriov live migration in rocky when we added the multiple port binding. | 13:54 |
| sean-k-mooney | melwitt: mriedem im not sure why yet but i think its due to how we create the dest binding so this might be a neutron error or nova error depending on the intended behavior | 13:55 |
| *** ttsiouts has quit IRC | 13:58 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667 | 14:00 |
| *** mriedem has quit IRC | 14:00 | |
| *** tbachman has quit IRC | 14:01 | |
| *** mhen has quit IRC | 14:01 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667 | 14:02 |
| *** tbachman has joined #openstack-nova | 14:03 | |
| *** tbachman has quit IRC | 14:04 | |
| *** amotoki_ is now known as amotoki | 14:04 | |
| *** mriedem has joined #openstack-nova | 14:05 | |
| mriedem | sean-k-mooney: super | 14:05 |
| *** mhen has joined #openstack-nova | 14:06 | |
| *** tbachman has joined #openstack-nova | 14:07 | |
| sean-k-mooney | mriedem: i think we need to set the vnic-type when creating the binding but i also was under the impression that neutorn was ment to copy that info form the source binding if not set so not sure its our bug. | 14:07 |
| sean-k-mooney | mriedem: ill file a bug but it should hopefully be a trival backport. that said "sriov migration" even when it works is not reallly supprot yet | 14:08 |
| sean-k-mooney | im hoping to work on that this cycle but we will see how thing go. | 14:09 |
| *** ttsiouts has joined #openstack-nova | 14:12 | |
| *** tbachman has quit IRC | 14:12 | |
| *** mlavalle has joined #openstack-nova | 14:12 | |
| *** tbachman has joined #openstack-nova | 14:12 | |
| *** med_ has quit IRC | 14:15 | |
| *** Luzi has quit IRC | 14:19 | |
| *** munimeha1 has joined #openstack-nova | 14:24 | |
| *** liuyulong has joined #openstack-nova | 14:29 | |
| *** liuyulong has quit IRC | 14:42 | |
| *** med_ has joined #openstack-nova | 14:42 | |
| jaypipes | sean-k-mooney: let's get down to brass tacks on https://review.openstack.org/#/c/603805/2/specs/stein/approved/device-discovery-policy-lib.rst. What is the likelihood that this will proceed? I have a number of concerns about the proposed implementation, but I don't really want to spend a bunch of time on reviewing it if you're not going to have time to pursue this and nobody else is either. | 14:44 |
| efried | jaypipes: I will pursue it to whatever extent sean-k-mooney doesn't have time. Depending which approach you (and/or others) prefer, I will pursue as appropriate. | 14:45 |
| jaypipes | efried: I thought you were more into pursuing Konstantinos' spec? | 14:46 |
| efried | jaypipes: They're both shooting at the same target. | 14:46 |
| *** ttsiouts has quit IRC | 14:46 | |
| efried | jaypipes: I like my baby better, possibly mostly because it's my baby (though I do have some logical reasons), but it's not a big gap. | 14:46 |
| efried | jaypipes: So whichever approach (the spirit, not the specifics yet - as I said, neither is perfectly ready for primetime yet) you and/or others prefer, I'm going to run with. | 14:47 |
| efried | But I want to run with one of them asap. | 14:48 |
| efried | fwiw, I already have code written for the schema in kosamara's spec. It's in nova-powervm atm, but trivial to port. | 14:49 |
| jaypipes | efried: I only have time this morning to review one of those specs. I will review kosamara's then. | 14:50 |
| efried | jaypipes: ack, thanks. | 14:51 |
| efried | note that a number of issues have been mentioned and acked, but not yet revised for, in previous patch sets. | 14:51 |
| sean-k-mooney | jaypipes: well that depend i would really like to pursue this but my primary focus this release will be fixing more live migration stuff specificaly for sriov | 14:53 |
| sean-k-mooney | jaypipes: spend time reviewing kosamara's proposal. | 14:54 |
| *** ttsiouts has joined #openstack-nova | 15:00 | |
| *** toabctl has joined #openstack-nova | 15:03 | |
| *** med_ has quit IRC | 15:03 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436 | 15:04 |
| stephenfin | johnthetubaguy: When you're about, fancy taking another look at https://review.openstack.org/#/c/456572 | 15:11 |
| sean-k-mooney | mriedem: so yes i just confirmed we need to pass the vnic type to the binding api but i also found another neutron bug | 15:13 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable nested allocation candidates in scheduler https://review.openstack.org/585672 | 15:14 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use provider tree in virt FakeDriver https://review.openstack.org/604083 | 15:14 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor allocation checking in functional tests https://review.openstack.org/607287 | 15:15 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Run ServerMovingTests with nested resources https://review.openstack.org/604084 | 15:15 |
| sean-k-mooney | gibi: so while macvtap sriov livemigration i was able to cause the same neutron db confilct and fush error so it looks liek tehre is definetly a neutron portbining bug at least on master | 15:15 |
| gibi | sean-k-mooney: good info | 15:19 |
| sean-k-mooney | i need to do more diging to see if this is just an issue for sriov or if i get the same behavior with just ovs so ill test that later today. | 15:20 |
| gibi | sean-k-mooney: good info | 15:20 |
| gibi | sean-k-mooney: oops | 15:20 |
| gibi | sean-k-mooney: I mean, thank you | 15:21 |
| sean-k-mooney | no worries this will be a blocker for me to enable sriov livemigration anyway so i need to figure out how to fix it one way or another | 15:22 |
| johnthetubaguy | stephenfin: what about uri being last in the list, leaving the mostly commonly changed ones first? Not sure I am actually that bothered by the order actually, kinda happy with whatever gets the later fixes in quickly, which is probably what you have now. | 15:22 |
| jaypipes | sean-k-mooney: ack, will do. thx. | 15:22 |
| *** dpawlik has quit IRC | 15:22 | |
| stephenfin | johnthetubaguy: Sure, I can do that. I'm writing an expanded message to better explain the reasoning for why this is necessary | 15:22 |
| *** ircuser-1 has joined #openstack-nova | 15:25 | |
| *** cdent has quit IRC | 15:27 | |
| *** dave-mccowan has quit IRC | 15:27 | |
| *** dave-mccowan has joined #openstack-nova | 15:28 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Gather 'live_migration_scheme', 'live_migration_inbound_addr' https://review.openstack.org/456572 | 15:29 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Convert 'live_migration_inbound_addr' to HostAddressOpt https://review.openstack.org/456573 | 15:29 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.openstack.org/456571 | 15:29 |
| stephenfin | johnthetubaguy: aaand done. Hopefully that's clearer | 15:29 |
| johnthetubaguy | stephenfin: cool, thanks | 15:30 |
| stephenfin | owalsh: You probably want to look at those (again) too ^ | 15:30 |
| owalsh | stephenfin: ack | 15:30 |
| johnthetubaguy | stephenfin: although now you have scheme first which basically says "please don't touch this config" | 15:31 |
| stephenfin | johnthetubaguy: I was just about to comment on https://review.openstack.org/#/c/456572/8/nova/conf/libvirt.py@278 about that | 15:32 |
| stephenfin | They're in URI order :) | 15:32 |
| stephenfin | I wonder if they should all be advanced? | 15:32 |
| stephenfin | *marked as advanced | 15:33 |
| stephenfin | Seeing as in many (most?) cases the driver can configure this automatically for us | 15:33 |
| johnthetubaguy | stephenfin: it should be marked advanced, yeah | 15:33 |
| johnthetubaguy | stephenfin: I can live with URI order then :) | 15:34 |
| sean-k-mooney | stephenfin: wait you are removing live_migration_uri | 15:34 |
| stephenfin | sean-k-mooney: Nope, I'm closing gaps to allow us to remove it in the future | 15:35 |
| sean-k-mooney | ... i guess i normally only change the user name so i could chagne to live_migration_username instead | 15:36 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ignore forcing of live migration for nested instance https://review.openstack.org/605785 | 15:36 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consider nested allocations during allocation cleanup https://review.openstack.org/606050 | 15:36 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ignore forcing of evacuation for nested instance https://review.openstack.org/606111 | 15:36 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Run negative server moving tests with nested RPs https://review.openstack.org/604125 | 15:36 |
| stephenfin | sean-k-mooney: yeah, that's the idea | 15:36 |
| sean-k-mooney | live_migration_username=root woudl be the same as live_migration_uri = qemu+ssh://root@%s/system | 15:37 |
| mdbooth | efried: Incidentally, I ran that test 250 times locally without failure. | 15:37 |
| efried | mdbooth: And you had been able to reproduce it some percentage of the time prior to the fix? | 15:38 |
| mdbooth | Yeah, before it failed 50%, as expected. | 15:38 |
| mdbooth | That's with *my* bug | 15:38 |
| owalsh | stephenfin: live_migration_tunnelled isn't related, is it? | 15:38 |
| mdbooth | With the original bug it failed about 1 in 10 locally | 15:38 |
| efried | mdbooth: Okay, then 250 passes is good enough for me. Thanks for your work on this. That bug has been a thorn in our collective sides for a long time. | 15:39 |
| openstackgerrit | Merged openstack/os-vif master: add nested DPDK VIF classes for kuryr-kubernetes https://review.openstack.org/604992 | 15:39 |
| stephenfin | owalsh: If tunneling is enabled, none of that configuration is used, right? | 15:39 |
| mdbooth | efried: Hasn't passed check queue once yet, though :) | 15:40 |
| owalsh | stephenfin: commit message says the opposite | 15:40 |
| stephenfin | whoops | 15:40 |
| mdbooth | Latest revision, that is | 15:40 |
| efried | mdbooth: But it's still on the first run? | 15:40 |
| mdbooth | Yeah | 15:40 |
| stephenfin | That should read "disabled" | 15:40 |
| efried | :) | 15:40 |
| mdbooth | efried: If it fails for any related reason I promise I'll shout about it :) | 15:40 |
| *** hamzy has quit IRC | 15:40 | |
| owalsh | stephenfin: ack, makes a lot more sense then :-) | 15:41 |
| *** ttsiouts has quit IRC | 15:42 | |
| sean-k-mooney | stephenfin: why are you defaulting to qemu+tcp and not qemu+ssh if the schema is not set? | 15:42 |
| *** macza has joined #openstack-nova | 15:43 | |
| sean-k-mooney | libvirt is typiclaly not exosed over tcp by default | 15:44 |
| stephenfin | sean-k-mooney: That was the case previously, right? (i.e. I haven't change anything) | 15:45 |
| sean-k-mooney | ya just looking at the old code. it was the case yes however the code comment is inccort in the old code too | 15:45 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Gather 'live_migration_scheme', 'live_migration_inbound_addr' https://review.openstack.org/456572 | 15:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Convert 'live_migration_inbound_addr' to HostAddressOpt https://review.openstack.org/456573 | 15:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.openstack.org/456571 | 15:46 |
| stephenfin | johnthetubaguy: Sorry, fancy hitting the first of those again. Typo fix pointed out by owalsh | 15:47 |
| *** macza_ has joined #openstack-nova | 15:47 | |
| sean-k-mooney | stephenfin: perhapes as a follow up we should have a seperate patch to change the default. or at least advise againts using tcp as that will copy the state unencrypted | 15:48 |
| *** sahid has quit IRC | 15:49 | |
| *** sahid has joined #openstack-nova | 15:49 | |
| *** gyee has joined #openstack-nova | 15:49 | |
| sean-k-mooney | stephenfin: ssh or libssh/libssh2 would be much better defaults imho | 15:51 |
| *** macza has quit IRC | 15:51 | |
| stephenfin | sean-k-mooney: Hmm, while I agree, I'm nervous to go changing defaults willy-nilly. I'm assuming ssh would require extra configuration for anyone relying on TCP? | 15:52 |
| stephenfin | *currently using TCP | 15:52 |
| stephenfin | And I imagine TripleO, Kolla etc. already tweak those knobs | 15:52 |
| sean-k-mooney | stephenfin: yes but to use TCP that person would have had to 1 expose libvirt unencyrted and unauthentated via tcp as it is disabled by default in ubunut/debian and rhel/fedora distros | 15:53 |
| sean-k-mooney | stephenfin: see https://libvirt.org/remote.html#Remote_transports | 15:53 |
| sean-k-mooney | tcp is expressed not recommended for production use which is why im uneasey about that being our defualt | 15:55 |
| stephenfin | sean-k-mooney: I can do that so but it'll warrant an Upgrade reno. ssh or tls as the default? | 15:55 |
| sean-k-mooney | stephenfin: kolla uses ssh not sure about tripplo | 15:55 |
| stephenfin | I think TripleO uses TLS if TLS everywhere is toggled but owalsh would need to confirm | 15:55 |
| artom | gryf, around? You wanted to ask about NUMA live migration? | 15:55 |
| *** Sigyn has quit IRC | 15:56 | |
| sean-k-mooney | stephenfin: i would go with ssh but it likely somthing we want to discuss perhaps in the nova team meeting | 15:56 |
| owalsh | stephenfin, sean-k-mooney: yea, TLS if it's enabled, otherwise ssh tunnelling | 15:56 |
| sean-k-mooney | or as a bug/bluepint | 15:56 |
| *** Sigyn has joined #openstack-nova | 15:57 | |
| sean-k-mooney | stephenfin: if we dont change the default it would be nice to atleast issue a wanrning on compute agent startup so that operators are very ware that they should not be using tcp by defualt | 15:59 |
| owalsh | sean-k-mooney: I wouldn't be surprised if it got #ifdef-ed out of libvirt by default too | 16:00 |
| efried | mdbooth: Oh, I meant to mention this earlier... | 16:00 |
| efried | we don't need context.nested, because that's implemented natively in py2 (I think as of 2.6?) and py3. | 16:01 |
| sean-k-mooney | owalsh: its compiled into the centos bin but its disabled in the libvirt deamon config so you have to expcily opt in to it | 16:01 |
| efried | mdbooth: s/context.nested/nested_contexts/ | 16:01 |
| mdbooth | efried: It's in py3? Where? | 16:02 |
| efried | mdbooth: natively | 16:02 |
| owalsh | sean-k-mooney: yes, but all of the random blogs on setting up migration enabled it | 16:02 |
| mdbooth | Called what? | 16:02 |
| efried | mdbooth: Called 'with' :) | 16:02 |
| mdbooth | efried: Oh, you mean the 'with-syntax' | 16:02 |
| efried | mdbooth: maybe I'm not understanding what you're trying to do with it. | 16:02 |
| sean-k-mooney | efried: really when did that get backported | 16:02 |
| mdbooth | Right, but that's *not* implemented in py2 | 16:02 |
| mdbooth | The optional bit, that is | 16:03 |
| sean-k-mooney | efried: we still have at least 1 place where we stub out compatiably in nova | 16:03 |
| mdbooth | You can't say: | 16:03 |
| *** cdent has joined #openstack-nova | 16:03 | |
| mdbooth | with X, Y (unless Y is None) | 16:03 |
| mdbooth | : | 16:03 |
| *** helenafm has quit IRC | 16:03 | |
| mdbooth | Basically the choices for optional composition are contextlib.nested, and ExitStack, and neither is available in both py2 and py3 | 16:04 |
| efried | mdbooth: oh, I was missing the filter | 16:04 |
| stephenfin | mdbooth: https://pythonclock.org/ | 16:04 |
| * stephenfin ducks | 16:05 | |
| mdbooth | stephenfin: Python 3 is the culprit here, because Python 2 was already here. | 16:05 |
| mdbooth | Deprecating a whole programming language was a monumentally stupid idea. | 16:05 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/ocata: [Stable Only] Add amd-ssbd and amd-no-ssb CPU flags https://review.openstack.org/607296 | 16:06 |
| mdbooth | efried: Incidentally, you can make Y a 'null context', i.e. a context manager which does nothing other than execute whatever it's given. | 16:09 |
| mdbooth | And py3 includes a contextlib.nullcontext | 16:09 |
| mdbooth | But py2 doesn't | 16:09 |
| mdbooth | So I could have written one, but it would still be duplicating code in py3 | 16:09 |
| sean-k-mooney | mdbooth: it could jsut delegate to the py3 verions on py3 and implement the 10 lines it takes on py2. | 16:10 |
| efried | or after the import: | 16:10 |
| efried | if not hasattr(contextlib, 'nullcontext'): | 16:10 |
| efried | contextlib.nullcontext = lambda x: x | 16:10 |
| efried | ? | 16:10 |
| sean-k-mooney | we cant drop py2 support for 2 more releases so i dont see an issue with it | 16:10 |
| efried | anyway, what you've done is foyne. | 16:10 |
| mdbooth | efried: That's not quite a nullcontext. You probably need at least 4 lines for it | 16:11 |
| efried | yeah yeah | 16:11 |
| mdbooth | efried: The point is there were a bunch of ways to do it, but none of them are as good as what you could do if we were only targetting a single programming language. | 16:11 |
| efried | yup, I get it. You had to choose among several ugly options. What you've done is foyne. | 16:12 |
| *** Bhujay has quit IRC | 16:12 | |
| artom | Weird - I'm trying to reproduce a bug where if an instance reschedules it gets two vifs - I'm trying to force a reschedule by just raising in build_and_run_instance one one of the two computes. Except *no instance ever lands on that compute* | 16:15 |
| artom | That host definitely doesn't get filtered out, it's enabled, etc | 16:15 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: DNM: Add some debug logging https://review.openstack.org/607301 | 16:15 |
| sean-k-mooney | artom: try raising right at the point the libvirt driver calls libvir to spawn the instance | 16:18 |
| gibi | mriedem: hi, do you have something for the notification subteam meeting? I don't have any | 16:18 |
| artom | sean-k-mooney, it's not even that. I've booted 10 instances with --min-count, and they've *all* gone to the "wrong" host | 16:19 |
| *** toabctl has quit IRC | 16:19 | |
| sean-k-mooney | artom: you didnt hit the auto host down on faiure codepath? | 16:21 |
| artom | sean-k-mooney, well, how would I see that? The hypervisor shows as up, same for the service | 16:21 |
| dansmith | artom: there's a weigher now | 16:23 |
| dansmith | for the fail count, | 16:23 |
| dansmith | but that will only take effect if you have failed once on that compute already | 16:24 |
| artom | dansmith, aha, thank you! | 16:24 |
| artom | That would explain it - now, presumably this fail count is hidden in the DB somewhere. | 16:24 |
| * artom goes spelunking. | 16:24 | |
| sean-k-mooney | dansmith: didnt you also add something last cycle to auto makre the state as disabled after 10 ish failed boots in addtion to the weigher | 16:25 |
| dansmith | sean-k-mooney: no, the weigher replaced the auto-down thing | 16:25 |
| sean-k-mooney | artom: im assuming openstack compute service list shoul the state as up and statuse as enabled | 16:25 |
| sean-k-mooney | dansmith: ah ok i guess that makes sense we had no auto up right so you had to manully go mark it up | 16:26 |
| dansmith | right | 16:27 |
| artom | dansmith, hrmm, does the fail count get reset on service restart? | 16:27 |
| artom | Btw, this would be a think we should probably expose in the hypervisors API or something | 16:27 |
| sean-k-mooney | artom: if its in memory you would have to restart the schduler | 16:28 |
| dansmith | it's in the host state | 16:28 |
| mriedem | gibi: no, although i do have something i can stick in your ear... | 16:28 |
| dansmith | so we should pull that each time in the filter scheduler, IIRC | 16:28 |
| mriedem | gibi: https://review.openstack.org/#/c/603930/6/nova/conductor/tasks/cross_cell_migrate.py@519 for the cross-cell resize stuff i'm adding a new conductor task and the conductor code relies on that old scheduler_utils.set_vm_state_and_notify method, which sends an error notification but those are all legacy | 16:29 |
| mriedem | https://github.com/openstack/nova/blob/cce3208cc28268e4b50e155c205bcab9f1da2a4b/nova/scheduler/utils.py#L600 | 16:29 |
| sean-k-mooney | artom: this is the filter https://github.com/openstack/nova/blob/master/nova/scheduler/weights/compute.py#L24 you could just set the weight multipler to 0 to disable it | 16:29 |
| mriedem | i'm not entirely sure how to make that generic thing useful for versioned notifications | 16:29 |
| dansmith | artom: you can just zero or invert the weight and see if that changes it | 16:29 |
| gibi | mriedem: looking | 16:29 |
| mriedem | artom: see compute_nodes.stats column | 16:30 |
| mriedem | which is a serialized dict i think.... | 16:30 |
| artom | mriedem, saw it, it's a JSON blob | 16:30 |
| mriedem | yup | 16:30 |
| mriedem | not a terrible idea to report that out of the os-hypervisors API | 16:30 |
| artom | Restarting sched and/or compute seems to do what I want (ie, effectively reset the failure count) | 16:30 |
| artom | mriedem, yeah eh? It would avoid more losers like me getting lost | 16:31 |
| gibi | mriedem: there is an open (and old) patch about that missing set of notifications https://review.openstack.org/#/c/482629/9/nova/scheduler/utils.py | 16:31 |
| gibi | mriedem: s/old/not reviewed by me for a while/ | 16:32 |
| mriedem | oh boy...that's a big one | 16:32 |
| gibi | mriedem: stephenfin was happy with that not so long ago | 16:32 |
| gibi | mriedem: anyhow I made a note to review this sooner than later | 16:33 |
| mriedem | definitely not a high priority, but thanks for noting this exists | 16:33 |
| gibi | mriedem: the burndown chart remembered that it exists not me :) | 16:35 |
| gibi | mriedem: anyhow I will not run the notification subteam meeting this week either then | 16:36 |
| *** sahid has quit IRC | 16:37 | |
| mriedem | gibi: ok | 16:39 |
| mriedem | gibi: do you want to just cancel the meeting? | 16:39 |
| mriedem | i mean, indefinitely | 16:40 |
| gibi | mriedem: that decision is forming slowly in my mind in the past weeks | 16:41 |
| gibi | mriedem: I think it would make sense to cancel it indefinitely | 16:41 |
| mriedem | go for it, | 16:44 |
| mriedem | just have to drop it from http://git.openstack.org/cgit/openstack-infra/irc-meetings/ | 16:44 |
| cdent | --meetings++ | 16:45 |
| gibi | mriedem: OK, and I drop a mail to ML just for the record | 16:45 |
| *** _hemna has joined #openstack-nova | 16:46 | |
| gibi | mriedem: fyi https://review.openstack.org/607314 | 16:50 |
| * sean-k-mooney tries to figure out if in c cdent is decrementing the temporay retruned by post incrament to incramenting the copy retruned by the predecmrent and then gives up | 16:52 | |
| *** dtantsur is now known as dtantsur|afk | 16:52 | |
| cdent | sean-k-mooney: I was "less meetings is good" in reponse to gibi. My syntax very bad. | 16:53 |
| sean-k-mooney | cdent: yes as a sane person that was my assumtion | 16:53 |
| sean-k-mooney | cdent: actully since your about. we still block live migration with config drive to fail right | 16:55 |
| gibi | cdent: :) | 16:56 |
| cdent | sean-k-mooney: that is not an area of expertise for me, but a hazy memory suggests that's the case | 16:56 |
| openstackgerrit | Merged openstack/nova stable/pike: Fix host validity check for live-migration https://review.openstack.org/590263 | 16:57 |
| sean-k-mooney | cdent: oh ok i had a vague memory that you were invovled in adding config drive at some point | 16:57 |
| sean-k-mooney | cdent: in either case it does which is what i was expecting | 16:57 |
| cdent | not me, unless I blacked it out | 16:57 |
| sean-k-mooney | cdent: yes that would be a sensable thing to do if you had worked on config drive :P | 16:58 |
| * cdent wonders what else he's forgotten | 16:58 | |
| sean-k-mooney | that abit unfiar to config drive as it actully works prettry in limited usescaes but live migration is not one of them | 16:59 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/ocata: Add check for invalid allocation amounts https://review.openstack.org/607320 | 16:59 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/ocata: Add check for invalid inventory amounts https://review.openstack.org/607321 | 16:59 |
| *** derekh has quit IRC | 17:03 | |
| *** jpena is now known as jpena|off | 17:14 | |
| *** hamzy has joined #openstack-nova | 17:27 | |
| *** mvkr has quit IRC | 17:31 | |
| cfriesen | So what's the process for getting a specless blueprint approved? One of my coworkers opened up https://blueprints.launchpad.net/nova/+spec/support-hpet-on-guest and there's code up as well. Should he be using the runway system once the blueprint is approved? It's just the one commit. | 17:32 |
| *** Sukhdev has joined #openstack-nova | 17:44 | |
| *** munimeha1 has quit IRC | 17:47 | |
| *** priteau has quit IRC | 17:47 | |
| *** dpawlik has joined #openstack-nova | 17:47 | |
| melwitt | cfriesen: usually when seeking specless blueprint approval, you can add it to the Open Discussion section of the next nova meeting agenda | 17:50 |
| cfriesen | melwitt: thanks, that works | 17:50 |
| melwitt | using the runway system after approval is good for attracting review attention | 17:50 |
| *** openstackgerrit has quit IRC | 17:51 | |
| gryf | artom: unfortunately, I'm afk right now. Which time zone are you in? | 17:51 |
| *** tssurya has quit IRC | 17:52 | |
| *** dpawlik has quit IRC | 17:53 | |
| *** hamzy has quit IRC | 17:55 | |
| *** openstackgerrit has joined #openstack-nova | 17:57 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Placement: Remove usage of get_legacy_facade() https://review.openstack.org/607336 | 17:57 |
| efried | melwitt, cdent: ^ | 17:58 |
| *** hamzy has joined #openstack-nova | 18:02 | |
| *** pcaruana has quit IRC | 18:02 | |
| *** mvkr has joined #openstack-nova | 18:05 | |
| *** dpawlik has joined #openstack-nova | 18:09 | |
| *** dpawlik has quit IRC | 18:14 | |
| *** imacdonn has quit IRC | 18:21 | |
| *** imacdonn has joined #openstack-nova | 18:21 | |
| *** openstackgerrit has quit IRC | 18:35 | |
| *** openstackgerrit has joined #openstack-nova | 18:43 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | 18:43 |
| artom | gryf, GMT-4 (NA east coast) | 18:48 |
| artom | gryf, email? | 18:48 |
| dansmith | mriedem: https://review.openstack.org/#/c/607296 | 18:52 |
| *** Sukhdev has quit IRC | 18:52 | |
| mriedem | done | 18:55 |
| *** ralonsoh has quit IRC | 18:56 | |
| *** Swami has joined #openstack-nova | 19:03 | |
| bauzas | dansmith: mriedem: a few other people interested in, I finally reworked the reshaper change https://review.openstack.org/599208 | 19:07 |
| bauzas | I'll test it on a devstack | 19:07 |
| bauzas | with a machine having pGPUs | 19:08 |
| mriedem | :( this is all half-baked https://review.openstack.org/#/q/topic:bug/1384637+(status:open+OR+status:merged) | 19:10 |
| mriedem | none of that plumbing was ever leveraged by the REST API | 19:10 |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Update blueprint name so spec matches launchpad https://review.openstack.org/607347 | 19:10 |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Dynamically find releases for move-implemented-specs https://review.openstack.org/592628 | 19:15 |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Add a script for counting blueprints https://review.openstack.org/581914 | 19:15 |
| cfriesen | if anyone feels like a spec review, the updated emulated TPM spec is up at https://review.openstack.org/#/c/571111 There are no API changes, the spec is really to allow discussion of the concept. | 19:17 |
| *** med_ has joined #openstack-nova | 19:18 | |
| sean-k-mooney | cfriesen: i assume the only enduser fasing change would be an image property or flavor extra spec to enable/request teh vtpm | 19:19 |
| cfriesen | sean-k-mooney: flavor extra spec, yes. the rest is virt driver backend stuff | 19:19 |
| *** tbachman has quit IRC | 19:24 | |
| *** hamzy has quit IRC | 19:25 | |
| *** hamzy has joined #openstack-nova | 19:26 | |
| sean-k-mooney | ill review it in detail tomorow but initall feedback is i would proably expect 2 extra_spec argument one to specify the tpm verion and another for the backend type. other then that it would be nice to support this via the image metadata too but after 5 mins skiming it it seams resonable | 19:28 |
| sean-k-mooney | i seam to recall form the PTG there were some live migration requirements too which i did not see explcitly in the spec. if so proabley a good idea to add them | 19:29 |
| *** awaugama has quit IRC | 19:31 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Fail count in API https://review.openstack.org/607352 | 19:33 |
| artom | dansmith, mriedem, ^^ really easy spec about the fail count discussion earlier | 19:34 |
| dansmith | hmm | 19:35 |
| bauzas | artom: I thought we said to deprecate os-hypervisors API ? | 19:35 |
| bauzas | at least not adding more to it | 19:35 |
| artom | bauzas, seriously? I had no idea. | 19:35 |
| artom | Nothing in the api-ref about it | 19:35 |
| dansmith | and that fail count is an internal value that can change | 19:35 |
| mriedem | artom wasn't at the ptg when os-hypervisors was discussed... | 19:36 |
| dansmith | so I'm kinda meh about exposing it and having people make up beliefs about what it means | 19:36 |
| artom | I mean, I'm not looking for +W fast track here :) | 19:36 |
| artom | So if there are legit issues, destroy at will | 19:36 |
| artom | We could find other ways of indicating the same information, if y'all agree the basic idea is worthwhile | 19:36 |
| bauzas | can't we just emit a notification ? | 19:37 |
| bauzas | stupid idea maybe | 19:37 |
| artom | And if y'all don't, I'm cool as well, NUMA live migration is quite enough for me ;) | 19:37 |
| *** dave-mccowan has quit IRC | 19:41 | |
| cfriesen | sean-k-mooney: thanks. to allow it in the image we'd have to use the "trait" as specified in the alternatives section. I'd be fine with that too, I just went with a resource because Eric Fried suggested it. :) | 19:51 |
| cfriesen | sean-k-mooney: live migration is fine, and cold migration is covered in the spec | 19:54 |
| artom | bauzas, notifications I don't think are very useful, but logs would definitely work | 19:55 |
| artom | To be honest, not sure why I didn't go there in the first place | 19:55 |
| cfriesen | artom: probably cause mriedem was talking about the os-hypervisors API this morning | 19:56 |
| artom | mriedem, see, totally your fault. | 19:56 |
| bauzas | I wasn't looking at the IRC discussion this 'US' morning | 19:57 |
| bauzas | what was the point ? | 19:57 |
| artom | The hell, we already log weighed hosts | 19:58 |
| sean-k-mooney | cfriesen: cool am i would have to read the spec properly to understand traits vs reousce comment but ill take your word for it. i had assumed we could have HW_VTPM_TYPE=emulated HW_VTPM_VERSION=2.0 image property pair and have nova construt the resouce request form that but perhaps there is a reason that would not work that i missed | 19:58 |
| mriedem | "(11:27:58 AM) artom: Btw, this would be a think we should probably expose in the hypervisors API or something" | 19:59 |
| artom | mriedem, sshh, let me have this | 19:59 |
| sean-k-mooney | bauzas: the context was artom was trying to recreate the "vm spawns with multiples" issue and injected a fault which resulted in all his instances landing on a host he was not expecting | 19:59 |
| * bauzas raises a fist at multiple-create again | 20:00 | |
| openstackgerrit | Merged openstack/nova stable/ocata: Fix instance evacuation with PCI devices https://review.openstack.org/605881 | 20:00 |
| openstackgerrit | Merged openstack/nova stable/ocata: Update nova network info when doing rebuild for evacuate operation https://review.openstack.org/605882 | 20:00 |
| sean-k-mooney | bauzas: idea was to help debuging perhaps expose it via hyperviors api but i would guess a debug level weigher log might be better in this case | 20:00 |
| artom | Ah, we just log the final sorted list, not the individual weights, nor the per-filter weights | 20:01 |
| bauzas | could we just have alaski back here and just him and me +2/+W a change deprecating multiple-create API ? | 20:01 |
| mriedem | oh god why would we allow attaching volumes to a resized server before it's confirmed/revert | 20:01 |
| mriedem | *reverted | 20:01 |
| mriedem | that's just asking for trouble | 20:02 |
| sean-k-mooney | mriedem: because we did not think that is what we were allowing at the time the code merged ? | 20:02 |
| melwitt | bauzas: users love the multi-create API | 20:03 |
| sean-k-mooney | mriedem: i cound half of the issue with the multiple port bindingings thing by the way. ill file a bug and upload a patch tomorow | 20:03 |
| mriedem | there is a forum session about the multi-create api | 20:03 |
| sean-k-mooney | *found | 20:03 |
| artom | mriedem, I feel like attaching anything to anything that isn't ACTIVE is asking for trouble | 20:03 |
| artom | Like, we should wrap the instance decorator around any method that as 'attach' in its name. | 20:04 |
| artom | *instance state decorator | 20:04 |
| sean-k-mooney | melwitt: do the love the multi create api or multi create support in the client/sdk/osc | 20:04 |
| sean-k-mooney | artom: i think the vm is active in this case on the dest | 20:05 |
| melwitt | mriedem: is it under a more broad topic? I don't see it | 20:05 |
| sean-k-mooney | artom: we just have not confimed it | 20:05 |
| artom | sean-k-mooney, it's still in VERIFY_RESIZE in the API tho | 20:06 |
| artom | I think is what mriedem means | 20:06 |
| melwitt | sean-k-mooney: the API, I think. | 20:06 |
| cfriesen | sean-k-mooney: I'd be open to something like that if people are looking for additional flexibility. making the version explicit would at least protect us if qemu ever supported a newer version | 20:06 |
| *** Sundar_ has joined #openstack-nova | 20:06 | |
| sean-k-mooney | cfriesen: you can always default it to 2.0 initally so its optional | 20:07 |
| *** dave-mccowan has joined #openstack-nova | 20:07 | |
| mriedem | bauzas: melwitt: https://www.openstack.org/summit/berlin-2018/vote-for-speakers#/22840 | 20:07 |
| bauzas | melwitt: I'm not against something elsewhere but not in the API :) | 20:08 |
| bauzas | that said, now the ship has sailed... | 20:08 |
| bauzas | I'm pretty sure we'd get lots of arguments if we deprecate it :) | 20:08 |
| sean-k-mooney | mriedem: i would assume the answer to there first quest is ther is no sla followed by there is no test coverage for that usecase | 20:08 |
| mriedem | i tested how you can kill the scheduler https://review.openstack.org/#/c/507918/ | 20:09 |
| mriedem | if that helps | 20:09 |
| sean-k-mooney | thats a lot of red from the ci | 20:10 |
| mriedem | it's an experiment | 20:10 |
| mriedem | not meant to run tempest | 20:10 |
| melwitt | cool experiment. I wonder could the oslo.messaging heartbeat be leveraged in this way? | 20:11 |
| sean-k-mooney | to kill the schduler proably | 20:11 |
| mriedem | yes, | 20:11 |
| mriedem | it passed once i set the rpc_response_timeout to 300 | 20:12 |
| mriedem | the timeout stuff with the select_destinations rpc call likely needs to use the long_rpc_timeout | 20:12 |
| mriedem | because if it times out, today we retry the rpc call up to max_attempts | 20:12 |
| mriedem | so it just compounds the problem | 20:12 |
| melwitt | yeah, sorry, I saw you increased the timeout and that worked -- that's why I asked. I wondered if we could use the heartbeat to help here | 20:13 |
| mriedem | i think yes, do that and drop the old retry_select_destinations decorator | 20:14 |
| melwitt | ++ | 20:14 |
| *** mlavalle has quit IRC | 20:29 | |
| *** dave-mccowan has quit IRC | 20:30 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | 20:33 |
| *** med_ has quit IRC | 20:36 | |
| openstackgerrit | Merged openstack/nova stable/ocata: unquiesce instance after quiesce failure https://review.openstack.org/605884 | 20:43 |
| *** mlavalle has joined #openstack-nova | 20:44 | |
| *** hamzy has quit IRC | 20:44 | |
| *** ttsiouts has joined #openstack-nova | 20:55 | |
| *** tbachman has joined #openstack-nova | 21:10 | |
| openstackgerrit | sean mooney proposed openstack/nova master: propagate vnic_type and profile on create port bindings https://review.openstack.org/607365 | 21:10 |
| *** ttsiouts has quit IRC | 21:10 | |
| *** ttsiouts has joined #openstack-nova | 21:11 | |
| *** Sundar_ has quit IRC | 21:14 | |
| sean-k-mooney | mriedem: im goint to repropose ^ later as a bug as i would like to backport it + some other related changes just and fyi but that will partly fix macvtap sriov. | 21:15 |
| *** tbachman_ has joined #openstack-nova | 21:15 | |
| *** tbachman has quit IRC | 21:16 | |
| *** tbachman_ is now known as tbachman | 21:16 | |
| sean-k-mooney | mriedem: this patch in partcalar is not super high on my list for the macvtap but per our PTG conversation ther are some other minor bug fixed need to make the codbase actully supprot what we said it would in the rocky release notes | 21:18 |
| sean-k-mooney | s/for the macvtap/for backporting but/ | 21:19 |
| mriedem | ok | 21:20 |
| openstackgerrit | Merged openstack/nova stable/queens: nova-manage - fix online_data_migrations counts https://review.openstack.org/605839 | 21:20 |
| *** awaugama has joined #openstack-nova | 21:21 | |
| bauzas | mriedem: heh, good news, the inventory reshape works | 21:22 |
| bauzas | mriedem: now, I'm testing the allocations move | 21:22 |
| mriedem | works in...functional tests? | 21:22 |
| mriedem | or a real env? | 21:22 |
| bauzas | mriedem: works on my devstack running on a physical machine :) | 21:23 |
| mriedem | le woot | 21:23 |
| bauzas | mriedem: http://paste.openstack.org/show/731303/ | 21:25 |
| melwitt | that reminds me, did someone create a catchall blueprint for all of the "closing the gaps with nested resource providers" work yet? | 21:26 |
| bauzas | oh oops, now you know my secret machine :p | 21:26 |
| bauzas | melwitt: I already have a vgpu-rocky BP that tracks all of the VGPU effort | 21:26 |
| mriedem | melwitt: probably this https://blueprints.launchpad.net/nova/+spec/use-nested-allocation-candidates | 21:26 |
| bauzas | mriedem: use-nested-alloc-candidates is only one of the gaps but meh | 21:27 |
| mriedem | i'll butt meh you | 21:27 |
| melwitt | ok, thanks. I'm looking to collect the related blueprints and make them dependencies for the vgpu theme | 21:27 |
| *** ttsiouts has quit IRC | 21:27 | |
| mriedem | you can't allocate against vgpu child providers without gibi's bp to get the scheduler stuff to work | 21:27 |
| bauzas | melwitt: mriedem: btw. I reproposed an approved spec for vgpu-stein https://review.openstack.org/#/c/602474/ | 21:28 |
| *** ttsiouts has joined #openstack-nova | 21:28 | |
| bauzas | mriedem: yup, I'll rebase the change once I'm done with testing the allocations migration | 21:28 |
| bauzas | mriedem: for this, we don't need the nested-a-c series | 21:28 |
| bauzas | nested-a-c series is only for creating a new instance | 21:29 |
| bauzas | not for a reshape | 21:29 |
| mriedem | sure but.... | 21:29 |
| mriedem | once we model vgpu as child providers, you can't create a new instance with vgpus until that nested-a-c bp is done | 21:29 |
| *** med_ has joined #openstack-nova | 21:29 | |
| bauzas | yeah hence my -W | 21:30 |
| melwitt | yeah, both blueprints will be part of the theme | 21:30 |
| bauzas | it should really be a -2 | 21:30 |
| openstackgerrit | Merged openstack/nova stable/ocata: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604451 | 21:30 |
| openstackgerrit | Merged openstack/nova stable/ocata: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600119 | 21:30 |
| bauzas | anyway, consumer generation is merged | 21:30 |
| bauzas | now, we only really need https://review.openstack.org/#/c/585672/ for the reshaper change, not all the follow-ups | 21:31 |
| bauzas | mriedem: ^ | 21:31 |
| *** cdent has quit IRC | 21:32 | |
| melwitt | mriedem: you're right on use-nested-allocation-candidates being the catchall L53 https://etherpad.openstack.org/p/nova-ptg-stein-priorities | 21:32 |
| mriedem | \o/ | 21:35 |
| *** moshele has joined #openstack-nova | 21:36 | |
| *** med_ has quit IRC | 21:45 | |
| *** tbachman has quit IRC | 21:51 | |
| bauzas | mriedem: FWIW, confirmed that a flavor asking for vgpus raises a NoValidHost once the reshape is done :) | 21:52 |
| bauzas | we all knew this, it's just an evidence :) | 21:52 |
| *** mriedem has quit IRC | 22:03 | |
| *** ttsiouts has quit IRC | 22:04 | |
| dansmith | bauzas: wait, what? | 22:12 |
| dansmith | oh, because we've reshaped to nested but can't yet schedule to nested? | 22:12 |
| melwitt | that was my assumption | 22:14 |
| openstackgerrit | Merged openstack/nova stable/ocata: Use ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa in tree https://review.openstack.org/581445 | 22:16 |
| openstackgerrit | Merged openstack/nova stable/ocata: Fix DB archiver AttributeError due to wrong table name attribute used https://review.openstack.org/599885 | 22:16 |
| openstackgerrit | Merged openstack/nova stable/ocata: Add unshelve instance error info to fault table https://review.openstack.org/583422 | 22:16 |
| openstackgerrit | Merged openstack/nova master: Remove unnecessary redirect https://review.openstack.org/605911 | 22:16 |
| openstackgerrit | Merged openstack/nova master: Replace usage of get_legacy_facade() with get_engine() https://review.openstack.org/606213 | 22:16 |
| openstackgerrit | Merged openstack/nova stable/rocky: Optimize AZ lookup during schedule_and_build_instances https://review.openstack.org/604378 | 22:16 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use nova-consoleauth only if workaround enabled https://review.openstack.org/607068 | 22:20 |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Add a script for counting blueprints https://review.openstack.org/581914 | 22:20 |
| *** moshele has quit IRC | 22:24 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Use nova-consoleauth only if workaround enabled https://review.openstack.org/607068 | 22:26 |
| *** med_ has joined #openstack-nova | 22:32 | |
| *** med_ has quit IRC | 22:51 | |
| *** icey has quit IRC | 23:05 | |
| *** awaugama has quit IRC | 23:11 | |
| *** icey has joined #openstack-nova | 23:12 | |
| *** mlavalle has quit IRC | 23:28 | |
| *** erlon has joined #openstack-nova | 23:36 | |
| *** macza_ has quit IRC | 23:37 | |
| *** macza has joined #openstack-nova | 23:37 | |
| *** macza has quit IRC | 23:42 | |
| *** Swami has quit IRC | 23:42 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Remove unnecessary redirect https://review.openstack.org/607400 | 23:49 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/queens: Remove unnecessary redirect https://review.openstack.org/607402 | 23:54 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/queens: Remove unnecessary redirect https://review.openstack.org/607402 | 23:57 |
| *** rcernin has joined #openstack-nova | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!