14:00:16 <PaulMurray> #startmeeting Nova Live Migration 14:00:17 <openstack> Meeting started Tue Jan 12 14:00:16 2016 UTC and is due to finish in 60 minutes. The chair is PaulMurray. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:20 <openstack> The meeting name has been set to 'nova_live_migration' 14:00:29 <tdurakov> hi 14:00:30 <andrearosa> hi 14:00:39 <shaohe_feng> hi 14:00:42 <PaulMurray> Hi all 14:01:08 <PaulMurray> just wait around a minute to see if any else comes 14:01:12 <davidgiluk> hi 14:01:54 <eliqiao> o/ 14:02:06 <PaulMurray> The agenda is on the meeting page: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration 14:02:13 <jlanoux_> o/ 14:02:48 <PaulMurray> I have been ill so I only got round to putting the agenda up a while ago 14:03:09 <tdurakov> PaulMurray, could you add ci for agenda please 14:03:27 <PaulMurray> tdurakov, yes, certainly 14:03:44 <PaulMurray> In fact lets do that first 14:03:47 <PaulMurray> #topic CI 14:04:06 <tdurakov> current status: https://review.openstack.org/#/c/252283/ 14:04:17 <tdurakov> this patch add ceph for hook 14:04:23 <tdurakov> review and merge needed 14:05:38 <PaulMurray> #action ALL please review https://review.openstack.org/#/c/252283/ 14:06:26 <eliqiao> seems all logic are in run_tests.sh, would that be better to split it out? 14:06:27 * PaulMurray the internet is being a little slow for me today 14:06:50 <tdurakov> eliqiao, sounds reasonable 14:07:09 <tdurakov> eliqiao, check Sebastien comment also 14:07:42 <eliqiao> oh, gold, already version 73, I am so late 14:08:02 * eliqiao :(s/gold/god 14:08:52 <PaulMurray> I was going to mention this in the next topic: https://review.openstack.org/#/c/227278/ failed the live migration CI 14:08:54 <pkoniszewski> o/ 14:09:24 <tdurakov> eliqiao, it could be splitted, Sebastien talking about reusing ceph from devstack/lib 14:09:39 <pkoniszewski> PaulMurray: its already fixed 14:09:43 <eliqiao> tdurakov: cool. 14:09:59 <PaulMurray> pkoniszewski, was that the live migration job or the review at fault? 14:10:27 <tdurakov> PaulMurray, what ps number? 14:10:48 <PaulMurray> 25 14:11:48 <PaulMurray> pkoniszewski, from your comment: 14:11:51 <PaulMurray> I'm currently investigating it, but so far it looks like this fail is not related to my changes. Tests without NFS worked correctly: 14:11:51 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration [66.216203s] ... ok 14:11:51 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration_paused [19.978373s] ... ok 14:11:52 <PaulMurray> With NFS one test passed and another one failed: 14:11:56 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration [15.971409s] ... FAILED 14:11:59 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration_paused [18.673453s] ... ok 14:12:02 <PaulMurray> Let's see the result on BLM with config drive patch 14:12:19 <pkoniszewski> PaulMurray: looks to me like it was a test issue, but Matt found another issue in another patch, no idea right now which one caused this 14:12:36 <pkoniszewski> I can see in my environment that everything works fine now 14:12:42 <pkoniszewski> so I pushed another patch 14:13:05 <PaulMurray> I saw that the config drive patch passed 14:13:42 <PaulMurray> so as we are discussing it..... 14:13:48 <PaulMurray> #topic Priority reviews 14:14:02 <PaulMurray> pkoniszewski, how is that series going? 14:14:12 <PaulMurray> you got one merged 14:15:05 <pkoniszewski> There is one concern from Daniel, but its just naming issue in old dicts, I need to dump data and respond to his review 14:15:51 <pkoniszewski> config drive patch is ready 14:16:01 <PaulMurray> #link https://review.openstack.org/#/q/topic:bp/block-live-migrate-with-attached-volumes 14:16:33 <PaulMurray> ok - we'll keep looking to review it 14:16:49 <PaulMurray> The next one I wanted to call out was 14:17:02 <PaulMurray> #link https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration 14:17:42 <pkoniszewski> so I need to go through code carefuly to see how andrearosa's proposition would work 14:17:55 <pkoniszewski> probably they are right and we can move logic down to drivers 14:18:09 <andrearosa> pkoniszewski: feel free to ping me if my comments are not clear 14:18:15 <pkoniszewski> my idea was to keep this at a high level (compute api) to be able to throw correct instance action 14:19:18 <pkoniszewski> this is async so there is no other option to notify about which action is taken at a lower level, or I don't know something about nova's notifications 14:23:42 <pkoniszewski> PaulMurray timed out :< 14:23:48 <eliqiao> :( 14:24:09 <pkoniszewski> let's move forward, not sure we will be able to finish the meeting or add links and so on 14:24:18 <pkoniszewski> #topic Bugs 14:24:29 <pkoniszewski> nah, it doesn't listen :< 14:24:57 <pkoniszewski> so just a kindly reminder that we are tracking bugs in etherpad - https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking 14:25:15 <pkoniszewski> most of them need reviews 14:26:08 <eliqiao> yeah, hard work 14:26:10 <pkoniszewski> there is a new series to deprecate migration flags config 14:26:29 <kashyap> pkoniszewski: Randomly chiming in, yeah, it avoids a lot of confusion 14:26:39 <eliqiao> pkoniszewski: I see, Dansims gives +2 on them. 14:27:01 <kashyap> Because, there are a lot of flags and tunables that libvirt allows, unless one really has tested all them, it's hard to remember which combination is suitable. 14:27:21 <pkoniszewski> so every review right now is appreciated, we are getting close to midcycle, I believe that we will be able to work on some changes and merge some of them at the midcycle 14:27:42 <eliqiao> pkoniszewski: seems they are all in high prority, get cores' eyes already. 14:28:17 <pkoniszewski> kashyap: exactly, this series will remove a lot of confusions around LM configuration and will give nova more flexibility to configure it on correctly 14:28:55 <pkoniszewski> a lot of patches are very close 14:29:20 <kashyap> Yeah, the bug that triggered it all - https://bugs.launchpad.net/nova/+bug/1441054 14:29:21 <openstack> Launchpad bug 1441054 in OpenStack Compute (nova) "live-migration --block-migrate fails with default libvirt flags" [Medium,Incomplete] - Assigned to Mathieu Rohon (mathieu-rohon) 14:29:54 <pkoniszewski> andrearosa: do you know what Paul wanted to say about midcycle? 14:29:55 <kashyap> In the time the bug was reported, and now, the said issue seems to be fixed there. 14:30:15 <pkoniszewski> it starts in a less than 2 weeks 14:30:27 <andrearosa> pkoniszewski: nope he is not in the office, I guess he just want to start to discuss about some topics we want to put in the agenda for the meetup 14:31:24 <pkoniszewski> okay, so here is the etherpad for midcycle - https://etherpad.openstack.org/p/mitaka-nova-midcycle 14:32:16 <eliqiao> pkoniszewski: thx, we should file something for LM 14:32:22 <pkoniszewski> live migration will be discussed on Tuesday 14:32:44 * andrearosa is trying to reach PaulMurray on different channels 14:32:46 <PaulMurray> Hi, am I back in? 14:32:49 <pkoniszewski> PaulMurray: is bad 14:32:52 <pkoniszewski> back * 14:32:52 <pkoniszewski> sorry 14:32:53 <pkoniszewski> :D 14:33:10 <eliqiao> pkoniszewski: will you attend mid-cycle? 14:33:14 <PaulMurray> Seems I got disconnected 14:33:15 <pkoniszewski> PaulMurray: we moved forward to midcycle, just started this topic and mentioned about etherpad 14:33:30 <PaulMurray> cool 14:33:46 <pkoniszewski> eliqiao: yes, I will participate 14:34:20 <eliqiao> pkoniszewski: cool. 14:34:28 <PaulMurray> tdurakov, did you get your visa? 14:34:45 <tdurakov> PaulMurray, yep, will attend mid-cycle 14:35:05 <PaulMurray> great - there's a reasonable turn out all round 14:35:24 <PaulMurray> there will be several people who work on libvirt and qemu there 14:35:43 <PaulMurray> so its a good time to bring up anything needed there 14:35:49 <tdurakov> great! 14:36:41 <tdurakov> pkoniszewski, could we discuss your patch and ci failure after this meeting? 14:36:55 <pkoniszewski> tdurakov: sure! 14:37:07 <PaulMurray> There are a few virt layer items at the bottom of our eatherpad 14:37:20 <PaulMurray> please add there if there is anything that I haven't included 14:37:43 <PaulMurray> see: https://etherpad.openstack.org/p/mitaka-live-migration 14:37:56 <PaulMurray> Live Migration Libvirt / qemu Work Items 14:38:13 <PaulMurray> This is to do with lining up work we may depend on in the future 14:38:27 <PaulMurray> the people coming may be able to take some on 14:39:10 <PaulMurray> #topic open discussion 14:39:23 <PaulMurray> Anything anyone wants to discuss? 14:39:47 <pkoniszewski> I see that there is series about deprecating migration flags in open discussion 14:40:14 <pkoniszewski> is owner of the series there? 14:40:26 <PaulMurray> I don't think so 14:40:51 <PaulMurray> its markmc 14:42:00 <PaulMurray> pkoniszewski, did you follow the thread? 14:42:04 <tdurakov> do we really need such constraint for operators? 14:42:08 <pkoniszewski> yes, I did 14:43:06 <pkoniszewski> tdurakov: what do you mean? 14:43:27 <eliqiao> hiden flgs from admin 14:43:35 <tdurakov> eliqiao, yep 14:44:05 <pkoniszewski> well, I don't think that there is reason to expose all these flags that we currently expose to operators 14:44:21 <pkoniszewski> we allow them, e.g., to misconfigure nova 14:44:22 <eliqiao> I commented to have a log for flags. 14:44:49 <tdurakov> from my sight it should be enough to have warning in logs 14:44:55 <eliqiao> markmc told it's hard to convert from bit to string.. and also havs warings.. 14:45:16 <eliqiao> tdurakov: hmm.. that would be not easy to findout all flags 14:45:24 <tdurakov> e.x. what about post-copy mode for live-migration? 14:46:12 <pkoniszewski> depends how it will impact performance of different workloads 14:46:35 <pkoniszewski> imo there should be another way to trigger post copy live migration, but I don't know much about its impact yet 14:46:45 <davidgiluk> pkoniszewski: What would you like to know 14:46:46 <eliqiao> Add a new switch flag to indicate to use post-copy or not, that what I can image onl. 14:46:48 <PaulMurray> ok - got back in 14:47:07 <tdurakov> pkoniszewski, afair we need to provide flag 14:47:33 <tdurakov> so if wee hide flags from operator - we need to provide some flag in cli/rest-api 14:47:50 <tdurakov> do we really need this?) 14:47:57 <PaulMurray> #chair andrearosa 14:47:58 <openstack> Current chairs: PaulMurray andrearosa 14:48:00 <eliqiao> ops. a new REST api ? 14:48:10 <pkoniszewski> eliqiao: just a flag 14:48:31 <tdurakov> eliqiao, if there is no way to change flags you need to change api 14:48:46 <pkoniszewski> tdurakov: I just don't know in which case you would want to trigger only post-copy live migrations 14:48:58 <pkoniszewski> I'm not a fan of all these static configurations in nova.conf 14:49:13 <eliqiao> pkoniszewski: add new options to live-migration CLI to passing flags, it make some scense to me. 14:49:15 <pkoniszewski> they are way too static for different kind of workloads and different sizes of VMs 14:49:25 <tdurakov> eliqiao, yep, new options 14:50:16 <tdurakov> but we recently going to remove disk-over-commit option from cli, no? 14:50:29 <eliqiao> tdurakov: cool, that's a good suggestion I think, (but maybe some one will say we don't need to expose low level details) 14:50:31 <pkoniszewski> its not used anymore afaik 14:50:47 <eliqiao> tdurakov: yeah, that's what I am working on 14:51:02 <tdurakov> it looks inconsistent imho 14:51:22 <tdurakov> to hide one flag and then add new one 14:51:27 <pkoniszewski> davidgiluk: I would like to know how post-copy affects databases and comparison to pre-copy 14:51:39 <tdurakov> but yes, this need to be discussed more 14:52:20 <davidgiluk> pkoniszewski: OK, PM for a discussion about performance and stuff, but the 'howto use' it is you must select it before the start of migration 14:52:42 <davidgiluk> pkoniszewski: The simple answer is that with postcopy the migration always completes, irrespective of the workload 14:53:04 <eliqiao> davidgiluk: do you konw the progress post-copy migration supported in libivrt, I don't find any patch about it. 14:53:14 <pkoniszewski> that's true, it ends in a finite time 14:53:21 <tdurakov> davidgiluk, tradeoff is no way to rollback it, right? 14:53:32 <eliqiao> yes, can not revert 14:53:34 <davidgiluk> eliqiao: jdenemar is currently tidying it up for posting, I need to sort out a couple of things for him on the qemu side 14:53:40 <PaulMurray> also slower vm if have to fetch 14:53:43 <pkoniszewski> tdurakov: yes, and in case of network failure you will need to reboot VM 14:53:54 <PaulMurray> alot 14:53:58 <davidgiluk> tdurakov: Right, once it's in postcopy mode you can't abandon the migrate since the destination has started writing to disk and sending packets etc 14:54:34 <davidgiluk> PaulMurray: Right, although normally the postcopy phase is only a short part of it, so that finishes pretty quickly 14:55:44 <eliqiao> davidgiluk: seems we are far away to support post-migration since it's not merged in libvirt yet? 14:56:03 <tdurakov> from my sight post-copy is one of examples where we need to decide, re we going to hide flags from conf and add extra cli/api params, or leave conf as-is 14:56:24 <davidgiluk> eliqiao: not merged in libvirt yet, but reasonably close; there's also a demo that Umea uni have of an implementation on openstack 14:56:37 <pkoniszewski> eliqiao: per my discussion with Daniel its very close and it's real to get it done in N/O-release of OpenStack 14:57:03 <eliqiao> pkoniszewski: do you mean in libvirt? 14:57:15 <pkoniszewski> tdurakov: we can just say in nova.conf that all live migrations will be post-copy or all live migrations will be pre-copy 14:57:19 <pkoniszewski> tdurakov: it's way too static 14:57:29 <davidgiluk> pkoniszewski: Yeh that's way too static 14:57:48 <pkoniszewski> tdurakov: so I strongly prefer to remove these flags from nova.conf, we need better approach 14:57:51 <andrearosa> pkoniszewski: you meant we can't 14:58:01 <pkoniszewski> andrearosa: yes, I meant that we can :D 14:58:04 <pkoniszewski> can't... 14:58:07 <tdurakov> :) 14:58:12 <andrearosa> ok, thanks 14:58:21 <eliqiao> almost done, sorry to interupt, can anyone help to review this topic #link https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/making-live-migration-api-friendly 14:58:25 <PaulMurray> We are coming to the end of time 14:58:37 <pkoniszewski> PaulMurray: just one more question 14:58:44 <PaulMurray> yes 14:58:51 <pkoniszewski> PaulMurray: will we discuss LM agenda for midcycle before midcycle? 14:58:55 <eliqiao> PaulMurray: that one is not discuess last week(I missed), just hight light 14:58:57 <pkoniszewski> like, next week or on a mailing list? 14:59:19 <PaulMurray> I tried to discuss it today when I got dropped 14:59:50 <PaulMurray> I'll put an eatherpad link on the live migration link 15:00:00 <PaulMurray> and we can list items there 15:00:05 <PaulMurray> then go over next week 15:00:09 <PaulMurray> sound ok? 15:00:14 <PaulMurray> I'll send an email 15:00:25 <pkoniszewski> sounds good! 15:00:27 <pkoniszewski> thanks 15:00:32 <PaulMurray> #endmeeting