13:00:58 <alexpilo_> #startmeeting hyper-v 13:00:59 <openstack> Meeting started Wed Jan 6 13:00:58 2016 UTC and is due to finish in 60 minutes. The chair is alexpilo_. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:01:03 <openstack> The meeting name has been set to 'hyper_v' 13:01:14 <alexpilo_> Morning! 13:01:17 <primeministerp> morning! 13:01:20 <claudiub> o/ 13:01:27 <sagar_nikam> Hi All, Happy new year !!!! 13:01:35 <claudiub> happy new year. :) 13:01:43 <alexpilo_> Happy New Year!! :-) 13:02:08 <atuvenie> o/ 13:02:27 <alexpilo_> up your "hands" folks, let's see who do we have today 13:02:38 <alexpilo_> lpetrut? 13:02:59 <alexpilo_> sagar_nikam: is sonu joining us? 13:02:59 <abalutoiu> o/ 13:03:10 <itoader> o/ 13:03:33 <sagar_nikam> yes 13:03:43 <sagar_nikam> he said he will join 13:03:54 <lpetrut> o/ 13:04:01 <alexpilo_> he sent a patch yesterday, we reviewed quickly to be able to discuss it today 13:04:09 <sagar_nikam> sure 13:04:13 <alexpilo_> but we can start with the other topics 13:04:20 <sagar_nikam> kvinod: Sonu joining ? 13:04:21 <alexpilo_> #topic FC support 13:04:39 <alexpilo_> lpetrut: any updates? 13:04:59 <kvinod> Sonu will join 13:05:02 <lpetrut> yep, so, all the os-win FC related patches have merged 13:05:42 <lpetrut> the CI and unit tests should be all green on the nova patches once we do an os-win version bump, so we should try to get those merged as soon as possible 13:06:17 <alexpilo_> sweet 13:06:26 <lpetrut> sagar: have you guys looked over those patches? 13:06:38 <lpetrut> sagar: it would be great if you could give it a try on your environment 13:06:45 <sagar_nikam> sure 13:06:56 <sagar_nikam> i will have somebody check it 13:07:21 <sagar_nikam> can you add me as reviewes 13:07:23 <lpetrut> thanks a lot, please let me know how it works for you 13:07:24 <lpetrut> sure 13:07:27 <sagar_nikam> reviewer 13:08:55 <alexpilo_> #link https://review.openstack.org/#/c/258617/ 13:08:57 <lpetrut> I know this is unrelated to FC, but here's the iSCSI refactoring patch: https://review.openstack.org/#/c/249291/ it's currently marked as WIP because we're currently working on the unit tests, but it would be great if you guys could give us some feedback on this as well 13:09:29 <lpetrut> most probably we'll get this merged today 13:09:30 <alexpilo_> there's still a -1 by walter here 13:09:40 <alexpilo_> anybody knows his IRC nic? 13:10:01 <alexpilo_> hemna, afaik 13:10:10 <lpetrut> yep, basically, he requested a few more info to be provided. such as whether multipath should be used, and the os_type and platform 13:10:12 <lpetrut> yep 13:10:12 <sagar_nikam> lpetrut: this has the fix for CHAP ? 13:10:12 <alexpilo_> his not on this channel anyway 13:10:35 <alexpilo_> lpetrut: let's stick to FC 13:10:35 <sagar_nikam> ok 13:10:40 <claudiub> hemna is on the #openstack-nova channel. 13:10:46 <alexpilo_> we'll switch topic to iSCSI soon 13:10:47 <lpetrut> my bad, sure 13:10:49 <sagar_nikam> and the fix for rescan ? 13:10:57 <alexpilo_> sagar_nikam: ^ :-) 13:11:02 <sagar_nikam> ok 13:11:34 <lpetrut> I guess I'm a bit over enthusiastic about this one :) 13:11:48 <lpetrut> any other questions related to FC? 13:12:00 <alexpilo_> there are no HP reviews on all Nova FC patches 13:12:02 <sagar_nikam> not from my side 13:12:10 <alexpilo_> only the last one in the chain 13:12:14 <sagar_nikam> kurt ? 13:12:18 <alexpilo_> we need reviews on all of them: 13:12:31 <alexpilo_> #link https://review.openstack.org/#/c/258614 13:12:32 <sagar_nikam> i thought he did the reviews 13:12:43 <alexpilo_> #link https://review.openstack.org/#/c/258615 13:12:52 <alexpilo_> #link https://review.openstack.org/#/c/260980 13:12:55 <lpetrut> just a side note: I'll add the vHBA support ASAP 13:13:07 <alexpilo_> just a quick not on vHBA 13:13:33 <alexpilo_> we have two separate options, passthrough and vHBA 13:14:06 <alexpilo_> vHBA is a more flexible feature and easier to implement as it doesn't require all the hassle required by passthrough 13:14:15 <alexpilo_> especially for live migration 13:14:17 <sagar_nikam> yes remember and i think we decided to first implement passthrough 13:14:25 <alexpilo_> but it has some very hard limitations: 13:14:34 <alexpilo_> 1) no boot from volume 13:14:53 <alexpilo_> 2) guest OS support requirement 13:15:00 <lpetrut> so we'll implement both, maybe passthrough as default, letting the user opt for vHBA by setting the bus type to FC 13:15:11 <alexpilo_> so for this reason it will be implemented separately as soon as the current patches merge 13:15:37 <sagar_nikam> ok 13:15:39 <alexpilo_> Im just recapping this here to make sure we keep focus on passthrough 13:16:04 <sagar_nikam> i think even kurt had suggested pass through... if i remember right 13:16:15 <alexpilo_> for the rest, priority is on getting Cloudbase and HP +1s 13:16:34 <alexpilo_> so that we can move the patches to the Nova etherpad queue 13:16:42 <sagar_nikam> alexpilo_: we have 3 patches ? 13:16:48 <sagar_nikam> that needs review ? 13:16:53 <alexpilo_> 4 13:17:17 <alexpilo_> the last one is the first link I posted (the one with hemna's review) 13:17:25 <sagar_nikam> 614,815 abd 980 13:17:40 <sagar_nikam> ok 13:17:40 <alexpilo_> 258617 13:18:00 <alexpilo_> anything else you'd like to add on FC? 13:18:08 <alexpilo_> sagar_nikam, lpetrut 13:18:16 <lpetrut> not on my side 13:18:30 <sagar_nikam> no, we will start the review, i think we will need Kurt to review all 4 13:18:39 <alexpilo_> sweet, tx! 13:18:42 <sagar_nikam> will let him know 13:18:55 <alexpilo_> #topic iSCSI 3par support 13:19:09 <alexpilo_> now is the time to get wild on iSCSI :-) 13:19:23 <primeministerp> buckwild 13:19:24 <sagar_nikam> 2 issues that we have seen in 3par iscsi, CHAP and rescan 13:19:29 <lpetrut> heh, so, lun rescanning is in place now 13:19:48 <sagar_nikam> lpetrut: already merged ? 13:19:58 <sagar_nikam> lun rescan 13:20:03 <lpetrut> nope, it's the patch I mentioned before 13:20:22 <lpetrut> passing CHAP credentials when logging in portals: not yet. As this was causing issues with other backends, I wanted to test this first. 13:20:43 <alexpilo_> sagar_nikam: this brings up the topic we discussed on Monday 13:21:00 <alexpilo_> using the FC 3par array for iSCSI testing as well 13:21:07 <primeministerp> *nod* 13:21:30 <primeministerp> We need hba's for the array 13:21:45 <alexpilo_> so, as soon as we have the additional HW in place, we will start testing this ASAP 13:22:17 <alexpilo_> primeministerp would you like to add something here? 13:22:22 <primeministerp> sagar_nikam, someone on your side was looking into that 13:23:37 <primeministerp> basically we're going to need the hba to add the iscsi functionality to the array 13:23:49 <primeministerp> we may need additional licensing as well 13:24:21 <sagar_nikam_> lpetrut: did you see kmbharath: patch 13:24:25 <sagar_nikam_> on CHAP 13:24:33 <sagar_nikam_> he had a fix for it many months back 13:24:46 <primeministerp> hmmm 13:25:05 <primeministerp> hopefully he'l reconnect 13:25:32 <lpetrut> yep, but I was thinking whether we can save time by not logging in the portal twice (once without CHAP creds, once with CHAP creds), and maybe use a flag on the Cinder side. Like "portals_requires_chap_auth" or something similar 13:25:33 <sagar_nikam_> i am connected 13:25:45 <primeministerp> ;) 13:26:00 <sagar_nikam_> kmbharath: your comments ? 13:26:11 <sagar_nikam_> on lpetrut: suggestion 13:26:12 <lpetrut> we could just push this into the volume connection info 13:26:25 <alexpilo_> sagar_nikam_: as previously discussed we need to ensure that this patch wont cause issues to other backedends 13:26:33 <kmbharath> Yes agreed, if we have a flag and do it , it would be better 13:26:43 <alexpilo_> as the 3par one seems to be the only one with this requirement 13:26:51 <sagar_nikam_> ok 13:26:56 <kmbharath> we had tested it on HP LeftHandnetwork and 3par earlier 13:27:02 <alexpilo_> the option is probably the only way to do that 13:27:08 <lpetrut> great, do you know by any change what backends require this? Is it just 3PAR, are all 3PAR backends requiring this? 13:27:21 <alexpilo_> but: 13:27:30 <sagar_nikam_> from our tests, only 3par 13:27:44 <alexpilo_> what if we have 2 backends, e.g. a 3par and another 3rd party one? 13:27:47 <sagar_nikam_> LHN/VSA worked without any change 13:27:57 <alexpilo_> one expecting portal logins and the other one failing? 13:28:22 <alexpilo_> the flag won't help, as it would force login on all of them 13:28:31 <lpetrut> no, why? 13:28:43 <alexpilo_> becuase the other backend would fail 13:29:05 <alexpilo_> an option (I believe suggested by lpetrut) would be to try the login and silently continue if it fails 13:29:29 <lpetrut> the other backend would not set this flag, and we would not use CHAP creds when logging in the portal, so it should not fail 13:29:34 <lpetrut> please correct me if I got something wrong 13:29:52 <kmbharath> can't we do check for backend type.... 13:29:55 <alexpilo_> that would require changes on the cinder side 13:30:09 <kmbharath> because its only 3par what we had seen needs the portal login 13:30:34 <sagar_nikam_> lpetrut: who will set the flag ? every cinder driver ? 13:30:51 <lpetrut> this can be optional, but the driver would set it when providing the connection info 13:30:59 <alexpilo_> or we could simply have a list of backends in the Nova driver with the drivers requiring portal login 13:31:08 <lpetrut> also, the connection info does not include the backend type at the moment 13:31:21 <alexpilo_> by making it an option, this could be configurable 13:31:35 <alexpilo_> lpetrut: d'oh, that'sa blocker 13:31:41 <sagar_nikam_> alexpilo_:this option looks better 13:32:15 <alexpilo_> lpetrut sagar_nikam_: let's bring this back to the witheboard and sync again next week 13:32:19 <sagar_nikam_> having list of backends in nova is better than cinder sending it in connection_info 13:32:31 <kmbharath> the target_iqn in connection info could help us to identify the backend 13:32:52 <lpetrut> kmbharath: umm, is this reliable enough? 13:33:23 <lpetrut> alexpilo_: ok, we can talk about this later so that we don't block the meeting for this 13:33:29 <sagar_nikam_> lpetrut: i think the iqn had 3par in it 13:33:41 <sagar_nikam_> ok, let move to next topic 13:33:46 <lpetrut> sagar_nikam: sure, just wanted to make sure that this happens all the time 13:33:54 <sagar_nikam_> can we have some networking discussion 13:33:56 <kmbharath> yes we had seen it everytime 13:34:04 <sagar_nikam_> sonu: is here i thinl 13:34:07 <alexpilo_> did sonu join? 13:34:17 <Sonu> I am listening.. 13:34:21 <sagar_nikam_> i saw him joining 13:34:24 <alexpilo_> great 13:34:39 <primeministerp> sagar_nikam: yes 13:34:44 <alexpilo_> #topic SGR RPC patch 13:34:55 * alexpilo_ fetches link... 13:35:27 <alexpilo_> #link https://review.openstack.org/#/c/263865/ 13:35:41 <alexpilo_> first, thanks Sonu for the patch 13:36:07 <alexpilo_> did you see claudiub's review? 13:36:22 <Sonu> I am reviewing the same. 13:36:33 <Sonu> Thanks for the comments. I will fix them and re-post 13:36:34 <alexpilo_> we prioritized it right away to be sure we could talk about it today 13:36:45 <Sonu> Thanks for that 13:36:52 <alexpilo_> claudiub Sonu: anything to add? 13:37:10 <Sonu> This patch is dependent on https://review.openstack.org/#/c/240577/ 13:37:31 <Sonu> there was a bug in base security groups driver, which I have fixed and Review is in progress. 13:37:36 <claudiub> could you then add a Depends-On: <change-id> here? 13:37:43 <alexpilo_> btw we need a BP for this 13:37:43 <Sonu> I will mark the dependency 13:37:53 <claudiub> cool ty. :) 13:37:55 <Sonu> Got it. I will work on the same. 13:38:17 <alexpilo_> sweet 13:38:28 <alexpilo_> moving to a broader topic: 13:38:38 <alexpilo_> #topic networking-hyperv improvements 13:39:00 <alexpilo_> let me share some of the design aspects related to this agent 13:39:11 <alexpilo_> there are a few improvements going on 13:39:57 <alexpilo_> we already talked about this a while back, when we talked about the multiprocessing patches that HP sent for SGR 13:40:22 <alexpilo_> we are currently finalizing the results, which will turn into a BP 13:40:41 <alexpilo_> 1) drop the port discovery loop and replace it with WMI events 13:41:09 <alexpilo_> 2) use PyMI 13:41:31 <alexpilo_> 3) replace associator queries with direct WQL queries 13:41:58 <alexpilo_> 4) parallelize all the things :) 13:42:28 <alexpilo_> this includes in particular ACL which are a big bottleneck 13:42:55 <kvinod> alexpilo_ : are you talking about new approach of multiprocessing 13:43:00 <alexpilo_> unlike the Juno patch that HP is using, this doesnt require multiprocessing 13:43:27 <kvinod> is this blueprint on multiprocessing going to be different from what we posted? 13:43:30 <alexpilo_> PyMI (unlike the old WMI + pywin32) is designed to work with multiple threads 13:43:55 <alexpilo_> kvinod it's quite different, especially in the implementation 13:44:27 <alexpilo_> this is the reason why it has been kept on hold 13:44:45 <kvinod> ok, then are you saying that HP's patch sets are not required and will not get merged? 13:45:01 <sagar_nikam_> alexpilo_: how are the tests in a scale environment with this new approach ? 13:45:04 <alexpilo_> surely not in the current status 13:45:13 <claudiub> so, I've tested the patchsets you've sent and there are a couple of issues. 13:45:17 <alexpilo_> kvinod: ^ 13:45:39 <Sonu> One biggest problem that we had encountered during scale was, too many port updates introducing delays in processing of new port additions. 13:45:44 <alexpilo_> sagar_nikam_: we are testing with Rally on scale 13:45:52 <claudiub> there are 2 big issues atm: 1. logging doesn't work, apparently, I've just noticed 30 minutes ago 13:46:11 <Sonu> thats one reason we separated addition of ports into a different workers scheduled on another CPU 13:46:34 <alexpilo_> Sonu: you can just use threads for that, no need for multiple processes 13:46:35 <claudiub> with the workers patch, logging is only done to stdout, the neutron-hyperv-agent.log is empty 13:47:02 <claudiub> and second, it seems that the agents die randomly during rally. 13:47:43 <claudiub> they freeze, leading to missing to report the alive state, leading to failing to spawn vms, as the neutron agents are considered to be dead and the neutron ports couldn't be bound 13:48:24 <Sonu> hmm thats a news :) 13:48:25 <kvinod> claudiub : we already noticed the logging issue and we solved it by making child process send message to parent process about port binding success or failure and parent process will log into log file 13:48:45 <alexpilo_> kvinod: that is unnecessary work 13:49:07 <claudiub> and thirdly, if there is any issue in binding a port over and over and over again, the neutron-hyperv-agent process will consume the whole cpu. 13:49:48 <kvinod> alexpilo_ : we did it that way due to limitation in logging framework as it doesnot works for multiprocessing 13:49:54 <alexpilo_> sorry for trinmming the discussion, as we have only 10' left 13:50:09 <claudiub> kvinod: yeah, i've seen that. if i start the process manually and see the std, i can see what happens in the child processes, including traces and so on. but there's nothing in the log file. 13:50:21 <claudiub> stdout* 13:50:36 <alexpilo_> the idea of parallel execution in the agent is of course the common goal here 13:50:48 <alexpilo_> Python's multiprocessing brings a lot of unnecessary drawbacks and there's no reason to use it 13:51:14 <alexpilo_> threads / green threads work perfectly fine as long as the underlying WMI calls are non-blocking 13:51:38 <alexpilo_> (otherwise we'd hit the GIL issue, which is I guess why you opted for multiprocessing) 13:52:14 <Sonu> yes you got it right 13:52:36 <alexpilo_> the discussion is anyway much broader, which is why this BP is taking a while 13:52:39 <claudiub> i'm currently trying out native threads, to see how it's going with those. 13:52:52 <Sonu> Vinod had tried all such possibilities 13:52:57 <sagar_nikam_> Sonu: Can you review the new approach if a patchset is available 13:53:13 <sagar_nikam_> and check how it will work 13:53:15 <Sonu> he can help you with his findings and observations 13:53:17 <claudiub> i haven't uploaded the native threads patch yet 13:53:23 <sagar_nikam_> based on our scale tests run on Juno 13:53:26 <alexpilo_> Sonu kvinod: that's why we wrote PyMI ;) 13:53:32 <Sonu> great 13:53:42 <alexpilo_> the main aspect here, is that the ACL API are simply terrible and no matter how you look at them they dont scale 13:53:43 <Sonu> I will have a look at the new patch set and try 13:54:02 <sagar_nikam_> claudiub: patchset not available yet ? 13:54:10 <kvinod> ok, please upload your patches 13:54:18 <claudiub> sagar_nikam_: native threads, not yet, still working on it. 13:54:23 <alexpilo_> so all this parallelization work is improving a bit the situation, but a more drastic approach will be needed 13:54:40 <alexpilo_> kvinod: expect the patches sometimes next week 13:55:19 <alexpilo_> 1) the OVS driver will become the preferred option as soon as conntrack will be available on our Windows port 13:55:29 <sagar_nikam_> sonu: kvinod: you had seen issues with security groups as well 13:55:35 <alexpilo_> as this is required for an SGR as well 13:55:51 <Sonu> have you seen the development - https://review.openstack.org/#/c/249337 13:56:05 <alexpilo_> 2) we're evaluating a complete rewrite of the ACL WMI API 13:56:27 <alexpilo_> Sonu: tep, that's what I'm referring to 13:56:48 <Sonu> this is OVS firewall being done once OVS conntrack is available 13:56:49 <alexpilo_> we need conntrack on Windows for that to work 13:57:09 <alexpilo_> which is our main goal for OVS 2.6 13:57:13 <primeministerp> Sonu: could we have an offline discussion about the HP networking hardware that supports protocol accelleration, I want to see if what it would take to add OVS and Native vswitch testing on the appropriate network hba, the ones we currently have in the HP 3Par ci only support accellerated iSCSI 13:57:13 <Sonu> got it. 13:57:34 <alexpilo_> 3 minutes to go 13:57:47 <Sonu> sure primeministerp 13:57:55 <Sonu> Sagar mentioned about it 13:58:01 <sagar_nikam_> primeministerp: sure i will connect you and Sonu: on this topic 13:58:02 <primeministerp> Sonu, awesome 13:58:07 <alexpilo_> changing topic, if you'd like to go on with this topic, please let's move to #openstack-hyper-v 13:58:10 <primeministerp> thanks sagar_nikam_ 13:58:17 <alexpilo_> #topic PyMI 13:58:32 <alexpilo_> so PyMI is feature complete for all OpenStack use cases 13:58:42 <alexpilo_> it has been tested under heavy load with Rally 13:58:45 <sagar_nikam_> that is a very good news 13:58:59 <primeministerp> woot 13:59:12 <alexpilo_> sagar_nikam_: do you think you could test it in your environments? 13:59:21 <sagar_nikam_> yes, we will 13:59:24 <alexpilo_> we tested, kilo, liberty and mitaka (master) 13:59:36 <sagar_nikam_> i need to get some slots from scale team to test this 13:59:58 <alexpilo_> all CIs are now switching to it, including nova, neutron, networking-hyperv, compute-hyperv and the cinder ones 14:00:08 <sagar_nikam_> most of our environments have also move to Liberty 14:00:15 <sagar_nikam_> need to find Juno 14:00:20 <sagar_nikam_> for testing pyMI 14:00:30 <alexpilo_> perfect tx! 14:00:37 <alexpilo_> time's up! :) 14:00:40 <alexpilo_> #endmeeting