16:00:09 #startmeeting Cinder 16:00:10 Meeting started Wed Apr 5 16:00:09 2017 UTC and is due to finish in 60 minutes. The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:14 The meeting name has been set to 'cinder' 16:00:29 hi 16:00:29 Hello 16:00:32 Courtesy ping: dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang1 tbarron scottda erlon rhedlind jbernard _alastor_ bluex karthikp_ patrickeast dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr mtanino yuriy_n17 karlamrhein diablo_rojo jay.xu jgregor baumann rajinir wilson-l reduxio wanghao thrawn01 16:00:32 chris_morrell watanabe.isao,tommylikehu mdovgal ildikov wxy viks ketonne abishop sivn breitz 16:00:39 hi 16:00:40 o/ 16:00:40 hi! 16:00:41 hi 16:00:43 o/ 16:00:44 hi 16:00:45 Chewbacca 16:00:49 Hi 16:01:01 Will give everyone a minute to join. Sean is is Tokyo so I am manning the ship today. 16:01:14 hi 16:01:14 lord help us 16:01:25 :P 16:01:25 hemna: Thanks for the vote of confidence. :-p 16:01:27 hi 16:01:35 hi 16:01:48 tbarron: Wow! Thanks for joining. 16:02:07 jungleboyj: I usually show up; don't feel too special. 16:02:13 Hi 16:02:18 tbarron: Awww. Ok. 16:02:36 Ok, we have a good showing. Lets get started. 16:02:47 #topic Announcements: 16:03:11 #link https://etherpad.openstack.org/p/cinder-spec-review-tracking 16:03:23 o/ 16:03:34 Weekly reminder of review tracking priorities for Pike. Please take a look at those and keep up with reviews. 16:03:47 Also, tomorrow is the cut-off for stable/newton patches. 16:03:57 I went through a bunch of things that were out there yesterday. 16:03:59 o/ 16:04:02 I will do another round today. 16:04:15 i just posted https://review.openstack.org/#/c/453697/ for Newton, seems like a good thing to get in 16:04:18 If there is anything I am missing please let me know and I will get to it todya or tomorrow. 16:04:35 eharney: Tab opened. 16:04:54 Any other ones that need immediate attention? 16:05:16 i've been eyeballing the list, it's mostly driver fixes 16:05:17 Ok, if others come up ping me in IRC please. 16:05:28 jungleboyj: we have a few newton backports that need a 2nd +2 16:05:42 https://review.openstack.org/#/c/452234/ 16:05:44 eharney: I have been pretty relaxed about letting those in. Figure if the maintainers want them...Ok. 16:05:55 https://review.openstack.org/#/c/453163/ 16:06:03 eharney: ^^ 16:06:08 https://review.openstack.org/#/c/451813/ 16:06:25 stable cores, please help! 16:06:48 jgriffith: If you and/or eharney could look at those it would be appreciated. 16:06:56 jungleboyj sure 16:07:02 jgriffith: Thank you! 16:07:13 thanks! 16:07:17 we can figure this out in #openstack-cinder later, but i dunno why "add replication" is a backport... 16:07:34 eharney: I will fill you in on that one. Look closer. ;-) 16:07:40 eharney: I can explain to you later, just a small portion of that 16:07:43 A bug fix that went in with that change. 16:07:55 I had them same heartburn when I saw it the other day. 16:08:05 Ok ... Anything else on this topic? 16:08:27 Ok, moving on. 16:08:38 #topic Updates from Oslo Meeting: 16:08:57 #link https://review.openstack.org/328692 16:09:07 The review above may break the gate for Cinder. 16:09:43 The change set the method method set_override's parameter enforce_type=True by default, this enforces developer to write valid test cases about config options. 16:10:08 When I get back next week I want to try to pull the change down and see if it breaks us. 16:10:22 Wanted to make everyone aware of the change if there were any immediate concerns. 16:10:34 it won't be allowed by upper-constraints requirements bounds until it passes w/ our unit tests, right? 16:10:50 eharney: I believe that is true. 16:11:21 I am not sure how much work will be involved to fix things up if it does break us. I think it is mostly busy work that I can tackle. 16:11:43 None-the-less, wanted to share the fact that that change is coming. 16:12:18 Also anohter note from the Oslo team. 16:12:33 With all the discussion and proposals around messaging ... 16:13:17 I wanted to make note that Ken Giusti is going to be doing a presentation on the different messaging backends and how to set up systems to run with multiple messaging queues. 16:13:34 May want to look for that in Boston. Sounds interesting. 16:13:58 So, that was what I had there. 16:14:21 #topic Extend Volume Discussion: 16:14:29 Now things get interesting. :-) 16:14:43 tommylikehu_: Has brought this functionality back up. 16:14:44 =) 16:14:53 :) 16:15:03 I am glad as it kind-of fell off the radar with other things going on in the community. 16:15:18 So, tommylikehu brought it up and mgagne has been helping out. 16:15:38 mgagne, have you tested out the os-brick extend_volume stuff yet ? 16:15:40 mgagne: Shared with following solution with mriedem : 16:15:47 #link https://gist.github.com/mgagne/9402089c11f8c80f6d6cd49f3db76512 16:16:05 mriedem: Was willing to consider the solution. 16:16:10 I'd like to verify that it still works :) 16:16:11 hemna: please take a look at the proposed implementation and see if it's considered "testing" os-brick. 16:16:37 BTW. https://blueprints.launchpad.net/nova/+spec/online-volume-extend-extension 16:16:43 Updated spec for cinder: https://review.openstack.org/#/c/453286/ 16:16:52 mgagne, can you just put up a cinder WIP ? 16:16:59 Updated spec for Nova: https://review.openstack.org/#/c/453272/ 16:17:00 then I can pull that and play with it 16:17:01 Some Trove folks had a patch up for this at one point as well (somewhere) 16:17:10 does the proposal from mgagne mostly line up with our discussions on this, or is it going a different direction? 16:17:30 hemna: sure, will do that by tomorrow 16:17:35 ok coolio 16:17:45 I think the people with the strongest feelings here are jgriffith and hemna 16:18:06 mgagne: That would be great. I would like to play with it as well. 16:18:29 the problem from my perspective has never been the Cinder side 16:18:35 I had hacked the cinder API to allow this while I worked on the os-brick piece a year or so ago when I wrote the brick code. 16:18:47 Nova has rejected the proposals to do the rescan in the past 16:19:08 jgriffith: Agreed. 16:19:18 as of yesterday, I think mriedem was happy with cinder doing a call into Nova to initiate the workflow on the nova side after cinder had resized the volume. 16:19:19 and there was some debate around how it should work in terms of notifications to Nova 16:19:22 jgriffith: the current proposal suggests reusing existing external-events API endpoint instead of introducing a new API call 16:19:25 mriedem: Seems to be ok with making changes if they are triggered from Cinder. 16:19:35 hemna and that was/is the problem :) 16:19:43 yah, as was mine too. 16:19:54 hemna how do you "call nova" from Cinder? 16:20:12 cinder already calls nova today 16:20:15 for migration and retype 16:20:15 Cinder does that now with swap volume 16:20:16 the same way we call it for other things now? 16:20:18 jgriffith: Isn't there a call back service. 16:20:25 There is the guy I was looking for. 16:20:36 nova doesn't want more apis like that, we have the external server events api that nova<>neutron use 16:20:50 if we're doing new things like that between nova<>cinder we use the same api with new event types 16:20:50 mriedem +1 16:21:38 ok, will post code to cinder so people can play with it and gather comments/feedback 16:21:43 eharney IIRC the only thing doing any interaction today is the instance-assisted snapshot stuff you worked on no? 16:22:04 suppose I could look for myself :) 16:22:04 jgriffith: and swap volume 16:22:11 but yes those are the two 16:22:15 #action mgagne To post WIP code for Cinder team to download and try. 16:22:19 retype/migration and guest-assisted snapshot 16:22:28 so maybe step 1: convert those to the new event system 16:22:36 step2: propose the extend call? 16:22:58 I don't want to have another case of *2 ways to do it in Cinder* that never gets cleaned up 16:22:59 i wouldn't waste time on converting snapshot/swap volume to the event api 16:23:13 those were never even tested in the gate until probably ocata 16:23:23 guest-assisted snapshot still isn't b/c it's fs-style backends only 16:23:25 mriedem alrighty then 16:23:29 mgagne, added a comment on your gist....FYI 16:23:41 thanks, will take a look 16:23:42 we have an NFS job testing assisted snaps now 16:23:48 mriedem: Yeah, lets work on getting this to go forward first. 16:23:59 eharney: yeah, but that was ocata 16:24:07 mriedem but I am going to voice my concern about adding yet another divergent path in Cinder on how to do things 16:24:11 but yeah we have that coverage now somewhere at least 16:24:19 mriedem voiced.. and done. Moving on 16:24:46 jgriffith: Sounds like it would be good to go back and fix that in the future. 16:24:51 Maybe get a bug open for that. 16:25:03 jungleboyj if I had a dollar for every time we said that and *didn't* do it :) 16:25:09 Agreed that we don't want to continue to diverge. 16:25:53 #action jungleboyj To open a bug for moving the snapshot/swap API to using events 16:25:57 jgriffith: :-) 16:26:07 It is an action. Has to happen now. 16:26:09 note, 16:26:17 swap is two callback apis :) 16:26:21 cinder -> nova -> cinder 16:26:23 it's the best 16:26:38 and if your policy doesn't line up, well, sucks to be you 16:27:17 mriedem: Noted. 16:28:08 So, I think we have visibility to the extend volume issue. We need to get everyone to look at the code that mgagne shares and we need reviews of the specs I linked above. 16:28:11 mriedem not sure I follow what you mean? 16:28:50 mriedem are you being sarcastic, or pointing out that without changing it on Nova the we're shit out of luck? 16:28:53 mriedem or both? 16:28:54 :) 16:29:33 i think policy.json 16:29:49 #action Try mgagne 's patch, review extend volume specs . 16:30:23 mriedem: Has disappeared behind the great firewall of China. 16:30:26 jgriffith: retype 16:30:27 in cinder, 16:30:33 calls swap_volume in nova, 16:30:38 and nova calls back to cinder to tell cinder that the swap is done 16:30:41 and complete the migration 16:30:46 i'm being serious 16:30:52 there was a bug in nova at one point, 16:31:03 mriedem yeah, I know about that sadly :( 16:31:18 where the nova and cinder policy files didn't line up such that a non-admin user initiating a swap volume in the comptue api (w/o cinder) would fail b/c of the callback to cinder, which is admin-only by default 16:31:29 swap volume used to be admin or owner in nova 16:31:31 it's admin-only now 16:31:32 by default 16:31:47 i had to dig through the history to figure out how swap volume even came about 16:31:51 mriedem I just wasn't sure as you stated you didn't think it was worth changing if that meant you didn't want it changed in Nova so if our *policy* was consistency and didn't align you were saying "suck to be you" 16:32:01 mriedem vish and avishay IIRC 16:32:17 mriedem: So that was a bug that has been fixed unless the end user changes the policy? 16:32:18 the good old days of wild west vendor specific apis 16:32:21 yeeehaWWW!!! 16:32:29 jungleboyj: the end user doesn't change the policy 16:32:34 the operator does 16:32:38 mriedem it was a *thing* with IBM 16:32:53 vish wasn't ibm 16:32:56 anyway 16:33:00 mriedem: Sorry, operator. or packager. 16:33:05 mriedem no, but avishay who owned/drove it was :) 16:33:13 and guest assisted snapshots was gluster 16:33:13 and red hat 16:33:16 mriedem but as you say... "anyway" 16:33:16 but we digress 16:33:30 mriedem that was actually eharney and russellb 16:33:33 i know 16:33:34 red hat 16:33:36 glusterf 16:33:57 So, does anyone have more on this topic or are we good with the action items going forward? 16:34:38 Was floating into the light so I think I'm good. 16:34:57 Ok, so we shall continue. 16:35:22 Thank you mgagne for helping with that code and to everyone else who is contributing to that effort. 16:35:45 =) 16:35:46 #topic Encrypted volumes and Glance: 16:35:52 eharney: You are up. 16:36:11 #link https://review.openstack.org/#/c/453342/ 16:36:18 sure, just wanted to see if anyone was interested in this work, i'll give some background... 16:36:21 #link https://review.openstack.org/#/c/453343/ 16:36:32 Go for it man! 16:36:34 upload-to-image is pretty good, volume encryption is pretty good, the combination of the two is pretty bad 16:37:18 and fortunately or unfortunately this used to "work", long ago, so people upgrade expecting this to be a functional workflow, and are disappointed 16:37:34 eharney oh, that's a bit horrid 16:38:10 eharney: Yuck. 16:38:28 it would have needed rework anyway to work with multiple encryption keys and barbican, as opposed to our conf key manager that has one key 16:38:55 so these patches attach a key id to images in glance when we upload them, and use that to find the encryption key when we download the image to a volume 16:39:09 there are a handful of other bugs and issues in there (see the dependent patches), but that's the main gist of it 16:39:39 without this, attempting that flow will get you either a) a doubly-encrypted volume that you can't read, or b) failure to create the volume from image 16:40:06 the patches are at the "works in devstack" phase now, i'll be polishing this up a lot and moving forward, unless anyone has big objections 16:40:16 eharney: Small patches, that is good. 16:40:24 Sounds like we need to get that fixed in pike though. 16:40:38 eharney have to say nice work disecting that 16:40:48 jgriffith: ++ 16:41:01 i have to test a lot of combinations of things, but the code isn't bad so far 16:41:10 eharney: Sounds like we should get those patches on the review priorities list? 16:41:16 also, would like to give thanks to LisaLi who started down this road a while ago 16:41:20 jungleboyj: yeah, probably so 16:41:44 they are loosely attached to the blueprint: https://blueprints.launchpad.net/cinder/+spec/improve-encrypted-volume 16:41:56 #action Get patches to fix Encrypted volumes and Glance on the review priority list. 16:42:02 this blueprint is basically "fix everything related to encryption" so i'd like to finish it out and not keep adding things to it, but this is there for now 16:42:27 that's about it from me 16:43:41 eharney: Thank you for the efforts there and for bringing it up. I will get the patches on the review list and make sure we follow up on that. 16:43:45 nice work 16:44:04 Wow Swanson is impressed. *gasping* ;-) 16:44:09 :) 16:44:21 jungleboyj, it happens! 16:45:03 :-) 16:45:15 Ok. So we have the action items there. 16:45:27 #topic Open Discussion 16:45:46 Anything else we need to discuss today or should we wrap this baby up? 16:46:01 Guys...I needed some feedback on the generate driver matrix 16:46:06 https://review.openstack.org/#/c/371169/ 16:47:06 karthikp: Ok, have a tab open for that and will take a look. 16:47:57 jungleboyj: Thanks Appreciate that! 16:48:07 It would be nice to see some feedback on blueprint proposing HA races fixes - https://review.openstack.org/#/c/429947/ 16:49:03 <_alastor_> Does anyone know the simplest way to get a project name in a driver if you have the project id? 16:49:29 Also my life would be much easier if we got https://review.openstack.org/#/c/417257/ (quota module split) merged or abandoned 16:49:41 bluex: Ok, Spec cutoff is coming up. I am hoping to get some time to get through Spec reviews. Will coordinate with smcginnis_vacati When he is back. I know he is pretty busy over the next few weeks but I am going to try to meet up with him to work through things. 16:49:47 End of next week. 16:50:01 awesome, thanks 16:50:35 bluex: Welcome. 16:50:46 _alastor_: Not sure. Anyone have feedback there? 16:50:55 jungleboyj: I thought we have until Pike-2 for spec merge? 16:51:26 <_alastor_> jungleboyj: It's not a huge deal if it's not possible. I'd just like to store name on the backend for easy human-readable mapping 16:51:32 jungleboyj: https://releases.openstack.org/pike/schedule.html#cinder-spec-freeze 16:51:37 june 7th 16:51:37 xyang: That will be here before we know it. ;-) 16:51:46 that's true:) 16:52:11 xyang: With the Forum in there and everything. That was more beating on myself to start reading specs which I haven't been doing so well at lately. ;-) 16:52:39 jungleboy: I need to review specs too:) 16:54:06 Also, a reminder that we don't yet have a docs liaison. 16:54:26 I could take that on if no one else is interested ... but, if someone would be interested ... 16:55:06 Anyone have anything else for today? 16:55:14 Also Pike-1 is around the corner. Please submit a patch to add CG capability to generic volume groups if you have not done so 16:55:24 xyang: ++ 16:55:38 Do we have to have those merge before Pike-1? 16:55:49 no, just submit a patch by Pike-1 16:56:03 so what happens if folks don't ? 16:56:08 xyang: Ok, good. Need to work on those reviews too. 16:56:09 should be merged by Pike-3 16:56:09 https://review.openstack.org/#/c/443364/ 16:56:18 Oh, never mind that link, then. 16:56:25 hemna: we'll open a bug 16:56:27 and if drivers don't update their code for the generic groups in PIke ? 16:56:46 They get a generic warning. ;-) 16:56:54 hemna: At PTG, we decided to submit a bug for that driver 16:57:01 ok that's it ? 16:57:10 just curious 16:57:16 hemna: yes, that was the decision at PTG 16:57:20 ok coolio. 16:57:34 does the driver stop functioning ? 16:57:40 wrt CG's ? 16:57:49 hemna: In Queens though, I want to remove the conversion part, so if they didn't convert, they lose the CG function 16:58:10 xyang: So the deadline is just encouraging everyone to get the work done. 16:58:14 hemna: actually in Pike, it will still work, because there's a conversion from CG to groups 16:58:15 ok, so CG will still work in Pike 16:58:25 jungleboyj, hemna: yes 16:58:32 do we throw a warning in the logs at startup 16:58:36 or during CG actions ? 16:58:57 hemna: I think we should probably do something like that 16:58:57 'CG has been deprecated, please migrate to generic groups." or something along those lines 16:59:06 for every CG call 16:59:09 hemna: deprecation of CG API is in Queens 16:59:10 $0.02 16:59:16 wait 16:59:23 you said you wanted to remove CG in queens 16:59:31 which means we need to deprecate it in Pike no ? 16:59:42 hemna: CG API will still work in Pike. In Queens we deprecate it 16:59:55 hemna: and we remove it when we can 16:59:59 and remove it later 17:00:00 ok 17:00:18 hemna: in Queens, I want to remove the piece of code in manager that convert a group object to a cg object 17:00:35 once that's removed, driver will lose CG function if they don't add it to groups 17:00:36 which would break CG ? 17:00:39 xyang, isn't that effectively removing it? 17:00:47 yah, so we need to deprecate it now then 17:00:49 Swanson: no 17:01:11 It is removal of the conversion. 17:01:11 xyang, That doesn't break CGs? 17:01:18 For drivers already added CG capability in Groups, that will not break CG 17:01:24 Oh, wait, forgot about the conversion thingy. 17:01:30 but it doesn't create groups 17:01:40 which then leaves us with what? 17:01:42 I'm confused 17:01:43 only if drivers don't care and don't implement groups, they won't have CG function in groups for free any more 17:01:56 Yikes. We are out of time. 17:01:59 seems like that could lead to upgrade problems 17:02:06 Deprecate now. 17:02:12 Lets take this over to openstack-cinder 17:02:13 hemna: upgrade is taken care of in Pike already 17:02:14 ok 17:02:21 *sigh* 17:02:28 jungleboyj, Unusually smoothly run meeting this week. Wonder what has changed? 17:02:28 hemna: there will be no entries in CG tables in pike 17:02:31 Thanks everyone for a productive meeting! 17:02:37 thanks 17:02:49 Swanson: Ouch, I won't tell Sean you said that. 17:02:55 It was going so well until the end. 17:03:04 Anyway, thanks everyone for your work on Cinder. 17:03:13 See you all next week. 17:03:18 #endmeeting