16:00:01 #startmeeting cinder 16:00:02 Meeting started Wed Feb 11 16:00:01 2015 UTC and is due to finish in 60 minutes. The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:03 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:05 The meeting name has been set to 'cinder' 16:00:06 hi! 16:00:07 hello all! 16:00:10 o/ 16:00:11 hi! 16:00:15 hello 16:00:23 Hi 16:00:27 hi o/ 16:00:43 hi, 16:00:44 Your weekly reminder about third party ci being required by march 19th if you have a cinder volume driver in Cinder http://lists.openstack.org/pipermail/openstack-dev/2015-January/054614.html 16:01:00 hi 16:01:03 if you have no idea what I'm talking about, and you have a volume driver in Cinder, talk to me today! 16:01:05 Hi 16:01:14 Hi 16:01:24 Also reminder, spec freeze is feb 15 16:01:34 hi 16:01:35 code freeze on march 10th 16:01:55 #link http://lists.openstack.org/pipermail/openstack-dev/2015-January/055719.html 16:02:07 hello 16:02:09 ok lets start! 16:02:11 hi 16:02:30 #topic Cinder state management 16:02:34 #link https://etherpad.openstack.org/p/cinder-enforcement-of-states 16:02:41 #link https://etherpad.openstack.org/p/cinder-kilo-persistance-workflow-status 16:02:49 oh hi :-P 16:02:57 agenda for today as well 16:02:59 #link https://wiki.openstack.org/wiki/CinderMeetings#Next_meeting 16:03:11 harlowja_at_home: go for it 16:03:12 o/ 16:03:39 harlowja_at_home: the original person leading it, is not present 16:03:46 ya, arg, where is he 16:04:05 should we circle back? 16:04:12 hi thingee 16:04:16 ah 16:04:20 vilobhmm: you're up 16:04:21 like magic, lol 16:04:32 why do we need both persisnance and microstates? 16:04:45 o/ 16:04:49 o/ 16:05:01 e0ne : i have outlined the differences here https://etherpad.openstack.org/p/cinder-kilo-persistance-workflow-status 16:05:25 converging both approaches somewhere is a better approach i feel 16:05:37 if not then persistence taskflow has many shortcomings 16:05:48 how so ;) 16:05:57 please see https://review.openstack.org/#/c/147879/ 16:05:57 so basically the information saved is very similar in both cases 16:06:11 dulek : agreed 16:06:29 :%s/many/few outlined in the review 16:06:39 and e0ne was able to create PoC of resuming tasks from saved microstate 16:06:54 the basic point is maintining state in centralized storage and able to recover from failure 16:07:24 actually, my PoC did the same what persistance taskflow does 16:07:40 vilobhmm: I don't agree with many statements you put in review but this is another discussion 16:07:55 and also to avoid local file system locks...micro-states helps us regarding both by introducing state transition based on the validation graph 16:08:02 hi 16:08:36 so few questions for thingee : what is your thoughs on going ahead regarding any cinder state management approaches 16:08:41 we should work to minimize the number of states / micro-states 16:08:44 So validating state transitions is good and taskflow's persistent BPs doesn't cover this topic 16:08:57 %s/thoughs/thoughts 16:09:05 dulek, agreed (that is cinder 'business logic') 16:09:09 dulek : agree 16:09:18 avishay: +1 16:09:23 vilobhmm: i'm asking _why_ we need both of them? imo, we need to choise between microstates w/o taskflow and taskflow+persistance 16:09:40 and thats why i am saying taking merits of both is a better way to go than saying x is not good or y is better 16:09:43 e0ne: practically we could combine them 16:10:10 e0ne: use persistence to save states and vilobhmm mechanism of transition graph for validations 16:10:18 dulek, +1 16:10:23 vilobhmm: So I'll repeat my thought, it was raised at the midcycle meetup that people weren't interested in continuing with taskflow. DuncanT was really concerned by what we gain. 16:10:25 dulek +2 16:10:37 thingee, except it seems people are no? ;) 16:10:56 vilobhmm: but people ignored me when I brought up the meeting notes it seems. 16:11:12 thingee: this means stopping further integration in kilo? 16:11:20 thingee : but didn't we decided to go with more taskflow in the paris design summit conclusion 16:11:23 all the above work seems to show that someone is interested, lol 16:11:27 dulek: yes 16:11:29 #link https://etherpad.openstack.org/p/cinder-meetup-winter-2015 16:11:49 thingee: what about the future? decision to be made on the summit? 16:11:53 harlowja_at_home: it's not of interest. It's we have bugs with the one thing that's using taskflow and no one is focusing on that 16:12:36 Hi, sorry I'm late 16:12:43 thingee: I've actually sent patch fixing 1408763 before the meeting. ;) Anyway I understand your concerns. 16:12:49 hi folks 16:12:55 dulek: jgriffith has some thoughts of a queue based system to start to get cinder back to a form that people who don't understand taskflow to be able to fix bugs again in the create volume flow 16:13:21 dulek: that's great, I'm really happy hear this. I'll have to see how you solved it though :) 16:13:56 so what about getting those people to understand taskflow? 16:13:59 thingee: you'll probably won't like it. ;) But lets see. 16:14:04 vilobhmm: anyways I don't think you want to hear my opinion. I'm confused by everything people are presenting me. It sounds like you guys are unsure as well. 16:14:14 thingee: any resource where I can read about this "queue based system"? 16:14:31 dulek: nothing yet. 16:14:43 dulek: this would be something in the summit and in L 16:14:50 dulek: if we actually go that route 16:15:03 thingee: okay, let's talk Kilo then 16:15:14 thingee, so what part of the stuff being discussed is confusing? what can we do better to explain? 16:15:16 thingee, DuncanT : dulek thanks for fixing the taskflow bug...so thingee now if the bug is fixed whats stopping for more taskflow...as harlowja mentioned why not get people understand taskflow..its starightforward has good documentation and the amount of time needed to not use it or remove it why can't we invest same time in understanding it 16:15:26 thingee : ok 16:15:31 vilobhmm: please talk with DuncanT. He was really concerned about persistence not allowing a task to be resumed on a different host. 16:15:33 i have a decent idea of how i'd like things to look, but unfortunately it's pretty far from where we are today, and taskflow unfortunately doesn't fit in that picture 16:15:50 for kilo we're going with microstates and leave persistence for summit discussion? Or any other idea? 16:15:53 thingee : sure will talk with DuncanT 16:15:56 harlowja_at_home: there's too much information and different approaches coming. I don't even know if the current spec we have is even valid anymore. 16:16:03 IMO if we can't continue a task on a different host we haven't gained much 16:16:23 thingee, vilobhmm, DuncanT: persisnance allows it 16:16:27 thingee, isn't that how normally things work? people have to iterate and agree on something to get somewhere (which does seem to be happening) 16:16:28 'Getting to know taskflow' seems to be an infinite task :-( Every time I think I have my head around it, somebody asks a question and I find I don't undersatnd it well enough again 16:16:30 thingee: so we decided to remove this constraint after consulting with hemna 16:16:46 avishay: I've read the reviews from dulek, I think the idea is to get there eventually, but there is an issue with sqlalchemy + taskflow, and what we support, etc. 16:16:57 thingee : for the microstates apprach i need few revies for https://review.openstack.org/#/c/124205/ which is all green...i have addressed all concerns posted by xyang 16:16:58 thingee: i see 16:17:39 vilobhmm: no doubt, taskflow has great documentation. If you see the documentation though that jgriffith wrote in that bug though ;), it was really complicated to fix an allocation issue in Cinder that was using taskflow 16:17:40 harlowja_At_home : +1 16:17:52 harlowja_at_home, that's how it's supposed to work afaik. 16:17:57 thingee : I can revisit it 16:18:01 and get it fixed 16:18:21 thingee :whats the plan for kilo 16:18:36 vilobhmm: harlowja_at_home was even very responsive with jgriffith on trying to make changes in taskflow to fix the problem. Not sure of the specifics, but it was still not resolved. 16:18:36 are we going with microstates if yes then can i have some serious attention towards https://review.openstack.org/#/c/124205/ 16:18:38 vilobhmm: we're talking about 1408763 bug, which I'm working on 16:18:59 vilobhmm: any of these state changes is too late for K. these require a bit of testing 16:19:12 if we can have something ready early L, that would be ideal, to allow gating on issues 16:19:16 dulek : sure ...the one we discussed in the nova mid cycle meetup 16:19:21 my biggest fears are: (1) we are already heavily reliant on a DB that doesn't scale horizontally in what is supposed to be a distributed system, and (2) a ridiculous number of state updates and rollbacks for each will make performance terrible and code unmanageable 16:19:27 I'd rather people focus on the cinder object patches 16:19:46 avishay: +1 16:20:16 avishay: +1 Good concern. 16:20:32 anyways that's my thought. vilobhmm hope you had a great trip and we missed you, but it's too late at this point. 16:20:47 avishay : with micro_states appraoch we are not storing huge amount of datat in db i am just storing the microstate the task was in 16:20:51 I'm happy that dulek and e0ne are pitching in though 16:20:53 which is just few bytes 16:21:13 ok, let's summarize it 16:21:14 thingee : what does early L means what date ? 16:21:15 okay, thingee, can we have your summary of this discussion and move on? 16:21:21 vilobhmm, harlowja_at_home: I have not decided on anything though. These are just issues raised by the cinder team 16:21:24 soren, prior to L1 ? 16:21:26 vilobhmm: so for create volume, how many updates do you estimate? i figure at least 10 16:21:28 avishay: +1 16:21:31 but would appreciate some serious reviews for https://review.openstack.org/#/c/124205/ 16:21:33 no cnahges for microstates and persisnance for K, right? 16:21:53 avishay : 5 for api and 4 for manager side 16:22:06 hemna: +1 16:22:12 avishay: We're trying to make a system with a massive number of moving parts and a very distributed state model work... some amount of central orchestration seems unavoidable. We are currently racey as all hell 16:22:18 so yes I would appreciate people also reviewing the links that I already put in the meeting notes. 16:22:40 thingee :thanks for the help for asking people to review it 16:22:52 #topic Local File Locks Removal from Volume Manager Update 16:22:53 thingee, undestandable, it just seems the cinder team doesn't have a clear vision of cinder, and people are trying to help, but getting pushed back because of that lack of a clear vision (which is sad), so visions are being formed (by dulek vilobhmm e0ne) and visions that will likely help overall, but then there is a greener pasture somewhere else (as always) 16:22:55 leeantho: you're up 16:23:03 Ok, so I have been working on looking into the removal of the local file locks in the cinder volume manager with help from hemna. 16:23:03 #link https://review.openstack.org/#/c/149894/ 16:23:10 #link https://blueprints.launchpad.net/cinder/+spec/remove-volume-manager-locks 16:23:15 DuncanT: i disagree. we don't need that many states and we should be "looser" - eventual consistency, for example, but let's move on... 16:23:17 #link https://review.openstack.org/#/c/153748/ 16:23:18 The idea was to remove the locks from the volume manager and instead check for a volume being in an 'ING' state like 'creating', 'detaching', etc. The API would throw a VolumeIsBusy error if a request was recieved and the volume had one of those statuses. 16:23:32 At this point I am looking for feedback on this approach mostly 16:23:33 avishay: We can talk after the meeting 16:23:41 DuncanT: yes 16:24:00 leeantho: Is that going to break people's tooling, and do we care? 16:24:21 leeantho: I fully support this. Not sure if we have time in L for it though in terms of testing. 16:24:28 soren, leeantho has a cinder-spec up that talks about the approach as well as a WIP cinder patch to show what happens FWIW 16:24:46 its most likely a L thing at this point 16:24:48 the throwing a VolumeIsBusy exception causes all sorts of breakage in Nova 16:24:58 leeantho: +1 16:25:01 which is the sticking point here and why we wanted to raise this up 16:25:04 so far problems I noticed were it breaking nova, and tempest 16:25:10 we didn't plan on putting this in K at this point 16:25:28 It's an API change, so I'd expect tempest to break 16:25:32 but simply wanted to get some feedback and figure out what to do with the nova breakage due to changing the expected responses to API calls 16:26:22 We need to patch nova to cope with both versions before we change cinder... or hide it all in cinder-client if that is even possible 16:26:56 makes sense 16:27:14 #agreed this is too late for K 16:27:14 DuncanT, Yes, it would be best if there was some way to not need a nova change, was hoping maybe there were some other ideas 16:27:22 The other option is to block in the API until the state changes, but I think that opens up too many DoS possibilities, so we are better changing the API 16:27:30 DuncanT, yah 16:27:49 leeantho: Nova's wrapping of cinder calls is really fragile, some eyes on that can only be a good thing IMO 16:27:51 those blocking calls could eventually timeout and result in the same problem 16:28:09 I've complained before as to how nova is calling cinder a few times 16:28:16 I think the model is broken IMHO 16:28:17 DuncanT, hemna, leeantho: I can help review the nova changes. I have worked on that layer a few times 16:28:22 hemna: Even if they don't timeout, it becomes trivial to use up all the API workers and DoS the system 16:28:30 DuncanT, +1 16:29:09 leeantho: I don't think there is a magical approach to avoid not touching nova unfortunately 16:29:10 hemna: +1 16:29:11 This will lead to some API breakage for scripts though, as well as nova/tempest (and horizon). I can't see any way to avoid that though 16:29:38 thingee, DuncanT I see, I can continue looking into the nova part then 16:29:46 leeantho: don't forget tripleo and heat as well 16:30:01 leeantho: I don't think it would concern them though 16:30:08 DuncanT, yah, that's what I was thinking as well. So I just wasn't sure what the plan was to changing the API and dealing with it. 16:30:32 #topic APIs for modifying volume and shapshot image metadata Update 16:30:37 hi 16:30:38 davechen: hey 16:30:39 hi all, 16:30:41 yeah, 16:30:43 #link https://review.openstack.org/#/c/136253/ 16:30:46 Even if we go more fined-grained, there are always going to be cases where we have to say 'nope' 16:30:49 #link https://review.openstack.org/#/c/147077/ 16:30:51 Nova is still looking at the internal state of volume objects to determine if it can take actions on them. It really should be asking Cinder, but that's a separate topic 16:30:55 Let me briefly summarize current status, 16:30:55 #link https://review.openstack.org/#/c/147726/ 16:31:02 #link https://review.openstack.org/#/c/149163/(snapshot) 16:31:04 The basic idea is provide commands and APIs for modifying the volume and snapshot image metadata. 16:31:27 I have submit one patch in Cinder Client, the patch provide the command lines to modify the image metadata for volume and volume-snapshot, define the API method as "action", so there will be no conflict with volume/snapshot API. 16:31:30 hemna: It should just try them, otehrwise there's always a time to check/time to use race 16:31:54 Patch is here: https://review.openstack.org/#/c/147077/, The command looks like: 16:32:04 cinder image-metadata � set 16:32:09 cinder image-metadata � unset 16:32:14 cinder snapshot-image-metadata � set 16:32:19 DuncanT, yes. 16:32:21 cinder snapshot-image-metadata � unset 16:32:49 Wrote a patch to address update and delete volume image metadata respectively, 16:32:50 Modify volume image metadata is addressed in this patch, https://review.openstack.org/#/c/147726/, 16:33:13 the API interfaces are defined in "cinder/api/contrib/volume_image_metadata.py" instead of "volume_actions.py" since "volume_image_metadata.py" is already existed and seems intends to show image metadata relevant with the specific volume. 16:33:33 davechen: Does that cover adding new metadata? (update of a none-existance key, I guess?) 16:33:33 And.. 16:33:35 Modify snapshot image metadata is addressed in this patch, https://review.openstack.org/#/c/149163/, the APIs is implemented in "cinder/api/contrib/snapshot_actions.py", the basic logic is the same with volume image metadata modification. 16:33:57 DuncanT: yes, covered 16:34:10 Snapshot metadata should not be editable IMO, snapshots are immutable 16:34:16 both for volume and snapshot. 16:34:41 DuncanT: good catch, i missed it 16:34:49 DuncanT: then why did you approve the spec? 16:34:58 thingee: I only just thought of it 16:35:04 DuncanT: ah fair 16:35:14 DuncanT: I am not sure, we design such a APIs in the SPEC. 16:35:16 thingee: That is one of the problems with specs, many things only become clear on further thought 16:35:48 yeah, how to keep specs from becoming lies ... 16:36:01 Currently, I am still working on porting Glance properties protection mechanism to Cinder, but I am not sure whether this could catch up with "K" release, maybe the part would be addressed in "L"? 16:36:13 i'm not sure, we allow changing a snapshot's name/description/metadata, no? 16:36:19 The snapshot contents can never change after it is created, so all of the arguments for why we need metadata editting become nonsense 16:36:50 DuncanT: unless there was some mistake, but you're probably right 16:36:56 DuncanT: 'normal' metadata should be editable for snapshot, just to be clear.. no? 16:36:59 avishay: That is all filing/cataloguing though, this alters the behaviour of the volume 16:37:03 davechen: can you explain how protection props in glance will prevent modify props that we would not want to mutable? 16:37:04 rushiagr: Yes 16:37:07 avishay: +1 16:37:08 want mutable* 16:37:36 davechen: I think we need the glance stuff too, else this becomes a route to break all of the billing properties :-( 16:38:00 thingee: seems it's like RBAC as well, but protect against metadata. I am trying to debug the code in the glance recently. 16:38:19 DuncanT: +1 16:38:24 I didn't think about that. 16:38:28 DuncanT: yeah, I will trying. 16:38:45 but any other big issue anyone see? 16:38:59 Before we consider merging this, I'm really curious on how we will prevent the point DuncanT just raised. 16:38:59 We use them for billing, as does rackspace, so it is an important concern for us 16:39:04 otherwise the rest lgtm 16:39:38 thingee: I am trying to provide a seperate patch to address that. 16:39:47 davechen: ok thanks 16:39:54 thingee: thanks. 16:40:05 #topic Return request ID to caller 16:40:10 anyother comments? 16:40:12 abhishekk: hi 16:40:16 hi 16:40:19 davechen: oh sorry :) 16:40:27 nothing... 16:40:33 thingee: I am creating a cross-project spec for this. 16:40:45 thingee: In the cross project, I am proposing solution of returning tuple containing response header and response body from all of the cinderclient api methods 16:41:00 abhishekk: so this requires a major version bump in cinderclient 16:41:08 it breaks all code using cinderclient today 16:41:09 thingee: right 16:41:18 too late for L 16:41:22 err K 16:41:30 I know my letters 16:41:50 thingee: I will discuss this in cross-project meet 16:41:50 but if we can get agreement cross project in L, lets do it 16:41:55 abhishekk: thanks 16:42:22 thingee: we are almost ready with the code 16:42:22 any others have questions/comments with this? 16:42:27 Can we collect anything else we'd like to make as a back-incompatible change in cinder-client at the same time? 16:42:35 what is the idea here? to be able to query a task later? 16:42:52 avishay: To be able to tie requests to logs 16:43:11 avishay: Makes QA a million times easier, and some customer support too 16:43:21 DuncanT: agreed 16:43:26 this will be great for debugging 16:43:33 abhishekk: ++ 16:43:35 DuncanT: +1 16:43:57 +1 16:44:02 i was hoping the idea included error reporting for tasks, that's all :) 16:44:09 agree that this is a good idea though 16:44:21 I will try to convince in cross-project meeting for ths 16:44:26 * this 16:44:28 abhishekk: did you implement changes to make this backward-compatible with the current api? 16:44:48 : yes 16:44:54 #agreed great idea, but needs to be approved in crossproject 16:44:54 gary-smith: The API is, the client changes, not so much 16:45:04 #action abhishekk to propose it in crossproject meeting 16:45:07 we have submitted patch for niva 16:45:10 will request IDs continue across projects? for example nova calling cinder for attach, everything has the same ID? or is that separate? 16:45:38 avishay: yes, we need to get one req-id throw all projects 16:45:39 avishay: I think you just get the info in the logs you need to join them up 16:45:58 avishay: One request has been discussed on the list and dismissed 16:46:04 DuncanT: ok 16:46:07 avishay: There were a bunch of issues 16:46:28 ok 16:46:36 anything else? 16:47:06 i will update the discussion from the cross-project 16:47:12 ok thank you abhishekk 16:47:19 #topic Driver filters and Goodness Functions 16:47:28 thingee: hi! 16:47:31 :) 16:47:33 #link http://lists.openstack.org/pipermail/openstack-operators/2015-February/006218.html 16:47:43 So I added a long comment to the review for this 16:48:14 thank you 16:48:26 DuncanT: link? 16:48:27 DuncanT: May I add you as reviewer? so you may take a look when you get chance, this may help me a lot, maybe there are some big mistakes with the IMPL but I didn't notice. 16:48:44 davechen: Sure 16:48:47 thingee: One sec 16:49:20 DuncanT: thanks a lot. 16:49:43 so I wanted people to be aware of this feature in K dev for Cinder. It provides great flexibility to the admin user. The documentation though and some concerns people raised are why I'm bringing it up 16:49:53 https://review.openstack.org/#/c/151353/ last comment 16:50:04 so thingee what else would you like to see in the documentation? 16:50:16 I can have leeantho work on adding any additional information you need 16:50:16 hemna: my comments are on the mailing list 16:50:28 and the review of the documentation itself 16:50:29 thingee, I saw your feedback on the openstack-manuals review and have started adding more details that will be in the next version 16:50:36 leeantho: thank you! 16:50:36 thingee : do we have a link for the same 16:51:01 DuncanT, saw your review. I'll work with gloria today to update the patch to provide the filter function and goodness function for get_volume_stats in the base driver. 16:51:04 so everyone gets it. 16:51:16 DuncanT: thanks for your write up. I'll read it to hopefully understand better 16:51:24 features going in K 16:51:28 hemna: +1000 16:51:36 hemna: that might be what's missing from all of this 16:51:57 hemna: I think that would be great 16:52:01 vilobhmm: https://launchpad.net/cinder/+milestone/kilo-3 16:52:05 ok perfect. I'll get on that today. 16:52:16 vilobhmm: it's not finalized...just rough 16:52:31 ok 16:52:44 hemna: I think it's fair to raise it on the ops list though 16:52:55 they're ultimately the users of this feature 16:53:19 hemna: and I encourage leeantho and you to be part of the discussion if one starts. leeantho sorry I didn't cc you 16:53:53 anything else? 16:54:15 #topic Open discussion 16:54:32 so it was brought up to me that we need yet another deadline date 16:54:53 YADD 16:55:13 I like to confuse people with dates, so lets do it. We need a date of when code should be proposed for specs. To allow enough time for reviewers. 16:55:17 #link https://launchpad.net/cinder/+milestone/kilo-3 16:55:36 taking a look at k-3, there aren't much bps that are not already in code review 16:56:03 I would like to propose code has to be posted by march 1st. That leaves 10 tens before code freeze, march 10th 16:56:11 do reviewers of cinder feel comfortable with that? 16:56:20 and do assignees feel comfortable with that? 16:56:25 +1 16:56:41 thingee: +1 16:56:53 10 tens...oh boy 16:56:55 10 days* 16:57:04 +1 16:57:05 * thingee drinks more coffee 16:57:07 thingee: are you going to create cinder-k3-priorities etherpad or we should look to https://launchpad.net/cinder/+milestone/kilo-3 for it? 16:57:13 thingee: THought you meant 10, 10 hour days. 16:57:18 :-) 16:57:21 e0ne: yes! xyang2 asked me about this too 16:57:33 thingee: :) 16:57:36 #action thingee to send out a k-3 priority etherpad for signup 16:57:48 jungleboyj: :) 16:57:56 We have a release in that timescale too, so if I am not paying attention to something you feel I should be, please poke me 16:58:10 * jungleboyj pokes DuncanT 16:58:17 Thanks, Jay 16:58:23 * thingee pokes DuncanT to pay attention to ALL of it ;) 16:58:25 DuncanT: Any time my friend. 16:58:37 thingee: i propose to add somewhere action item to release cinderclient in the end of K 16:58:42 and three votes for this YADD 16:58:54 Yadda yadda yadda 16:59:01 e0ne: That is a good reminder. 16:59:03 e0ne: +1 16:59:08 thank you, I will 16:59:27 Honestly, an interim release allows for more testing, though I dunno what has gone in of late 16:59:32 #action thingee to set reminder for cut and propose of new cinderclient version 16:59:46 thingee: thanks:) 16:59:55 and maybe a link to it in the cinder channel message.. 17:00:00 #endmeeting