16:09:36 #startmeeting Cinder 16:09:37 Meeting started Wed Dec 11 16:09:36 2013 UTC and is due to finish in 60 minutes. The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:09:38 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:09:39 avishay: Everyone is bailing out here for Christmas already. Not be just an IBM thing. :-) 16:09:41 The meeting name has been set to 'cinder' 16:09:46 hi all 16:09:51 woop 16:09:53 hi 16:09:58 hey 16:10:04 hello 16:10:05 Howdy! 16:10:08 better late than never 16:10:09 agenda today folks https://wiki.openstack.org/wiki/CinderMeetings 16:10:15 bswartz: not always :) 16:10:17 o/ 16:10:30 thingee: you have the floor 16:10:36 haha 16:10:47 #topic Ideas with extensions 16:10:57 #link https://etherpad.openstack.org/p/cinder-extensions 16:11:04 Oh no, thingee is talking to himself. 16:11:40 thingee: so an extension could add a new flow, or inject into existing flows, yes? 16:12:00 inject into an existing flow 16:12:07 yes 16:12:23 or a new one, right? 16:12:54 so that's debatable. I thnk things get more complex then. 16:12:59 i see 16:13:02 so reading the etherpad, does multi-attach fall into core since its part of "attach it" basic function 16:13:08 because you could end up with different results 16:13:30 kmartin: hehe i asked same question on the summit, but seems it isn't 16:13:38 If it is a new flow, how do users know when to invoke it and how? If you are modifying a flow, how do you allow two exensions to modify the same flow? 16:13:51 where would retype go? it's related to types, but it has it's own flow. where does migration go? replication can definitely inject into create_volume. 16:15:38 caitlin56: exactly. there maybe other extensions injecting to that flow...and then you got a whole mess if an extension just does a new flow. 16:15:49 sorry guys, ym irssi session is really laggy atm 16:16:14 i like the idea 'core feature should be avaiable (cannot be disabled/turned off) everywhere'. 16:16:24 thingee: that's my concern, the phrase "injection" brings back ancient memories of DOS TSR routines. 16:16:34 +1 winston-1 16:16:44 winston-d: +1 16:16:57 avishay: so I see migration as a new flow that could be registered with the manager. 16:17:01 DOS TSR routines: -1000 16:17:32 thingee: OK so that's what i meant by an extension adding a new flow 16:17:40 hey hi all! 16:17:44 rushiagr: hi 16:17:56 avishay: Thanks. I just need a good example to get to that point :) 16:18:04 thingee: :) 16:18:22 so harlowja and I have been talking about ways to inject beginning and end 16:18:26 that's pretty simple 16:18:38 i like the idea in terms of code, the DB is where it gets tricky (and also conflicting extensions as caitlin56 mentioned) 16:19:27 * DuncanT forgot the meeting, sorry 16:19:30 when you talk middle..I have this sort of silly idea that you take the the core tasks...whatever task is in the middle, you inject before that. I think this might lead to problems where the core tasks can change..and then extensions would break. 16:19:32 need tempest tests that try all combinations? 16:19:34 avishay: we can learn from past mistakes, the key is to prevent extension X from modifying the chain as previously modified by extension Y while assuming it was unmodifed. 16:20:21 You have things like FC zone manager that'll basically need to change how attach/detach works. 16:20:52 caitlin56: thingee: yep 16:21:57 thingee, do you have a wiki on what it means to be an extension and how one codes it up? I'm not clear on the distinction. 16:22:05 anyways, this is kind of what I've been throwing around. 16:22:35 I agree, we don't have good docs to explain what an extension is 16:22:43 hemna: not yet. I wanted to start with just rexplaining my idea since there was a big disagreement at the summit 16:22:47 henma: there isn't anything I could find. That might be a first step -- fully document what is there today. 16:22:59 I wanted to start with everyone agreeing what is core 16:23:09 if we can agree on what core is, the rest are extensions to cinder. 16:23:28 well I thought the real disagreement was simply around disallowing extensions to touch/change the db schema 16:23:46 thingee: your list of core seems good to me 16:24:32 henma: with some solid examples on a wiki page there might be less opposition to specific restrictions. Show us how it would work with the restriction. 16:24:34 hemna: yeah and I just want people on the same page of what core is. I really see the future of cinder is mostly just talking about extensions. Very rarely does something new come into core..it's just improvements to core mostly. 16:24:35 hemna: i agree that it would be cool to not have them change schema, but i think we need a better alternative than joins or metadata 16:24:58 avishay, +1 16:25:25 not that i have any bright ideas :) 16:25:30 heh 16:25:37 the FC Zone Manager is a grey area 16:25:46 which makes it a good discussion point 16:25:59 avishay: my only concern is when you have extensions changing the model schema to say volumes. What happens when that plugin gets disabled... 16:26:40 thingee, could be a use at your own risk documentation issue 16:26:46 Neutron is dealing with that right now since vendors dictate what the core table schemas look like. 16:26:47 Where can I read about why multi-attach can't be part of core? 16:27:19 thingee, disabling in that case breaks cinder 16:27:42 once you enable one of those plugins it can't get disabled, unless there is a migration that gets run on disable ? 16:27:42 is this blocking multi-attach? 16:27:44 pain 16:28:19 avishay: no. This will not block anything because of how fast it's progressing. Extensions should continue with development. 16:28:24 One question I have: could you define an extension that made some new type of volume available where you could *only* access that volume with the extension? 16:28:28 thingee: ok good 16:28:38 avishay: the problem will just get worse, but we'll get there when we get there. 16:29:02 thingee: we have to rewrite all the code anyway for taskflow, and all the unit tests for mock :) 16:29:22 caitlin56: I see that volume still part of the basic idea of what cinder thinks a volume is. it just has added information with it. 16:29:29 if you cinder list, it shows up. 16:29:32 you do* 16:30:28 So you use extensions to add new operations/etc., not totally new things that surpass the base. That's livable, especially if well explained on a wiki page. 16:30:53 That makes more sense to me. 16:31:04 ok so it sounds like no one disagrees with this and I'm ok with continuing to spend time with find alternatives to joins, what to do with extensions that require a complete rewrite of a flow like FC zone manager attach/detach...and provide a useful example of how this all works. 16:31:39 thingee: sounds good to me 16:31:42 the persistent data thing I'm really unsure about atm. 16:31:46 at the summit avishay discussed the idea of having a new type of volume that would be the passive end of a mirrored relationship. Is that something that an extension could handle? 16:31:55 bswartz: That discussion about multi-attach being core has popped up many times, and come out with several different answers. Not sure the discussion has been documented anywhere. A big part of it not being core is that many installations might want to turn it off, since it is hard to use correctly and *will* generate support load and data corruption 16:32:15 DuncanT: thanks for the explanation 16:32:22 DuncanT: +1 16:32:31 caitlin56: the passive volume will not be visible to users, so maybe that will ease things? 16:33:15 avishay: interesting, I'd like to hear how you use it if it isn't visible, but after this topic is finished. 16:33:33 #todo thingee needs to figure out 1) ext persistent data 2) work with ext that require redoing an entire flow 3) provide a useful example 16:33:36 caitlin56: https://etherpad.openstack.org/p/icehouse-cinder-continuous-volume-replication-v2 16:33:49 thanks folks for the input 16:34:00 thingee: thanks for the effort 16:34:08 if anyone is interested with working on this with me, please ping me 16:34:18 another impact of extension is that they sometimes modified internal RPC APIs 16:34:39 winston-1: mmm good point 16:35:02 winston-1: yes good point. noted on the etherpad 16:35:25 #topic icehouse-2 progress 16:35:40 even if you can turn off some extension, but it already modified RPC APIs, which means backwards compatibility is messed up 16:35:47 DuncanT: https://blueprints.launchpad.net/cinder/+spec/filtering-weighing-with-driver-supplied-functions 16:35:56 I think you started on this from discussions yesterday? 16:36:40 thingee: Yup, I've got the evaluatior done, the filter itself seems to work. Unit test still need sorting, but I'm waiting for Avisay's branch to be merged since he re-wrote them all 16:37:04 winston-1: so should we just disallow extensions to make backward incompatible changes? 16:37:19 DuncanT: do you mind marking that as started? 16:37:29 Sure 16:37:55 Done 16:38:01 thanks 16:38:09 rushiagr: extensions will need to *add* methods. And those methods would be disabled/disappaear when the extension is disabled. But they shouldn't *change* an existing method. 16:38:13 avishay: you're waiting jenkins failures for https://review.openstack.org/#/c/44881/ ? 16:38:16 volume retype 16:38:52 thingee: just put up a new version that addressed all issues so far (i think). yea, jenkins sucks today. 16:38:54 caitlin56: oh 16:39:05 thingee: it works and is ready for re-review 16:39:15 caitlin56: thats messy 16:39:23 great, so everyone should help out out on the retype review. :) 16:39:32 #link https://review.openstack.org/#/c/44881 16:39:39 is Abhishek here? 16:39:43 :) 16:40:29 caitlin56: err.. that can make things messy while disabling extensions 16:40:42 thingee: Refactor code for delete volume using TaskFlow 0.1 ? 16:40:54 avishay: yes. 16:41:06 alright going to assume not 16:41:10 thingee: avishay I will try to look at 44881 later today. 16:41:17 thingee: as far as i understood, all taskflow-related work is blocked on the create_volume patch currently in the queue 16:41:21 jungleboyj: thanks! 16:41:24 jungleboyj: thanks. I should too since I already went through it once and have an env ready 16:41:46 #Open topic 16:41:56 #topic open topic 16:41:58 :) 16:42:07 so any bps people want to discuss? 16:42:22 avishay: will review retype tomorrow, err, later today 16:42:31 winston-1: thanks a lot 16:42:51 thingee: https://blueprints.launchpad.net/cinder/+spec/deprecate-chance-and-simple-schedulers 16:42:51 we should also give create_volume w/ taskflow a high priority. it looks ok to me, but i'm not an expert. 16:43:06 avishay: I'll have a look at it too. I always found reasons to skip reviewing it so far :) 16:43:21 rushiagr: awesome 16:44:14 * jungleboyj needs to look at some of the longer patches soon. 16:44:49 avishay, I reviewed the taskflow -> 0.1.1 yesterday and it looks good minus a few minor tweaks 16:44:54 winston-1: looks like you were the one stopping it :) https://review.openstack.org/#/c/60690/ 16:45:03 thingee: i'm working on a patch to actually replace chance/simple scheduler with filter scheduler. 16:45:12 hemna: yep saw that 16:45:14 oh ok 16:45:36 I'll try looking at the retyping review today 16:45:42 winston-d: is it clear which filters to use in those cases? everything except capablities? 16:45:45 thingee: not really sure if anyone still using chance/simple scheduler 16:46:00 winston-d, avishay: I noticed the replace the schedulers isn't in I-2 16:46:35 thingee: it is now :) 16:47:29 avishay: i think every filters in default filter list should be included, even capablities filter. it's about weigher 16:47:30 has jgriffith mentioned when he is getting to the brick work ? 16:47:49 I'm going to make my usual point about back-compatibility here... 16:47:52 BTW, all, please stop rechecking...poor jenkins is broken, it won't help and only adds load 16:47:55 winston-d, avishay: ok so this bp captures deprecating...is there another for replacing with filter? 16:48:19 winston-d: do you know why the filters are in two different folders in cinder 16:48:20 hemna: I was wondering that myself...I have not heard anything on it yet 16:48:21 thingee: so winston-d's suggestion is not to deprecate, but replace them with filter scheduler under the covers 16:48:38 DuncanT: ^ 16:48:39 avishay: got it 16:48:46 I spoke with him at the start of I-1, and he said he slated it for I-1, but I guess he got busy 16:49:05 avishay: That's fantastic, as long as running config don't stop running, I'm entirely happy 16:49:16 DuncanT: yea sounds good to me too 16:49:33 xyang__: yes, i do. i can explain that offline in #openstack-cinder if you like 16:49:45 winston-d: ok, thanks 16:49:48 DuncanT, did you get a chance to write the BP on the scheduluer driver methods as discussed at HK ? 16:49:49 bswartz: do you need anything for https://blueprints.launchpad.net/cinder/+spec/multiple-capability-sets-per-backend ? 16:50:22 hemna: https://blueprints.launchpad.net/cinder/+spec/filtering-weighing-with-driver-supplied-functions ? 16:50:29 nice 16:50:31 thanks 16:50:32 avishay, DuncanT replacing scheduler patches should be ready in a day or 2. 16:50:45 bswartz: i like that BP 16:51:19 winston-d: I've got patches in a holding pattern waiting on Avishay's unit test rewrite to land 16:52:10 winston-d: cool. note also that 'chance' has that silly ignore_hosts thing...not sure if it needs to stay 16:52:30 thingee: since you're the resident mock expert, can you review that one and free up DuncanT ? 16:52:56 avishay: are you going to pass the bp onto winston-d then? 16:53:00 avishay: sure 16:53:03 thingee: yep 16:53:09 avishay: i prefer no -> 'ignore_hosts', force_hosts 16:53:16 thingee: BTW, mock is a million times better than mox 16:53:22 avishay: I think dosaboy has taught me a thing or two from previous reviews with mock, but I'll take it :) 16:53:40 thingee: :) 16:54:20 bswartz: ping me later about the bp? or jgriffith whenever he wakes up :) 16:54:26 thingee: reassigned to winston-d 16:54:39 thingee, DuncanT, avishay, hemna do you guys really want to make scheduler aware pools inside one back-end? 16:55:18 thingee: sorry I got distracted 16:55:26 winston-d: i think it will help scalability 16:55:28 thingee: yes I'm working on that one 16:56:22 winston-d: I'm not sure I follow. Though I've worked / reviewed a little with scheduler. 16:56:25 pools inside back-ends, small pools inside pools, smaller poools ... 16:56:56 winston-d: I'm not sure of the point, but I do remember there being talk of it 16:57:07 thingee: well, whether there's oone pool or multiple pools within a back-end, it's transparent to scheduler, for now. 16:57:24 bswartz: can you update the BP with its motivation please? 16:57:29 winston-d: almost 16:57:35 avishay: yes 16:57:45 DuncanT, bswartz do you remember if we have any consensus about that? 16:57:48 avishay, bswartz: yeah I just want to know if its been started yet 16:57:58 it was marked unknown 16:58:16 it's transparent until the scheduler picks the wrong backend and there's a failure 16:58:24 winston-d, the pools are part of volume types for our backend 16:58:29 winston-d: one backend should be able to manage multiple pools 16:58:57 xyang__: i'm fine with that, but not sure if pools should be expose to Cinder 16:59:38 here's one motivation: suppose a backend has 2 pools with 50GB space each -- it reports 100GB capacity to the scheduler, if the scheduler sends a 100GB create request, it will fail. It would be better if the backend could just report that it has 2 pools of 50GB each 17:00:02 alright lets take the discussion for pools into #openstack-cinder 17:00:04 we're out of time 17:00:07 that's a simple example, there are more complicated ones -- I'll update teh BP though 17:00:08 #endmeeting