17:00:42 <ildikov> #startmeeting cinder-nova-api-changes
17:00:43 <openstack> Meeting started Mon Dec 19 17:00:42 2016 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:46 <openstack> The meeting name has been set to 'cinder_nova_api_changes'
17:01:05 <ildikov> scottda DuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh lyarwood
17:01:14 <mriedem> o/
17:01:22 <scottda> hey
17:01:34 <ildikov> hi :)
17:01:41 <hemna> tough
17:02:05 <johnthetubaguy> o/
17:02:31 <ildikov> let's wait one more minute and then we can start
17:04:05 <ildikov> on Cinder side we have the dependency of the new API patch merged, so hopefully jgriffith is busy with the API patch atm
17:04:32 <ildikov> johnthetubaguy: did you have a chance to go through the comments on the Nova side patch?
17:04:46 * johnthetubaguy hides in the naughty corner
17:05:16 * ildikov does not know what to say on the year's last meeting to this :)
17:05:27 <johnthetubaguy> heh
17:05:53 <ildikov> johnthetubaguy: we discussed it briefly and the main comment from mriedem was to not initiate any volume state changes in Cinder from Nova
17:06:41 <ildikov> johnthetubaguy: so far everyone agreed to remove that direction form your spec
17:07:06 <johnthetubaguy> OK, I am not sure I know what that means
17:07:06 <ildikov> johnthetubaguy: but we can discuss it further today if you feel the need
17:07:10 <mriedem> johnthetubaguy: yeah the main issue i had was there are several parts that rely on nova setting a vol attachment resource status to error in cinder when we hit a problem, and i thought nova would just delete the vol attachment
17:07:44 <johnthetubaguy> ah, delete vs update into an error state
17:07:44 <mriedem> similar to nova rolling back and calling os-terminate_connection today on a failed attach
17:07:57 <mriedem> yeah i don't like nova controlling resource state in cinder
17:08:05 <mriedem> i don't think anyone does
17:08:26 <johnthetubaguy> I think I wanted a "tombstone" for an operator to do checks/cleanup
17:08:28 <mriedem> i think you were doing it in part for evacuate making queries for attachments in error state,
17:08:33 <ildikov> it also gives more items to clean up later which is not that fortunate either
17:08:36 <mriedem> but there are probably other ways to handle that
17:08:37 <johnthetubaguy> ah, yeah, evacute
17:09:03 <johnthetubaguy> what was the replacement for evacuate?
17:09:04 * smcginnis strolls in but stays at the back of the room
17:09:05 <mriedem> it's been a couple of weeks since i left comments so they are in there
17:09:29 <mriedem> i had some questions about evacuate related to the error state thing, but don't remember what they were now
17:09:44 <mriedem> but was thinking there were alternatives, or just not needing the error thing
17:09:45 <johnthetubaguy> oh, use the migration object instead
17:10:16 <mriedem> not sure what you mean by replacement for evacuate, but yeah we use the migration object to track state for evacuate now
17:10:30 <mriedem> i think i thought about that at one point too, we could store some data in the migration object
17:10:49 <mriedem> like we have old/new instance type in the migration object
17:10:56 <mriedem> if we needed, we could have old/new vol attachment in there too
17:11:46 <johnthetubaguy> so I should go through that, and undo the error state stuff
17:12:15 <johnthetubaguy> so tomorrow is my last day before next year
17:12:26 <johnthetubaguy> I should try get that done tomorrow
17:12:39 <johnthetubaguy> but been pushing on neutron v2 refactoring instead
17:13:28 <ildikov> johnthetubaguy: the pretty high level Cinder spec: http://specs.openstack.org/openstack/cinder-specs/specs/ocata/add-new-attach-apis.html
17:13:56 <mriedem> johnthetubaguy: i'd say if you have thoughts, just make a note in the spec review as a reminder for when you get back, but keep trucking on the neutronv2 conductor stuff
17:14:00 <ildikov> johnthetubaguy: just in case you would need to double check what calls we ended up with finally
17:14:02 <mriedem> because that's actually in plan for ocata
17:14:33 <johnthetubaguy> mriedem: yeah, I think thats probably the better approach
17:15:32 <johnthetubaguy> I can't think of why delete the attachment wouldn't be enough
17:15:55 <johnthetubaguy> I think I was worrying too much about cinder knowing about all the state changes on the Nova side
17:16:11 <mriedem> which we don't want cinder to know about
17:16:34 <smcginnis> mriedem: +1
17:16:41 <ildikov> I'm more of a fan of single responsibility as much as we can achieve that here
17:16:43 <mriedem> from what i remember i think the only case that it maybe made some kind of sense we evacuate cleanup, but again, it's been 2 weeks and i think we have alternatives
17:16:51 <ildikov> so what mriedem says :)
17:16:52 <smcginnis> I think Cinder should have minimal awareness of what's happening with consumers.
17:16:54 <mriedem> s/we/was/
17:18:05 <ildikov> mriedem: having alternatives sounds good, we can go into details if needed on the next meeting in Jan
17:18:45 <johnthetubaguy> I think I don't really understand the failure modes around disconnect that well at this point
17:19:01 <johnthetubaguy> I guess if its not detached, we leave it attached, if it is detatched, we detach it
17:20:19 <johnthetubaguy> anyways, simpler sounds like a better starting point
17:20:51 <ildikov> johnthetubaguy: +1
17:21:24 <ildikov> johnthetubaguy: when we'll have the Cinder API merged or close to that state we can also do some PoC, like what jgriffith started earlier
17:22:42 <johnthetubaguy> the tricky bit, I think, is going to be the smooth transition to the new API
17:23:30 <ildikov> you mean from the point of view of the already created volumes?
17:24:44 <smcginnis> johnthetubaguy: Or just all the changes required on the nova side, microversions, etc.?
17:25:43 <johnthetubaguy> yeah, already created volumes
17:25:57 <johnthetubaguy> and when to use the new API part way through operations like live-migration
17:26:45 <johnthetubaguy> we have a mix of old a new compute nodes out there, etc
17:27:00 <johnthetubaguy> we have patterns to follow, its just tricky
17:27:18 <ildikov> we can switch to the new API, when all the computes are upgraded
17:27:19 <mriedem> johnthetubaguy: we can globally lock that out based on minimum nova-compute service version in the deployment if needed
17:27:24 <ildikov> that should be doable
17:27:37 <ildikov> mriedem: yep, that's what I meant too
17:27:52 <mriedem> we can also check per-compute service versions if we want to be fancy, but that's probably a more complicated dance
17:27:57 <johnthetubaguy> mriedem: yeah, we might have to do that per operation though
17:28:15 <johnthetubaguy> well, I mean at the start of long running operations, like live-migrate
17:28:20 <johnthetubaguy> so we don't swap half way through
17:28:27 <ildikov> old volumes should have attachment_id already, it's a question what other info we would need for live migrate for instance as we will create a new attachment for the new host anyhow
17:28:31 <johnthetubaguy> but the BDM pattern I proposed in the spec should do that
17:28:44 <ildikov> so the question here is removing the old one I guess
17:29:15 <johnthetubaguy> apparently, not all old volumes have an attachment id already
17:29:31 <mriedem> johnthetubaguy: do you mean attachment id in nova or cinder?
17:29:33 <johnthetubaguy> if they were attached before we created attachment ids
17:29:36 <johnthetubaguy> in cinder I mean
17:29:45 <mriedem> johnthetubaguy: i think we're talking about different things
17:30:01 <mriedem> i've been told in previous meetings that each volume attached has something in the cinder volume_attachments table already
17:30:16 <johnthetubaguy> only past a certain point in time, I believe
17:30:22 <ildikov> I don't remember when that got introduced, but should be there for a while now
17:30:23 <mriedem> is havana old enough?
17:30:32 <ildikov> it is for me!
17:30:38 <mriedem> hemna wrote the schema migration so he'd know
17:31:21 <hemna> hey
17:31:28 <hemna> sorry guys, been on multiple meetings at once
17:32:32 <hemna> ok correct.
17:32:35 <johnthetubaguy> according to the user survey, 16% of users in October 2015 were on Havana, 12% are on older releases
17:32:39 <mriedem> johnthetubaguy: so for old volume attachments / BDMs we know we'll need to 'upgrade' those to the new model somehow
17:32:40 <hemna> each attachment is supposed to have a volume_attachment entry
17:32:47 <johnthetubaguy> mriedem: yeah, basically
17:32:59 <hemna> and all the original cinder volume table information was migrated when the volume_attachment table was created.
17:33:00 <johnthetubaguy> mriedem: I think we previously agreed a cinder-manage cmd to fix that up
17:33:00 <mriedem> johnthetubaguy: i left some comments on that in the spec too i believe
17:33:12 <mriedem> johnthetubaguy: i was hoping for something more automatic
17:33:20 <mriedem> i.e. online data migration
17:33:26 <johnthetubaguy> mriedem: we can call all that from the online data migrations stuff
17:33:33 <mriedem> because the spec says something about a nova-manage command calling a cinder-manager command, and that seems bad to me
17:33:57 <johnthetubaguy> so the trade off was vs adding an API thats just for the upgrade hump
17:34:12 <mriedem> i think we can migrate old attachments on (1) first access after rolling new code or (2) a periodic task in the computes for vols attached to instances on that upgraded compute
17:34:40 <mriedem> then maybe after n-2 or whatever we drop that compat code
17:35:00 <johnthetubaguy> yeah, I am thinking the same, except its nova-manage cmds to match the DB work
17:35:20 <mriedem> if nova-manage has to cast to the compute it's not going to be fun
17:35:26 <mriedem> *rpc call to the compute
17:35:43 <mriedem> when i was reading the spec i thought there was some reason we'd need to call to the compute - to get the volume connector from brick
17:35:47 <johnthetubaguy> mriedem: true, its really about staggering the system load on upgrade
17:35:50 <mriedem> which is needed in the update attachment for existing attachments
17:36:10 <mriedem> we do'nt have the connector from nova-manage, and you can only git it off the compute
17:36:19 <johnthetubaguy> yeah
17:36:20 <mriedem> so you're talking about adding an rpc call from nova-manage to compute i think
17:36:34 <mriedem> which is why i was talking about doing something more automatic
17:36:34 <johnthetubaguy> yeah, thats a "bit" evil :(
17:36:38 <mriedem> or background tasks
17:36:39 <mriedem> right
17:36:43 <johnthetubaguy> anyways, we can work through that
17:37:05 <mriedem> yeah it's all in the spec :)
17:37:12 <ildikov> :)
17:37:13 <mriedem> my wedding gift to you
17:37:19 <johnthetubaguy> mriedem: :)
17:37:34 <scottda> ha. That's right congratulations johnthetubaguy
17:38:03 <johnthetubaguy> actually just got the usb stick with all the photos on today, so thats some fun for tonight
17:38:29 <ildikov> johnthetubaguy: I guess you're asking me now to cut the meeting short :)
17:39:04 <johnthetubaguy> I can't go in the living room right now, apparently, all I can here is present wrapping noises, and the odd bit of singing
17:39:06 <johnthetubaguy> so I am good for now
17:39:14 <hemna> :)
17:39:31 <ildikov> :)
17:39:35 <johnthetubaguy> anyways, we derailed fast there :)
17:39:52 <ildikov> is there anything for the Nova spec we can/should discuss now?
17:40:03 <johnthetubaguy> cinder cmd vs cinder API?
17:40:06 <ildikov> *anything else
17:40:13 <johnthetubaguy> vs cinder lib
17:40:30 <johnthetubaguy> for that transition bit
17:40:58 <johnthetubaguy> maybe its such a mess if there is no API to "upgrade" an attachment, we should just allow us to call update?
17:41:31 <johnthetubaguy> (and we ignore the case where there is no attachment_id)
17:42:03 <ildikov> my guess here is that it's either update or create a new attachment and remove the old
17:42:18 <johnthetubaguy> oh... create a new one
17:42:29 <ildikov> I'm slightly unsure at this point what update will do with an old attachment
17:42:58 <hemna> I think the idea was to just call update
17:43:20 <ildikov> I can clarify this with jgriffith later
17:43:35 <ildikov> but I think the API should be able to cover this
17:44:17 <jgriffith> sorry... late :(
17:44:21 <ildikov> I wonder what the load will be during an upgrade and whether that can cause any issues
17:44:25 <jgriffith> been trying to deploy OpenStack
17:44:32 <jgriffith> I weep for our users
17:44:32 <scottda> ha
17:44:58 <jgriffith> johnthetubaguy update is basically a "finalize" and make the actual connection
17:45:06 <ildikov> jgriffith: now that you mentioned users I might accept this as an excuse :)
17:45:09 <jgriffith> create *can* do that at the same time if you like
17:45:18 <mriedem> i have to drop - sev1 in prod, ttyl
17:45:22 <jgriffith> but that doesn't work for Nova
17:45:45 <ildikov> mriedem: thanks for joining, ttyl
17:46:12 <ildikov> jgriffith: we are agonizing on an upgrade process
17:46:24 <jgriffith> ildikov how come?
17:46:38 <ildikov> jgriffith: in the sense of switching to the new Cinder API and get old volume attachments updated
17:47:08 <jgriffith> should be ok with a manage script
17:47:19 <jgriffith> we've minimized the differences between the two
17:47:27 <jgriffith> in terms of what's in the DB etc
17:47:33 <ildikov> jgriffith: you mean on the Nova or the Cinder side?
17:47:48 <jgriffith> remember a while back we had this whole rework to try and make them compatible
17:48:16 <jgriffith> ildikov Cinder side, Nova was a bit uglier (ie nova --> cinder OR cinder-1)
17:48:25 <johnthetubaguy> there are concerns on Nova having to directly call cinder-manage
17:48:39 <jgriffith> johnthetubaguy I wouldn't have Nova do that if it were me
17:48:51 <johnthetubaguy> basically because we need to get the connector from the specific nova-compute node
17:49:17 <jgriffith> johnthetubaguy what i'm saying is that Nova should just check which version of the API cinder supports and use the latest call
17:49:21 <johnthetubaguy> I am ok with nova-manage doing nasty things, but we can't really cheat here
17:49:36 <jgriffith> as far as existing attachments and things like the stored attachment id... for that you do a get-attachments and find it
17:49:37 <johnthetubaguy> jgriffith: this is about all the existing volumes, and trying to move them so we only use the new API version
17:49:47 <johnthetubaguy> then drop the support in Nova for the old API
17:50:02 <johnthetubaguy> right, the connector isn't stored today
17:50:02 <jgriffith> johnthetubaguy sure
17:50:12 <johnthetubaguy> so that needs pushing up
17:50:28 <johnthetubaguy> and connection_info I guess
17:50:32 <jgriffith> johnthetubaguy all you need to do after an upgrade (like nova-migrate) is call cinder "attachments_get_all_by_volume"
17:50:50 <jgriffith> johnthetubaguy that will give you the connector info and everything that you would get if you were using the "new" API calls
17:50:58 <johnthetubaguy> but cinder doesn't have the info from os-brick?
17:51:31 <jgriffith> johnthetubaguy Oh!  Yeah... that case
17:51:37 <jgriffith> johnthetubaguy so that's doable
17:51:55 <johnthetubaguy> yeah, we just haven't agreed on how yet
17:52:07 <jgriffith> johnthetubaguy we use the old call for detach etc, and just make sure we use all new calls for attach creation
17:52:19 <jgriffith> johnthetubaguy I had some code that did this... it even worked :p
17:52:36 <johnthetubaguy> I think thats what ildikov was suggesting, that seems possible
17:53:09 <jgriffith> johnthetubaguy I do think we can make this work, I'll need to get the latest update to Cinder done, and get a running cloud again to move forward
17:53:32 <ildikov> johnthetubaguy: partially as I was thinking about how to update and I think it's not what jgriffith meant
17:54:12 <jgriffith> ildikov very hard to say "what I meant". even for me :)
17:54:23 <johnthetubaguy> I like the delete/create approach though
17:54:34 <johnthetubaguy> means we do it via the API
17:54:40 <johnthetubaguy> but its not APIs just created for upgrade
17:54:57 <ildikov> jgriffith: hmm, good opportunity for me to plant ideas in your head now it seems :)
17:55:02 <johnthetubaguy> its probably a create/delete, but thats a detail for later
17:55:18 <jgriffith> johnthetubaguy I think we can solve this with the cinder api calls
17:55:35 <ildikov> I think Cinder API calls as a target is good in general
17:55:35 <jgriffith> johnthetubaguy it's annoying for a couple releases to support both but I think it's certainly possible
17:56:07 <ildikov> jgriffith: as we have users still on Havana I wonder how many is that "couple"... :(
17:56:33 <ildikov> jgriffith: it does not make it impossible of course, it was just a note
17:56:42 <johnthetubaguy> well, maybe they just don't need the delete, because the attachment doesn't exist
17:57:09 <johnthetubaguy> it looked like > 20% users where on havana or earlier in December 2015 user server
17:57:17 <johnthetubaguy> so its very likely they still have volumes attached
17:57:22 <johnthetubaguy> and want to upgrade at some point
17:57:36 <jgriffith> johnthetubaguy they can't upgrade H-->O anyway
17:57:44 <johnthetubaguy> totally
17:57:50 <jgriffith> johnthetubaguy I can barely upgrade M-->N
17:58:11 <johnthetubaguy> its just more, folks are using that thing in production
17:58:56 <johnthetubaguy> jgriffith: well, I kinda think people should use OpenStack as a service, but I would say that
17:59:08 <jgriffith> johnthetubaguy well, I do think it's completely doable, and it will work a hell of a lot better than things like Keystone V3 and Glance and .......
17:59:35 <johnthetubaguy> jgriffith: totally agreed, its totally doable
17:59:39 <jgriffith> johnthetubaguy after this lab move and redeploy I would agree with you!
18:00:12 <jgriffith> We've built something that requires PS to deploy at this point
18:00:57 <johnthetubaguy> well, there are many competing deploy solutions, I guess
18:01:20 <johnthetubaguy> doing it yourself is crazy hard
18:02:03 <johnthetubaguy> anyways, at some point we should discuss private cloud deploy optimizations, but anyways
18:03:29 <ildikov> johnthetubaguy: sounds a bit like discussing what to do with global warming at this point to me
18:03:32 <johnthetubaguy> I guess we are out of time
18:03:43 <ildikov> johnthetubaguy: and that too :)
18:03:52 <jgriffith> and lucky me... I get to go to a meeting on Open Source contribution policy :)
18:03:52 <johnthetubaguy> ildikov: yeah, thats true
18:04:01 <ildikov> does anyone here would like to have the first meeting on the 2nd?
18:04:03 <johnthetubaguy> jgriffith: I am sorry
18:04:20 <ildikov> or the 9th sounds just fine?
18:04:22 <johnthetubaguy> thats a public holiday for most folks right?
18:04:42 <ildikov> jgriffith: should I create a reason for you to miss that? :)
18:05:01 <scottda> I think the 9th
18:05:11 <scottda> Not much will be changed by the 2nd
18:05:15 <ildikov> jgriffith: unless you're the one holding it :)
18:05:54 <johnthetubaguy> scottda: a very good point
18:05:56 <ildikov> scottda: yeap, my thought exactly, just wanted to give a chance for objection if there would happen to be any :)
18:06:12 <johnthetubaguy> happy holidays, when that happens for folks
18:06:31 <ildikov> Happy Holidays everyone!
18:06:48 <ildikov> Talk to you on the 9th the latest!
18:07:27 <scottda> yup, have a good one all
18:07:43 <ildikov> Thanks for all your efforts this year and have some well deserved fun and rest in the upcoming two weeks! :)
18:08:26 <ildikov> #endmeeting