16:00:35 <jungleboyj> #startmeeting Cinder
16:00:36 <openstack> Meeting started Wed Jul 17 16:00:35 2019 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:39 <openstack> The meeting name has been set to 'cinder'
16:01:02 <jungleboyj> Today's agenda:
16:01:06 <jungleboyj> #link https://etherpad.openstack.org/p/cinder-train-meetings
16:01:10 <geguileo> hi! o/
16:01:17 <eharney> heyo
16:01:20 <jungleboyj> Courtesy ping:  jungleboyj whoami-rajat rajinir lseki carloss pots woojay erlon geguileo eharney rosmaita enriquetaso e0ne smcginnis davidsha walshh_ xyang hemna _hemna
16:01:20 <davidsha> Hey
16:01:22 <pots> morning!
16:01:46 <Luzi> o/
16:01:51 <jungleboyj> @!
16:01:51 <_pewp_> jungleboyj ( ´ ▽ ` )ﾉ
16:01:58 <enriquetaso> o/
16:02:10 <rosmaita> o/
16:02:21 <whoami-rajat> Hi
16:03:10 <jungleboyj> Hey everyone.  Wait one more minute to see if we get anyone else.
16:04:06 <jungleboyj> Ok.  Guessing this is it for today.
16:04:24 <jungleboyj> I know Sean is at a conference and didn't think he could make it.
16:04:41 <jungleboyj> #topic Announcements
16:04:59 <jungleboyj> The usual reminder that we are almost to Milestone 2.
16:05:10 <jungleboyj> #link https://releases.openstack.org/train/schedule.html
16:05:34 <jungleboyj> So, we have Spec freeze next week and driver merge deadline.
16:05:44 <jungleboyj> I have more discussion on specifics there later.
16:06:03 <jungleboyj> Also a reminder that we have the deadline to show drivers trying to run with Py3.7y
16:06:08 <jungleboyj> *Py3.7
16:06:30 <jungleboyj> I know people have been working on this as I have heard frustration in the channel.
16:06:51 <jungleboyj> Is there anyone that has their CI working with Py3.7 that is here?
16:07:22 <pots> I've got it working but not posting comments just yet.
16:07:29 <jungleboyj> pots:  Awesome.
16:07:58 <jungleboyj> Would encourage those of you that have it working to keep an eye on the Cinder channel/mailing lists so you guys might be able to help each other.
16:08:39 <pots> A word of caution to others, I tried moving up to bionic to get Python 3.7 support but found that the kernel crashed when starting nested VMs, so I reverted to Xenial and just used the deadsnakes ppa for Python 3.7.
16:09:34 <jungleboyj> I will be going through the CI results in the next couple of weeks to see who all is running with Py3.7.  I have seen at least one py3.5.  I am not going to mark a driver unsupported for that, but by the end of Train it will need to be py3.7 or it will get marked.
16:10:41 <jungleboyj> Thanks for the info pots
16:10:56 <jungleboyj> Lets me to the main agenda
16:11:24 <jungleboyj> #topic stable releases update (and controversy)
16:11:28 <jungleboyj> rosmaita:
16:11:37 <rosmaita> the update is we've cut a bunch of releases from the stable branches, see the etherpad for details
16:11:43 <rosmaita> #link https://etherpad.openstack.org/p/cinder-releases-tracking
16:11:50 <rosmaita> now the controversy
16:11:57 <rosmaita> cinder stable/rocky and stable/queens are being held up over a question about a backport to stable/rocky
16:12:07 <rosmaita> backport in question is "Declare multiattach support for HPE MSA"
16:12:14 <rosmaita> #link https://review.opendev.org/#/c/639867/
16:12:22 <rosmaita> backport was allowed by cinder team because there wasn't a code change, just a change to a flag to indicate multiattach support
16:12:31 <rosmaita> the objection is: has the already existing code been tested with stable/rocky?
16:12:40 <rosmaita> the change in https://review.opendev.org/#/c/639867/ is to the dothill driver, which isn't a currently supported driver
16:12:46 <rosmaita> but it *is* the base class for supported drivers that run 3rd party CI: Lenovo FC/ISCSI and HPMSA FC/ISCSI
16:12:59 <rosmaita> but, unless i'm reading the logs incorrectly, those CIs are not currently testing multiattach on master (so I doubt they are testing it in the stable branches)
16:13:06 <rosmaita> recent test logs so you can check my work:
16:13:08 <jungleboyj> pots ^^^
16:13:13 <rosmaita> #link http://os-logs.tristero.net/hpmsa-ci/93/668793/13/HPMSA-FC/console.log.out
16:13:17 <rosmaita> #link http://os-logs.tristero.net/hpmsa-ci/93/668793/13/HPMSA-iSCSI/console.log.out
16:13:27 <rosmaita> #link http://os-logs.tristero.net/lenovo-ci/93/668793/13/Lenovo-iSCSI/console.log.out
16:13:33 <rosmaita> #link http://os-logs.tristero.net/lenovo-ci/93/668793/13/Lenovo-FC/console.log.out
16:13:46 <rosmaita> i thik we have two options here:
16:13:52 <rosmaita> (option-1) revert the change from stable/rocky
16:14:00 <rosmaita> mriedem has already put up a patch for that
16:14:07 <rosmaita> #link https://review.opendev.org/#/c/670086/
16:14:14 <rosmaita> (option-2) the proposed release is 13.0.6; change it to 13.1.0 to indicate a new feature
16:14:41 <rosmaita> that's a lot to digest, so i'll wait a few minutes to hear what people think
16:15:34 <jungleboyj> rosmaita:  Thanks.  So I would like to hear from pots what level of testing they did since he is the one who worked on that.
16:16:15 <eharney> the version bump option is kind of appealing to dodge the issue of landing a change in stable and then undoing it -- but as part of this... how much do we know about how well this works?  has it been thoroughly tested?
16:16:40 <rosmaita> i'm worried about the test coverage, tbh
16:17:01 <jungleboyj> eharney:  I am ok with the version bump.  I am curious if this was backported based on customer request.
16:17:23 <rosmaita> we may need to update the driver doc that sean just added to include you must run the multiattach tests in gate to be allowed to turn the flag on in a driver
16:18:03 <pots> The CI isn't testing it explicitly no.  And yes, it was a customer request to backport it.
16:18:04 <jungleboyj> rosmaita:  That is a good thought.
16:18:09 <eharney> we've found that just passing multiattach tempest tests is not sufficient to know that the feature even works at a basic level
16:18:14 <jungleboyj> pots:  Did you test it manually?
16:18:26 <eharney> so it would be good to hear about hands-on testing with it
16:19:09 <rosmaita> agree
16:19:51 <jungleboyj> rosmaita:  The 3rd Party CI requirement sounds like a good midcycle topic.
16:20:17 <rosmaita> ok, i will add it to the etherpad
16:20:21 <pots> i did some manual testing but frankly I don't remember enough that i think it should count.  i would welcome some guidance into what would be considered adequate testing.
16:20:57 <eharney> the scenario that broke on a handful of drivers was, attach to two instances, detach from one, and then make sure I/O continues to work as expected on the other instance
16:21:16 <jungleboyj> eharney:  ++
16:21:26 <jungleboyj> rosmaita:  Added.
16:21:27 <eharney> at least three drivers failed to do that without falling apart, which is a pretty minimal requirement for this to be a working feature
16:22:21 <jungleboyj> Agreed.
16:22:22 <LiangFang> o/
16:22:51 <pots> i'll check that out today.
16:23:04 <jungleboyj> pots:  Ok.  Thank you.
16:23:24 <rosmaita> we can hold up the release another week
16:23:34 <jungleboyj> rosmaita: eharney  If pots manually verifies the environment then are we ok with going for the 13.1.0 bump?
16:24:01 <rosmaita> i think that would be a good compromise
16:24:11 <jungleboyj> We probably shouldn't have backported it, but it was kind of a bug that they didn't have the flag for what was supported.
16:24:14 * jungleboyj sighs
16:24:30 <eharney> is the idea there that's also a promise of "we won't backport this kind of change in the future"?
16:24:45 <rosmaita> i think we should discuss that
16:24:52 <jungleboyj> eharney:  Yeah, I think we should probably say that.
16:25:06 <rosmaita> i think with the drivers we have a different situation that with the "regular" code
16:25:11 <rosmaita> *than
16:25:48 <jungleboyj> Yeah. That is a longer future discussion.
16:25:52 <rosmaita> what i mean is that it may be ok to backport some feature-esque changes, nothing major
16:25:53 <eharney> it was quite nice when we were a little more liberal with backports to driverfixes branches... not having that as an option leaves kind of a gap for what to do now
16:26:13 <jungleboyj> eharney:  Yeah,  I was just thinking that.
16:26:24 <jungleboyj> The longer lived branches are a blessing and a curese.
16:26:33 <rosmaita> so the key point is that we're not promising anything by what we do with stable/rocky for this next release
16:26:57 <jungleboyj> rosmaita:  Yes.
16:26:59 <eharney> right
16:27:13 <rosmaita> ok, cool, just want to make sure that's in the minutes
16:27:14 <jungleboyj> #action pots to test the requested multi-attach scenario
16:27:35 <jungleboyj> #action we will go with 13.1.0 once the results have been reported.
16:27:49 <rosmaita> The People don't seem to be demanding new stable/rocky and stable/queens releases ATM, so i think it is ok to wait
16:27:53 <jungleboyj> Everyone ok with that response?
16:28:06 <jungleboyj> rosmaita:  ++
16:28:16 <rosmaita> i am good with that response
16:28:40 <eharney> yeah
16:28:49 <jungleboyj> Ok.  Cool.  Yay for resolving controversy.
16:28:56 <rosmaita> that's all from me, then
16:29:11 <jungleboyj> rosmaita:  Thank you for keeping up with those releases.
16:29:22 <rosmaita> np, glad to stir things up
16:29:29 <jungleboyj> #topic Spec:  Leverage compression hardware accelerator
16:29:34 <jungleboyj> LiangFang:
16:29:37 <LiangFang> hi
16:29:45 <jungleboyj> Hello
16:29:50 <LiangFang> I have reworked with the spec
16:30:06 <jungleboyj> #link https://review.opendev.org/#/c/652275/
16:30:16 <LiangFang> it seems the spec will be refused to check-in on 7-22?
16:30:49 <LiangFang> July 22 is the latest day for train release, right/
16:31:08 <jungleboyj> LiangFang:  I wouldn't say that.  It depends on if eharney and rosmaita are ok with the updates you have made.
16:31:24 <rosmaita> i will read through this afternoon
16:31:31 <LiangFang> thanks
16:31:55 <LiangFang> eharney: I and Mazheng did test for NFS and ceph as backend
16:32:05 <jungleboyj> I will do the same.  I know we talked about this at the PTG and I thought had worked through the majority of the issues.  Hope we can get it merged.
16:32:06 <LiangFang> it should work
16:32:25 <eharney> it works because those drivers just don't end up using the hardware compressor path at all?
16:33:16 <LiangFang> the volume will be mapped to a path
16:33:39 <rosmaita> LiangFang: i do have a question about nova -- i guess we need to be sure that nova can handle a compressed container_format gracefully -- i'm not sure what it does now if you have a container_format it doesn't understand
16:33:41 <LiangFang> and this path is a normal file path
16:34:13 <eharney> i don't think that's accurate for nfs with snapshots, but i might be missing something
16:35:46 <LiangFang> rosmaita: nova with new container_format, we will do test today, thanks
16:35:55 <LiangFang> and let you know
16:36:00 <rosmaita> LiangFang: thanks!
16:36:53 <LiangFang> eharney: to just use nfs as backend storage, no snapshots yet
16:37:22 <LiangFang> we only tested nfs without snapshots
16:38:27 <LiangFang> eharney: do you mean a volume is created from NFS, and then we create snapshot on this volume, and then upload this volume as image?
16:38:28 <jungleboyj> Ok.  So, I think there some updates needed to the spec and we will review.
16:38:58 <jungleboyj> And some testing done.
16:39:06 <eharney> LiangFang: yes
16:39:10 <eharney> i'll check out the spec again
16:39:19 <LiangFang> eharney: ok, we will do this test today
16:39:37 <LiangFang> eharney: thanks
16:39:57 <jungleboyj> Changes need to be merged by 7/25
16:40:09 <LiangFang> jungleboyj: OK
16:40:22 <jungleboyj> So, we will do one last check-in on these things next Wednesday.
16:40:27 <LiangFang> we will do our best to catch train release
16:40:45 * jungleboyj giggles.  Catch the Train
16:41:17 <LiangFang> :)
16:41:31 <jungleboyj> Ok.  So lets move on.
16:41:41 <LiangFang> currently we have 2 engineer working on this
16:42:01 <jungleboyj> #topic Review prep for Milestone 2.
16:42:16 <jungleboyj> So, we have a couple of drivers that I have been pinged about:
16:42:24 <jungleboyj> First, re-adding the Infortrend driver.
16:42:34 <jungleboyj> Their CI is running and passing again.
16:42:47 <jungleboyj> #link https://review.opendev.org/#/c/523659/21
16:43:10 <jungleboyj> The vendor indicates that they plan to continue support.
16:43:23 <jungleboyj> They have updated the driver to support the new exception handling.
16:43:33 <jungleboyj> In driver instead of in cinder/exception.
16:43:44 <jungleboyj> I am ok with adding it back in unless others have objections.
16:44:09 <jungleboyj> Know it is a little late but given that it is a re-add I don't feel a need to nit-pick.
16:44:34 <jungleboyj> I voted in the driver.  Please add your votes.
16:44:51 <jungleboyj> #link https://review.opendev.org/#/c/671195/
16:44:58 <jungleboyj> Now this one is more fun.  :-)
16:45:04 <jungleboyj> pots: You want to explain?
16:45:21 <pots> Sure.  So I need to add a driver for Seagate storage arrays.
16:45:41 <pots> Seagate acquired dothill a few years ago and took over maintenance of the dothill driver and the drivers that subclass it.  The dothill driver became unsupported at that point, which causes a little confusion because other drivers subclass the dothill driver.
16:45:55 <pots> Rather than adding a new Seagate driver and subclassing the dothill driver, I submitted a patch (WIP) to re-enable the dothill driver and rename it to "Seagate".
16:46:09 <pots> The patch is cosmetic changes only, and includes documentation, release note and support matrix updates, and patches to the the HPMSA and Lenovo drivers to use the new class names.
16:46:44 <pots> So the end result is that the dothill driver is gone, replaced by a "supported" Seagate driver.
16:47:34 <pots> This seemed like a better long term solution than just subclassing the dothill driver again.
16:47:46 <jungleboyj> So, I think that this is actually a good change as it is getting rid of the weird driver that isn't a driver.
16:48:02 <jungleboyj> The driver has been tested all along as part of the HPE MSA and Lenovo CIs.
16:48:14 <jungleboyj> pots:  Will add a CI job for it.
16:48:23 <jungleboyj> They have been active and kept their CI running.
16:48:48 <jungleboyj> So, it is a little late in the game to 'add' the driver but I think it is low risk and there is a good argument for it.
16:49:04 <eharney> this is ok, but i suspect that in the long run it will be easier to keep all this clean by moving most of the code to a common class that all three can inherit from instead of inheriting drivers from other drivers
16:49:08 * jungleboyj shakes my finger at people for coming in at the last minute.  ;-)
16:49:28 <eharney> i wouldn't hold it up for that now though
16:49:32 <rosmaita> i agree with eharney
16:49:47 <eharney> this is bound to cause some confusion and messes later
16:49:49 <jungleboyj> eharney:  Well, that is how it currently is and it has caused confusion because we have one driver that can't actually be run by itself.
16:49:59 <rosmaita> we can worry about it when seagate decides to stop supporting it :P
16:50:02 <eharney> it's not
16:50:12 <jungleboyj> smcginnis: Didn't like what we are currently doing.
16:50:15 <eharney> because the thing it was inheriting from currently was still a "driver"
16:50:17 <jungleboyj> eharney: Ok, I don't understand.
16:50:18 <eharney> iirc
16:50:34 <jungleboyj> eharney:  No, it was just a dothill library.
16:50:41 <jungleboyj> Lenovo and HPE MSA used it.
16:50:56 <eharney> named DotHillISCSIDriver
16:51:09 <pots> the dothill driver is an actual driver, it just contains extra code now to prevent it from being invoked without subclassing.
16:51:21 <jungleboyj> pots:  Right.
16:51:56 <jungleboyj> eharney:  So what are you proposing for the future?
16:52:00 <rosmaita> it is definitely a plus that it will be a "real" driver and have CI
16:53:09 <eharney> the proposal is to make a library of the common shared code that isn't a driver
16:53:37 <jungleboyj> rosmaita: Ok.  So we are short on time.  It sounds like we are in agreement that this is good for now and making a library for the shared code in the future could be good.
16:53:45 <jungleboyj> eharney:  Ok.
16:53:56 <rosmaita> jungleboyj: that is my position in a nutshell
16:54:07 <jungleboyj> Something else to discuss for the mid-cycle.  How we would like that to happen.
16:54:49 <jungleboyj> Ok, specs that are waiting to merge.
16:55:33 <jungleboyj> #link https://review.opendev.org/#/c/651480/  The Untyped vol to default vol type is close.
16:55:45 <jungleboyj> Would be nice if we could agree and merge that.
16:56:07 <jungleboyj> I think the volume rekey is good to go:
16:56:19 <jungleboyj> #link https://review.opendev.org/#/c/664973/
16:56:34 <jungleboyj> There are two small comments that need to be addressed and then we can merge it.
16:56:42 <eharney> yeah i'll tweak the "work items" list
16:56:52 <jungleboyj> eharney:  Ok.  Great and then I will merge it.
16:57:17 <jungleboyj> Last.  The Image Encryption sepc:
16:57:28 <jungleboyj> #link https://review.opendev.org/#/c/608663/
16:57:40 <jungleboyj> So, I think it is down to whether eharney 's comments are addressed.
16:58:08 <Luzi> eharney, did you read mhen
16:58:14 <Luzi> s comment
16:58:23 <eharney> my only outstanding concern was more about the general question of having multiple ways that images are encrypted, but we've collectively decided that that is ok, so i'm ok with this one
16:59:10 <eharney> removed my -2, will give it another read-over soon
16:59:28 <Luzi> eharney, thank you
17:00:26 <jungleboyj> eharney:  Ok.  Cool.  So, I will read through it after the meeting.  We will try to get this merged for you Luzi
17:00:42 <eharney> Luzi: thanks for tolerating all of my complaining on this one :)
17:01:01 <jungleboyj> If you can provide updated patch once we have made comments.
17:01:12 <jungleboyj> :-)
17:01:23 <jungleboyj> Ok.  Where did the time go.  It is 12:01 already.
17:01:34 <jungleboyj> So, thank you everyone for a productive meeting.
17:01:38 <Luzi> jungleboyj, i will do so
17:01:42 <jungleboyj> Lets get specs and drivers merged.
17:01:49 <jungleboyj> Look forward to talking to you next week.
17:02:07 <jungleboyj> Reminder to check the etherpad on the mid-cycle for next month.  I am booked.
17:02:22 <jungleboyj> #link https://etherpad.openstack.org/p/cinder-train-mid-cycle-planning
17:02:38 <jungleboyj> Don't forget to vote for Shanghai  presentations!
17:02:47 <jungleboyj> Talk to you all next week.
17:02:51 <jungleboyj> #endmeeting