Friday, 2015-05-29

*** BitSmith has joined #openstack-manila01:17
*** vbellur has joined #openstack-manila01:43
*** vbellur has quit IRC01:48
*** BitSmith has quit IRC01:51
*** zhidong has joined #openstack-manila04:09
*** rraja has joined #openstack-manila04:13
*** sks has joined #openstack-manila04:29
*** Triveni has joined #openstack-manila04:55
*** zhidong has quit IRC05:00
*** Triveni_ has joined #openstack-manila05:40
*** Triveni has quit IRC05:43
*** nkrinner has joined #openstack-manila05:44
*** Maike has joined #openstack-manila06:10
*** sks has quit IRC06:33
*** lpetrut has joined #openstack-manila06:38
*** kaisers1 has joined #openstack-manila06:39
*** sks has joined #openstack-manila06:48
*** Maike_ has joined #openstack-manila06:57
*** Maike_ has quit IRC06:59
*** Maike_ has joined #openstack-manila06:59
*** Maike has quit IRC07:00
*** kaisers2 has joined #openstack-manila07:03
*** kaisers2 has quit IRC07:06
*** Maike_ has quit IRC07:52
*** Maike has joined #openstack-manila08:02
*** Maike_ has joined #openstack-manila08:10
*** Maike has quit IRC08:14
*** Maike has joined #openstack-manila08:14
*** kaisers1 has quit IRC08:15
*** Maike_ has quit IRC08:15
*** lpetrut has quit IRC08:47
openstackgerritValeriy Ponomaryov proposed openstack/manila: Transform share and share servers statuses to lowercase  https://review.openstack.org/18652008:58
openstackgerritValeriy Ponomaryov proposed openstack/python-manilaclient: Add rw functional tests for shares  https://review.openstack.org/18021909:07
*** Maike has quit IRC09:19
*** lpetrut has joined #openstack-manila09:36
*** lpetrut1 has joined #openstack-manila09:39
*** lpetrut has quit IRC09:41
openstackgerritsac proposed stackforge/puppet-manila: Add share driver options.  https://review.openstack.org/18669909:46
*** Maike has joined #openstack-manila10:14
*** vbellur has joined #openstack-manila11:08
*** lpetrut1 has quit IRC11:12
*** sgotliv has quit IRC11:53
*** ganso_ has joined #openstack-manila11:53
*** rraja has quit IRC11:54
*** nkrinner has quit IRC12:02
*** rcallawa_ has joined #openstack-manila12:02
*** rcallawa_ has quit IRC12:05
*** nkrinner has joined #openstack-manila12:05
*** marcusvrn has joined #openstack-manila12:14
*** lpetrut has joined #openstack-manila12:38
*** bswartz has quit IRC12:39
*** marcusvrn1 has joined #openstack-manila12:40
*** fthiagogv has joined #openstack-manila12:41
*** marcusvrn has quit IRC12:44
*** lpabon has joined #openstack-manila12:44
*** marcusvrn1 has quit IRC12:48
*** marcusvrn has joined #openstack-manila12:48
*** sks has quit IRC12:51
*** marcusvrn has quit IRC12:55
*** marcusvrn has joined #openstack-manila13:08
*** nkrinner has quit IRC13:10
*** xyang1 has joined #openstack-manila13:11
*** lpabon has quit IRC13:12
*** akshai has joined #openstack-manila13:15
openstackgerritsac proposed stackforge/puppet-manila: Add share driver options.  https://review.openstack.org/18669913:24
*** mtanino has joined #openstack-manila13:26
*** cknight has joined #openstack-manila13:29
*** sgotliv has joined #openstack-manila13:30
*** vbellur has quit IRC13:36
*** bswartz has joined #openstack-manila13:44
bswartzvponomaryov u_glide ganso_: ping13:46
vponomaryovbswartz: pong13:46
bswartzI had a conversation with ganso_ yesterday about share migration and the Manila DB13:46
bswartzwe both agreed that 2 database records during the migration makes more sense than trying to use a single DB record13:47
bswartzthere is too much state that we would need to store while the migration was in progress, without forcing every driver to change and have a special create_migration_destination_share() method13:48
bswartzand because we want the fallback migration method to work with all drivers, it's cleaner to just create a second share temporarily13:48
bswartzwe will have the same issue that cinder does regarding ownership of the temporary share13:49
bswartzand the same issues regarding the IDs of the two shares13:49
bswartzthere are 2 proposals to address the ID problem:13:49
bswartz1) require all drivers to support a change-ID operation for shares, which will be much easier than in the past due to private driver share data13:50
bswartz2) implement a second ID column in the database so every share has an "API ID" and a "Driver ID" which are different and the manager deals with mapping between the two13:50
ganso_bswartz: pong13:51
bswartzOption 2 is the least disruptive for drivers, but could require a lot of change in manila core -- also cknight pointed out that 2 IDs would make manual debugging a lot more confusing13:51
bswartzOption 1 seems like it could work, and driver could use the generic driver as a model for how to handle mapping of IDs to the object on the backend13:52
vponomaryov2) is redundant, you already can store such values in private driver storage13:52
vponomaryovfor any amount of drivers13:53
bswartzvponomaryov: the idea behind option 2 would be that drivers would NEVER see the API ID, and users would NEVER see the driver ID13:53
bswartzwe would have to implement some magic to ensure that13:53
vponomaryovAPI ID is current uuid of a share13:53
vponomaryov?13:53
ganso_options 2 would be enforcing the separate IDs13:53
bswartzAPI ID is what the user sees through the REST API13:53
vponomaryovok, but not as new DB columns13:54
ganso_option 1 we would be providing the means and guidelines for drivers, but we would not be enforcing them to correctly use it13:54
bswartzwell we would need to store both13:54
bswartzin a migration this is what would happen13:54
bswartzUser creates a a new share, it gets API ID 1 and Driver ID 213:54
bswartzUser does an action which causes the share to migrate13:54
*** lpabon has joined #openstack-manila13:54
bswartzManager creates a new share with API ID 1 and Driver ID 313:55
bswartzmanager copies the data13:55
bswartzmanager deletes the first share13:55
bswartzthe second share becomes visible13:55
bswartzin this model, the driver never realizes that share 2 and share 3 are considered the same by the user13:56
*** timcl has joined #openstack-manila13:56
bswartzand the user never realizes that there were temporarily 2 shares with ID 113:56
bswartzall of the complexity is hidden inside the manager13:56
u_glideoption 2 looks destructive for manila13:57
ganso_^ in fact there was never 2 shares with the same API ID...13:57
bswartzOption 2 is what the cinder team has done without realizing it13:58
bswartzthey had some bugs, but they were able to fix them13:58
vponomaryovbswartz: looks like we came to situation when we need to rework DB schema - separate "data" part of a share and "identification". It will be useful for migration and future replication13:58
ganso_vponomaryov: +113:59
u_glideit's better to move ownership of private driver data for particular share from one driver to another13:59
bswartzI can see benefits of having 2 IDs per share13:59
ganso_we also cannot bet everything on private driver storage13:59
bswartzbut the downside is complexity and confusion for developers13:59
bswartzu_glide: private driver storage is intended to be for a single driver only13:59
bswartznetapp's private data won't make any sense to HDS, and vice versa14:00
bswartzduring a share migration, we would need to wipe the private data to avoid confusion in the driver code14:00
u_glidevponomaryov:+1 good idea14:01
vponomaryovbswartz: I guess if we do solution for future "replication", then migration will be case of it14:01
ganso_bswartz: as we discussed yesterday, as far as I know, HDS backend would never be able to read netapp's private driver storage14:01
ganso_bswartz: u_glide can confirm this14:02
vponomaryovganso_: false14:02
vponomaryovganso_: driver takes ID and requests data by IT14:02
vponomaryovganso_: so, if you provided share ID, driver can read/write data14:02
u_glidebswartz: sometimes it could, for example when migrate share from "old" company solution to "new"14:03
u_glidevponomaryov is wrong, private driver data isolated on host level14:03
*** Maike_ has joined #openstack-manila14:03
bswartzu_glide: I'm not aware any drivers that need to do that today though14:04
*** Maike_ has quit IRC14:04
bswartzI think it's best to start from the assumption that nothing outside the driver that created the private share data will ever read it14:04
u_glidebswartz: agree14:04
*** Maike_ has joined #openstack-manila14:04
*** mtanino has quit IRC14:05
u_glideI think we should take approach proposed by vponomaryov and carefully design new changes14:05
*** rushil has joined #openstack-manila14:05
ganso_vponomaryov:  https://review.openstack.org/#/c/176877/14/manila/share/drivers_private_data.py LN 9214:06
bswartzI'm not clear on how vponomaryov's proposal differs from mine14:06
vponomaryovbswartz, u_glide, ganso_: approach is following: create new DB table for "instances"  of a share taht will be related to "identity" share table (existing)14:06
bswartzoh you're proposing new table too14:06
bswartzcknight: ping14:06
vponomaryovbswartz, u_glide, ganso_: in that case we will be able to migration as subcase of replication14:06
cknightbswartz: pong14:07
vponomaryov*to do14:07
bswartzcknight read history14:07
*** Maike has quit IRC14:07
vponomaryovganso_: yes, I forget for that time about filter by host, sorry14:07
bswartzcknight: vponomaryov is proposing the 2 UUID option on steroids14:07
u_glidevponomaryov idea about new "share_instance" DB  table sounds14:07
u_glidegood14:08
bswartzI think it's worth considering14:08
bswartzit's not conceptually different from my 2 UUID proposal, but it's cleaner in terms of implementation14:09
bswartzand given that cinder has proven that the 2 UUID option can work, I'm not very nervous about it if we implement it early in liberty and test the hell out of it14:09
bswartzI'll have to give some thought to how this might work with replication14:10
u_glidebut this new table will introduce new concept in manila - "instance of share", so proposal are different14:10
bswartzyes but we would never expose it directly to the user, right?14:10
vponomaryovexactly possibility to have more than one instance of a share is an issue for migration case14:11
vponomaryovbswartz; only by multiple export locations14:11
u_glidebswartz: right14:11
*** sgotliv has quit IRC14:11
u_glidevponomaryov: +114:11
bswartzwell hold on14:11
bswartzthe export locations would be tied to the instances, not the user visible share14:11
bswartzin the migration case, you would need to actually mount the 2 instances and copy data from one to the other14:12
bswartzthe user would only ever see the export locations for the "active" instance of a share14:12
vponomaryovmount one export location to another14:12
vponomaryov/to/and/14:12
u_glidebswartz: instance of share could be 'hidden' for user14:13
ganso_we need to also consider what effort will be required from driver vendors, the best would be to minimize that14:13
ganso_multiple export locations are going to be exposed to users, instances should not... instances need to be managed by core code for migration/replication purposes14:14
*** lpetrut1 has joined #openstack-manila14:15
ganso_I think the instances table can be implemented a way that requires minimum, if any, effort from drivers14:15
*** lpetrut has quit IRC14:16
ganso_there is still one small detail14:16
bswartzyes I think if we implemented share instances then drivers would just interact with those and they would look identical to existing shares from the driver's perspective14:17
vponomaryovshare instances list should be possible to get by admin14:17
vponomaryovas he is resource maintainer14:17
bswartzyeah the admin should be able to see it all14:18
ganso_currently, driver vendors use API ID for their physical shares, for instance: as soon as the request to create a share reaches the driver code, it needs to decide on an ID to store in its backend, so it uses the API ID.14:18
ganso_how do we make they use the instance ID?14:19
bswartzganso_: when it reached the driver the field would just be called "ID"14:20
bswartzthe manager would hide everything that the driver didn't need to know14:20
ganso_bswartz: but would it be in fact the instance ID?14:20
bswartzin this proposal, yes14:20
vponomaryovganso_: oh, it is not a problem at all. Drivers still will get param "share", but it will have additional attributes14:20
*** rushil has quit IRC14:21
vponomaryov"identity" share will have subattrs for instances of shares14:21
vponomaryovso, driver interface should be the same14:21
vponomaryovdrivers just can start consider new attrs14:21
vponomaryovat some moment14:21
ganso_so we are not going to trick drivers by providing the instance ID instead of API ID? they are going to have to update the code14:22
ganso_ /s/the code/their code14:22
cknightganso_: "trick drivers" == "make driver authors miserable"14:23
*** rcallawa_ has joined #openstack-manila14:23
cknightganso_: If the UUID from 'manila list' doesn't match what I see on my backend storage, that's gonna be awful.14:23
ganso_cknight: that is the problem14:24
*** rcallawa_ has quit IRC14:24
ganso_in the migration demo I displayed at design summit, share X which previously had a export location of "/shares/X", after migration, it had "/shares/Y"14:25
vponomaryovwait, what tricks do you mean? With implementation of instances no tricks will be needed14:25
ganso_Y would be other ID we are talking about14:25
vponomaryovganso_: you provided export location it has nothing to ID14:26
ganso_the generic driver back then used the API ID as export location name14:26
vponomaryovand?14:27
u_glidevponomaryov:  +1 :)14:27
vponomaryovit does not make you use two different IDs14:27
vponomaryovfor one share14:27
vponomaryovmount volume to old path and that all14:28
vponomaryovin migration logic14:28
ganso_when we are going to create the new share on destination backend, unless we skip the API creating the new ID and use the same API ID, with a new instance ID, the driver will create a new share and post updates to the original API ID DB entry the source share is owner of14:28
cknightUnless there is a use case we haven't considered, having a separate shares instances table seems overcomplicated.  I still prefer requiring drivers to be able to change the UUID of a share.  It's not complicated, and they can either use the private storage mechanism or direct backend calls to achieve that.14:29
*** rushil has joined #openstack-manila14:30
bswartzcknight: vponomaryov suggested that it might benefit the replication case as well14:30
u_glide cknight: but we also need solution compatible with AZ and replication14:30
bswartzI haven't given that much thought yet14:30
cknightbswartz, u_glide: understood.  Let's consider everything before deciding this one.14:31
vponomaryovbswartz: I bet you will agreee after giving it a thought14:31
bswartzyeah I need to give this a lot of thought14:31
bswartzvponomaryov: I have a separate question14:31
bswartzhttps://github.com/openstack/manila/blob/master/manila/db/sqlalchemy/models.py#L18314:32
bswartzwhat is the purpose of the availability_zone column in the shares table?14:32
vponomaryovnone14:32
cknightvponomaryov: Isn't that a Cinder holdover?14:32
vponomaryovleft-over from cCinder14:32
u_glide cknight: +114:32
bswartzokay that's what I suspected14:32
bswartzjust checking14:32
cknightbswartz: Might as well leave it there for now, since removing it is a breaking change, and it might be needed in the near term anyway.14:33
vponomaryovlet it be, for future purpose14:33
bswartzI just wanted to make sure it wasn't used for anything -- we need a plan for how to handle AZs and it might involve simply reusing that column14:33
cknightIf we do choose to remove it, we can do so safely after adopting microversions.14:34
vponomaryovbswartz: AZs are expected to be attr of "instance" table14:34
bswartzvponomaryov: we haven't agreed to that proposal yet ;-)14:35
bswartzit's an interesting idea with a lot of implications14:35
vponomaryovbut it is obvious14:35
u_glideseems like we need detailed plan for feature sprints :)14:35
*** esker has joined #openstack-manila14:35
ganso_yes, we need a proposal14:35
bswartzI'm sensitive to cknight's concern about developer confusion with additional IDs14:35
u_glide/s/feature/future14:36
vponomaryovbswartz: why additional?14:36
u_glideinstance id's will be hidden for drivers14:36
vponomaryovbswartz: instance IDs will be used for export locations, and identity ID for user-facing get/delete , etc...14:36
ganso_as far as I understood the instance table would be managed by core code only14:36
bswartzvponomaryov: today when we create a share using the netapp driver, you can run "manila list" and see the ID, and go to the backend and see a flexvol with a name matching that ID14:37
bswartzthis is a convenience for developers14:37
cknightganso_: If the instances table is used for replicated shares, the drivers would definitely need to see it.14:37
bswartzthat would go away with the split ID scheme14:37
ganso_API request ID X => core code gets instance ID A using API ID X => provide ID A to drivers, since it will be the driver's ID14:38
bswartzcknight: maybe or maybe not -- if we do things cleverly it might be avoidable14:38
u_glidebswartz: admin API manila share-instance-list will address this problem14:38
vponomaryovu_glide: +billion14:38
ganso_bswartz: +1 I think that needs to go away14:39
bswartzthat's an option14:39
vponomaryovcknight: don't forget about admin API with possibility to see instances14:39
bswartzganso_: what needs to go away?14:39
*** Maike_ has quit IRC14:39
vponomaryovcknight: ops will use it too14:39
ganso_u_glide: +1 +1 +1 lol14:39
ganso_bswartz: the convenience14:42
ganso_bswartz: sorry did not see your message14:42
ganso_cknight: btw, my message above about providing ID A to drivers, so it can use as its ID, is what I called "tricking" drivers14:44
ganso_cknight: I would prefer if drivers had a way of saying "for API ID X, I have the ID A which I store in my private driver storage, so I know all about it"14:45
ganso_cknight: vponomaryov's idea of continue to provide the API ID X but with additional attributes that contains the active instance ID A would need a small update to driver code, same as private driver storage14:46
ganso_(is everyone receiving my messages or has my IRC dropped and I have not yet realized?)14:47
vponomaryovits ok14:47
ganso_vponomaryov: thanks14:47
vponomaryovs/its/it's/ =)14:48
cknightganso_: thanks, I think I understand.  I'm just hesitant to introduce a whole new table and extra set of IDs just to solve the migration problem.  I'd like to see us consider additional use cases, such as replication, to see if there are other reasons to add the extra layer of complexity.14:54
*** esker has quit IRC14:59
*** esker has joined #openstack-manila15:00
*** absubram has joined #openstack-manila15:02
*** absubram has quit IRC15:04
*** absubram has joined #openstack-manila15:04
*** mtanino has joined #openstack-manila15:12
openstackgerritsac proposed stackforge/puppet-manila: Add share driver options.  https://review.openstack.org/18669915:18
*** lpetrut1 has quit IRC15:24
*** chen12 has quit IRC15:44
*** chen12 has joined #openstack-manila15:45
*** lpetrut has joined #openstack-manila15:51
*** absubram has quit IRC15:51
*** absubram has joined #openstack-manila15:56
*** timcl has quit IRC15:57
*** rushil has quit IRC15:58
*** dhellmann has joined #openstack-manila16:08
dhellmannbswartz: I would like to cut a release of python-manilaclient on Monday to unblock the work on dropping the oslo namespace packages. The changes that will be released are listed in http://paste.openstack.org/show/245107/ -- Does that work for you?16:10
bswartzdhellmann: from what branch?16:10
dhellmannbswartz: master16:10
dhellmannas 1.2.016:10
bswartz1.1.X is the liberty series16:11
bswartzwhy not call it 1.1.1?16:11
dhellmannsemver16:11
bswartzsemver?16:11
dhellmannyou have feature additions and requirements changes so the minimum version needs to be bumped -- http://docs.openstack.org/developer/pbr/semver.html16:12
bswartzhmm16:12
bswartzokay that works for me16:12
dhellmannwe're trying to get all of the libraries, including clients, to standardize on the way we convey info through the version numbers16:12
bswartzdoes anything need merging beforehand?16:12
dhellmannthat's up to you, I'll cut from HEAD on Monday morning US Eastern16:13
bswartzokay16:13
dhellmanncool, thanks!16:13
bswartzI will be unreachable at that time but go ahead and push a tag16:14
dhellmannok, I'll find some of the other cores if we run into issues16:14
*** timcl has joined #openstack-manila16:28
*** esker has quit IRC16:38
*** lpetrut has quit IRC16:40
*** lpabon has quit IRC16:44
*** lpabon has joined #openstack-manila16:50
*** rushil has joined #openstack-manila16:52
*** lpabon has quit IRC16:54
*** xyang1 has quit IRC17:02
*** vbellur has joined #openstack-manila17:02
*** rushil has quit IRC17:13
*** rushil has joined #openstack-manila17:19
*** vponomaryov has quit IRC17:25
*** sgotliv has joined #openstack-manila17:42
*** sgotliv has quit IRC17:42
*** sgotliv has joined #openstack-manila17:42
ganso_bswartz: ping17:48
bswartzganso_: pong17:49
*** xyang1 has joined #openstack-manila17:50
ganso_bswartz: Do you know what is the liberty milestone deadlines for new drivers and new core features?17:50
bswartzhttps://wiki.openstack.org/wiki/Liberty_Release_Schedule17:50
bswartzL-3 (September 3) will be the feature freeze17:50
ganso_yea but for Manila, I can assume that everything, drivers and features, will be accepted by L3?17:51
bswartznew features have to be submitted and mergeable 2 weeks before that to get review time17:51
ganso_ok, so L-3 then17:51
bswartzyou can never assume something will be accepted -- we'll only merge code that we're happy with17:51
bswartzbut there are no special early deadlines17:51
ganso_yes, sure17:51
ganso_thanks!17:52
bswartzfor a major new feature though, we would want to split it into pieces and merge them throughout the release17:52
bswartza big giant feature at the last minute is going to make us too nervous17:52
bswartzI'm hoping to have something working for replication before august17:53
bswartzthat means we have to sort out the AZ issues too before then17:53
ganso_and migration even before that17:54
bswartzand we definitely need to reach a conclusion on single ID vs multiple ID and the concept of "share instances" in the next few weeks17:54
ganso_yes17:54
ganso_btw, data copy service... this is also going to be mandatory, right?17:54
bswartzso far I'm not convinced that share instances are any help for replicated shares17:54
bswartzganso_: data copy service it nice to have, but something that should go in early if we're going to do it in this release17:55
bswartzit might be a good starting place because we can get it in mergeable shape with no dependencies17:55
*** lpetrut has joined #openstack-manila18:12
*** lpabon has joined #openstack-manila18:16
*** sgotliv has quit IRC18:18
*** lpabon has quit IRC18:21
*** lpabon has joined #openstack-manila18:48
*** fthiagogv has quit IRC18:55
*** vbellur has quit IRC19:02
*** cknight has quit IRC19:15
*** cknight has joined #openstack-manila19:19
*** lpabon has quit IRC19:37
*** timcl has quit IRC20:02
*** openstack has joined #openstack-manila20:05
*** openstack has joined #openstack-manila20:30
*** rushil has quit IRC20:43
*** Triveni_ has quit IRC20:56
*** openstackgerrit has quit IRC20:59
*** openstackgerrit has joined #openstack-manila20:59
*** bswartz has quit IRC21:10
*** lpetrut has left #openstack-manila21:28
*** openstack has joined #openstack-manila21:31
*** marcusvrn has quit IRC21:53
*** akshai has quit IRC21:53
*** mdenny has quit IRC21:54
*** cknight has quit IRC22:17
*** lpabon has joined #openstack-manila22:28
*** lpabon has quit IRC22:42
openstackgerritOpenStack Proposal Bot proposed openstack/manila: Updated from global requirements  https://review.openstack.org/18694522:45
openstackgerritOpenStack Proposal Bot proposed openstack/python-manilaclient: Updated from global requirements  https://review.openstack.org/18694722:53
*** sgotliv has joined #openstack-manila23:06
*** sgotliv has quit IRC23:07
*** ganso_ has quit IRC23:14
*** xyang1 has quit IRC23:52
*** bswartz has joined #openstack-manila23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!