19:01:56 #startmeeting swift 19:01:57 Meeting started Wed Jun 12 19:01:56 2013 UTC. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:58 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:00 The meeting name has been set to 'swift' 19:02:12 welcome all. agenda (or topics) for today are on https://wiki.openstack.org/wiki/Meetings/Swift 19:02:40 agenda is a plus for future meetings also :) 19:02:46 first up, reminder about the hong kong summit 19:02:55 hong kong summit 19:02:56 nov 5-8: http://www.openstack.org/summit/openstack-summit-hong-kong-2013/ 19:02:56 us visa: http://hongkong.usconsulate.gov/acs_hkvisa.html 19:03:09 get your passport and visa ready if you haven't yet 19:03:21 welcome 19:03:29 #info get travel arrangements ready for the HK summit 19:03:43 "They must have a U.S. passport valid for at least six months" 19:03:45 interesting 19:04:03 also, I believe hotel blocks open up this week (according to an email sent last week), so that stuff is coming up soon too 19:04:04 oh I guess that means doesn't expire within 6 months 19:04:16 creiht: I think that's common to most international travel 19:04:22 creiht: standard procedure - same for pepole going to the US 19:04:23 yep 19:04:23 when is the call for papers? 19:04:44 notmyname: yeah my first thought was that it had to have been valid for 6 months 19:05:00 shri: not sure on the conference side. the summit (tech) side will be much closer to the event 19:05:08 shri: normally about a month out 19:05:12 * creiht needs to figure out if he would like to visit anywhere else in china 19:05:18 creiht: you need to be at least 6 month old also 19:05:27 creiht: let me answer that for you: yes 19:05:27 davidhadas: hehe 19:05:54 ok, next release 19:05:57 #topic next release 19:06:06 we had a bunch of good patches merged yesterday 19:06:39 torgomatic is working on a affinity on writes patch right now. that's the last major part of global clusters, and once that lands I'd like to do a release 19:06:52 it's been a while, and we've had lots of other stuff land too 19:06:56 notmyname: how about trying to get account acls in before next release? 19:07:15 its in the oven... 19:07:22 for certain values of "right now", at least. Actual right now, I'm here in a meeting. ;) 19:07:23 that's my next question: what else needs to land? any other outstanding patches that are in gerrit now? 19:07:31 all infrustructure is there now that we merged get_info 19:07:37 notmyname: should we give some time with an RC before we release so that those region features can be tested fairly well before release? 19:08:06 creiht: yes, to some extent. we're testing it as we go with a global clsuter we have 19:08:25 notmyname: I imagine dfg would like to get his slo of slos patch in 19:08:31 but he has been on vacation 19:08:31 ya 19:08:39 he should be back soonish 19:08:43 I'd like to have the final patches land next week and then do a release on June 27 19:08:52 that's 2 weeks from tomorrow 19:09:06 that seems reasonable 19:09:13 notmyname: a bit pressed fro acount acls - I assume it would take 2 weeks to score 19:09:45 So few more days would be great 19:09:58 If we get it great - if not, next time 19:10:20 davidhadas: when do you expect to submit it? 19:10:25 davidhadas: I imagine there will be room for more releases before Havana 19:10:39 yes, there will be 19:10:56 I will try to send first version before weekend - working now on tests and need to close one or two items 19:11:11 ok, great 19:11:21 * portante working towards getting series of patches for DiskFile refactoring into the release following June 27th 19:11:34 Lets target adding it in 19:11:39 the next release will be 1.9 19:12:07 I don't think 1.8.1 is appropriate, but I'm willing to hear arguments 19:12:18 agreed 19:12:20 sounds sexy to me 19:12:30 davidhadas: that's what we're going for ;-) 19:12:47 notmyname: no 2.0? :) 19:13:07 not yet :-) 19:13:16 creiht: lets do the 2.* inline with an openstack release 19:13:30 global clusters seems like a big enough feature to bump to 2.0 19:13:42 not sure if there will be a feature more important than that in a while 19:13:51 but I have no idea if after 1.9 its 2.0 or 1.a or 1.10 19:13:51 unless we are going to do like 1.10 1.11 etc 19:14:04 I'm fine with 1.10, 1.11, etc 19:14:11 that's the nice thing about integers: there are so many of them 19:14:13 notmyname: what would constitute 2.0? 19:14:15 perhaps save 2.0 bump to API changes? 19:14:25 creiht: not sure yet, but I'd tend to agree with portante 19:14:32 meh 19:14:34 :) 19:14:34 breaking changes 19:14:39 yes 19:14:55 well nothing would break because you would still have backwards compat :) 19:15:02 we've got some other stuff planned that may constitute a more major version bump :-) 19:15:14 do tell? 19:15:19 not yet :-) 19:15:24 notmyname: meh 19:15:25 ;0 19:15:25 :) 19:15:30 let's move on though, to other topics for today 19:15:44 notmyname: if it is that super awesome, then it can be 3.0 :) 19:15:49 #info next release targeted as 1.9 for June 27 19:16:03 Am I still connected - last message I see is from 22:12 19:16:06 creiht: it's teh awesomz!!11!!! 19:16:22 davidhadas: looks like lag 19:16:27 #topic API specs 19:16:35 https://wiki.openstack.org/wiki/Swift/API 19:16:39 davidhadas: I see your message, though if you're not seeing mine, I don't know how much good this reply will do 19:16:54 I've added some stuff to the API wiki page 19:17:10 mostly comments based on when things were implemented 19:18:02 I'd love more feedback on the wiki page 19:18:06 did we arrive at a decision for how to gauge what is in or out? 19:18:20 I got disconnected 19:18:24 from the last meeting, I thought notmyname was considering options 19:18:57 What is the current topic? 19:19:08 API specs 19:19:08 portante: ya, but I didn't hear from anyone :-) 19:19:08 I'm becoming convinced that we realy just need a decision to be made 19:19:20 like "everything in folsom or essex is v1.0" 19:19:28 if only we had some kind of leader to make that decision ;) 19:19:32 ;0 19:19:33 hmm ;-) 19:19:45 that could make a good decision? 19:19:46 ;) 19:19:59 * portante ow! 19:20:05 I'll put together a more formal proposal around that and send it to the mailing list 19:20:08 creiht: ouch! 19:20:14 I kid I kid :) 19:20:24 lets call clay and vote 19:20:47 api v1.0 is so 3 years ago 19:21:00 #action notmyname to make a formal proposal to the mailing list with decisions made as to what's in and out 19:21:26 ok, next topic 19:21:36 #topic internal API changes 19:21:36 How backwards compatible do we need to be with internal (i.e. non-user-facing) interfaces? This question was inspired by https://review.openstack.org/22820, which changes the format of a data structure in the WSGI environment that other middlewares may use, though none in the Swift codebase do. 19:21:49 who proposed that item? ^ 19:21:53 o/ 19:21:57 that's my issue 19:22:09 I'm just trying to get a sense of what people feel is important to have compatibility on 19:22:15 obviously, the client API must not change 19:22:38 and obviously, if I rename a method in swift.common.utils or something, that's okay 19:22:53 but isn't that talking about on disk data? 19:23:02 torgomatic: its just the same since someone may use it 19:23:14 acls stored in metadata on disk 19:23:25 * clayg snikers at "obviously" when coming from the 204->200 point the clients at the spec guy 19:23:33 we can't not change anything because somebody somewhere may change it 19:23:34 lol 19:23:39 we simply cant have such restrictions assuming we want to grow and prosper 19:23:40 torgomatic: is that a good way to change something and have release notes for next release? 19:23:47 portante: the on-disk ACLs aren't the contentious part there; it was the changes to the WSGI environment 19:23:57 so, just curious what people usually care about 19:24:11 portante: the on-disk ACLs are backwards-compatible in that change 19:24:13 in the past, we have chosen to keep things for backwards compatibility with a deprecation warning 19:24:20 and then take it out in a future version 19:24:34 gah, that review is huge - do I need to understand that issue to under the current topic? 19:24:41 any version should support growing from previous one(s) with whatever is on disk 19:24:58 But I am not sure if one or ones and how many ones 19:25:00 we've always had upgrade in place as well 19:25:06 if it is a data format issue 19:25:17 clayg: the issue is: an earlier version of that patch changed the value of env['keystone.identity'] that the middleware was emitting (leaking?) 19:25:19 upgrade plans are important (eg ring formats, pickle->json) 19:25:23 * portante wonders if there is a good summary somewhere about that discussion 19:26:07 torgomatic: yeah but so that's the same thing as changing a method name in utils right? 19:26:08 One does not simply run an upgrade script on a swift cluster :) 19:26:30 clayg: seemed that way to me, but chmouel disagreed 19:26:35 hence this discussion :) 19:26:42 clayg: not so sure, because a filter can be written and used outside of the make swift code base 19:26:45 chmouel: wtf? 19:26:54 if there are data changes that are cluster wides, the updates *must* happen in place 19:26:59 heh 19:26:59 that is why we are working on the DiskFile refactoring 19:27:06 creiht: definitely 19:27:09 portante: yeah middleware that's not in core is tough to keep up with master 19:27:47 I think we should still keep backwords compatible functions that are depricated for a release or two so that outside utilities can catch up 19:28:00 portante: but I mean this is way not as bad as something like swob, or the fact that our memcche client gripes about a deprecated kwarg "timeout" 19:28:13 We should have defined APIs - e.g. the auth API is a good example which we agree to not change (only extend - unless we do a major release for example 2.0) 19:28:14 clayg: heu i'm talking about the middleware 19:28:15 portante: but isn't the difference witht he DiskFile stuff that it is introducing a new API that is relatively stable (ie we agree to to change it). no such agreement is there with env vars 19:28:19 the api that middleware is based on 19:28:21 and get into the chain 19:28:28 not swift.common.utils and such 19:29:02 notmyname: I think that there is still an implicit agreement to try to play well together 19:29:10 of course 19:29:13 notmyname: yes, but, as an example, gluster today has to track the code changes made to make sure we are not broken by any code change assumptions 19:29:14 But ethe rest... there is no difference between what's in mmcache, whats in environment and whats in the runtime as for function names, parameters etc 19:29:42 the external-middleware models will have to do the same, and to verify one did not miss anything is a real pain 19:30:01 * portante shouts up from the whole we dug 19:30:21 portante: so the DiskFile should fix that hopefully as we are making it an official API that will not change unless we declare ands warn of that change etc. 19:30:31 agreed 19:30:55 so we have to be careful to define clearly what external filters can expect for their API 19:31:12 heh, well it's not even going to be "official official" - it just reduces the surface area a bit 19:31:13 * portante look another API definition popping up there too 19:31:31 So as far as I can see there are only two levels - either an official API - in which case we proactivly make sure we can change before we change or at least pre-warn, and all the rest where we continue as usual and try to be fair 19:31:33 like if you "reach around" the abstraction you're prolly gunna get bent 19:31:37 but not prevent progress 19:31:45 clayg: true 19:32:19 so the issue on this one is the keystone.identity env var. seems that the var is already "scoped" by name to be keystone. I can't imagine that eg swiftstack auth should require keystone's data structure to not change 19:32:25 yeah the internal api thing scares me a bit 19:32:48 I don't want to hamper innovation inside of swift 19:32:56 davidhadas_: well having a good abstraction that's more or less stable *accelerates* progress, and not just disk file, if we renamed "get_logger" or "readconf" we'd break a bunch of stuff 19:33:04 for example, what if we decide to rewrite the object server in C to get better io? 19:33:14 that api abstraction totally goes away 19:33:29 I would have to review the discussions, more but can somebody tell report on what existing middleware might get burned by the keystone.* wsgi change? 19:33:34 It would seem more important to define the REST api between the services first 19:33:36 creiht: yeah but we already have a loose internal api that allows that 19:33:41 clayg: agreed. But whatever we decide to fixate - we need to think about it before we fixate it and than say it out load 19:33:42 creiht: not it's "external" one. ya, that 19:33:43 heh yeah 19:33:59 like define what the object server REST api is 19:34:01 I dunno, I like the Linux kernel model: syscalls Shall Not Change(tm), and if an internal API gets changed, all callers get updated by the changer 19:34:14 creiht: isn't that what' portante is doing with LFS? :-) 19:34:25 if you're maintaining a driver outside of the kernel, it's up to you to keep up with internal API changes 19:34:26 torgomatic: but that's only for stuff in the kernel right? 19:34:30 * portante hopes he is doing that 19:34:48 notmyname: I'm not entirely sure... I was under the impression that it was an internal api from a code perspective 19:34:57 but then again I'm only going by what I hear you guys talk about 19:35:27 creiht: it will end being external once we get more than one backend implemented that is available to the community 19:35:32 end up 19:35:45 portante: ok, I guess I need to look at it closer then 19:36:00 Lets create a wiki page where we define what we preceive as an external API that extensions can rely to not change without prewarning 19:36:14 I think we're pretty loose on "defined" api's (previous topic was, yeah we should have one for the *public client api*) - i'm not sure what we're going to decide here 19:36:17 creiht: thanks, that would be great 19:36:32 perhaps "internal" vs "external" is confusing? an API s external, but it doesn't mean it's available to an end user 19:36:37 ... except maybe - yes continue to try and not break stuff other poeople are probably using 19:36:40 it's all perspective 19:36:52 basically, I want to know what sorts of non-client-breaking changes are likely to piss off other Swift devs if they get approved 19:36:53 notmyname: true 19:37:20 portante: of course my list of things to look at is ever increasing :/ 19:37:28 torgomatic: so lets document that 19:37:28 torgomatic: easy, the one that's make some other swift dev have to change their code 19:37:31 torgomatic: so how do we answer that? 19:37:58 notmyname: well, I was hoping to do that with a quick straw poll here, just to get a feel for things 19:38:23 torgomatic: go for it 19:38:48 sure 19:39:17 torgomatic: can you frame it succinctly? 19:39:23 the straw poll that is? 19:39:30 so, who here gets annoyed when things change in the WSGI environment? how about utility functions? other stuff? 19:39:30 straw man poll? 19:39:37 torgomatic: but it's impossible even your lowest bar "and obviously, if I rename a method in swift.common.utils or something, that's okay" is *NOT* ok - you can't change the name of get_logger 19:39:49 lol 19:40:07 is there *anything* else you can think of that we could *ALL* agree is totally safe to change? 19:40:27 does all this come down to "don't break things we know people are using and have good release ntoes"? 19:40:36 notmyname: +1 19:40:43 I think it is a lot like how we have handled api things 19:40:48 additive is fine 19:40:55 notmyname: sure, could be 19:40:56 and it may be that we don't know what people are using until a review 19:40:59 if you are going to radically change something, then we need some extra care 19:41:01 @notmyname, a product in my view should only insist on the confirmation of the external API 19:41:15 anything developed based on internal methods are at risk anyway. 19:41:23 creiht: agreed, but realizing that we can be a litle more free with non-client things 19:41:26 but is middleware internal methods? 19:41:27 I think chmouel has to defined why he thinks the wsgi environ is scared or a similar +2 is likely to come up 19:41:29 there is no one there to ensure it will be always there like that. 19:41:39 it's not obvious to me why that would be consumed by anyone except the guy setting it 19:41:51 clayg: I'd say whatever other middleware which is non core would use 19:41:51 portante: IMO, yes. middleware is an implementation detail not a definition of "optional" 19:41:52 we should not define that many APIs, it will be a big problem down the road for a product. 19:42:00 notmyname: sure, I just expect a "best effort" at making sure we don't make changes that are likely to break things 19:42:10 I agree 19:42:20 alright 19:42:22 torgomatic: what else would you like to see? 19:42:22 and that goes for those that review the changes as well 19:42:23 creiht: good for me 19:42:23 ok 19:42:29 notmyname: I'm done 19:42:29 but we allow other folks to write middleware and use with core, right? 19:42:31 * davidhadas_ volnteers to try and create a list 19:42:42 * portante willing to be done 19:42:44 with typos! 19:42:54 #agreed don't make changes that are likely to break things 19:42:55 :-) 19:42:57 portante: I've done that with many a middleware sure 19:43:05 lol 19:43:18 #topic DiskFile status 19:43:25 portante: can you give a 3 minute summart? 19:43:25 I think we have been fairly reasonable so far... there have been a couple of misses, but I think we have learned from them 19:43:28 notmyname: you skipped some items 19:43:28 sumary? 19:43:34 yes 19:43:35 in the agenda 19:43:37 davidhadas_: I'm not going in order 19:43:44 working through test case failures 19:44:03 chmouel: so this was a *real* breaking change between the keystone-client middleware and the keystone-auth middleware in swift? 19:44:08 davidhadas_: (I want enough time to discuss yours, so a quick DiskFile overview should work) 19:44:18 stumbled on quarantine semantics of when an object gets quarantined 19:44:30 chmouel: like ours was getting changed to look for a variable somewhere that the upstream middleware wasn't putting it? 19:44:30 based on refactoring of the APIs 19:44:45 working to propose that topic separately 19:44:46 clayg: which one are you talking about the review about the ACL ? 19:44:57 working on refactoring the reader code like the writer 19:44:59 chmouel: clayg: cna you discuss in #openstack-swift? 19:45:15 portante: is that your next patch? 19:45:17 clayg: breaking users who use env['keystone.identity'] defined in swifft, not keystone headers 19:45:42 next patch will likely be the quarantine refactoring 19:45:48 then reader done like writer 19:46:10 then API definition (this is private, this is public, this is reference implmemetation kinda stuff) 19:46:25 ok 19:46:40 would love input on quarantine issue from someone willing to listen on the problem 19:46:46 that is the status for DiskFile 19:46:53 portante: ok thanks 19:47:09 let's discuss quarantine issues in IRC or in a review 19:47:17 IRC=#openstack-swift 19:47:20 great 19:47:34 #topic path control 19:47:39 Path Control: Create an open interface to control the path used by Swift within devices (where within the device a/c/o/tmp/qurentined/async are placed). 19:47:39 Make sure any code approaching the device go via a central function (e.g. storage_directory() in utils) allowing monkey patching it or offering other ways to extend it 19:47:39 While we do it , we can extend storage_directory() to also append the basedir and device since this seem to be required by all its users. 19:47:46 I'm not sure really what this is? davidhadas_? 19:48:06 I wanted to hear your views on this one 19:48:48 this feels like it will end up being related to the "backend" in use one we get the DiskFile refactoring, and DatabaseBroker refactoring, in place 19:48:51 have a single function in utils that determines where stuff lives on disk? seems reasonable. maybe even something good to have in DiskFile itself 19:48:54 first , si getting all swift code handling the path to go via a central place - we can use this for ring doubling where we want to control the path 19:49:07 notmyname: agreed 19:49:23 davidhadas_: what's your concern? 19:49:48 second I want te the path control function to be made external API such that installations can decide to change it at will 19:49:58 Essentially, when one looks at the three servers, there is an "understood" way of dealing with things on disk unites DatabaseBroker classes and DiskFile 19:50:03 If this is ok with everyone I have no concerns 19:50:16 davidhadas: can you share a use case to help us understand it better? 19:50:41 *that unites 19:50:46 davidhadas: it sounds like something reasonable to be in DiskFile (eg portante doesn't want to name things the same way on disk in gluster) 19:51:39 notmyname: or even for XFS today, where we would want to use a different path for tempfiles that take advantage of XFS allocation groups behaviors 19:51:42 1. Disk doubling. 2. I can do SAIO with multiple swift regionss on one with this 3. We may decide to use a special path for containers and trat it with more cache based on that path 19:52:07 creiht: I've got a hypothetical use case of having different storage policies using different top-level directories (objects/ and objects-reduced-redundancy/ and etc) 19:52:15 davidhadas: can't that be done today with device paths and mount_check = False? 19:52:17 3 is kind of hardware and system specific - but different pepole may have their own 19:52:56 This feels very much tied to backends for DiskFile 19:52:59 portante: No - this allows chnaging the path on the fly as well 19:53:04 but when wearing bear glasses ... 19:53:13 I guess I was more cuious why you would change it on the fly? 19:53:26 * portante echos creiht 19:53:42 portante: it is related to backends - I think it is a different issue from DiskFile - BUt I also think it it can be combined to a threesome with DiskFIle and DB 19:54:32 creiht: it asallows for exampel to create two swift domains in one - as you know we are into domains 19:54:34 :) 19:54:39 a backend implementation might allow for this, but not sure changing specifically to get flexibility for the current implementation is the right way to go 19:54:48 Different domains for example may have different QoS 19:55:00 I guess it is difficult for me to visualize this without seeing some code 19:55:01 that most likely can be done with device paths, no? 19:55:14 davidhadas: I'd like to see the way things are stored more abstracted. eg what if my storage doesn't use directories? in that way, this proposal seems to be most appropriate under something like DiskFile rather than a sibline to it 19:55:30 sibling 19:55:53 creiht: I am in the same boat with you 19:56:11 notmyname: I cant see that, but I am ok with having this combined as long as I am not forced to implement a DiskFile simply to control the path 19:56:33 davidhadas: a DiskFile that allows you to control the path? ;-) 19:56:34 davidhadas: one should be able to subclass an implementation 19:56:50 portante: sure np 19:57:10 gluster is doing that for the current implementation 19:57:30 we just tweak a few things and reuse the reader 19:57:31 davidhadas: does all that sound ok to you? questions answered? 19:57:47 yes 19:57:51 great 19:57:57 thanks for bringing it up 19:58:26 reminder: next meeting is in 2 weeks, get your passport ready if you are going to HK, release tentatively on June 27 19:58:31 thanks for your time 19:58:33 have a great day 19:58:36 #endmeetign 19:58:38 #endmeeting