19:00:36 #startmeeting swift 19:00:37 Meeting started Wed May 29 19:00:36 2013 UTC. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:38 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:40 The meeting name has been set to 'swift' 19:01:00 @notmyname, good afternoon 19:01:06 hellow everyone 19:01:22 things to cover today: next swift release, API, summit, and a few other things 19:01:23 hello 19:01:31 ahoy 19:01:37 hi 19:01:40 first, a few simple things 19:01:40 hey 19:01:42 helo 19:01:44 hello 19:01:50 #topic housekeeping/misc 19:02:19 The naming for the next OpenStack release (the "I" release) is open for a vote. 19:02:28 'the poll is at https://launchpad.net/~openstack/+poll/i-release-naming 19:02:58 on the topic of openstack releases, the next summit is in hing kong 19:03:03 hong kong 19:03:15 dates are nov 5-8 19:03:15 http://www.openstack.org/summit/openstack-summit-hong-kong-2013/ 19:03:38 and if you are a US citizen, here's your visa info: http://hongkong.usconsulate.gov/acs_hkvisa.html 19:04:01 if you are planning on attending, make sure your passport is up to date. it can take up to 2 months to get one, so do it now 19:04:48 a little more specific to swift now, let's talk about the next release of swift 19:05:03 #topic next swift release 19:05:22 I'd like to cut the next release when the final global clusters features land 19:05:42 @notmyname, for US citizen , you do not need visa if you stay there less than 90 days. 19:05:51 litong: correct 19:05:57 but you do need a valid passport. 19:06:00 the proxy affinity is the major one, but I've also got my eye on the cross-region replication 19:06:33 any comments or questions on when we should do the next release? is that a good plan or should we do something else? 19:07:32 I'll assume either everyone agrees or I've dropped out of IRC again... ;-) 19:07:42 I'd like to see https://review.openstack.org/30797 merged before the next release 19:08:02 I'd like to see thread pools make the next release 19:08:05 @notmyname, cross-region replication is a good one. 19:08:10 or at least, I'd like to see that bug fixed; doesn't have to be my patchset 19:08:23 notmyname: whats the tentative release date for these features to be completed? 19:08:28 torgomatic: portante: agreed on both 19:08:42 shri1: there is no date as of yet. it's done when it's done 19:08:42 *if these features have to be completed 19:08:49 I see 19:08:51 shri1: sooner is better than later 19:09:11 if anyone can review this one (just need one more +1 to merge), that will be really nice. https://review.openstack.org/#/c/22569/ 19:09:15 but aside from the 6 month openstack cycle, we aren't doing any sort of time-based releases 19:09:56 notmyname: do you want to keep the DiskFile and DatabaseBroker out of the next release to not delay it? 19:10:11 so basically, all this comes down to doing reviews 19:10:19 (what's new?) 19:10:24 ;) 19:10:40 @notmyname, yes, yes, yes. review, please 19:10:44 #topic reviews 19:10:48 speaking of reviews... 19:12:02 just a quick note, especially for core reviewers: if someone has already given a +2 and there is a minor issue (eg typo in docs or commit message), it probably shouldn't be given a -1 by another reviewer. that resets the previous +2 and slows everything down 19:12:51 quick follow-up commits could be a good way to address small issues like that, as long as master remains deployable 19:13:16 just something to keep in mind and try to see if we can speed up the review time 19:13:37 #topic bot in channel 19:13:50 * portante ack 19:14:10 last quick thing: I added the openstackgerrit bot to #openstack-swift. anyone think it should be removed? 19:14:22 it reports on new patches and merges 19:14:34 but if it gets too annoying, we can easily remove it 19:14:42 * torgomatic likes having the bot there 19:14:47 * portante agrees 19:15:10 I think it is a good thing. 19:15:27 cool. I like it too. let's keep it until it gets to be a problem 19:16:01 ogelbukh1: glad to see you. I'm interested in the mirantis patches wrt global clusters, but they are all marker WIP 19:16:23 ogelbukh1: basically, I want the functionality to be in the next release 19:16:39 and I doubt many people have looked at them since they are marked WIP 19:17:33 is there a bp/wiki entry for the global clusters work? 19:18:01 dosaboy: https://blueprints.launchpad.net/swift/+spec/multi-region 19:18:17 notmyname: cool thanks 19:18:26 portante: ready to talk about DiskFile? 19:18:35 sure 19:18:38 #topic DiskFile refactor 19:18:43 portante: go for it 19:19:06 so how many folks have heard about this effort? 19:19:17 o/ 19:19:23 I not that notmyname and torgomatic have looked at? 19:19:42 o/ 19:19:47 (quiet crowd today) 19:20:17 From April's summit, we are working to refactor the DiskFile and DatabaseBroker to define them as "public" classes so that we can have various implementations of those classes 19:20:26 i've heard of it 19:20:31 the existing implementation would be the first 19:20:50 to date, we've proposed https://review.openstack.org/#/c/30051/ 19:21:04 which is a stab at refactoring just DiskFile to define the APIs 19:21:29 It still has 18 unit test failures, so it is a work-in-progress for sure 19:21:58 but I am seeing folks to take a look at the changes to see if they make sense and if they are in a direction we want to go in 19:22:09 aside from the minor comments I left, I like the direction of it 19:22:35 I like the way it's going 19:22:52 notmyname has suggested that I pull out the "reader" work, which is sister work to the "writer" work performed in https://review.openstack.org/#/c/27149/ 19:23:05 that might make it easier to review 19:23:07 i haven't yet, but i'm interested in taking a look and will provide any feedback that i can think of 19:23:19 alexpecoraro: thanks 19:23:51 I personally think pulling the "reader" work out would be a good thing, but want to balance too many things to review against too large a change to review 19:25:10 The next question is how to take this in pieces so that it does not straddle a release and possibly cause problems 19:25:39 I would like each piece to be fully functional, but still, not a fan of half-way for a release 19:25:42 portante: well, anything that lands should be distinct enough to work itself 19:25:48 is it really a problem if it straddles a release? I mean, at each step, Swift is still working 19:25:57 eg extracting the reader doesn't break anything 19:26:02 true enough 19:26:41 if folks are fine with that, then let's work to target the "reader" work pulled out of the next release, and then the full API definition to follow 19:26:52 I'm ok with that 19:26:55 anyone else have comments? 19:27:13 certainly I am available for questions afterwards 19:27:39 when it is in, how one can replace with a different implementation? 19:28:13 ah, em, ah, well, er, so, still working on those mechanics. :) 19:28:20 heh 19:28:35 worst-case (ie now), monkey-patching, right? 19:28:42 right 19:29:01 @portante, I am thinking about LTFS. 19:29:03 best case, defined backends associated with a know string 19:29:10 LFS patch? 19:29:13 litong? 19:29:20 right. 19:29:28 Linear Tape File System 19:29:36 The DiskFile and DatabaseBroker refactoring work is going to be a subclass of that 19:29:44 very low cost storage. 19:29:47 ah, yes 19:29:48 I see now 19:30:03 in thoery, we should be able to do that. 19:30:33 yes 19:30:33 swift cluster can be a mixed disk , tape, very fast storage and some what slow storage. 19:30:34 I've got a couple of uses too (the most obvious is optimizing some XFS-specific things, if possible) 19:30:59 I'd like to use it to let me audit for missing objects on non-XFS filesystems 19:31:18 well, more quickly audit ;-) 19:31:32 to that end, it would be helpful to get the thread pools work in before this work 19:31:43 notmyname: true; right now, though, it takes until another update invalidates hashes.pkl before anyone notices a missing file 19:31:45 so the question is, when this patch is in, will it be every node with same driver or object node can have mixed drivers. 19:31:49 I would like to make sure this refactoring is considered on top of that 19:31:52 so let's say "update in bounded time" :) 19:32:51 Also, regarding the DatabaseBroker side, the work that David Haddas did for the "db_file" exists code, would be useful 19:32:51 litong: that will be determined by the yet-to-be-defined inclusion mechanism, I'm usre 19:33:13 portante: so what you're saying is "do more reviews" ;-) 19:33:14 Also, zaitcev proposed a patch there as well 19:33:18 ;) 19:33:25 * portante smiles quietly to himself 19:33:56 #agreed do more reviews 19:34:03 (now it's official) 19:34:11 ;) 19:34:13 ok, shall we move on? 19:34:18 I'm good 19:34:24 #topic swift API 19:34:33 last thing to discuss today 19:34:38 https://wiki.openstack.org/wiki/Swift/API 19:35:25 this is a description of what I believe is the subset of functionality that can be defined as "swift" and that we can therefore call swift API v1 19:35:52 but there are a couple of outstanding questions, and I'm not sure who has actually looked at the page (there certainly haven't been many edits) 19:36:00 @notmyname, totally agree with the first 2 items. ACL and Auth. 19:36:34 multi-range is not supported on large objects 19:36:56 the goal is not to restrict functionality but to find the subset of functionality that currently-deployed swift clusters (ie from the swift source code) support 19:37:03 we could have that, if the patch got reviewed. I worked on that for 4 months, 19:37:17 litong: right, I think that distinction was lost in my last edit 19:37:32 So ACLs is interesting, because the code chose key names stored in the metadata on disk already, right? 19:37:52 can we have that back? I mean the multi-range on large objects. 19:37:56 Most auth filters rely on that format, don't they? 19:38:21 portante: yes, probably 19:39:25 would it cause a problem to define ACLs as part of the v1 API? 19:39:53 it is kinda asymetrical, since an auth filter would not be part of the v1 API 19:40:18 @portante, once it is in API, it alomst feel like required. some company may want to ACL completely different way. 19:40:44 but this is what it is today, these ACLs are stored on disk that way 19:40:44 I mean "do ACL differently." 19:41:04 @portante, you do not have to use it. 19:41:19 I want to have a v2 API that has pretty much all of the functionality in the codebase, but I don't think we should ever deprecate v1. This is to give a baseline of what can be called an implementation of the swift api 19:41:26 you can add your own ACL at the proxy server any way you want. 19:42:24 portante: since ACL definitions are only meaningful to a particular auth system, it doesn't make sense to me to define an ACL format as part of the swift api (especially a v1 api) 19:42:48 @notmyname, not in v1, v2, or any other version. 19:42:55 perhaps v1 and v2 are bad names. perhaps "basic functionality" and "complete functionality" are better 19:43:20 litong: perhaps. I'm not sure I'm willing to go that far yet 19:43:23 it conflicts with separation of concerns (design principals) 19:43:50 no matter how you define it , someone won't like it 19:44:19 what are the metadata key names used by the tempauth and other auth system? 19:44:23 I'd like to agree that the wiki page is an accurate reflection of what we as a group can call the v1 API 19:44:27 do we have to define a names apced for the v1 API then? 19:44:55 portante: I'd just leave X-Container-Read and X-Container-Write out of it 19:45:13 the other questions I had were around tempurl and large object support 19:45:21 okay 19:46:26 I like them both. I want to have a definition that says both "here is something that defines swift" but also not so restrictive that existing clusters aren't swift any more 19:46:53 so at this very moment, I'm leaning towards not including them. but I go back and forth 19:47:01 is tempurl purely a filter? 19:47:23 Or does it rely on some code in the app server side? 19:47:32 it is implemented as such, but that is immaterial to its inclusion 19:48:24 hmm, messy 19:49:02 I think it's important to separate the implementation decisions (ie middleware or not) from the functionality 19:49:19 eg dynamic large objects vs static large objects. one is middleware and the other isn't 19:49:24 notmyname: agreed 100% 19:50:03 well, if I have an implementation that can use the existing tempurl filter, then my implementation could say it supports it 19:50:05 to me the question is "when someone claims they have a swift cluster, what functionality should my client assume exists as a baseline?" 19:50:30 if I don't, then I have to implement it (like Ceph folks can use filters) 19:51:24 portante: correct. and in your case, when Red Hat says that gluster supports the Swift API, what does that mean for clients? 19:51:44 today, we support all the filters because we reuse most of the code 19:51:55 as you should! 19:51:58 ;-) 19:52:00 ;) 19:52:23 the way I see it, if a cluster lets me set X-Account-Meta-Temp-Url-Key on my account, and then do some HMAC magic to make a URL like /v1/AUTH_me/con/obj?temp_url_sig=X&temp_url_expires=Y and that url *works*, then that cluster supports Swift signed URLs 19:52:30 it's all from the client's perspective 19:52:34 yes 19:52:45 okay, I'm easily convinced 19:53:04 but should that be in the swift v1 api? :-) 19:53:12 I don't care if the implementation is using swift.common.middleware.tempurl verbatim, hacked up, or if it's got a hand-rolled proxy server written in node.js and befunge in front that handles it ;) 19:53:26 @notmyname, if that is really useful to customers, yes it should be in. 19:53:34 we are defining an API after all. 19:53:38 * torgomatic thinks signed requests *should* be in the Swift v1 API 19:53:57 IOW, should any clusters out there that don't support it today no longer be able to be called a swift cluster? 19:54:25 litong: usefulness isn't the criteria IMO, since we are only formalizing an API post-facto 19:55:16 and we're running out of time, so it seems that there isn't a broad consensus 19:55:22 hmmm, its the post-facto thingy that has me wondering here, not the core principle (which I like) 19:56:11 portante: basically, I don't want so write a tester against the v1 spec and have mercado libre, wikimedia, vimeo, rackspace cloud files, or HP fail it :-) 19:56:18 and all the others 19:56:20 yes 19:56:36 that is why I am thinking it should be based on the app server code vs filter code 19:56:42 for post-facto definition 19:56:47 not for the principle or the future 19:57:02 define v1 based on that, immediately define v2 based on the principle 19:57:03 are we starting the API on the wrong foot? 19:57:43 I think stick with the principal is important. we can not change an api in v2 which was just defined in v1. 19:57:45 litong: it is the foot we are on already, I think. 19:58:08 we should continue this in the -swift channel 19:58:09 we won't be, if I understandthings, we'll only be adding to the API 19:58:11 k 19:58:16 we're out of time 19:58:25 #agreed to nothing yet 19:58:37 thanks for your time everyone 19:58:42 #endmeeting