20:01:39 <notmyname> #startmeeting swift community meeting
20:01:40 <openstack> Meeting started Mon Oct  1 20:01:39 2012 UTC.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:01:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:01:42 <openstack> The meeting name has been set to 'swift_community_meeting'
20:01:59 <notmyname> welcome and thanks for coming
20:02:09 <notmyname> there is an agenda at http://wiki.openstack.org/SwiftOct1Meeting
20:02:11 <notmyname> #link http://wiki.openstack.org/SwiftOct1Meeting
20:02:24 <notmyname> and and etherpad doc at http://etherpad.openstack.org/SwiftOct1Meeting
20:02:25 <notmyname> #link http://etherpad.openstack.org/SwiftOct1Meeting
20:02:57 <notmyname> Goal for this meeting: 1) wider dissemination of stuff to work on in swift and 2) determine what people will be working on over the next 6 months (openstack grizzly). This is also prep for the summit and how talks are scheduled and planned there.
20:03:44 <notmyname> my vision is for swift to be used by everyone, every day, whether they know it or not.
20:04:07 <notmyname> great example of how this is being done today is through wikipedia and the recent iphone 5 event
20:04:44 <notmyname> anyone who goes to wikipedia sees the media content served from a swift cluster, and the gdgt's liveblog of the iphone 5 event used swift (RAX cloud files) to serve up the content
20:05:13 <notmyname> To help guide the discussion for today's meeting, I've listed out a bunch of ideas on the agenda and etherpad
20:05:20 <notmyname> These ideas have come from users and swift devs
20:05:31 <notmyname> Just to cover the bases, just because I listed something here doesn't mean it will get implemented, nor does the absence of an item mean it won't get implemented.
20:05:53 <notmyname> I'll be goving over each category in turn. please add your items as you see fit, and if you are interested in the feature, please add your name/org so we can keep a rough poll of what's important to who.
20:06:21 <notmyname> I think there are 4 broad areas for swift development
20:06:28 <notmyname> he first is focusing on new contributors
20:06:47 <notmyname> This helps swift in 2 ways. One, it adds momentum to the project by adding more devs and dev teams as contributors. Two, the process of making the codebase more friendly to new devs simply makes the codebase better for existing devs too.
20:07:10 <notmyname> A good way to help new devs get involved is to provide and improve our getting started docs. Docs about the code organization, overall system design, and improvements around setting up and testing a dev environment would go a long way to facilitating new devs coming into the project.
20:07:23 <notmyname> Would anyone like to work on this, or is there anything to add in this category?
20:08:02 <notmyname> The second major area for swift development is focusing on new deployers. Deployers are the people who actually choose swift over alternatives and keep it running in production day in and day out. The features deployers look for have to do with automation, performance, and operational costs.
20:08:05 <zaitcev> I think it's the usual place for new people to start.
20:08:26 <notmyname> zaitcev: docs or deployer concerns?
20:08:27 <ogelbukh> we've got some docs on swift internals here at Mirantis
20:08:31 <zaitcev> docs
20:08:45 <ogelbukh> some of them we've published to our blog
20:09:05 <notmyname> ogelbukh: cool. anything you can do to add some of those to the docs in the codebase?
20:09:07 <ogelbukh> others mostly internal materials and kb stuff
20:09:46 <zykes-> hoii notmyname !
20:10:18 <ogelbukh> notmyname: if we can break down into subsections
20:10:20 <notmyname> if anyone is interested in or working on the deployment category of things, please add your name/org there too
20:10:34 <ogelbukh> I think we can try to list anything we have
20:10:40 <notmyname> ogelbukh: great
20:11:46 <zaitcev> Deployment is also common to work on, I think. Here's one thingie, installs Swift among other things - https://github.com/derekhiggins/os-installer
20:12:22 <notmyname> zaitcev: that's cool
20:12:56 <tongli> guys, I've started looking at the multi-range support.
20:12:59 <briancline> is there interest enough in better docs for folks running swift on alternative *nix platforms, in addition to what's there today?
20:13:02 <ogelbukh> regarding deployment, it's interesting to know the status of multi-region
20:13:25 <tongli> I think that the getting started dev guide and few experienced swift developers taking turns really monitor the questions regarding the swift develoment will be good.
20:13:34 <notmyname> briancline: there are several things that have come up from time to time, but the core dev mostly see ubuntu+xfs
20:14:05 <notmyname> tongli: go ahead and add your name to the etherpad please
20:14:33 <notmyname> briancline: and the openstack CI testing isn't very broad as far as  differing platforms goes
20:14:59 <notmyname> briancline: so the biggest concerns are around supporting the code and regressions and reliable testing for it
20:15:23 <notmyname> The third major category of swift development focuses on client apps. As nice as a REST-based API is, the vast majority of users will never touch HTTP. Good client apps bring a healthy ecosystem to the project and help swift thrive.
20:15:29 <tongli> john, you mean put my name after the feature?
20:15:38 <notmyname> What can we do to provide tools an information to the people making the client apps so that they add swift support to their products?
20:15:47 <notmyname> tongli: ya, I saw you did. thanks
20:16:01 <notmyname> One way is to encourage community language bindings (Rackspace has several, as do other deployers like SoftLayer and HP). We can also provide docs around how to design good swift clients and how to design data sets to be used with swift.
20:16:41 <notmyname> I don't want to say that all swift needs is better docs, but there are areas where better docs would help ;-)
20:17:19 <ogelbukh> it's hard to overestimate the importance of docs )
20:17:27 <adrian_smith> Another way to encourage client apps would be CORS compatibility
20:17:27 <briancline> notmyname: is that due to lack of easily accessible expertise with alternative platforms and even distributions? would also be interested in hearing more offline about the details on support concerns
20:17:49 <pandemicsyn> api docs with some sample apps, or maybe links to opensource apps might be handy
20:17:59 <notmyname> briancline: ya, that's part of it
20:18:23 <adrian_smith> Has there been anything done on this recently?
20:18:29 <notmyname> pandemicsyn: good idea, can you add it to the etherpad
20:18:54 <notmyname> I'll be using the etherpad to facilitate a similar discussion at the summit
20:18:57 <adrian_smith> Last I checked we were looking to put this kind of thing out into a separate project rather than keep in in the core project.
20:19:06 <tongli> how about the idea I had above to have one take turns to supervise the doc and answer very technical questions, say like 1per a month?
20:19:18 <notmyname> adrian_smith: there was a patch for it (was it yours?) but it never got merged
20:19:34 <notmyname> adrian_smith: I don't immediately see why it would need to be in a separate project
20:19:35 <Guest80024> some analysis / comparison to other storage solutions out there might be helpful to increase adoption
20:19:47 <adrian_smith> notmyname: it was yes. I meant to follow up and put it in a separate project but just haven't had a chance yet.
20:20:09 <notmyname> adrian_smith: ah ok. ya, I'd love to see full CORS support for swift
20:20:34 <scotticus> notmyname: working on it.
20:20:43 <adrian_smith> notmyname: just rereading your review now, https://review.openstack.org/#/c/6909/
20:20:49 <notmyname> scotticus: CORS? cool
20:21:17 <notmyname> scotticus: you and adrian_smith should talk ;-)
20:21:35 <adrian_smith> notmyname: I'll try and take your comments into the next changeset
20:22:15 <notmyname> moving to the next big category...
20:22:18 <notmyname> The fourth major area is a focus on new end users. The end users are ultimately why we're working on this. This category is for things that are user-facing. Users want features to meet a specific pain point or to open up new use cases for the storage. For example, on-disk encryption is a pain point for some users for compliance reasons. On the other hand, a tiered storage cluster would allow swift to solve some new use cases.
20:23:21 <ogelbukh> we've done a design for on-disk encryption as a part of joint effort with Webex
20:23:32 <notmyname> ogelbukh: ya, I came across your blog post this morning
20:23:34 <ogelbukh> there's a link to blog on it
20:23:43 <notmyname> ogelbukh: do you have an implementation for it yet?
20:24:04 <ogelbukh> not ready to publish it yet
20:24:06 <notmyname> ogelbukh: also, I'd love for you to submit a proposal to talk about it at the design summit (summit.openstack.org)
20:24:12 <ogelbukh> but we're working on it
20:24:13 <adrian_smith> ogelbukh: could you post a link to your post. sounds interesting. love to see how you manage keys.
20:24:33 <notmyname> Finally, there's the "catch-all" of "make swift more awesome". These are things like code refactorings, speed improvements, and system support. These are issues that get a lot of attention from the swift devs since it's what we deal with on a daily basis. These are things that will always be improved and will make up the largest number of commits in any swift release.
20:24:38 <ogelbukh> http://www.mirantis.com/blog/openstack-swift-encryption-architecture/
20:24:48 <creiht> yeah the encryption part is pretty easy, the key management is where things get sticky
20:25:13 <ogelbukh> yes, that's the hardest part
20:25:25 <ogelbukh> for the initial implementation it's very simple
20:25:30 <creiht> I've wondered if it would deserve its own service
20:25:42 <glange> keys as a service?
20:25:43 <ogelbukh> we need pluggable back-end there eventually
20:26:00 <judd7> is keystone a hard requirement these days?
20:26:13 <notmyname> judd7: for swift? it's never been a requirement for running swift
20:26:14 <creiht> I imagine keys are going to be more important in the other projects
20:26:19 <ogelbukh> not really hard, i believe
20:26:25 <zaitcev> Keystone is not a requirement to run Swift but some distros may pull it in as a dependency.
20:26:26 <Guest80024> you're thinking of having keystone serve out keys?
20:26:41 <CrackerJackMack> Guest20200: seems like a fit
20:26:42 <judd7> it's in the name. :)
20:26:46 <creiht> lol
20:26:59 <notmyname> hehe
20:27:39 <Guest80024> might make sense - having tenant level keys + user level keys might make much sense
20:27:42 <judd7> As long as we need ANOTHER key management project in the universe, it might as well be in keystone.
20:28:07 <judd7> Unless the keystone devs want to make that a pluggable backend too.
20:28:18 <judd7> Which is fine.  Would help corporate adoption.
20:28:27 <creiht> all things are pluggable in openstack right?
20:28:29 <creiht> :)
20:28:34 <notmyname> creiht: isnt' it all middleware?
20:28:43 <redbo> encryption is "easy"
20:29:02 <judd7> "keep your keys in verisign's or whomsoever's safe."
20:29:16 <judd7> "we default to sqlite.
20:29:21 <creiht> judd7: yeah
20:29:35 <creiht> I haven't done very much research in that area though
20:30:25 <notmyname> is there any work being done on this already in keystone?
20:30:26 <judd7> I might pull something together.  Dell has/needs that data for our own efforts.
20:30:50 <notmyname> judd7: cool. also, submit a talk for it for the summit :-)
20:30:53 <judd7> And by "pull together" I merely mean names/versions of customers who have at least asked us about it.
20:31:33 <redbo> there are some big concerns I have about encryption
20:31:41 <Guest80024> seems that mirantis is already doing a talk about their arch
20:32:00 <ogelbukh> redbo: could you please share some?
20:32:04 <notmyname> Guest80024: ogelbukh: not in the technical tracks though
20:32:33 <notmyname> what else in the feature list or category list would you like to discuss before moving on to the summit plans?
20:32:34 <redbo> if it's implemented as middleware, I'm assuming you'll lose the ability to serve ranges inexpensively
20:33:02 <redbo> as opposed to something like an encrypted block device
20:33:07 <adrian_smith> unique-as-possible data placement is really interesting
20:33:13 <Guest80024> (reading the blog) - they encrypt each block seperatly
20:33:27 <Guest80024> not block - each chunk.
20:33:39 <ogelbukh> yes, it's by-chunk
20:33:46 <judd7> CPU++
20:33:52 <adrian_smith> it really opens the door for distributed clusters
20:33:54 <notmyname> redbo: ogelbukh: we've got an implementation at swiftstack that uses LUKS devices
20:34:24 <notmyname> adrian_smith: ya, the unique-as-possible is really cool. it's already in swift, but it needs to be expanded somewhat for the global clusters
20:34:42 <redbo> my other concern is mostly CPU usage
20:35:01 <ogelbukh> should be actually faster then middleware encryption
20:35:12 <ogelbukh> however, it's all or nothing
20:35:20 <Guest80024> can you elaborate on unique-as-possible?
20:35:31 <Guest80024> (i.e. link?)
20:35:38 <adrian_smith> notmyname: sure. it's a really attractive feature. looking forward to seeing where it goes
20:35:40 <ogelbukh> some range of options here )
20:36:23 <adrian_smith> Guest80024: described here, http://swiftstack.com/blog/2012/09/27/top-three-swift-features-in-openstack-folsom/
20:36:38 <Guest80024> thx!
20:36:43 <notmyname> adrian_smith: thanks
20:37:04 <Guest80024> there was some talk about dedup within swift...
20:37:08 <notmyname> so, anything else before we move on to the summit plans? (summit plans discussion should be short)
20:37:17 <Guest80024> is that on anyone's radar?
20:37:37 <tongli> what do you mean by remove webob?
20:37:39 <notmyname> Guest80024: it's possible (anything's possible!) but probably unlikely
20:38:02 <notmyname> tongli: redbo has an outstanding patch to do so. basically, webob keeps breaking us with every version bump
20:38:06 <Guest80024> there was a Msc dude in portogal that started circulating a proposal that got some traction on the ML
20:38:08 <notmyname> tongli: it's a pain to amange
20:38:41 <tongli> i c. I remember once that the HEAD operation failed. had to apply a patch.
20:38:50 <notmyname> tongli: and has annoying "magic properties" that do stuff when you don't expect it to (like content-type and charset setting)
20:39:05 <pandemicsyn> https://review.openstack.org/#/c/12186/ - swob
20:39:56 <notmyname> ok, summit plans
20:40:10 <notmyname> I hope you are all going to the summit, but I know some of you won't be there
20:40:11 <tongli> quite big change. so it will be merged?
20:40:33 <notmyname> please submit your talk proposals at http://summit.openstack.org
20:40:42 <notmyname> you can also see what's currently proposed there
20:40:53 <Guest80024> nexenta put out a detailed proposal for the dedup stuff.. not sure it ever went anywhere: http://etherpad.openstack.org/P9MMYSWE6U
20:40:58 <notmyname> We've got a swift track all day on monday (8 slots) and possibly 2 more slots on tuesday morning. The sessions are intendend to be for community feature dev work, but in the past we have also had talks on design overviews and in-depth discussions on how existing features work.
20:41:37 <notmyname> there are some additional workshop sessions that will cover some swift topics, so there should be some very good swift content at this summit
20:42:31 <notmyname> is anyone planning on submitting talks that hasn't done so already (we've only had 3 people submit talks so far)
20:42:45 <Guest80024> can i float an idea here?
20:43:03 <notmyname> I'd like to see one from ogelbukh on the encryption. perhaps someone from rax or adrian_smith on CORS
20:43:05 <notmyname> Guest80024: sure
20:43:17 <Guest80024> automatic ring adjustment for add/remove disks, with timed, gated adjustment of disk weights
20:43:34 <tongli> I would love to see someone talks about the ring generation in details and how we can make that automated in the future.
20:43:52 <tongli> @Guest80024, are we thinking about the same thing?
20:44:01 <Guest80024> there are a few tools out there that do generation.
20:44:15 <Guest80024> the problem is maintaining.... and adjusting to changes in available disks/servers
20:44:15 <notmyname> pandemicsyn: arent' you working on soemthing like that? the ring-server?
20:44:18 <judd7> Guest80024: can you list some of those tools?
20:44:19 <ogelbukh> personally I won't be able to attend, but mirantis will do the talk about it
20:44:20 <tongli> yes,
20:44:48 <tongli> I would like to see a talk about ring and ring related session.
20:44:49 <notmyname> ogelbukh: cool. please try to get something on the summit page, even a placeholder ASAP
20:45:03 <pandemicsyn> notmyname: ring server just lets you add/remove/manage devices , doesn't manage rebalance/deployments
20:45:07 <ogelbukh> on the ring generation, we've done some things in puppet
20:45:21 <ogelbukh> based on current stuff but slightly modified
20:45:30 <zaitcev> http://swiftstack.com/blog/2012/04/09/swift-capacity-management/ says that SwiftStack's cluster management has "add gradually" mode
20:45:32 <judd7> I did (in a previous life) ring management with chef+rundeck.  It was cool.
20:45:33 <Guest80024> I was talking about crowbar
20:45:37 <notmyname> tongli: we're doing a meetup in sunnyvale next week about that :-)
20:46:03 <Guest80024> but it's not yet updated for the smarter folsom ring
20:46:13 <tongli> yeah, in Raleigh NC, can not attend.
20:46:38 <notmyname> Please go to your org and campaign for what is important to you. And bring other ideas to the summit. :-)
20:46:45 <Guest80024> (and IRC is being stupid about letting me use my handle for some reason.. its aabes_)
20:46:51 <tongli> it will be nice to have some one write up something for a session not only about the ring in current implementation, but also look at the future and some options to improve
20:47:13 <notmyname> tongli: ya, we can look into something about that
20:47:19 <Guest80024> if there's interest in having a discussion about my suggestion, I'll submit a summit session.
20:47:21 <tongli> great.
20:47:42 <notmyname> Guest80024: I'm interested. please submit it :-)
20:47:45 <notmyname> And now to the last item on the agenda: are there any outstanding concerns or questions about swift that I can answer or work on, on your behalf?
20:48:15 <Guest80024> better deployment consideration documentation?
20:48:27 <zaitcev> notmyname: I added it to Etherpad - do you know if gholt is going to continue on Swiftly?
20:48:42 <notmyname> gholt: around and want to answer?
20:48:53 <pandemicsyn> zaitcev: not sure if gholt is around today (hes got bird flu or something)
20:49:01 <glange> notmyname: he's been out sick today
20:49:04 <notmyname> zaitcev: swiftly is a community or ecosystem thing
20:49:04 <pandemicsyn> but we use swiftly alot at $RAX
20:49:25 <zaitcev> I'm sick of the client CPU burn, so I was going to have a look. Not sure if it should go into python-swiftclient or perhaps he's already solved it.
20:49:58 <notmyname> zaitcev: ya, talks around the client certainly would be appropriate at the summit too
20:50:07 <Guest80024> haven't looked recently - how's the swift support in fog?
20:50:16 <zaitcev> Not coming to Summit this time sadly
20:50:45 <torgomatic> zaitcev: the client CPU burn can, in principle, be cleaned up provided that it doesn't have to run on Python <= 2.4
20:51:12 <zykes-> still swift meeting notmyname ?
20:51:19 <notmyname> zykes-: ya
20:52:02 <notmyname> 9 minutes left. anything I can address, either with swift or openstack-related? also, feel free to email me at me@not.mn
20:52:07 <creiht> notmyname: I may have another one, but I can't talk about it yet :)
20:52:11 <creiht> (summit talk)
20:52:29 <notmyname> creiht: cool
20:52:35 <swifterdarrell> zaitcev: torgomatic: I think I've fixed at least one Queue busy-waiting in python-swiftclient's swift
20:52:37 <notmyname> creiht: when will you be able to?
20:52:42 <Guest80024> notmyname: fog support for swift?
20:53:04 <notmyname> Guest80024: I'm not a ruby dev, so I haven't kept up to date on fog support
20:53:20 <Guest80024> just checked, doesn't seem to be there.. there's rax and s3
20:53:29 <notmyname> Guest80024: RAX == swift
20:53:36 <Guest80024> keystone auth?
20:53:42 <notmyname> assuming it let's you set the auth endpoint
20:53:45 <creiht> notmyname: be able to?
20:53:54 <zaitcev> swifterdarrell, I'll check if it's the same. I saw a problem with SSL enabled.
20:54:06 <creiht> I have an idea of something that I want to talk about, but have to run it by some people here first
20:54:07 <notmyname> creiht: able to talk about it
20:54:14 <notmyname> ok, let me knwo
20:54:19 <creiht> ahh... it is probably best that I don't yet :)
20:54:32 <notmyname> oh the suspense ;-)
20:54:36 <creiht> hah
20:54:46 <judd7> We wont tell anyone.
20:54:56 <notmyname> ok, let's go ahead and wrap this up so that the next meeting can start
20:55:07 <briancline> don't worry, this definitely isn't logged
20:55:18 <swifterdarrell> zaitcev: may be unrelated--I was referring to CPU burn related to busy-waiting on threads/Queues; one place has been changed and as torgomatic pointed out, there are several left to fix
20:55:20 <judd7> Thanks notmyname, pandemicsyn
20:55:27 <swifterdarrell> zaitcev: but that's probably unrelated to something re: SSL?
20:55:28 <notmyname> Thanks for your time. We all work for different organizations. Please go to your org and campaign for what is important to you. And bring other ideas to the summit. Looking forward to seeing you there.
20:56:03 <notmyname> #endmeeting