21:00:22 <notmyname> #startmeeting swift
21:00:25 <openstack> Meeting started Wed Jun 29 21:00:22 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:28 <openstack> The meeting name has been set to 'swift'
21:00:34 <notmyname> who's here for the swift team meeting?
21:00:36 <briancline> o/
21:00:37 <jrichli> o/
21:00:38 <timburke> o/
21:00:40 <bkeller`> o/
21:00:41 <mmotiani> Hi
21:00:42 <kota_> hi
21:00:43 <pdardeau> o/
21:00:44 <cschwede> o/
21:00:45 <dmorita> o/
21:00:45 <mattoliverau> o/
21:00:47 <kei_yama> o/
21:00:48 <cutforth> o/
21:00:54 <mathiasb> o/
21:00:58 <kmARC> o/
21:01:02 <tdasilva> helo
21:01:04 <sgundur1> hi
21:01:43 <notmyname> acoles_: torgomatic: ping
21:02:15 <notmyname> I'm guessing acoles_ might me getting back online shortly. might need to give him a few minutes :-)
21:02:32 <notmyname> but before we get to the crypto stuff...
21:02:38 <notmyname> welcome, everyone
21:02:41 <notmyname> #topic hackathon
21:02:54 <torgomatic> ohai
21:02:58 <notmyname> the hackathon in san antonio is coming quickly. just a week and a half from now
21:03:14 <mattoliverau> \o/
21:03:16 <notmyname> today I set up an etherpad to start collecting topics
21:03:18 <notmyname> #link https://etherpad.openstack.org/p/swift_SA_hackathon_2016
21:03:26 <notmyname> so feel free to add your ideas to that
21:03:49 <notmyname> also, if you haven't booked your hotel yet, DO IT NOW! the room block is about to expire
21:04:07 <notmyname> hurricanerix: (are you here?) anything to add about the hackathon?
21:04:14 <notmyname> any questions from anyone about the hackathon?
21:04:53 <dmorita> Do we have to get a car to get to RackSpace?
21:04:54 <mattoliverau> doesn't look like it
21:05:00 <dmorita> Or shuttle is provided?
21:05:17 <notmyname> I'm told there is a bus from that hotel to the RAX office
21:05:24 <dmorita> nice :)
21:05:29 <mattoliverau> dmorita: there is a shuttle if you stay at the aloft
21:05:42 <notmyname> it's not quite like acoles_ had for us in bristol, I think
21:05:45 <notmyname> mattoliverau: what's it like?
21:05:52 <notmyname> I mean, will it even seat all of us?
21:06:09 <bkeller`> Do they have Uber out there?
21:06:23 <pdardeau> bkeller`: yes
21:06:24 <tdasilva> does it run multiple times?
21:06:55 <mattoliverau> yeah, it runs on the hour, you just have to ask at the front desk, or call to get picked up
21:07:34 <mattoliverau> its a minibus, so probably seats 20 or so.
21:07:44 <notmyname> ok
21:07:45 <mattoliverau> maybe less, I've never counted
21:08:11 <mattoliverau> if there are more they might run a little more frequent
21:08:19 <notmyname> I'm goign to plan on using the shuttle and uber while I'm there and not get a rental car, FWIW
21:08:30 <mattoliverau> +1
21:08:38 <dmorita> ok :)
21:08:48 <mattoliverau> if all else fails you can get a shuttle to the airport and hire a car
21:08:53 <jrichli> I'll have a car - cause i am drivign from Austin.  so i can take a few passengers
21:09:03 <notmyname> the hotel is *very* close to the airport
21:10:07 <notmyname> I'm looking forward to it. coming right off of finishing crypto, it will be a great launching point, of sorts, for the next things :-)
21:10:20 <notmyname> well, that, and I just really like seeing everyone at hackathons :-)
21:10:35 <briancline> the folks working on crypto deserve an uberchopper ride
21:10:40 <notmyname> any other questions (that I'll punt to mattoliverau and pdardeau to answer)?
21:10:58 <jrichli> briancline: :-)
21:11:00 <notmyname> briancline: lol
21:11:29 <notmyname> ok, let's move on to crypto
21:11:33 <notmyname> #topic crypto
21:11:51 <notmyname> acoles_: sent me a message saying he's having some network problems and will be on ASAP
21:12:03 <notmyname> #link http://not.mn/swift/swift_community_dashboard.html
21:12:07 <notmyname> that page has 2 links on it
21:12:17 <notmyname> the 2nd is a nice dashboard that timburke made
21:12:27 <notmyname> IMO, crypto seems *really* close to landing
21:12:30 <notmyname> acoles: hi!
21:12:51 <notmyname> acoles: just started the crypto topic. you didn't miss any of that
21:12:52 <acoles> hi sorry, got myself tethered
21:13:22 <notmyname> acoles: did those people in bussels cut your internet connection already?
21:13:29 <mattoliverau> lol
21:13:31 <acoles> seems that way ;)
21:13:48 <notmyname> acoles: can you give a summary of where we are with crypto-review?
21:14:17 <acoles> I think we are in good shape (although I have not checked for -ve reviews in last hour)
21:14:34 <acoles> Since last week we have addressed the concerns over etag encryption
21:14:47 <acoles> and the concerns over constraints checking
21:15:23 <acoles> and the final "last minute" change to have just 2 middleware filters (keymaster + encryption) looks pretty simple imho
21:15:33 <acoles> and that will make config life much easier ^^
21:16:01 <acoles> So first four patches have +2s , then we have patch 328208
21:16:12 <acoles> hello patchbot?
21:16:27 <notmyname> patch 328208
21:16:27 <patchbot> notmyname: https://review.openstack.org/#/c/328208/ - swift (feature/crypto-review) - Enable object body and metadata encryption
21:16:31 <acoles> yay
21:16:41 <acoles> that is the meat of the encryption feature ^^
21:16:48 <jrichli> nice!
21:16:48 <notmyname> for the etag, we're now storing the hmac of the etag and using that for comparisons in conditional requests. it's a clever update that means we simplify some of the crypto code (no more derived IVs) and still support the full API
21:17:31 <jrichli> hmac was a great idea acoles.  glad we have a better solution there now
21:17:36 <notmyname> for the constraints checking, we're now storing user metadata keys encrypted in the [transient] sysmeta space. this means we don't have to worry as much about constaints being violated or overwriting the user namespace
21:18:07 <notmyname> and the unification of decrypter and encrypter should make deployments easier for operators (less pipeline management)
21:18:20 <acoles> so all eyes on that patch 328208, and I am primarily interested in whether we are making any mistakes in what we write to disk or how we do config i.e. things that are hard to go back and change
21:18:21 <patchbot> acoles: https://review.openstack.org/#/c/328208/ - swift (feature/crypto-review) - Enable object body and metadata encryption
21:18:30 <notmyname> although there's a good question from mathias and response from timburke about it
21:19:36 <notmyname> acoles: right. other stuff can be changed later on master without worrying nearly as much about migrations and upgrades
21:20:00 <acoles> oic, I will reply to that too, but IMHO the decrypter has to drive when the keymaster returns keys because only the decrypter understand when it is decrypting metadata from a POST vs data from a PUT
21:20:33 <notmyname> acoles: jrichli: although it's not in the docs (which is ok IMO), I'd like to add something to the CHANGELOG about "crypto should probably be enabled for new clusters instead of added to existing clusters". how do you feel about that?
21:20:56 <clayg> notmyname: +1
21:21:04 <notmyname> (in addition to the lots of other words I need to write int he changelog about crypto)
21:21:11 <clayg> but really as long as it gets in before the next *release*
21:21:13 <clayg> it's just words
21:21:50 <notmyname> right. that's what i was looking forward to. for whenever the next release happens. probably won't be right after the crypto branch lands, but I don't expect it to be too long
21:22:01 <acoles> notmyname: it begs the question "why, does it break existing cluster?". We've worked very hard to not break existing clusters.
21:22:43 <jrichli> yes, i was trying to somehow say that we'd want to communicate that you *could* do it.  I am not sure why we want to discourage it, TBH
21:22:58 <notmyname> acoles: true. my reasoning is that adding it to a new cluster is the only way to be sure that all the data is actually encrypted. not that I expect it to break on existing clusters. just trying to give guidance about when to use it for best results
21:23:34 <acoles> notmyname: so those words sound appropriate "adding it to a new cluster is the only way to be sure that all the data is actually encrypted"
21:23:42 <notmyname> jrichli: ok. so something like "while you can enable this for existing clusters, best practice would be to create a crypto cluster and migrate your data"
21:23:56 <jrichli> notmyname: i like that
21:24:20 <timburke> fwiw, i think there are two pieces missing that should make operators hesitate to turn it on in existing clusters: first, a process to go around encrypting unencrypted data, and second, some way of monitoring that process's progress
21:24:21 <notmyname> yeah, it's similar to my thinking on "hey, you shoudl encrypt client side instead of trusting swift to do it for you; but if you don't here's this crypto feature we wrote for you"
21:24:26 <tdasilva> a copy would encrypt right
21:24:27 <tdasilva> ?
21:24:34 <clayg> notmyname: I'd prefer the recommendation around "whild you *can*" to be a little stronger
21:24:59 <notmyname> I don't know the right words yet. I was just asking about the general idea to find everyone's comfort with that
21:25:08 <pdardeau> anyone have an idea how much extra overhead there is for encryption?
21:25:11 <notmyname> I expect it to be a few weeks before I actually have to write the words
21:25:16 <notmyname> pdardeau: non-zero
21:25:16 <timburke> and if we ever get torgomatic's audit watchers patch landed, it would be useful for that second half, as well as monitoring progress of key-rotation (once *that's* implemented)
21:25:17 <acoles> tdasilva: yes
21:25:22 <clayg> I have *no* understanding of a use-case where you only want to turn on encyrpiton for the whole cluster - but only want to encrypt some of your data
21:25:37 <torgomatic> heh, I should resurrect that
21:25:44 <notmyname> timburke: it'll be great in the future!
21:26:07 <pdardeau> clayg: wouldn't the rma drives count?
21:26:08 <torgomatic> last time I submitted it, though, it clogged up zuul pretty well, so I've been really hesitant to touch it
21:26:09 <timburke> everything is gonna be just grand in the future!
21:26:14 <notmyname> pdardeau: in seriousness, crypto is expected to have some degree of cost, but that hasn't been exactly quantified
21:26:15 <tdasilva> clayg: I don't understand what you said
21:26:15 <torgomatic> (and yes, it worked on my machine (tm))
21:26:17 <clayg> pdardeau: i'm looking at this - it's like *a bunch* - although I don't think it's the encryption as much as all the other stuff to support it (md5 in the proxy, footers, chunked transfer, mime-doc, etc)
21:26:48 <clayg> pdardeau: can't rma drives that have unencrypted data on them?  so "upgrading" is non-sensical for that use-case
21:27:06 <notmyname> tdasilva: ^
21:27:14 <pdardeau> clayg: sure you can, but i thought that was the primary motivation behind adding it
21:27:23 <clayg> tdasilva: about "upgrading" a cluster to encrypt "everything" is a mis-feature?
21:27:23 <acoles> pdardeau: clayg: my measurements also showed the extra md5's to cost more than the encryption
21:27:33 <clayg> or non-feature; depending how you look at it?
21:27:38 <clayg> acoles: yes :'(
21:28:26 <clayg> pdardeau: I think encrypting everything as an operator feature is totally a thing - that's why I suggest you steal some capacity out of your existing cluster and plan a migration
21:29:17 <notmyname> right, because if only *some* of your data in encrypted, then you can't RMA any of the drives in the cluster because there might be plaintext data on it
21:29:47 <tdasilva> but you might not care about the data that is unencrypted
21:29:56 <notmyname> then RMA anyway ;-)
21:30:22 <acoles> notmyname: all: so back to the near term (this week) - I need feedback on the change in patch https://review.openstack.org/335641, ideally in next 12 hours so I can squash it in (or not), then barring anything that is broken, I hope we are close to done.
21:31:01 <clayg> tdasilva: maybe if you have some use-cases that are *not* on your swift because of lack of encyrption, and once you turn it on they can start using it - encyrption of the new data from the old use-cases is just ... for fun?
21:31:02 <notmyname> acoles: right before the meeting I had said in -swift that aside from the few small comments in docs and that patch, I'm pretty happy with it all
21:31:15 <acoles> notmyname: oic. thanks
21:31:18 <tdasilva> clayg: yeah, that's what i was thinking
21:31:41 <notmyname> acoles: I still need to grok the conversation on the unification patch ( timburke's reply). but yeah
21:32:05 <notmyname> ok, so here's a question for everyone, based on what acoles jsut said
21:32:08 <clayg> tdasilva: ok, but I think that lends better to either standing up an *encrypted* cluster for those use-cases; or moving to encryption on/off per-policy
21:32:29 <notmyname> who here will expect to be able to leave a review on teh crypto branches within the next 12 hours?
21:32:36 * notmyname will
21:32:42 <timburke> o/
21:32:56 * mattoliverau will
21:33:08 * cschwede too
21:33:09 <kota_> o/
21:33:24 <mattoliverau> it is only 730am here, so I have all day :)
21:33:39 <torgomatic> I just left feedback before the meeting, so probably nothing new from me
21:33:45 <kota_> mattoliverau: 6:30 for me ;)
21:34:00 * jrichli will review the most recent patch from acoles for sure
21:34:10 <notmyname> ok, great :-)
21:34:14 * mattoliverau is more east then the land of the rising sun.. i still find that crazy
21:34:19 <cschwede> kota_ and mattoliverau are cheating! ;)
21:34:26 <notmyname> acoles: so 12 hours or when all these people leave a review. whichever is first :-)
21:34:29 <mattoliverau> cschwede: :P
21:34:36 <kota_> lol
21:35:26 <notmyname> acoles: and I'm expecting that there will likely be just one more patch chain push. then the final one should be able to get all the approvals and we'll do a merge commit. ideally on my friday
21:35:49 <acoles> ok thanks everyone, so that govesme tomorrow to process comments and then...yeah, that ^^
21:35:58 <acoles> gives me*
21:36:06 <notmyname> early next week, those of us in the US will be celebrating the anniversary of the first brexit, so might not be online (I'll be unavailable until my tuesday pm)
21:36:29 <clayg> lol
21:36:31 <mattoliverau> brexit v1
21:36:34 <notmyname> so let's get it landed by friday :-)
21:36:34 <clayg> notmyname: have you been saving that one?
21:36:40 <acoles> notmyname: "culturally insensitive holiday" ;)
21:36:41 <notmyname> clayg: yes ;-)
21:36:47 <cschwede> ymmd!
21:37:35 <notmyname> anything else from anyone on crypto for this meeting today?
21:38:13 <clayg> i'm freakishly dissappointed at my inability to find *BUGS*
21:38:26 <acoles> clayg: PM me and I'll share a few
21:38:27 <notmyname> clayg: have you met acoles? ;-)
21:38:28 <clayg> I keep beating on it with all kinds of crazy chaos crap and and can't get any tracebacks
21:38:29 <briancline> lol, first brexit. before it was cool
21:38:29 <jrichli> its acoles's fault!
21:39:01 <notmyname> for expectations-setting, after crypto lands, I want to get some of the other bugfixes landed and some things waiting like the container sync updates. then after that do a release
21:39:07 <acoles> I'm just the editor, I just type what timburke dictates ;)
21:39:19 <clayg> the closest I got was filling up my drives and getting some currpt metadata that seems to makes the objects unreadable; haven't quite sussed that one out - but christ - the drives were out of bytes
21:39:31 <jrichli> lol.  timburke rulz
21:39:33 <clayg> all hail timburke !
21:39:44 <timburke> clayg clearly should have gotten involved earlier :P
21:39:51 <acoles> notmyname: did the api docs land yet?
21:39:54 <clayg> no - this is perfect
21:40:13 <notmyname> acoles: no. that's another thing. I'll look at those later this afternoon, in fact
21:40:27 <notmyname> ok, that's all that's on the agenda for this week
21:40:39 <notmyname> thank you, everyone, for reviewing the crypto work
21:40:55 <notmyname> and thanks expecially to acoles for managing it and timburke and jrichli for pouring so much work into it
21:41:05 <notmyname> #endmeeting