21:00:40 #startmeeting swift 21:00:40 Meeting started Wed May 11 21:00:40 2016 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:41 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:43 The meeting name has been set to 'swift' 21:00:46 who's here for the swift team meeting? 21:00:50 me 21:00:55 me too! 21:00:58 hi 21:00:59 o/ 21:01:00 o/ 21:01:01 happy hump day everyone! 21:01:04 me 21:01:07 o/ 21:01:08 \o/ 21:01:10 o/ 21:01:19 hello 21:01:20 hello 21:01:24 o/ 21:01:30 joeljwright: quick note, https://review.openstack.org/#/c/313812/ has one +2 so that's good! 21:01:32 here 21:01:55 welcome, everyone 21:02:01 notmyname: I'll go add my +1 21:02:13 what a crazy week in the land of openstack (we'll get to details on that in a bit) 21:02:25 agenda for this week is 21:02:26 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:36 #topic crypto update 21:02:50 jrichli: acoles: you're up. what's going on in crypto? 21:03:19 yeah, acoles - you're up ;-) 21:03:35 progress is being made! and it looks like copy middleware may land real soon now which will be great 21:03:47 great 21:04:07 the priority review list is up to date https://wiki.openstack.org/wiki/Swift/PriorityReviews 21:04:27 great 21:04:31 we have two priority patches there - one is to wrap object data keys which we decided on in Austin 21:04:53 acoles: jrichli: is there something we should focus on as a group that we can have done by next week's meeting? 21:05:01 the other is a fix to how we handle conditional GETs so as to not break unencrypted objects 21:05:28 copy, copy, copy 21:05:53 :-) 21:06:11 notmyname: copy, yes. and patch 314740 because I would like to know sooner if anyone objects to my solution there 21:06:12 acoles: https://review.openstack.org/#/c/314740/ - swift (feature/crypto) - crypto - don't break conditional GETs on unencrypt... 21:06:13 top priority is https://review.openstack.org/#/c/156923/ 21:06:14 notmyname: patch 156923 - swift - Refactor server side copy as middleware 21:07:23 notmyname: apart from that we have some cleanup patches (timburke is doing fantastic job reviewing those) and some work to do on more tests 21:07:44 sorry, I -1 it (copy)... oh there is already a new version of copy.. I need to look at that again this morning then :) 21:07:44 tdasilva and timburke have been going back and forth on the copy middleware patch. and I think mattoliverau is on it as well 21:07:59 mattoliverau: yeah, they were just talking about it a few hours ago 21:08:11 just pushed a new patch set about 2 hours ago 21:08:18 notmyname: a stretch goal would be to starting to prepare small chain to propose to master, on feature/crypto-review as discussed in Austin 21:08:18 mattoliverau: made the changes you suggested 21:08:23 by next week 21:08:28 acoles: wow, that's great :-) 21:08:40 great, I'll take a look at copy first thing then :) 21:08:41 acoles: do you need help from me on that? 21:08:53 notmyname: *stretch* goal relies on reviews happening, and I meant Weds next week ;) 21:08:53 acoles: I'm assuming we'll use a different branch name for that... 21:09:03 acoles: i'm finishing up reviewing the key wrapping patch 21:09:19 notmyname: I'll ned help to create the feature branch - feature/crypto-review ?? 21:09:40 tdasilva: thanks! 21:10:22 acoles: ok, I can do that very easily as soon as it's needed. (technically I don't do it, but someone in -infra does. but we've done it several times before) 21:10:51 notmyname: ok. well, not quite yet...I may be being overly optimistic 21:10:51 I'm excited about the progress being made. looking forward to it :-) 21:11:06 +1 21:11:15 acoles: ok. I've got enough other email threads to keep me busy until then :-) 21:11:38 anything else on crypto to bring up? 21:11:42 notmyname: what topic would they be on ? ;) 21:11:59 notmyname: not from me 21:12:24 hi 21:12:49 #topic [new|no]-specs process update 21:13:06 ok, let's start the craziness that's been going on 21:13:53 so in their post-summit report, there's a plan from -infra to deprecate using the wiki and currently all new accounts are disabled. so that kinda stinks for our ideas page 21:14:47 however, that news prompted some response and now there's some more interest in finding a lightweight think to either replace the existing wiki or to streamline it to make it still work 21:15:13 and ttx sent an email out this morning about documenting the current wiki use cases 21:15:18 there's an etherpad somewhere... 21:15:32 #link https://etherpad.openstack.org/p/wiki-use-cases 21:15:40 good news is that's swift's use cases are already there 21:16:06 but if you really like wikis and the functionality they provide, there's a new way you can get involved and spend time working on openstack stuff :-) 21:16:33 I'll keep an eye on it and let you know what happens and what if anything needs to change with our new no-specs ideas page stuff 21:16:44 #topic golang update 21:16:55 ok, here's where the really crazy stuff has been happening :-) 21:17:18 we're 8 days and 130 messages in to a ML thread about using golang 21:17:33 it's either hilarious or will drive you to serious drinking ;-) 21:17:39 py3 asyncio uvloop and an aiofile interface will save python! https://nikhilm.github.io/uvbook/filesystem.html#filesystem-operations 21:17:53 you need popcorn to read it 21:18:03 ... or at least make it a sufficent approximation of the golang runtime that when pypy gets py3 support we'll be w/i a few points ;) 21:18:27 clayg: now all we need is a decent thread scheduler for those async calls, and we're golden! 21:19:05 notmyname: if we can use the libuv calls it'll be in c and dispatched to uvloop directly so we're set - 3 years - tops 21:19:13 so to quote the princess bride, "let me explain....no there is too much. let me sum up" 21:19:21 lol 21:19:48 there's been some crazy diversions down off-topic things in the thread. so let's ignore that 21:19:48 lol 21:20:09 point 1 is that we aren't the only project looking at using golang in our codebase 21:20:51 that's good, because it means we can avoid some of the "oh it's swift and they're *special*". no. we can work cross-project and get something that is good for us and good for everyong 21:21:41 * morgan lurks. 21:22:02 point 2 is that torgomatic has done a good job dropping some technical knowledge and experience in there. that help shut down the "oh well I spend a few minutes looking at it and you obviously haven't though about it hard enough" 21:22:09 so, thanks torgomatic 21:22:19 (although he's not here) 21:22:27 torgomatic is ey 21:22:27 +1 21:22:28 *key 21:22:42 point 3 is that the real questions and reasons for the thread are starting to be raised 21:23:03 namely, the important stuff is about "how do we actually make all this work in the CI/gate/docs/tests/etc" 21:23:32 stuff like dependency management and discovering and building docs and what a release actually looks like when you've got compiled code 21:23:51 yesterday I went to the -infra team meeting where those questions were specifically brought up 21:24:13 https://etherpad.openstack.org/p/golang-infra-issues-to-solve is the current scratch pad for asking and answering a lot of these questions 21:25:13 there's currently some exploration about different ways to do dependency management in golang (the golang community doesn't yet have "One Way It Must Be Done" so there's lots of confusion out there) 21:25:37 :( 21:25:50 3 things being considered (linked in the etherpad) are godep, glide, and govendor. mordred proposed a patch for each on the hummingbird branch just so we can look at them and see what it looks like 21:26:32 another good question (which I'm hoping to get better answers from dfg et al on) is about how to do unit tests and prublish the results 21:26:34 sounds like a barrel of monkies 21:27:03 so in summary, there's a lot going on, mostly distracting things, but some really good progress overall (I think/hope) 21:27:24 some of these will be figured out as we go and a few need to be known up-front 21:27:37 (hee hee) as we go 21:27:51 notmyname: we run tests with "make test" as far as publishing the results- idk 21:27:53 oh the puns! 21:28:02 dfg_: cool. in a Makefile? 21:28:05 ya 21:28:10 dfg_: perfect 21:28:53 so overall, don't get disheartened about the openstack craziness (I mean that in the nicest possible way). there's some good questions being raised and a whole lot of interest in finding good solutions 21:29:43 and in a lot of ways, we get to have a pretty big influence because we've actually got some golang code and a vague plan of how/when to merge that 21:30:04 so any questions on all of that? 21:31:18 nope, I'll continue eathing my popcorn and watch :P 21:31:36 ok. feel free to ping me any time (publicly or privately) if you want to talk about it or are concerned or need to vent or whatever 21:31:57 #topic rolling upgrade test status update 21:32:09 cschwede couldn't make it but had this update... 21:32:35 still waiting on the infra patch, and has a promise of a review soon 21:32:41 #link https://etherpad.openstack.org/p/swift-rolling-upgrade-multinode-testing 21:33:07 so not much else to say there. patch 304465 I think is the one that's blocking further work 21:33:07 notmyname: https://review.openstack.org/#/c/304465/ - openstack-infra/devstack-gate - Use subnodes for Swift storage nodes in a multinod... 21:33:27 #topic symlinks 21:33:35 jrichli: you're up. talk to us about symlinks 21:34:05 There are some summaizing statements at the top of https://etherpad.openstack.org/p/swift_symlinks 21:34:14 There is some great history centered around the POST discussions in an etherpad linked from there. 21:34:31 each of the different ideas were given a label. It would be good to get everyone's feedback on what approaches to rule out, and which to focus on. 21:34:51 acoles had thoughts on the plane ride back from the summit. I was interested in hearing whether that aligns with 'Proposal symlink-POST-4' 21:34:52 havent' we talked about that in tokyo, bristol, and austin? 21:35:32 * clayg goes to check if he's still "winning" 21:35:44 yes - that etherpad catures some of the past history. its a good place to start if you weren't around for those talks. I know we had some new comers to the topic in austin 21:35:54 I haven't had much time to think about it more 21:36:12 ok. my concern is just that it's written down so we don't have to keep having the same discussion 21:36:13 clayg: you and acoles could add notes from your Friday discussion :-) 21:36:33 jrichli: yeah I need to quick before acoles proves me wrong again! 21:37:02 right. it'd be good to have a summary of "this idea is winning" - like clayg said 21:37:13 Do we have a clear idea of what the symlink use case(s) are that we're trying to address? in particular whether the client can and will make requests to the symlink only or to the target (only/mostly) or both? 21:37:50 cos clayg convinced me that good use cases lead to winning solutions 21:37:54 oh oh oh! start with the use-case! I like it! 21:38:07 isn't our first use case tiering? 21:38:19 i think one of those etherpads had started with that ... one sec 21:38:19 notmyname: nope - it's manual policy migration 21:38:29 ! 21:38:38 notmyname: I have a name, i like the name, my cdn likes the name, but I want the data to be in another container - COPY + PUT symlink 21:38:47 clayg: ok. s/tiering/policy migration/ 21:39:12 i remember also talking about a new object versioning using symlinks 21:39:13 notmyname: well then there's also cluster assisted policy migration which is something ntt is looking at 21:39:20 notmyname: I think the important differentiation is client involvement vs automated server side tiering 21:39:23 oh right versioning 21:39:25 hmm ... i dont see use cases spelled out 21:39:34 acoles: indeed it is 21:40:01 and whether the client is aware of the target, and could therefore expect to make requests to it 21:40:04 tdasilva: yeah. I get asked about that one nearly every day now (internally from product people) 21:40:24 * tdasilva needs to start reviewing timburke's patch 21:40:41 yes please! 21:40:43 notmyname: versioning or swift3 improvments related to object versions? 21:40:44 ok so we've got 2 initial use cases: 1) manual policy migration 2) versioning 21:40:58 clayg: mostly the s3 compat stuff and whatever's needed to get that :-) 21:41:01 and where it really matters is POST handling, which the etherpads draw attention to. 21:41:06 notmyname: everytime i ask timburke about symlinks he says it's useful but not required 21:41:20 ah, tiering also an use case still 21:41:33 it's under development though. 21:41:39 kota_: auto tiering after manual tiering 21:41:46 yump 21:41:47 btw: we are simulating symlinks using SLO with only one segment now. so symlinks is nice but "not needed" :) 21:41:47 oh 21:41:48 yup 21:42:10 onovy: us too :) 21:42:17 how does a post to a SLO manifest work :D 21:42:17 yes 21:42:24 clayg: notmyname: i can get something functional for swift3 following https://review.openstack.org/#/c/214922/ - it will be oh-so-much-better if we have symlinks, but that'll basically come along for free 21:42:24 timburke: patch 214922 - swift - Add "history" mode to versioned_writes middleware 21:42:33 clayg: don't know, we are not posting to it :) 21:42:39 onovy: good call 21:42:42 clayg: that was my thought too. seems like we are all solving similar problems for parts 21:43:01 jrichli: parse error "solving similar problems for parts" - cut off? 21:43:25 there are some consistency issues both topics face 21:43:44 solution could be similar for both? 21:44:30 jrichli: notmyname: there's so much written down and referenced in those eather pads - it's really hard to put it all in your head :'( 21:44:38 jrichli: maybe but the solutions could be quite different depending on the use case/requirements. 21:44:42 clayg: +1 21:46:19 acoles: yes. we should not feel restricted to solve in the same way. 21:46:20 jrichli: e.g. If the client has no awareness of the target then I *think* that POSTing to the symlink only is workable. But if the client wants to GET the target direct then you need a different solution. 21:46:50 jrichli: but choosing one problem to solve first might help progress to a solution 21:47:02 comes down to the use case right? 21:47:14 notmyname: yeah, clayg is right on that! 21:47:17 agreed. so focus on manual now? 21:47:24 so where's that written down? manual policy migration 21:48:02 notmyname: it *was* in the spec i believe - it was like *the* example that sam gave for symlinks before everyone thing else and their dog realized they needed something similar as well? 21:48:14 unless timburke convinces us that we should do it for versioning first (assuming that's a different implementation) 21:48:14 who edits the eatherpad! name yourself! 21:48:25 me 21:48:37 specs gave us traceability, bring back specs :P 21:48:41 jrichli: i just ment on the right hand side where you can type in your name 21:48:58 acoles: you didn't acctually like specs - you like the *idea* of specs 21:49:01 https://specs.openstack.org/openstack/swift-specs/specs/in_progress/symlinks.html 21:49:22 clayg: lol. i added it 21:49:54 jrichli: lol - you see the little "people" icon in the upper right? 21:49:56 does the naming in etherpad persist? i always seem to lose the identities from the right column 21:50:10 jrichli: nice 21:50:16 clayg: sorry, thought i had that already 21:50:23 it probably forgot 21:50:30 yeah, i had to reboot today 21:50:39 acoles: everytime i use eatherpad it seems to know i'm clayg? 21:50:45 oic 21:50:57 so only as long as I have this browser open (which is foreverish) 21:50:58 and i upgraded my browser 21:51:07 holy cow! cutting edige 21:51:11 clayg: but I'm not sure I always know you are clayg (well, apart from the typos :P) 21:51:21 the color changes sometimes 21:51:38 burn 21:51:45 :D 21:51:48 lolz 21:51:50 :D 21:52:05 * acoles apologizes 21:52:19 :D :D 21:52:45 ok, so where are we? 21:52:54 notmyname: clayg can't spell 21:52:59 it's *super* funny 21:53:03 lol 21:53:04 yeah, I knew that ;-) 21:53:20 well you spell diferently to acoles and I :P 21:53:36 ok ok. symlinks 21:53:44 anyway, back to manual symlink policy migration use case - imho POST handling becomes hard (impossible?) if the client expects to get the same result from the link or the target - but is that what the use case requires? I'm not sure 21:54:48 seems like a part of discussion in auto-tiering from m_kazuhiro 21:55:03 he might have thought for that. 21:55:30 seems like reading over the etherpad and recording your thoughts is the next thing 21:55:35 on https://etherpad.openstack.org/p/swift_symlinks 21:56:03 sounds good 21:56:04 does that seem reasonable for everyone? 21:56:19 sure 21:56:24 kota_: m_kazuhiro's proposal is what I would describe as server side tiering - client does not know about targets 21:56:37 or server side migration 21:56:41 acoles: ok 21:56:51 In my auto-tiering, client will not touch the target object (reffered by symlink), so client will get same result anytime. 21:57:08 m_kazuhi_: right, thanks 21:57:10 m_kazuhi_: gotcha, thanks 21:57:28 #topic open discussion 21:57:32 any update on hackathon dates/location? 21:57:33 anything else to bring up this week? 21:57:44 joeljwright: Real Soon Now(tm) 21:57:58 oh i wanted to say i have no strong preference on the location of the hackathon so i'm putting my vote up for bribe 21:58:13 clayg: lol 21:58:16 lol 21:58:21 clayg: I will contact you separately ;) 21:58:30 we'll start the bidding at one can of domestic beer? Do I hear a draft beer at a pub near the hack-a-thon? 21:58:43 did I understand correctly from the low hanging fruit session that now is the best time for any refactoring and cleanup patches? 21:58:45 clayg: one can of "America"? 21:59:09 acoles: i believe both suggested locations were in N.A. 21:59:19 actually turns out that the location will be mostly determined based on schedule. there's several things going on this summer, so finding the gap is the trick 21:59:35 rdopiera: there's never a "best" time - just that "right before release" is the "worst" time? 21:59:36 evergreen question: what about io nice? 21:59:40 rdopiera: what'd you have in mind? 21:59:46 onovy: we should flipping merge that! 21:59:53 onovy: only needs one more +2! 21:59:59 doo eet! 22:00:17 srly, you just check out the change edit your config and run a few shell commands to make sure the stuff got applied 22:00:38 we're at time 22:00:42 clayg: there are some patches in the quque that fix style and enable pep8 rules, there is also the mox removal thing, some patches for removing calls to locals(), globals() and exec(), etc. 22:00:46 if you want to be mean you can try setting bogus/garbage in the config and complain that aside from *maybe* an error in the log there's little indication it's not doing anything useful 22:00:47 go review the ionice patch :-) 22:01:13 thanks for coming, everyone. thanks for your work on swift 22:01:17 #endmeeting