21:00:46 #startmeeting swift 21:00:47 Meeting started Wed Jun 19 21:00:46 2019 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:51 The meeting name has been set to 'swift' 21:01:00 who's here for the swift meeting? 21:01:03 o/ 21:01:06 o/ 21:02:15 o/ 21:02:31 sounded like kota_ wouldn't be able to join, and rledisez is still moving 21:02:55 let's get started! 21:03:10 agenda's pretty short 21:03:13 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:03:27 so, let's do some updates 21:03:30 #topic py3 21:03:43 all the unit tests have been approved! 21:04:22 just waiting on locale and tempurl to make their way through the gate 21:04:43 last blocker for a release is being able to run func tests under py2 against services running py3 21:06:13 that requires a couple small fixups 21:06:14 https://review.opendev.org/#/c/665487/ to make func tests not break prefixes part-way through a codepoint 21:06:36 https://review.opendev.org/#/c/665746/ to get some things in s3api to be native strings 21:06:41 wow, cool nice work. Sorry I'm been away most the last week (since before the last meeting) and still playing catchup here and at work. 21:07:36 Is the priority reviews page up to date? 21:07:38 https://review.opendev.org/#/c/665773/ to make ECAppIter happy, but i might be willing to known-issues it for now 21:08:00 mattoliverau, mostly. i'll make sure it is shortly after the meeting 21:08:03 oh looks like it is. nice 21:08:05 cool 21:08:15 a bunch of stuff landed just this morning :-) 21:09:03 https://review.opendev.org/#/c/662546/ is more meaty, required to work around https://bugs.python.org/issue37093 so we can send and receive non-ascii header names 21:10:03 i think once we get those in, i'll tag 2.22 21:10:31 Awesome 21:10:43 TAG IT 21:11:02 next steps are things like getting a py3 dsvm job (https://review.opendev.org/#/c/653548/) and porting func tests (there's a list on https://wiki.openstack.org/wiki/Swift/PriorityReviews) 21:11:09 clayg, REVIEW IT 21:11:10 :P 21:11:22 seems reasonable 21:11:47 lol 21:12:14 after those, we try to suss out what kind of a test plan we'd like to see to declare full support 21:12:21 well, that and probe tests 21:12:44 anybody have questions on py3 or what needs doing there? 21:13:47 #topic lots of small files 21:13:55 alecuyer, how's it going? 21:14:29 I have written more tests for vfile.py, the classes this time. That was slow and I restarted once, but I hope to have something "reviewable" soon 21:14:42 yay! 21:14:55 I pushed it here https://review.opendev.org/#/c/666378/ but I don't think it's worth looking at yet :) 21:15:58 what can we do to help? more reviews, more patches, more manual testing? 21:16:57 Well if someone wants to take a quick look , just to check if it's very wrong, but otherwise I think I just have to write these tests :) 21:17:03 i really need to try out https://github.com/swiftstack/vagrant-swift-all-in-one/pull/80 sometime ... 21:17:14 sounds good 21:17:50 #topic open discussion 21:18:00 anything else to bring up? 21:18:35 i know clayg's been looking at symlinks and versioning and s3api lately 21:18:59 it's gettin there 21:19:21 the hardlink to symlink thing is getting so annoying 21:20:35 should we give a little background on it to help mattoliverau and alecuyer catch up? 21:21:06 yes please :) 21:21:27 hardlink is "symlink with md5 of target" or is that very wrong 21:21:35 that's it 21:21:51 the neat thing this does is allow us to keep some information uptodate in the container listing 21:22:13 so normally when you put a symlink your container listings just show these zero byte objects with the symlink_target key 21:22:43 md5 of target, sounds like a bitch to keep up to date in the symlink 21:22:50 but when you KNOW this symlink doesn't work unless the target doesn't change you can keep the hash (and the content-length) in the listing and you know it's valid (at least for the lifetime of the hardlink) 21:23:08 yeah so the use-case was actually versioned writes 21:23:53 so versioned writes today does a lot of moving data around to keep the "active" objects actually "in" the container... but with symlinks we can solve for data protection requirements in place 21:24:26 and with using hardlinks we can even make listings of the version'd container looking reasonable (as far as etags & bytes) 21:24:29 so that's all great 21:25:02 it also resolves some race conditions when you have concurrent overwrites to the same (biggish) object 21:25:26 somewhere along the way the idea of "use hardlinks to point to the versions" got sideways when the the "version" was actually a symlink... just cause when we fleshed out hardlinks we didn't really define the behavior for hardlink to symlink... 21:26:02 right as "a bunch of data movement" wasn't bad enough - there's also the fact that it takes awhile and leaves wide race windows 21:26:20 lol, wow. 21:26:23 that's a case we need to support then ? 21:26:24 symlink based versions is awesome, and with hardlinks there's very little downside 21:27:29 the hardlink to symlink use-case? currently I have versioned writes just doing symlinks when the target in the version'd container is itself a symlink... and that may be the way to go if we can't come up with a definition for the behavior of hardlink to a symlink that we WANT 21:27:48 alecuyer, i like your thought process :-) 21:28:03 i think part of it was that we have existing functional tests with versioning and symlinks, and we want to make sure they still pass 21:28:15 but now I think I see the case for it with clay's explanation ;) 21:28:44 the usecase for *hardlinks* or specifically for hardlinks *to symlinks* 21:29:15 ah, hardlinks I had no doubts, hardlinks to symlinks I meant 21:29:41 I think the use-case for hadnlinks is pretty solid - the etag validation is nice, but the benifit is mostly the container listings - by making more concret the relationship with the link and the target it because reasonable to maintain state about what that hardlink points to... 21:29:43 so that's great 21:29:50 yeah hardlinks to symlinks 🤮 21:30:25 oh yeah, ok so for versioned writes hardlinks sound great.. once the edge cases are thought through. Tho a random symlink with a pointer to an object elsewhere, with an md5 of the target seems hard to keep up to date. Unless you know the target never changes. 21:30:27 it just ends up akward in versioned writes having to track if it's versioning an object (write a hardlink) or a symlink (write a symlink) 21:32:35 but hey maybe people actually want that. You'll then know if somethings changed. maybe there are usecases you'd actaully want to use that. Like a tamper seal 21:33:33 I think there's definately use-cases for symlinks and hardlinks 21:33:48 But I haven't looked at the code, so only making assumptions on the discussion here. I should go take a look :) 21:33:56 versioned writes definately prefers hardlinks because it makes the redirection more transparent 21:34:19 everything I have posted is rusty - i'll try to push up some current revisions soon 21:34:19 I'm lost now, I thought you woulnd't update the md5 (if the target changes the link is broken ?) but I should try it and read the code :) 21:34:28 it's that one right ? https://review.opendev.org/#/c/633094/ 21:34:33 no, i need to push 21:34:39 idk why tim brought it up 21:34:42 :P 21:34:54 by next week i'll have some patches 21:35:08 because it's interesting? ;) 21:35:09 mainly 'cause you mentioned it in channel this morning ;-) 21:35:10 the end game is s3api compatible object versioning support 21:35:25 and i know that we want to be reviewing it soonish 21:35:58 sooner we can get other people thinking about it, too, the better 21:36:49 anything else we want to discuss? 21:38:16 all right, then we oughta let mattoliverau get breakfast and alecuyer get to bed :D 21:38:28 thanks :) 21:38:30 thank you all for coming, and thank you for working on swift! 21:38:35 #endmeeting