Friday, 2017-12-08

*** threestrands has joined #openstack-swift00:01
*** manous_ has quit IRC00:03
*** itlinux has joined #openstack-swift00:03
*** mat128 has quit IRC00:04
kota_clayg: AFAIK, that is from your suggestion that you were working there with tdasilva...00:13
kota_To redirect, that's nice to have non conditional response always for Symlink object, AFAIK00:15
*** openstackgerrit has joined #openstack-swift00:17
*** manous_ has joined #openstack-swift00:17
openstackgerritAlistair Coles proposed openstack/swift feature/deep: Add probe test to delete then revive a sharded container  https://review.openstack.org/52655200:17
kota_Oh you mind on proxy controllers obj. I should be back to there.00:21
kota_One possible point is because ec case... but anyway i need my laptop to check my thought.00:23
kota_Mobile is too hard to review the code.00:24
*** rcernin has quit IRC00:25
*** rcernin has joined #openstack-swift00:25
claygyeah so the 412 response *already* includes the X-Object-Sysmeta-Symlink-Target00:29
claygkota_: the behavior is correct, but the implementation has some unfortunate side effects - luckily it looks like it will be easy to clean up - may not be a blocker!00:30
timburkeyup. so mware can do whatever it wants with that -- try again at the new location, 3xx with a Location header, w/e00:30
kota_clayg: sounds nice. Thanks!00:31
timburkeoh... i was thinking that happened in https://review.openstack.org/#/c/357559/ -- must've been thinking of another change00:32
patchbotpatch 357559 - swift - Include object sysmeta in POST responses (MERGED)00:32
timburkei know that we *needed* that for symlinks... so we could 307... maybe we were already sending back the meta? always had been? *shrug*00:33
kota_clayg, timburk: if you leave anything more you find on Symlink, i will try to address them today (in my day time)00:33
*** manous_ has quit IRC00:35
timburkedo we... include user meta in 416/412/304 responses?00:35
claygtimburke: we kinda shouldn't...00:36
claygswift cli does not support upload -m00:38
claygtimburke: we DO include user-meta in 412 response00:39
timburkehttps://review.openstack.org/#/c/481202/00:39
patchbotpatch 481202 - python-swiftclient - Allow --meta on upload00:39
claygtimburke: has an answer for everything00:42
*** csmart has joined #openstack-swift00:50
timburkeoh man, this is great -- now that i've seen it, i can't *not* hit bug 173684000:50
openstackbug 1736840 in OpenStack Object Storage (swift) "Suffix-range requests for 0-byte objects produce invalid responses" [Undecided,In progress] https://launchpad.net/bugs/1736840 - Assigned to Samuel Merritt (torgomatic)00:50
timburkegot another, different traceback testing symlinks -- but it's not coming up out of the middleware, so we're better off addressing it separately00:51
torgomatictimburke: there's a partial fix up already; you could see if that helps00:51
torgomaticEC's still busted, but I'm working on it00:52
timburkeyeah, it's the ec side that caused the trouble. with a replicated policy for the symlink, it all workd like a champ!00:52
openstackgerritAlistair Coles proposed openstack/swift feature/deep: Add object count and bytes to ShardRange repr  https://review.openstack.org/52656100:54
*** tovin07_ has joined #openstack-swift00:55
claygacoles: you're killing it on the deep containers!00:59
*** gyee has quit IRC01:01
acolesclayg: TBH I am mosty salvaging snippets from otherwise abandoned patches01:01
claygso you're killing... patches... or at least putting them out of their misery?  still seems like a noble and worthwhile endevour01:02
claygkudos!01:02
timburkeall right, i think i'm getting it... it boils down to the is_success check in _recursive_get_head -- if you provide a `if-none-match: <empty-md5>` header, the object server responds 304 which makes its way all the way back to the client01:03
claygtimburke: so I'm landing somewhere around here -> https://gist.github.com/clayg/6969949a611add5c004a6d4efe3c6f1f01:03
clayg... and we can drop all the conditional handling in the proxy & obj server (or so say the unit & func tests)01:03
acolesfeels a bit like a reunion party in this channel today :)01:04
claygbut it doesn't really seem like a data model problem... and i'm not even sure it makes a difference during rolling upgrade01:04
claygmy gut is leaning towards follow on patch - but of course I'd be happy to see it squashed01:04
claygi still need to look into the probe test (which is for container sync, and not failure modes related to split brain while POSTing symlinks sadly)01:05
claygand I have the worry about the cross account container sync symlinking01:05
claygsigh - thank you tim - it is *absolutely* a rolling upgrade cruft challange if we don't fix it out the gate01:08
claygwait... is it01:08
claygold proxies would be expecting the 200's - so if your storage layer upgrades you're boned01:10
timburkei think i'm liking clayg's gist -- m_kazuhiro and kota_, are you ok with object layer reporting 3xx/4xx statuses, or would you have monitoring that will think something's gone wrong?01:27
clayghow could you possibly infer something bad from a metric that's totally dependent on client behavior?01:28
claygalso... should symlink=get requests not be conditional?  why can't I stick a if-modified-since on a symlink=get request?01:29
openstackgerritClay Gerrard proposed openstack/swift master: Do not leak symlinks to proxy & obj layer  https://review.openstack.org/52656401:31
timburkeif i see a spike in *anything* i'd want to investigate ;-) whole bunch of 499s? what's up with that? suddenly i'm getting 10x my normal # of requests? better get some more servers up and figure out if this is supposed to be the new normal01:31
timburkeanyway, we could later add some x-backend-* header that says "hey, if you find any of this list of metadata, turn off conditional responses" to clean up the status codes *without* the obj server needing to know about symlinks01:32
timburkesymlink=get is *always* conditional01:33
claygoh oh - yeah I'm not sure why I have such trouble with that boolean there01:34
claygx-backend-conditional-override-metadata: x-object-sysmeta-symlink-target[, ...]01:36
*** m_kazuhiro has joined #openstack-swift01:38
m_kazuhirogood morning01:43
openstackgerritSamuel Merritt proposed openstack/swift master: Fix suffix-byte-range responses for zero-byte EC objects.  https://review.openstack.org/52656501:48
*** two_tired has joined #openstack-swift02:00
m_kazuhiroclayg: timburke : You discussed symlink's conditional response on irc. What is your main concerning point? (I'm reading irc dicssion logs now.)02:04
kota_I'm here at my office02:04
claygm_kazuhiro: patch 52656402:05
patchbothttps://review.openstack.org/#/c/526564/ - swift - Do not leak symlinks to proxy & obj layer02:05
m_kazuhiroclayg: Thank you. I'm reading the patch now...02:06
kota_clayg: my quick look, it looks pretty nice.02:06
kota_let me run unit/func with the patch02:07
clayghopefully zuul will get back to us on that soon too02:07
kota_yeah02:08
timburkeonly functional delta should be in the response codes getting logged, but the *proxy* statuses still reflect what's going out to the client02:08
kota_btw, does swiftstack stops the third party jobs? I didn't see it appears to the ci results.02:08
kota_recently02:08
kota_at first, unittests runs as OK, Ran 5459 tests in 61.143s. *GREAT*02:12
claygcharz: notmyname: can we add a note to check on community CI results?02:12
charz@clayg We were doing VM migration last two weeks.02:14
claygunderstood - carry on good sir - thank you02:14
charzkota_: I'll try to bring the CI server ASAP.02:15
kota_charz: thanks! the information from your CI is really helpful to find the bugs, on the probe in particular02:16
claygm_kazuhiro: what was the reasoning behind sticking the client x-symlink-target and x-symlink-target-account headers into a single piece of sysmeta?02:19
m_kazuhiroclayg: At implementation time, I thought single piece is simple...02:21
m_kazuhiroclayg: But as you wrote on gerrit, it makes container-sync implementation complexed.02:22
*** mat128 has joined #openstack-swift02:23
claygyeah... but there's no getting around container-sync knowing about symlinks02:24
claygI'm just wondering how sad we'll be down the road if we don't support the concept of an explicitly "free" or "accountless" symlink in the data-model02:28
claygbytes on disk are the worst02:28
kota_clayg: so if we store x-symlink-target and x-symlink-target-account as separate sysmeta, it's easy to handle what container-sync to transfer?02:29
kota_just transfer x-symlink-target-account if it exists, e.g.02:29
claygthat was what made me notice... but i'm not sure I have a concern02:30
claygit's just - you can *tell* if a user said 'x-symlink-target: c/o' or they said 'x-symlink-target: c/o' & 'x-symlink-target-account: a' but a happened to be in the same account they stored the symlink in at the time...02:30
claygyou don't ever really have "accountless" symlinks - but we kind of support them anyway - or at least container-sync does... but implicitly02:31
kota_true02:31
kota_let me try to write the sample in today's work?02:32
claygsorry, i'm not sure it's important yet - just noticing and thinking02:32
claygI feel a *little* more sure the conditional request thing is a good idea02:32
kota_hehe02:32
claygi need to break for dinner tho i'm afraid02:32
kota_tbh, I and m_kazuhiro had conversation on that, and we're also not sure on that. Good point you noticed ;)02:33
claygi made much good progress!  I think timburke and zaitcev are looking too02:33
claygwe will merge it!!02:33
claygvery soon!02:33
claygyay!!!02:33
claygm_kazuhiro: YOU ARE HERO!02:33
claygthe end is in sight02:33
m_kazuhiroclayg: Thank you!02:34
kota_clayg: one thing02:34
kota_even we divide the header, could we have the symlink handle (translate from the sysmeta to the user meta for the remote cluster) in the conatiner-sync code?02:35
kota_i had an opinion, i'd love to make it in because if we do not do so, we should modify the pipeline of internal client that can cause operational failures.02:36
*** JimCheung has quit IRC02:37
*** JimCheung has joined #openstack-swift02:50
*** JimCheung has quit IRC02:54
*** links has joined #openstack-swift03:42
charzkota_: clayg the community CI are back, let me know if you still see problems on it.03:43
kota_charz: nice, thanks!03:43
*** m_kazuhiro has quit IRC04:37
*** mat128 has quit IRC04:43
*** rcernin has quit IRC05:11
*** threestrands has quit IRC05:24
*** threestrands has joined #openstack-swift05:39
*** threestrands has quit IRC05:39
*** threestrands has joined #openstack-swift05:39
*** armaan has joined #openstack-swift06:01
*** armaan has quit IRC06:02
*** rcernin has joined #openstack-swift06:11
*** cshastri has joined #openstack-swift06:13
*** two_tired has quit IRC06:23
*** threestrands has quit IRC06:28
*** cbartz has joined #openstack-swift06:29
*** m_kazuhiro has joined #openstack-swift06:52
-openstackstatus- NOTICE: Due to some unforseen Zuul issues the gate is under very high load and extremely unstable at the moment. This is likely to persist until PST morning07:04
*** ChanServ changes topic to "Due to some unforseen Zuul issues the gate is under very high load and extremely unstable at the moment. This is likely to persist until PST morning"07:04
*** amito has quit IRC07:06
*** seongsoocho has quit IRC07:07
*** serverascode has quit IRC07:07
*** seongsoocho has joined #openstack-swift07:07
*** onovy has quit IRC07:08
*** kencjohnston has quit IRC07:08
*** serverascode has joined #openstack-swift07:08
*** tdasilva has quit IRC07:08
*** cshastri has quit IRC07:11
*** onovy has joined #openstack-swift07:11
*** tdasilva has joined #openstack-swift07:13
*** kencjohnston has joined #openstack-swift07:13
*** csmart has quit IRC07:14
*** armaan has joined #openstack-swift07:17
*** armaan has quit IRC07:21
*** m_kazuhiro has quit IRC07:41
*** hseipp has joined #openstack-swift07:53
*** ukaynar has quit IRC07:57
*** ukaynar has joined #openstack-swift07:58
openstackgerritKota Tsuyuzaki proposed openstack/swift master: More cleanup for Clay's great work  https://review.openstack.org/52661407:58
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Testing Implementation to devide symlink headers  https://review.openstack.org/52661507:58
*** ukaynar has quit IRC08:03
*** tesseract has joined #openstack-swift08:20
*** armaan has joined #openstack-swift08:45
*** armaan has quit IRC08:53
*** armaan has joined #openstack-swift08:55
*** armaan has quit IRC08:57
*** itlinux has quit IRC09:01
*** hseipp has quit IRC09:05
*** amito has joined #openstack-swift09:06
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Testing Implementation to devide symlink headers  https://review.openstack.org/52661509:07
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Testing Implementation to devide symlink headers  https://review.openstack.org/52661509:13
kota_clayg, timburke: for the conditional response of symlink, i like the clay's patch. then, probably https://review.openstack.org/#/c/526614/ is better to squash in.09:15
patchbotpatch 526614 - swift - More cleanup for Clay's great work09:15
kota_and https://review.openstack.org/#/c/526615/ is my trying to separate the symlink target sysmeta to cont+object and account.09:16
patchbotpatch 526615 - swift - Testing Implementation to devide symlink headers09:16
kota_it looks to work well but i need to circle back again to check though the whole patch set.09:17
kota_whole of the patch.09:17
kota_it makes container-sync change smaller so that you may like it and i'm fine to change the symlink to do so.09:17
kota_i'll up tomorrow too to keep track the discussion so that it'll be happy you would check them while i asleep.09:18
kota_tomorrow, it's weekend though, i have to do small things (release) for storlets anway, bercause zuul is too slow today :,(09:19
kota_if you find anything, please let me (or m_kazuhiro) know.09:19
kota_clayg, timburke, zaitcev: ^^^^09:20
kota_thx09:20
*** csmart has joined #openstack-swift09:21
*** cshastri has joined #openstack-swift09:26
*** hseipp has joined #openstack-swift09:47
*** cbartz has quit IRC09:48
*** ukaynar has joined #openstack-swift09:58
*** ukaynar has quit IRC10:03
*** cbartz has joined #openstack-swift10:04
*** tovin07_ has quit IRC10:23
*** hoonetorg has quit IRC10:44
*** hoonetorg has joined #openstack-swift10:45
*** openstackgerrit has quit IRC11:17
*** links has quit IRC11:28
*** kei_yama has quit IRC11:37
*** links has joined #openstack-swift11:41
*** tesseract has quit IRC11:50
*** tesseract has joined #openstack-swift11:51
*** silor has joined #openstack-swift11:57
*** ukaynar has joined #openstack-swift11:59
*** ukaynar has quit IRC12:04
*** cshastri has quit IRC12:34
*** mat128 has joined #openstack-swift13:16
*** mat128 has quit IRC13:24
*** openstackgerrit has joined #openstack-swift13:30
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Testing Implementation to devide symlink headers  https://review.openstack.org/52661513:30
*** mat128 has joined #openstack-swift13:30
*** mat128 has quit IRC13:40
*** tesseract has quit IRC13:47
*** tesseract has joined #openstack-swift13:50
*** ukaynar has joined #openstack-swift14:00
*** ChanServ changes topic to "https://bugs.not.mn/project/Swift | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Review Dashboard: http://not.mn/reviews.html"14:01
-openstackstatus- NOTICE: The issues have been fixed, Zuul is operating fine again but has a large backlog. You can recheck jobs that failed.14:01
*** ukaynar has quit IRC14:04
*** mat128 has joined #openstack-swift14:05
*** mat128 has quit IRC14:05
*** mat128 has joined #openstack-swift14:35
*** links has quit IRC14:37
*** vinsh has quit IRC14:46
*** rcernin has quit IRC14:55
zaitcevWhat's "devide" in this case? "Decide"? "Divide"?15:10
*** hseipp has quit IRC15:40
openstackgerritMerged openstack/swift feature/deep: Use the same hash and created_at value in shard db  https://review.openstack.org/51648815:43
*** hseipp has joined #openstack-swift15:45
*** cbartz has quit IRC15:58
*** ukaynar has joined #openstack-swift15:59
*** itlinux has joined #openstack-swift16:14
*** mat128 has quit IRC16:27
*** zaitcev has quit IRC16:33
*** itlinux has quit IRC16:42
*** itlinux has joined #openstack-swift16:43
*** zaitcev has joined #openstack-swift16:46
*** ChanServ sets mode: +v zaitcev16:46
timburkeon p 526615 -- what do we want the behavior to be for cross-account copies? given a valid symlink in one account, i guess copying it (with a ?symlink=get) should produce different results depending upon which headers were sent when the symlink was PUT?16:50
patchbothttps://review.openstack.org/#/c/526615/ - swift - Testing Implementation to devide symlink headers16:50
timburkeit feels a little weird to me though... like it's an extra layer of complication for clients -- they have to do a ?symlink=get GET *first* before they can know what effect the COPY will have, at which point they may as well do the PUT themselves16:50
*** d0ugal has quit IRC16:52
zaitcevYea, I'm looking at it too.16:52
zaitcevIn fact, when I PUT, my supplied account header goes straight into sysmeta (in both implementations). Then, as I do GET, it gets into the traversal request now. Doesn't it allow me to peek into ANY account?16:53
*** d0ugal has joined #openstack-swift16:54
*** d0ugal has quit IRC16:54
*** d0ugal has joined #openstack-swift16:54
timburkekind of. but the request is (or at least, should be, *needs to be*) re-authed as we follow redirects. you can create a symlink that points somewhere you can't access (or doesn't exist), but trying to follow it should 403 (or 404)16:55
timburkeat least, that's what my mental model says :-)16:56
zaitcevI need to look closer.16:56
openstackgerritAlistair Coles proposed openstack/swift feature/deep: Add object count and bytes to ShardRange repr  https://review.openstack.org/52656116:56
*** itlinux has quit IRC17:00
*** itlinux has joined #openstack-swift17:05
*** gyee has joined #openstack-swift17:14
*** JimCheung has joined #openstack-swift17:19
*** hseipp has quit IRC17:27
*** tesseract has quit IRC17:29
*** ukaynar has quit IRC17:32
claygheyoh!17:33
*** ukaynar has joined #openstack-swift17:41
*** openstackgerrit has quit IRC18:03
*** openstackgerrit has joined #openstack-swift18:21
openstackgerritAlistair Coles proposed openstack/swift feature/deep: Add probe test to delete then revive a sharded container  https://review.openstack.org/52655218:21
claygtimburke: it seems like patch 476992 should have a closes-bug associated with it?18:59
patchbothttps://review.openstack.org/#/c/476992/ - swift - Make If-None-Match:* work properly with 0-byte PUTs18:59
openstackgerritClay Gerrard proposed openstack/swift master: Do not leak symlinks to proxy & obj layer  https://review.openstack.org/52656419:10
timburkeclayg: yeah, probably should write up a bug for it...19:11
openstackgerritClay Gerrard proposed openstack/swift master: Testing Implementation to divide symlink headers  https://review.openstack.org/52661519:12
*** __david has joined #openstack-swift19:26
claygzaitcev: no worries then - i probably had extra context from discussing the patch in channel with kota...19:38
claygzaitcev: it's definitely divide - it takes the x-symlink-target and x-symlink-target-account and divide's them into *separate* metadata instead of cramming them into the single absolute path.19:39
zaitcevOh. I see... I thought the division was about subdividing sysmeta and meta, but I wasn't sure.19:42
openstackgerritMerged openstack/python-swiftclient master: Allow --meta on upload  https://review.openstack.org/48120219:51
*** ChanServ has quit IRC20:17
*** ChanServ has joined #openstack-swift20:24
*** barjavel.freenode.net sets mode: +o ChanServ20:24
claygI'm pondering versioned writes + symlinks20:25
claygtimburke: I think you've advocated most strongly for the symlink_path in the container listings... but it's causing me some worry on patch 232162 and patch 526615'20:26
patchbothttps://review.openstack.org/#/c/232162/ - swift - Symlink implementation.20:26
patchbothttps://review.openstack.org/#/c/526615/ - swift - Testing Implementation to divide symlink headers20:26
timburkeclayg: my main reason for it is just that, if we don't do it now, *we'll never have it, ever* -- see the resistance (or at any rate, apathy) to https://review.openstack.org/#/c/337960/20:32
patchbotpatch 337960 - swift - Send correct SLO ETag for container updates20:32
*** itlinux has quit IRC20:55
*** itlinux has joined #openstack-swift20:59
*** JimCheung has quit IRC21:01
*** JimCheung has joined #openstack-swift21:01
*** ukaynar has quit IRC21:31
*** ukaynar has joined #openstack-swift21:32
*** ukaynar has quit IRC21:35
*** ukaynar has joined #openstack-swift21:35
*** silor has quit IRC21:36
claygtimburke: sure... but in this case it's causing some specific complications... (leakage if symlinks is removed & ambiguity of absolute and relative symlinks)21:43
*** rcernin has joined #openstack-swift21:44
claygin case anyone else is totally psyched about this - python-swiftclient `swift post <cont> <symlink>` automatically follows the 307 and you end up applying the metadata to BOTH which is basically the ideal behavior.  It's amazing when HTTP clients and services do the right thing.21:58
claygit's almost as if this whole "RESTful interface" isn't total bullshit ๐Ÿ˜…21:59
*** itlinux has quit IRC22:01
*** rcernin has quit IRC22:03
timburkewhooo! our requests dependency actually worked in our favor!22:03
*** rcernin has joined #openstack-swift22:03
*** ukaynar has quit IRC22:07
*** ukaynar has joined #openstack-swift22:08
openstackgerritSamuel Merritt proposed openstack/swift master: Fix small error in a doc string  https://review.openstack.org/52679122:09
claygtimburke: in VW, in history mode - what does one *do* with the zero byte marker?22:09
clayghttps://gist.github.com/clayg/0103ef60243c0a174002c4e1b8c9e56622:11
timburkeit serves as a record of the fact that there was a client request to delete that key22:12
claygand what is `swift_versions_deleted=1` all about?  it's not mentioned on https://docs.openstack.org/swift/latest/overview_object_versioning.html22:13
timburkeit's something only middleware can set, because of the `;swift`-ness of it. so we *know* it came from versioned_writes22:15
timburkeon the multiple-delete behavior, i was largely cribbing behavior from http://docs.aws.amazon.com/AmazonS3/latest/dev/RemDelMarker.html22:15
*** saint_ has quit IRC22:16
*** itlinux has joined #openstack-swift22:16
*** ianychoi_ has joined #openstack-swift22:17
*** itlinux has quit IRC22:18
*** ianychoi has quit IRC22:19
kota_zaitcev: divide, i meant. Sorry22:33
* kota_ is looking at log22:34
claygdamn you flakey tests; lp bug #172435622:35
openstackLaunchpad bug 1724356 in OpenStack Object Storage (swift) "test_part_swapping_problem is flakey" [Medium,Confirmed] https://launchpad.net/bugs/172435622:35
claygbummer - I can't figure out how to create a symlink with swift cli - for zero-byte object I normally do `echo | swift upload test - --object-name zero`22:44
claygbut that results in chunked upload that just happens to not send any bytes - and the symlink middleware explcitly wants `content_length == 0` for PUT22:45
claygI tried to add the header - but it seems to throw off swiftclients guts... `sendall() argument 1 must be convertible to a buffer, not ReadableToIterable` too bad22:45
timburkewait, what's that last part? i wanna see a traceback!23:09
timburkefor the first, have you tried just touching a file? or uploading /dev/null?23:09
claygtimburke: https://gist.github.com/clayg/2a713f1b7a5b3cc44973411579db5b2123:16
zaitcevkota_: Clay has set me straight. Truth to be told, I tried to discuss "ใƒใƒˆ้Šƒใฎใ‚นใ‚นใƒก" a few days ago... the symbols look too similar! So yeah23:44
kota_That's nice. Thanks clayg23:47
kota_zaitcev: make sense, i googled ใƒใƒˆๅ……ใฎใ‚นใ‚นใƒก23:49
*** ukaynar has quit IRC23:49
kota_symbol is similar but the mean is completely different23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!