Tuesday, 2015-03-24

openstackgerritSamuel Merritt proposed openstack/swift: EC: support multiple ranges for GET requests  https://review.openstack.org/16657600:02
*** jrichli has joined #openstack-swift00:09
*** annegentle has quit IRC00:14
claygson of a bitch00:16
*** ho has joined #openstack-swift00:19
mattoliverauclayg: you ok?00:26
torgomaticman vs. code; code is winning00:31
notmynamemattoliverau: I'm sitting next to him. he's cursing like a sailor at the code. I think he's pretty happy, actually00:31
mattoliveraunotmyname: aah, the 'son of a bitch' was in a moment of solving/understanding the problem.. I know that feeling :)00:33
homorning!00:37
egonI sit across from a guy from south america, so I get the equivalent in spanish.00:38
egonit's pretty funny, because he didn't think anyone would understand anything he said.00:38
*** kota_ has joined #openstack-swift00:38
openstackgerritJanie Richling proposed openstack/swift: Allow middleware to override metadata header checking  https://review.openstack.org/16380600:41
openstackgerritJanie Richling proposed openstack/swift: FakeFooters middleware  https://review.openstack.org/16551700:46
*** zhill has quit IRC00:46
hoI would like to have a procedure for building a saio with ec. does anyone have this info?00:49
mattoliverauho: morning00:49
homattoliverau: morning!00:49
mattoliverauho: if you checkout the EC branch of the code, the SAIO documentation contained is up to date regarding setting it up for EC00:49
jrichlimorning00:50
mattoliveraujrichli: evening :)00:50
jrichlimattoliverau: well, yes... thanks!00:50
homattoliverau: thanks!00:50
mattoliverauho: sample files etc.. just make sure your on the EC branch when followining the instructions and you should be fine, ask if you need any help00:51
hojrichli: hello :-)00:51
jrichliho: hello!00:52
homattoliverau: thanks for your help.00:53
clayga'ight tox - do your thing00:58
*** annegentle has joined #openstack-swift01:14
*** Tyger has quit IRC01:16
*** Tyger has joined #openstack-swift01:17
*** annegentle has quit IRC01:20
openstackgerritTim Burke proposed openstack/python-swiftclient: Compare each chunk of large objects when uploading  https://review.openstack.org/16104301:36
pelusemattoliverau, small tweak on https://review.openstack.org/#/c/166307/01:39
claygI either really don't understand patch 159205 - or I should have stuck a -1 on it - because I *really* don't understand acoles_away's reasoning for having in-process functests test a pre-configured on disk ring?01:39
patchbotclayg: https://review.openstack.org/#/c/159205/01:39
openstackgerritKota Tsuyuzaki proposed openstack/swift: Small ec diskfile refactor  https://review.openstack.org/16675401:42
*** annegentle has joined #openstack-swift02:16
*** annegentle has quit IRC02:21
*** haomaiwang has joined #openstack-swift02:28
*** panbalag has quit IRC02:46
*** zul has quit IRC02:47
*** yuan has quit IRC02:48
*** zul has joined #openstack-swift02:59
*** ppai has joined #openstack-swift03:01
*** gyee has quit IRC03:06
*** bkopilov has quit IRC03:29
openstackgerritpaul luse proposed openstack/swift: Erasure Code Reconstructor  https://review.openstack.org/13187203:31
openstackgerritJanie Richling proposed openstack/swift: WIP - Provides a simple skeleton of middleware for encryption feature. depends on review 165517 for FakeFooters as a work-around until EC is merged to master, and support for footers can be used from there.  https://review.openstack.org/15790703:41
notmynametorgomatic: ok, looking at your patches now (no tequila, but I do have ice cream)03:42
jrichlioops - i guess i need a space after the title03:42
jrichlis/space/line/03:42
notmynamejrichli: ya, a blank line03:42
openstackgerritJanie Richling proposed openstack/swift: WIP - Provides a simple skeleton of middleware for encryption feature.  https://review.openstack.org/15790703:43
notmynamejrichli: much better :-)03:43
jrichlilol03:44
notmynamejrichli: oh, I ended up getting the CODE keyboard from wasdkeyboards03:44
notmynamejrichli: cherry mx clear switches03:44
jrichliyou happy with it?03:44
notmynameit's different, definitely. I'm still, at times, teaching my fingers how to use it03:45
notmynameI got a wrist rest, and I think that's essential. so much better after that.03:45
notmynameoverall, yes, I'm pretty happy with it03:45
jrichligood, good.  BTW, did you all decide on a T-shirt design?03:46
notmynameyup. 2 actually. well, same design but different colors with different color shirts03:47
*** yuan has joined #openstack-swift03:48
notmynamepeluse: nice! I was just sitting down to look at that patch (conditional responses03:48
jrichlicool.  I will be excited to see them!  I saw you have lots of talks lined up.  Will you need more than 2 swift shirts?03:48
notmynamelol. I have enough swift and openstack shirts to last for about 3 full weeks of conferences03:48
jrichligreat, no worries then.03:49
notmynamehmm... I don't like how `git review -d 1234` and `git branch -d 1234` pretty much mean the exact opposite things03:53
notmynamemattoliverau: what are you looking at for the rest of the day?03:57
mattoliveraunotmyname: I'm currently playing with ranged gets, then might get back on PUT refactor and then work my way down one of the patch set chains.03:58
notmynamemattoliverau: awesome! I'm looking at the single range gets right now too03:58
notmynamethanks!03:59
mattoliverauyay twins :P03:59
notmynameI'm eating ice cream. are you? (not climbing any volcanos, though)03:59
mattoliveraunotmyname: lol, damn volcanos.. actaully I blame the icecream :P04:00
mattoliveraunotmyname: no ice cream for me.. but I *could* go get some for afternoon tea ;P04:01
notmynamemattoliverau: isn't the better question "why wouldnt' you go get ice cream?"04:01
mattoliveraunotmyname: true, that is the better question :P04:02
jrichliok, now that I am hungry for sweets, I am going to bed.  night!04:02
mattoliveraujrichli: lol, night04:02
notmyname:-)04:03
*** jrichli has quit IRC04:03
notmynamemattoliverau: I've got "FAILED (SKIP=12, errors=5, failures=4)" for functests with the single-range patch on top of the ifmatch patch. looking metter!04:04
notmyname*better04:04
mattoliveraunotmyname: so it looks like the Melbourne Linux User Group wants me to talk about swift.. So was thinking doing the normal intro to swift talk, any chance I can steal your slide deck? If not I'll steal what I remember and use presentty ;)04:04
mattoliveraunotmyname: nice :) I'm manually testing it first, trying to trick it and throwing data at it.04:05
notmynamemattoliverau: I'd be happy to give you pointers or help, but you should make your own talk instead of use mine :-)04:05
mattoliveraukk, just thought I'd ask :)04:05
notmynamehmm...a lot of ifmatch functests errors04:06
mattoliveraunotmyname: torgomatic has another patch to get EC working with If-[None]-Match04:07
mattoliverauI wonder if it'll work if this was rebased off that?04:07
notmynameya, didn't you +2 it and peluse just marked it +2/+1?04:07
mattoliveraunotmyname: yup, it worked well :)04:07
notmynameah, interesting. it would be because my git-fu seems to be lacking04:08
mattoliveraunotmyname: k, so I'll play with presentty after this EC rush is over (RE: Presentation) :)04:08
notmynameI thought I was still working off of that patch. turns out I'm not04:08
*** km has quit IRC04:09
notmynamegot it. rerunning tests04:10
peluseparty04:13
*** annegentle has joined #openstack-swift04:18
notmynamecool. "errors=2, failures=2" with ifmatch + single range patch on functests04:19
openstackgerritMerged openstack/swift: Refactoring the PUT method  https://review.openstack.org/15682504:19
mattoliverau^^ looks like I don't need to look at that next :P04:19
*** annegentle has quit IRC04:23
kota_notmyname, mattoliverau: hi, for now I could be a volunteer to review EC patches so what is highest priority?04:23
notmynamekota_: great! and thanks04:24
notmynamekota_: let me pull up the list04:25
kota_notmyname: Thanks, priority reviews on wiki hides all of them for EC, right now :P04:26
notmynameheh, I think I just added "go look on the gerrit dashboard" since it's changing so rapidly :-)04:26
mattoliveraukota_: you can start with starred by PTL list on the dashboard (should match priority reviews).04:26
notmynamekota_: mattoliverau and I are looking at the range patches, so I think we've got those04:27
notmynamekota_: the patch chain starting with https://review.openstack.org/#/c/165188/ is pretty important04:27
notmynamekota_: that would be a great place to start04:27
openstackgerritMerged openstack/swift: EC: Support If-Match and If-None-Match on GET+HEAD  https://review.openstack.org/16657904:28
notmynamekota_: or https://review.openstack.org/#/c/143791/ is a stand-alone patch (no dependencies) that is good to look at04:28
kota_ya, I missed the line priority reviews are starred. Ok, I will start to walk through from 16518804:28
kota_notmyname, mattoliverau: thanks!04:29
notmynamemattoliverau: hrm. still getting one if-match error (but no range errors)04:32
mattoliveraunotmyname: well that's better then before :P04:33
*** silor has joined #openstack-swift04:41
notmynamegit st04:48
openstackgerritJohn Dickinson proposed openstack/swift: better functional test for ranged GETs  https://review.openstack.org/16709204:49
notmynamemattoliverau: I had that patch in patch set 5 for the single range GETs but looks like torgomatic dropped it. there it is independently so the patch chain doesn't get messed up04:50
mattoliveraunotmyname: ta04:50
* mattoliverau is waiting for functional tests to complete04:51
claygmattoliverau: I'm going to screwing with the diskfile->ec-probe test patch chain shortly04:57
*** bkopilov has joined #openstack-swift04:57
claygmattoliverau: I'm all done moving code, got a beer cracked open and working my way change-by-change via tox04:58
mattoliverauclayg: living the dream :)04:59
*** ppai_ has joined #openstack-swift05:00
*** ppai has quit IRC05:01
notmynameok, I've looked enough at the single ranges for now (although I'm slightly suspicious about no new unit tests in the proxy object controller)05:03
notmynameI'm looking at the weird ifmatch bug still on HEAD of ec right now05:03
clayghrmmm... so https://xkcd.com/1205/ says you can only spend about an hour reviewing https://review.openstack.org/#/c/166104/ - but I still think it would help when you're going to be running unittests dozens of times like I am now05:03
claygnotmyname: ranged requests are stupid - so is copy - and so is if-match - this is crunch time - we need to get the reconstructor and the proxy refactor done05:04
notmynameclayg: what do you need?05:04
claygwe can noodle around with proxy features while people are reviewing important stuff :P05:04
notmynameheh. I thought you were all over that! :-)05:04
claygnotmyname: yeah we're trying - i should plan on syncing with paul and acoles hi-bandwidth in the am05:05
notmynamekk05:05
claygnotmyname: did the master put method extraction refactor merge?05:05
claygnotmyname: or is it not starred - or is my dashboard old?05:05
notmynamelooks like it did05:06
claygoh shit!05:06
mattoliverauclayg: yup about 10 mins ago05:06
claygdid it include the fix for https://bugs.launchpad.net/swift/+bug/66795605:07
openstackLaunchpad bug 667956 in OpenStack Object Storage (swift) "chunked transfer client disconnect causes unhandled value error in object server" [Low,Confirmed]05:07
claygbha, or a test for swob.Request.is_chunked - you guys are too easy05:07
claygi stop paying attention for two minutes and ya'll merge anything - I seriously don't get the in-process tests running against an on-disk ring change - just because 9/10 of acoles ideas are gold - doesn't mean he can't throw out a turd occasionally as much as the rest of us05:08
claygoh it did include the finally for conn in conn close - nice - no tests for it though05:09
claygwell i guess a few mocks had to get updated to have a close method - so I guess that probably means someone was calling it accidently ;)05:10
claygneway - i'm happy it got merged - i was worried about that change05:10
claygnotmyname: what's going on with the master merge to feature/ec -> https://review.openstack.org/#/c/166558/05:10
notmynameclayg: bug in the infra CI system. should be fixed tomorrow05:11
notmynameclayg: tl;dr is that global-requirements doesn't have a branch called feature/ec and so it failed and was treated as a test failure. the fix is to gracefully fall back to using master, I think. anyway, they're on it and I'm told it's good for tomorrow05:13
openstackgerritClay Gerrard proposed openstack/swift: Cleanup ECDiskFile classes  https://review.openstack.org/16709705:14
claygwoooh!  1 down 5 more to go!05:15
notmynameclayg: this is the start of a new patch chain that should be starred, right?05:16
*** annegentle has joined #openstack-swift05:19
claygman, sometimes I be hacking and then like I get all annoyed when my vim commands don't work in browsers and shit05:19
claygzz - make this line the center of the screen damnit!05:19
claygglange: hey, i are you activily looking into https://bugs.launchpad.net/swift/+bug/1435506 or just raising the issue?05:22
openstackLaunchpad bug 1435506 in OpenStack Object Storage (swift) "Account DB policy_stat Table Needs Some More Creating" [Undecided,New]05:22
claygglange: because I've been trying to take a stab at https://bugs.launchpad.net/swift/+bug/1424108 but somehow we don't seem to be experiencing the issue05:23
openstackLaunchpad bug 1424108 in OpenStack Object Storage (swift) "KeyError: 'storage_policy_index' in _really_merge_items swift/account/backend.py" [Undecided,New]05:23
*** annegentle has quit IRC05:23
*** reed has quit IRC05:26
*** Novtopro has joined #openstack-swift05:27
*** km has joined #openstack-swift05:28
claygoh man top post on HN is great -> https://bugzilla.redhat.com/show_bug.cgi?id=120285805:29
openstackbugzilla.redhat.com bug 1202858 in squid "restarting squid results in deleting all files in hard-drive (rm -rf /*)" [Urgent,On_qa] - Assigned to psimerda05:29
*** zaitcev has quit IRC05:34
openstackgerritClay Gerrard proposed openstack/swift: Add commit method to interface of DiskFile  https://review.openstack.org/16520805:36
claygkota_: don't get too far on that patch chain - i'm rebasing - you could tell me what you thinkg about https://review.openstack.org/16709705:40
kota_clayg: what?05:45
notmynameclayg is changing everything!05:46
kota_ah, great05:46
claygkota_: my bad - i thought you had just put a review comment on https://review.openstack.org/#/c/165188/ - so i was just giving you the heads up that I'm moving some stuff around05:46
kota_clayg: Ah, ok. I'll look at the cleanup,05:48
*** Novtopro has quit IRC05:49
kota_clayg: I hope it would be addressed my review comment. And if everything is ok, I'll revert my patch. So please let me take a time to look at.05:49
openstackgerritClay Gerrard proposed openstack/swift: Multiple Fragment Archive support for suffix hashes  https://review.openstack.org/15963705:51
claygacoles_away: ^ I think I got all of your spelling and other fixups in there05:52
claygkota_: which patch - don't revert - we need all the good code we can get!05:53
kota_clayg: ah, I might be in a fog, current chain of EC staff starts from 165188?05:53
claygkota_: take your time - let me know if I can help :)05:53
claygkota_: only just as of about 45 mins ago ;)05:54
claygkota_: that's what I was trying to give you the heads up on :)05:54
kota_clayg: I could make my misunderstanding you said about https://review.openstack.org/#/c/166754/ but it seems not correct.05:54
kota_166754 based on the chain and so...05:55
kota_clayg: ok, I'll start again from https://review.openstack.org/#/c/165188/05:57
kota_clayg: Thanks to make my heads up on :)05:58
*** nshaikh has joined #openstack-swift05:58
*** km has quit IRC06:02
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Imported Translations from Transifex  https://review.openstack.org/16710606:06
notmynameI'm turning in for the night06:15
*** pcaruana has quit IRC06:41
*** chlong has quit IRC06:42
*** Bsony has joined #openstack-swift06:57
*** chlong has joined #openstack-swift07:03
openstackgerritClay Gerrard proposed openstack/swift: Add Fragment Index filter support to ssync  https://review.openstack.org/16518807:10
*** annegentle has joined #openstack-swift07:20
*** annegentle has quit IRC07:25
*** tsg has quit IRC07:28
*** ppai_ has quit IRC07:37
*** navid__ has joined #openstack-swift07:38
*** mmcardle has joined #openstack-swift07:49
*** ppai_ has joined #openstack-swift07:49
*** nshaikh has quit IRC07:59
*** navid__ has quit IRC07:59
*** nshaikh has joined #openstack-swift08:00
*** kota_ has quit IRC08:04
*** jordanP has joined #openstack-swift08:07
*** chlong has quit IRC08:12
*** kota_ has joined #openstack-swift08:14
*** annegentle has joined #openstack-swift08:21
*** annegentle has quit IRC08:26
*** tellesnobrega has quit IRC08:28
*** joeljwright has joined #openstack-swift08:30
*** tellesnobrega has joined #openstack-swift08:40
*** Bsony has quit IRC08:41
*** geaaru has joined #openstack-swift08:49
*** ppai_ has quit IRC08:53
*** jistr has joined #openstack-swift09:04
*** ppai_ has joined #openstack-swift09:06
*** tab___ has quit IRC09:07
*** acoles_away is now known as acoles09:14
claygmorning acoles!09:16
acolesclayg: good morning, hows things?09:16
*** h_m has quit IRC09:16
*** egon has quit IRC09:19
openstackgerritClay Gerrard proposed openstack/swift: Erasure Code Reconstructor  https://review.openstack.org/13187209:24
*** egon has joined #openstack-swift09:24
claygacoles: just peachy!09:24
portantemorning clayg09:26
portantemorning acoles09:26
portanteyou guys still up?09:26
claygwell for acoles I think it's morning09:26
claygi guess *technically* it's morning for me to :\09:27
portante:(09:27
claygmeh09:27
cschwedelooks like some swift devs are affected by insomnia… Good Morning folks!09:30
acolesportante: hi, what are you doing around at this time? ;)09:30
*** nshaikh has quit IRC09:31
kota_cschwed: lol, anyways, good morning guys :)09:32
kota_woops, s/cschwed/cschwede/09:33
hogood morning!09:36
*** Bsony has joined #openstack-swift09:38
kota_ho: good evening :)09:38
openstackgerritClay Gerrard proposed openstack/swift: wip: ec reconstructor probe test  https://review.openstack.org/16429109:41
openstackgerritClay Gerrard proposed openstack/swift: Erasure Code Reconstructor  https://review.openstack.org/13187209:41
claygok, and that should be working rebuild and revert09:41
claygpeluse: ^09:41
*** km has joined #openstack-swift09:42
hokota_: :-)09:46
*** nshaikh has joined #openstack-swift09:59
openstackgerritMartin Kletzander proposed openstack/swift: Fix common misspellings  https://review.openstack.org/16684310:08
*** nshaikh has joined #openstack-swift10:10
openstackgerritMerged openstack/swift: EC: support single ranges for GET requests  https://review.openstack.org/16362010:11
openstackgerritMerged openstack/swift: Make get_dev_path() treat mount_check literally...  https://review.openstack.org/16630710:11
openstackgerritMerged openstack/swift: better functional test for ranged GETs  https://review.openstack.org/16709210:11
hokota_: btw thanks for the review today :)10:12
kota_ho: sure :)10:13
portanteacoles: can't sleep thinking of how hard you folks are working ... ;)10:17
acolesportante:  are we making too much noise? :P10:19
*** haomaiwang has quit IRC10:21
*** ppai_ has quit IRC10:30
*** Bsony_ has joined #openstack-swift10:34
*** Bsony has quit IRC10:35
*** acoles is now known as acoles_away10:41
*** ppai_ has joined #openstack-swift10:43
openstackgerritAlistair Coles proposed openstack/python-swiftclient: Allow reading from object body on download  https://review.openstack.org/15529110:45
*** Bsony has joined #openstack-swift10:46
*** silor has quit IRC10:47
*** Bsony has quit IRC10:47
*** Bsony_ has quit IRC10:48
*** acoles_away is now known as acoles10:52
*** Bsony has joined #openstack-swift10:52
*** ho has quit IRC10:57
*** Bsony_ has joined #openstack-swift10:57
openstackgerritMerged openstack/swift: Add swift-recon feature to track swift-drive-audit error count  https://review.openstack.org/16388910:59
*** Bsony has quit IRC10:59
*** Bsony_ has quit IRC11:02
*** Bsony has joined #openstack-swift11:05
*** Bsony_ has joined #openstack-swift11:13
*** Bsony has quit IRC11:16
*** Bsony_ has quit IRC11:18
*** panbalag has joined #openstack-swift11:22
*** delattec has quit IRC11:28
*** cdelatte has quit IRC11:28
*** Bsony has joined #openstack-swift11:31
*** km has quit IRC11:32
*** Bsony has quit IRC11:36
*** zul has quit IRC11:40
*** zul has joined #openstack-swift11:41
*** jamielennox is now known as jamielennox|away11:45
pelusewow, I just missed clay by a few hrs I guess11:48
*** silor has joined #openstack-swift12:01
*** cdelatte has joined #openstack-swift12:03
tdasilvagood morning12:05
*** Bsony has joined #openstack-swift12:09
*** chlong has joined #openstack-swift12:11
*** chlong has quit IRC12:15
*** silor has quit IRC12:17
*** Bsony has quit IRC12:20
pelusemorning12:22
*** kota_ has quit IRC12:27
*** silor has joined #openstack-swift12:29
*** jistr is now known as jistr|class12:30
*** chlong has joined #openstack-swift12:32
*** chlong has quit IRC12:35
*** chlong has joined #openstack-swift12:36
*** octavian1001 has joined #openstack-swift12:38
octavian1001hello12:39
*** chlong has quit IRC12:39
octavian1001how can I change the access level of a given container using the api from http://developer.openstack.org/api-ref-objectstorage-v1.html12:40
octavian1001from dashboard (Horizon) I can change the access level from private to public12:41
octavian1001but I don't see any option to change it from the API12:41
tdasilvaoctavian1001: you would use the headers X-Container-Write and X-Container-Read12:44
tdasilvaoctavian1001: this blog post also has some good info: https://swiftstack.com/docs/cookbooks/swift_usage/container_acl.html12:44
*** bogdan_ has joined #openstack-swift12:45
octavian1001thanks12:45
*** octavian1001 has quit IRC12:55
*** chlong has joined #openstack-swift12:56
mandarineHi everyone13:09
mandarineI might have a question about replication13:10
mandarineDo the rsync logs ever stop to run crazy ?13:10
mandarineI've been waiting for about 25 minutes to synchronize just 55GB and it does not seem to stop yet13:11
ctennisYou can check the output of the replication process and see how far along it is, rsync is just a byproduct of what it's doing.13:12
ctennisgrep for "partitions" in the swift log13:12
mandarinethanks13:15
openstackgerritpaul luse proposed openstack/swift: Cleanup ECDiskFile classes  https://review.openstack.org/16709713:15
mandarineI'll wait for it to appear again, then13:15
*** ppai_ has quit IRC13:16
mandarine"1h remaining"13:17
mandarinethanks you very much13:17
*** bogdan_ has quit IRC13:21
*** Gue______ has joined #openstack-swift13:30
*** fthiagogv has joined #openstack-swift13:31
*** Gue______ has quit IRC13:32
openstackgerritpaul luse proposed openstack/swift: Add commit method to interface of DiskFile  https://review.openstack.org/16520813:34
acolespeluse: ^^ did much change? just reviewing that one13:35
*** Bsony has joined #openstack-swift13:35
peluseacoles, rebase is all.  its D changed in the test code13:38
peluseparent I mean, not dependency13:38
acolespeluse: cool. thx13:39
pelusejust rebasing the whole chain on up right now to fix the patch the patch conflict at the head13:39
acolespeluse: ok i have a bunch of comments on https://review.openstack.org/167097 that are going to be on last version but I'm sure clayg can live with that13:39
peluseacoles, cool.  I've scanned all of the new ones and wil review in depth after I get them all rebased13:41
*** nshaikh has quit IRC13:48
*** jistr|class is now known as jistr13:53
pelusethis sucks13:58
acolespeluse: ?13:58
pelusefixing rebase shit on https://review.openstack.org/#/c/159637/  - have to step out for an hour or so.  will finish when i get back.  who knew making it required that your target device/path actually exist would cause so many problems in this new code13:59
openstackgerritLorcan Browne proposed openstack/swift: Add lowest option to swift-recon disk usage output  https://review.openstack.org/16723614:12
*** jrichli has joined #openstack-swift14:18
openstackgerritThiago da Silva proposed openstack/swift: versioned writes middleware  https://review.openstack.org/13434714:37
*** reed has joined #openstack-swift14:42
*** EmilienM|PTO is now known as EmilienM14:51
*** annegentle has joined #openstack-swift14:51
openstackgerritLorcan Browne proposed openstack/swift: Add lowest option to swift-recon disk usage output  https://review.openstack.org/16723614:52
notmynamegood morning15:25
*** Gue______ has joined #openstack-swift15:28
*** bkopilov has quit IRC15:31
*** mahatic has joined #openstack-swift15:31
notmynamelooks like https://review.openstack.org/#/c/167097/2 is the starting point for today15:32
*** annegentle has quit IRC15:33
openstackgerritpaul luse proposed openstack/swift: Multiple Fragment Archive support for suffix hashes  https://review.openstack.org/15963715:36
*** tsg has joined #openstack-swift15:37
*** ygbo has joined #openstack-swift15:37
*** gyee has joined #openstack-swift15:39
openstackgerritpaul luse proposed openstack/swift: Multiple Fragment Archive support for suffix hashes  https://review.openstack.org/15963715:40
*** annegentle has joined #openstack-swift15:43
pelusenotmyname, yup... it and everyone up the chain needed a little work after a late night landing - almost done getting them all up to snuff gain then I'll circle back and throw in my 2 cents FWIW15:45
notmynamegreat15:45
*** ygbo has left #openstack-swift15:45
tdasilvapeluse, notmyname: what's the plan with merging master to feature/ec? I imagine the next one might be a bit messy with the PUT refactoring now in master15:48
*** bkopilov has joined #openstack-swift15:49
pelusetdasilva, we've got at least a half dozen or more to land before crossing that bridge...15:50
*** annegentle has quit IRC15:50
tdasilvapeluse: oh ok, cool15:50
notmynamewell, the master to ec is different, right?15:51
notmynameor, rather, we shoudl be able to make progress on https://review.openstack.org/#/c/166558/ today15:51
notmynameI hope15:51
notmynamepeluse: or what do you want to do there?15:52
*** rdaly2 has joined #openstack-swift15:52
pelusenotmyname, yeah, would be nice to land that one as is however if we have to create a new one it will pick up the PUT refactor in whcih case we're better off waiting until we have the current WIP chain landed I think15:53
notmynamepeluse: maybe that one there can be rechecked and then land on ec. it's from before the PUT refactor so should be easier15:54
openstackgerritLorcan Browne proposed openstack/swift: Add lowest option to swift-recon disk usage output  https://review.openstack.org/16723615:59
notmynameok, coffee cup empty. time to go to the office16:01
*** EmilienM is now known as EmilienM|PTO16:02
tdasilvamm...just noticed probe tests failing on my local master branch16:03
*** jistr has quit IRC16:05
*** _ahale has quit IRC16:13
*** ahale has joined #openstack-swift16:14
*** ahale has quit IRC16:15
*** chlong has quit IRC16:16
cschwedetdasilva: which test is failing? running probetests on master now, checking16:17
*** Gue______ has quit IRC16:22
cschwedetdasilva: all probetests passed on my VM, saw only 5 skipped tests.16:23
*** bkopilov has quit IRC16:25
*** Bsony has quit IRC16:30
claygacoles: oh no, the Mixin messes with your ide?!16:34
*** Gue______ has joined #openstack-swift16:35
*** annegentle has joined #openstack-swift16:37
claygpeluse: so you got the whole chain rebased all the way up through the probe tests?16:38
*** GustavoJoseRuiz1 has joined #openstack-swift16:38
*** annegent_ has joined #openstack-swift16:38
peluseclayg, not yet16:39
claygwhat happend in the multi-fi suffix hashes?16:39
claygcan i help16:39
peluseI'm on https://review.openstack.org/#/c/159637/2416:39
peluseyeah, that one :)16:40
peluseI'm down to jsut a few failing ones.  what happened was that simple change to get_dev_path() is messing with all of the new tests16:40
peluseit landed right after you pushed the last set of changes16:40
pelusegive me a few minutes....16:41
*** annegent_ has quit IRC16:42
*** annegentle has quit IRC16:43
claygacoles: I don't think you submitted your draft comments to patch set 1 - you found a test that I dropped?16:43
peluseactualy that one is OK now.  all fixed up.  the one I'm fighting is https://review.openstack.org/#/c/165188/16:43
claygpeluse: ok, let me know if you get stuck16:45
acolesclayg: oops, just hit submit16:45
claygnotmyname: yeah if the master->feature/ec is working again I'd like to get the proxy put method extraction merge fix out of the way so we can start looking at minimizing churn in the proxy next16:46
openstackgerritpaul luse proposed openstack/swift: Add Fragment Index filter support to ssync  https://review.openstack.org/16518816:47
GustavoJoseRuiz1holaaaaaaaaaaaaaaaa16:47
peluseclayg, ^^ there are 2 still 2 broken tests in diskfile but I can't fix them without going back and reviewing earlier patches...16:47
peluseso there it is for now16:48
claygGustavoJoseRuiz1: Que onda?16:48
claygpeluse: ok - maybe I should take a stab at them then :P16:49
GustavoJoseRuiz1de donde eres?16:49
peluseclayg, yes, please!  could easily be a rebase issue between that one and the one before it... lots of code moving around16:49
claygGustavoJoseRuiz1: I'm in SF, what's up is about the only spanish i know :P16:50
*** Gue______ has quit IRC16:54
*** Gue______ has joined #openstack-swift16:55
tdasilvacschwede: thanks for checking...it's probably my VM,16:55
*** GustavoJoseRuiz1 has left #openstack-swift16:55
* notmyname is back online16:58
claygcschwede: thanks for the fix on https://github.com/swiftstack/vagrant-swift-all-in-one/pull/15 - but I don't understand why it isn't already picked up with the rest of the depends?16:58
*** tsg has quit IRC16:59
claygcschwede: I noticed I had "fixed" all of the other source installs to use pip install -e . instead of python setup.py develop at some point - so I'm trying that on a fresh up now16:59
cschwedeclayg: you’re welcome! i don’t know, it failed using the vagrant-all…, but worked manually. actually the important part was to install it before swift itself17:00
acolesclayg: peluse: i'm trying out jrichli's policy-specific func test patch, i want to verify that it is NOT using the default - can you think of a way to disable a policy  so tests would break if that policy was accessed?17:00
claygcschwede: i don't know sounds fishy - it's in requirements.txt - something must be wonky17:01
claygacoles: hack the object-server X-Backend-Storage-pOlicy-Index to throw up?17:02
cschwedeclayg: yes, that’s true. i wasn’t able to reproduce the problem when i installed it manually, it happened only when i provisioned the VM from scratch17:02
claygacoles: I think it's in get_name_and_placement17:02
acolesclayg: yeah, thats the best idea i had, i can't think of any way to do it via config17:02
acolesclayg: thx17:03
cschwedeclayg: i will check if it works when i do a „pip install -r requirements.txt“ before the swift installation17:04
cschwedeclayg: if i remembered correctly, it was an error in the pyeclib installation itself17:04
cschwedeclayg: thus the dependency was recognized, but failed to install (some gcc errors)17:04
claygoh crazy!?17:05
cschwedebut as i said - i couldn’t reproduce this manually17:05
claygpeluse: which tests were failing on the ssync patch?17:06
peluseone sec17:08
claygpeluse: it doesn't seem to be in test.unit.obj?17:08
peluseTestECDiskFile.test_yield_hashes()17:09
peluseTestECDiskFile.test_yield_hashes_suffixes()17:09
claygoh, heh WOMM17:09
claygacoles: so what are you going to do about your IDE and the Mixin's?  I thought i was being so cleaver :'(17:10
acolesclayg: well i could just live with the hate :/17:10
claygpeluse: oh, nm - totally fails for me - I've got it!17:10
pelusesweet!17:11
peluseacoles, which IDE do you use?17:11
claygacoles: well, what ide is it - aren't Mixin's like a not so uncommon python thing?17:11
acolesclayg: i tried making the Mixin class inherit TestCase but then the tests in that class get run (and fail of course)17:12
acolespycharm17:12
claygacoles: right - which methods specifically does it think are unresolved?17:12
peluseall this talk of Mixin is making me thirsty....17:12
acolesself.assertEqual etc17:12
acolesso what does your ide do? it can't know that those methods will get inherited from another superclass17:13
claygacoles: vim doesn't complain?17:14
*** annegentle has joined #openstack-swift17:14
claygacoles: anyway - what if we make a TestMixinABC in test.unit.utils and add unittest.Testcase abstract methods to it as we go?17:14
clayghttp://stackoverflow.com/questions/28010960/are-mixin-classes-abstract-base-classes17:14
acolesclayg: huh, but vim does complain if you type self.banana()?17:15
claygnah, that shits to hard in python - you never know at runtime if you'll have a bananna or not!17:16
claygor test.unit.__init__ w/e17:16
cschwedeclayg: so, just tested. it would be possible to simply do a „pip install -r requirements.txt“ before, that works too. i’m now running it again on master, will send you the bug report afterwards17:18
acolesclayg: true, which makes me wonder what heuristics pycharm is applying17:19
claygpeluse: both of those tests were replaced and moved - not sure in which change exactly - maybe this one - anyway it looks like i'm just going to delete them - they're covered in the per-manager tests17:19
claygacoles: well it's obviously trying really hard - it can't know that the runner doesn't execute the non-TestCase class and it "just works"17:20
claygacoles: but I like the idea of hinting that the MixIn is an abstract class that can't be used unless you define methods xyz (or subclass from someone who does)17:21
notmynamepeluse: I'm told that -infra is ready. I just rechecked the master->ec patch17:21
peluseclayg, great.  I saw several copies (well >1) so my next step was going to be to back track and review if there were suposed to be or not :)17:21
pelusenotmyname, cool17:21
claygacoles: how many methods do you need to add to the MixIn to get the warnings to go away?  assertTrue, assertFalse, assertEqual?17:21
acolesclayg: i think right now i would rather live with the noise in my ide than churning/reviewing the tests again. maybe after kilo, unless you think its trivial17:22
* peluse out for 90 min or so...17:23
acolesclayg: min, i'm on a different branch. fail() from memory17:25
claygpeluse: I think you messed up the rebase on the ssync change - there's a bunch of tests that moved from the diskfile to the manager test case that are duplicated now17:26
cschwedeclayg: error on master: http://paste.openstack.org/show/196089/17:27
peluseclayg, OK, about to walk out the door.  I can re-do it when I get back.  The only required tweaks are where I changed a lot of 'dev' to 'sda1'17:27
cschwedethe problem is in line 191: ==> default: Ran python setup.py develop && pip install -r test-requirements.txt returned 117:27
peluseor whatever its defined to in one of those classes self.expected1 or something like that17:27
cschwedeswapped commands are working17:28
acolesclayg: assertRaises and fail seem to be it. but hey, its not a priority, i can most likely turn down the heat in pycharm and stop whining17:28
clayghey do you guys remember back when you used to install python code with setup.py develop - and then pbr broke everything and now we type 'pip install -e .' like that's how it supposed to work?17:29
acolesclayg: like, i don't 'hate' that you did it, its a useful pattern, just wish we could inherit from TestCase without causing the tests to be run17:30
claygacoles: *right* - someone said that nose would respect a __test__ = False class attribute - but it didn't work for me17:30
* cschwede afk dinner, brb17:31
*** zhill has joined #openstack-swift17:32
acolesclayg: hmmm, well maybe i'll dig some more some time. btw thanks for the changes to write_durable/commit17:34
acolesok lets see if we can verify jrichli 's cool func test patch17:34
*** annegentle has quit IRC17:34
notmynameclayg: oh is that what I'm supposed to do? I end up with `python setup.py develop` after switching between master and feature/ec17:34
claygpeluse: ok, looks like you're out for a min - i have the ssync patch fixed up like i want - i'm going to try and apply some of the fixes acoles and kota requested into one rebase17:35
claygnotmyname: I think that's fine for getting the bin scripts pointing at the right versions - but python setup.py develop just doens't work for dependency resolution anymore - pbr has ruined something and no one seems to care :'(17:36
*** Bsony has joined #openstack-swift17:36
claygthat's not fair17:36
claygI think we all care - but trying to fix it such a terrible mess because the obvious conclusion is just to stop using it - which for infra is a non-starter - so we give up17:37
acolesclayg: notmyname : i noticed recently that i have to repeat setup.py after running tox. but i do get easily confused :/17:38
claygwhat?!17:38
claygi hate python17:38
openstackgerritMerged openstack/swift: Imported Translations from Transifex  https://review.openstack.org/16710617:38
claygredbo: dfg: are you guys full on the go-go-rack-labs now or still chipping away at hummingbird?17:39
claygglange: you filed a bug about the policy stat table - i was thinking I should try and help - do you need any help?17:39
acolesclayg: re 'i hate python' - i can't remember if i hit submit for my comment that we should make our variable names hungarian e.g. strTimestamp :D17:40
claygacoles: we should stop having vairables called timestamp that are strings17:41
torgomaticthe penalty for actually doing so is fifty goulashes17:41
*** Bsony has quit IRC17:41
claygtorgomatic: like... the soup?17:42
notmynameclayg: looks like dfg is adding some hummingbird-specific stuff into their deployment of swift17:42
torgomaticclayg: none other17:42
acolestorgomatic: sounds worse than the lashes :)17:42
torgomatic:)17:42
*** annegentle has joined #openstack-swift17:43
acolesclayg: yeah, i know, and stop using ts for tombstones and timestamps17:44
*** mmcardle has quit IRC17:44
dfgnotmyname: you still get emails about pull requests to our prod stuff?17:44
claygacoles: re the [Tt]est in the name - I'm pretty sure unittests's test runner needs the class to be subclassed from unittest.TestCase - but nosetest will pick up anything that has name with test in it (or even is a subclass of TestCase)17:44
notmynamedfg: it's a public repo17:44
dfgnotmyname: ya i know17:45
notmynamedfg: so yeah, I'm subscribed, just to see what you're up to and running17:45
claygdfg: you never call17:45
claygdfg: we worry about you17:45
*** tsg has joined #openstack-swift17:45
dfgwe're still experimenting with stuff and will have somehting to show about hummingbird fairly soon.17:46
claygacoles: i don't know what to do about ts - that's a damn convienient variable name :\17:46
claygdfg: yay yay yay!17:46
*** tsg_ has joined #openstack-swift17:47
*** rdaly2 has quit IRC17:50
acolesclayg: til from man "functions and classes whose names match testMatch and TestCase subclasses with any name will be loaded and executed as tests"17:50
*** tsg has quit IRC17:50
dfgnotmyname: if that stuff pans out i'll clean it up and submit it to upstream, once things are a little more solidified on our end. which should be soon.17:50
notmynamedfg: great17:51
claygacoles: i'm not sure if you telling me I'm full of it - or if you agree?17:52
claygacoles: i like your change to test _get_ondisk_file instead of manager.get_ondisk_files17:53
acolesclayg: lol. the man page agrees with you so i do too!18:00
acolesclayg: ok good (re test)18:01
*** Gue______ has quit IRC18:02
*** jamielennox|away is now known as jamielennox18:04
openstackgerritPrashanth Pai proposed openstack/swift: Refactor server side copy as middleware  https://review.openstack.org/15692318:05
*** jordanP has quit IRC18:10
InAnimaTehey all, in a bit of a pickle. i have a cluster that is over 90% full and really need some relief asap18:13
InAnimaTesome of the users have started removing some stuff...however because of (im guessing) the reclaim age, its taking a while for stuff to actually be removed.18:14
InAnimaTeis there any way i can speed this up to almost instantly remove items ?18:14
*** annegentle has quit IRC18:16
InAnimaTenote that im running 1.8.018:20
ctennisInAnimaTe: the bytes are freed if users are actually deleting objects18:24
*** reed has quit IRC18:24
ctennisif they are deleting the account and expecting the objects to be cleaned up, then you need to run the reaper a little more aggressively18:24
InAnimaTehmm that doesn't seem to be what im seeing18:26
ctenniswhen an object is deleted, a 0 byte tombstone is written out18:27
notmynameInAnimaTe: the reclaim age _must_ be longer than your replication cycle time (and the time it takes you to handle failed hardware) so that data isn't resurrected18:27
InAnimaTectennis: so the next time the replicator is run, it reclaims the space utilized by whatever object existed at where that tombstone is?18:28
InAnimaTenotmyname: but if everything is running 100% fine, then that time should be short18:29
InAnimaTeat least i would think18:29
ctennisInAnimaTe: no, that space should be reclaimed instantly on DELETE - only if you have objects in handoff locations due to drive failures or other replication issues would oyu have extra data that replication needs to clean up18:30
ctennisInAnimaTe: in that case then, yes replication would take care of cleaning it up18:30
*** annegentle has joined #openstack-swift18:30
ctennisInAnimaTe: any idea how long your replication cycle time is?18:30
notmynameInAnimaTe: yes. it's what ctennis just said. the reclaim age is to allow replication to clean up handoffs and ensure that it isn't missed18:31
notmynameInAnimaTe: also, check your quarantine directory to ensure that there isn't a lot of stuff there taking up space18:31
openstackgerritClay Gerrard proposed openstack/swift: Add Fragment Index filter support to ssync  https://review.openstack.org/16518818:34
openstackgerritClay Gerrard proposed openstack/swift: Cleanup ECDiskFile classes  https://review.openstack.org/16709718:34
openstackgerritClay Gerrard proposed openstack/swift: Multiple Fragment Archive support for suffix hashes  https://review.openstack.org/15963718:34
openstackgerritClay Gerrard proposed openstack/swift: Add commit method to interface of DiskFile  https://review.openstack.org/16520818:34
*** reed has joined #openstack-swift18:35
claygpeluse: I was going to rebase the ec-recon - but I'm honestly not sure which version of the disk/dir check stuff you want?18:42
claygpeluse: can you take over rebasing from https://review.openstack.org/#/c/131872/37 onward?18:42
peluseclayg, thanks, I'll take recon as I'll rebase and work on it as well18:42
claygok, i'm heading into the office18:43
peluseso yes for sure, and thanks thanks thanks!18:43
*** annegentle has quit IRC18:45
*** fthiagogv has quit IRC18:46
*** lpabon has joined #openstack-swift18:48
openstackgerritpaul luse proposed openstack/swift: Erasure Code Reconstructor  https://review.openstack.org/13187218:50
*** jkugel has joined #openstack-swift18:53
acolesjrichli: i got the policy-specific func tests to run :) needed some tweaks - i hacked my proxy to return 500 if the default policy was used and then used SWIFT_TEST_POLICY=<non-default> and all but two tests pass!18:57
acolesjrichli: the two fails are tests that explicitly setup policy headers to best across policies and so run foul of my proxy hack18:58
acoless/best/test/18:58
jrichliacoles: sounds good!18:59
tdasilvaacoles, jrichli: are the two tests that are failing copy related?19:00
acolestdasilva: TestCrossPolicyObjectVersioning19:00
acolestdasilva: NB they are only failing in this context because I deliberately broke the proxy to reject default policy19:01
acolestdasilva: i wanted to convince myself that jrichli patch really did cause all the tests to use a specified policy19:01
*** jistr has joined #openstack-swift19:02
tdasilvaacoles: ok...just wondering because I saw patch 162890 and I was wondering if that would help jrichli's patch19:02
patchbottdasilva: https://review.openstack.org/#/c/162890/19:02
acolestdasilva: also note i was working on master with replication policies19:02
tdasilvaacoles: oh ok, I was going to test on feature/ec19:03
acolestdasilva: maybe you could take a look at my comment on https://review.openstack.org/#/c/166097/ and see what you think about making TestCrossPolicyObjectVersioning use the specified_policy rather than a random choice19:03
tdasilvaacoles: sure, looking now19:04
acolestdasilva: jrichli : the random choice in those tests means that even if we specify a policy, the specified policy may not be used at all in those tests19:04
acolestdasilva: so i suggest SWIFT_TEST_POLICY should override the random-ness19:04
openstackgerritpaul luse proposed openstack/swift: wip: ec reconstructor probe test  https://review.openstack.org/16429119:04
acolestdasilva: i also wonder if those tests should be duplicated with reversed policies w.r.t. the version source and target??19:05
*** geaaru has quit IRC19:06
acolesjrichli: anyway, i feel pretty positive about that! thx!19:06
tdasilvaacoles: ok, caught up19:07
tdasilvaacoles: I see no reason to testing the reversed way too19:08
tdasilvaacoles: just a question: would we need two env variables to specify two policies?19:08
acolestdasilva: my thinking was that (initially at least) we just want to be sure that the specified policy is exercised by all tests19:09
acolestdasilva: i guess to be thorough, maybe cross policy tests should be run using all combinations of policies??19:10
*** torgomatic has left #openstack-swift19:11
acolestdasilva: or at least policy types, but that imho is a separate issue from jrichli patch19:11
*** torgomatic has joined #openstack-swift19:11
*** ChanServ sets mode: +v torgomatic19:11
tdasilvaacoles: agreed19:11
*** silor has quit IRC19:11
*** silor has joined #openstack-swift19:12
tdasilvaacoles: trying to locate the randomness you mentioned19:12
acolestdasilva: i'll get a link19:12
tdasilvait's that select() call right?19:13
tdasilvaacoles: ^19:13
acolestdasilva: yes, thats it19:13
tdasilvareturn random.choice(self)19:13
acolesyup19:14
acolestdasilva: so maybe that becomes return specified_policy or random.choice() ??19:14
tdasilvaacoles: sounds like a good idea, but then in the case of TestCrossPolicyObjectVersioingTest the second policy selected will just be a random one?19:17
tdasilvaI think that would be ok, right?19:17
acolestdasilva: yes, random but excluding the specified_policy. i think that is ok. we at least force the specified_policy to be used as one of the test policies.19:18
*** silor has quit IRC19:19
acolestdasilva: a follow on might be to swap the policies in the test wrt versions, so we then know the specified_policy has been tested as a source and a target for versions19:19
acoles'swap' as in repeat test with policies swapped19:19
tdasilvaacoles: yes, sounds good19:20
acolesjrichli: so is this making sense to you ^^?19:20
tdasilvaacoles: so, jrichli's patch is in feature/ec atm, are you guys planning to move it to master?19:21
acolestdasilva: :D jrichli has been asking that, idk!19:21
acolespeluse: what do you think?^^ this https://review.openstack.org/#/c/166097 could go on master, or stay on feature/ec?19:22
tdasilvaacoles: the other issue i'm concerned with that patch is the issue of copying between two different policy types19:23
tdasilvaI think jrichli ran into that issue, right?19:23
acolestdasilva: not sure19:24
acolestdasilva: whats your concern?19:24
jrichlitdasilva: I ran into an issue where an ECController was being used for a non-EC policy19:25
tdasilvajrichli: right, that's where I think this patch 162890 might come in handy19:25
patchbottdasilva: https://review.openstack.org/#/c/162890/19:25
*** panbalag has quit IRC19:27
tdasilvaacoles, jrichli: so I'm thinking that jrichli's patch might need to be dependent on that19:27
jrichliif the patch goes on master, the tests will pass.19:27
jrichliand we could merge to ec from there19:27
*** panbalag has joined #openstack-swift19:28
tdasilvawell...but not until the other patch is merged19:28
acolestdasilva: jrichli : thats a bug with ec though not functional testing, right?19:28
jrichlii thought the other patch only caused issues on the ec branch.  is that not correct?19:28
tdasilvajrichli: yes, ec only19:28
*** tellesnobrega has quit IRC19:29
acolesbbiab19:29
*** gyee has quit IRC19:30
tdasilvaacoles: sorry i'm being confusing...I guess the point I'm trying to make is that motivation of jrichli patch is to allow testing with EC policies, but we won't really be able to test that until this other problem of copying between two policy types is fixed19:30
*** jistr has quit IRC19:31
jrichlitdasilva: oic.19:31
*** tellesnobrega has joined #openstack-swift19:33
jrichlitdasilva acoles: so basically, we may as well keep it on ec19:34
tdasilvajrichli: it makes sense to me, but I could be over thinking this :P19:36
*** morganfainberg is now known as needslesscoffee19:39
acolestdasilva: got it. but we could merge jrichli patch before ec copy is fixed. the tests will just fail *if* SWIFT_TEST_POLICY=<ec-policy>. correct?19:40
*** Nadeem has joined #openstack-swift19:41
tdasilvaacoles: yes, true19:41
acolestdasilva: i have to go, thanks for your help!19:44
acolesjrichli: tdasilva : have a good afternoon/evening!19:44
jrichliacoles: you too19:44
tdasilvaacoles: thank you! have a good one19:44
claygpeluse: looks like we're still waiting on jenkins for the ec-probe branch - but otherwise looking good - thanks!19:45
*** acoles is now known as acoles_away19:46
clayglet's merge something!  jrichli's func tests?  ec fix copy?  something related to torgomatic's range fixes?19:49
claygoh oh or maybe patch 167097?!19:49
patchbotclayg: https://review.openstack.org/#/c/167097/19:49
claygjust throwin' that out there19:49
tdasilvalol19:49
jrichli:-)19:49
*** needslesscoffee is now known as morganfainberg19:50
jrichliclayg: I still have one more bit to do on the func tests.  I probably wont get to work on that tonight, tho.19:51
claygjrichli: sounds like a good reason to merge it now!19:51
jrichlilol!19:51
petertr7notmyname: I'm interested in discussing an idea at tomorrow's weekly meeting. Is it okay if I just add it as an item on the (wiki) agenda?19:55
*** lpabon has quit IRC19:56
notmynamepetertr7: what's up?19:56
petertr7Hi! I just had an idea I wanted to discuss, I mentioned it here yesterday and figured maybe the meeting was a good place to open it up19:56
petertr7It's regarding python-swiftclient19:57
petertr7And how large files are handled19:57
petertr7I wanted to allow users, say uploading a 10gb file, to stop their upload and continue their upload at a later time without reuploading successfully uploaded segements19:57
notmynamepetertr7: ya, that sounds pretty useful. I'd guess the client would need to be smart enough to detect that previously uploaded chunks are available and not re-upload them19:58
petertr7I just wanted to get a feel for whether the community would find it useful or not19:58
petertr7I implemented something locally19:58
notmynamecool19:58
petertr7outside of python-swiftclient19:59
petertr7which does an md5 checksum and compares it to the etag of potentially uploaded segments19:59
petertr7So for example, a 10gb file segmented into 1gb chunks are uploaded and interrupted19:59
petertr7I'd attempt to look at etags for the first 1gb, then the 2gb and stop when the etags don't exist or are not matching20:00
petertr7and then continue uploading at that chunk20:00
notmynameright, makes sense20:01
petertr7Would this be something I could work on and contribute to python-swiftclient?20:01
notmynamepetertr7: so, yes, I think it would be great. is it something you're planning on working on?20:01
notmynameyes! :-)20:01
petertr7Awesome, thanks!20:02
petertr7One thing I haven't ironed out in my head is when to do these checks20:02
claygnotmyname: how do i close a bug -> https://bugs.launchpad.net/swift/+bug/66795620:03
openstackLaunchpad bug 667956 in OpenStack Object Storage (swift) "chunked transfer client disconnect causes unhandled value error in object server" [Low,Confirmed]20:03
petertr7In my local script I do the check every time I attempt to upload any large file20:03
joeljwrightpetertr7: it's possible that the continue upload logic would be closely related to a patch that's currently being reviewed for skip-identical and large objects20:03
joeljwrightpetertr7: https://review.openstack.org/#/c/161043/20:04
notmynamepetertr7: definitely joeljwright is the one to talk to :-)20:04
petertr7joeljwright: Neat! Thank you!20:04
petertr7notmyname: Thanks for your time! I appreciate it!20:05
notmynamejoeljwright: also, I've been getting bugged about doing a new release of swiftclient20:05
joeljwrightnotmyname: that would be good - but it seem like everyone is too focused on EC patches at the moment :)20:05
claygpeluse: sweet looks like jenkins is cool with the probe tests too - are they passing?20:06
notmynamejoeljwright: last time I asked about that, the answer was "wait until https://review.openstack.org/#/c/130339/ lands"20:06
*** agentle has joined #openstack-swift20:06
joeljwrightI'd definitely like to see that land20:06
notmynamejoeljwright: yes, there's a huge focus on ec at the moment, but the actual release is pretty simple (trivial, actually)20:06
notmynamejoeljwright: all i have to do is make a singed tag and push it up. that's it. takes about 10 seconds20:07
joeljwrightnotmyname: and the skip-identical patch I just mentioned would be nice to have20:07
notmynameya20:07
claygnotmyname: peluse: so like if we can't get the current master (including the PUT method extraction) merged to feature/ec like by EOB - we *literally* have a problem.20:07
notmynamejoeljwright: so the question is if we wait until those land or just tag what's there now20:07
claygnotmyname: peluse: please let me know if there is anything more useful I could do to help with that than say just whining like a little baby20:07
* notmyname goes to get the grease for the squeaky wheel ;-)20:08
joeljwrightnotmyname: I think the problem is not the effort in creating the release, but the effort required to review the nice pending patches20:08
claygjoeljwright: stupid patches, always needing reviews20:08
joeljwrightnotmyname: might be better to release what's there now20:08
notmynamejoeljwright: yup. exactly20:08
claygjust *one more thing*20:08
claygFOREVAR20:08
joeljwrightclayg: :D20:08
peluseclayg, I am more than willing to do it20:09
joeljwrightnotmyname: there is one patch already merged that I'd love to see released (fixes a bug that leaves orphaned segments)20:09
pelusenotmyname, do you know whats up with the requirements.txt problem, should I just try a fresh one w/latest master?20:09
notmynamepeluse: it's getting sorted. they had a bug in their fix20:10
pelusenotmyname, ahh OK.20:10
peluseclayg, once we can do that (a simple merge again with a requirements change) I'll take the big one - well, I'll attack it and whether its a clean tackle or not is yet to be seen20:10
notmynamewhich is what clayg was just whining about. but it looks like the current patch needs to be replaces with a new one20:10
notmyname"whining" said with love20:11
pelusenotmyname, yeah, we can land it then a  new one jsut to make sure the requiments chnnge takes20:11
pelusesince the current one is super simple20:11
pelusebut I guess in the mantime I will shelve my unit test work and start a merge so its ready when things "work" on the jenkins side... starting now20:11
notmynamepeluse: thanks. that's what I was about to say :-)20:12
*** cdelatte has quit IRC20:16
notmynamepeluse: ok, I'm told it takes about an hour to rebuild and deploy the worker images so the fix is out20:17
notmynamepeluse: and fungi is working on it20:17
pelusecool20:17
notmynameso that gives time for getting a new master->ec merge20:18
pelusefixing merge conflicts now... only 32 of them so not horrible I guess :)20:18
claygpeluse: so... we're all in a holding pattern waiting on infra before we attempt the merge?  oh.. or we should work on a new merge now?20:18
claygpeluse: wow :)20:18
peluseclayg, I'm resolving conflicts now and will push it when told its ready20:18
* clayg hugs peluse 20:19
peluse"it" meaning the jenkins fix of course20:19
notmynamepeluse: actually push it whenever it's ready. and if -infra isn't ready by then, we'll simply recheck it20:20
peluseOK20:21
* peluse wonders if clayg will be in a huging mood when he has to rebase the patch chain on feature/ec :)20:21
*** trunk has joined #openstack-swift20:21
claygoh right :\20:21
claygwell maybe it's no so bad - that change should mostly be out of the proxy - well hell I guess I don't really know what's coming in with master20:22
claygpeluse: anyway you're right - we have too much un-merged code20:22
claygpeluse: but I can't find anything I didn't write worth merging ;P   (j/k)20:23
peluseheh20:29
peluseI do indeed agree though20:29
claygsee tdasilva gets it20:33
tdasilvalol20:33
tdasilvaclayg: go merge obj. versioning :P20:33
*** reed has quit IRC20:42
notmynamejoeljwright: I'm about to go to a meeting, but I'm not opposed to tagging/releasing python-swiftclient 2.3.2 today20:46
claygtdasilva: oh good one20:46
tdasilva;-)20:47
*** nellysmitt has joined #openstack-swift20:47
claygtdasilva: i sorta think i was either putting that off because functionally testing the upgrade was a pita - or there was something about container vs. object request routing that I keep trying to refactor everytime I look at it20:47
notmynamejoeljwright: I know it's getting late for you. please let me know if you have any concerns if a 2.3.2 release were to happen20:48
claygtdasilva: oh wait - what happend - it's like way simpler now?20:48
tdasilvaclayg: what do you mean?20:48
claygtdasilva: like maybe everything goes through the context thing now?20:49
claygtdasilva: idk, just seems simpler since the last time I looked at it?20:50
tdasilvaclayg: I think that was based on a suggestion you made20:50
tdasilvaclayg: so now there's container_request and object_request20:51
tdasilvaremember __call__ was very long??20:51
claygtdasilva: I think you're just plying to my egomania trying to get reviews20:56
tdasilvalol20:56
claygtdasilva: ... could work20:56
tdasilvaclayg: no, i actually just found the comment too20:56
tdasilvahttps://review.openstack.org/#/c/134347/21/swift/common/middleware/versioned_writes.py20:56
tdasilvaclassicsnail: L34720:56
tdasilvaclayg: L34720:57
tdasilvaclassicsnail: sorry20:57
joeljwrightnotmyname: sorry I was picking my wife up from the station20:57
*** mahatic has quit IRC20:59
joeljwrightI don't think I'd have any concerns about a 2.3.2 release21:00
joeljwrightthere are certainly quite a few fixes that would be nice to have in the released version… I just might be bugging you for a 2.3.3 release sooner than you'd like21:01
*** joeljwright1 has joined #openstack-swift21:05
*** joeljwright has quit IRC21:06
joeljwright1notmyname: and my laptop chose that moment to lock up21:06
*** mmcardle has joined #openstack-swift21:06
*** openstackgerrit has quit IRC21:07
*** openstackgerrit has joined #openstack-swift21:07
*** kevinc_ has joined #openstack-swift21:11
tdasilvajrichli: still around?21:13
jrichliye21:13
jrichlis21:13
tdasilvajrichli: hi, i started playing with that functional test change that acoles_away was talking about, will try to send you something later, ok?21:14
jrichlisounds good, thanks!21:15
peluselooks like I picked the wrong week to stop sniffin' glue...21:17
jrichlilol21:17
tdasilvahaha21:17
pelusetdasilva, I'm merging into feature/ec if you hadn't gathered (you PUT refactor)... fun stuff!21:19
*** reed has joined #openstack-swift21:20
tdasilvapeluse: mmm...i can imagine...I almost wish we had landed that other patch on feature/ec first21:20
tdasilvathen it would be easy to merge21:20
pelusebah, it'll sort out :)21:21
tdasilvaok, let me know if I can help with anything21:21
claygman, our test suite is going to get so much better when we don't care about python 2.6 anymore21:22
claygwho cares about python 2.6 again?21:22
* torgomatic sure doesn't21:25
mattoliverauMorning, wow, that was a lot of scroll back to read!21:25
openstackgerritSamuel Merritt proposed openstack/swift: EC: support multiple ranges for GET requests  https://review.openstack.org/16657621:28
openstackgerritSamuel Merritt proposed openstack/swift: EC: better failure handling for GETs  https://review.openstack.org/16740621:28
*** jrichli has quit IRC21:31
pelusehey cores:  quick request, please don't land anything on feature/ec til I'm done the the current merge from master.  Gracias!21:38
mattoliverauhai21:39
*** jkugel has quit IRC21:41
*** jkugel has joined #openstack-swift21:41
openstackgerritClay Gerrard proposed openstack/swift: Update contianer sync to use internal client  https://review.openstack.org/14379121:43
*** mmcardle has quit IRC21:43
claygmattoliverau: yeah idk why everyone is so chatty today21:43
*** jkugel has quit IRC21:46
*** agentle has quit IRC21:48
*** Cipher45 has quit IRC21:48
*** wolsen has quit IRC21:48
*** tristanC has quit IRC21:48
*** agentle has joined #openstack-swift21:49
*** Cipher45 has joined #openstack-swift21:49
*** wolsen has joined #openstack-swift21:49
*** tristanC has joined #openstack-swift21:49
*** sc has quit IRC21:50
*** agentle has quit IRC21:50
*** sc has joined #openstack-swift21:52
*** zhill has quit IRC21:53
*** dfg has quit IRC22:06
*** dfg has joined #openstack-swift22:07
*** hurricanerix has quit IRC22:07
*** hurricanerix_ has joined #openstack-swift22:07
*** joeljwright1 has quit IRC22:12
notmynamepeluse: what's the status of a new merge from master?22:14
*** zhill has joined #openstack-swift22:14
pelusenotmyname, its going pretty good if you can believe it22:20
notmynamecool. -infra says things should be ready from their side22:20
peluseGET/PUT working fine.  half dozen or so tests failing with one signature and another set with just one other signature.  Digging into those now22:21
claygtorgomatic's good idea of the hour - X-If-Not-Match-Fragment-Index-List22:27
claygalso, i feel like the commit message on patch 167406 could be a popular theme this week22:28
patchbotclayg: https://review.openstack.org/#/c/167406/22:28
claygtdasilva: I really wanted to review versioned writes - but I just can't - i need to keep looking at some ec stuff - sorry - maybe next week between rebase's of the merge to master chain!22:30
tdasilvaclayg: no worries22:31
notmynameok, I'm tagging python-swiftclient 2.4.022:32
notmyname(there hasn't been a release since september!)22:33
*** shri has joined #openstack-swift22:35
notmynamedone22:35
shriHi, I have a question about the upstart scripts for various swift processes that are installed with the debian packages.22:37
shriI see that these scripts don't have a respawn clause in case the process dies. Any particular reason for that?22:38
peluseOK, bout to do last tox and then if it works "fire in the hole"22:41
peluseoh crap... have to go pick up son from school... BRB :)22:41
*** zhill has quit IRC22:44
mattoliveraupeluse: Lol, it's like going to an ad break just before the cliff hanger :P22:49
*** Nadeem has quit IRC22:55
peluseheh, back and tox'ing...22:58
openstackgerritpaul luse proposed openstack/swift: Merge master to feature/ec  https://review.openstack.org/16742923:05
notmynamepeluse: yay23:05
notmynameI'll abandon the other one (if you haven't already)23:06
pelusejust did!23:06
notmynameThat Was Easy23:06
notmynameI'm tracking the jobs on http://status.openstack.org/zuul/23:07
peluseheh, good luck trying to read the obj.py diff on that bad boy in gerrit!23:07
peluseme too23:08
*** vinsh has joined #openstack-swift23:09
*** welldannit has joined #openstack-swift23:11
peluseBAH!  requirements failed23:12
notmynamepeluse: on it23:12
pelusecool23:13
notmynamepeluse: but the good news is that we have the merge commit done now23:13
notmynameso it should "just" be a recheck23:14
claygyeah that proxy.controller.obj diff looks wonky :\23:15
claygtorgomatic: can you look at patch 167429 when you get a chance and guess at if the ObjController changes look sane?23:16
patchbotclayg: https://review.openstack.org/#/c/167429/23:16
torgomaticclayg: sure, give me 15 minutes23:16
peluseindeed23:16
peluseclayg, well, its wonky because of all the movement23:17
peluseI didn't bring over the repl specific _store_object() so what will land will be the putter/mime method (refactored) for both23:18
claygpeluse: oh i know - but the line diffs seem strange too - i guess because maybe - did we already merge something close to that refactoring onto feature/ec?23:18
peluseif you really want to separate the paths we *could* have different _store_object() methods23:18
claygpeluse: oic - I had taken a different approach in patch 164950 - yeah i split ec stuff into ECObjectController next to GET23:19
patchbotclayg: https://review.openstack.org/#/c/164950/23:19
claygpeluse: I still think that's the way to go - but it can come after we merge them together then pull them back apart23:19
peluseahh geeze, didn't even know that was there :)23:20
*** chlong has joined #openstack-swift23:20
claygpeluse: oh it's fine - i don't think it would have helped anything23:21
pelusewell its just software, we can always change it again later :)23:21
notmynamepeluse: I've got to head home, so I'll be offline for about an hour. what do you need from me for the merge? I want to see that land today23:28
peluseI guess just lean on infra dudes to fix whatever is messed up.  the code I think is fine23:29
notmynameok23:29
notmynamepeluse: "zuul estimates 5 minutes until it merges, then roughly an hour for me to get the worker images replaced again"23:31
notmynamefrom fungi in -infra23:31
peluseOK, do we just recheck no bug in a few hrs?23:32
notmynamepeluse: yup23:32
peluseBTW, just to confirm.  All unit, functional and probetests pass on the merge patch23:32
*** chlong has quit IRC23:35
*** chlong has joined #openstack-swift23:35
InAnimaTeanyone know how long python-swiftclient stores the auth url?23:42
InAnimaTesigh, i mean the x-storage-url23:42
*** trunk has quit IRC23:47
*** dencaval has joined #openstack-swift23:48
*** ho has joined #openstack-swift23:49
*** gyee has joined #openstack-swift23:54
*** zhill has joined #openstack-swift23:55
*** zaitcev has joined #openstack-swift23:56
*** ChanServ sets mode: +v zaitcev23:56
*** vinsh has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!