| *** jistr has quit IRC | 00:00 | |
| *** jistr has joined #openstack-swift | 00:01 | |
| *** abhitechie has joined #openstack-swift | 00:21 | |
| *** tonanhngo has quit IRC | 00:30 | |
| *** tonanhngo has joined #openstack-swift | 00:32 | |
| *** tonanhngo has quit IRC | 00:37 | |
| *** jlvillal is now known as jlvillal_pto | 00:42 | |
| *** zhurong has joined #openstack-swift | 00:43 | |
| *** vint_bra has joined #openstack-swift | 00:45 | |
| *** winggundamth has joined #openstack-swift | 00:46 | |
| *** bkopilov has quit IRC | 01:27 | |
| kota_ | good morning | 01:30 |
|---|---|---|
| *** abhitechie has quit IRC | 02:02 | |
| *** abhitechie has joined #openstack-swift | 02:03 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Update implementation for DELETE of autoamted tiering https://review.openstack.org/454423 | 02:07 |
| *** vint_bra has quit IRC | 02:11 | |
| *** vint_bra has joined #openstack-swift | 02:13 | |
| *** Sukhdev has quit IRC | 02:15 | |
| *** vint_bra has quit IRC | 02:23 | |
| *** vint_bra has joined #openstack-swift | 02:31 | |
| *** abhitechie has quit IRC | 02:49 | |
| *** gkadam-mtg has joined #openstack-swift | 03:12 | |
| *** gkadam-mtg is now known as gkadam | 03:13 | |
| mattoliverau | kota_: morning | 03:15 |
| *** ianychoi has quit IRC | 03:36 | |
| *** Sukhdev has joined #openstack-swift | 03:40 | |
| *** ianychoi has joined #openstack-swift | 03:42 | |
| *** Sukhdev has quit IRC | 03:44 | |
| *** _JZ_ has quit IRC | 03:46 | |
| *** links has joined #openstack-swift | 03:47 | |
| *** zhurong has quit IRC | 03:51 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Fix StopIteration if no more nodes in DB Replication https://review.openstack.org/454447 | 03:57 |
| *** zhurong has joined #openstack-swift | 04:03 | |
| *** klrmn has quit IRC | 04:07 | |
| *** jamielennox is now known as jamielennox|away | 04:18 | |
| *** dja has joined #openstack-swift | 04:26 | |
| *** JimCheung has quit IRC | 04:29 | |
| kota_ | hmmm | 04:33 |
| kota_ | mattoliverau: o/ | 04:33 |
| kota_ | in my saio vm, the probe test doesn't work as expected | 04:33 |
| kota_ | idk, why revive_drive fails with OSError Is a directory? | 04:34 |
| kota_ | it looks like, the renamer fails because the target already exists and it's dir | 04:35 |
| kota_ | but looking at the swift.probe.common, kill_drive should move the dir to <device> + 'X' so it should not be there | 04:36 |
| kota_ | or just mount/unmount | 04:36 |
| kota_ | I'm using mount_check=false so it should be handled just as a dir... | 04:40 |
| clayg | kota_: mattoliverau: PavelK: you guys rock! thanks for running with patch 454174 | 04:45 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 04:45 |
| kota_ | clayg:^^ | 04:45 |
| kota_ | clayg: and i'm in trouble with probe above, will look at deeply, sorry, just now I'm going to another meeting :/ | 04:46 |
| mattoliverau | kota_: yeah ok. Not sure how smart the probe tests are when you just move a dir, is that going to raise a NotMounted? | 04:46 |
| mattoliverau | kota_: I was using a SAIO that has drives to unmount. | 04:47 |
| mattoliverau | I should loop back and continue looking at runway, because the way notmyname has designed it, it'll give you drives to umount | 04:47 |
| *** rcernin has joined #openstack-swift | 04:48 | |
| kota_ | mattoliverau: er - you mean, mount_check=False won't work yeah? | 04:49 |
| kota_ | mattoliverau: maybe that's true. | 04:49 |
| kota_ | so we need to check the options before runnig probe and SKIP if it's False? | 04:50 |
| kota_ | ah, if no mount_check, something like DiskFile class calls mkdirs to dig the *device* dir to reach the object/part/suffix/hash... | 04:53 |
| kota_ | make sense :/ | 04:54 |
| *** rcernin has quit IRC | 04:55 | |
| *** rcernin has joined #openstack-swift | 04:55 | |
| kota_ | let me make sure the intermediate state on running probe... | 04:55 |
| *** rcernin is now known as rcernin|wfh | 04:55 | |
| kota_ | mattoliverau: thanks for your advice | 04:57 |
| *** dja has quit IRC | 05:07 | |
| *** PavelK has joined #openstack-swift | 05:13 | |
| *** ChubYann has quit IRC | 05:19 | |
| kota_ | ok, that was true | 05:39 |
| kota_ | oops, mistake | 05:40 |
| kota_ | still looking | 05:40 |
| kota_ | weird, the db replicator test should touch only containers dir but it looks objects dir there | 05:45 |
| *** gkadam is now known as gkadam-afk | 05:48 | |
| *** m_kazuhiro has joined #openstack-swift | 05:50 | |
| m_kazuhiro | good morning | 05:50 |
| mattoliverau | m_kazuhiro: morning | 05:51 |
| mattoliverau | kota_: thanks for looking :) | 05:51 |
| m_kazuhiro | mattoliverau: morning! | 05:51 |
| kota_ | hmm... the reason why the objects dir exists was we're running self.replicators (i.e. also object-replicator) run_once | 05:53 |
| kota_ | I pugged unncecessary object-replicator run, but it stil... making "tmp" dir??? | 05:54 |
| mattoliverau | kota_: oh yeah, the probe test run replicators runs them all | 05:54 |
| mattoliverau | or whatever its called | 05:54 |
| kota_ | m_kazuhiro: and mount_check=False object-replicator/server will make the device there :/ | 05:55 |
| kota_ | it causes an error when the probe test tries to call revive_device! | 05:55 |
| kota_ | the device already there! | 05:55 |
| kota_ | gah | 05:55 |
| *** zhurong has quit IRC | 05:55 | |
| kota_ | m_kazuhiro: sorry, that's for mattoliverau^^ | 05:56 |
| *** cbartz has joined #openstack-swift | 05:58 | |
| kota_ | ah, ok. if device is missing, that will call complete_rsync rpc and no mount check there, that will create tmp file and then rename. | 06:05 |
| kota_ | so imo we should not run the probe with mount_check = False | 06:05 |
| *** zhurong has joined #openstack-swift | 06:16 | |
| mattoliverau | right, ok, so we'll need to add a a skip then :( | 06:18 |
| *** silor has joined #openstack-swift | 06:19 | |
| *** silor1 has joined #openstack-swift | 06:26 | |
| kota_ | :mattoliverau: yes, and it looks like the environ builder using vagrant-swift-all-in-one goes to mount_check=false in default | 06:26 |
| kota_ | idk, we can cofigure that | 06:27 |
| mattoliverau | kota_: I think this is why notmyname wants runway to make lvm partitions for devices so there is something to unmount | 06:28 |
| *** silor has quit IRC | 06:28 | |
| *** silor1 is now known as silor | 06:28 | |
| kota_ | mattoliverau: cool, is it in.. where? | 06:28 |
| *** pcaruana has joined #openstack-swift | 06:34 | |
| clayg | tdasilva: you tried to +2 EmilienM's patch 453424 and I poored water on the whole thing no it looks to fizzle out - I feel bad! | 06:36 |
| patchbot | https://review.openstack.org/#/c/453424/ - swift - Add sem-ver flag so pbr generates correct version (ABANDONED) | 06:36 |
| *** hseipp has joined #openstack-swift | 06:38 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: fixup! Add container sharding to Swift containers https://review.openstack.org/450491 | 06:42 |
| mattoliverau | kota_: https://github.com/swiftstack/runway it used ansible to build an env. Step 1 will be to make it create SAIO, then later use LXC or Docker containers so better simulate a swift cluster on one node. | 06:45 |
| kota_ | thx! | 06:47 |
| clayg | kota_: mattoliverau: for the probe test on the db unmount with mount_check = false - do you just need this -> https://gist.github.com/clayg/411374f07801bf9d2cd70aee0bf75936 | 06:53 |
| clayg | a/c server didn't get some love the obj server did back during EC to support a missing directory as "unmounted" when mount_check = false | 06:54 |
| kota_ | clayg: looks what I'm looking for, try it | 06:55 |
| clayg | SKIP: /etc/swift/account.ring.gz has 4 devices instead of 8 | 06:55 |
| clayg | let me re-configure ;) | 06:55 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift master: Container drive error results double space usage on rest drives https://review.openstack.org/454174 | 06:56 |
| clayg | PavelK: you're online! | 06:57 |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Refactor diskfile check implementation of automated tiering https://review.openstack.org/453548 | 06:57 |
| clayg | hi | 06:57 |
| mattoliverau | oh yeah, I made it 8, cause the 4 doesn't really test the first case. As you take one down, you wont really see it fill the cluster as there isn't enough drives to fill. | 06:57 |
| mattoliverau | well to see the effect anyway | 06:57 |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Update implementation for DELETE of autoamted tiering https://review.openstack.org/454423 | 06:57 |
| clayg | mattoliverau: oh... are the *other* tests still happy with 8? | 06:58 |
| mattoliverau | oh umm, yeah good question, I just wrote them so I could play with some patches :) | 06:59 |
| *** silor has quit IRC | 06:59 | |
| PavelK | clayg: yes, sometimes | 07:01 |
| clayg | PavelK: wait... whoa... you fixed the stop iteration (like patch 454447) | 07:01 |
| patchbot | https://review.openstack.org/#/c/454447/ - swift - Fix StopIteration if no more nodes in DB Replication | 07:01 |
| clayg | but instead of a seperate test - you add it the other fix to the end of the test that's covering the other thing :'( | 07:02 |
| PavelK | clayg: :-) Yes, I realize that I can fix it simply | 07:02 |
| clayg | but it's still two things ;) | 07:03 |
| clayg | minimal/simply is not "small line count" it's "does only one thing" ??? maybe? | 07:04 |
| PavelK | clayg: I believe that if I broke something then I should fix it. | 07:05 |
| PavelK | I understand what is "one thing" and sometimes I agree | 07:06 |
| clayg | that is a reasonable sentiment! perhaps I don't recgonize the coupling - me goes to look further | 07:06 |
| *** tesseract has joined #openstack-swift | 07:07 | |
| PavelK | ... but when my patch is merge before StopIteration fixed then the replication is not correct for small count of devices... | 07:07 |
| clayg | hrm.. it seems like next(more_nodes) could always raise StopIteration with small counts of devices - your chagne to fix the critical bug doesn't make it obviously worse - still thinking | 07:08 |
| * mattoliverau has to go, as I'm a single parent tonight, so need to go pickup the bub. o/ | 07:09 | |
| clayg | I honestly hadn't gotten to Kota's patch - he had the stand alone test for just the stopiteration issue - so I was looking forward to understanding that more | 07:09 |
| *** jaosorior_away is now known as jaosorior | 07:09 | |
| kota_ | clayg: re: the container-fix: we need also https://gist.github.com/bloodeagle40234/3a3f51a7b2584c6475df0bc76c6f7dad to prevent the failure with mount_check=False | 07:11 |
| clayg | s'ok - i really just want to re-grok the new patch set and do the review - hopefully I can understand - kota_ seemed to be having issues with the new probe test so I was going to try focus on that - the bug and fix is well understood - it's very bad and IMHO would be easy to merge w/ the probetest in the follow-up | 07:11 |
| kota_ | clayg: account-replicator/object-replicator also same mount_check issue and then they digging the device dirs | 07:11 |
| clayg | kota_: oh yes I see - because my crappy half assed fix didn't fix the account server | 07:11 |
| kota_ | clayg: yes | 07:12 |
| clayg | hrm... I had hoped maybe object-replicator was ok, actually hrm... why does one db_replicator have issue and not the other | 07:12 |
| kota_ | clayg, PavelK: and on the stop iteration issue, I think PavelK minds that, right? so the "next" call for iterator can raise StopIteration which is not handled anyware | 07:13 |
| kota_ | (i saw the comment on https://review.openstack.org/#/c/448480/ which is the latest one. | 07:13 |
| patchbot | patch 448480 - swift - Container drive error results double space usage o... | 07:13 |
| clayg | kota_: but that is existing issue yes? Even if we don't fix the over iteration issue with the existing regression? | 07:13 |
| kota_ | clayg: ah, yes. that is | 07:14 |
| clayg | kota_: YES! thank you for picking that up in the follow-up! beautifly done! | 07:14 |
| kota_ | clayg: it's exisiting issue which should be sparated. | 07:14 |
| kota_ | separated | 07:15 |
| clayg | ah, yes, ok - probetest - must focus :P | 07:15 |
| clayg | mattoliverau: i know you're gone but now i get SKIP: /etc/swift/object.ring.gz has 8 devices instead of 4 | 07:16 |
| clayg | first it was account has too few, now object has too many! :P | 07:16 |
| kota_ | oh | 07:19 |
| kota_ | I noticed NOW, PavelK squashed up my patch into that | 07:19 |
| clayg | kota_: /me snickers | 07:19 |
| clayg | but not the test! poor PavelK. | 07:19 |
| clayg | I feel like this totally normal person has showed up and is trying to help us - and is freaked out because we act like aliens and have strange occult incantations | 07:21 |
| clayg | "follow-up" "probetest" "irc" - it seems normal to us :D | 07:22 |
| clayg | kota_: does os.rename(old, new) fail OSError: [Errno 21] Is a directory? | 07:23 |
| kota_ | clayg: that is | 07:24 |
| kota_ | clayg: and "new" exists as dir | 07:24 |
| kota_ | as a dir | 07:24 |
| clayg | ugh, it's a bunch of cleanup - and he dir_check stuff in the container (and account) needs unitests - we should pull the probetests into a seperate change | 07:25 |
| clayg | it'd be a good change on it's own tho | 07:25 |
| clayg | some debate on how to handle required_devices - like you can't have a config that works for all the probests? seems wonky | 07:25 |
| kota_ | clayg: yeah, if including that into the patch, it seems too big to fix *THE BUG* | 07:26 |
| clayg | womp womp | 07:26 |
| clayg | it's not even working for me :'( | 07:26 |
| clayg | PavelK: we need to respin w/o the probe tst | 07:26 |
| clayg | PavelK: what can we do help you? | 07:27 |
| clayg | :'( | 07:28 |
| * kota_ is going to grab a new refill of iced coffee | 07:28 | |
| clayg | it's late for me - i really only stayd on hoping to +2 the bugfix - i could -1 it again I suppose | 07:29 |
| kota_ | clayg: :( | 07:29 |
| *** geaaru has joined #openstack-swift | 07:29 | |
| kota_ | clayg: sorry and thanks for working untill too late time for you | 07:29 |
| clayg | kota_: aww you're too kind | 07:30 |
| clayg | i saw PavelK's response in email today while chasing something else - after the kids where in bed it was like "man, i'm sure he's getting frustrated - that sucks" and also ... like I totally wanna get this fix merged :P | 07:30 |
| PavelK | clayg: I already added Kota's unit test to patch 454174 in form that was in patch 448480 ... | 07:33 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 07:33 |
| patchbot | https://review.openstack.org/#/c/448480/ - swift - Container drive error results double space usage o... | 07:34 |
| clayg | In both of those I see test_replicate_object_with_exception in Kota's I see test_replicate_object_with_exception_run_out_of_nodes | 07:36 |
| PavelK | test_replicate_object_with_exception_run_out_of_nodes is the tail of test_replicate_object_with_exception... see test_replicate_object_with_exception | 07:37 |
| PavelK | see https://review.openstack.org/#/c/454174/2..3/test/unit/common/test_db_replicator.py | 07:37 |
| patchbot | patch 454174 - swift - Container drive error results double space usage o... | 07:37 |
| clayg | i did notice that, and prefer the seperate test; I've seen that desirable expressiveness in unitests refered to as DAMP "Descriptive And Meaningful Phrases" it's part of a spectrum and different from DRY | 07:37 |
| clayg | .... but ... | 07:38 |
| clayg | that's nothing | 07:38 |
| clayg | it's fine | 07:38 |
| PavelK | ok, it can be separate test. what about probe tests? | 07:38 |
| clayg | if you fix two things your commit message should say fixed to things | 07:38 |
| clayg | " * drive-by cleanup unrelated orthoganal bug when there's too few devices" | 07:38 |
| PavelK | I have not kota's probe tests without errors... | 07:39 |
| clayg | or seperate change/test is fine | 07:39 |
| clayg | the probe test i think came from matt and... it's not really robust | 07:39 |
| PavelK | right, from Matt... | 07:39 |
| clayg | we acctually can not / should not land patch 454174 with the probetest | 07:39 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 07:39 |
| clayg | it just requires more work - it's a good idea - but it'll take some more hammering on - which would slow us down | 07:40 |
| clayg | so it gets cut | 07:40 |
| clayg | CUT CUT CUT | 07:40 |
| PavelK | clayg: fine | 07:40 |
| clayg | lol - i'm truely sorry | 07:40 |
| clayg | honestly tho for me - stopiteration and probetests aside, after the stop the bleeding change lands - i want to figure out the cleanup/remediation change? | 07:42 |
| clayg | AFAIK if you're over replicated it seems to sit there until disks are remounted? Does that sound right? | 07:42 |
| clayg | PavelK: also it's fine on these patches to say "here's a thing it fixes a b and c - you want im in seperate changes - have at it" | 07:43 |
| clayg | it's not rude. helpful. we'd be happy to help with grunt work pushing around text in gerrit and git | 07:44 |
| PavelK | clayg: when overreplicated - you mean when there is only 1 primary unmounted but there are more then one handoff replica? | 07:46 |
| clayg | yes, from the state even if the change - in my test it did not go back to 3 replicas? | 07:46 |
| clayg | ... until i remounted the primary ... | 07:46 |
| clayg | if you think it should I can retest | 07:47 |
| clayg | acctually i sort of plan to regardless - but felt like landing the first thing should be first | 07:48 |
| *** cshastri has joined #openstack-swift | 07:50 | |
| PavelK | clayg: The right solution deppends on failure behaiour. If you expect that next primary fail or is flapping then it may be useful that the next replica has an old DB prepared to be just updated. | 07:52 |
| PavelK | But when ops soves flapping drive and we do not expect early failure then it is better to clean that old extra replica | 07:54 |
| acoles | good morning | 07:56 |
| charz | acoles: good morning! | 07:56 |
| acoles | charz: o/ | 07:56 |
| m_kazuhiro | acoles: good morning! | 07:58 |
| PavelK | clayg: So I preffer to clean the old replica (the extra handoff is not synced) as mattoliverau noticed in patch 448480 | 07:59 |
| patchbot | https://review.openstack.org/#/c/448480/ - swift - Container drive error results double space usage o... | 07:59 |
| openstackgerrit | junboli proposed openstack/swift master: Fixed get ring name from recon cli https://review.openstack.org/448449 | 08:00 |
| kota_ | acoles: o/ | 08:02 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift master: Container drive error results double space usage on rest drives https://review.openstack.org/454174 | 08:05 |
| clayg | acoles: i feel like I just saw you? no way you're back | 08:06 |
| acoles | clayg: no way you're not gone?!? ;) | 08:06 |
| acoles | apparently I slept | 08:06 |
| PavelK | clayg or kota_: I hope that patch 454174 is correct now. Probe tests will be finished later with cleanup (I would like to finish it later), so there are only unit tests. Is it correct to have one patch or StopIteration exception should be separate patch? | 08:09 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 08:09 |
| clayg | PavelK: you can N things in one patch - but you should enumerate all of them in the commit message | 08:12 |
| clayg | now that I'm *aware* of all the things - if I see something missing I can jus fix the commit message without having to ask | 08:13 |
| clayg | "why is this here - it doesn't seem related to commit message and only test that fails witout it is one new test you added" | 08:13 |
| clayg | ah - i can see you are way ahead of me | 08:14 |
| clayg | PavelK: i feel like we're turning you into alien - before long you won't even notice how weird it is | 08:14 |
| PavelK | kota_: sorry, I missed your followup patch so I used your unit test today. But I feel that when my patch breaks it then it should be fixed in one | 08:17 |
| PavelK | clayg: please if you don't understand my commit message or you see my bad English, correct me... | 08:19 |
| clayg | PavelK: commit looks great, re-reading your awesome bug report I realized you explained how the db over-replication increases one db every cycle | 08:19 |
| openstackgerrit | junboli proposed openstack/swift master: Fixed get ring name from recon cli https://review.openstack.org/448449 | 08:23 |
| PavelK | THANKS | 08:23 |
| kota_ | PavelK: nice work, I'll circle back it again. | 08:23 |
| clayg | PavelK: ZOMG - *and* seperate tests!? you're just showing off | 08:24 |
| *** oshritf__ has joined #openstack-swift | 08:29 | |
| clayg | PavelK: iirc there was at least one just little "cleanup" from the first patch that looked pretty good - can I help you in anyway with the gerrit tooling for doing a change set that's based on another change set? | 08:30 |
| kota_ | hmm... i noticed we may want one more assertion at test_replicate_object_with_exception because it doesn't fail with the change | 08:32 |
| clayg | doh! | 08:33 |
| clayg | kota_: good catch! | 08:33 |
| kota_ | i think.. somthing like, replication from replicator.ring.devs[-1]? | 08:33 |
| clayg | but... over all? are you positive or negative on the change? follow up? | 08:34 |
| clayg | :P | 08:34 |
| kota_ | IIUC, the patch looks good for over all, if the assertion works well as i expected | 08:34 |
| kota_ | i think, it's ok to fix the stop iteration too in a patch. that is not so big one. | 08:35 |
| *** oshritf__ has quit IRC | 08:36 | |
| clayg | tfw you have your saio in *just* the right failure mode - then accidently ctrl-r nose a probetest :'( | 08:36 |
| kota_ | clayg: sorry, what's "tfw"? | 08:37 |
| clayg | "that feel when" | 08:37 |
| clayg | or sometimes "that face when" | 08:37 |
| clayg | it's a statement of empathy with the human^Wswift-developer condition | 08:37 |
| *** oshritf__ has joined #openstack-swift | 08:39 | |
| *** zhurong has quit IRC | 08:41 | |
| kota_ | thanks for the explanation ;-) | 08:43 |
| kota_ | PavelK: https://gist.github.com/bloodeagle40234/213df5b2ee8567958fe745172d609875 will fail w/o the change but succeed w/ the change | 08:45 |
| * kota_ is realizing we want more devices the fake ring to fail as I expected at first but it seems one of cleanup things | 08:46 | |
| * kota_ is thinking and it's worth if we trace device id feeded to the _repl_to_node to assert the state | 08:47 | |
| kota_ | but IMO, land it fist as soon as we can. | 08:48 |
| *** kirill_ has joined #openstack-swift | 08:53 | |
| kirill_ | hi, guys! I am trying to figure out how to DELETE account with swiftclient.service.SwiftService. the .delete() will remove all child containers, but not the account itself. | 08:55 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift master: Container drive error results double space usage on rest drives https://review.openstack.org/454174 | 08:55 |
| PavelK | kota_: thanks for test | 08:55 |
| clayg | PavelK: was there another bug with node syncing to itself | 09:14 |
| clayg | PavelK: ahah! if is_handoff_node and node['id'] == node_id: | 09:27 |
| *** oshritf__ has quit IRC | 09:27 | |
| clayg | in my working tree that is if node == local_dev: | 09:28 |
| *** oshritf__ has joined #openstack-swift | 09:29 | |
| clayg | i need to go to bed, and i'm not gunna fix tests | 09:29 |
| clayg | so i'ma jus leave this here -> https://gist.github.com/clayg/e03558e557097debde1d99d2f5b43cb1 | 09:30 |
| *** sputnik13 has quit IRC | 09:40 | |
| *** sputnik13 has joined #openstack-swift | 09:41 | |
| dewanee_ | clayg, regarding .lock : do I expect to see them never released? | 09:42 |
| dewanee_ | I mean I have .lock files that have all the same date: when the cluster was started | 09:43 |
| dewanee_ | almost a year ago | 09:43 |
| kota_ | clayg: please take a rest | 09:43 |
| kota_ | clayg, PavelK: i may be misunderstanding the patch, I was trying to trace the node id feeded to repl_to_node with https://gist.github.com/bloodeagle40234/34cc09f70b299f97ef4f8b684d6e9bc7 | 09:44 |
| kota_ | clayg, PavelK: but the case that the first handoff tries to replicate looks odd for now, it looks like it was trying to repl to itself. | 09:45 |
| kota_ | clayg: anyway, take a rest at first and I hope you could continue to work on your Fri, tommorow, it's my today. | 09:45 |
| * kota_ is not sure if it's safe that the replicator tries to replicate to itself | 09:52 | |
| PavelK | clayg: if node == local_dev AND is_handoff_node because you sometimes need to move DB to another path | 09:53 |
| *** zhurong has joined #openstack-swift | 09:54 | |
| *** zhurong has quit IRC | 09:55 | |
| acoles | dewanee_: the lock files are created first time an object is put in a partition, they are used for taking flock when needed, the existence of the .lock file does not necessarily mean the lock is taken | 10:01 |
| acoles | dewanee_: so expect to see them | 10:02 |
| PavelK | kota_: primary nodes tries replicate with self if path should be changed. Handoffs replicates to self... This is not critical and is fixed in 448480 (and will be ready to review later). | 10:02 |
| PavelK | Is it safe? IDK | 10:04 |
| kota_ | PavelK: a primary doesn't try to sync to itself, IIRC even in the current master | 10:06 |
| PavelK | There is a special case with replica count = 1 where primary node replicates with self - so I believe when handoff replicates with self then nothing bad happens | 10:08 |
| *** jaosorior has quit IRC | 10:11 | |
| *** silor has joined #openstack-swift | 10:12 | |
| PavelK | kota_: you are right. If you have replica count 2 and reballance then when DB path should be changed then it sync with the second primary and deletes local copy. Isn't it a bug? | 10:15 |
| *** jaosorior has joined #openstack-swift | 10:15 | |
| *** links has quit IRC | 10:16 | |
| *** foutatoro has joined #openstack-swift | 10:16 | |
| kota_ | PavelK: sorry, I couldn't get it | 10:17 |
| *** silor1 has joined #openstack-swift | 10:17 | |
| *** silor has quit IRC | 10:18 | |
| *** silor1 is now known as silor | 10:18 | |
| *** m_kazuhiro has quit IRC | 10:20 | |
| *** gkadam-afk is now known as gkadam | 10:24 | |
| kota_ | PavelK: I think I need to lean more about the special case | 10:24 |
| PavelK | kota_: a variable `shouldbehere` become True if path is changed. When one primary node replicates successfully to other primaries then it deletes local copy. Then you have missing 1 replica until another primary pushes DB to this primary. But this is another case that can be fixed later. | 10:24 |
| PavelK | kota_: you are right - learn more or do not replicate with self. I'll skip it. | 10:25 |
| kota_ | ah - to use strict word, when *partition* was changed | 10:26 |
| kota_ | is that? | 10:26 |
| kota_ | no? | 10:27 |
| kota_ | looking the code... | 10:27 |
| *** JimCheung has joined #openstack-swift | 10:29 | |
| kota_ | PavelK: so sounds like, IF the partition changed, _repl_to_node sounds safe because the path was changed | 10:29 |
| kota_ | and IF the partition NOT changed, idk if it's safe.... | 10:30 |
| *** tesseract has quit IRC | 10:30 | |
| *** links has joined #openstack-swift | 10:32 | |
| PavelK | if patition is not changed then there "should" be no change.. But I'm afraid of DB update... | 10:34 |
| *** JimCheung has quit IRC | 10:34 | |
| kota_ | PavelK: yup, i'm with you | 10:35 |
| kota_ | so the crazy guilty is "shouldbehere" has 2 meanings | 10:35 |
| kota_ | a) this is at handoff b) this is at primary node but another partition | 10:36 |
| kota_ | i need someone to help me to know why the case 'db can exists in wrong partition' can occur... | 10:37 |
| kota_ | hmmm.... git blame shows much older bug fix https://github.com/openstack/swift/commit/fef2afd927159af00b6e0c44310b0c7227ed5cc3 :( | 10:40 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift master: Container drive error results double space usage on rest drives https://review.openstack.org/454174 | 10:41 |
| PavelK | kota_: I added "Handoff device is not replicated with self." fix to the patch. But Clay will be sad because there is not extra unit test... | 10:42 |
| PavelK | kota_: I belive that db is in bad path after reballance only... | 10:43 |
| *** oshritf__ has quit IRC | 10:43 | |
| *** tesseract has joined #openstack-swift | 10:44 | |
| kota_ | alright, the change looks fine, probably my test https://gist.github.com/bloodeagle40234/34cc09f70b299f97ef4f8b684d6e9bc7 will pass | 10:45 |
| kota_ | with a chane https://gist.github.com/bloodeagle40234/34cc09f70b299f97ef4f8b684d6e9bc7#file-tracing_repl_to_node-diff-L45 to [1,2,3] | 10:45 |
| kota_ | right? | 10:45 |
| kota_ | s/chane/change/ | 10:45 |
| kota_ | let's do this | 10:46 |
| kota_ | nice, it works | 10:48 |
| *** cbartz has quit IRC | 10:49 | |
| *** jaosorior has quit IRC | 10:53 | |
| *** jaosorior has joined #openstack-swift | 10:53 | |
| *** jaosorior has quit IRC | 10:53 | |
| *** jaosorior has joined #openstack-swift | 10:56 | |
| *** cshastri has quit IRC | 10:56 | |
| *** cbartz has joined #openstack-swift | 10:56 | |
| *** cshastri has joined #openstack-swift | 10:57 | |
| PavelK | kota_: do you see unit tests in patch 454174 enaught? or is something missing? | 10:58 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 10:58 |
| kota_ | patch set 6? | 10:59 |
| kota_ | I don't think you should merge my test into yours because I changed a bunch of your tests | 10:59 |
| *** jaosorior_ has joined #openstack-swift | 11:00 | |
| kota_ | mainly, your test only asserted the call counts but mine tries to assert the node id which called via _repl_to_node | 11:00 |
| kota_ | I think mine is an improvement of yours | 11:00 |
| *** jaosorior has quit IRC | 11:01 | |
| kota_ | and I'll keeps mine at gist and then publish my comment at the gerrit for other reviewrs 'hey you can check the sanity here' | 11:01 |
| PavelK | I understand that it is not necessary to test replication order. I'm asking if there is something missing or I can leave (for today) the patch for reviewers | 11:02 |
| kota_ | perhaps, it's enough and then review as separeted, or ... we can squah if someone thought "do it now" | 11:02 |
| kota_ | imo, it should not be now | 11:03 |
| kota_ | we are changing a lot of things so quickly in the patch so keep it simple to review again easy for other reviewers. just IMHO. | 11:04 |
| *** links has quit IRC | 11:04 | |
| *** links has joined #openstack-swift | 11:05 | |
| PavelK | kota_: OK | 11:05 |
| kota_ | ok, push my comment to the gerrit. I think that can land as it is but sorry, i still need to know about my concern. | 11:11 |
| openstackgerrit | junboli proposed openstack/swift master: Fixed get ring name from recon cli https://review.openstack.org/448449 | 11:12 |
| kota_ | PavelK: and I'm running out of time today, have to leave my office | 11:12 |
| kota_ | PavelK: I'm thinking that change is my priority so that I'll look at earlier in the next week | 11:13 |
| * kota_ is leaving his office | 11:16 | |
| * kota_ and offline | 11:16 | |
| *** openstackgerrit has quit IRC | 11:18 | |
| *** silor1 has joined #openstack-swift | 11:19 | |
| *** silor has quit IRC | 11:20 | |
| *** silor1 is now known as silor | 11:20 | |
| *** pcaruana has quit IRC | 11:22 | |
| *** gkadam has quit IRC | 11:30 | |
| *** jaosorior_ is now known as jaosorior | 11:37 | |
| *** cshastri_ has joined #openstack-swift | 11:50 | |
| *** cshastri has quit IRC | 11:52 | |
| *** silor1 has joined #openstack-swift | 12:08 | |
| *** jaosorior has quit IRC | 12:09 | |
| *** jaosorior has joined #openstack-swift | 12:09 | |
| *** silor has quit IRC | 12:12 | |
| *** silor1 is now known as silor | 12:12 | |
| *** cshastri_ is now known as cshastri | 12:16 | |
| *** gkadam has joined #openstack-swift | 12:24 | |
| *** gkadam has quit IRC | 12:44 | |
| *** kei_yama has quit IRC | 12:52 | |
| *** PavelK has quit IRC | 12:53 | |
| *** dja has joined #openstack-swift | 12:57 | |
| *** catintheroof has joined #openstack-swift | 13:03 | |
| *** openstackgerrit has joined #openstack-swift | 13:05 | |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift master: Container drive error results double space usage on rest drives https://review.openstack.org/454174 | 13:05 |
| openstackgerrit | Christopher Bartz proposed openstack/swift master: Allow DLO PUT to prefix-based tempurls https://review.openstack.org/454716 | 13:14 |
| *** winggundamth has quit IRC | 13:18 | |
| *** _JZ_ has joined #openstack-swift | 13:22 | |
| *** chlong has joined #openstack-swift | 13:33 | |
| *** SkyRocknRoll has joined #openstack-swift | 13:46 | |
| *** zhurong has joined #openstack-swift | 13:52 | |
| *** zhurong has quit IRC | 13:53 | |
| *** xinli has joined #openstack-swift | 13:58 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 14:12 | |
| *** SkyRocknRoll has quit IRC | 14:14 | |
| *** amoralej is now known as amoralej|lunch | 14:20 | |
| *** links has quit IRC | 14:25 | |
| *** cshastri has quit IRC | 14:33 | |
| *** Dinesh_Bhor has quit IRC | 14:34 | |
| *** stradling has joined #openstack-swift | 14:37 | |
| *** amoralej|lunch is now known as amoralej | 14:50 | |
| *** klrmn has joined #openstack-swift | 14:53 | |
| *** winggundamth has joined #openstack-swift | 14:59 | |
| *** winggundamth has quit IRC | 15:04 | |
| *** rcernin|wfh has quit IRC | 15:04 | |
| *** kirill_ has quit IRC | 15:06 | |
| *** catintheroof has quit IRC | 15:06 | |
| *** catintheroof has joined #openstack-swift | 15:06 | |
| *** catintheroof has quit IRC | 15:07 | |
| *** foutatoro has quit IRC | 15:10 | |
| *** catintheroof has joined #openstack-swift | 15:15 | |
| *** catintheroof has quit IRC | 15:16 | |
| *** catintheroof has joined #openstack-swift | 15:26 | |
| *** catintheroof has quit IRC | 15:27 | |
| *** cbartz has quit IRC | 15:30 | |
| notmyname | good morning | 15:37 |
| *** chsc has joined #openstack-swift | 15:40 | |
| *** chsc has joined #openstack-swift | 15:40 | |
| *** jaosorior has quit IRC | 15:44 | |
| *** silor has quit IRC | 15:54 | |
| notmyname | I just noticed that topics on http://forumtopics.openstack.org have been updated. based on comments left there, it would seem that we've got 3 topics approved (ops feedback session, small-file optimization, container sharding), and the others are not currently approved (but not rejected?) | 15:56 |
| *** silor has joined #openstack-swift | 16:11 | |
| *** ChubYann has joined #openstack-swift | 16:11 | |
| *** JimCheung has joined #openstack-swift | 16:11 | |
| *** ediardo has joined #openstack-swift | 16:33 | |
| *** winggundamth has joined #openstack-swift | 16:35 | |
| *** tesseract has quit IRC | 16:42 | |
| *** hseipp has quit IRC | 16:44 | |
| *** winggundamth has quit IRC | 16:47 | |
| *** catintheroof has joined #openstack-swift | 16:51 | |
| *** catintheroof has quit IRC | 16:57 | |
| *** sgundur has joined #openstack-swift | 17:19 | |
| *** xinli has quit IRC | 17:21 | |
| *** amoralej is now known as amoralej|off | 17:21 | |
| *** Renich has joined #openstack-swift | 17:28 | |
| *** dja has quit IRC | 17:44 | |
| *** chlong_ has joined #openstack-swift | 17:46 | |
| *** Sukhdev_ has joined #openstack-swift | 17:54 | |
| *** geaaru has quit IRC | 17:56 | |
| *** mvk has quit IRC | 18:08 | |
| *** zaitcev has quit IRC | 18:17 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Fix UnicodeDecodeError in reconstructor _full_path function https://review.openstack.org/452750 | 18:23 |
| *** neonpastor has quit IRC | 18:28 | |
| *** xinli has joined #openstack-swift | 18:31 | |
| *** silor has quit IRC | 18:34 | |
| *** silor has joined #openstack-swift | 18:41 | |
| acoles | notmyname: I am wondering if bug 1679175 should be critical? - the new probe tests in this patch ^^ show that the reconstructor can hang on master :/ | 18:47 |
| openstack | bug 1679175 in OpenStack Object Storage (swift) "Reconstructor raises UnicodeDecodeError when reverting an obj with non-ascii chars in path" [High,In progress] https://launchpad.net/bugs/1679175 - Assigned to Alistair Coles (alistair-coles) | 18:47 |
| openstackgerrit | John Dickinson proposed openstack/python-swiftclient master: respect bulk delete page size and fix logic error https://review.openstack.org/454849 | 18:48 |
| notmyname | acoles: sounds likely | 18:48 |
| * notmyname needs to step outside for a bit | 18:49 | |
| *** silor has quit IRC | 18:51 | |
| *** openstackgerrit has quit IRC | 19:19 | |
| *** openstackgerrit has joined #openstack-swift | 19:23 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Fix encoding issue in ssync_sender.send_put() https://review.openstack.org/452112 | 19:23 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Make diskfile persist all metadata as encoded utf8 https://review.openstack.org/454860 | 19:23 |
| acoles | rledisez: please let me know your thoughts on squashing this ^^ into your patch (also note that I updated the bug report - it's not the obj name that triggers the UnicodeDecodeError AFAICT) | 19:27 |
| *** chlong_ has quit IRC | 19:37 | |
| *** chlong has quit IRC | 19:37 | |
| *** xinli has quit IRC | 19:40 | |
| *** chlong has joined #openstack-swift | 19:42 | |
| *** zaitcev has joined #openstack-swift | 19:53 | |
| *** ChanServ sets mode: +v zaitcev | 19:53 | |
| * notmyname back for a while | 20:09 | |
| notmyname | if you're like me and like the never-ending quest of finding the perfect font for coding, you might like http://sourcefoundry.org/hack/ | 20:30 |
| *** oshritf has joined #openstack-swift | 20:33 | |
| *** _JZ_ has quit IRC | 20:34 | |
| *** stradling has quit IRC | 20:48 | |
| *** oshritf has quit IRC | 20:50 | |
| *** mvk has joined #openstack-swift | 21:04 | |
| *** chlong has quit IRC | 21:08 | |
| clayg | "But Clay will be sad because there is not extra unit test | 21:11 |
| *** catintheroof has joined #openstack-swift | 21:12 | |
| clayg | " <- YUP | 21:12 |
| clayg | gah, ok i picked up from the logs you were aware that you change would cause handoff to not replicate to itself and you see this as future work | 21:15 |
| clayg | shit man, i just blew an hour | 21:15 |
| *** catintheroof has quit IRC | 21:16 | |
| clayg | I hope Pavel is a really patient guy or something on pach 454174 | 21:42 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Fix default FakeRing max_more_nodes https://review.openstack.org/454898 | 21:52 |
| clayg | kota_: fyi patch 454898 | 21:52 |
| patchbot | https://review.openstack.org/#/c/454898/ - swift - Fix default FakeRing max_more_nodes | 21:52 |
| *** amoralej|off has quit IRC | 21:55 | |
| *** tongli has joined #openstack-swift | 22:34 | |
| *** tongli has quit IRC | 22:45 | |
| *** Renich___ has joined #openstack-swift | 23:19 | |
| *** Renich has quit IRC | 23:22 | |
| kota_ | clayg: thx | 23:27 |
| kota_ | It looks like you found something | 23:27 |
| *** Renich___ has quit IRC | 23:31 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Follow up tests for get_hashes regression https://review.openstack.org/454920 | 23:40 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!