| *** tqtran has quit IRC | 00:04 | |
| clayg | notmyname: it's a common problem - most people are not in australia | 00:04 |
|---|---|---|
| mattoliverau | well with Trump, I suspect that to change somewhat, at least with regards to Americans :P | 00:11 |
| *** vint_bra has quit IRC | 00:12 | |
| *** kei_yama has quit IRC | 00:30 | |
| *** kei_yama has joined #openstack-swift | 00:32 | |
| notmyname | http://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html | 00:34 |
| timburke | notmyname: idk, indexing doesn't sound so "simple" to me :P | 00:48 |
| notmyname | well, like most of their API, it's pretty complicated (ie hard to type in curl) | 00:50 |
| notmyname | I'm curious about the implementation | 00:50 |
| notmyname | is it simply the first bytes that return on an key read, and they filter once they have those? | 00:51 |
| notmyname | are they building a per-user index of stuff that they're querying before the read? | 00:51 |
| notmyname | the eventual consistency callout implies the second | 00:51 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Remove ring md5 integration check from recon unittests https://review.openstack.org/404500 | 00:51 |
| notmyname | or maybe they've got some per-server or per-drive fast cache where they store the tags for the keys on that drive. but it might have some async component, thus giving some possible consistency windows | 00:52 |
| notmyname | like our async pendings | 00:52 |
| notmyname | implying that extra value in s3 isn't as much clever programming as it is clever hardware selection (+ volume purchasing) allowing them a more time-budget per request to do interesting things like this | 00:53 |
| notmyname | extra value == new exciting features | 00:54 |
| openstackgerrit | Matthew Oliver proposed openstack/swift: Quarantine malformed database schema SQLite errors https://review.openstack.org/405031 | 00:54 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Remove ring md5 integration check from recon unittests https://review.openstack.org/404500 | 00:55 |
| *** nikivi has joined #openstack-swift | 00:55 | |
| notmyname | mattoliverau: thanks for picking up that bug! | 00:55 |
| clayg | i hate changes that are only refactoring :'( | 00:56 |
| mattoliverau | notmyname: nps. Am debugging and watching containers shard.. and so while I was in the db broker anyway thought I might as well fix it. | 00:57 |
| notmyname | perfect :-) | 00:57 |
| *** dmorita has quit IRC | 00:58 | |
| *** catintheroof has joined #openstack-swift | 01:05 | |
| *** kei_yama has quit IRC | 01:05 | |
| *** kei_yama has joined #openstack-swift | 01:06 | |
| clayg | notmyname: why would you assume they're doing anything different that you would do? something in process and consistency troller to put metadata in an index (yeah probably shared per user) | 01:07 |
| clayg | the integration is sweet tho -> That is, you cannot create a policy to grant or deny a user permissions to delete or override an existing object based on its existing tags. | 01:07 |
| *** catintheroof has quit IRC | 01:08 | |
| notmyname | clayg: well, I'm not exactly sure how *I'd* do it :-) | 01:11 |
| notmyname | probably one of those ways | 01:11 |
| * mattoliverau is going to lunch | 01:13 | |
| mattoliverau | well really Bec and Lucy are at the beach playing in the sand.. so I'm going to take a break and have a beach walk, play with the daughter and find something to eat. be back in about an hour | 01:13 |
| mattoliverau | its a hard life :P | 01:13 |
| clayg | notmyname: i probably don't understand some nuanced part of what's confusing you - to mee it looks like glossed up and scaled out version of exactly what timur is doing with elastic search (like add in something that *attempts* to do it on ingest i.e. middleware + some api integration with index queries vs. direct access to the indexes) | 01:15 |
| clayg | oh.. and they probably use some cool internal Lucene based index that's all geo'd up in their internal business and multi-tenant so they can have seperate indexes for each account (or maybe per bucket, and then you make at most 100 queries per account if you need to do cross bucket stuff?) | 01:16 |
| *** dmorita has joined #openstack-swift | 01:18 | |
| notmyname | yeah, that seems like what timur is doing and what we've talked about overall in the community. it's a reasonable way to do it | 01:18 |
| notmyname | but a different way, only reasonable if you have control over the hardware, would be to have something more directly integrated into the actual servers themselves instead of a different external service | 01:19 |
| *** nikivi has quit IRC | 01:19 | |
| notmyname | like if their DCs have Awesome(tm) networks and/or protocols --meaning new requests aren't expensive -- then putting an NVMe card in each storage server to index the tags on the keys on that server might be simpler | 01:20 |
| *** dmorita has quit IRC | 01:21 | |
| notmyname | eg if it's not an expensive operation to have an api server make a request to a storage server and check the tags against some policy | 01:21 |
| *** dmorita has joined #openstack-swift | 01:21 | |
| notmyname | alternatively, i could have just been playing with hardware too much lately, and that's my current hammer ;-) | 01:22 |
| *** dmorita has quit IRC | 01:24 | |
| *** dmorita has joined #openstack-swift | 01:24 | |
| notmyname | oh, and timur's stuff is and will be totally awesome | 01:25 |
| clayg | notmyname: but the tags are an aggregation - you can split that up across every storage server in s3? well maybe i don't know the science behind distributed indexes - seems like there'd be a diminishing returns spreading it around if you'd expect only 1/1K servers to have a hit for given key | 01:25 |
| *** dmorita has quit IRC | 01:25 | |
| *** dmorita has joined #openstack-swift | 01:26 | |
| notmyname | what do you mean by "tags are an aggregation"? they're stored per key right? | 01:29 |
| notmyname | (AWS key == swift object) | 01:29 |
| notmyname | oh, they use "object" in their doc. i'll keep using that | 01:29 |
| notmyname | what do you mean by "tags are an aggregation"? they're stored per object right? | 01:29 |
| notmyname | hmm..it does look like the bucket listings could be filtered by tag. well, they imply that but don't give much info | 01:31 |
| notmyname | otherwise it seems like a very object-centric design | 01:32 |
| notmyname | I don't see anything that is "show me all my objects that are color=blue". just modifications on requests that also include the actual object name itself | 01:32 |
| notmyname | oh, wait. not even the lifecycle stuff is aggregate. "lifecycle management in which you can specify tag-based filter, in addition to key name prefix" | 01:34 |
| *** jamielennox is now known as jamielennox|away | 01:34 | |
| notmyname | oh, but the next line is "list objects in your bucket by tag" | 01:34 |
| notmyname | hmm | 01:34 |
| *** isotope has quit IRC | 01:35 | |
| openstackgerrit | Merged openstack/python-swiftclient: Fix 'url' to 'URL' https://review.openstack.org/379379 | 01:38 |
| openstackgerrit | Merged openstack/swift: Multi Swift - Multiple Swift clusters on same h/w https://review.openstack.org/393794 | 01:38 |
| openstackgerrit | Merged openstack/swift: Show team and repo badges on README https://review.openstack.org/402856 | 01:39 |
| clayg | mattoliverau: you know I've never really noticed the corrupted and malformed db's in our test suite before - and right there under test/unit/common/ too!? | 01:40 |
| kota_ | yeah, exactly, http://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html says that we can filter but... i cannot find the api in http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html | 01:40 |
| kota_ | notmyname:^^ | 01:40 |
| kota_ | that *will* be available in the future (like v3) api? or sort of hidden parameter? | 01:41 |
| clayg | yeah I'm sure they store the keys on the object the same way they do any object metadata | 01:42 |
| clayg | also note that to change a tag on an object you have to do a server side copy ;) | 01:42 |
| *** m_kazuhiro has joined #openstack-swift | 01:43 | |
| *** dmorita has quit IRC | 01:50 | |
| openstackgerrit | Merged openstack/python-swiftclient: Show team and repo badges on README https://review.openstack.org/402853 | 01:56 |
| *** adu has joined #openstack-swift | 01:59 | |
| *** kei_yama has quit IRC | 02:03 | |
| *** m_kazuhiro has quit IRC | 02:06 | |
| *** kei_yama has joined #openstack-swift | 02:10 | |
| *** klrmn has quit IRC | 02:11 | |
| *** adu has quit IRC | 02:12 | |
| *** wanghua has joined #openstack-swift | 02:22 | |
| *** catintheroof has joined #openstack-swift | 02:24 | |
| *** supamatt has joined #openstack-swift | 02:36 | |
| supamatt | question on swift | 02:37 |
| *** bkopilov has quit IRC | 02:38 | |
| *** clu_ has quit IRC | 02:38 | |
| mattoliverau | clayg: I built a db with a malformed schema.. secret is to build one, the turn on editable schma PRAGMA, then exit the scema via an SQL update to the sqlite_master table. Will attach it to the patch, but look at adding it to a test in my patch as well. | 02:50 |
| mattoliverau | s/exit/edit/ | 02:50 |
| mattoliverau | whats up supamatt | 02:50 |
| *** jamielennox|away is now known as jamielennox | 02:54 | |
| supamatt | mattoliverau: If a failure to insert is complete (all replicas failed) and a 503 is returned, will the key be present or absent in the container listing? | 02:54 |
| supamatt | swift object key that is | 03:03 |
| mattoliverau | supamatt: if they all failed then the update to the container server wouldn't have been fired off. However swift wont return success unless it get quorum from the storage nodes. So there is a chance that 1 succeeded but still caused a failure, if one make it, it would trigger a container update so the objedct might appear in the list.. and if nothing happens to the < quorum replicas that made it, they could get | 03:06 |
| mattoliverau | replicated around and actually exist in the cluster. | 03:06 |
| supamatt | So yes, it's possible then. | 03:07 |
| mattoliverau | yes, when thinking the logic through in my head ;) | 03:08 |
| supamatt | That's baddd maybe? | 03:09 |
| supamatt | depends on the data you have in there | 03:09 |
| *** kei_yama has quit IRC | 03:11 | |
| *** kei_yama has joined #openstack-swift | 03:12 | |
| mattoliverau | well its good if the data you wanted to put in, actually goes in :P. Tho I might have to go look at the code to see if what I say is actually true ;) | 03:13 |
| *** jamielennox is now known as jamielennox|away | 03:17 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift: Quarantine malformed database schema SQLite errors https://review.openstack.org/405031 | 03:17 |
| mattoliverau | clayg: ^^ now with a malformed schema db for your testing pleasure | 03:17 |
| *** jamielennox|away is now known as jamielennox | 03:26 | |
| *** catintheroof has quit IRC | 03:28 | |
| *** catintheroof has joined #openstack-swift | 03:28 | |
| *** catintheroof has quit IRC | 03:28 | |
| *** catintheroof has joined #openstack-swift | 03:30 | |
| *** catintheroof has quit IRC | 03:30 | |
| *** catintheroof has joined #openstack-swift | 03:31 | |
| *** tqtran has joined #openstack-swift | 03:31 | |
| *** catintheroof has quit IRC | 03:31 | |
| *** tqtran has quit IRC | 03:37 | |
| *** jamielennox is now known as jamielennox|away | 03:47 | |
| *** psachin has joined #openstack-swift | 03:47 | |
| notmyname | supamatt: are you asking about the object write? if that fails on all replicas and 503 is given to the client, what happens? | 03:50 |
| notmyname | or what? where's the failure? | 03:50 |
| mattoliverau | turns out I was wrong, it wont happen because we're (swift) is smart and checks for minimum puts (i remember the reorganising patch now that I'm looking at the code) at each chuck and before the final commit. | 03:53 |
| mattoliverau | notmyname: turns out is was a customer questiion, and supamatt's a racker. They we're worried about whether a container update could happen if the object servers are too busy to put the obj. | 03:55 |
| notmyname | ah | 03:55 |
| notmyname | nope | 03:55 |
| *** bkopilov has joined #openstack-swift | 04:02 | |
| *** thurloat has quit IRC | 04:04 | |
| *** PsionTheory has joined #openstack-swift | 04:14 | |
| *** thurloat has joined #openstack-swift | 04:15 | |
| *** deep_1 has joined #openstack-swift | 04:22 | |
| *** jamielennox|away is now known as jamielennox | 04:33 | |
| *** PsionTheory has quit IRC | 04:49 | |
| *** xionchen_ has quit IRC | 04:54 | |
| *** links has joined #openstack-swift | 05:00 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:07 | |
| *** tqtran has joined #openstack-swift | 05:34 | |
| *** qwertyco has joined #openstack-swift | 05:38 | |
| *** qwertyco has quit IRC | 05:38 | |
| *** qwertyco has joined #openstack-swift | 05:39 | |
| *** tqtran has quit IRC | 05:39 | |
| *** m_kazuhiro has joined #openstack-swift | 06:03 | |
| *** ppai has joined #openstack-swift | 06:12 | |
| *** dmorita has joined #openstack-swift | 06:18 | |
| *** dmorita has quit IRC | 06:22 | |
| *** m_kazuhiro has quit IRC | 06:34 | |
| *** ChubYann has quit IRC | 06:36 | |
| *** tqtran has joined #openstack-swift | 06:36 | |
| *** tqtran has quit IRC | 06:41 | |
| *** _JZ_ has quit IRC | 06:44 | |
| cschwede | just stumbled upon missing recon metrics in http://docs.openstack.org/developer/swift/admin_guide.html - and remembered i was already looking at them | 06:52 |
| cschwede | and indeed, there’s a fix for that! patch 343321 is an easy review ;) | 06:52 |
| patchbot | https://review.openstack.org/#/c/343321/ - swift - Add missing recon metrics to admin_guide | 06:52 |
| *** sams-gleb has joined #openstack-swift | 07:12 | |
| *** sams-gleb has quit IRC | 07:12 | |
| *** sams-gleb has joined #openstack-swift | 07:12 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Safe cleanup listdir for db replicator https://review.openstack.org/405134 | 07:14 |
| clayg | mahatic_: in patch https://review.openstack.org/#/c/331601/7/swift/common/db_replicator.py@116 - os.rmdir(suff_dir) changes the state of the filesystem but not the string values in the list referred to by the variable suffixes | 07:17 |
| patchbot | patch 331601 - swift - remove empty db hash and suffix directories | 07:17 |
| clayg | mattoliverau: that is sick | 07:22 |
| clayg | re: malformed_schema.db | 07:22 |
| clayg | mattoliverau: well replicated objects are auto commited - it only takes one successful 200 from the backend to rebuild the object - so that one server that returned the 200 can do the container update | 07:28 |
| clayg | mattoliverau: on the EC side once the proxy thinks an object is durable ("enough" frags responded) then it sends commit and anyone still alive can send their container updates | 07:29 |
| mahatic_ | clayg: hmm if there are no more suffixes (and it's an empty partition), that's what it's supposed to do right? delete the part dir if it's empty? | 07:29 |
| clayg | ... either way container is always updated after successful object PUT - so the higher risk would be an object PUT successfully and then die before the container update - which is why the conatiner update is distributed | 07:29 |
| mahatic_ | empty part dir* | 07:30 |
| clayg | mahatic_: IME empty part dirs *are* cleaned up... patch 138524 | 07:34 |
| patchbot | https://review.openstack.org/#/c/138524/ | 07:34 |
| *** hseipp has joined #openstack-swift | 07:37 | |
| *** ppai has quit IRC | 07:39 | |
| mahatic_ | clayg: right, and with this change https://review.openstack.org/#/c/331601/7/swift/common/db_replicator.py@99 we might end up again with an empty part dir which is not cleaned up | 07:44 |
| patchbot | patch 331601 - swift - remove empty db hash and suffix directories | 07:44 |
| mahatic_ | clayg: Not sure how possible it is in reality, but in a recursive clean up, we might end up there | 07:45 |
| mahatic_ | clayg: http://paste.openstack.org/show/591062/ -> linie#99 is gonna clean up afb and afc and then move onto 460, leaving empty 450 | 07:55 |
| *** ppai has joined #openstack-swift | 07:55 | |
| mahatic_ | that is what I think would happen | 07:56 |
| *** cschwede has quit IRC | 08:05 | |
| *** tmoreira has quit IRC | 08:07 | |
| *** ppai has quit IRC | 08:07 | |
| clayg | mahatic_: put an empty part dir in one of your container datdirs and run the replicator a couple of times - I think it's cleaned up by line 91 | 08:09 |
| clayg | ... but that's not really my point - I get what your saying about trying to clean it up in the same pass - but I don't feel that your acknowledging to me that you recognize the values of the variable suffix doesn't change with line 99 - the code is not right - maybe that is obvious and the conversation moved on from implementation to behavior - unclear | 08:11 |
| mahatic_ | line 91 will clean it up, I think I was trying getting it cleaned up in one pass replicator | 08:11 |
| mahatic_ | clayg: right! | 08:11 |
| clayg | ok, i don't think it important to catch it in the same pass - but if you wanted to do that you'd need another listdir - value held in suffixes would not reflect cleared out empty suffix dirs at line 115 otherwise | 08:12 |
| mahatic_ | clayg: and correct, my bad, I did not acknowledge that suffix variable doesn't change and that line of code is wrong! | 08:13 |
| mahatic_ | clayg: right, thanks. If it's not that important, one more listdir is not worth it I think | 08:15 |
| *** oshritf has joined #openstack-swift | 08:19 | |
| *** ppai has joined #openstack-swift | 08:19 | |
| *** rledisez has joined #openstack-swift | 08:21 | |
| clayg | yeah i'm fine with it cleaning up as it goes, I think the listdir would mostly come back with a bunch of suffixes - i'd rather save the listdir for the next pass when we would expect to do something with all the suffixes we found | 08:24 |
| clayg | ... or something | 08:24 |
| *** Jeffrey4l_ has quit IRC | 08:26 | |
| *** jcook has quit IRC | 08:28 | |
| mahatic_ | okay | 08:28 |
| *** Jeffrey4l_ has joined #openstack-swift | 08:30 | |
| *** csmart has quit IRC | 08:33 | |
| *** jcook has joined #openstack-swift | 08:34 | |
| *** rcernin has joined #openstack-swift | 08:34 | |
| clayg | mahatic_: please feel free to squash any code from patch 405134 as well if you do another rev | 08:35 |
| *** csmart has joined #openstack-swift | 08:35 | |
| patchbot | https://review.openstack.org/#/c/405134/ - swift - Safe cleanup listdir for db replicator | 08:35 |
| *** Guest66666 has quit IRC | 08:36 | |
| clayg | mahatic_: i dropped the implementation of test_empty_suffix_and_hash_dirs_get_cleanedup that was based on the arcane test_roundrobin_datadirs | 08:36 |
| *** zaitcev has quit IRC | 08:36 | |
| clayg | i just use the file system and assert the behaviors rather than the implementation | 08:37 |
| *** rcernin has quit IRC | 08:37 | |
| *** rcernin has joined #openstack-swift | 08:37 | |
| *** Guest66666 has joined #openstack-swift | 08:38 | |
| *** geaaru has joined #openstack-swift | 08:38 | |
| *** sams-gleb has quit IRC | 08:40 | |
| mahatic_ | clayg: sure. from a glance I think it has neat refactoring, plus it also fixes https://bugs.launchpad.net/swift/+bug/1646362 | 08:40 |
| openstack | Launchpad bug 1646362 in OpenStack Object Storage (swift) "spurious files next to hashdir don't get cleaned up" [Undecided,New] | 08:40 |
| clayg | mahatic_: also feel free to *not* do another rev - I put a +2 on it because I think it fixes the bug and should merge - if there's any cleanup that needs to happen *before* it can merge and fix the bug - I didn't see it | 08:40 |
| *** sams-gleb has joined #openstack-swift | 08:40 | |
| mahatic_ | clayg: I will do that, maybe you could swap the author/co-author on it (I think I'm sure I'll mess it up) | 08:41 |
| clayg | mahatic_: I screwed up on that bug - i should have bugged the db-replicator behavior seperatly from the object-replicator behavior | 08:41 |
| mahatic_ | clayg: ah okay, well that sounds fine too | 08:41 |
| clayg | mahatic_: hell, i probably should have *just* refactoed/cleaned up the test and spurious rmdir calls from your change in the follow up then gone after the hashdir is file fix in another change :'( | 08:42 |
| clayg | so many fucking bugs | 08:42 |
| clayg | mahatic_: as long as we close lp bug #1583719 i'm happy | 08:43 |
| openstack | Launchpad bug 1583719 in OpenStack Object Storage (swift) "empty db hash/suffix directories can pile up" [Medium,In progress] https://launchpad.net/bugs/1583719 - Assigned to Mahati Chamarthy (mahati-chamarthy) | 08:43 |
| clayg | i have some long since rebalanced off parts laying on this old disks in this one cluser - but they damn part dirs won't die because they have some empty suffixes in them because who knows how long ago some db's got quarantined | 08:44 |
| *** amoralej|off is now known as amoralej | 08:44 | |
| mahatic_ | clayg: I'm thinking of a way that maybe could make it easy for you. I could split your follow up if you'd wish that, just that it will be more clear on what it's doing | 08:45 |
| *** sams-gleb has quit IRC | 08:45 | |
| clayg | mahatic_: sounds *amazing* - i'll take it! | 08:45 |
| mahatic_ | clayg: great! | 08:45 |
| clayg | mahatic_: just ping me in irc what you need me to review or add me in gerrit - i'll probably catch it in emails | 08:46 |
| mahatic_ | yup sure | 08:46 |
| clayg | thanks | 08:46 |
| mahatic_ | np, thanks too! | 08:46 |
| clayg | acahcaphcaph cschwede is not in channel!? | 08:47 |
| clayg | wasn't there some critical replication bug fix related to suffix listdirs we still needed to land? | 08:48 |
| clayg | lp bug #1644807 | 08:51 |
| openstack | Launchpad bug 1644807 in OpenStack Object Storage (swift) "hashes.pkl suffix update is not deterministic and might be never triggered" [High,In progress] https://launchpad.net/bugs/1644807 - Assigned to Christian Schwede (cschwede) | 08:51 |
| clayg | s/be never/never be/ - it even *sounds* like cschwede ;) | 08:53 |
| *** sams-gleb has joined #openstack-swift | 08:55 | |
| *** asettle has joined #openstack-swift | 08:55 | |
| *** Guest36 has joined #openstack-swift | 08:59 | |
| *** acoles_ is now known as acoles | 09:00 | |
| *** hseipp has quit IRC | 09:02 | |
| *** ppai has quit IRC | 09:06 | |
| acoles | good morning | 09:06 |
| *** sams-gleb has quit IRC | 09:07 | |
| *** sams-gleb has joined #openstack-swift | 09:08 | |
| *** wer has quit IRC | 09:08 | |
| *** hseipp has joined #openstack-swift | 09:10 | |
| *** sams-gleb has quit IRC | 09:10 | |
| *** sams-gleb has joined #openstack-swift | 09:10 | |
| *** hogepodge has quit IRC | 09:11 | |
| *** sams-gleb has quit IRC | 09:12 | |
| *** sams-gleb has joined #openstack-swift | 09:12 | |
| *** hogepodge has joined #openstack-swift | 09:12 | |
| mahatic_ | acoles: o/ | 09:12 |
| *** wer has joined #openstack-swift | 09:13 | |
| *** sams-gleb has quit IRC | 09:17 | |
| *** ppai has joined #openstack-swift | 09:23 | |
| *** tmoreira has joined #openstack-swift | 09:26 | |
| *** rcernin has quit IRC | 09:29 | |
| *** rcernin has joined #openstack-swift | 09:30 | |
| *** hogepodge has quit IRC | 09:31 | |
| *** hogepodge has joined #openstack-swift | 09:32 | |
| *** wanghua has quit IRC | 09:38 | |
| *** Guest36 has quit IRC | 09:40 | |
| *** gabor_antal has joined #openstack-swift | 09:45 | |
| *** SkyRocknRoll has quit IRC | 09:46 | |
| *** davideagle_ has joined #openstack-swift | 09:53 | |
| *** openstack has joined #openstack-swift | 10:04 | |
| *** rcernin has quit IRC | 10:06 | |
| *** rcernin has joined #openstack-swift | 10:07 | |
| *** ppai has joined #openstack-swift | 10:09 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:16 | |
| *** SkyRocknRoll has quit IRC | 10:20 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:21 | |
| *** psachin has quit IRC | 10:22 | |
| *** SkyRocknRoll has quit IRC | 10:29 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add missing expirer recon metric to admin_guide https://review.openstack.org/405265 | 10:33 |
| *** mvk has quit IRC | 10:33 | |
| *** tmoreira has quit IRC | 10:33 | |
| *** sams-gleb has joined #openstack-swift | 10:33 | |
| *** tmoreira has joined #openstack-swift | 10:34 | |
| *** dmorita has joined #openstack-swift | 10:34 | |
| *** sams-gleb has quit IRC | 10:35 | |
| *** sams-gleb has joined #openstack-swift | 10:35 | |
| *** psachin has joined #openstack-swift | 10:36 | |
| *** aj701_ has joined #openstack-swift | 10:36 | |
| openstackgerrit | Mahati Chamarthy proposed openstack/swift: Safe cleanup listdir for db replicator https://review.openstack.org/405134 | 10:38 |
| *** patchbot has quit IRC | 10:39 | |
| *** tonyb has quit IRC | 10:39 | |
| *** aj701 has quit IRC | 10:39 | |
| *** dmorita has quit IRC | 10:39 | |
| *** sams-gleb has quit IRC | 10:39 | |
| *** tmoreira has quit IRC | 10:42 | |
| *** patchbot has joined #openstack-swift | 10:43 | |
| *** tonyb has joined #openstack-swift | 10:43 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:43 | |
| *** tmoreira has joined #openstack-swift | 10:44 | |
| *** SkyRocknRoll has quit IRC | 10:44 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:45 | |
| *** cschwede has joined #openstack-swift | 10:45 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift: Remove redundant codes from unit test of object updater https://review.openstack.org/405271 | 10:48 |
| *** ChanServ sets mode: +v cschwede | 10:51 | |
| openstackgerrit | Mahati Chamarthy proposed openstack/swift: Cleanup spurious files next to hashdir https://review.openstack.org/405274 | 10:58 |
| mahatic_ | clayg: ^^ there you go 405134 and 405274 | 11:00 |
| *** mvk has joined #openstack-swift | 11:09 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift: Rename ObjectUpdater variables in unit test to appropriate names https://review.openstack.org/405281 | 11:10 |
| *** bapalm has quit IRC | 11:10 | |
| *** bapalm has joined #openstack-swift | 11:15 | |
| *** deep_1 has quit IRC | 11:43 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Cleanup tests from deterministic suffix updates https://review.openstack.org/405295 | 11:56 |
| *** bkopilov has quit IRC | 12:09 | |
| *** catintheroof has joined #openstack-swift | 12:25 | |
| *** deep_1 has joined #openstack-swift | 12:34 | |
| *** SkyRocknRoll has quit IRC | 12:41 | |
| *** catintheroof has quit IRC | 12:48 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Move documented reclaim_age option to correct location https://review.openstack.org/374419 | 12:56 |
| *** vint_bra has joined #openstack-swift | 13:00 | |
| *** jamielennox is now known as jamielennox|away | 13:00 | |
| *** catintheroof has joined #openstack-swift | 13:13 | |
| *** catinthe_ has joined #openstack-swift | 13:18 | |
| *** catintheroof has quit IRC | 13:19 | |
| *** amoralej is now known as amoralej|lunch | 13:21 | |
| *** kei_yama has quit IRC | 13:29 | |
| *** ppai has quit IRC | 13:29 | |
| *** qwertyco has quit IRC | 13:47 | |
| *** dmorita has joined #openstack-swift | 13:52 | |
| *** bkopilov has joined #openstack-swift | 13:52 | |
| *** dmorita has quit IRC | 13:56 | |
| *** jpich has joined #openstack-swift | 14:01 | |
| *** rcernin has quit IRC | 14:10 | |
| *** rcernin has joined #openstack-swift | 14:11 | |
| *** StraubTW has joined #openstack-swift | 14:15 | |
| openstackgerrit | Mahati Chamarthy proposed openstack/swift: Confirm receipt of SLO PUT with etag https://review.openstack.org/390901 | 14:16 |
| mahatic_ | ^ needs some eyes before I run into merge conflict with tests.py again ;) | 14:19 |
| * mahatic_ is taking off mon and tue. will be on/off IRC tomm | 14:19 | |
| *** amoralej|lunch is now known as amoralej | 14:21 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Cleanup tests from empty suffix quarantined db fix https://review.openstack.org/405134 | 14:30 |
| clayg | mahatic_: I think patch 331601 is ready for a +A - the follow in patch 405134 is pretty clean at this point I think | 14:32 |
| patchbot | https://review.openstack.org/#/c/331601/ - swift - remove empty db hash and suffix directories | 14:32 |
| patchbot | https://review.openstack.org/#/c/405134/ - swift - Cleanup tests from empty suffix quarantined db fix | 14:32 |
| clayg | mahatic_: i split up the bugs so patch 405274 can close the spurious file issue - I think it'll take a little more review to agree on all the right behaviors | 14:32 |
| patchbot | https://review.openstack.org/#/c/405274/ - swift - Cleanup spurious files next to hashdir | 14:32 |
| *** manous has joined #openstack-swift | 14:33 | |
| clayg | mahatic_: note that lp bug #1646502 is totally wide open for a confirmation or failing unittest or even a fixing patch! I think it's unrelated to your work in the container db walker | 14:34 |
| openstack | Launchpad bug 1646502 in OpenStack Object Storage (swift) "spurious files next to object hashdir trigger bad quarantine" [Undecided,New] https://launchpad.net/bugs/1646502 | 14:34 |
| *** psachin has quit IRC | 14:34 | |
| mahatic_ | clayg: all-nighter? | 14:36 |
| cschwede | clayg: are you sleeping sometimes? | 14:36 |
| cschwede | mahatic_: heh, same thought :) | 14:36 |
| mahatic_ | clayg: ah, you've moved the unlink part as well while splitting up. I thought since it's put a function in place for rmdir calls, it should go with patch 405134. But either way is fine | 14:37 |
| patchbot | https://review.openstack.org/#/c/405134/ - swift - Cleanup tests from empty suffix quarantined db fix | 14:37 |
| mahatic_ | maybe it more belongs in patch 405274 as it's handling the unexpected files in suffixdir as well | 14:38 |
| patchbot | https://review.openstack.org/#/c/405274/ - swift - Cleanup spurious files next to hashdir | 14:38 |
| clayg | mahatic_: ah, that might could have been added there i see - but i'm pretty sure that's closer to the spurious files fix - not the empty dir fix | 14:38 |
| clayg | and regardless it wasn't "stright forward cleanup over patch 331601" - which is really what i was going for with patch 405134 | 14:39 |
| patchbot | https://review.openstack.org/#/c/331601/ - swift - remove empty db hash and suffix directories | 14:39 |
| patchbot | https://review.openstack.org/#/c/405134/ - swift - Cleanup tests from empty suffix quarantined db fix | 14:39 |
| clayg | cschwede: what do you wanna do about the deterministic hashdir fix's tests? | 14:39 |
| mahatic_ | clayg: ack on lp bug #1646502. I'd like to take a better look at it. will only get to do it next (i'm off work until wed) | 14:39 |
| openstack | Launchpad bug 1646502 in OpenStack Object Storage (swift) "spurious files next to object hashdir trigger bad quarantine" [Undecided,New] https://launchpad.net/bugs/1646502 | 14:39 |
| mahatic_ | clayg: yeah, I see that | 14:39 |
| jrichli | mahatic_, clayg: I'll retest and check-out patch 331601 today | 14:40 |
| patchbot | https://review.openstack.org/#/c/331601/ - swift - remove empty db hash and suffix directories | 14:40 |
| mahatic_ | jrichli: thanks | 14:40 |
| cschwede | clayg: what do you mean? i see you already added a cleanup? i was thinking to add sth to probetests | 14:43 |
| cschwede | clayg: and probably your cleanup should be squashed into the patch itself? | 14:45 |
| clayg | "sth"? | 14:50 |
| clayg | yeah we can squash it down - we need to get someone else to review it - onovy maybe? | 14:50 |
| jrichli | mahatic_: np! Maybe clayg decided to go to sleep finally ... nope, still awake. Crazy man! | 14:50 |
| mahatic_ | clayg: gotcha! I more often than not end looking up the acronyms you use. I think "sth" = something | 14:52 |
| mahatic_ | s/end/end up | 14:52 |
| mahatic_ | ;) | 14:52 |
| cschwede | clayg: i mean i am looking into modifying the test_replication_servers_working.py test, and ensuring that a hashes.pkl with one missing suffix gets replicated finally | 14:52 |
| clayg | ah, wow - ok | 14:53 |
| cschwede | clayg: i think this bug (together with the random fix of the missing suffix in hashes.pkl) is quite serious, and i want to make sure there is no regression of this in the future | 14:54 |
| *** sams-gleb has joined #openstack-swift | 14:55 | |
| *** daemontool has joined #openstack-swift | 15:00 | |
| *** links has quit IRC | 15:01 | |
| *** stevemar has joined #openstack-swift | 15:21 | |
| *** stevemar_znc has joined #openstack-swift | 15:29 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fixups for ghost listing fix https://review.openstack.org/405450 | 15:32 |
| clayg | cschwede: everyone was so jazzed up about those invalid files being faster somehow we forgot that we're messing with a critically important component? probetests are always good | 15:34 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift: WIP: optimize hash calculation when suffix hash invalidated https://review.openstack.org/402043 | 15:34 |
| *** silor has joined #openstack-swift | 15:37 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Remove duplicate code in test_diskfile.py https://review.openstack.org/405455 | 15:39 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift: WIP: optimize hash calculation when suffix hash invalidated https://review.openstack.org/402043 | 15:41 |
| acoles | clayg: cschwede I have been thinking that it would be good to have a unit test that mocks HASH_PATH_*FIX and does a series of PUTs, POSTs, DELETEs checking the absolute result of a REPLICATE request as it goes in an attempt to prevent any variation in suffix hashes calculations | 15:42 |
| *** links has joined #openstack-swift | 15:48 | |
| *** silor1 has joined #openstack-swift | 15:49 | |
| *** silor has quit IRC | 15:51 | |
| *** silor1 is now known as silor | 15:51 | |
| *** oshritf has quit IRC | 15:53 | |
| *** isotope has joined #openstack-swift | 15:53 | |
| *** isotope has quit IRC | 15:56 | |
| *** krypto has joined #openstack-swift | 16:00 | |
| krypto | hello all i want to remove a swift node from my cluster permanently for other purpose ,right now there are 5 servers total.Can i simply power the server and will it re balance automatically? | 16:02 |
| *** geaaru has quit IRC | 16:07 | |
| *** stevemar_znc is now known as topol_ | 16:12 | |
| *** rcernin has quit IRC | 16:13 | |
| *** _JZ_ has joined #openstack-swift | 16:15 | |
| *** topol_ is now known as topol | 16:20 | |
| *** silor has quit IRC | 16:23 | |
| *** sams-gleb has quit IRC | 16:23 | |
| *** sams-gleb has joined #openstack-swift | 16:24 | |
| *** sams-gleb has quit IRC | 16:28 | |
| notmyname | good morning | 16:34 |
| notmyname | krypto: no, you need to remove its drives from the ring, rebalance, and push the new ring | 16:35 |
| notmyname | krypto: swift will treat a powered-down node as unavailable (as opposed to a durability risk) and it won't automatically rebalance that | 16:36 |
| *** siva_krish has joined #openstack-swift | 16:40 | |
| *** rcernin has joined #openstack-swift | 16:50 | |
| *** links has quit IRC | 16:53 | |
| *** links has joined #openstack-swift | 16:53 | |
| *** dmorita has joined #openstack-swift | 16:54 | |
| *** oshritf has joined #openstack-swift | 16:54 | |
| notmyname | FYI https://github.com/blog/2288-git-2-11-has-been-released (and it's now available for macos via brew). I like the indent-heuristic features | 17:01 |
| *** davideagle_ has quit IRC | 17:03 | |
| notmyname | not as useful for day-to-day in the way openstack works with branches, but the `git log 53d4d2130086c656f47e3a7cb5d6c411e8306546^-1` syntax is interesting too | 17:04 |
| notmyname | (that commit is the merge that brought in storage policies, and the ^-1 tells git log to only show the commits that the referenced merge commit brought in | 17:04 |
| *** jpich has quit IRC | 17:07 | |
| *** krypto has quit IRC | 17:13 | |
| *** asettle has quit IRC | 17:24 | |
| *** deep_1 has quit IRC | 17:26 | |
| *** oshritf has quit IRC | 17:28 | |
| *** klrmn has joined #openstack-swift | 17:29 | |
| *** isotope has joined #openstack-swift | 17:31 | |
| *** diogogmt has joined #openstack-swift | 17:34 | |
| *** chlong has joined #openstack-swift | 17:45 | |
| openstackgerrit | Merged openstack/swift: Add missing expirer recon metric to admin_guide https://review.openstack.org/405265 | 17:52 |
| *** bkopilov has quit IRC | 17:54 | |
| openstackgerrit | Merged openstack/swift: Ensure update of the container by object-updater https://review.openstack.org/99598 | 17:54 |
| *** manous has quit IRC | 17:57 | |
| *** bkopilov has joined #openstack-swift | 17:58 | |
| *** siva_krish has quit IRC | 17:59 | |
| *** siva_krish has joined #openstack-swift | 18:01 | |
| *** dmorita has quit IRC | 18:07 | |
| *** dmorita has joined #openstack-swift | 18:11 | |
| *** mvk has quit IRC | 18:17 | |
| *** henryn has joined #openstack-swift | 18:17 | |
| henryn | good morning all | 18:18 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Further improvement to deterministic suffix re-hashing tests https://review.openstack.org/405559 | 18:19 |
| *** amoralej is now known as amoralej|off | 18:20 | |
| *** sams-gleb has joined #openstack-swift | 18:21 | |
| acoles | cschwede: clayg: left my comments on patch 402324, would be +2 IFF Clay's follow up for tests is squashed in. | 18:23 |
| patchbot | https://review.openstack.org/#/c/402324/ - swift - Fix non-deterministic suffix updates in hashes.pkl | 18:23 |
| *** silor has joined #openstack-swift | 18:37 | |
| *** acoles is now known as acoles_ | 18:38 | |
| *** tqtran has joined #openstack-swift | 18:39 | |
| *** ChubYann has joined #openstack-swift | 18:40 | |
| *** silor1 has joined #openstack-swift | 18:46 | |
| *** silor has quit IRC | 18:47 | |
| *** silor1 is now known as silor | 18:47 | |
| *** henryn has quit IRC | 18:48 | |
| *** daemontool has quit IRC | 18:55 | |
| *** mvk has joined #openstack-swift | 19:14 | |
| *** sams-gleb has quit IRC | 19:22 | |
| *** sams-gleb has joined #openstack-swift | 19:22 | |
| *** sams-gleb has quit IRC | 19:27 | |
| *** david-lyle_ has joined #openstack-swift | 19:28 | |
| *** davideagle_ has joined #openstack-swift | 19:28 | |
| *** henryn has joined #openstack-swift | 19:29 | |
| *** david-lyle has quit IRC | 19:31 | |
| *** openstackgerrit has quit IRC | 19:32 | |
| *** openstackgerrit has joined #openstack-swift | 19:33 | |
| *** openstackgerrit has quit IRC | 19:36 | |
| *** davideagle_ has quit IRC | 19:38 | |
| *** hseipp has quit IRC | 19:43 | |
| *** jamielennox|away is now known as jamielennox | 19:53 | |
| *** david-lyle_ is now known as david-lyle | 20:01 | |
| *** henryn has quit IRC | 20:06 | |
| *** silor has quit IRC | 20:08 | |
| *** dmorita has quit IRC | 20:09 | |
| *** links has quit IRC | 20:09 | |
| *** dmorita has joined #openstack-swift | 20:09 | |
| *** dmorita has quit IRC | 20:10 | |
| *** dmorita has joined #openstack-swift | 20:13 | |
| *** dmorita has quit IRC | 20:17 | |
| *** dmorita has joined #openstack-swift | 20:19 | |
| *** dmorita has quit IRC | 20:26 | |
| *** henryn has joined #openstack-swift | 20:27 | |
| *** isotope has quit IRC | 20:30 | |
| *** catinthe_ has quit IRC | 20:34 | |
| *** isotope has joined #openstack-swift | 20:42 | |
| *** openstack has joined #openstack-swift | 20:45 | |
| *** siva_krish has quit IRC | 20:51 | |
| *** henryn has quit IRC | 20:58 | |
| *** siva_krish has joined #openstack-swift | 20:58 | |
| *** dmorita has joined #openstack-swift | 20:58 | |
| *** openstackgerrit has joined #openstack-swift | 21:04 | |
| openstackgerrit | Merged openstack/swift: remove empty db hash and suffix directories https://review.openstack.org/331601 | 21:04 |
| *** rcernin has quit IRC | 21:10 | |
| *** nexusz99 has joined #openstack-swift | 21:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 21:17 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 21:18 |
| pdardeau | this is probably a dumb question, is use of jumbo frames the norm for swift deployments? | 21:21 |
| notmyname | pdardeau: I've heard that question before. let me ask around | 21:24 |
| pdardeau | notmyname: thanks! | 21:26 |
| notmyname | pdardeau: ok, I got an answer from some our support people :-) | 21:26 |
| notmyname | short answer: good technical idea, generally bad operationally | 21:27 |
| notmyname | so it's probably not worth it | 21:28 |
| pdardeau | notmyname: thanks! i've heard similar things in the past, but wanted to see if that's still the case | 21:29 |
| notmyname | operationally it's not great because it requires the whole networking infrastructure to be set up with it. and, to quote our support person, "they complain swift isn't working and we ask if the MTUs are the same everywhere and then two weeks later it's 'oh yeah, someone changed a switch config' or something" | 21:30 |
| *** siva_krish has quit IRC | 21:30 | |
| notmyname | so unless you've got decent controls over that sort of thing and are tuning for a specific workload, it's not worth the operational overhead | 21:31 |
| ahale | yeah, im pretty sure when i was with rackspace we didn't do any of that sort of thing as it was too much hassle | 21:33 |
| notmyname | oh hi ahale! | 21:33 |
| ahale | o/ | 21:34 |
| *** Jeffrey4l_ has quit IRC | 21:34 | |
| *** siva_krish has joined #openstack-swift | 21:35 | |
| *** Jeffrey4l_ has joined #openstack-swift | 21:37 | |
| pdardeau | ahale: thx for sharing | 21:44 |
| *** dmorita has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack/pyeclib: Cleanup lint errors in test suite https://review.openstack.org/404457 | 21:45 |
| *** dmorita has joined #openstack-swift | 21:45 | |
| *** diogogmt has quit IRC | 21:45 | |
| clayg | timburke: i'm not so sure the rmdir is not a big deal - it might be? maybe a script that will put a roundrobin_datadirs into a tight loop; while another script does some direct container PUTs? | 21:47 |
| notmyname | clayg: good morning! | 21:48 |
| clayg | looking lock_path it does look like maybe the mkdir/open(lockpath) could blowup | 21:48 |
| timburke | clayg: yeah, our directory locking seems like it could be problematic :-( | 21:49 |
| clayg | the ENOTEMPTY in roundrobin (should such a thing occur) would be addressed in patch 405274 if we ever make it that far | 21:49 |
| patchbot | https://review.openstack.org/#/c/405274/ - swift - Cleanup spurious files next to hashdir | 21:49 |
| clayg | timburke: i'm not aware of ever having seen that race in traceback - it seems like other things (container db reclimation) are cleaning suffix dirs currently | 21:51 |
| clayg | so if it *is* a problem it'd just be making it more frequent | 21:51 |
| clayg | ... it's possible there's some other interaction with container PUT that resolves the race | 21:51 |
| tdasilva | is clayg in japan or australia? | 21:52 |
| clayg | jrichli: missed that about the erorr case test in all the mocks - mocks are definately the way to handle the error case - a targeted test to catch the OSError line would be the ideal! | 21:53 |
| jrichli | yes, that would be better | 21:54 |
| timburke | clayg: that reminds me: why are we so gung-ho to delete things when we don't know wtf the are? am i the only one that gets nervous doing that? | 21:54 |
| jrichli | timburke: I was nervous too, but the rmdir doesn't succeed if its non-empty. And a random empty dir seems ok to remove. | 21:55 |
| clayg | timburke: we have to make progress, i'm not that *gung-ho* about deleting spurious files next to suffix dirs - but I'd rather than not stop the partition from being balanced off the node | 21:55 |
| clayg | so i'm looking to the object-replicator for inspiration | 21:56 |
| clayg | it seems to quarantine spurious hashdir-files and while it ignores spurious suffixdir-files it *will* recursively destory them (and the hashes.pkl they sit next too) when it delete's the handoff partdir after sync (n.b. sync on moves suffixdirs never spurious suffixdir-files) | 21:57 |
| clayg | it's a crap system - but it works | 21:57 |
| jrichli | maybe I need to look again: I was only focused on recent changes made, and I dont see where we are removing more files than we had before | 21:57 |
| timburke | jrichli: yeah, the empty dirs are fine. i'm looking at the patch clayg just linked | 21:58 |
| jrichli | timburke: oh right, thx! | 21:58 |
| clayg | timburke: i'm very much hoping someone has better ideas ;) | 21:58 |
| clayg | timburke: if there's part you like but are less sure on other's we can try to get mahatic_ to split it up some more | 21:59 |
| clayg | i'm mostly just glad to have lp bug #1583719 closed - hopefully we didn't wreak anything else | 21:59 |
| openstack | Launchpad bug 1583719 in OpenStack Object Storage (swift) "empty db hash/suffix directories can pile up" [Medium,In progress] https://launchpad.net/bugs/1583719 - Assigned to Mahati Chamarthy (mahati-chamarthy) | 21:59 |
| timburke | idk, emit a warning? quarantine it? do something so the operator knows that these things we expected to all be directories had a file in 'em | 22:00 |
| mattoliverau | morning | 22:00 |
| clayg | no why what happened! patch 393276 | 22:00 |
| patchbot | https://review.openstack.org/#/c/393276/ - pyeclib - ISA-L Cauchy support | 22:00 |
| timburke | ahahaha must've been me. but it's also you :P | 22:01 |
| clayg | timburke: object-replicator just ignores them and then delete's them when it removes the partition - what's wrong with that pattern? I don't want parts to not balance until operator "does something" - I'd be ok if we could trick the quarantine machinery into eating them | 22:01 |
| *** sams-gleb has joined #openstack-swift | 22:02 | |
| clayg | os.path.join(suffixdir_path_file, 'some-pretend-hashdir', 'some-pretend-filename') might work? did you see lp bug #1646502 | 22:02 |
| openstack | Launchpad bug 1646502 in OpenStack Object Storage (swift) "spurious files next to object hashdir trigger bad quarantine" [Undecided,New] https://launchpad.net/bugs/1646502 | 22:02 |
| clayg | timburke: so... are you going to do that roundrobin datadir and container PUT race script test? | 22:03 |
| *** openstackgerrit has quit IRC | 22:03 | |
| clayg | timburke: or do you think we can wait and see if the tracebacks happen in the wild? in either race I feel like a) roundrobin datadirs will throw and try again next pass b) container server will throw and 500 (but other container replicas probably won't) | 22:04 |
| *** clu_ has joined #openstack-swift | 22:05 | |
| *** sams-gleb has quit IRC | 22:06 | |
| *** sams-gleb has joined #openstack-swift | 22:07 | |
| jrichli | timburke: do we create the .lock in such a way that it will create sub-dirs if they don't exist? | 22:10 |
| jrichli | and if we do, it seems like an OS thing at that point for the race | 22:10 |
| *** sams-gleb has quit IRC | 22:11 | |
| clayg | omg the stuff people get away with when here is no linting!? 3 spaces, srly!? ;) | 22:12 |
| clayg | jrichli: there is no system call for open this file and create the dirs if they don't exist | 22:13 |
| jrichli | ok :-) | 22:13 |
| clayg | jrichli: timburke: gah the more I look at common/db.py the more I see stuff that will get mad when the container-replicator removes empty dirs from under it :'( | 22:17 |
| jrichli | which rmdirs are risky? some already existed IIRC | 22:19 |
| jrichli | like the one we just merged? | 22:20 |
| *** vint_bra has quit IRC | 22:20 | |
| clayg | jrichli: yeah timburke pointed out that I should probably be more scared about the rmdir race that I was (since we were going to be doing more rmdirs!) | 22:24 |
| clayg | but... i wasn't scared... and now I'm thinking he was probably right and I should be scared | 22:25 |
| jrichli | so he said just a traceback. is that because there are retries? | 22:25 |
| jrichli | or would something fail and not get retried? | 22:25 |
| clayg | ... but i'm not writing the script to test if the new roundrobin_datadirs + heavy container PUT load will race - so maybe I'm not *that* scared | 22:25 |
| clayg | jrichli: unclear how it will fail w/o testing | 22:25 |
| clayg | jrichli: might not fail at all in practice - could be entirely theoretical | 22:26 |
| clayg | but it's a *good theory*! | 22:26 |
| jrichli | agreed | 22:27 |
| jrichli | i have wondered before about having a heavy load test in the gate. i guess it would too resource intensive. | 22:29 |
| jrichli | i know, in this case, you'd have to have those empty dirs too | 22:29 |
| jrichli | but it just made me think of it | 22:29 |
| jrichli | well, i guess in theory they'd be created under this high load | 22:30 |
| jrichli | just run replication more than usual | 22:31 |
| *** chlong has quit IRC | 22:50 | |
| timburke | clayg: jrichli: ok, that's about as tight a loop as i think i can manage http://paste.openstack.org/show/591185/ | 22:51 |
| timburke | i'm not gonna worry about it. can't for the life of me get the race to trigger; not with threads, not with processes | 22:51 |
| clayg | timburke: is he best! | 22:54 |
| jrichli | timburke: glad you aren't worried. I am still running a test of my own :-) | 22:55 |
| *** isotope has quit IRC | 22:59 | |
| clayg | timburke: that script is great - for me the tipping point is ~35 microseconds of sleep between lock_path's mkdir and open (sleep(0.000035)) does will cause it trip in for loop | 23:07 |
| jrichli | so, i am looking at storage logs more than I usually do. I am seeing a bunch of HEADs and GETs to .misplaced_objects. is that normal when you replicate? | 23:07 |
| jrichli | like HEAD /sdb2/701/.misplaced_objects | 23:08 |
| clayg | that's just the reconciler doing it's thing | 23:09 |
| jrichli | clayg: whew :-) | 23:09 |
| clayg | it's sort of like the object expirer | 23:09 |
| clayg | maybe it's frequency is less or something idk | 23:09 |
| timburke | clayg: by "trip", do you mean to say that the loop didn't run? or that it would actually hit an OSError during open()? | 23:09 |
| timburke | 'cause that last one is what i was worried about, but i just didn't see it | 23:10 |
| *** isotope has joined #openstack-swift | 23:12 | |
| clayg | yeah with some sleep in there you can hit the OSError - that's how I knew the code was exercising the race | 23:12 |
| clayg | it's in there - just NBD in practice | 23:12 |
| timburke | oh, i see what you were saying now! got it | 23:12 |
| clayg | if this code can't even hit hit a replicator vs a container have only a snowballs chance in hell (meaning well totally get that traceback some day - but it won't be as soon as this sha makes it out to prod) | 23:13 |
| *** StraubTW has quit IRC | 23:13 | |
| *** isotope has quit IRC | 23:16 | |
| jrichli | I still have some ideas for testing i'll do later for fun. but i'm goin to eat dinner now | 23:17 |
| clayg | do eet! | 23:17 |
| clayg | nom nom | 23:17 |
| *** kei_yama has joined #openstack-swift | 23:30 | |
| *** siva_krish has quit IRC | 23:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!