| *** Administrator_ has quit IRC | 00:04 | |
| *** Administrator_ has joined #openstack-swift | 00:05 | |
| *** mvk has quit IRC | 00:12 | |
| *** dja_ is now known as dja | 00:16 | |
| *** catintheroof has quit IRC | 00:19 | |
| *** chsc has quit IRC | 00:21 | |
| *** mvk has joined #openstack-swift | 00:25 | |
| *** cdelatte has joined #openstack-swift | 00:30 | |
| *** cdelatte has quit IRC | 00:30 | |
| *** mvk has quit IRC | 00:34 | |
| *** mvk has joined #openstack-swift | 00:34 | |
| mattoliverau | If we auto include dlo, we should def auto include slo especially if it's what we keep recommending over dlo :) | 00:48 |
|---|---|---|
| timburke | either way, client still ought to be robust to it not being in the pipeline, and definitely shouldn't go confusing things by sending what's essentially a sysmeta header | 00:50 |
| *** mvk has quit IRC | 00:51 | |
| *** zhurong has joined #openstack-swift | 00:55 | |
| mattoliverau | timburke: yeah, totally. | 00:56 |
| mattoliverau | if it aint there, don't use it | 00:57 |
| *** mvk has joined #openstack-swift | 01:03 | |
| *** gyee has quit IRC | 01:04 | |
| *** m_kazuhiro has joined #openstack-swift | 01:08 | |
| m_kazuhiro | good morning | 01:08 |
| *** mvk has quit IRC | 01:10 | |
| mattoliverau | m_kazuhiro: morning | 01:10 |
| *** mvk has joined #openstack-swift | 01:10 | |
| m_kazuhiro | mattoliverau: morning! | 01:10 |
| *** klrmn has quit IRC | 01:12 | |
| kota_ | good morning | 01:15 |
| kota_ | welcome back timburke | 01:15 |
| timburke | thanks kota_! and good morning | 01:16 |
| timburke | good morning m_kazuhiro! | 01:16 |
| m_kazuhiro | timburke: good morning! | 01:16 |
| kota_ | timburke: and thanks for looking at patch 454447 but it was squashed up into patch 454174 | 01:16 |
| patchbot | https://review.openstack.org/#/c/454447/ - swift - Fix StopIteration if no more nodes in DB Replication | 01:16 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 01:16 |
| kota_ | timburke: sorry, i should have abandoned the patch before you had wasted time for it. | 01:17 |
| mattoliverau | kota_: morning | 01:17 |
| kota_ | mattoliverau: morning | 01:17 |
| timburke | kota_: not much time :-) and i think i like how that patch dealt with it... i just need to look at tests | 01:18 |
| timburke | it just always feels a little weird to deal in StopIterations explicitly | 01:18 |
| kota_ | timburke: yup, that have a little history we had discussion how we could imporove/cleanup the code | 01:19 |
| kota_ | timburke: but for now, IMO, we want to fix a bug (or some bugs) with small changes and *want* to try to cleanup/refactor all db_replicator iteration | 01:20 |
| kota_ | after bug fixes | 01:20 |
| kota_ | the patch 454174 actually fixes two bugs, a) stop to bleed the db can be located every drives b) stop to raise bare StopIteration when we ran out of devices (i.e. no more handoffs) | 01:22 |
| patchbot | https://review.openstack.org/#/c/454174/ - swift - Container drive error results double space usage o... | 01:22 |
| kota_ | and it seems clayg is trying to fix some test refactoring at patch 454898 | 01:22 |
| patchbot | https://review.openstack.org/#/c/454898/ - swift - Fix default FakeRing max_more_nodes | 01:22 |
| *** JimCheung has quit IRC | 01:31 | |
| *** Sukhdev has quit IRC | 01:33 | |
| kota_ | thanks mattoliverau to looki at that! | 01:35 |
| mattoliverau | kota_: nps, patch looks small and clean. And pretty much fixes the bleeding in the way clay and I both did in the bug :) | 01:36 |
| *** gmann has joined #openstack-swift | 01:46 | |
| gmann | notmyname: can you check query/answer for bulk delete request - https://review.openstack.org/#/c/409634/8 | 01:48 |
| patchbot | patch 409634 - tempest - Separate object-storage bulk operation service cli... | 01:48 |
| gmann | notmyname: query url would not contain any path to list of objects to be deleted right? its in request body what i understood from code- http://git.openstack.org/cgit/openstack/openstack-manuals/tree/doc/user-guide/source/cli-swift-bulk-delete.rst#n16 | 01:49 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Tolerate RFC-compliant ETags https://review.openstack.org/455488 | 01:52 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Skip checksum validation on partial downloads https://review.openstack.org/455489 | 01:52 |
| kota_ | it looks Boston Summit Forum has been scheduled | 01:55 |
| kota_ | and all proposed sessions for swift look there | 01:56 |
| * kota_ is looking at http://lists.openstack.org/pipermail/openstack-dev/2017-April/115174.html | 01:57 | |
| mattoliverau | kota_: yeah, one of them twice! | 01:58 |
| mattoliverau | but yeah, looks like it's all in there | 01:58 |
| mattoliverau | Now I just have to hope I can make it to boston.. otherwise, timburke will need to handle sharding for me | 01:58 |
| * timburke hides | 01:59 | |
| kota_ | mattoliverau: I hope you will be there ;-) | 01:59 |
| mattoliverau | timburke: good choice ;) | 02:00 |
| mattoliverau | kota_: +1 | 02:00 |
| *** chsc has joined #openstack-swift | 02:02 | |
| *** chsc has joined #openstack-swift | 02:02 | |
| *** SkyRocknRoll has joined #openstack-swift | 02:03 | |
| *** chsc has quit IRC | 02:07 | |
| timburke | mattoliverau: doesn't the logger.error already rather imply ERROR? | 02:08 |
| timburke | i'd be partial toward removing it from the other locations, but that'd probably muddy the patch a bit | 02:09 |
| mattoliverau | yes it does, but the line below also has it.. so consistency | 02:09 |
| mattoliverau | or remove ERROR on the other lines is fine too :P | 02:09 |
| mattoliverau | exactly | 02:09 |
| mattoliverau | but really rewording would be nice, and make consistent.. easiest is to add the ERORR, and then wouldn't potentially muddy any translations, if that is in fact how gettext works ? | 02:10 |
| timburke | i was actually thinking that pushing an ERROR version and then another patch to strip them all out would cause some needless churn for the translators, but then, i've never really had to deal with i18n much... | 02:13 |
| *** catintheroof has joined #openstack-swift | 02:16 | |
| mattoliverau | well, we could at least raise the patch and then ask in 118n, we could always abandon if it causes too much work. or maybe they wont care, update the po files, but translations _may_ stay the same, depends on how the translation tracks log changes. | 02:18 |
| timburke | i think if we land the string change fairly quickly after the critical fix, it comes out a wash for them? *shrug* | 02:19 |
| timburke | ...and then there's the question of whether we're going to get on the http://lists.openstack.org/pipermail/openstack-dev/2017-March/114191.html bandwagon... | 02:22 |
| timburke | between that and clayg's well-reasoned response on https://review.openstack.org/#/c/318989/ i wonder whether *anything* should be marked for translation in swift, but we should start translating *swiftclient* instead... | 02:26 |
| patchbot | patch 318989 - swift - Mark more end-user-facing strings for translation | 02:26 |
| timburke | unless we want to get some Accept-Language support going... ugh. | 02:26 |
| timburke | well, anyway, i oughta get going before my wife starts to wonder what happened to me | 02:29 |
| timburke | good night everybody! | 02:29 |
| *** catintheroof has quit IRC | 02:34 | |
| *** catintheroof has joined #openstack-swift | 02:35 | |
| *** catintheroof has quit IRC | 02:40 | |
| *** klrmn has joined #openstack-swift | 02:54 | |
| *** SkyRocknRoll has quit IRC | 03:31 | |
| *** edisonxiang has joined #openstack-swift | 03:33 | |
| *** gkadam has joined #openstack-swift | 03:39 | |
| *** gkadam is now known as gkadam-afk | 03:44 | |
| *** chlong has quit IRC | 03:47 | |
| *** SkyRocknRoll has joined #openstack-swift | 03:49 | |
| *** m_kazuhiro_ has joined #openstack-swift | 04:00 | |
| *** dfflanders_ has joined #openstack-swift | 04:01 | |
| *** m_kazuhiro has quit IRC | 04:02 | |
| *** chsc has joined #openstack-swift | 04:03 | |
| *** chsc has joined #openstack-swift | 04:03 | |
| *** Dinesh_Bhor has quit IRC | 04:07 | |
| *** gkadam-afk is now known as gkadam | 04:11 | |
| *** SkyRocknRoll has quit IRC | 04:11 | |
| *** Sukhdev has joined #openstack-swift | 04:14 | |
| notmyname | hello, world | 04:15 |
| mahatic | notmyname: o/ | 04:16 |
| mahatic | good morning | 04:16 |
| notmyname | I finally made it to Ann Arbor (bunch of delays getting out of SFO), but now it'd after midnight, so I was just checking what had happened today | 04:16 |
| * mahatic learns where Ann Arbor is | 04:18 | |
| *** psachin has joined #openstack-swift | 04:31 | |
| *** Dinesh_Bhor has joined #openstack-swift | 05:11 | |
| *** Sukhdev has quit IRC | 05:12 | |
| *** klrmn has quit IRC | 05:12 | |
| *** rcernin has joined #openstack-swift | 05:15 | |
| kota_ | mahatic: morning, notmyname: good... midnight? | 05:21 |
| * kota_ learns where Ann Arbor is too | 05:21 | |
| mahatic | kota_: o/ | 05:24 |
| *** cshastri has joined #openstack-swift | 05:33 | |
| *** psachin has quit IRC | 05:34 | |
| *** ChubYann has quit IRC | 05:38 | |
| *** zaitcev has quit IRC | 05:46 | |
| *** spotz is now known as spotz_zzz | 05:46 | |
| *** spotz_zzz is now known as spotz | 05:48 | |
| *** jaosorior_away is now known as jaosorior | 05:55 | |
| *** chsc has quit IRC | 05:56 | |
| *** adriant has quit IRC | 06:02 | |
| *** sudorandom has quit IRC | 06:29 | |
| *** sudorandom has joined #openstack-swift | 06:30 | |
| *** psachin has joined #openstack-swift | 06:31 | |
| *** chsc has joined #openstack-swift | 06:38 | |
| *** chsc has joined #openstack-swift | 06:38 | |
| *** chsc has quit IRC | 06:46 | |
| *** tesseract has joined #openstack-swift | 06:47 | |
| *** mgagne has quit IRC | 06:52 | |
| *** JimCheung has joined #openstack-swift | 07:00 | |
| *** m_kazuhiro_ has quit IRC | 07:04 | |
| *** winggundamth has joined #openstack-swift | 07:04 | |
| *** JimCheung has quit IRC | 07:04 | |
| *** pcaruana has joined #openstack-swift | 07:06 | |
| mathiasb | good morning! | 07:10 |
| *** geaaru has joined #openstack-swift | 07:26 | |
| mattoliverau | mathiasb: morning | 07:31 |
| acoles | good morning | 07:47 |
| mattoliverau | acoles: morning | 07:49 |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift master: Container drive error results double space usage on rest drives https://review.openstack.org/454174 | 07:50 |
| acoles | mattoliverau: hi how's things? you gotta get to Boston now :) | 07:53 |
| mattoliverau | lol, yeah, sharding is scheduled.. so lets cross fingers :) | 07:53 |
| *** ianychoi has quit IRC | 07:54 | |
| *** m_kazuhiro has joined #openstack-swift | 07:54 | |
| mattoliverau | I'm all good. I easter this week and the in-laws are coming up as it's school holidays.. so I have a feeling I'll be a little distracted this week. But good | 07:54 |
| kota_ | acoles: morning | 07:54 |
| mattoliverau | acoles: how are things with you? | 07:54 |
| *** ianychoi has joined #openstack-swift | 07:54 | |
| acoles | kota_: hi | 07:55 |
| acoles | mattoliverau: good thanks, got a long weekend coming up for Easter here too. And the sun is still shining! | 07:56 |
| mattoliverau | \o/ nice. Cyclone up north has cleared up, so we're back to nice beach weather here.. so easter is shaping up nicely here too :) | 07:57 |
| mattoliverau | k, I'm going to go do the normal baby dinner, bath, bed rutine. bbl | 08:01 |
| acoles | fun time! | 08:01 |
| *** dfflanders_ has quit IRC | 08:03 | |
| *** cbartz has joined #openstack-swift | 08:07 | |
| mahatic | acoles: good morning | 08:07 |
| acoles | mahatic: o/ | 08:08 |
| mahatic | I haven't gotten around much on this - https://review.openstack.org/#/c/449310/, but do have some comments in draft. Looks like there has been much progress! | 08:08 |
| patchbot | patch 449310 - swift - Add id to RingBuilder to differentiate rings in co... | 08:08 |
| acoles | mahatic: thanks, all comments welcome. that will hopefully squash into the parent but I thought it helpful to leave it separate while things are changing | 08:09 |
| mahatic | acoles: yeah, latter seems better for now | 08:10 |
| *** ianychoi has quit IRC | 08:20 | |
| *** bikmak has joined #openstack-swift | 08:20 | |
| kota_ | acoles, mahatic: I'm also just about looking at the patch progress | 08:20 |
| *** ianychoi has joined #openstack-swift | 08:21 | |
| kota_ | sorry, I didn't have a time to look at in recent 2 days, I'll check the recent difference tomorrow morning | 08:21 |
| kota_ | i think, we can use uuid published at the save and it seems ok to verify for preventing same uuid in the component rings, right? | 08:22 |
| * kota_ had quick veiw | 08:22 | |
| acoles | kota_: NP. I tried to address the comments around using an ID. And I found a bug in compose_builder | 08:22 |
| kota_ | acoles: for patch 441921? | 08:22 |
| patchbot | https://review.openstack.org/#/c/441921/ - swift - Add Composite Ring Functionality | 08:22 |
| acoles | kota_: yes, assign uuid when saving is the safest way to guarantee the id is persisted | 08:23 |
| * kota_ assuming the *yes* is to re: when uuid will be made. | 08:23 | |
| acoles | kota_: for now I left the UUID patch as a follow on, but just for clarity so we can see what is changing. It should be squashed when we are happy. | 08:24 |
| acoles | kota_: you assume correctly :) | 08:24 |
| kota_ | acoles: ok, awesome progress. thanks | 08:24 |
| acoles | kota_: I hope to work some more on https://review.openstack.org/#/c/453827/ soon. | 08:25 |
| patchbot | patch 453827 - swift - WIP Add CompositeRingBuilder class | 08:25 |
| kota_ | acoles: yup, that's fantastic too | 08:26 |
| *** ianychoi has quit IRC | 08:35 | |
| *** ianychoi has joined #openstack-swift | 08:38 | |
| *** derekjhyang has joined #openstack-swift | 08:43 | |
| openstackgerrit | junboli proposed openstack/swift master: Fixed get ring name from recon cli https://review.openstack.org/448449 | 08:48 |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** kei_yama has quit IRC | 09:49 | |
| *** kei_yama has joined #openstack-swift | 09:51 | |
| *** PavelK has joined #openstack-swift | 10:06 | |
| *** m_kazuhiro has quit IRC | 10:07 | |
| *** zhurong has quit IRC | 10:10 | |
| *** zhurong has joined #openstack-swift | 10:11 | |
| dewanee_ | anyone has experience in integrating swift and keystone roles? Apparently if a user has read/write access to a container belonging to a project but no other rights he/she cannot delete multiple files | 10:27 |
| dewanee_ | just one at a time | 10:27 |
| dewanee_ | which is ridicolous | 10:28 |
| *** ianychoi has quit IRC | 10:36 | |
| *** ianychoi has joined #openstack-swift | 10:38 | |
| *** kei_yama has quit IRC | 11:22 | |
| *** mvk has quit IRC | 11:49 | |
| *** ianychoi has quit IRC | 12:21 | |
| *** ianychoi has joined #openstack-swift | 12:26 | |
| *** gkadam has quit IRC | 12:40 | |
| *** PavelK has quit IRC | 12:48 | |
| *** stradling has joined #openstack-swift | 12:48 | |
| *** zhurong has quit IRC | 12:54 | |
| *** winggundamth has quit IRC | 13:12 | |
| *** RobGThai has joined #openstack-swift | 13:14 | |
| *** RobGThai has quit IRC | 13:15 | |
| *** RobGThai has joined #openstack-swift | 13:15 | |
| RobGThai | Hi guys, I have issue with Swift Proxy failing upload request a lot. | 13:15 |
| RobGThai | Has anyone got similar experiences before? | 13:16 |
| *** JimCheung has joined #openstack-swift | 13:17 | |
| *** gabor_antal has quit IRC | 13:19 | |
| *** gabor_antal has joined #openstack-swift | 13:20 | |
| *** JimCheung has quit IRC | 13:21 | |
| *** Dinesh_Bhor has quit IRC | 13:22 | |
| *** Dinesh_Bhor has joined #openstack-swift | 13:24 | |
| *** _JZ_ has joined #openstack-swift | 13:25 | |
| *** mvk has joined #openstack-swift | 13:33 | |
| *** JimCheung has joined #openstack-swift | 13:33 | |
| *** JimCheung has quit IRC | 13:37 | |
| *** vint_bra has joined #openstack-swift | 13:42 | |
| *** SkyRocknRoll has joined #openstack-swift | 13:51 | |
| *** flwang has quit IRC | 13:57 | |
| *** ianychoi has quit IRC | 14:05 | |
| *** jaosorior is now known as jaosorior_away | 14:06 | |
| *** ianychoi has joined #openstack-swift | 14:06 | |
| *** flwang has joined #openstack-swift | 14:10 | |
| *** silor has joined #openstack-swift | 14:17 | |
| *** chlong has joined #openstack-swift | 14:21 | |
| -openstackstatus- NOTICE: latest base images have mistakenly put python3 in some places expecting python2 causing widespread failure of docs patches - fixes are underway | 14:27 | |
| *** jordanP has joined #openstack-swift | 14:28 | |
| *** SkyRocknRoll has quit IRC | 14:35 | |
| *** jordanP has quit IRC | 14:37 | |
| *** edisonxiang has quit IRC | 14:38 | |
| *** edisonxiang has joined #openstack-swift | 14:39 | |
| *** jordanP has joined #openstack-swift | 14:39 | |
| *** cshastri has quit IRC | 14:40 | |
| -openstackstatus- NOTICE: we have rolled back centos-7, fedora-25 and ubuntu-xenial images to the previous days release. Feel free to recheck your jobs now. | 14:48 | |
| *** ianychoi has quit IRC | 14:48 | |
| *** ianychoi has joined #openstack-swift | 14:52 | |
| *** jlvillal_pto is now known as jlvillal | 14:52 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:58 | |
| *** klrmn has joined #openstack-swift | 15:01 | |
| *** rcernin has quit IRC | 15:05 | |
| hugokuo | morning ~! | 15:21 |
| *** xinli has joined #openstack-swift | 15:25 | |
| *** psachin has quit IRC | 15:25 | |
| *** joeljwright has joined #openstack-swift | 15:25 | |
| *** ChanServ sets mode: +v joeljwright | 15:25 | |
| *** joeljwright has quit IRC | 15:26 | |
| *** joeljwright has joined #openstack-swift | 15:27 | |
| *** ChanServ sets mode: +v joeljwright | 15:27 | |
| timburke | good morning | 15:27 |
| joeljwright | timburke: hey | 15:28 |
| *** chlong_ has joined #openstack-swift | 15:29 | |
| *** chlong has quit IRC | 15:32 | |
| *** geaaru has quit IRC | 15:37 | |
| * timburke should go look at that pre-/post-amble patch again... | 15:39 | |
| *** neonpastor has joined #openstack-swift | 15:39 | |
| joeljwright | timburke: yes please! | 15:39 |
| joeljwright | :D | 15:39 |
| *** vinsh has quit IRC | 15:43 | |
| *** vinsh has joined #openstack-swift | 15:43 | |
| *** chlong has joined #openstack-swift | 15:43 | |
| *** tonanhngo has joined #openstack-swift | 15:47 | |
| *** Administrator_ has quit IRC | 15:51 | |
| *** Administrator_ has joined #openstack-swift | 15:52 | |
| timburke | dewanee_: i'm starting to regret having done https://review.openstack.org/#/c/190887/ ... the trouble is that when there are enough objects, python-swiftclient switches from per-object deletes to bulk deletes in an effort to minimize WAN requests. however, the authorization gets confused since that involves a POST to the account | 15:53 |
| patchbot | patch 190887 - python-swiftclient - Use bulk-delete middleware when available (MERGED) | 15:53 |
| timburke | i'm thinking i should revisit the idea of having a --bulk option and requiring that users opt-in to the bulk-delete behavior | 15:53 |
| *** vinsh_ has joined #openstack-swift | 15:57 | |
| dewanee_ | timburke, that's actually a useful feature but at the same time it can give the issues I was writing about before | 15:58 |
| *** jistr is now known as jistr|mtg | 15:59 | |
| dewanee_ | what would then be the minimum permission needed to have it work? | 16:00 |
| *** vinsh has quit IRC | 16:00 | |
| dewanee_ | we know it works as admin or however you want to call the role | 16:01 |
| *** ianychoi has quit IRC | 16:07 | |
| timburke | :-/ looking at the bulk-delete code, i'm not actually sure what went wrong -- it seems to just translate the one account POST into a whole bunch of object DELETEs, each of which should individually be authorized. is bulk left of auth in the proxy pipeline? | 16:08 |
| timburke | either way, i should see about getting a patch up to require a flag to trigger bulk deletes, and in the mean time, you can do some manual batching, or downgrade to swiftclient<3.0.0, or (if you feel like hacking up swiftclient a bit) find _should_bulk_delete in service.py and have it always return False | 16:09 |
| *** ianychoi has joined #openstack-swift | 16:12 | |
| *** d0ugal has quit IRC | 16:18 | |
| *** gyee has joined #openstack-swift | 16:25 | |
| *** JimCheung has joined #openstack-swift | 16:28 | |
| *** pcaruana has quit IRC | 16:29 | |
| *** jistr|mtg is now known as jistr | 16:37 | |
| *** chsc has joined #openstack-swift | 16:41 | |
| *** joeljwright has quit IRC | 16:45 | |
| clayg | timburke: patch 190887 took seven months to get merged; and 4 core reviewers and a couple of other folks; and you're saying it may not have been the best idea to add the behavior in the first place? | 16:45 |
| patchbot | https://review.openstack.org/#/c/190887/ - python-swiftclient - Use bulk-delete middleware when available (MERGED) | 16:45 |
| clayg | ... or you just mean the part to make it "automatic" | 16:45 |
| clayg | most of the code would have been the same even if it was --explicit? | 16:45 |
| RobGThai | Quick question, should I use SwiftService in python-swiftclient to write an API for user to upload file into Swift? | 16:47 |
| clayg | oh, it's all torgomatic's fault "I'd prefer not having two ways to delete things; what about adding this stuff to the normal "swift delete" command and using bulk ops if available and single requests otherwise?" | 16:47 |
| timburke | clayg: the automatic part. torgomatic's comment of "The --bulk flag could be good, but I don't see it as essential." may have been off the mark | 16:48 |
| clayg | RobGThai: yeah, SwiftService is totally supported and even documented! https://docs.openstack.org/developer/python-swiftclient/service-api.html | 16:48 |
| RobGThai | I have a problem using it in production. | 16:48 |
| RobGThai | It tried to create container every request. | 16:48 |
| clayg | RobGThai: joeljwright I'm sure would love you to give it a spin, report bugs, provide feedback and *gasp* patches for docs and missing features/bugs! | 16:48 |
| clayg | RobGThai: what kind of request? | 16:49 |
| RobGThai | Causing Account sync to fail because it's locking a lot. | 16:49 |
| clayg | ... account sync ... | 16:49 |
| RobGThai | At least that's what the error seems to say. | 16:50 |
| RobGThai | Is that something related? Or is it something completely different? | 16:51 |
| *** winggundamth has joined #openstack-swift | 16:53 | |
| *** stradling has quit IRC | 16:54 | |
| cbartz | Hello. Anyone here for reviewing patch 454716 ? Thx.... | 16:57 |
| patchbot | https://review.openstack.org/#/c/454716/ - swift - Allow DLO PUT to prefix-based tempurls | 16:57 |
| *** mgagne has joined #openstack-swift | 17:00 | |
| clayg | RobGThai: well, post some code in a gist or something - i'm not sure what error you're seeing or from where? | 17:00 |
| clayg | RobGThai: there may be an option to restructure the calls so you get only the requests you want | 17:01 |
| *** stradling has joined #openstack-swift | 17:02 | |
| RobGThai | There's a way? | 17:02 |
| RobGThai | clayg https://gist.github.com/RobGThai/42eba4c73f78ee48c9c92e76231ac42f | 17:03 |
| RobGThai | It's pretty straight forward, really. | 17:04 |
| timburke | i don't think there is a way to avoid the container PUT through the service api -- although i'd totally +2 a patch to add an option to bypass it! https://github.com/openstack/python-swiftclient/blob/3.3.0/swiftclient/service.py#L1427-L1437 | 17:04 |
| *** cbartz has quit IRC | 17:07 | |
| dewanee_ | if you are curious that's what the proxy reply: cold-k4-30 proxy-server: xxx.xxx.xxx.xxx 10.129.31.231 11/Apr/2017/17/04/45 DELETE /v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/user%3Fbulk-delete%3D1 HTTP/1.0 403 | 17:08 |
| dewanee_ | if a user has the ability to create containers and manage acl it works | 17:09 |
| dewanee_ | but that's what we wanted to avoid in production | 17:09 |
| dewanee_ | I don't know if it makes more sense this way.. | 17:10 |
| timburke | dewanee_: is there any logging from auth_token or keystoneauth middlewares? another option would be to just remove bulk from the pipeline; the client will only try it we see bulk delete is available | 17:11 |
| timburke | i'll also try to reproduce locally later, but i'm afraid i'm a bit tied up at the moment... | 17:11 |
| *** ChubYann has joined #openstack-swift | 17:15 | |
| RobGThai | The reason I wanted to avoid doing that is because the LockTimeout. | 17:19 |
| RobGThai | I'm under impression that PUT container causing account to sync/replicate. | 17:20 |
| dewanee_ | timburke, I'll try to provide some additional info next week. Bye all | 17:25 |
| *** mvk has quit IRC | 17:25 | |
| timburke | dewanee_: sounds good. hopefully some of the suggestions will help in the meantime | 17:25 |
| *** tesseract has quit IRC | 17:26 | |
| *** stradling has quit IRC | 17:50 | |
| *** mvk has joined #openstack-swift | 17:57 | |
| *** stradling has joined #openstack-swift | 18:10 | |
| *** Sukhdev has joined #openstack-swift | 18:26 | |
| *** mgagne_ has joined #openstack-swift | 18:35 | |
| *** mgagne_ is now known as Guest89317 | 18:35 | |
| clayg | RobGThai: I'm not sure the locktimeout is strictly "because" of the PUT container requests... are you running your account and container databases on the same devices as the object data? rotational media? (i.e. NOT ssds) | 18:35 |
| clayg | RobGThai: anyway, you *can* give self.service.upload a list of *multiple* objects - in that case there should only be the *one* PUT container (which should be fast/cheap) | 18:37 |
| clayg | oh... sorry you're already giving it "A list of treasure to upload" | 18:37 |
| RobGThai | We are using it to back end-user upload tho. | 18:38 |
| clayg | so yeah, i'm not sure the single container PUT is really so much of an error - the *client* doesn't see LockTimeout - what is the problem from the client side? | 18:38 |
| RobGThai | So bundling multiple files isn't really the option. | 18:38 |
| RobGThai | It took too long then timeout | 18:38 |
| RobGThai | Like over a minute. | 18:39 |
| clayg | oic, the container create tries to update the account syncronously ... maybe you need to tune the account/container workers? | 18:39 |
| RobGThai | By tuning you mean account sync worker? | 18:40 |
| RobGThai | We currently set it to 32 on our 16 CPU node. | 18:41 |
| clayg | does that particulary account have many millions of containers or something? | 18:41 |
| RobGThai | We created about 100 containers per day. | 18:42 |
| clayg | the syncronous account update on container put should timeout after node_timeout - you could tune down the container node_timeout - but honestly even on spinning disks I don't think i've seen accounts hit timeouts much | 18:43 |
| RobGThai | Right now it's about tens of thousand at most | 18:43 |
| clayg | 10-100K is *fine* | 18:43 |
| clayg | probably many 100K is fine - even on spinning disk - but if the disks are the same as the object disks - it just depends on the client access and ongoing replication | 18:44 |
| RobGThai | Even if I have around 20K files upload per day? | 18:44 |
| RobGThai | We separated objects into itsown disk. | 18:45 |
| clayg | with containers that have 4M - 10M rows or more - when they have to stop adding entries to .pending and insert them into the index it can make the request slow (which can cause other updates to that container to hit the locktimeout trying to add entries to the .pending) | 18:45 |
| clayg | the same process holds for the account - but it's much less common - the cardinality of objects in containers tends to be many order of magnitude greater than the cardinaility of containers in the account | 18:46 |
| clayg | ... but everyone's data schema is different - so I just thought i'd ask | 18:46 |
| *** Guest89317 has quit IRC | 18:46 | |
| RobGThai | The maximum number of file in one container is about 150K. | 18:47 |
| clayg | anyway - your worker count sounds fine - you could try to increase it if you have a particularlly high request count - but it sounds like maybe it's just disk i/o contention in general (which eventlet servers hate, grrr) | 18:47 |
| clayg | RobGThai: 150K is fine! | 18:47 |
| clayg | RobGThai: so but you have objects & accounts & containers all on the same disks yes? | 18:47 |
| RobGThai | Account & container on one with object on another. | 18:48 |
| clayg | oh... that's good! | 18:48 |
| clayg | and the one disk (per node?) that is for account & container are... ssd or rotational? | 18:49 |
| RobGThai | Yes, rotational disk. | 18:49 |
| RobGThai | One account/container disk and ten object disks per node. | 18:50 |
| RobGThai | No RAID. | 18:50 |
| clayg | that should be a pretty good setup from my experience - i'm sorry you're having trouble :\ | 18:54 |
| clayg | OTOH, I'm not sure I would expect there would "never" be locktimeout on the accounts & containers - but I would expect you to see it *more* on containers than accounts - so that's weird maybe | 18:55 |
| RobGThai | We also see it on containers. | 18:56 |
| RobGThai | We didn't expect a perfect scenario though. It just unable to serve our 150K uploads per day right now. | 18:57 |
| clayg | well - it could mean your request rate to that layer is generating more iops than those spindles want to support - esp if you only have same number of devices as replicas :\ | 18:57 |
| clayg | well - that part is weird - even if a udpate fails here and there it should only take *seconds* to timeout at most - and you *could* tune those down | 18:57 |
| RobGThai | We have 4 nodes with 3 replica. | 18:58 |
| clayg | ... but the requests should *finish* regardless, and that work load seems pretty nominal available resources :\ | 18:58 |
| RobGThai | Everything I read so far seems to be saying that. | 18:58 |
| RobGThai | Could it be happening to a specific version? | 18:59 |
| RobGThai | We are on Mitaka. | 19:00 |
| clayg | well... idk - it could be close-ish - 150K ops is a ~ hundred a second yeah? with write amplification across three spindles... each sqlite update is going to be a couple of disk iops at best (read/modify/write page) | 19:00 |
| clayg | updating to a later version always helps | 19:00 |
| clayg | has the load been ramping up over time? did you use to see less timeouts? either way if you expect usage to increase over time... | 19:01 |
| clayg | i need to step away - the timeout is telling you the databases are hitting some contention and they're busy - you can throw hardware at it - you can try to tweak/tune - you can try to change the access pattern | 19:01 |
| RobGThai | The load has been about the same since the last two months AFAIK. | 19:01 |
| clayg | ... and it only recently started? if it was fine then it went south something changed :P | 19:02 |
| clayg | sorry, i really need to step away - i'm not 100% sure the issue is any more complicated than load at this point tho - GL! | 19:02 |
| RobGThai | Thanks clayg. Appreciate the help. | 19:02 |
| *** openstackgerrit has joined #openstack-swift | 19:18 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient master: Updated from global requirements https://review.openstack.org/89250 | 19:18 |
| *** mgagne_ has joined #openstack-swift | 19:18 | |
| *** mgagne_ is now known as Guest4366 | 19:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 19:20 |
| *** mgagne has quit IRC | 19:24 | |
| *** Guest4366 is now known as mgagne | 19:25 | |
| *** mgagne has quit IRC | 19:25 | |
| *** mgagne has joined #openstack-swift | 19:25 | |
| *** winggundamth has quit IRC | 19:29 | |
| timburke | bug or feature? doing a DELETE with ?multipart-manifest=delete to something that's not an SLO doesn't delete anything | 19:32 |
| timburke | the dlo case may be a bit special, but it'd be kinda nice if i could have a single client request to delete an object, and if it's an SLO, to clean up the segments while we're at it | 19:33 |
| *** ianychoi has quit IRC | 19:34 | |
| *** ianychoi has joined #openstack-swift | 19:35 | |
| *** pxwang has joined #openstack-swift | 19:51 | |
| *** silor has quit IRC | 19:54 | |
| *** RobGThai has quit IRC | 20:00 | |
| *** brnelson has joined #openstack-swift | 20:18 | |
| clayg | timburke: sounds like a bug to me? do you get a 4XX of some kind I guess? | 20:27 |
| timburke | clayg: nope! because we're piggy-backing off the bulk deleter, i guess? 200 OK, with 400 Bad Request needing to be parsed out of the *body* | 20:28 |
| * clayg has a conniption | 20:30 | |
| timburke | looks something like http://paste.openstack.org/show/606201/ | 20:30 |
| *** SkyRocknRoll has quit IRC | 20:31 | |
| clayg | do you suppose it's always been like that - or some kind of regression? I think maybe we shouldn't have hi-jacked DELETE in this way :\ | 20:32 |
| clayg | that's probably come up before tho | 20:32 |
| clayg | TFW after 5 mins troubleshooting a partition placement hash mis-match issue through backend logs and with swift-get-nodes you realize the original path just had a typo :'( | 20:42 |
| *** vint_bra1 has joined #openstack-swift | 20:47 | |
| *** dmellado has quit IRC | 20:48 | |
| *** vint_bra has quit IRC | 20:50 | |
| brnelson | I'm sure we all have stories like that. And it's even more fun when you're stumped and ask a coworker for advice on the problem, and they instantly spot the typo. | 20:56 |
| *** stradling has quit IRC | 21:01 | |
| *** pxwang has quit IRC | 21:03 | |
| *** chlong has quit IRC | 21:09 | |
| *** chlong_ has quit IRC | 21:09 | |
| *** catintheroof has joined #openstack-swift | 21:19 | |
| *** catintheroof has quit IRC | 21:21 | |
| *** catintheroof has joined #openstack-swift | 21:21 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Update reno for stable/ocata https://review.openstack.org/434785 | 21:58 |
| clayg | timburke knows how to reno! | 22:03 |
| timburke | maybe? i guess we'll see what happens. does make me a little sad that https://review.openstack.org/#/c/455439/ isn't landed though... | 22:05 |
| patchbot | patch 455439 - openstack-infra/project-config - swift: Skip long running dsvm jobs for unrelated c... | 22:05 |
| clayg | timburke: I hear in zuul3 we get to have that stuff in our own repo! | 22:06 |
| clayg | tdasilva: do you know someone to poke to help timburke with patch 455439 | 22:07 |
| patchbot | https://review.openstack.org/#/c/455439/ - openstack-infra/project-config - swift: Skip long running dsvm jobs for unrelated c... | 22:07 |
| timburke | yay, more yaml! | 22:07 |
| clayg | yaml is the best | 22:07 |
| clarkb | fwiw we are trying to avoid those rules, they create constant confusion "why do not get any test results" or "why isn't my change merging" | 22:10 |
| * clayg bests clarkb already posted more details on the review | 22:11 | |
| clarkb | no, I haven't | 22:11 |
| * clayg owes me a dollar :'( | 22:11 | |
| clarkb | basically the current situation is correct and hard to break at the expense of generating a few extra tests here and there | 22:11 |
| clayg | i'm not sure if you mean '-1 don't do this' or '+1 meh, we have another thing that will make this so much more awesome you don't even..." | 22:12 |
| clayg | ah, so the first | 22:12 |
| clarkb | its more +0 current setup is correct, proposed change will potentially break you when you end up in situation where no tests run | 22:12 |
| clayg | you should *totally* put that on the review so timburke can abandon | 22:12 |
| clarkb | I can right the +0 :) | 22:12 |
| clayg | oic, idk, sounds bad, if you have experience that says we don't want what we think we want ... oh yeah maybe that's a +0 | 22:13 |
| clayg | timburke: do you follow why no tests running would be bad? maybe clarkb can guide you. I think the says running dsvm on doc changes is *fine* | 22:13 |
| clarkb | clayg: posted | 22:15 |
| timburke | i think i follow -- if you write so many of these rules that some changes don't trigger any tests *at all*, there's nothing for zuul to verify, so you'll never get a +1/+2, so it'll never merge | 22:17 |
| clarkb | correct | 22:17 |
| timburke | still seems like it might be worth it when specifically scoped to the hour+ jobs, but if we're worried about people coming along later wanting to prevent unit tests from running on doc-only changes or the like, maybe it's best to not even have the precedent | 22:17 |
| clarkb | and this happens often enough people show up in the infra channel and we spend time debugging only to have a sad when we discover it | 22:17 |
| clarkb | and then I try to get them to undo the rules they have and they don't want to then I get extra sad :) | 22:18 |
| clarkb | but if you have a specific use case or concern beyond generic wasted use cases ahppy to reconsider | 22:18 |
| timburke | just the wall time. all the other jobs complete in <10 mins, those ones can be as long as an hour | 22:19 |
| clarkb | timburke: I found what I think is an actual functional issue with it so I -1'd. I'm happy to +2 if thats fixed and you want to reduce rtt but won't go too crazy on optimizing when all the jobs run | 22:23 |
| timburke | clarkb: good call. i was mainly cribbing from others -- but i guess cinder, designate, horizon, etc. don't have tox targets hit by the dsvm tests? | 22:27 |
| clarkb | timburke: or if they do they are in error. Not sure which :) | 22:27 |
| clarkb | timburke: but not all projects have set up their functional tests that way so possible its not actually a problem for them | 22:27 |
| timburke | cool. i'll go abandon | 22:29 |
| clarkb | I'm happy to put it in if the argument is reducing turn around (I wasn't sure how much time would be saved) | 22:30 |
| timburke | the good news is, https://review.openstack.org/#/c/434785/ has already produced http://docs-draft.openstack.org/85/434785/2/check/gate-swift-releasenotes/af9ae98//releasenotes/build/html/ ! | 22:30 |
| patchbot | patch 434785 - swift - Update reno for stable/ocata | 22:30 |
| clarkb | but as a general rule ya I prefer to avoid skip ifs | 22:30 |
| clayg | timburke: and the version # on http://docs-draft.openstack.org/85/434785/2/check/gate-swift-releasenotes/af9ae98//releasenotes/build/html/current.html is wrong because EmilienM didn't get to add the magic commit in the gitlog? | 22:32 |
| timburke | hmm... then maybe i'll scope it more tightly (*only* the doc trees) and hope that the improved turn-around time leads to better docs since fixes will seem "cheaper" | 22:32 |
| timburke | clayg: not sure | 22:36 |
| timburke | huh. nova's version number on https://docs.openstack.org/releasenotes/nova/unreleased.html seems kinda insane -- a 15.0.0.0 RC when 15.0.2 was released as part of ocata? | 22:37 |
| mattoliverau | morning | 22:37 |
| *** xinli has quit IRC | 23:17 | |
| timburke | huh. that's a new one for me: http://logs.openstack.org/85/434785/2/check/gate-swift-tox-xfs-tmp-func-post-as-copy-ubuntu-xenial/052bd6a/console.html#_2017-04-11_22_02_36_453001 | 23:17 |
| *** vint_bra1 has quit IRC | 23:23 | |
| *** kei_yama has joined #openstack-swift | 23:29 | |
| *** catintheroof has quit IRC | 23:29 | |
| *** catintheroof has joined #openstack-swift | 23:30 | |
| *** catintheroof has quit IRC | 23:30 | |
| *** tonanhngo has quit IRC | 23:36 | |
| *** chsc has quit IRC | 23:37 | |
| *** tonanhngo has joined #openstack-swift | 23:43 | |
| *** jamielennox is now known as jamielennox|away | 23:48 | |
| *** tonanhngo has quit IRC | 23:48 | |
| *** jamielennox|away is now known as jamielennox | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!