| *** hosanai has joined #openstack-swift | 00:01 | |
| *** ChanServ sets mode: +v hosanai | 00:01 | |
| *** mweshi has joined #openstack-swift | 00:09 | |
| *** mweshi has quit IRC | 00:10 | |
| *** mweshi has joined #openstack-swift | 00:11 | |
| *** mweshi has quit IRC | 00:16 | |
| *** diogogmt has quit IRC | 00:22 | |
| *** tqtran has quit IRC | 00:25 | |
| *** mingyu has joined #openstack-swift | 00:41 | |
| *** mingyu has quit IRC | 00:45 | |
| *** mmcardle has joined #openstack-swift | 01:02 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Support multi-range GETs for static large objects. https://review.openstack.org/352657 | 01:03 |
|---|---|---|
| *** cdelatte has quit IRC | 01:04 | |
| notmyname | torgomatic: fun! | 01:05 |
| torgomatic | ...and on that note, I'm going home :p | 01:08 |
| *** klrmn has quit IRC | 01:09 | |
| *** mmcardle has quit IRC | 01:13 | |
| *** trananhkma has joined #openstack-swift | 01:17 | |
| *** kota_ has quit IRC | 01:20 | |
| *** kota_ has joined #openstack-swift | 01:22 | |
| *** ChanServ sets mode: +v kota_ | 01:22 | |
| zhengyin | good morning, everyone | 01:24 |
| zhengyin | ChanServ,koat_:good morning, kota, ChanServ | 01:25 |
| *** chlong has joined #openstack-swift | 01:27 | |
| zaitcev | guys | 01:42 |
| *** amitkqed has quit IRC | 01:43 | |
| *** amitkqed has joined #openstack-swift | 01:43 | |
| zaitcev | Do we have people running with fast-post nowadays? object_post_as_copy=false I mean | 01:44 |
| MooingLemur | I'm not | 01:50 |
| zaitcev | I'm going to enable it for new installations. | 01:50 |
| zaitcev | The question is, does the container sync work with it now? | 01:51 |
| MooingLemur | That's one of the reasons I had it set to false | 01:51 |
| zaitcev | I see... | 01:51 |
| MooingLemur | I don't know if it can work with container sync now, but at installation time it didn't | 01:52 |
| *** mingyu has joined #openstack-swift | 01:56 | |
| *** mingyu has quit IRC | 02:00 | |
| *** diogogmt has joined #openstack-swift | 02:17 | |
| *** klrmn has joined #openstack-swift | 02:22 | |
| *** mmcardle has joined #openstack-swift | 02:46 | |
| *** mmcardle has quit IRC | 02:49 | |
| jrichli | zaitcev, MooingLemur: the latest swift does allow fast-post with container sync. See merged patch 295739 | 02:56 |
| patchbot | jrichli: https://review.openstack.org/#/c/295739/ - swift - Docs: Container sync does not require POST-as-COPY (MERGED) | 02:56 |
| *** mingyu has joined #openstack-swift | 02:57 | |
| *** mingyu has quit IRC | 03:02 | |
| jrichli | the commit message there explains that it was patch 135380 that actually allowed for the combination to work | 03:02 |
| patchbot | jrichli: https://review.openstack.org/#/c/135380/ - swift - Update container on fast-POST (MERGED) | 03:02 |
| *** mmcardle has joined #openstack-swift | 03:15 | |
| *** mmcardle has quit IRC | 03:16 | |
| *** tqtran has joined #openstack-swift | 03:22 | |
| *** acoles_ is now known as acoles | 03:23 | |
| *** acoles is now known as acoles_ | 03:26 | |
| *** tqtran has quit IRC | 03:27 | |
| *** hurricanerix has quit IRC | 03:30 | |
| *** ppai has joined #openstack-swift | 03:57 | |
| mahatic_ | good morning | 04:01 |
| *** chromatin has quit IRC | 04:05 | |
| *** klrmn has quit IRC | 04:08 | |
| *** links has joined #openstack-swift | 04:11 | |
| *** jith_ has quit IRC | 04:11 | |
| *** mingyu has joined #openstack-swift | 04:12 | |
| *** nadeem has joined #openstack-swift | 04:15 | |
| *** diogogmt has quit IRC | 04:16 | |
| openstackgerrit | KATO Tomoyuki proposed openstack/swift: Remove the duplicated word 'be' https://review.openstack.org/352694 | 04:18 |
| *** psachin has joined #openstack-swift | 04:20 | |
| jrichli | mahatic_: good morning! | 04:27 |
| mahatic_ | jrichli: o/ | 04:28 |
| *** mingyu has quit IRC | 04:33 | |
| *** chrichip has joined #openstack-swift | 04:33 | |
| *** mingyu has joined #openstack-swift | 04:35 | |
| *** nadeem has quit IRC | 04:48 | |
| *** abhitechie has joined #openstack-swift | 04:52 | |
| *** abhinavtechie has joined #openstack-swift | 05:02 | |
| *** abhitechie has quit IRC | 05:02 | |
| charz | kota_: just one question for swift3, did you remember v4 signature with virutal-host style work or not? | 05:03 |
| *** ppai has quit IRC | 05:04 | |
| *** david-lyle_ has joined #openstack-swift | 05:11 | |
| *** david-lyle has quit IRC | 05:13 | |
| *** vint_bra has joined #openstack-swift | 05:15 | |
| *** rcernin has joined #openstack-swift | 05:21 | |
| *** ChubYann has quit IRC | 05:35 | |
| *** vint_bra has quit IRC | 05:36 | |
| *** ppai has joined #openstack-swift | 05:41 | |
| openstackgerrit | Merged openstack/swift: Raise 412 response on expirer https://review.openstack.org/326903 | 05:50 |
| *** mweshi has joined #openstack-swift | 05:54 | |
| *** mweshi has quit IRC | 05:55 | |
| *** mweshi has joined #openstack-swift | 05:56 | |
| *** _JZ_ has joined #openstack-swift | 06:06 | |
| *** maestropandy has joined #openstack-swift | 06:10 | |
| *** tqtran has joined #openstack-swift | 06:25 | |
| *** tqtran has quit IRC | 06:29 | |
| *** chlong is now known as chlong|mtg | 06:32 | |
| *** mmcardle has joined #openstack-swift | 06:36 | |
| *** geaaru has joined #openstack-swift | 06:38 | |
| *** pcaruana has joined #openstack-swift | 06:39 | |
| *** mingyu has quit IRC | 06:40 | |
| *** mingyu has joined #openstack-swift | 06:40 | |
| *** silor has joined #openstack-swift | 06:43 | |
| *** tesseract- has joined #openstack-swift | 06:45 | |
| *** mingyu has quit IRC | 06:45 | |
| *** maestropandy has left #openstack-swift | 06:48 | |
| *** tqtran has joined #openstack-swift | 06:52 | |
| *** mweshi has quit IRC | 06:59 | |
| *** abhitechie has joined #openstack-swift | 07:05 | |
| *** abhinavtechie has quit IRC | 07:05 | |
| *** silor1 has joined #openstack-swift | 07:13 | |
| charz | okay, I think that's a bug in swift3 | 07:14 |
| *** silor has quit IRC | 07:16 | |
| *** silor1 is now known as silor | 07:16 | |
| *** oshritf has joined #openstack-swift | 07:18 | |
| *** daemontool has joined #openstack-swift | 07:21 | |
| *** rledisez has joined #openstack-swift | 07:25 | |
| *** daemontool_ has joined #openstack-swift | 07:28 | |
| *** daemontool has quit IRC | 07:28 | |
| *** david-lyle has joined #openstack-swift | 07:43 | |
| *** david-lyle_ has quit IRC | 07:46 | |
| *** takashi has joined #openstack-swift | 07:51 | |
| *** tqtran has quit IRC | 07:58 | |
| charz | kota_: timburke https://review.openstack.org/#/c/352755/ | 08:05 |
| patchbot | charz: patch 352755 - swift3 - Fix 403 SignatureDoesNotMatch when use v4 signatur... | 08:05 |
| *** mingyu has joined #openstack-swift | 08:08 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/swift: [WIP] Testing latest u-c https://review.openstack.org/318441 | 08:10 |
| *** SkyRocknRoll has joined #openstack-swift | 08:13 | |
| *** mvk has quit IRC | 08:26 | |
| *** geaaru has quit IRC | 08:38 | |
| *** acoles_ is now known as acoles | 08:43 | |
| *** maestropandy has joined #openstack-swift | 08:48 | |
| *** geaaru has joined #openstack-swift | 08:51 | |
| *** mvk has joined #openstack-swift | 08:53 | |
| *** oshritf has quit IRC | 08:57 | |
| *** SkyRocknRoll has quit IRC | 08:59 | |
| *** maestropandy1 has joined #openstack-swift | 08:59 | |
| *** maestropandy1 has left #openstack-swift | 08:59 | |
| *** maestropandy has quit IRC | 09:00 | |
| *** ouchkernel has quit IRC | 09:02 | |
| *** ouchkernel has joined #openstack-swift | 09:08 | |
| *** chlong|mtg has quit IRC | 09:10 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:16 | |
| *** SkyRocknRoll has quit IRC | 09:22 | |
| *** oshritf has joined #openstack-swift | 09:37 | |
| *** hosanai has quit IRC | 09:37 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:40 | |
| *** chlong|mtg has joined #openstack-swift | 09:47 | |
| *** early has quit IRC | 10:06 | |
| *** early has joined #openstack-swift | 10:06 | |
| openstackgerrit | Merged openstack/swift: Remove the duplicated word 'be' https://review.openstack.org/352694 | 10:15 |
| *** mvk has quit IRC | 10:25 | |
| openstackgerrit | Merged openstack/swift: Make the logger information format https://review.openstack.org/351132 | 10:27 |
| openstackgerrit | Merged openstack/swift: [install-guide] Include environment-networking https://review.openstack.org/346925 | 10:27 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 10:32 |
| *** mvk has joined #openstack-swift | 10:38 | |
| *** chromatin has joined #openstack-swift | 11:15 | |
| *** maestropandy has joined #openstack-swift | 11:16 | |
| *** maestropandy has quit IRC | 11:20 | |
| *** thebloggu has joined #openstack-swift | 11:26 | |
| *** thebloggu has left #openstack-swift | 11:27 | |
| *** thebloggu has joined #openstack-swift | 11:34 | |
| *** abhitechie has quit IRC | 11:52 | |
| *** pkovar has joined #openstack-swift | 12:01 | |
| *** SkyRocknRoll has quit IRC | 12:09 | |
| *** diogogmt has joined #openstack-swift | 12:14 | |
| *** trananhkma has quit IRC | 12:16 | |
| *** Jeffrey4l has quit IRC | 12:23 | |
| *** maestropandy has joined #openstack-swift | 12:23 | |
| *** maestropandy has left #openstack-swift | 12:23 | |
| *** kei_yama has quit IRC | 12:31 | |
| *** bill_az has joined #openstack-swift | 12:35 | |
| *** baojg has joined #openstack-swift | 12:43 | |
| *** catintheroof has joined #openstack-swift | 12:46 | |
| tdasilva | good morning | 12:47 |
| *** baojg has quit IRC | 12:49 | |
| *** pkovar has left #openstack-swift | 12:57 | |
| *** baojg has joined #openstack-swift | 12:59 | |
| *** klamath has joined #openstack-swift | 13:00 | |
| *** klamath has quit IRC | 13:00 | |
| *** zul_ has quit IRC | 13:00 | |
| *** klamath has joined #openstack-swift | 13:00 | |
| *** vint_bra has joined #openstack-swift | 13:01 | |
| *** vinsh has quit IRC | 13:01 | |
| *** baojg has quit IRC | 13:03 | |
| *** takashi has quit IRC | 13:04 | |
| *** zul_ has joined #openstack-swift | 13:05 | |
| *** links has quit IRC | 13:06 | |
| *** vint_bra has quit IRC | 13:06 | |
| *** vint_bra has joined #openstack-swift | 13:15 | |
| *** permalac has joined #openstack-swift | 13:18 | |
| *** vint_bra has quit IRC | 13:20 | |
| *** diogogmt has quit IRC | 13:33 | |
| *** bill_az_ has joined #openstack-swift | 13:35 | |
| *** Jeffrey4l has joined #openstack-swift | 13:41 | |
| *** Jeffrey4l has quit IRC | 13:42 | |
| *** vinsh has joined #openstack-swift | 13:44 | |
| *** cdelatte has joined #openstack-swift | 13:59 | |
| *** jistr is now known as jistr|debug | 14:00 | |
| *** airdisa has joined #openstack-swift | 14:04 | |
| *** ppai has quit IRC | 14:04 | |
| *** lcurtis has joined #openstack-swift | 14:11 | |
| thebloggu | I updated an openstack swift installation (with RDO) from liberty to mitaka. Even though everything seems to work fine after the upgrade I'm getting errors in the logs (openstack-swift-proxy) about liberasurecode missing shared dependencies (dynamic linking errors) but I do have the liberasure and python-pyeclib packages installed. From what I could tell from two different bug reports it seems to be the logging handling in pyeclib that tries to lo | 14:16 |
| thebloggu | ad the backends even if you don't use them and prints an error (https://bugzilla.redhat.com/show_bug.cgi?id=1351725 https://bugzilla.redhat.com/show_bug.cgi?id=1351725) | 14:16 |
| openstack | bugzilla.redhat.com bug 1351725 in openstack-swift "Swift proxy log: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory" [Medium,Assigned] - Assigned to zaitcev | 14:16 |
| thebloggu | Sorry, the second link should be https://bitbucket.org/tsg-/liberasurecode/issues/28/libshssso-no-such-file-or-director | 14:17 |
| *** chsc has joined #openstack-swift | 14:25 | |
| *** _JZ_ has quit IRC | 14:25 | |
| *** diogogmt has joined #openstack-swift | 14:26 | |
| *** asettle has joined #openstack-swift | 14:36 | |
| *** Jeffrey4l has joined #openstack-swift | 14:42 | |
| *** catintheroof has quit IRC | 14:47 | |
| *** catintheroof has joined #openstack-swift | 14:48 | |
| *** mingyu has quit IRC | 14:50 | |
| *** mingyu has joined #openstack-swift | 14:51 | |
| *** catintheroof has quit IRC | 14:53 | |
| *** oshritf has quit IRC | 14:56 | |
| *** _JZ_ has joined #openstack-swift | 14:58 | |
| *** hurricanerix has joined #openstack-swift | 14:59 | |
| *** vint_bra has joined #openstack-swift | 15:01 | |
| *** arch-nemesis has joined #openstack-swift | 15:04 | |
| *** portante has quit IRC | 15:08 | |
| *** portante has joined #openstack-swift | 15:11 | |
| *** rfinn has joined #openstack-swift | 15:16 | |
| *** links has joined #openstack-swift | 15:16 | |
| *** asettle is now known as Apsettle | 15:22 | |
| *** jistr|debug is now known as jistr | 15:23 | |
| *** catintheroof has joined #openstack-swift | 15:25 | |
| *** Apsettle is now known as asettle | 15:30 | |
| *** mingyu_ has joined #openstack-swift | 15:36 | |
| *** mingyu has quit IRC | 15:39 | |
| *** nadeem has joined #openstack-swift | 15:39 | |
| *** nadeem has quit IRC | 15:40 | |
| *** nadeem has joined #openstack-swift | 15:44 | |
| jrichli | thebloggu: you may also be interested in https://bitbucket.org/tsg-/liberasurecode/issues/23/spurious-messages-in-syslog | 15:44 |
| lcurtis | hello all...still trying to wade through disk i/o and rebalance contention issues | 15:45 |
| lcurtis | would there be a case whereby if disk utilization above 85% for example, auditor and replicator never idle? | 15:46 |
| lcurtis | i am getting arguments on team that we should not be above 55% full on disks | 15:46 |
| lcurtis | since when we do we see handoffs occur on ingest | 15:46 |
| *** pcaruana has quit IRC | 15:48 | |
| thebloggu | jrichli: yes, thank you. so there's currently no option available, in case you don't use those backends, to omit those errors right? | 15:49 |
| *** rcernin has quit IRC | 15:50 | |
| *** tesseract- has quit IRC | 15:50 | |
| jrichli | thebloggu: Seems to be the case. You know as much as I do on this issue. | 15:52 |
| *** psachin has quit IRC | 15:54 | |
| *** psachin has joined #openstack-swift | 15:55 | |
| *** SkyRocknRoll has joined #openstack-swift | 15:57 | |
| *** mingyu_ has quit IRC | 16:03 | |
| *** mingyu has joined #openstack-swift | 16:04 | |
| *** SkyRocknRoll has quit IRC | 16:16 | |
| *** psachin has quit IRC | 16:17 | |
| *** nadeem has quit IRC | 16:22 | |
| *** nadeem has joined #openstack-swift | 16:27 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:30 | |
| *** nadeem has quit IRC | 16:32 | |
| *** mmcardle has joined #openstack-swift | 16:35 | |
| timburke | thanks charz -- i'll take a look today | 16:39 |
| notmyname | good morning | 16:39 |
| *** rledisez has quit IRC | 16:39 | |
| *** nadeem has joined #openstack-swift | 16:40 | |
| charz | timburke: thx, that's a bit wired. My thought is that should have same behavior like v2. | 16:40 |
| *** asettle has quit IRC | 16:51 | |
| *** asettle has joined #openstack-swift | 16:52 | |
| *** asettle has quit IRC | 16:56 | |
| *** asettle has joined #openstack-swift | 17:05 | |
| *** mvk has quit IRC | 17:09 | |
| *** asettle has quit IRC | 17:10 | |
| *** mmcardle has quit IRC | 17:18 | |
| *** asettle has joined #openstack-swift | 17:19 | |
| *** asettle has quit IRC | 17:23 | |
| *** klrmn has joined #openstack-swift | 17:23 | |
| *** geaaru has quit IRC | 17:23 | |
| *** mingyu has quit IRC | 17:28 | |
| *** zul_ has quit IRC | 17:30 | |
| *** asettle has joined #openstack-swift | 17:32 | |
| *** zul has joined #openstack-swift | 17:34 | |
| *** mvk has joined #openstack-swift | 17:36 | |
| openstackgerrit | David Goetz proposed openstack/swift: go: make moveparts work for both kinds of ring change https://review.openstack.org/353038 | 17:36 |
| *** daemontool_ has quit IRC | 17:36 | |
| *** asettle has quit IRC | 17:38 | |
| timur | clayg: did we not release the new version of liberasure code and pyeclib that makes use of it? | 17:39 |
| *** zul has quit IRC | 17:40 | |
| timur | thebloggu: the new version of liberasure code (1.2.0 -- https://github.com/openstack/liberasurecode/releases/tag/1.2.0) and pyeclib (1.2.1 -- https://github.com/openstack/pyeclib/releases/tag/1.2.1) resolve the spurrious errors related to discovering the backends | 17:42 |
| *** pcaruana has joined #openstack-swift | 17:44 | |
| *** ChubYann has joined #openstack-swift | 17:45 | |
| timur | I guess it's now a matter of bumping the redhat packages | 17:45 |
| *** asettle has joined #openstack-swift | 17:46 | |
| thebloggu | timur: great. does these updates usually make it to the current release repositories? or will only be released with the next openstack version? | 17:47 |
| timur | I'm not actually sure about the release process for those :\ | 17:48 |
| timur | I only fixed the actual code, but not sure how/when it gets packaged | 17:48 |
| thebloggu | timur: sure, thank you for your help. i'll probably check with the packagers (rdo in this case) then | 17:49 |
| timur | anytime! sounds good | 17:49 |
| openstackgerrit | Merged openstack/swift: Fix Python 3 issues in diskfile https://review.openstack.org/347390 | 17:51 |
| *** silor has quit IRC | 17:51 | |
| *** asettle has quit IRC | 17:51 | |
| *** zul has joined #openstack-swift | 17:52 | |
| *** asettle has joined #openstack-swift | 17:59 | |
| *** asettle has quit IRC | 18:04 | |
| *** asettle has joined #openstack-swift | 18:05 | |
| openstackgerrit | Merged openstack/swift: Fix repeated OPTIONS in swift-dispersion-report manpage https://review.openstack.org/349910 | 18:07 |
| openstackgerrit | Merged openstack/swift: Fix gettext_ calls https://review.openstack.org/336743 | 18:07 |
| openstackgerrit | Merged openstack/swift: NIT: fixing inconsistent naming of OpenStack Swift https://review.openstack.org/351388 | 18:16 |
| openstackgerrit | Merged openstack/swift: Have py35 tox env match py34 https://review.openstack.org/337802 | 18:20 |
| timburke | go torgomatic! merge it all! | 18:21 |
| *** acoles is now known as acoles_ | 18:25 | |
| *** silor has joined #openstack-swift | 18:26 | |
| *** thebloggu has quit IRC | 18:31 | |
| *** bill_az has quit IRC | 18:32 | |
| *** silor has quit IRC | 18:33 | |
| openstackgerrit | Merged openstack/swift: Raise ValueError if empty value coming into encrypt_header_val https://review.openstack.org/337000 | 18:34 |
| *** oshritf has joined #openstack-swift | 18:39 | |
| *** mmcardle has joined #openstack-swift | 18:39 | |
| *** links has quit IRC | 18:40 | |
| *** david-lyle has quit IRC | 18:41 | |
| *** david-lyle has joined #openstack-swift | 18:42 | |
| *** treyd has quit IRC | 18:44 | |
| *** alpha_ori has quit IRC | 18:44 | |
| *** SkyRocknRoll has quit IRC | 18:45 | |
| *** alpha_ori has joined #openstack-swift | 18:50 | |
| *** rcernin has joined #openstack-swift | 18:51 | |
| *** treyd has joined #openstack-swift | 18:51 | |
| *** mmcardle has quit IRC | 18:59 | |
| *** tqtran has joined #openstack-swift | 19:03 | |
| *** tqtran has quit IRC | 19:09 | |
| *** catintheroof has quit IRC | 19:12 | |
| *** kragniz has joined #openstack-swift | 19:14 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Support multi-range GETs for static large objects. https://review.openstack.org/352657 | 19:25 |
| *** rfinn has quit IRC | 19:26 | |
| *** rfinn has joined #openstack-swift | 19:36 | |
| *** david-lyle_ has joined #openstack-swift | 19:41 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: add reminder how to run debug func tests https://review.openstack.org/351873 | 19:42 |
| *** david-lyle has quit IRC | 19:43 | |
| openstackgerrit | Merged openstack/swift: Make swift-oldies py3-compatible https://review.openstack.org/344963 | 19:48 |
| *** nadeem has quit IRC | 20:00 | |
| *** nadeem has joined #openstack-swift | 20:03 | |
| *** bill_az_ has quit IRC | 20:06 | |
| *** david-lyle_ is now known as david-lyle | 20:12 | |
| *** cdelatte has quit IRC | 20:24 | |
| *** airdisa has quit IRC | 20:28 | |
| *** asettle has quit IRC | 20:29 | |
| *** asettle has joined #openstack-swift | 20:30 | |
| *** asettle has quit IRC | 20:30 | |
| *** asettle has joined #openstack-swift | 20:31 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** dfflanders has joined #openstack-swift | 20:35 | |
| *** Jeffrey4l has joined #openstack-swift | 20:36 | |
| openstackgerrit | David Goetz proposed openstack/swift: go: make moveparts work for both kinds of ring change https://review.openstack.org/353038 | 20:37 |
| *** gyee has joined #openstack-swift | 20:37 | |
| openstackgerrit | Nandini Tata proposed openstack/swift: Add man page for swift.conf https://review.openstack.org/353097 | 20:38 |
| openstackgerrit | David Goetz proposed openstack/swift: go: make moveparts work for both kinds of ring change https://review.openstack.org/353038 | 20:39 |
| *** nadeem has quit IRC | 20:39 | |
| *** nadeem has joined #openstack-swift | 20:39 | |
| openstackgerrit | David Goetz proposed openstack/swift: go: make moveparts work for both kinds of ring change https://review.openstack.org/353038 | 20:43 |
| *** mmcardle has joined #openstack-swift | 20:43 | |
| *** mmcardle has quit IRC | 20:53 | |
| clayg | i have no idea what Alexandr Porunov on the ML is talking about - why would you want to *rename* the stats? how do you keep a mapping of the semantic value to your new made up name? | 21:14 |
| clayg | does he mean like host prefixes and stuff? I don't know how OpenTSDB werx - isn't it just a metrics dump? | 21:14 |
| clayg | what cli utility? | 21:14 |
| clayg | timur: i released liberasure stuff - it's awesome now | 21:14 |
| clayg | timur: zaitcev knows stuff about rdo | 21:15 |
| clayg | timur: you can even try something like this, watch | 21:16 |
| clayg | zaitcev: when are you going to release the rdo fixes for bugzilla.redhat.com bug 1351725 (already assigned to you!) | 21:16 |
| openstack | bug 1351725 in lrt-crashes "lrt test found crash file _usr_lib_arm-linux-gnueabihf_indicator-network_indicator-network-service.32011.crash" [Critical,New] https://launchpad.net/bugs/1351725 | 21:16 |
| clayg | yeah, no patchbot | 21:16 |
| zaitcev | timur, clayg: I am responsible for the bug, but I don't have a solution. | 21:16 |
| zaitcev | well, except "upgrade to latest" | 21:18 |
| clayg | zaitcev: acctually those don't look like same syslog messages we were seeing? | 21:18 |
| ahale | idk about that thread or opentsdb but i rename a load of metrics from with account in them to something human readable. like long uuid -> "glance" | 21:18 |
| *** oshritf has quit IRC | 21:19 | |
| clayg | ahale: and that's like a graphite config thing right? IIRC it has that sweetness built in... but that's a good point - we do some similar stuff because our nodes report via uuid and the controller has to patch metrics to clusters | 21:20 |
| zaitcev | I need to look at this more seriously. I have a test cluster that was throwing some messages about Jerasure missing, but they went away with liberasurecode 1.1.0 or something like that. | 21:21 |
| *** oshritf has joined #openstack-swift | 21:21 | |
| notmyname | there was some talk about getting rid of the warnings when known backends weren't installed | 21:21 |
| ahale | oh yeah, thats all graphitey stuff, was just thinking :)( | 21:22 |
| ahale | he's probably best off hacking up informant middleware or something, thats what i do when i want more metrics since never switched when swift got statsd | 21:22 |
| notmyname | ahale: we could write a generic stats-er plugin framework for swift, and you could use whatever metrics emitter plugin you want. then you could do statsd or opentsdb or whatever | 21:23 |
| clayg | timur: zaitcev: oh nm, yeah that looks right | 21:24 |
| ahale | that sounds cool | 21:24 |
| *** rfinn has quit IRC | 21:25 | |
| notmyname | ahale: ...and when I say "swift" I really mean "all of openstack", of course | 21:25 |
| ahale | oh so like you install all of oslo and end up only being able to send to gnocci ? | 21:26 |
| clayg | ahale: why you are even awake? i hope you're drinking. | 21:27 |
| ahale | lol | 21:28 |
| timur | notmyname: the warnings do go away with the latest, since it should no longer blindly pollute syslog with ERROR level messages. I don't know what else to do there aside from getting everyone to use the latest and greatest 1.2/1.2.1 releases | 21:31 |
| *** zigo has quit IRC | 21:32 | |
| *** zigo has joined #openstack-swift | 21:35 | |
| *** admin6 has joined #openstack-swift | 21:40 | |
| clayg | does anyone know how gerrit works can explain what the little yellow dot next to "Parent(s)" means on patch 319387? | 21:49 |
| patchbot | clayg: https://review.openstack.org/#/c/319387/ - swift - swift-ring-builder output corrected for ipv6 | 21:49 |
| clayg | like it means *something* is out of date - does it *need* a rebase? | 21:49 |
| *** rfinn has joined #openstack-swift | 21:49 | |
| timur | clayg: mouse over it. alt text "not current - rebase possible" | 21:50 |
| timur | I think it would need to be rebased to merge if it's fast-forward only on merges? | 21:51 |
| timur | although rebases *during* the review break the delta between patchets in terrible ways :\ | 21:51 |
| clayg | timur: i'd guess we merge --no-ff i *always* see the merge commits | 21:51 |
| timur | oh, then it should be ok? I think gerrit will complain if it has a merge conflict when trying to merge it in | 21:52 |
| clayg | timur: yeah idk, there was a comment about some diff polltion that was confusing to the reviewer - i just saw the yellow dot and I normally think that means you have to throw chicken bones at it or something | 21:52 |
| timur | hahahaha | 21:52 |
| timur | "arise chicken!" | 21:52 |
| ntata | clayg, so, I rebased it once when the ip validations methods got moved.. | 21:53 |
| pdardeau | ntata: that may be all well and good, but what did you do with the chicken bones? :-) | 21:54 |
| clayg | ok, so I think maybe the yellow dot is fine, but I don't understand I think shashi's comment about the previous change - but I think she was just confused - so that's ok | 21:56 |
| *** mmcardle has joined #openstack-swift | 21:59 | |
| *** rcernin has quit IRC | 22:02 | |
| *** asettle has quit IRC | 22:04 | |
| clayg | ntata: i still feel guilty about `header_line, print_dev_f = _make_display_device_table(builder)` :\ | 22:05 |
| *** diogogmt has quit IRC | 22:05 | |
| clayg | I think it *works* great - the result looks fabulous | 22:05 |
| admin6 | Hi there, I have plenty of these logs into the object reconstructor : object-reconstructor: 10.10.2.53:6000/s03z3ecd01/34782 Unexpected response: ":ERROR: 0 '15 seconds: /srv/node/s03z3ecd01/.lock’ ». Do you know what they means? | 22:05 |
| clayg | so normally I wouldn't care *at all* - I have no obvious idea how to make it better | 22:05 |
| clayg | but... i feel guilty that maybe you think "oh maybe this mess of closures and functions returning functions is what clay thinks is good looking code" - it wasn't that | 22:06 |
| clayg | i just threw it over the wall because I was getting frustrated at how difficult it was to draw some good lines for method extraction | 22:07 |
| clayg | anyway, sorry - i'm not sure if my point is to subtle for irce | 22:08 |
| clayg | *irc | 22:08 |
| ntata | I think it this way too - I definitely like the idea of separating _make_display_device_table rather than cluttering *that huge formatting code just to print neatly* in the method that's constructing the output | 22:11 |
| clayg | yeah, something has to be better than nothing | 22:12 |
| clayg | ntata: ok, i'd like someone else to think about adding the stub files to the tests tree - i like that too - but it's new | 22:13 |
| admin6 | I also have a lot of reconstruct error like this after my last rebalance : object-reconstructor: Unable to get enough responses (8/9) to reconstruct 10.10.2.54:6000/s04z4ecd08/54781/AUTH_Zoo/20160808_O1/129617/21662868/original.JPG policy#1 frag#11 with ETag 9d8bb41cac43ecf770e20906cd86153b. Which let me think that the object is lost and can’t be reconstructed. However, I can do a GET on the object and it is perfectly valid. Could that be linked | 22:13 |
| admin6 | the Unexpected response error? | 22:13 |
| clayg | timburke: you have opinions on things - patch 319387 puts files that aren't .py in test/unit/cli - you care? | 22:14 |
| patchbot | clayg: https://review.openstack.org/#/c/319387/ - swift - swift-ring-builder output corrected for ipv6 | 22:14 |
| *** sebl_ has joined #openstack-swift | 22:14 | |
| ntata | I agree! It does need some thought with gate tests | 22:14 |
| clayg | admin6: oh yah, that's all related, yuk | 22:15 |
| clayg | admin6: soo.... basically... i think reconstruction is like currently probably super busy and there's a good chance it will eventually work itself out :\ | 22:16 |
| clayg | i mean the timeout is pointing at "overloaded" - but the problem is really a efficiency issue | 22:16 |
| admin6 | clayg: yeah the reconstructor seem to be very busy, but it is taking about 250 hours to perform a full reconstruction pass | 22:17 |
| clayg | admin6: do you do anything kind of partition placement monioring? like dispersion-report or like http://docs.openstack.org/developer/swift/admin_guide.html#checking-handoff-partition-distribution | 22:18 |
| clayg | admin6: rofl - sorry it's not funny :'( | 22:18 |
| clayg | it's really not funny :'( | 22:19 |
| clayg | admin6: how many devices do you have in the EC cluster - what was the extent of the capacity change? | 22:19 |
| admin6 | clayg: here is my ring definition : 262144 partitions, 12.000000 replicas, 1 regions, 4 zones, 48 devices, 0.01 balance, 0.00 dispersion | 22:21 |
| timburke | clayg: i'm not opposed. might want to make it more clear that these "stub" files are intended for use in test_ringbuilder (maybe have a 'test_ringbuilder' directory?) but this seems like a thing that might be nice for some of the ring rebalance tests | 22:21 |
| *** oshritf has quit IRC | 22:22 | |
| clayg | timburke: oh good point - i sorta thought the module and class name went in the file names | 22:23 |
| clayg | oh, i had done something sorta even more stupid -> https://review.openstack.org/#/c/345674/ | 22:24 |
| patchbot | clayg: patch 345674 - swift - crazy idea to check output (ABANDONED) | 22:24 |
| clayg | maybe the implict coupling of the test name and the stub file name is less useful than just assert(output, './path/to/stub') or something... | 22:25 |
| clayg | idk | 22:25 |
| clayg | timburke: we can probably change it later if more files start to grow and we need better organization | 22:26 |
| admin6 | clayg: here is the dipsersion verbose result : http://paste.openstack.org/show/552780/. I had 32 devices with a weight of 4000 each before rebalance, and I just add 16 disk with a weight of 400 each into the last rebalance. | 22:26 |
| clayg | ok, so it's a pretty decent capacity increase when it's all said and done? +50% | 22:27 |
| clayg | so you haven't had a rebalance finish since you pushed out this ring yeah? you're seeing 250 from the estimation log line? | 22:28 |
| admin6 | clayg: yes, but only 5% more capacity for this rebalance, which seem not to much to be able to reconstruct, isn’t it? | 22:29 |
| clayg | admin6: yeah I think you did a good job trying to manage this as a conservative adjustment | 22:30 |
| admin6 | clayg: yes, here a two reconstuctor log example : STACO2 object-reconstructor: 161445/202505 (79.72%) partitions of 2/12 (16.67%) devices reconstructed in 117100.77s (1.38/sec, 212h remaining) | 22:30 |
| admin6 | STACO1 object-reconstructor: 742821/762666 (97.40%) partitions of 7/8 (87.50%) devices reconstructed in 341089.37s (2.18/sec, 16h remaining) | 22:31 |
| clayg | how many files/gb's are we talking about here? How much real data is %5? | 22:31 |
| *** klamath has quit IRC | 22:32 | |
| admin6 | This EC ring is currently filled with 42TB of data, so 5% means 2,1TB | 22:33 |
| clayg | ok, so it's been running 32 hours - oic, it's not *really* 80% done - it's only parsed 2/12 devices :\ | 22:33 |
| admin6 | clayg: yes, 32 hours to parse only 2 over 12 devices, so as the last 4 devices are only 10% weight of the other, it is still more than 150h to perform the full reconstruction. | 22:36 |
| admin6 | clayg: do you have suggestion to reduce the overload of the reconstructor process? and it seems antithetical but do you have suggestion to accelerate it ? | 22:39 |
| *** vint_bra has quit IRC | 22:40 | |
| admin6 | clayg; anf BTW, what exactly means the concurrency value for the reconstructor? | 22:40 |
| *** vint_bra has joined #openstack-swift | 22:40 | |
| *** itlinux has joined #openstack-swift | 22:42 | |
| *** lcurtis has quit IRC | 22:45 | |
| *** vint_bra has quit IRC | 22:45 | |
| *** mmcardle has quit IRC | 22:45 | |
| *** itlinux has quit IRC | 22:46 | |
| admin6 | clayg: also on these ACO servers, the swift-container replicator is always taking 99% of CPU, and there are 24 (number of core on these servers) swift-container-server processes that take more than 10% CPU. is that normal? | 22:49 |
| clayg | it's ultimately the number of jobs that an instance of the reconstructor will run at once | 22:54 |
| clayg | what do you have concurrency set to? I was going to suggest you increase it - obvs. :D | 22:55 |
| admin6 | clayg: ok, when I first set the concurrency to 8 (which was with the default value before) it speed up the number of partition per second up to 8. but now the concurrency is still set to 8 but the artition per second goes down to around 2/sec | 22:56 |
| admin6 | I set it to 8 about 2 weeks ago | 22:57 |
| clayg | admin6: I think the default is 1? which is too low for almost any real deployment :D | 22:58 |
| admin6 | clayg, that’s what i discover 2 weeks ago ;-) | 22:58 |
| clayg | admin6: ok, so you have 8 "outgoing" syncs per node - times four nodes - that's only 32 outgoing syncs across the whole cluster | 22:59 |
| clayg | admin6: there's also some tunring on the incoming side | 22:59 |
| clayg | you have seperate replication network? | 22:59 |
| clayg | well... at least seperate processes (replication object servers) | 22:59 |
| clayg | or they all talk to the same ports in the ring? | 22:59 |
| clayg | "replication_concurrency" - check out the notes in the sample config for the object-server | 23:00 |
| admin6 | clayg: replication is done on same port but on a different ip address, however it currently a fake as both ip adress are on the same card plugged on a 10GB switch | 23:01 |
| clayg | I think you have 32 outgoing *jobs* (which i *guess* will only ever need to hold a single SSYNC connection - most of the chatter is either REPLICATION requests or GETs for fragments) | 23:02 |
| clayg | ... but if you're running the default replication_concurrency of 4 and one_per_device = True - you only have 16 slots available for those 32 jobs? | 23:02 |
| clayg | admin6: ok, but the *processes* it's the same object-server *process* as your cluster/proxies talk to? | 23:03 |
| clayg | most of our deployments use the same physical network - but it's important with ssync that it's a different object-server instance (we spin up a /etc/swift/object-server/2.conf for replication on a different port) | 23:03 |
| clayg | either way - you probably need to tune up replication_concurrency on the object server (which would *ideally* be a seperate process than the once your proxies/clients talk to) - and watch out for lp bug #1446873 | 23:04 |
| openstack | Launchpad bug 1446873 in OpenStack Object Storage (swift) "ssync doesn't work with replication_server = true" [Medium,Confirmed] https://launchpad.net/bugs/1446873 | 23:04 |
| admin6 | clayg: yes, I’ve only one object server instance per server | 23:05 |
| admin6 | clayg: do you have a good link to follow about how to separate object server process? and do you also suggest me to increase the concurrency of the object reconstructor ? | 23:07 |
| clayg | ok, well just keep turning up reconstructor concurrency and object-server replication_concurrency until it a) doesn't help b) makes your clients notice | 23:08 |
| *** chsc has quit IRC | 23:08 | |
| admin6 | clayg: ok, that seems a good way of finding the best concurrency value :-D | 23:09 |
| clayg | admin6: so the concurrency of the reconstructor and replication_concurrency need to match - they're both per server - so you *could* just try 8, 8 - 16, 16 - 32, 32 | 23:10 |
| clayg | somewhere around device count per node I can't imagine it really getting much better - but even as high as 4x device count *might* accidently sort of keep helping | 23:10 |
| clayg | but at that point you have to turn off replication_one_per_device | 23:11 |
| clayg | be sure to report back! | 23:11 |
| clayg | and i'm *guessing* well before that you'll have to spot and move to seperate replication servers | 23:11 |
| admin6 | clayg: Ok; many thanks, one last question : on these ACO servers, the swift-container replicator is always taking 99% of CPU, and there are 24 (number of core on these servers) swift-container-server processes that take more than 10% CPU. is that normal? | 23:12 |
| clayg | this is the best doc on seperate replication servers -> http://docs.openstack.org/developer/swift/replication_network.html | 23:12 |
| clayg | admin6: http://docs.openstack.org/developer/swift/admin_guide.html#checking-handoff-partition-distribution is also worth a read | 23:13 |
| clayg | the "status" lines from the replciator/reconstructor aren't as good of a proxying during rebalance as watching the handoff parts get moved across the cluster | 23:13 |
| clayg | so if you can monitor your handoff counts per device and sum it up across the cluster you can do a better job predicting how many are left and the rate at which they're being reoved | 23:14 |
| admin6 | clayg: ok. I’ll try to digest all that. many thanks for your great help and expertise. | 23:16 |
| clayg | admin6: RE container processes - you mean like 99% of *one* cpu? i don't have a production node handy at the moment - but those things are probably somewhat CPU busy and single thread - so mabye? | 23:16 |
| admin6 | clayg: yes 99% of one cpu. | 23:17 |
| clayg | idk, seems like they should be doing some cpu... are your container dbs on the object disks? is there a lot of load in wait or all user? | 23:17 |
| clayg | err.. sorry some i/o | 23:17 |
| clayg | obs. | 23:17 |
| clayg | and np | 23:17 |
| *** diogogmt has joined #openstack-swift | 23:18 | |
| admin6 | clayg: account and container db are on separated SSD devices | 23:19 |
| clayg | hrmm... ok | 23:19 |
| *** david-lyle has quit IRC | 23:20 | |
| admin6 | clayg: and this EC ring is adding about 1,2TB of new data per day, so that should load the container a bit | 23:21 |
| clayg | container load is per object not per byte (obs.) but 1TB a day is a good growth curve! hopefully we can get this rebalance tuned to keep up - but it could be tight :\ | 23:22 |
| admin6 | clayg: yes. thats the source of my questions. being able to make the ring grow at least as far as it filled in ;-) | 23:24 |
| admin6 | as fast | 23:24 |
| *** kei_yama has joined #openstack-swift | 23:27 | |
| *** vinsh has quit IRC | 23:35 | |
| clayg | admin6: fwiw i spotted checked a few nodes with decent workload and similar setup - and I don't have any container-replicators running hot - dunno the issue, you might monitor it and put it on the list to loop back too after the ingest/rebalance stuff | 23:39 |
| *** david-lyle has joined #openstack-swift | 23:39 | |
| admin6 | clayg: ok. I’ll try to investigate that deeper also. | 23:40 |
| *** nadeem has quit IRC | 23:49 | |
| *** rfinn has quit IRC | 23:50 | |
| *** hosanai has joined #openstack-swift | 23:50 | |
| *** ChanServ sets mode: +v hosanai | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!