Monday, 2017-08-28

*** baojg has quit IRC00:33
*** vint_bra has joined #openstack-swift00:48
*** tovin07_ has joined #openstack-swift01:00
kota_good morning01:45
kota_hello mattoliverau01:45
mattoliveraukota_: morning o/01:45
openstackgerritMatthew Oliver proposed openstack/swift master: Container sharding docs  https://review.openstack.org/43832202:02
openstackgerritMatthew Oliver proposed openstack/swift master: Add container sharding to Swift containers  https://review.openstack.org/49828502:02
kota_container sharding is getting progressed, nice!02:02
mattoliveraukota_: yeah, slowly :P finally getting back into it after the craziness that has been the last few months02:05
*** hoonetorg has quit IRC02:43
*** itlinux has quit IRC02:45
*** kiennt has joined #openstack-swift02:54
*** hoonetorg has joined #openstack-swift03:00
*** silor has joined #openstack-swift03:07
*** abhinavtechie has joined #openstack-swift03:07
*** abhitechie has quit IRC03:10
*** kei_yama has quit IRC03:26
*** mrhillsman has left #openstack-swift03:31
*** gkadam has joined #openstack-swift03:33
mahaticgood morning03:37
mahaticmathiasb: I'm late to the party, congrats on patch 364878 merge!03:40
patchbothttps://review.openstack.org/#/c/364878/ - swift - Retrieve encryption root secret from Barbican (MERGED)03:40
mattoliveraumahatic: morning03:56
mahaticmattoliverau: hello03:56
*** vint_bra has quit IRC03:57
*** kei_yama has joined #openstack-swift03:59
*** links has joined #openstack-swift04:00
*** silor has quit IRC04:21
*** kei_yama has quit IRC04:26
*** silor has joined #openstack-swift04:47
*** kiennt has quit IRC04:47
*** kiennt has joined #openstack-swift04:48
*** kei_yama has joined #openstack-swift04:51
*** jaosorior has joined #openstack-swift05:18
*** rcernin has joined #openstack-swift05:30
*** silor has quit IRC05:34
*** geaaru has joined #openstack-swift05:47
*** abhitechie has joined #openstack-swift05:55
*** abhinavtechie has quit IRC05:57
*** abhinavtechie has joined #openstack-swift06:14
*** abhitechie has quit IRC06:15
openstackgerritMatthew Oliver proposed openstack/swift master: Add container sharding to Swift containers  https://review.openstack.org/42390606:17
openstackgerritMatthew Oliver proposed openstack/swift master: Container sharding docs  https://review.openstack.org/43832206:17
*** bkopilov has joined #openstack-swift06:29
*** silor has joined #openstack-swift06:34
*** pcaruana has joined #openstack-swift06:36
*** gkadam_ has joined #openstack-swift06:45
*** gkadam has quit IRC06:47
*** hseipp has joined #openstack-swift06:56
*** cschwede has joined #openstack-swift06:59
*** ChanServ sets mode: +v cschwede06:59
*** oshritf has joined #openstack-swift07:01
openstackgerritlidong proposed openstack/python-swiftclient master: Update links in README  https://review.openstack.org/49833507:04
*** silor has quit IRC07:20
*** d0ugal has joined #openstack-swift07:30
*** tesseract has joined #openstack-swift07:31
*** silor has joined #openstack-swift07:45
acolesgood morning07:50
*** silor has quit IRC08:08
*** d0ugal has quit IRC08:35
*** jaosorior has quit IRC08:39
*** kiennt_ has joined #openstack-swift08:46
*** kiennt has quit IRC08:49
*** jaosorior has joined #openstack-swift08:50
*** abhinavtechie has quit IRC09:12
mathiasbthanks, mahatic! :)09:24
*** gabor_antal has quit IRC09:28
*** cschwede has quit IRC09:40
*** cschwede has joined #openstack-swift09:42
*** ChanServ sets mode: +v cschwede09:42
acolestorgomatic: clayg thanks for getting https://review.openstack.org/#/c/460073/ into shape09:53
patchbotpatch 460073 - swift - Add probe test for ssync of unexpired metadata to ... (MERGED)09:53
*** mat128 has joined #openstack-swift09:57
*** silor has joined #openstack-swift10:02
*** kiennt has joined #openstack-swift10:11
*** kei_yama has quit IRC10:12
*** kiennt_ has quit IRC10:13
*** silor has quit IRC10:19
*** silor1 has joined #openstack-swift10:19
*** silor1 is now known as silor10:21
*** kiennt has quit IRC10:27
*** tovin07_ has quit IRC10:28
*** silor has quit IRC10:34
openstackgerritMatthew Oliver proposed openstack/swift master: Small readability change to test_keystoneauth  https://review.openstack.org/49820210:35
*** CowboyPr1de has joined #openstack-swift10:37
CowboyPr1deGood morning all. I added a new region/storage node to swift cluster this weekend and am wondering if my replica count needs to be increased? https://pastebin.com/UYVZb31710:38
*** d0ugal has joined #openstack-swift10:40
openstackgerritMerged openstack/swift master: Small readability change to test_keystoneauth  https://review.openstack.org/49820210:51
*** jamielennox has quit IRC10:51
*** jamielennox has joined #openstack-swift10:52
*** baojg has joined #openstack-swift11:27
*** gabor_antal has joined #openstack-swift11:31
*** szaher has quit IRC11:32
*** szaher has joined #openstack-swift11:36
*** bkopilov has quit IRC11:39
*** lifeless has quit IRC12:15
*** lifeless has joined #openstack-swift12:17
*** cschwede has quit IRC12:24
*** cschwede has joined #openstack-swift12:26
*** ChanServ sets mode: +v cschwede12:26
*** catintheroof has joined #openstack-swift12:36
*** mat128 has quit IRC13:03
*** gkadam_ has quit IRC13:04
*** lucasxu has joined #openstack-swift13:05
*** SkyRocknRoll has joined #openstack-swift13:21
*** bkopilov has joined #openstack-swift13:37
*** mat128 has joined #openstack-swift13:56
*** mat128 has quit IRC13:57
*** abhitechie has joined #openstack-swift14:08
*** baojg has quit IRC14:11
*** baojg has joined #openstack-swift14:14
*** links has quit IRC14:22
*** geaaru has quit IRC14:52
*** gyee has joined #openstack-swift14:56
*** mat128 has joined #openstack-swift15:08
*** mat128 has quit IRC15:11
*** geaaru has joined #openstack-swift15:13
*** oshritf has quit IRC15:30
*** jistr is now known as jistr|off|mtg15:33
*** jistr|off|mtg is now known as jistr15:33
*** links has joined #openstack-swift15:33
notmynamegood morning15:42
notmynameCowboyPr1de: why are you thinking you need to add replicas?15:44
notmynameCowboyPr1de: based on the pastebin, you've got 2 region and about 1/6 of the data has a replica in the 2nd region (1 of six servers, all with 4 drives of equal weight)15:45
notmynameCowboyPr1de: there's nothing wrong with that output. but I don't know if it matches your goals or not15:46
*** pcaruana has quit IRC15:47
*** mat128 has joined #openstack-swift16:01
*** rcernin has quit IRC16:10
*** vint_bra has joined #openstack-swift16:18
*** hseipp has quit IRC16:22
*** klrmn has quit IRC16:23
*** links has quit IRC16:27
timburkegood morning16:29
*** tesseract has quit IRC16:30
*** abhinavtechie has joined #openstack-swift16:33
*** dmsimard has joined #openstack-swift16:35
dmsimardtimburke: Hi! Re: https://bugs.launchpad.net/openstack-ansible/+bug/170722016:35
openstackLaunchpad bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Low,In progress] - Assigned to Andy McCrae (andrew-mccrae)16:35
timburkedmsimard: hi! yeah, i remember that guy!16:36
*** abhitechie has quit IRC16:36
dmsimardtimburke: We'd be happy to update libec to 1.5.0 (from 1.4.0) in RDO to fix that issue for Ocata but the current upper-constraints pins it to 1.4.0: https://github.com/openstack/requirements/blob/0651044b20227dad1d8247be47614a2447c21793/upper-constraints.txt#L28616:36
dmsimardWe'd rather not bump our version of the package ahead of upstream, could you submit a bump to u-c ?16:37
timburkehappily16:37
notmynametimburke: any dependency restrictions that adds for pyeclib?16:38
dmsimardtimburke: Thanks ! Add me on the review ? dms [at] redhat16:38
timburkenotmyname: maybe? i was gonna propose both. we keep the interface stable enough that it shouldn't present any troubles for swift, and i don't know of any other openstack projects using either16:39
*** mat128 has quit IRC16:39
notmynametimburke: ya, makes sense. and now at the start of a cycle is a reasonable time to update it16:39
timburkeactually, now that i think about it more, i think requirements repo *only* has pyeclib :-)16:40
claygheyoh!16:41
notmynamegood morning clayg16:42
claygacoles: so i have this probe test for expired objects and revert (and w/ unexpired metadata w/e)16:42
timburkedmsimard: looks like pyeclib got updated as part of https://github.com/openstack/requirements/commit/a76a545 (which is included in pike) -- i guess you just need it back on ocata?16:43
claygbut it has *two* dependents (patch 498107 and patch 498107)16:44
patchbothttps://review.openstack.org/#/c/498107/ - swift - Make X-Backend-Replication consistent for HEAD16:44
claygumm... the other is p 45807016:44
patchbothttps://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers16:44
acolesclayg: looking...16:45
claygI don't even know how to express that in gerrit?  I bunch of Depends-On I guess... but how should I handle getting my working tree in order?  two merge commits follow by the change?16:45
claygi guess I can just start with figuring out what's up with the dsvm environment and p 45807016:46
patchbothttps://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers16:46
dmsimardtimburke: right, the ocata upper-constraints still has 1.4.016:48
acolesclayg: yeah I think rebase on one and Depends-On the other, but that is only useful if the 'other' is a weak dependency (i.e. you don't need the code changes for your patch to work, you just want to enforce a merge order)16:48
dmsimardtimburke: so it's a bump of upper-constraints from stable/ocata that we need16:48
*** rcernin has joined #openstack-swift16:48
acolesclayg: better may be to tell me which of the others I should review and try to get one merged ASAP16:49
notmynameclayg: acoles: if you do need to push a patch to gerrit that has 2 parents, I can teach you how (note that only cores can propose patches like that)16:49
claygwooooooo16:50
acolesnotmyname: oh, that's interesting16:50
notmynamehint: it's exactly the same as proposing merge commits to a feature branch. so you've both already done it before ;-)16:50
claygwell i need to get both of the parents working - then yeah if they're not merged I'll want to purpose the "why" patch (with two parents)16:50
notmynameIIRC `gerrit review -R ...`16:50
claygreivew *-R* - what sorcery!?16:51
timburkehttps://review.openstack.org/#/c/498521/ if anyone else cares17:01
patchbotpatch 498521 - requirements (stable/ocata) - Update PyEClib17:01
*** mat128 has joined #openstack-swift17:03
*** chlong has joined #openstack-swift17:07
*** prometheanfire has joined #openstack-swift17:14
prometheanfirewhat's with the pyeclib bump in ocata? https://review.openstack.org/49852117:14
patchbotpatch 498521 - requirements (stable/ocata) - Update PyEClib17:14
*** klrmn has joined #openstack-swift17:15
timburkeprometheanfire: there are some segfaults that have caused issues for downstream (see also https://bugs.launchpad.net/openstack-ansible/+bug/1707220), but RDO (at least) feels uncomfortable advancing beyond what's listed in (ocata's) upper-constraints17:34
openstackLaunchpad bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Low,In progress] - Assigned to Andy McCrae (andrew-mccrae)17:34
prometheanfireya, I can understand limiting to UC17:37
prometheanfirecan you update the review with bug links and better comments as to why?17:37
*** cschwede has quit IRC17:41
*** __david_ has joined #openstack-swift17:41
*** baojg has quit IRC18:09
*** baojg has joined #openstack-swift18:14
*** itlinux has joined #openstack-swift18:23
timburkemattoliverau: if you want https://review.openstack.org/#/c/464217/ you may need to go review https://review.openstack.org/#/c/464216/ too :-)18:32
patchbotpatch 464217 - swift - domain_remap: stop mangling client-provided paths18:32
patchbotpatch 464216 - swift - domain_remap: be more careful about client-path ma...18:32
*** silor has joined #openstack-swift18:37
*** SkyRocknRoll has quit IRC18:43
*** silor has quit IRC18:44
claygthat moment when you realize the problem is "you're just not thinking 4th dimensionally!"18:53
*** catintheroof has quit IRC19:04
__david_We've been working on adding a storage policy to an existing set of swift clusters in order to use multi-region functionality. The existing clusters are 3 copy with ACO rings only referencing local ACO nodes. In a staging setup with two clusters, I have a default policy in each cluster with ACO rings pointing to local nodes ( just like production) and then a new storage policy has been added with 4 copies. with equal number19:13
__david_As the account and container rings are not multi-region, you cannot access the objects on the remote cluster by default. I have been looking at mutli-region / storage policy docs and talks but I can't see any specific guidance about this.  If I PUT objects to multi-region container in region 1 and then 'seed' the same containers (POST with storage policy metadata) in region 2, I can stat objects region 2 directly. Am I miss19:13
*** catintheroof has joined #openstack-swift19:17
clayg__david_: multi-region clusters are a single flat name space - metadata operations happen across the WAN to account & container database replicas distributed globally.19:19
clayg__david_: if some of the remote replcias can't be reached for a metadata update those records are stored locally and trasfered async19:19
claygmost metadata updates can happen out of band from the object data transfer19:20
claygall operations (data & metadata) are robust to the unavailability of remote replicas19:20
clayg__david_: maybe https://docs.openstack.org/swift/latest/admin_guide.html#geographically-distributed-swift-considerations ???19:24
timburkeclayg: on p 458070 -- why do we check_mount in account/container server, but for obj we do it down in diskfile? should that patch instead be messing with swift/common/db.py?19:39
patchbothttps://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers19:39
claygthe databases don't really have a diskfile-like layer abstraction - ask zaitcev about it sometime.19:40
claygtl;dr "no that patch shouldn't do that; yes some other useful patch in the future might move some things around around in that area"19:41
zaitcevtimburke: Peter Portante had a particular dislike for excessive mount checking and wanted to get rid of it or at least attenuate it. He saw too much of it happening in strace, I gather.19:44
timburkei kinda feel like the broker would be an appropriate place for it? but what do i know...19:44
zaitcevYes, IIRC I moved it too, because obviously the mount checking is implemented differently for non-posix back-ends.19:45
* portante had a particular love for performance19:45
timburkezaitcev: so does this mean we should be getting it out of diskfile, too? hoist it up to obj/server.py? i still don't have the insight of "this *obviously* belongs *here*"19:47
__david_Thanks clayg.  Adding a SSD storage policy or a 2-copy storage policy in a local cluster is straight-forward,  but retro-filling a multi-region policy to two independent  clusters is somewhat mind-bending .19:49
__david_If we choose to not turn our existing AC rings into multi-region rings, will that create problems?  Region 2 from my example above has objects which are not reflected in the container listing in that region. De-coupling this fundamental tenant of Swift seems wrong, but I can't dete rmine another path without massive changes for our existing tenants/containers.19:49
claygtimburke: it's obvious when you look at the diff - this was were ht was - so continuing to do it here when the goal is to keep the behavior the same for mount_check = true is "obviously correct" - if I moved in this patch that'd be doing two things - the second thing is ok, but it's a side show - not having check_dir behavior in database servers is stupid/annoying and causing problems with testing in19:51
claygenvironments that use dirs (e.g. vsaio, devstack, etc)_19:51
claygi've almost got the devstack change working - i'll wip that patch for now19:52
claygFWIW zaitcev's patches to make the db brokers a better abstraction were great19:53
claygwe just never managed to get the merged19:53
timburkehaving a BaseDatabaseServer that subclasses BaseStorageServer feels weird to me. or rather, it feels weird that we only have this extra layer of abstraction so we can switch between check_mount and check_dir when we have a very similar switch in https://github.com/openstack/swift/blob/2.15.1/swift/obj/diskfile.py#L1193-L119819:54
*** wantstofail has joined #openstack-swift19:55
timburkewait, so the failure was *legit*? i was real close to just saying "recheck" ... maybe it's better i *don't* review it...19:56
*** dmsimard has left #openstack-swift19:58
claygyeah devstack doesn't create the device dirs :'(20:00
claygthe account/container server would create the device dirs - so when the object-server required they be there... THEY WERE!20:01
claygpretty cool20:01
claygi don't know how to help you "feel better" about having a class that implements a method that's used by two subclasses?20:01
* clayg shrugs20:01
claygit seems better than having to sublcasses both implement the same method20:01
claygbut if *that's* thing you wanna try to spin a diff for today - all I can hope for is follow up :\20:02
*** mat128 has quit IRC20:03
timburkeso what about places like https://github.com/openstack/swift/blob/2.15.1/swift/container/updater.py#L103 ? should *they* be doing check_dir too?20:06
claygyeah ideally!  although afaik we mostly only use the "break device" stuff for the 507 behavior20:07
*** baojg_ has joined #openstack-swift20:07
timburkeif we need that logic in account, container, and object servers, it seems like it belongs in BaseStorageServer. if we won't put it there because object server already has it covered on account of diskfile, it seems very strange to me that we aren't addressing the account/container servers in a similar way20:07
claygtimburke: I'll agree it's not ideal - can't build a cathedral with mud & spackling!20:09
*** baojg has quit IRC20:09
claygwhat we *really* need is a generic abstraction for all mounted filesystem based storage subsystems that both the sqlitedb and xfs .data file implementations can inherit from20:11
notmynamehttps://en.wikipedia.org/wiki/Great_Mosque_of_Djenné20:11
notmyname"mud cathedral"20:12
notmynameclayg: yes! like an "under" diskfile ;-)20:12
claygor we could just say - this method call right here with the ismount - it needs to also with the isdir - if we stuff this code in like this it will 1) do that and b) we should be able to grok it enough to change it later.20:15
notmynamewould it help rledisez and his LOSF work?20:15
claygnot really, no20:23
claygtimburke: there's also that change that adds the magic file check for mounts ... something for containers cschwede was working on...20:26
claygI guess instead of a base class we could just let it be a pure function that takes self.root and self.mount_check ???20:26
claygif nothing else it's weird that I moved self.mount_check = and not self.root =20:27
timburkehttps://review.openstack.org/#/c/466255/ ?20:27
patchbotpatch 466255 - swift - Make mount_check option usable in containerized en... (MERGED)20:27
timburkethere's also https://review.openstack.org/#/c/311703/ ...20:29
patchbotpatch 311703 - swift - Respect check_mount setting in recons disk_usage20:29
torgomatic__david_: are you trying to merge two completely separate Swift clusters together by adding the same storage policy with the same devices to both of them?20:34
__david_torgomatic - we want to add the ability to have containers that are replicate to another region. The existing accounts/containers that are local in each cluster we want to remain local.20:36
torgomatic__david_: sounds like you should be looking at container sync, then20:36
claygor eran's patches for container sync anyway :\20:36
mattoliverautimburke: ok I'll take a look today :) had a thought about how to improve the sharer.. I'll flush it out, but of it needs to wait until sharer 2.0 then that's ok too :)20:46
*** lucasxu has quit IRC20:46
mattoliverau*sharder20:46
__david_Thanks torgomatic and clayg20:47
*** chlong has quit IRC20:53
*** wantstofail has quit IRC21:03
*** tonyb has quit IRC21:04
*** catintheroof has quit IRC21:09
*** tonyb has joined #openstack-swift21:11
*** geaaru has quit IRC21:15
claygtimburke: i'm going to re-spin p 458070 with depends on p 49860421:23
patchbothttps://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers21:23
patchbothttps://review.openstack.org/#/c/498604/ - openstack-dev/devstack - Create correct directory layout for swift on purpose.21:23
claygtimburke: any final thoughts on how to make the BaseDatabaseStorageServer thing suck less?21:23
*** chlong has joined #openstack-swift21:26
*** chlong has quit IRC21:31
*** rcernin has quit IRC21:32
*** chlong has joined #openstack-swift21:43
openstackgerritClay Gerrard proposed openstack/swift master: Always require device dir for containers  https://review.openstack.org/45807022:10
*** itlinux has quit IRC22:28
*** prometheanfire has left #openstack-swift22:29
*** abhinavtechie has quit IRC22:54
*** vint_bra has quit IRC23:14
*** itlinux has joined #openstack-swift23:28
*** itlinux has quit IRC23:28
*** kei_yama has joined #openstack-swift23:30
*** chlong has quit IRC23:34
*** alenavolk has joined #openstack-swift23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!