Friday, 2016-07-29

*** NM has joined #openstack-swift00:00
*** diogogmt has quit IRC00:00
*** Suyi_ has quit IRC00:07
*** hosanai has joined #openstack-swift00:08
*** ChanServ sets mode: +v hosanai00:08
*** NM has quit IRC00:10
*** adu has joined #openstack-swift00:21
*** tqtran has quit IRC00:25
MadOtisback00:29
notmynameMadOtis: basically, I've been trying to dig into keystone's source to find what's happening. it's fully of plugin interfaces and class inheritance and getters/setters, so it's slow going00:35
MadOtisBeen there, have the cross-eyes to prove it.  :)00:35
notmynamebut basically, if keystone doesn't think you've set the auth version, then it tries to discover it, and that's what's throwing the error00:35
notmynameI don't know why the error is being thrown, but the issue is really why it doesn't think the auth version has been set00:36
MadOtisgotcha00:36
notmynamewell, that's what was happening when you had v3 set00:36
notmynameI assume that if you use v2, everything still works and it's just a different api to the same thing, but I really don't know that to be true00:36
*** adu has quit IRC00:37
notmynamebut the 401 error you got with the v2 seems more promising than the 503 with v300:37
notmynamealso, AFAIK, the swift client supports auth v3 with keystone, so I'm not sure what's going on there either00:37
notmynameso I think what needs to happen is to properly specify the auth version so it doesn't do the discovery00:45
*** jamielennox is now known as jamielennox|away00:49
*** trananhkma has joined #openstack-swift00:51
*** m_kazuhiro has joined #openstack-swift00:56
*** gyee has quit IRC00:57
*** dmorita has quit IRC01:06
*** dmorita has joined #openstack-swift01:07
*** dmorita has quit IRC01:08
*** dmorita has joined #openstack-swift01:08
*** asettle has joined #openstack-swift01:22
*** asettle has quit IRC01:27
openstackgerritTim Burke proposed openstack/swift: Expose encryption status in GET/HEAD/PUT/POST responses  https://review.openstack.org/34860301:30
openstackgerritTim Burke proposed openstack/swift: Expose encryption status to client  https://review.openstack.org/34860401:30
*** jamielennox|away is now known as jamielennox01:34
*** ukaynar has joined #openstack-swift01:46
*** klrmn has quit IRC02:01
*** mmcardle has joined #openstack-swift02:15
*** mmcardle has quit IRC02:19
*** jamielennox is now known as jamielennox|away02:32
*** ukaynar has quit IRC02:54
*** dmorita has quit IRC03:00
*** dmorita has joined #openstack-swift03:05
*** dmorita has quit IRC03:10
*** bill_az has quit IRC03:26
openstackgerritgengchc2 proposed openstack/swift: modify the home-page info with the developer documentation  https://review.openstack.org/34871103:34
*** MadOtis has quit IRC03:40
*** chsc has joined #openstack-swift04:11
*** links has joined #openstack-swift04:13
*** links has quit IRC04:13
*** links has joined #openstack-swift04:13
openstackgerritzheng yin proposed openstack/swift: Use variable to make code simple  https://review.openstack.org/34872404:22
openstackgerritMatthew Oliver proposed openstack/swift: Fix memcached exception out of range stacktrace  https://review.openstack.org/21996304:25
*** psachin has joined #openstack-swift04:35
*** KriskK has joined #openstack-swift04:39
*** McMurlock1 has joined #openstack-swift04:42
*** KriskK has quit IRC04:43
*** McMurlock1 has quit IRC04:44
*** chsc has quit IRC05:00
*** flwang1 has quit IRC05:06
*** amitkqed has quit IRC05:19
*** amitkqed has joined #openstack-swift05:20
*** zaitcev has quit IRC05:24
*** SkyRocknRoll has joined #openstack-swift05:47
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/swift: [WIP] Testing latest u-c  https://review.openstack.org/31844105:49
*** KriskK has joined #openstack-swift06:00
*** geaaru has joined #openstack-swift06:08
*** oshritf has joined #openstack-swift06:14
*** KriskK_ has joined #openstack-swift06:21
*** KriskK has quit IRC06:23
*** KriskK_ is now known as KriskK06:23
*** oshritf has quit IRC06:36
*** tesseract- has joined #openstack-swift06:37
*** maestropandy has joined #openstack-swift06:40
*** hseipp has joined #openstack-swift06:43
*** pcaruana has joined #openstack-swift06:45
*** dmorita has joined #openstack-swift06:57
*** dmorita has quit IRC07:02
*** rledisez has joined #openstack-swift07:22
*** sheel has joined #openstack-swift07:32
*** silor has joined #openstack-swift07:32
*** flwang1 has joined #openstack-swift07:34
*** SkyRocknRoll has quit IRC07:54
*** silor has quit IRC07:57
*** SkyRocknRoll has joined #openstack-swift08:12
*** KriskK has quit IRC08:22
*** daemontool has joined #openstack-swift08:44
*** ouchkernel has quit IRC09:03
*** KriskK has joined #openstack-swift09:03
*** ouchkernel has joined #openstack-swift09:10
*** oshritf has joined #openstack-swift09:16
*** oshritf has quit IRC09:22
*** KriskK has quit IRC09:24
*** oshritf has joined #openstack-swift09:25
*** maestropandy has quit IRC09:25
*** KriskK has joined #openstack-swift09:27
openstackgerritPeter Lisák proposed openstack/swift: Recon's get_diskusage and get_unmounted return all devs.  https://review.openstack.org/34880309:32
*** permalac has joined #openstack-swift09:33
*** psachin has quit IRC09:35
*** oshritf has quit IRC09:39
*** ellen has joined #openstack-swift09:50
*** trananhkma has quit IRC10:08
*** psachin has joined #openstack-swift10:08
*** hosanai has quit IRC10:14
*** KriskK has quit IRC10:15
*** sheel has quit IRC10:16
*** jordanP has joined #openstack-swift10:16
*** maestropandy has joined #openstack-swift10:18
openstackgerritMarek Kaleta proposed openstack/python-swiftclient: Add copy object method  https://review.openstack.org/28020010:51
openstackgerritBéla Vancsics proposed openstack/swift: Reduce code duplication  https://review.openstack.org/26776710:56
*** psachin has quit IRC11:04
*** psachin has joined #openstack-swift11:05
openstackgerritMarek Kaleta proposed openstack/swift: Extend swift-init config handling documentation  https://review.openstack.org/29390411:06
*** manousd has joined #openstack-swift11:11
*** m_kazuhiro has quit IRC11:11
*** manous has joined #openstack-swift11:12
*** manousd has quit IRC11:16
*** asettle has joined #openstack-swift11:20
*** KriskK has joined #openstack-swift11:24
*** cdelatte has joined #openstack-swift11:27
*** maestropandy has quit IRC11:27
*** maestropandy has joined #openstack-swift11:43
*** maestropandy has left #openstack-swift11:43
*** abhitechie has joined #openstack-swift11:56
*** NM has joined #openstack-swift12:02
*** abhitechie has quit IRC12:05
*** abhitechie has joined #openstack-swift12:05
*** mhu has quit IRC12:27
*** kei_yama has quit IRC12:33
*** mhu has joined #openstack-swift12:34
*** zhengyin has quit IRC12:35
*** abhitechie has quit IRC12:39
*** abhitechie has joined #openstack-swift12:44
*** abhinavtechie has joined #openstack-swift12:48
*** abhitechie has quit IRC12:49
*** SkyRocknRoll has quit IRC12:52
*** maestropandy1 has joined #openstack-swift12:52
*** bill_az has joined #openstack-swift12:52
*** maestropandy1 has left #openstack-swift12:56
*** psachin has quit IRC12:59
*** maestropandy has joined #openstack-swift13:00
*** maestropandy has left #openstack-swift13:04
*** links has quit IRC13:07
*** zhengyin has joined #openstack-swift13:17
*** NM has quit IRC13:18
*** silor has joined #openstack-swift13:20
*** KriskK has quit IRC13:23
openstackgerritMerged openstack/swift: Simple code and improve the readability of the code  https://review.openstack.org/34814913:26
*** silor1 has joined #openstack-swift13:29
*** silor has quit IRC13:31
*** silor1 is now known as silor13:31
*** catintheroof has quit IRC13:42
ntataclayg, Thanks for rebasing and all the help with reviews for patch 347546 :)13:53
patchbotntata: https://review.openstack.org/#/c/347546/ - swift - Moved ipv4 & ipv6 validations to the common utils13:53
*** StraubTW has joined #openstack-swift13:57
*** abhinavtechie has quit IRC14:03
*** KriskK has joined #openstack-swift14:15
*** abhinavtechie has joined #openstack-swift14:22
*** lcurtis has joined #openstack-swift14:25
*** asettle has quit IRC14:32
*** psachin has joined #openstack-swift14:39
*** david-lyle has joined #openstack-swift14:42
*** david-lyle has quit IRC14:47
*** siva_krish has joined #openstack-swift14:54
*** siva_krish1 has joined #openstack-swift14:55
*** siva_krish has quit IRC14:58
*** jistr is now known as jistr|call15:10
*** klamath has joined #openstack-swift15:10
*** klamath has joined #openstack-swift15:11
*** diogogmt has joined #openstack-swift15:11
*** jistr|call is now known as jistr15:17
*** arch-nemesis has joined #openstack-swift15:19
*** david-lyle has joined #openstack-swift15:23
*** KriskK has quit IRC15:24
lcurtiscould someone suggest some ways of moving to erasure coding from an existing huge swift cluster15:27
lcurtisrebalancing is killing us15:27
lcurtisam i better off creating a new cluster?15:28
*** NM has joined #openstack-swift15:30
tdasilvalcurtis: Interestingly OVH has proposed a talk for the next summit titled: "How to migrate a large Swift cluster to Erasure Coding?"15:33
tdasilvabut I don't know if OVH guys hang out here15:33
lcurtis:)15:33
lcurtisthanks tdasilva15:34
*** zhengyin has quit IRC15:34
*** zhengyin has joined #openstack-swift15:35
*** klrmn has joined #openstack-swift15:36
*** bill_az has quit IRC15:44
*** manous has quit IRC15:44
*** Suyi_ has joined #openstack-swift15:50
*** chsc has joined #openstack-swift15:51
*** klrmn has quit IRC15:52
notmynamegood morning15:52
notmynamelcurtis: you should talk to rledisez15:52
notmynamelcurtis: what's the rebalancing problems you're seeing?15:52
*** nadeem has joined #openstack-swift15:54
*** nadeem has quit IRC15:55
*** nadeem has joined #openstack-swift15:55
*** manous has joined #openstack-swift15:57
rledisezhello lcurtis15:57
rledisezyes, there is a major problem of rebalancing EC cluster15:57
rledisezwhat EC config have you? (fragment number)15:58
rlediseznotmyname: the problem is mainly that when you rebalance, the reconstructor can decide to run a SYNC job because it does not find the fragments in place (because the corresponding REVERT job has not been done yet). so, you end up with n+m requests instead of one (n+m == fragments)15:59
rlediseznotmyname: for us, it's 15 instead of 115:59
rledisezso it creates a huge IO load on the cluster15:59
rlediseznotmyname: for now, we disable SYNC when we rebalance15:59
*** pcaruana has quit IRC16:01
*** admin6 has quit IRC16:18
*** hseipp has quit IRC16:21
*** tesseract- has quit IRC16:30
lcurtishello <rledisez>16:30
lcurtiswe are not using ec right now..using replicas16:30
*** bill_az has joined #openstack-swift16:30
lcurtisour servers under constant load during rebalance16:30
*** permalac has quit IRC16:31
lcurtiswe also seem to only be able to use 60% of usable disk space ..the more data we have on disks, the more handoffs we seem to generate16:31
lcurtiswith 3x replicas...trying to improve on cost / tb16:32
lcurtisso thinking of switching to ec16:32
*** jamielennox|away is now known as jamielennox16:32
lcurtisthinking of just creating new cluster16:32
lcurtisthis coming from perspective of it mgr...not as hands on as i used to be16:33
notmynamelcurtis: how many drives per server do you have in the object servers today?16:33
lcurtishi notmyname ...3616:33
*** tqtran has joined #openstack-swift16:34
lcurtisrledisez does disabling sync help?16:34
rledisezlcurtis: if you're not running EC yet, this is not your problem16:35
rledisezlcurtis: we have the same kind of hardware (36 disks, 6TB each). we use them up to 95% without major problem in replica mode16:35
rledisezdoes handoff appear when you're rebalancing only, or all the time ?16:35
*** tqtran_ has joined #openstack-swift16:35
*** psachin has quit IRC16:36
*** dmorita has joined #openstack-swift16:37
notmynamerledisez: I thought you had those 2/2TB server configs16:37
lcurtis<rledisez> the handoffs appear during rebalance16:37
lcurtismostly16:38
*** SkyRocknRoll has joined #openstack-swift16:38
rlediseznotmyname: we have multiple kind of hardware. we used to have 2*2TB, then 36*6TB. we're going to move to 36*2TB for replica and keep 36*6TB for EC (different offers, different perf ;))16:38
*** tqtran has quit IRC16:38
rledisezlcurtis: maybe it's rsync that is to IO intensive. rsync stat each files before moving it. So it creates lot of IO, can create timeout on object-servers, etc...16:39
rledisezlcurtis: did you try ssync ? iot can really help for that. we have good experience with it to save IO during rebalance (but it's slower, so better not be in a hurry)16:39
*** dmorita has quit IRC16:40
*** dmorita has joined #openstack-swift16:41
*** ukaynar has joined #openstack-swift16:41
*** klrmn has joined #openstack-swift16:43
lcurtiswill have to check it out16:44
lcurtisso does ec save you lot of disk space?16:44
*** manous has quit IRC16:47
rledisezlcurtis: it depends on the choice you make for the number of fragments. you can achieve an overhead of 1.1 if you want. it's a balance between saving money, risk you can take, number of servers you'll have, ...16:48
lcurtisi see16:49
*** zaitcev has joined #openstack-swift16:50
*** ChanServ sets mode: +v zaitcev16:50
*** manous has joined #openstack-swift17:03
*** jordanP has quit IRC17:05
*** rledisez has quit IRC17:13
*** manous has quit IRC17:14
*** klrmn has quit IRC17:15
*** klrmn has joined #openstack-swift17:20
*** KriskK has joined #openstack-swift17:24
*** abhinavtechie has quit IRC17:32
*** manous has joined #openstack-swift17:32
*** klrmn has quit IRC17:32
*** itlinux has quit IRC17:35
*** nadeem has quit IRC17:35
*** jamielennox is now known as jamielennox|away17:35
*** KriskK_ has joined #openstack-swift17:46
*** KriskK has quit IRC17:46
*** manous has quit IRC17:47
*** KriskK has joined #openstack-swift17:50
*** KriskK_ has quit IRC17:50
*** links has joined #openstack-swift17:53
claygzaitcev: i'm still confused on patch 291959 if you wanted to put the +A on it or wait for another set of eyes because I touched it?17:56
patchbotclayg: https://review.openstack.org/#/c/291959/ - swift - Handle IPv6 addresses in swift-get-nodes.17:56
claygseems pretty low risk, fix an old bug17:57
claygalso sets up for ntata's patch 34754617:57
patchbotclayg: https://review.openstack.org/#/c/347546/ - swift - Moved ipv4 & ipv6 validations to the common utils17:57
claygntata: maybe you could look at torgomatic's fix and give it a thorough review?17:58
zaitcevclayg: I hoped for someone to look, but I am confident as-is18:02
zaitcevclayg: it's just maybe I'm too lenient. I was +2 for the old version with duplication too.18:03
zaitcevMy litmus test is, and please forgive the unpleasant hypothesis, what if 1906 earthquakes kills everyone in San Frrancisco and I have to maintain this code, would it be okay to add this patch. That duplication looked completely okay to me...18:05
notmynamelol18:05
claygnice ;)18:07
*** timur has joined #openstack-swift18:17
timurI'm finally at the point of adding the container databases crawler for ElasticSearch, but not sure how to structure it. Should it be a new daemon added to the Swift repository or should it be a separate entity? Should I start by adding a write up of what it is/how it's implemented somewhere first?18:22
*** KriskK has quit IRC18:22
*** timur has quit IRC18:23
*** timur has joined #openstack-swift18:24
jrichlitimur: fwiw, I would appreciate getting a chance to see any documentation you may have.18:35
notmynametimur: there's 2 ways you could do it, depending on what state it's in18:36
notmynameif it's much more at the "I've got a prototype, but it needs to be rewritten" stage, then writing up the ideas/thinking/plans and linking to them on https://wiki.openstack.org/wiki/Swift/ideas is best18:37
notmynamehowever, if it's at the "here's something that's pretty good and is close to what should land upstream" stage, then making sure your patch has good in-tree docs is best18:38
*** KriskK has joined #openstack-swift18:47
timurwell, I guess my first question is whether this should live in Swift or be in a separate repository. It certainly interacts with Swift, but is not really a core part of it. Maybe laying out the current design on the wiki would be the best first step. I'll work on that18:47
*** dmorita has quit IRC18:49
notmynameyeah, it's a good question18:49
*** vint_bra has joined #openstack-swift18:50
*** hoonetorg has quit IRC18:56
ntataclayg, sure.. I'll take a look at it18:59
*** geaaru has quit IRC19:04
claygtorgomatic: cschwede: is there anyway I could get you to think about and chime in on lp bug #1605301 and/or patch 34552419:06
openstackLaunchpad bug 1605301 in OpenStack Object Storage (swift) "min_part_hours should not larger than 255" [Medium,In progress] https://launchpad.net/bugs/1605301 - Assigned to Cheng Li (shcli)19:06
patchbotclayg: https://review.openstack.org/#/c/345524/ - swift - min_part_hours should not larger than 25519:06
*** fifieldt has quit IRC19:07
claygit's possible I'm totally off base, the whole premise seems sane on the surface - but then I started to think about how we use this akward little trick with min_part_hours 9999 in practice...19:07
claygahale: briancline: you guys ever run a rebalance with an impossibly high min_part_hours for any reason?19:07
*** vint_bra has quit IRC19:09
claygtimur: where did you land on the overlap of swift3-sync and metadata-indexer?19:09
*** vint_bra has joined #openstack-swift19:10
claygtimur: are each of them going to be seperate daeamons independently finding/talking to container dbs?19:10
*** NM has quit IRC19:11
claygtimur: are you happy with your scribble space for trcking row_id sync points and using the container api's - or do you think someting with a tighter coupling of the ContainerBroker api's and a sync_point table would be an important improvement long term?19:11
*** NM has joined #openstack-swift19:11
timurclayg: for now I'm going with the separate daemon approach and separate scratch space in /var/lib/. Longer term, it may be more useful to use something else as scratch space, but I don't think it's an immediate problem19:15
claygit's local disk?  so multiple instances of the daemons don't need to share state about what they've already worked on?  how do you handle loosing a node/daemon?19:18
*** fifieldt has joined #openstack-swift19:18
*** links has quit IRC19:20
*** dmorita has joined #openstack-swift19:20
claygi think ideas page is reasonable next step19:20
timurclayg: the daemons don't share any state today. When a daemon is lost, a new one can be added and it will start from row 0 again, but hopefully catch up quickly as most of the time it will be checking that the row has already been replicated. For the Swift to S3 case, at least, that seems fine, as the actual data transfer should dominate the HEAD requests. For ES, I suspect the catching up wouldn't19:21
timurbe too painful either, but haven't done anything to measure it.19:21
timurAlright, I'll start with the ideas page and then we can talk further about how this works19:21
*** silor has quit IRC19:23
*** manous has joined #openstack-swift19:38
openstackgerritMerged openstack/swift: Add doc entry to check partition count  https://review.openstack.org/32748919:38
*** asettle has joined #openstack-swift19:58
*** cdelatte has quit IRC20:00
*** asettle has quit IRC20:03
*** manous has quit IRC20:08
*** manous has joined #openstack-swift20:22
*** vint_bra has quit IRC20:24
*** manous has quit IRC20:27
*** nadeem has joined #openstack-swift20:28
*** klamath has quit IRC20:32
*** SkyRocknRoll has quit IRC20:46
*** nadeem has quit IRC20:54
*** Suyi_ has quit IRC20:54
*** Suyi_ has joined #openstack-swift20:54
*** dmorita has quit IRC20:56
*** dmorita has joined #openstack-swift21:00
notmynamethe interesting ML emails always come after lunch on friday (pacific time)21:00
notmynamehere's what I'm referring to21:07
notmynamehttps://review.openstack.org/#/c/349068/  <-- this proposed change will significantly affect every openstack project21:07
patchbotnotmyname: patch 349068 - governance - describe a process for managing community-wide goals21:07
*** vint_bra has joined #openstack-swift21:08
*** siva_krish1 has quit IRC21:10
zaitcevThe official goal of Linux is "world domination and penguins".21:14
zaitcevAt least that's what Linus answered when someone requested him to formulate it.21:14
openstackgerritMerged openstack/swift: Split up backend/sysmeta header/footer preference tests  https://review.openstack.org/34699221:22
openstackgerritMerged openstack/swift: Additional test for container override etag preferences  https://review.openstack.org/34768421:22
openstackgerritMerged openstack/swift: Use self.ts more consistently in obj.test_server  https://review.openstack.org/34699321:22
openstackgerritMerged openstack/swift: Handle IPv6 addresses in swift-get-nodes.  https://review.openstack.org/29195921:25
openstackgerritMerged openstack/swift: Moved ipv4 & ipv6 validations to the common utils  https://review.openstack.org/34754621:25
*** KriskK has quit IRC21:29
*** NM has quit IRC21:30
*** nadeem has joined #openstack-swift21:32
*** ukaynar_ has joined #openstack-swift21:34
*** ukaynar has quit IRC21:36
*** daemontool has quit IRC21:41
*** ukaynar_ has quit IRC21:49
*** _JZ_ has joined #openstack-swift21:50
*** manous has joined #openstack-swift21:52
*** dmorita has quit IRC21:53
*** chsc has quit IRC21:53
*** manous has quit IRC21:57
*** _JZ_ has quit IRC22:04
*** vint_bra has quit IRC22:04
*** dmorita has joined #openstack-swift22:08
*** StraubTW has quit IRC22:09
*** manous has joined #openstack-swift22:10
*** dmorita has quit IRC22:12
*** dmorita has joined #openstack-swift22:12
*** dmorita has quit IRC22:16
*** dmorita has joined #openstack-swift22:18
*** lcurtis has quit IRC22:30
*** manous has quit IRC22:31
*** nadeem has quit IRC22:37
*** nadeem has joined #openstack-swift22:38
claygwhoooo it's merge friday!22:38
*** manous has joined #openstack-swift22:44
*** manous has quit IRC23:05
*** manous has joined #openstack-swift23:19
*** pgbridge has quit IRC23:22
*** manous has quit IRC23:23
*** dmorita has quit IRC23:31
*** dmorita has joined #openstack-swift23:32
*** daemontool has joined #openstack-swift23:34
*** manous has joined #openstack-swift23:35
*** NM has joined #openstack-swift23:37
*** Suyi_ has quit IRC23:38
*** NM has quit IRC23:40
*** manous has quit IRC23:45
*** NM has joined #openstack-swift23:45
*** vinsh has quit IRC23:54
*** NM has quit IRC23:57
*** NM has joined #openstack-swift23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!