Monday, 2017-02-06

*** Jeffrey4l has joined #openstack-swift00:02
*** gabor_antal_ has joined #openstack-swift00:09
*** gabor_antal has quit IRC00:09
*** sams-gleb has joined #openstack-swift00:20
*** klrmn has quit IRC00:24
*** sams-gleb has quit IRC00:25
*** ukaynar has joined #openstack-swift00:29
openstackgerritKota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support  https://review.openstack.org/21916500:33
*** jistr has quit IRC01:00
*** jistr has joined #openstack-swift01:00
*** klrmn has joined #openstack-swift01:09
*** tqtran has joined #openstack-swift01:23
*** tqtran has quit IRC01:28
*** Jeffrey4l_ has joined #openstack-swift01:51
*** Jeffrey4l has quit IRC01:53
*** zhugaoxiao has joined #openstack-swift01:53
*** tanee is now known as tanee_away02:01
*** tanee_away is now known as tanee02:01
*** tanee is now known as tanee_away02:01
*** tanee_away is now known as tanee02:01
*** ukaynar has quit IRC02:09
*** ukaynar has joined #openstack-swift02:10
*** sams-gleb has joined #openstack-swift02:23
*** sams-gleb has quit IRC02:27
*** ukaynar has quit IRC03:08
*** chenhuayi has joined #openstack-swift03:13
*** psachin has joined #openstack-swift04:04
*** chenhuayi has quit IRC04:04
*** links has joined #openstack-swift04:09
*** sams-gleb has joined #openstack-swift04:24
*** sams-gleb has quit IRC04:29
*** chenhuayi has joined #openstack-swift04:31
*** david-lyle has joined #openstack-swift04:46
*** winggundamth has quit IRC04:48
*** david-lyle has quit IRC04:50
*** david-lyle_ has joined #openstack-swift04:50
*** david-lyle_ is now known as david-lyle04:52
*** winggundamth has joined #openstack-swift05:02
*** david-lyle has quit IRC05:21
*** tqtran has joined #openstack-swift05:24
*** tqtran has quit IRC05:29
*** chenhuayi has quit IRC06:13
*** links has quit IRC06:19
*** sams-gleb has joined #openstack-swift06:26
*** sams-gleb has quit IRC06:31
*** links has joined #openstack-swift06:35
*** remix_tj has quit IRC06:39
openstackgerritKota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support  https://review.openstack.org/21916506:39
*** remix_tj has joined #openstack-swift06:43
*** winggundamth_ has joined #openstack-swift06:46
*** klrmn has quit IRC06:52
*** dmorita has joined #openstack-swift06:53
*** dmorita has quit IRC07:00
*** dmorita has joined #openstack-swift07:05
*** links has quit IRC07:05
*** dmorita has quit IRC07:07
*** dmorita has joined #openstack-swift07:09
*** chenhuayi has joined #openstack-swift07:09
*** tesseract has joined #openstack-swift07:12
*** links has joined #openstack-swift07:17
*** dmorita has quit IRC07:25
*** dmorita has joined #openstack-swift07:30
*** ChubYann has quit IRC07:33
openstackgerritKazuhiro MIYAHARA proposed openstack/swift master: Swift Automated Tiering  https://review.openstack.org/28705707:42
*** dmorita has quit IRC07:47
*** m_kazuhiro has joined #openstack-swift07:49
*** sams-gleb has joined #openstack-swift07:49
m_kazuhiroGood morning07:51
*** dmorita has joined #openstack-swift07:54
*** dmorita has quit IRC07:55
*** pcaruana has joined #openstack-swift07:55
*** dmorita has joined #openstack-swift07:55
openstackgerritSachin Patil proposed openstack/swift master: Fix swift-get-nodes arg parsing for missing ring  https://review.openstack.org/40601207:57
openstackgerritNam Nguyen Hoai proposed openstack/swift master: Fix_typo: "subsitute" -> "substitute"  https://review.openstack.org/42955408:03
*** dmorita has quit IRC08:05
*** pcaruana has quit IRC08:07
*** pcaruana has joined #openstack-swift08:07
*** Shashikant86 has joined #openstack-swift08:14
*** rledisez has joined #openstack-swift08:22
*** rledisez has quit IRC08:22
*** rledisez has joined #openstack-swift08:22
*** geaaru has joined #openstack-swift08:31
openstackgerritKota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support  https://review.openstack.org/21916508:33
*** tanee is now known as tanee_away08:34
*** tanee_away is now known as tanee08:37
*** links has quit IRC08:45
*** links has joined #openstack-swift08:45
*** lifeless has quit IRC08:54
*** lifeless has joined #openstack-swift08:55
*** Shashikant86 has quit IRC09:09
*** oshritf has joined #openstack-swift09:12
*** Shashikant86 has joined #openstack-swift09:17
*** tqtran has joined #openstack-swift09:26
*** tqtran has quit IRC09:31
*** links has quit IRC09:35
*** Shashikant86 has quit IRC09:36
*** Shashikant86 has joined #openstack-swift09:45
*** winggundamth has quit IRC09:45
*** winggundamth has joined #openstack-swift09:46
*** nikivi has quit IRC09:49
*** jcaron has joined #openstack-swift09:57
*** dmorita has joined #openstack-swift10:05
*** jordanP has joined #openstack-swift10:05
openstackgerritKazuhiro MIYAHARA proposed openstack/swift master: Swift Automated Tiering  https://review.openstack.org/28705710:08
*** chenhuayi has quit IRC10:09
*** dmorita has quit IRC10:09
*** vint_bra has joined #openstack-swift10:17
*** sams-gleb has quit IRC10:17
*** jcaron has left #openstack-swift10:21
*** jcaron has joined #openstack-swift10:21
*** vint_bra has quit IRC10:29
*** sams-gleb has joined #openstack-swift10:35
*** silor has joined #openstack-swift10:44
*** chenhuayi has joined #openstack-swift10:45
*** m_kazuhiro has quit IRC10:46
*** Shashikant86 has quit IRC10:47
*** Shashikant86 has joined #openstack-swift10:49
*** Shashikant86 has quit IRC10:58
*** silor1 has joined #openstack-swift10:58
*** silor has quit IRC10:58
*** silor1 is now known as silor10:58
*** Shashikant86 has joined #openstack-swift11:07
*** Shashikant86 has quit IRC11:16
*** kei_yama has quit IRC11:19
*** links has joined #openstack-swift11:21
*** winggundamth has quit IRC11:21
*** Shashikant86 has joined #openstack-swift11:23
*** links has quit IRC11:25
*** winggundamth has joined #openstack-swift11:33
*** vint_bra has joined #openstack-swift11:35
*** vint_bra has quit IRC11:39
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Fix test_replicator assertion  https://review.openstack.org/42967312:11
*** early has quit IRC12:12
openstackgerritKota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support  https://review.openstack.org/21916512:13
*** chenhuayi has quit IRC12:22
*** early has joined #openstack-swift12:28
*** Shashikant86 has quit IRC12:29
*** winggundamth has quit IRC12:42
*** winggundamth has joined #openstack-swift12:45
*** winggundamth has quit IRC12:52
*** oshritf_ has joined #openstack-swift12:52
kota_clayg, timburke: I'm now writing some comments on https://review.openstack.org/#/c/425493 but i have to leave my office for now asap, so i will push my comment later at home12:54
patchbotpatch 425493 - swift - Make the reconstructor handoffs_first work (and us...12:54
*** oshritf has quit IRC12:56
*** oshritf_ has quit IRC12:56
*** Shashikant86 has joined #openstack-swift12:58
*** oshritf_ has joined #openstack-swift12:58
*** catintheroof has joined #openstack-swift13:03
*** oshritf_ has quit IRC13:04
*** Shashikant86 has quit IRC13:05
*** oshritf_ has joined #openstack-swift13:18
openstackgerritJean Caron proposed openstack/swift master: Wait for each putter to be done before disconnection's error handling  https://review.openstack.org/42969713:18
*** Shashikant86 has joined #openstack-swift13:21
*** tqtran has joined #openstack-swift13:27
*** Jeffrey4l_ has quit IRC13:32
*** tqtran has quit IRC13:32
*** Jeffrey4l_ has joined #openstack-swift13:34
*** dims_ has quit IRC13:47
*** dims has joined #openstack-swift13:48
*** ma9 has joined #openstack-swift13:49
*** ma9 has left #openstack-swift13:50
*** Jeffrey4l_ has quit IRC13:58
*** vint_bra has joined #openstack-swift13:58
*** vint_bra has quit IRC14:03
*** Shashikant86 has quit IRC14:07
*** chlong has joined #openstack-swift14:18
*** Shashikant86 has joined #openstack-swift14:19
*** Shashikant86 has quit IRC14:25
*** Shashikant86 has joined #openstack-swift14:27
*** _JZ_ has joined #openstack-swift14:42
*** Shashikant86 has quit IRC14:43
*** Shashikant86 has joined #openstack-swift14:44
*** sams-gle_ has joined #openstack-swift15:19
jrichlijcaron: hello.  I am interested in reproducing the error you reported recently.15:20
jrichliyou said it was easier to reproduce using dlo/slo15:20
jrichlihow likely is it to manifest even with the steps on the bug report?15:21
jrichlimaybe i need to run several times or in different ways?15:21
*** sams-gleb has quit IRC15:23
*** sams-gle_ has quit IRC15:24
jcaronjrichli: Hi, yes slo/dlo involves more connections to the object-server. so there is more chances to see the leak.15:25
jcaronon our infrastruture running swift-2.12.0, we hit the bug almost everytime we kill the client connection15:25
jcaronjrichli: the policy storage has to be Erasure coding15:26
jrichlijcaron: ok, thank you.  Yes, I made sure to run with EC.  It is good to know the version where this exsits.  I'll do more research.15:26
openstackgerritGábor Antal proposed openstack/swift master: Fixed indentation in copy.py  https://review.openstack.org/42975015:35
*** sams-gleb has joined #openstack-swift15:36
*** cdelatte has joined #openstack-swift15:38
openstackgerritGábor Antal proposed openstack/swift master: Fixed indentation in common/ and obj/  https://review.openstack.org/42975015:40
*** Shashikant86 has quit IRC15:48
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Fix test_replicator assertion  https://review.openstack.org/42967316:19
*** oshritf_ has quit IRC16:20
openstackgerritKota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support  https://review.openstack.org/21916516:21
* kota_ is going to his bed hoping that changes fixes everything16:24
kota_g' night16:24
jrichligood night, kota_ !16:26
*** pcaruana has quit IRC16:38
*** remix_tj has quit IRC16:39
*** acoles_ is now known as acoles16:39
*** remix_tj has joined #openstack-swift16:41
notmynamegood monring16:43
*** sams-gle_ has joined #openstack-swift16:45
*** sams-gleb has quit IRC16:45
*** oshritf has joined #openstack-swift17:02
*** oshritf has quit IRC17:05
*** dmorita has joined #openstack-swift17:06
*** dmorita has quit IRC17:10
*** dims has quit IRC17:11
*** chsc has joined #openstack-swift17:15
*** dims has joined #openstack-swift17:18
*** tesseract has quit IRC17:18
*** psachin has quit IRC17:20
*** tqtran has joined #openstack-swift17:29
*** oshritf has joined #openstack-swift17:30
*** JimCheung has joined #openstack-swift17:33
*** tqtran has quit IRC17:34
*** oshritf has quit IRC17:35
*** klrmn has joined #openstack-swift17:36
*** gmmaha has quit IRC17:52
*** dmorita has joined #openstack-swift18:01
*** jordanP has quit IRC18:01
notmynamereminder, here's the draft logo for swift from the foundation. http://d.not.mn/swift_draft_logo.png18:03
notmynameplease give me feedback if you have something to say18:03
notmyname(I already got feedback from timburke -- a good color change suggestion)18:04
*** dmorita has quit IRC18:04
*** dmorita has joined #openstack-swift18:04
jrichlinotmyname: I am curious what the other team logos look like - for a comparison18:04
notmynameyou know as much as me. just whatever's been posted on the ML18:06
notmynamethis one is thematically similar to the others18:06
notmyname(strong lines, geometrical, sharp angles, bold solid colors)18:06
jrichliok, i will check out the mailing list.  i figured that there may be some sort of theme and wanted to see how it fit.18:06
*** tqtran has joined #openstack-swift18:06
* jrichli is off to lunch18:06
*** gmmaha has joined #openstack-swift18:10
*** dmorita has quit IRC18:15
*** dmorita has joined #openstack-swift18:16
*** dmorita has quit IRC18:16
*** dmorita has joined #openstack-swift18:16
*** gmmaha has quit IRC18:19
*** gmmaha has joined #openstack-swift18:20
*** gmmaha has quit IRC18:22
*** gmmaha has joined #openstack-swift18:22
claygheyoh!18:23
claygkota_: thanks!18:23
*** pcaruana has joined #openstack-swift18:36
*** winggundamth_ has quit IRC18:37
*** oshritf has joined #openstack-swift18:40
*** rledisez has quit IRC18:45
*** chlong has quit IRC18:52
*** acoles is now known as acoles_18:52
*** ChubYann has joined #openstack-swift18:53
*** bkopilov has quit IRC19:01
*** bkopilov_ has quit IRC19:01
*** geaaru has quit IRC19:01
*** chlong has joined #openstack-swift19:08
claygmahatic: I'm loving all these reviews you're doing lately - spot on the patch 425493 (nice catch!)19:10
patchbothttps://review.openstack.org/#/c/425493/ - swift - Make the reconstructor handoffs_first work (and us...19:10
*** pcaruana has quit IRC19:11
*** oshritf has quit IRC19:13
*** gmmaha has quit IRC19:16
*** vint_bra has joined #openstack-swift19:27
*** vint_bra has quit IRC19:30
jrichliclayg: are you thinking that somebody would use find_handoff_parts.py in order to find override partitions so they could feed these to replicator/reconstructor to do 'handoffs only'?19:37
claygjrichli: i've used scripts *like* that to do different masochistic things when I *really* need to be super particular about organizing my rebalance IO19:43
claygjrichli: I monitor the *count* of handoff parts constantly - it's a big driver for our automatic ring rebalancing - https://youtu.be/ger20cqOypE?t=156419:44
*** vint_bra has joined #openstack-swift19:45
jrichlioh, i see. so this would be more about the monitoring of progress, in this case19:45
claygi mean I think you gotta do *something* - ovh says they just use dispersion-report19:46
claygthe idea is "when is this rebalance cycle finished" isn't something a single node nodes19:47
claygwhen you're running handoffs_*first* some nodes get the wrong idea - with handoffs_*only* operations can leaverage the monitoring they're *already doing* to decide when to push a ring - to also introduce a config change at the end of the cycle after the last ring change19:48
*** caiobrentano has joined #openstack-swift19:48
*** jordanP has joined #openstack-swift19:49
claygi think "the right thing" is open to debate - and different operators may have different experience/feedback - I'm sure there's lots of good ideas that we could pursue19:49
*** tonanhngo has joined #openstack-swift19:51
*** zaitcev has joined #openstack-swift19:53
*** ChanServ sets mode: +v zaitcev19:53
claygbut specifically in the example of that reconstructor change (where master is *obviously* broken) the behiavor of the replicator is being held up as the gold standard - but a) we didn't always have handoffs_first b) when chuck added it was better than nothing; but didn't really work for what we needed c) the current behavior acctually came from operational experience/feedback that was acctually *asking* for handoffs only19:53
jrichlisorry if this is a silly question: but are you proposing to also introduce the "handoffs only" to replication?  or just reconstruction?19:53
claygsetting aside names and how to handle config option life cycle ...19:54
claygI think we *want* handoffs only mode19:54
claygI mean we sort of *know* we do - we sort of always knew we did - in the reconstructor more than ever we really need it!19:54
claygI don't think there's much value in supporting handoffs first behavior (where some nodes start syncing primaries that are sitting on handoffs)19:55
jrichliI am pretty sure that SL has had to run with handoffs_first with replication in the past.  I wonder if briancline has any insights from that.19:55
claygjrichli: i put this topic on the meeting for this week - i really hope we can get some more input feedback!19:56
jrichlinice19:56
claygbriancline: is a great example - his team was one of the first ones to point out that you should *never* leave handoffs_first turned on if you're not in a priority rebalance situation19:57
claygand I also know that they are definately monitoring rebalance - I think via dispersion-report19:57
claygso... I was trying to make the case that operations already uses replicator handoffs_first they way I'm expecting reconstructor handoffs_first to be used.19:59
clayg... but maybe it'd just be better if everyone did handoffs_only and called it handoffs_only20:00
jrichlii think i prefer the clearly stated "only" plan as well, fwiw20:00
claygmy only other cavet is that I don't want the replicator to have handoffs_first *and* handoffs_only - handoffs_first is stupid - the only useful thing is does is as an approximation of handoffs_only20:02
claygand I don't want to try to implement handoffs_first in the reconstructor - it's nonsense20:02
jrichliright.  that backwards compatible thing is always impeding progress!20:04
claygI think handoffs_first should just be an alias to handoffs_only - maybe with a warning20:04
clayg"Since you said handoffs_first, I know you want to priortize rebalance (which I know you're montioring cluster wide) - so I went ahead and put this node in handoffs_only mode - you should updating your tooling and turn off handoffs_first/only when you detect the rebalance is done"20:06
claygshould do the trick20:06
jrichlisounds reasonable. but yea ... when we know no handoffs remain, we gotta shout about it!!20:07
claygcool - ill leave it in a holding pattern till meeting20:08
jrichliJust one more thought: would it be crazy to automatically turn that mode off when no handoffs remain?  Ops would have to restart when they wanted to be in that mode again anyway20:11
claygthat's the little graph thing - node a may thing it's "done" - but really the "cluster" is still every much in rebalance mode20:12
jrichlioh, right.20:13
*** silor has quit IRC20:24
*** jordanP has quit IRC20:28
*** Jeffrey4l has joined #openstack-swift20:44
*** cdelatte has quit IRC20:46
*** vint_bra has quit IRC20:49
*** Jeffrey4l has quit IRC20:49
*** chlong has quit IRC20:49
*** Jeffrey4l has joined #openstack-swift20:51
*** Jeffrey4l has quit IRC20:55
*** Jeffrey4l has joined #openstack-swift20:56
*** Jeffrey4l has quit IRC20:57
*** Jeffrey4l has joined #openstack-swift20:58
*** vint_bra has joined #openstack-swift20:59
*** Jeffrey4l has quit IRC21:03
*** Jeffrey4l has joined #openstack-swift21:06
*** Jeffrey4l has quit IRC21:11
*** vint_bra has quit IRC21:16
*** Jeffrey4l has joined #openstack-swift21:18
*** caiobrentano_ has joined #openstack-swift21:24
*** tonanhngo has quit IRC21:24
*** tonanhngo_ has joined #openstack-swift21:24
*** caiobrentano has quit IRC21:27
*** Jeffrey4l has quit IRC21:28
openstackgerritTim Burke proposed openstack/swift master: Add abort flag to replicator/reconstructor  https://review.openstack.org/42901521:31
*** Jeffrey4l has joined #openstack-swift21:36
*** mkaminski has joined #openstack-swift22:00
*** mkaminski has quit IRC22:07
*** sams-gleb has joined #openstack-swift22:10
*** sams-gle_ has quit IRC22:12
*** caiobrentano has joined #openstack-swift22:23
*** sams-gleb has quit IRC22:41
openstackgerritTim Burke proposed openstack/swift master: Add abort flag to replicator/reconstructor  https://review.openstack.org/42901522:41
openstackgerritMerged openstack/swift master: Fix_typo: "subsitute" -> "substitute"  https://review.openstack.org/42955422:44
*** caiobrentano has quit IRC22:51
*** cdelatte has joined #openstack-swift22:53
*** cdelatte has quit IRC22:55
*** _JZ_ has quit IRC23:05
*** caiobrentano has joined #openstack-swift23:05
*** caiobrentano has quit IRC23:12
notmynameso I found this email interesting (at least interesting enough to want to share)...23:21
notmynamethe next three lines are excerpts...23:22
notmynameOn behalf of the Product Work Group and the OpenStack Foundation, we’re requesting your input on your team’s top features in the Ocata release and features/ themes likely in future releases for the roadmap.23:22
notmynameThe OpenStack Foundation’s executive staff begins briefings with media and analysts next Monday on Ocata...23:22
notmynameDuring the media briefings, we’ll highlight our message that OpenStack provides one platform to manage virtual machines, bare metal, and container orchestration frameworks.23:22
notmyname //end quote//23:22
notmynamewhat I find interesting is the messaging that the foundation is pushing23:22
timburkewhich part? the one-platform-one-project thing we've seen a decent bit of recently, or the complete and utter disregard for storage in favor of compute provisioning?23:24
*** kozhukalov_ has joined #openstack-swift23:27
*** AndyWojo_ has joined #openstack-swift23:27
*** DuncanT_ has joined #openstack-swift23:27
claygi'm all out of quips today - he said "interesting" not "surprising"23:27
*** cargonza_ has joined #openstack-swift23:28
timburkeoh, i find neither part *surprising*...23:29
*** Anticime1 has joined #openstack-swift23:29
*** dosaboy_ has joined #openstack-swift23:30
*** ktsuyuzaki has joined #openstack-swift23:32
*** ChanServ sets mode: +v ktsuyuzaki23:32
*** Raymii_ has joined #openstack-swift23:32
*** chsc has quit IRC23:32
*** rsFF- has joined #openstack-swift23:33
notmynameused to, the foundation said openstack was (paraphrased) "software to make cloud infrastructure, basically consisting of compute, storage and networking"23:34
*** DuncanT has quit IRC23:34
*** Raymii has quit IRC23:34
*** Anticimex has quit IRC23:34
*** kota_ has quit IRC23:34
*** dosaboy has quit IRC23:34
*** rsFF has quit IRC23:34
*** kozhukalov has quit IRC23:34
*** cargonza has quit IRC23:34
*** AndyWojo has quit IRC23:34
*** Raymii_ is now known as Raymii23:34
notmynamethe change in "interesting"23:34
*** vint_bra has joined #openstack-swift23:34
*** kozhukalov_ is now known as kozhukalov23:36
*** DuncanT_ is now known as DuncanT23:36
*** cargonza_ is now known as cargonza23:37
notmynames/in/is/23:37
*** AndyWojo_ is now known as AndyWojo23:37
*** vint_bra has quit IRC23:37
zaitcevDOWN WITH QUAN... er, NEWTON23:43
timburkezaitcev: want me to go ahead and update that commit message to say that it (only?) affects chunked transfers? and, if memory serves, replicated policies?23:45
timburke(thanks for looking at that, by the way)23:46
zaitcevtimburke: I think it would be more helpful to write something like "The ValueError is thrown up by the MIME parser of Eventlet that blindly does int(rfile.readline().split(b";", 1)[0], 16)"... or something23:47
zaitcev(+ during chunked transfers)23:48
ktsuyuzakigood morning23:49
timburkegood morning! nick change?23:50
zaitcevohayou gozaimasu23:50
ktsuyuzakizaitcev: ohayo!23:50
ktsuyuzakioh what? I'm using secandary nick?23:50
*** ktsuyuzaki is now known as kota_23:51
kota_timburke: thanks! now I changed my nick to my default23:51
timburkei mean, either one seems reasonable to me... :-)23:51
kota_timburke: and thanks for quick review on my test fix ;-)23:52
kota_but it looks like my global ec patch still got another failure which seems freaky one :/23:53
kota_I should try to resolve that23:53
timburkeyeah! like i said, i saw that test pop a few times so when i saw the patch, i thought "ooooohhhhh...... of course"23:53
timburkeyeah, i need to finally make another pass on that...23:53
*** catintheroof has quit IRC23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!