Thursday, 2017-05-04

*** adriant has joined #openstack-swift00:05
*** tone_zrt has joined #openstack-swift00:15
*** vint_bra has quit IRC00:15
*** gyee has quit IRC00:29
*** zhurong has joined #openstack-swift00:31
*** tovin07_ has joined #openstack-swift00:46
*** klrmn1 has quit IRC01:02
*** zhurong has quit IRC01:16
*** zhurong has joined #openstack-swift01:26
*** mingyu has joined #openstack-swift01:41
*** mingyu has quit IRC01:46
*** lcurtis has joined #openstack-swift01:47
*** winggundamth has joined #openstack-swift01:53
*** zhongjun has joined #openstack-swift01:55
*** klrmn has joined #openstack-swift02:14
*** zhurong has quit IRC02:34
*** zhurong has joined #openstack-swift02:36
*** vint_bra has joined #openstack-swift02:43
*** vint_bra has quit IRC02:56
*** lcurtis has quit IRC03:06
*** vint_bra has joined #openstack-swift03:09
*** tdasilva- has quit IRC03:11
*** blairo has quit IRC03:11
*** flwang has quit IRC03:11
*** pdardeau has quit IRC03:11
*** ChubYann has quit IRC03:11
*** tovin07_ has quit IRC03:16
*** tovin07_ has joined #openstack-swift03:17
*** ChubYann has joined #openstack-swift03:18
*** pdardeau has joined #openstack-swift03:19
*** flwang has joined #openstack-swift03:20
*** vint_bra has quit IRC03:20
*** tdasilva has joined #openstack-swift03:23
*** blair has joined #openstack-swift03:28
*** mingyu has joined #openstack-swift03:32
*** mingyu has quit IRC03:38
*** klrmn has quit IRC03:53
*** zhurong has quit IRC04:12
*** klrmn has joined #openstack-swift04:21
*** geaaru has quit IRC04:23
*** cebruns has quit IRC04:34
*** zhurong has joined #openstack-swift04:36
*** mingyu has joined #openstack-swift04:36
*** Sukhdev has joined #openstack-swift04:36
*** cebruns has joined #openstack-swift04:39
*** mingyu has quit IRC04:41
*** klrmn has quit IRC04:41
*** flwang has quit IRC04:56
*** Sukhdev has quit IRC05:05
*** arunman has joined #openstack-swift05:08
*** links has joined #openstack-swift05:18
*** clayg has quit IRC05:22
openstackgerritNgo Quoc Cuong proposed openstack/swift master: Trivial fix docstring and typos  https://review.openstack.org/46237905:29
*** gkadam has quit IRC05:31
openstackgerritNgo Quoc Cuong proposed openstack/swift master: Trivial fix docstring and typos  https://review.openstack.org/46237905:34
*** gkadam has joined #openstack-swift05:34
*** Dinesh_Bhor has quit IRC05:41
*** Dinesh_Bhor has joined #openstack-swift05:41
*** chsc has joined #openstack-swift05:41
*** chsc has joined #openstack-swift05:41
*** Dinesh_Bhor has quit IRC05:43
*** Dinesh_Bhor has joined #openstack-swift05:48
*** ChubYann has quit IRC05:52
*** gkadam has quit IRC06:00
*** gkadam has joined #openstack-swift06:01
*** abhitechie has joined #openstack-swift06:04
*** mingyu has joined #openstack-swift06:11
*** mingyu has quit IRC06:16
*** mingyu has joined #openstack-swift06:25
*** mingyu has quit IRC06:41
*** mingyu has joined #openstack-swift06:42
*** goutham has quit IRC06:43
*** pcaruana has joined #openstack-swift06:45
*** chsc has quit IRC06:50
*** arunman has quit IRC06:52
*** mingyu has quit IRC06:55
*** mingyu has joined #openstack-swift06:56
*** arunman has joined #openstack-swift07:02
*** jamielennox is now known as jamielennox|away07:06
*** geaaru has joined #openstack-swift07:19
*** tesseract has joined #openstack-swift07:29
acolesgood morning07:45
f0owhile I'm here, I have an issue with swift+ceilometer. I'm sure I'm just missing out some python module or so, here's the error: https://paste.debian.net/plain/93069507:47
*** hseipp has joined #openstack-swift07:48
*** hseipp has left #openstack-swift07:49
acolesnotmyname: I'm sure there are some english people in this channel who have no idea what is going on ;-)07:49
acolestimburke: mahatic: thanks for rechecks07:49
openstackgerritNgo Quoc Cuong proposed openstack/swift master: Trivial fix docstring and typos  https://review.openstack.org/46237907:51
mahaticacoles: good morning, np07:52
acolesmahatic: o/08:02
*** jaosorior_away is now known as jaosorior08:14
*** abhitechie has quit IRC08:25
*** abhitechie has joined #openstack-swift08:25
*** cbartz has joined #openstack-swift08:35
*** hseipp has joined #openstack-swift08:43
*** hseipp has left #openstack-swift08:44
f0oFound my issue, outdated python libs08:52
f0oGot a amqp issue now; inequivalent arg 'auto_delete' for exchange 'swift' in vhost '/': received 'false' but current is 'true' ; is there a config flag for this? documentation for this is quite sparse08:52
*** adriant has quit IRC08:57
f0oI think it's the same bug as https://bugs.launchpad.net/ceilometermiddleware/+bug/1673738 - I can't config it because ceilometermiddleware doesnt look for it09:05
openstackLaunchpad bug 1673738 in ceilometermiddleware "ceilometermiddleware should read oslo configs" [Undecided,New]09:05
*** zhurong has quit IRC09:06
*** zhurong has joined #openstack-swift09:15
*** tovin07 has quit IRC09:18
*** links has quit IRC09:45
*** remix_tj has joined #openstack-swift10:02
*** mvk has quit IRC10:03
*** tovin07_ has quit IRC10:04
*** zhurong has quit IRC10:07
*** abhitechie has quit IRC10:08
*** mingyu has quit IRC10:13
*** mingyu has joined #openstack-swift10:15
*** jordanP has joined #openstack-swift10:18
*** hseipp has joined #openstack-swift10:21
openstackgerritKaren Chan proposed openstack/swift master: Store version id if restoring object from archive  https://review.openstack.org/43752310:21
*** hseipp has left #openstack-swift10:22
*** zhurong has joined #openstack-swift10:24
*** mvk has joined #openstack-swift10:30
*** arunman has quit IRC10:31
*** arunman has joined #openstack-swift10:33
*** mingyu has quit IRC10:36
*** mrhillsman has quit IRC10:48
*** mrhillsman has joined #openstack-swift10:48
*** mingyu has joined #openstack-swift10:49
*** jordanP has quit IRC11:00
openstackgerritKaren Chan proposed openstack/swift master: Store version id when copying object to archive  https://review.openstack.org/43752311:07
*** winggundamth_ has joined #openstack-swift11:33
*** winggundamth has quit IRC11:35
*** lcurtis has joined #openstack-swift11:36
*** mingyu has quit IRC11:45
*** mingyu has joined #openstack-swift11:45
*** catintheroof has joined #openstack-swift11:47
*** catintheroof has quit IRC11:51
*** lcurtis has quit IRC11:57
*** mingyu has quit IRC11:58
openstackgerritAlistair Coles proposed openstack/swift master: Use setUpModule instead of setup for module level unit test setup  https://review.openstack.org/46252012:01
acolestimburke: ^^ figuring setup vs setUpModule finally shone a light onto something that had bugged me for quite some time12:04
*** mingyu has joined #openstack-swift12:09
*** mingyu has quit IRC12:14
*** klamath has joined #openstack-swift12:19
*** jordanP has joined #openstack-swift12:19
*** klamath has quit IRC12:19
*** klamath has joined #openstack-swift12:19
*** mingyu has joined #openstack-swift12:20
*** zhurong has quit IRC12:30
*** mingyu has quit IRC12:33
*** mingyu has joined #openstack-swift12:35
*** cebruns has quit IRC12:43
*** cebruns has joined #openstack-swift12:50
*** vint_bra has joined #openstack-swift12:53
*** vint_bra has quit IRC12:54
*** mingyu has quit IRC13:14
*** mingyu has joined #openstack-swift13:16
*** chlong has joined #openstack-swift13:32
*** xinli has joined #openstack-swift13:38
*** zhurong has joined #openstack-swift13:41
openstackgerritMerged openstack/swift master: Trivial fix for decrypter docstrings  https://review.openstack.org/46207513:45
*** mingyu has quit IRC13:47
*** mingyu has joined #openstack-swift13:48
*** zhurong_ has joined #openstack-swift13:50
*** zhurong has quit IRC13:51
*** gkadam has quit IRC13:53
*** bkopilov has quit IRC13:56
*** jamielennox|away is now known as jamielennox13:58
*** chlong has quit IRC14:01
*** NM has joined #openstack-swift14:02
*** zhurong_ has quit IRC14:12
*** chlong has joined #openstack-swift14:13
*** catintheroof has joined #openstack-swift14:20
*** catinthe_ has joined #openstack-swift14:23
*** catintheroof has quit IRC14:24
*** _JZ_ has joined #openstack-swift14:31
*** hseipp has joined #openstack-swift14:39
*** vint_bra has joined #openstack-swift14:45
*** lcurtis has joined #openstack-swift14:52
*** tongli has joined #openstack-swift14:52
*** jerrygb has joined #openstack-swift15:07
*** NM has quit IRC15:12
*** lcurtis_ has joined #openstack-swift15:13
*** chlong has quit IRC15:36
notmynamegood morning15:40
*** SkyRocknRoll has joined #openstack-swift15:41
*** gyee has joined #openstack-swift15:50
openstackgerritAlistair Coles proposed openstack/swift master: Add read and write affinity options to deployment guide  https://review.openstack.org/46261915:50
*** jaosorior has quit IRC15:54
*** jordanP has quit IRC15:57
*** jordanP has joined #openstack-swift16:02
*** SkyRocknRoll has quit IRC16:04
*** vinsh has quit IRC16:05
*** mvk has quit IRC16:11
lcurtis_hello everyone...does anyone have advice on backfilling object servers with larger drives?16:12
lcurtis_as an experiment, we weighted a node to zero and it has only rebalanced 20% over the last month or so16:12
lcurtis_thinking of doing server to server rsync from 4T > 6T drives16:13
*** cbartz has quit IRC16:13
*** lcurtis has quit IRC16:23
*** hseipp has quit IRC16:27
*** mrhillsman has quit IRC16:35
*** chsc has joined #openstack-swift16:37
*** chsc has joined #openstack-swift16:37
notmynamelcurtis_: if you replaced the drive but kept it at the same mountpoint, all you'd need to do is change the weight of the drive16:39
notmyname(+rebalance +redeploy rings +repeat)16:40
*** pcaruana has quit IRC16:40
lcurtis_notmyname: thanks ... will try drive at a time16:42
lcurtis_it seems that with the number of objects we have per node -- taking extreme amount of time16:43
lcurtis_also we have min part hours at 144 (i think)16:43
lcurtis_will take long time16:43
lcurtis_need to do something a little more aggressive16:44
notmynamethe rebalance speed is something that's been worked on a bunch over the past year (and there is a large body of work that is still to be done). make sure you're using a recent version of the code. that itself can help (eg the rebalance operation is smarter)16:45
notmynamealso, you can use handoffs_only mode for replication. and also check that you've got enough rsync max connections to handle the replication load16:45
lcurtis_awesome...will check these out16:46
lcurtis_thnx16:46
*** mrhillsman has joined #openstack-swift16:46
notmynamebut do not rebalance and push rings faster than a replication cycle time (regardless of what min_part_hours is). a clever automation tool/product will observe the replication cycle time and only rebalance/redeploy rings once per cycle16:46
*** geaaru has quit IRC16:47
*** winggundamth_ has quit IRC16:48
timburkehuh. `python -m unittest test.unit.proxy.test_server.TestProxyServer.test_exception_occurred` -> OK, but `python -m unittest test.unit.proxy.test_server.TestProxyServer.test_exception_occurred | cat` -> FAILED16:49
notmynametimburke: weird16:49
*** NM has joined #openstack-swift16:49
timburkei guess python picked up on the UTF-8 support when writing straight to the terminal, but fell back to plain old ASCII for talking to a pipe?16:50
notmynametimburke: python or nose?16:51
notmynameoh, hmm.. you're runnign python itself16:52
timburkelook again -- no nose, all std lib16:52
*** JimCheung has joined #openstack-swift16:52
notmynameyeah16:52
timburkeno idea whether nose would exhibit the same behavior16:52
timburke(oh, and good morning everyone!)16:53
notmynametimburke: what's the specific assert that's failing?16:55
timburkeUnicodeEncodeError gets thrown down in the log handling. but TIL about the PYTHONIOENCODING env var!16:56
timburke`PYTHONIOENCODING=utf8 python -m unittest test.unit.proxy.test_server.TestProxyServer.test_exception_occurred | cat` works, but feels like an awkward way to try to write a program where you only ever want a particular encoding...16:57
*** lcurtis has joined #openstack-swift16:57
notmynametimburke: no I mean which specific unittest method is throwing the error. some `self.assert*()` thing16:58
*** psachin has joined #openstack-swift16:58
timburkeand i'm say, no, it doesn't fail with an assertion error. the call to logging.LoggerAdapter.exception(self, msg, *args, **kwargs) down in swift/common/utils.py throws a UnicodeEncodeError16:59
notmynameoh17:00
notmynamehere's an interesting comment in the unittest code17:00
notmyname            # don't switch to '{}' formatting in Python 2.X17:00
notmyname            # it changes the way unicode input is handled17:00
notmynamein unittest/case.py line 424 in the _formatMessage() method17:01
notmynametimburke: good luck!17:01
notmyname;-)17:01
*** arunman has quit IRC17:03
openstackgerritAlistair Coles proposed openstack/swift master: Enable per policy proxy config options  https://review.openstack.org/44824017:04
*** SkyRocknRoll has joined #openstack-swift17:05
openstackgerritAlistair Coles proposed openstack/swift master: Enable per policy proxy config options  https://review.openstack.org/44824017:12
*** lcurtis has quit IRC17:16
*** psachin has quit IRC17:20
notmynameFYI swift turns 7 on may 17 (ie in two weeks). I count swift's birthday as when it was first put into production17:22
MooingLemurnifty, hard to believe I've had it in production for just over 417:24
*** klrmn has joined #openstack-swift17:25
*** tesseract has quit IRC17:30
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements  https://review.openstack.org/8873617:35
*** ChubYann has joined #openstack-swift17:39
*** mingyu has quit IRC17:41
*** jordanP has quit IRC17:42
*** gyee has quit IRC17:45
openstackgerritTim Burke proposed openstack/swift master: Add message to account-creation assertion  https://review.openstack.org/46268717:51
notmynameacoles: is your per-policy config patch broken on probe tests, or is it the community cluster that's broken? https://8b86aea46fb38e6450f2-0e5f4c086da474abc1df58826577db2f.ssl.cf1.rackcdn.com/448240/6542/probetests/console.txt17:54
*** xinli has quit IRC17:55
acolesnotmyname: argh17:59
timburkefwiw, running locally now...17:59
acolesnotmyname: patch is broken for container sync17:59
acolesthe exuberance of getting all the other tests to pass blinded me to that :/18:00
notmynamelol18:01
acolesthe overview section of the community cluster probe test results is wonky, shows 0 fails when there were in fact 7. But the failures are reported below. https://review.openstack.org/#/c/177837/18:02
patchbotpatch 177837 - swift - Remove _ensure_flush() from SSYNC receiver (MERGED)18:02
acolesoops18:02
*** Sukhdev has joined #openstack-swift18:03
acolesprobe test results -> https://8b86aea46fb38e6450f2-0e5f4c086da474abc1df58826577db2f.ssl.cf1.rackcdn.com/448240/6542/probetests/result.html18:03
notmynamethat is so much easier to look at. why have I only been looking at the raw logs?18:06
acolesI'll look at fixing it later, dinner time now18:07
*** Sukhdev has quit IRC18:20
*** tonanhngo has joined #openstack-swift18:28
*** xinli has joined #openstack-swift18:31
*** mingyu has joined #openstack-swift18:41
timburkenotmyname: fyi https://review.openstack.org/#/c/462708/18:59
patchbotpatch 462708 - releases - Add release-note links for swift18:59
*** vint_bra has quit IRC19:07
*** clayg has joined #openstack-swift19:13
*** vint_bra has joined #openstack-swift19:14
*** JimCheung has quit IRC19:35
*** JimCheung has joined #openstack-swift19:35
*** JimCheung has quit IRC19:39
*** NM has quit IRC19:39
*** NM has joined #openstack-swift19:46
*** silor has joined #openstack-swift19:47
notmynametimburke: this time for real?19:58
timburkei really hope so19:59
timburkelooks promising: http://docs-draft.openstack.org/08/462708/1/check/gate-releases-docs-ubuntu-xenial/352e62b//doc/build/html/teams/swift.html20:00
*** JimCheung has joined #openstack-swift20:07
*** JimCheung has quit IRC20:11
*** SkyRocknRoll has quit IRC20:19
*** JimCheung has joined #openstack-swift20:19
*** NM has quit IRC20:20
*** silor has quit IRC20:24
openstackgerritMerged openstack/swift master: Remove the unused import code  https://review.openstack.org/46136020:33
*** silor has joined #openstack-swift20:33
*** silor has quit IRC20:40
*** jerrygb has quit IRC21:35
*** adriant has joined #openstack-swift21:36
openstackgerritAlistair Coles proposed openstack/swift master: Enable per policy proxy config options  https://review.openstack.org/44824021:39
acolesnotmyname: ^^ that *should* work, but I now need to add more tests ... tomorrow though!21:40
acolesgood night21:40
*** mingyu has quit IRC21:43
notmynamegood night21:46
*** mingyu has joined #openstack-swift21:46
*** klamath has quit IRC21:48
*** klamath has joined #openstack-swift21:50
*** mvk has joined #openstack-swift21:51
*** flwang has joined #openstack-swift21:52
*** mingyu has quit IRC22:09
*** mingyu has joined #openstack-swift22:09
*** vint_bra has quit IRC22:10
*** klamath has quit IRC22:11
*** mingyu has quit IRC22:13
*** flwang has quit IRC22:20
*** jerrygb has joined #openstack-swift22:37
*** jerrygb has quit IRC22:41
*** JimCheung has quit IRC23:00
*** catinthe_ has quit IRC23:08
*** lcurtis_ has quit IRC23:20
*** _JZ_ has quit IRC23:37
*** chsc has quit IRC23:37
*** cebruns has quit IRC23:39
*** cebruns has joined #openstack-swift23:42
*** xinli has quit IRC23:43
*** vint_bra has joined #openstack-swift23:43
*** vint_bra has quit IRC23:44
*** JimCheung has joined #openstack-swift23:47
*** cebruns has quit IRC23:54
*** vint_bra has joined #openstack-swift23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!