Monday, 2018-02-26

*** ^andrea^ has joined #openstack-swift00:07
*** geaaru_ has quit IRC00:15
*** tovin07_ has joined #openstack-swift00:46
*** cshastri has joined #openstack-swift01:11
*** briancline has quit IRC02:15
*** briancline has joined #openstack-swift02:16
*** tovin07 has joined #openstack-swift03:07
*** itlinux has quit IRC03:26
*** psachin has joined #openstack-swift03:33
*** kei_yama has quit IRC03:41
*** threestrands has quit IRC03:47
*** kei_yama has joined #openstack-swift04:03
*** StevenK has quit IRC04:21
hugokuohi guys04:21
hugokuoanyone knows what would cause this?04:21
*** StevenK has joined #openstack-swift04:21
hugokuoFeb 26 03:36:53 node1 object-replicator: Lockup detected.. killing live coros.04:21
*** links has joined #openstack-swift04:45
*** SkyRocknRoll has joined #openstack-swift05:19
*** SkyRocknRoll has quit IRC05:20
*** cshastri_ has joined #openstack-swift06:02
*** links has quit IRC06:04
*** links has joined #openstack-swift06:04
*** cshastri has quit IRC06:07
*** cshastri_ has quit IRC06:07
*** armaan has quit IRC06:18
*** armaan has joined #openstack-swift06:18
*** cshastri_ has joined #openstack-swift06:19
*** armaan has quit IRC06:38
*** armaan has joined #openstack-swift06:38
*** armaan has quit IRC07:02
*** armaan has joined #openstack-swift07:02
kota_good morning Dublim07:24
kota_hugokuo: IIRC, that is caused when monitor process of object-replicator find nothing progressed while the monitoring cycle (maybe every 5 minutes or 15 minutes?)07:26
kota_Dublin!07:26
hugokuomorning07:27
hugokuokota_: Are you in Dublin?07:27
kota_yes i am here07:28
hugokuokota_: So I assume that there're rsync clients are sending data to other hosts. But they take more than the timeout due to the limited bandwidth.07:29
hugokuoIn this case, object-replicator doesn't get any update by rsync clients so it thought it's hanging then terminated it.07:29
mattoliverauMorning07:30
hugokuoThe sequel will be lots of rsync client processes in running. But those are not managed by replicator anymore. The restarted object-replicator spawns new rsync clients.07:31
kota_oic07:31
hugokuoThe total rsync clients would be double than the object-replicator concurrency.07:31
hugokuoIt makes the total bandwidth shared by more rsync clients. The result would be more and more rsync in a host.07:32
kota_that might happen, because replicator cannot detect the progress of rsync.07:32
hugokuoright.07:32
hugokuothat's painful for a cluster has bigger size objects/segment.07:33
kota_i'm wondering we could change the detect timeframe to be increased....07:33
hugokuoI'm considering it as a bug tho. Swift doesn't handle this part very well especial a cluster has regional setup.07:33
*** pcaruana has joined #openstack-swift07:33
hugokuoThe WAN suppose to be slower in general07:34
hugokuoIf not kill old rsync client processes, it'll get worse and worse.07:34
hugokuoBy the end, the total object-server and rsync processes will hit the maximum openfiles/nproc07:35
hugokuofurther more, other swift workers won't be able to folk subprocesses anymore. Then the worker will be terminated.07:35
hugokuoIt's kind like a chain-effect.07:36
hugokuoclayg: notmyname:^^ Do you think my assumption is reasonable or not?07:40
*** hseipp has joined #openstack-swift07:45
openstackgerritmelissaml proposed openstack/swift master: Fix the incorrect reference links  https://review.openstack.org/54792307:49
*** rcernin has quit IRC07:49
kota_hugokuo: I'll look at around the protocol today, but at this morment, I should go breakfast before PTG will begin.07:50
timburkegood morning, dublin07:53
timburkekota_: hugokuo: you guys reminded me of https://review.openstack.org/#/c/429015/ -- i wonder if there are still valuable ideas there that should get rebased...08:01
patchbotpatch 429015 - swift - Add abort flag to replicator/reconstructor08:01
*** tesseract has joined #openstack-swift08:15
*** hseipp has quit IRC08:15
*** hseipp has joined #openstack-swift08:17
*** cbartz has joined #openstack-swift08:40
*** links has quit IRC09:00
*** m_kazuhiro has joined #openstack-swift09:01
*** links has joined #openstack-swift09:06
*** armaan has quit IRC09:06
*** armaan has joined #openstack-swift09:10
*** m_kazuhiro has quit IRC09:11
*** zhurong has joined #openstack-swift09:12
*** hoonetorg has quit IRC09:18
*** mvk has quit IRC09:20
*** armaan has quit IRC09:29
*** armaan has joined #openstack-swift09:29
*** hoonetorg has joined #openstack-swift09:32
*** armaan_ has joined #openstack-swift09:36
*** ashok_n has quit IRC09:36
*** armaan has quit IRC09:39
notmynamegood morning09:40
notmynameat the PTG, we're in room 661 today09:40
notmynamehttps://etherpad.openstack.org/p/Swift_Operational_Feedback is now linked from our main PTG etherpad10:01
notmynamehttps://etherpad.openstack.org/p/Dublin_PTG_Swift  is the main PTG etherpad10:01
*** tovin07_ has quit IRC10:06
*** zaitcev has joined #openstack-swift10:06
*** ChanServ sets mode: +v zaitcev10:06
notmynametdasilva: cschwede_: you are in the fast-forward upgrades room?10:07
tdasilvayep10:07
notmynamezaitcev: we're in 661 as a home base for now10:07
zaitcevnotmyname, thanks a lot10:07
notmynametdasilva: cool! means I don't need to be there :-)10:07
tdasilvanotmyname: lol, it's been hard to follow10:08
zaitcevthere's no space left to sit in fast-upgrade room and they talk quetly, so I cannot hear the discussion10:08
notmynametdasilva: it would probably be helpful if you say, "why can't you just be like swift" ;-)10:08
notmynameand by "helpful" I mean "not helpful" ;-)10:08
zaitcevActually10:08
tdasilvalol, yeah, that's a really nice way to make new friends10:09
notmynameheh10:09
notmynameand mattoliverau is in the "first contact" sig room, so we've got that covered too10:09
tdasilva"first contact"?10:09
zaitcevI am quite convinced that it must be possible to be like Swift, even with DB schema changes. It's just that nobody is willing to put work towards it: always support 2 nodes talking to each other across N and N+1.10:09
zaitcevWell, they depend on MySQL anway10:10
zaitcevAnd Rabbit10:10
* zaitcev migrates to 66110:12
*** zaitcev has quit IRC10:12
*** armaan_ has quit IRC10:17
timburkenotmyname: https://governance.openstack.org/tc/goals/rocky/index.html10:17
*** armaan has joined #openstack-swift10:18
notmynamethanks10:19
*** zaitcev has joined #openstack-swift10:20
*** ChanServ sets mode: +v zaitcev10:20
*** geaaru has joined #openstack-swift10:23
*** armaan_ has joined #openstack-swift10:26
*** charz has quit IRC10:26
*** hugokuo has quit IRC10:26
*** timburke has quit IRC10:26
openstackgerritTim Burke proposed openstack/swift master: py3: port common/storage_policy.py  https://review.openstack.org/54018810:26
openstackgerritTim Burke proposed openstack/swift master: py3: port common/memcached.py  https://review.openstack.org/54018910:26
openstackgerritTim Burke proposed openstack/swift master: py3: port common/wsgi.py  https://review.openstack.org/54019010:26
openstackgerritTim Burke proposed openstack/swift master: py3: Port more cli tools  https://review.openstack.org/54796910:26
*** armaan_ has quit IRC10:27
*** armaan_ has joined #openstack-swift10:27
*** armaan has quit IRC10:28
*** charz has joined #openstack-swift10:31
*** timburke has joined #openstack-swift10:32
*** ChanServ sets mode: +v timburke10:32
kota_m_kazuhiro: done on the one patch.10:32
*** hugokuo has joined #openstack-swift10:32
notmynamemattoliverau: how's the first contact room?10:33
*** armaan has joined #openstack-swift10:36
openstackgerritTim Burke proposed openstack/swift master: Tighten up assertions around expirer's concurrency  https://review.openstack.org/54711410:37
*** armaan_ has quit IRC10:39
notmynamewow. this is a giant room with very soft-spoken people10:42
*** armaan has quit IRC10:46
*** armaan has joined #openstack-swift10:47
openstackgerritTim Burke proposed openstack/swift master: Don't log tracebacks when we know where they're going to end  https://review.openstack.org/54798110:49
*** armaan_ has joined #openstack-swift10:49
*** zhurong has quit IRC10:50
*** armaan has quit IRC10:52
*** kei_yama has quit IRC10:58
*** m_kazuhiro has joined #openstack-swift11:05
m_kazuhirogood morning11:05
m_kazuhiromattoliverau: kota_: Do you have time to review patch 543796 ? This is small patch (38 lines) for expirer general task queue and is added +2 by tim.11:08
patchbothttps://review.openstack.org/#/c/543796/ - swift - Fix expirer's invalid task object names in unit tests11:08
openstackgerritMerged openstack/swift master: Fix the incorrect reference links  https://review.openstack.org/54792311:12
openstackgerritMerged openstack/swift master: Remove confusing assertion from expirer's unit test  https://review.openstack.org/54378411:13
openstackgerritMerged openstack/swift master: Tighten up assertions around expirer's concurrency  https://review.openstack.org/54711411:13
notmynamehmm... all this time when they were talking about hierarchical project quotas, I thought it was about openstack projects (eg nova->glance->swift). turns out it's about the *keystone* project (nee tenant) hierarchy11:26
notmynametimburke: how are things in 661?11:35
timburkenotmyname: torgomatic and acoles are here now! kota_ came back, too11:36
timburketdasilva and cschwede_ are here11:36
notmynamethat sounds like fun11:37
*** ianychoi has quit IRC11:40
*** zhurong has joined #openstack-swift11:40
*** ianychoi has joined #openstack-swift11:40
*** rcernin has joined #openstack-swift11:40
*** armaan_ has quit IRC11:45
*** armaan has joined #openstack-swift11:46
timburkeand now we've got a clayg!11:47
notmynametdasilva: cschwede_: in the upgrades room, was there any talk of "checkpoint releases"?11:47
cschwede_notmyname: no (at least not until we left)11:49
*** cshastri_ has quit IRC11:49
*** mvk has joined #openstack-swift11:49
cschwede_notmyname: on the other side, every release is kind of a checkpoint release, right? i mean you still need to switch to release N+1, N+2, until you finally are at N+3 - it's not that you skip an intermediate release if you upgrade from N to N+X11:50
cschwede_where X is > 111:51
notmynamecschwede_: well, you can with swift :-)11:51
cschwede_notmyname: we just make ops life too easy allowing that ;)11:52
*** hseipp has quit IRC11:55
*** zhurong has quit IRC12:07
openstackgerritTim Burke proposed openstack/swift master: py3: port common/storage_policy.py  https://review.openstack.org/54018812:13
openstackgerritTim Burke proposed openstack/swift master: py3: port common/memcached.py  https://review.openstack.org/54018912:13
openstackgerritTim Burke proposed openstack/swift master: py3: port common/wsgi.py  https://review.openstack.org/54019012:13
openstackgerritTim Burke proposed openstack/swift master: py3: Port more cli tools  https://review.openstack.org/54796912:13
openstackgerritSamuel Merritt proposed openstack/swift master: Fix suffix-byte-range responses for zero-byte EC objects.  https://review.openstack.org/52656512:18
openstackgerritTim Burke proposed openstack/swift master: Make If-None-Match:* work properly with 0-byte PUTs  https://review.openstack.org/47699212:23
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Kill rsync coros when lockup detector tries to kill the process  https://review.openstack.org/54801812:26
*** mvk has quit IRC12:27
kota_hugokuo: (and Martin) patch 548018 may be able to mitigate to increase such a zombie rsync procs but I'm not sure this is fine fix for you because that will halt the in-sync rsync process.12:28
patchbothttps://review.openstack.org/#/c/548018/ - swift - Kill rsync coros when lockup detector tries to kil...12:28
kota_as timburke suggested now in f2f, waiting rather than kill process would be nice to have.12:32
*** zaitcev has quit IRC12:33
timburkeideally, we have the replicator read some output, stick it in a buffer, send a heartbeat, read some output, stick it in a buffer, send a heartbeat, ... and we only go murdering rsync & coros when we haven't received a heartbeat in a while12:35
*** m_kazuhiro has quit IRC12:36
openstackgerritTim Burke proposed openstack/swift master: py3: port common/memcached.py  https://review.openstack.org/54018912:39
openstackgerritTim Burke proposed openstack/swift master: py3: port common/wsgi.py  https://review.openstack.org/54019012:39
openstackgerritTim Burke proposed openstack/swift master: py3: Port more cli tools  https://review.openstack.org/54796912:39
*** armaan has quit IRC12:44
*** armaan has joined #openstack-swift12:44
*** SkyRocknRoll has joined #openstack-swift13:05
openstackgerritTim Burke proposed openstack/swift master: Make If-None-Match:* work properly with 0-byte PUTs  https://review.openstack.org/47699213:13
*** links has quit IRC13:19
*** armaan has quit IRC13:25
*** armaan has joined #openstack-swift13:26
notmyname#swift book Suite 661-MonP113:26
notmynamemattoliverau: how's the newcomer session?13:35
claygmattoliverau: where ARE you?!13:36
torgomaticWhen a lockup is detected, the only sane thing to do is to kill the entire process. You can't just kill co-routines or anything like that; if one of your coroutines is holding a lock and you wipe that coro away, then the lock will never be released.13:40
torgomaticHonestly, it's nuts to me that we have the whole kill-coroutines-and-hope thing going now. Does it actually help things?13:41
*** zhurong has joined #openstack-swift13:44
mattoliverauYeah it's ok.. chatting, I'm heckling as much as I can. Once we finish here, Ill need to find out where y'all are hiding :)13:45
*** mvk has joined #openstack-swift13:55
notmynamemattoliverau: still in 66113:56
*** mvk has quit IRC14:02
*** mvk has joined #openstack-swift14:09
*** Supun has joined #openstack-swift14:13
claygi was gunna look at patch 54801814:13
patchbothttps://review.openstack.org/#/c/548018/ - swift - Kill rsync coros when lockup detector tries to kil...14:13
claygtorgomatic: yeah, it could be good money at bad14:14
claygtorgomatic: but what else are we gunna do?14:14
torgomaticclayg: detect a lockup and blow up the whole replicator process14:14
kota_so should NOT we kill the rsync coros?14:14
clayg^ right!?14:15
claygtorgomatic: yeah... i mean I guess we could agree on that here and do it... can we restart tho?14:15
claygtorgomatic: kota_: if we added the per-disk "workers" model it's easy to throw away the process and let it restart14:15
kota_the main issue is the lockup detector will cause a lot of rysnc processes because the parent eventlet thread has gone (and the next will up to call a new rsync).14:15
kota_clayg: +114:16
*** hseipp has joined #openstack-swift14:16
torgomaticyeah, throwing away rsync processes is good, but (kill coros + kill rsyncs) is not necessarily enough to return us to a good state14:16
torgomatickilling the whole replicator and letting systemd restart it is a (heavy) way to return to a good state14:17
notmynameit's a long way to the nally room14:19
openstackgerritAlexandre Lécuyer proposed openstack/swift master: Add optional parameters to common/utils fallocate()  https://review.openstack.org/54803414:21
*** saint_ has joined #openstack-swift14:25
*** zhurong has quit IRC14:27
*** links has joined #openstack-swift14:27
timburkeclayg: https://github.com/Pylons/webob/blob/1.7.4/webob/request.py#L152-L18514:32
*** m_kazuhiro has joined #openstack-swift14:38
timburketdasilva: https://github.com/openstack/swift3/blob/master/swift3/request.py#L926-L93614:40
openstackgerritKota Tsuyuzaki proposed openstack/swift feature/s3api: Fix functests name address swift3 -> s3api migration  https://review.openstack.org/54803814:42
notmynametdasilva: timburke: clayg: added my +A to the backport14:42
notmynameand +2. it may have to have that too in order to trigger the gate14:43
*** zhurong has joined #openstack-swift14:43
tdasilvanotmyname: thanks14:46
*** hseipp has quit IRC14:48
openstackgerritTim Burke proposed openstack/swift master: Add in-process cross-policy func tests  https://review.openstack.org/53665814:53
DHEis lab/testing the only reason you would run multiple instances of the object server, or is it useful in real world deployments?14:54
torgomaticUsually just for testing.15:02
*** psachin has quit IRC15:02
torgomaticFor real deployments, you'll want to run one object server with 2-4 worker processes per disk.15:03
*** hseipp has joined #openstack-swift15:07
notmynameeveryone still in the same 661 room?15:08
kota_notmyname: almost of them. clayg went to somewhere (idk)15:10
notmynamethanks. I kept the room on the schedule then15:11
kota_notmyname: appreciated!15:13
openstackgerritKota Tsuyuzaki proposed openstack/swift feature/s3api: Merge remote-tracking branch 'remotes/origin/master' into feature/s3api  https://review.openstack.org/54805215:17
*** Supun has quit IRC15:23
*** zaitcev has joined #openstack-swift15:34
*** ChanServ sets mode: +v zaitcev15:34
*** Supun has joined #openstack-swift15:39
*** links has quit IRC15:42
*** armaan has quit IRC15:48
*** armaan has joined #openstack-swift15:49
notmynameI left by bag and coat in the room15:51
openstackgerritMerged openstack/swift feature/s3api: Merge s3api func tests  https://review.openstack.org/53783815:59
*** zhurong has quit IRC16:00
tdasilvanotmyname: we are still here16:00
openstackgerritAlexandre Lécuyer proposed openstack/swift master: Add optional parameters to common/utils fallocate()  https://review.openstack.org/54803416:01
*** pcaruana has quit IRC16:01
*** ^andrea^ has quit IRC16:04
*** itlinux has joined #openstack-swift16:08
openstackgerritKazuhiro MIYAHARA proposed openstack/swift master: Fix expirer's invalid task object names in unit tests  https://review.openstack.org/54379616:11
mattoliverauFyi donuts and beer on level 516:26
notmynamemattoliverau: now?16:30
mattoliverauYeah, well about 10 mins ago when I was there. Beer is apparently near the rego desk16:32
kota_mattoliverau: beer!16:33
*** zhurong has joined #openstack-swift16:44
*** cbartz has quit IRC16:50
*** gyee has joined #openstack-swift16:51
*** tonyb_ is now known as tonyb16:56
*** tesseract has quit IRC16:59
*** zhurong has quit IRC17:01
*** d0ugal has joined #openstack-swift17:07
*** armaan has quit IRC17:07
*** zaitcev has quit IRC17:07
*** armaan has joined #openstack-swift17:10
*** openstack has joined #openstack-swift17:16
*** ChanServ sets mode: +o openstack17:16
*** m_kazuhiro has quit IRC17:16
*** SkyRocknRoll has quit IRC17:17
*** d0ugal has quit IRC17:24
*** Supun has quit IRC17:25
*** Supun has joined #openstack-swift17:25
*** mvk has quit IRC17:25
*** hseipp has quit IRC17:32
*** hseipp has joined #openstack-swift17:32
*** ukaynar has joined #openstack-swift17:34
*** hseipp has quit IRC17:54
openstackgerritMerged openstack/swift feature/s3api: Fix functests name address swift3 -> s3api migration  https://review.openstack.org/54803818:06
openstackgerritMerged openstack/swift master: py3: port common/storage_policy.py  https://review.openstack.org/54018818:12
openstackgerritTim Burke proposed openstack/swift feature/deep: Tell the new broker that it's active before replicating  https://review.openstack.org/54807618:17
openstackgerritTim Burke proposed openstack/swift feature/deep: Ignore found/created shards when shrinking  https://review.openstack.org/54807718:17
*** neonpastor has quit IRC18:18
*** neonpastor has joined #openstack-swift18:20
*** neonpastor has quit IRC18:22
*** neonpastor has joined #openstack-swift18:25
*** gerardgine has joined #openstack-swift18:28
*** geaaru has quit IRC18:34
*** neonpastor has quit IRC18:39
*** mvk has joined #openstack-swift18:43
*** armaan has quit IRC19:00
*** ukaynar has quit IRC19:00
*** armaan has joined #openstack-swift19:00
*** ukaynar has joined #openstack-swift19:00
*** ukaynar has quit IRC19:01
*** ukaynar has joined #openstack-swift19:07
*** raginbaji has joined #openstack-swift19:12
*** gerardgine has quit IRC19:22
*** gerardgine has joined #openstack-swift19:22
*** gerardgine has quit IRC19:26
*** gerardgine has joined #openstack-swift19:26
*** Supun has quit IRC19:32
*** ukaynar has quit IRC19:32
*** ukaynar has joined #openstack-swift19:33
*** ukaynar_ has joined #openstack-swift19:35
*** ukaynar has quit IRC19:35
*** ukaynar_ has quit IRC19:51
*** geaaru has joined #openstack-swift19:53
*** gyee has quit IRC19:56
*** neonpastor has joined #openstack-swift19:57
*** ukaynar has joined #openstack-swift20:02
*** _alastor__ has quit IRC20:08
*** armaan has quit IRC20:27
*** armaan has joined #openstack-swift20:28
*** armaan has quit IRC20:41
*** armaan has joined #openstack-swift20:41
*** armaan has quit IRC21:02
*** armaan has joined #openstack-swift21:02
*** zhurong has joined #openstack-swift21:07
*** gerardgine has quit IRC21:09
*** gerardgine has joined #openstack-swift21:13
*** zhurong has quit IRC21:20
*** gerardgine has quit IRC21:22
openstackgerritMerged openstack/swift feature/deep: Get accurate listings while sharding shards  https://review.openstack.org/54766221:23
*** gerardgine has joined #openstack-swift21:34
*** rcernin has quit IRC21:37
*** ukaynar has quit IRC21:38
*** jistr has quit IRC22:06
*** jistr has joined #openstack-swift22:06
*** ggine has joined #openstack-swift22:07
*** gerardgine has quit IRC22:08
*** ggine has quit IRC22:13
*** rcernin has joined #openstack-swift22:23
*** gerardgine has joined #openstack-swift22:24
*** itlinux has quit IRC22:42
*** ukaynar has joined #openstack-swift22:55
*** gerardgine has quit IRC23:06
*** kei_yama has joined #openstack-swift23:24
openstackgerritMerged openstack/swift master: Fix expirer's invalid task object names in unit tests  https://review.openstack.org/54379623:27

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!