Wednesday, 2019-11-06

*** rcernin_ has joined #openstack-swift00:05
*** rcernin has quit IRC00:09
*** timburke_ has joined #openstack-swift00:25
*** ChanServ sets mode: +v timburke_00:25
*** timburk__ has joined #openstack-swift00:29
*** ChanServ sets mode: +v timburk__00:29
*** timburke_ has quit IRC00:30
*** gyee has quit IRC00:33
openstackgerritTim Burke proposed openstack/swift stable/rocky: Fix stable gate  https://review.opendev.org/69310200:42
openstackgerritMerged openstack/swift stable/stein: Fix stable gate  https://review.opendev.org/69300700:51
*** timburk__ has quit IRC00:57
*** timburke_ has joined #openstack-swift01:02
*** ChanServ sets mode: +v timburke_01:02
*** timburk__ has joined #openstack-swift01:07
*** ChanServ sets mode: +v timburk__01:07
*** timburke_ has quit IRC01:07
kota_i got the reachability to irc!01:14
*** timburk__ has quit IRC01:33
*** nanzha has joined #openstack-swift01:37
*** timburke_ has joined #openstack-swift01:40
*** ChanServ sets mode: +v timburke_01:40
*** openstackstatus has joined #openstack-swift01:44
*** ChanServ sets mode: +v openstackstatus01:44
*** mwheckmann has joined #openstack-swift01:48
viks___My swift-recon shows the following.01:50
viks___```01:50
viks___Distribution Graph:01:50
viks___  1%    3 *********************************************************************01:50
viks___  8%    3 *********************************************************************01:50
viks___ 16%    1 ***********************01:50
viks___ 17%    1 ***********************01:50
viks___ 21%    1 ***********************01:50
viks___ 23%    1 ***********************01:50
viks___ 24%    1 ***********************01:50
viks___ 25%    1 ***********************01:50
viks___ 26%    1 ***********************01:50
viks___ 28%    1 ***********************01:50
viks___ 31%    2 **********************************************01:50
viks___Just wanted to confirm if this is normal.. because 2 disks are around 90%. Rest all well below. Or should all disks fill rate is almost around same range?01:52
timburke_seongsoocho: hi!02:05
seongsoochohi~02:05
timburke_https://github.com/swiftstack/vagrant-swift-all-in-one02:06
claygviks___: if your rings have good balance, all disks should be about the same amount full02:08
claygviks___: varience in the 2-5% is normaly 7-10 is quite high but maybe not indicating a problem02:09
claygmost likely you've changed rings (added node?) and are waiting to rebalance - you probably need to tune your replicators (workers & concurrency)02:09
viks___clayg:  Here is my object ring details:02:17
viks___```02:17
viks___object.builder, build version 92, id 1e06239cf36344a1934df49cd2c980fb02:17
viks___262144 partitions, 3.000000 replicas, 1 regions, 2 zones, 15 devices, 0.00 balance, 0.00 dispersion02:17
viks___The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)02:17
viks___The overload factor is 0.00% (0.000000)02:17
viks___Ring file object.ring.gz not found, probably it hasn't been written yet02:17
viks___Devices:   id region zone    ip address:port replication ip:port  name weight partitions balance flags meta02:17
viks___            8      1    1 10.255.252.24:6200  172.16.252.24:6200 disk3 100.00      52498   -0.0002:17
viks___            0      1    1 10.255.252.24:6200  172.16.252.24:6200 disk4 100.00      52499    0.0002:17
viks___            1      1    1 10.255.252.24:6200  172.16.252.24:6200 disk5  99.00      51974    0.0002:17
viks___            2      1    1 10.255.252.24:6200  172.16.252.24:6200 disk6 100.00      52500    0.0002:17
claygviks___: balance and dispersionlook good!02:19
timburke_clayg: basically, have the get_more_nodes call include enough context that we could pre-populate https://github.com/openstack/swift/blob/2.23.0/swift/common/ring/ring.py#L411-L41602:20
claygviks___: there's a "find" command in https://docs.openstack.org/swift/latest/admin_guide.html#checking-handoff-partition-distribution you should run on your nodes02:23
*** tkajinam has joined #openstack-swift02:24
claygall the disks in your ring say that should have 52K parts (that's kind of a lot btw)02:26
claygso if you have one disk that is 99% full - but it has 100K parts the problem is just that it has a bunch of handoffs - it's replicator should be trying to shed them to the correct primaries02:26
viks___clayg:  In my case disk7 is having 92% ...  rest all below that as seen from the distribution graph... do not have 99% filled disk..02:30
clayghow many partition directories are on disk7?02:31
timburke_seongsoocho: https://bugs.launchpad.net/swift/+bug/1835324 is pretty small, should be easy to reproduce the issue02:32
openstackLaunchpad bug 1835324 in OpenStack Object Storage (swift) "Internal client's delete_object always logs 499 for 404s" [Low,Confirmed]02:32
timburke_https://bugs.launchpad.net/swift/+bug/1634382 is one clayg's wanted for a while02:32
openstackLaunchpad bug 1634382 in OpenStack Object Storage (swift) "return x-backend-storage-policy-index header from object server" [Wishlist,Confirmed] - Assigned to Christian Hugo (christianhugo)02:32
seongsoochotimburke_:  ok. I will try it.02:33
timburke_https://bugs.launchpad.net/swift/+bug/1693149 is surely a good idea, and the Ring code (as opposed to RingBuilder) already has it, so https://github.com/openstack/swift/commit/c6cebb6 would give you a good idea of what would be involved02:34
openstackLaunchpad bug 1693149 in OpenStack Object Storage (swift) "RingBuilder.save isn't atomic" [Wishlist,Confirmed]02:34
*** alexlecuyer-ch has joined #openstack-swift02:35
viks___clayg:02:37
viks___```02:37
viks___sudo find -L /srv/node/disk7 -maxdepth 3 -type d -wholename "*objects/*" | wc -l02:37
viks___3945102:37
viks___```02:37
timburke_so, thinking about server reloads, we've got https://github.com/eventlet/eventlet/pull/590 (so our HUP currently kills the connection) and https://review.opendev.org/#/c/691423/ should let us have seamless reloads with the same socket rather than needing to close & re-open02:39
patchbotpatch 691423 - swift - Seamlessly reload servers with SIGUSR1 - 8 patch sets02:39
claygviks___: that's odd, that seems like significantly LESS partitions than the ring has assigned to it?!  but you say it's nearly full!?02:43
clayghow many parts are on the other less full disks?  Maybe I'm missing something 🤔02:43
openstackgerritTim Burke proposed openstack/swift stable/stein: Authors/changelog for 2.21.1  https://review.opendev.org/69069902:46
*** diablo_rojo has joined #openstack-swift02:47
viks___clayg: should it match exactly the same number which is in the  ring  ?02:48
claygviks___: when the cluster is settled and stable - yes!02:48
viks___the below is for 18% filled disk:02:49
viks___```02:49
viks___sudo find -L /srv/node/disk6 -maxdepth 3 -type d -wholename "*objects/*" | wc -l02:49
viks___3991802:49
viks___```02:49
clayg!!!  So you have one disk with 39K parts thats 92% full - and other disk with 39K parts that's only 17% full!?02:49
openstackclayg: Error: "!!" is not a valid command.02:49
claygviks___: are these disks the same GB?02:50
viks___yes...02:50
*** alexlecuyer-ch has quit IRC02:50
*** alexlecuyer-ch has joined #openstack-swift02:50
viks___i'm checking on vagrant vm's.. so all the disk sizes are 2GB02:52
claygvagrant?  vms?  2GB disks... with 39K parts on them!?  viks___ i'm getting confused 🤣02:53
viks___just for testing purpose i have created this setup.... i mean i'm using part_power 18...  is that the problem? i mean i should'nt use that at all? can you clarify?02:57
claygtesting is great!03:09
claygso i guess that makes sense - part power 18 is too big to get even distribution unless you have a millon or so objects in your "cluster"03:11
claygand it sounds like you'd fill up your disks before then03:11
claygpart power 10 would probably be better for 15 disks03:11
clayg👍03:11
viks___clayg: Oh ok...Thanks a lot..  Just to understand,  how to migrate from 18 part power to 10 without fresh resetup.. is it doable?03:16
zaitcevNo03:22
clayg👋 @ zaitcev 😁03:25
claygzaitcev: rledisez says he thinks we can still make put+post work03:26
zaitcevclayg: does anyone need it? we managed to complete py3 with changes to eventlet.03:27
claygI thought I'd given up - but then timburke_ reminded me we can't get rid of eventlet unless we fix EC PUT to not need the 100-continue behavior and then rledisez convineced me that we could probably make it work03:27
viks___zaitcev:  clayg..  Ok... Thanks..03:27
claygviks___: 😢 sorry - it took us a long to get a story for part power *increase* - going the other direction has less usecases in production03:28
viks___clayg: Ok03:29
zaitcevThe remaining hurdle was actually... hmm... I forgot now. I remember you brought up some issue with a behavior change, but I cannot remember what it was.03:30
zaitcevAh03:30
zaitcevIt was the lack of -HUP03:31
zaitcevBefore, if HUP came while PUT was proceeding, it completed. But now, PUT completes, but POST does not, and the temp file disappears. On-disk consistency is not violated, but users see 500.03:32
zaitcevDid Romain resolve it?03:32
*** psachin has joined #openstack-swift03:33
claygwell, he had some idea about it otmp somewhere then link somewhere then move it later - i didn't quite follow - but he convinced me it would work03:33
zaitcevNo, that was his suggestion for another issue: getting into the same inode group as the directory.03:33
zaitcevSo theoretically he proposed a path, but I didn't implement it, only wrote down.03:34
*** psachin has joined #openstack-swift03:35
claygzaitcev: sounds like you understood him better than me - I thought it was a NEW idea03:36
*** alexlecuyer-ch has quit IRC03:36
zaitcevThere may be a misunderstanding. Did you go to Shanghai?03:38
zaitcevor we can hash it out on IRC03:38
openstackgerritTim Burke proposed openstack/python-swiftclient master: Rename "tests" directory to be "test" like in the swift repo  https://review.opendev.org/68777303:40
openstackgerritTim Burke proposed openstack/python-swiftclient master: v1auth: support endpoint_data_for() api  https://review.opendev.org/68777403:40
timburke_zaitcev: yeah, clayg and rledisez are both in Shanghai. along with kota_, alecuyer, cschwede and i03:43
*** alexlecuyer-ch has joined #openstack-swift03:44
*** alexlecuyer-ch has joined #openstack-swift03:45
*** tkajinam has quit IRC03:45
timburke_sounded like the idea was to have the PUT use O_TMPFILE to get the file in the right inode group, flush, and rather than keeping the fd around, go link it into the tmp directory. hand the client back the tmpfile name, so when the POST comes along we can go open it up again, add metadata, link it into place in the hashdir03:46
zaitcevSo who is getting the HUP - proxy or object server?03:48
claygseongsoocho: https://github.com/swiftstack/vagrant-swift-all-in-one/issues/8703:50
claygseongsoocho: I filed that, feel free to add detials or put your diff up as a PR!03:50
seongsoochoclayg:  OK thanks!03:50
alexlecuyer-chhttps://etherpad.openstack.org/p/swift-ptg-shanghai03:52
timburke_zaitcev: i think we've convinced ourselves that the troublesome case is the object-server -- proxy *should* just keep processing until we've sent the reply back to the client03:54
timburke_clayg: so i think that status field in the account_stat table actually *does* still matter... https://github.com/openstack/swift/blob/master/swift/account/backend.py#L269-L27703:58
*** diablo_rojo__ has joined #openstack-swift04:03
*** diablo_rojo has quit IRC04:04
openstackgerritTim Burke proposed openstack/python-swiftclient master: Rename "tests" directory to be "test" like in the swift repo  https://review.opendev.org/68777304:04
openstackgerritTim Burke proposed openstack/python-swiftclient master: v1auth: support endpoint_data_for() api  https://review.opendev.org/68777404:04
*** mwheckmann has quit IRC04:07
*** diablo_rojo__ has quit IRC04:12
openstackgerritRomain LE DISEZ proposed openstack/swift master: proxy: stop sending chunks to objects with a Queue  https://review.opendev.org/69311604:14
*** alexlecuyer-ch has quit IRC04:22
*** timburke_ has quit IRC04:25
*** timburke_ has joined #openstack-swift04:27
*** ChanServ sets mode: +v timburke_04:27
*** timburke_ has quit IRC04:32
*** timburke_ has joined #openstack-swift05:06
*** ChanServ sets mode: +v timburke_05:06
*** alexlecuyer-ch has joined #openstack-swift05:10
*** alexlecuyer-ch has quit IRC05:12
*** alexlecuyer-ch has joined #openstack-swift05:19
alexlecuyer-cha few notes about how we may use larger (12+TB) on object servers https://etherpad.openstack.org/p/swift-ptg-shanghai-large-drives05:34
*** irclogbot_0 has quit IRC05:34
*** irclogbot_1 has joined #openstack-swift05:38
*** tkajinam has joined #openstack-swift05:39
*** diablo_rojo has joined #openstack-swift05:40
*** diablo_rojo has quit IRC05:52
*** diablo_rojo has joined #openstack-swift05:53
*** timburke_ has quit IRC05:54
*** alexlecuyer-ch has quit IRC05:54
*** pcaruana has joined #openstack-swift05:59
*** timburke_ has joined #openstack-swift06:02
*** ChanServ sets mode: +v timburke_06:02
mattoliverauhey, it's cool to see more of you on IRC. I assume the PTG side of things has finally started ;)06:17
*** nanzha has quit IRC06:24
mattoliveraualecuyer: interesting. Wish I was there.06:25
*** nanzha has joined #openstack-swift06:26
*** timburke_ has quit IRC06:26
*** timburke_ has joined #openstack-swift06:27
*** ChanServ sets mode: +v timburke_06:27
*** timburke_ has quit IRC06:28
*** timburke_ has joined #openstack-swift06:28
*** ChanServ sets mode: +v timburke_06:28
*** timburk__ has joined #openstack-swift06:29
*** ChanServ sets mode: +v timburk__06:29
*** timburke_ has quit IRC06:33
*** alexlecuyer-ch has joined #openstack-swift06:49
*** alexlecuyer-ch has quit IRC06:51
*** alexlecuyer-ch has joined #openstack-swift06:51
*** nanzha has quit IRC06:58
*** nanzha has joined #openstack-swift07:00
*** tkajinam has quit IRC07:08
openstackgerritAlexandre Lécuyer proposed openstack/swift feature/losf: Erase object header if we fail to register it to the index-server  https://review.opendev.org/69312707:11
*** diablo_rojo__ has joined #openstack-swift07:18
*** fungi has quit IRC07:20
*** diablo_rojo__ has quit IRC07:20
*** diablo_rojo__ has joined #openstack-swift07:20
*** diablo_rojo has quit IRC07:20
*** diablo_rojo__ is now known as diablo_rojo07:21
*** fungi has joined #openstack-swift07:24
claygmattoliverau: yes, we miss you!!!07:26
openstackgerritRomain LE DISEZ proposed openstack/swift master: proxy: stop sending chunks to objects with a Queue  https://review.opendev.org/69311607:33
*** diablo_rojo__ has joined #openstack-swift07:34
*** diablo_rojo__ has quit IRC07:35
*** diablo_rojo__ has joined #openstack-swift07:35
*** diablo_rojo has quit IRC07:35
*** diablo_rojoooooo has joined #openstack-swift07:37
*** diablo_rojo__ has quit IRC07:38
openstackgerritSeongSoo Cho proposed openstack/swift master: Fix bug internal client's delete_object logs 499  https://review.opendev.org/69312807:40
*** diablo_rojo__ has joined #openstack-swift07:41
*** diablo_rojoooooo has quit IRC07:45
*** diablo_rojo__ has quit IRC07:47
*** rdejoux has joined #openstack-swift08:04
*** rcernin_ has quit IRC08:06
*** tesseract has joined #openstack-swift08:08
openstackgerritSeongSoo Cho proposed openstack/swift master: Fix bug internal client's delete_object logs 499  https://review.opendev.org/69312808:18
*** nanzha has quit IRC08:35
*** nanzha has joined #openstack-swift08:36
openstackgerritSeongSoo Cho proposed openstack/swift master: Fix bug internal client's delete_object logs 499  https://review.opendev.org/69312808:52
openstackgerritTim Burke proposed openstack/python-swiftclient master: v1auth: support endpoint_data_for() api  https://review.opendev.org/68777408:55
*** timburk__ has quit IRC08:55
openstackgerritAlexandre Lécuyer proposed openstack/swift feature/losf: Erase object header if we fail to register it to the index-server  https://review.opendev.org/69312708:57
*** alexlecuyer-ch has quit IRC08:57
*** mikecmpbll has joined #openstack-swift09:13
zigoIn the doc, I can read:09:46
zigo"The auto_shard option is currently NOT recommended for production systems and shoud be set to false"09:46
zigoIs this still truth?09:46
*** timburke_ has joined #openstack-swift09:46
*** ChanServ sets mode: +v timburke_09:46
zigotimburke: ^09:47
openstackgerritTim Burke proposed openstack/swift master: Fix bug internal client's delete_object logs 499  https://review.opendev.org/69312809:49
timburke_zigo: eh, what? sorry, not on my bouncer at the moment09:52
*** timburke_ has quit IRC09:55
*** timburke_ has joined #openstack-swift09:56
*** ChanServ sets mode: +v timburke_09:56
zigotimburke_: I was wondering if "The auto_shard option is currently NOT recommended for production systems and shoud be set to false" in the doc is still valid, or if auto_shard is ok in production these days.09:56
*** nanzha has quit IRC09:57
*** nanzha has joined #openstack-swift09:57
*** timburke_ has quit IRC09:57
*** timburke_ has joined #openstack-swift09:58
*** ChanServ sets mode: +v timburke_09:58
zigotimburke_: I was wondering if "The auto_shard option is currently NOT recommended for production systems and shoud be set to false" in the doc is still valid, or if auto_shard is ok in production these days.09:58
timburke_i don't know of anyone currently running it in pod with that turned on. i know there are some concerns about what might happen during rebalances10:01
timburke_the auto-sharding relies on replica index to choose a leader, so during a rebalance there's a chance that you'd get a second node thinking it ought to be the leader...10:01
timburke_i thinki like the plan outlined in https://review.opendev.org/#/c/667579/ though!10:02
patchbotpatch 667579 - swift - sharding: first attempt at _elect_leader - 5 patch sets10:02
timburke_sorry, gotta go -- dinner time10:03
zigoHave a good evening!10:03
*** timburke_ has quit IRC10:04
*** ianychoi has quit IRC10:13
*** timburke_ has joined #openstack-swift10:46
*** ChanServ sets mode: +v timburke_10:46
*** timburke_ has quit IRC10:51
*** rcernin_ has joined #openstack-swift10:55
*** rcernin_ has quit IRC11:31
*** timburke_ has joined #openstack-swift11:50
*** ChanServ sets mode: +v timburke_11:50
*** timburke_ has quit IRC11:54
openstackgerritMerged openstack/swift master: Fix bug internal client's delete_object logs 499  https://review.opendev.org/69312812:14
*** tomha has joined #openstack-swift12:29
*** timburke_ has joined #openstack-swift12:31
*** ChanServ sets mode: +v timburke_12:31
*** timburke_ has quit IRC12:35
*** mikecmpbll has quit IRC12:39
*** mikecmpbll has joined #openstack-swift12:52
*** timburke_ has joined #openstack-swift13:01
*** ChanServ sets mode: +v timburke_13:01
*** timburke_ has quit IRC13:06
*** timburke_ has joined #openstack-swift14:34
*** ChanServ sets mode: +v timburke_14:34
*** alexlecuyer-ch has joined #openstack-swift14:41
*** timburk__ has joined #openstack-swift14:45
*** ChanServ sets mode: +v timburk__14:45
*** alexlecuyer-ch has quit IRC14:45
*** timburke_ has quit IRC14:45
*** renich has joined #openstack-swift14:48
*** alexlecuyer-ch has joined #openstack-swift14:48
*** alexlecuyer-ch has quit IRC14:48
*** mikecmpbll has quit IRC14:56
*** renich_ has joined #openstack-swift15:02
*** renich has quit IRC15:03
*** timburk__ has quit IRC15:16
*** timburke_ has joined #openstack-swift15:26
*** ChanServ sets mode: +v timburke_15:26
*** timburke_ has quit IRC15:30
*** gyee has joined #openstack-swift15:45
*** timburke_ has joined #openstack-swift16:06
*** ChanServ sets mode: +v timburke_16:06
*** timburke_ has quit IRC16:11
*** nanzha has quit IRC16:40
*** psachin has quit IRC17:06
*** renich has joined #openstack-swift17:14
*** renich_ has quit IRC17:15
*** rdejoux has quit IRC17:38
openstackgerritGilles Biannic proposed openstack/swift master: probe: Add test for syncing a delete when the remote 404s  https://review.opendev.org/69146317:43
openstackgerritGilles Biannic proposed openstack/swift master: Stop retrying every deletes in container-sync  https://review.opendev.org/69172917:43
*** renich has quit IRC18:24
DHEis there an ideal version of python for swift? if I'm installing a new system just for swift and had full flexibility in which version I'm using, what's suggested? 2.7? 3.6?19:09
*** viks___ has quit IRC19:27
*** nanzha has joined #openstack-swift20:07
*** nanzha has quit IRC20:12
openstackgerritMerged openstack/swift stable/rocky: Fix stable gate  https://review.opendev.org/69310220:16
*** timburke_ has joined #openstack-swift21:08
*** ChanServ sets mode: +v timburke_21:08
*** pcaruana has quit IRC21:27
*** renich has joined #openstack-swift22:15
*** tesseract has quit IRC22:28
*** rcernin has joined #openstack-swift22:53
*** threestrands has joined #openstack-swift23:07
*** renich has quit IRC23:16
*** timburke_ has quit IRC23:54
*** timburke_ has joined #openstack-swift23:54
*** ChanServ sets mode: +v timburke_23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!