Tuesday, 2017-09-19

*** chlong has joined #openstack-swift00:20
*** mat128 has joined #openstack-swift00:28
*** tovin07_ has joined #openstack-swift00:41
timburkemattoliverau: fyi, i've got a patch that puts back a call to broker.get_info() in https://review.openstack.org/#/c/504981/1/swift/container/sharder.py -- maybe should have broke that out as its own thing, but the rest of the patch is fairly straight-forward so i figured... eh, why not?00:59
patchbotpatch 504981 - swift (feature/deep) - Move sharding header translation to the proxy00:59
timburkei think the if and comment will make me less likely to want to remove it again later :-)00:59
mattoliveraulol00:59
mattoliverauI'll take a look :)01:00
*** klrmn has quit IRC01:03
timburkeif you merge it before acoles wakes up, he'll never need to figure out why hash_path is raising `TypeError: sequence item 0: expected string, NoneType found` ;-)01:03
timburkealso, i think that probe test should be ready to merge now. still not sure why our time mock bled over from container/sharder.py to common/db_replicator.py though01:06
timburkemaybe it has to do with us importing the time module in both, rather than importing just the time *function*?01:07
* timburke shrugs01:07
*** Sukhdev has quit IRC01:12
*** cshastri has joined #openstack-swift01:21
*** itlinux has joined #openstack-swift01:48
*** chlong has quit IRC02:03
mattoliverautimburke: oh yeah, you need to call get_info to get the .container and .account. that burnt me once :)02:18
*** itlinux has quit IRC02:37
-openstackstatus- NOTICE: Gerrit is being restarted to feed its insatiable memory appetite02:41
*** itlinux has joined #openstack-swift04:03
*** mat128 has quit IRC04:10
*** itlinux has quit IRC04:36
*** cnf has joined #openstack-swift04:45
*** mabrams has joined #openstack-swift04:54
*** gyee has quit IRC04:58
*** ChubYann has quit IRC05:52
*** Jeffrey4l has quit IRC06:05
*** mwheckmann has quit IRC06:05
*** afazekas has quit IRC06:05
*** dhellmann has quit IRC06:05
*** _d34dh0r53_ has quit IRC06:05
*** StevenK has quit IRC06:05
*** dr_gogeta86 has quit IRC06:05
*** tdasilva has quit IRC06:05
*** acoles has quit IRC06:05
*** d34dh0r53 has joined #openstack-swift06:05
*** dhellmann has joined #openstack-swift06:05
*** skudlik has joined #openstack-swift06:05
*** Jeffrey4l has joined #openstack-swift06:05
*** StevenK has joined #openstack-swift06:05
*** afazekas has joined #openstack-swift06:05
*** dr_gogeta86 has joined #openstack-swift06:05
*** dr_gogeta86 has quit IRC06:05
*** dr_gogeta86 has joined #openstack-swift06:05
*** csmart has quit IRC06:09
*** mwheckmann has joined #openstack-swift06:10
*** tdasilva has joined #openstack-swift06:10
*** acoles has joined #openstack-swift06:10
*** barjavel.freenode.net sets mode: +v acoles06:10
*** kei_yama has quit IRC06:34
*** kei_yama has joined #openstack-swift06:43
*** hseipp has joined #openstack-swift06:52
*** hseipp has quit IRC06:52
*** hseipp has joined #openstack-swift06:52
*** hseipp has quit IRC06:53
*** hseipp has joined #openstack-swift06:57
*** hseipp has quit IRC06:57
*** hseipp has joined #openstack-swift06:57
*** hseipp has quit IRC06:57
*** hseipp has joined #openstack-swift06:57
*** rcernin has joined #openstack-swift07:18
*** csmart has joined #openstack-swift07:19
*** pcaruana has joined #openstack-swift07:22
-openstackstatus- NOTICE: Post jobs are not executed currently, do not tag any releases07:24
*** ChanServ changes topic to "Post jobs are not executed currently, do not tag any releases"07:24
*** tesseract has joined #openstack-swift07:24
*** cbartz has joined #openstack-swift07:25
*** cschwede has joined #openstack-swift07:34
*** ChanServ sets mode: +v cschwede07:34
*** oshritf has joined #openstack-swift07:38
openstackgerritChenghui Yu proposed openstack/swift master: Update os-api-ref>=1.4.0  https://review.openstack.org/50509507:41
*** mingyu has joined #openstack-swift07:42
*** SkyRocknRoll has joined #openstack-swift07:45
*** geaaru has joined #openstack-swift07:47
acolesgood morning07:54
mahaticacoles: o/ good morning07:58
acoleshi mahatic07:58
acoleslol mattoliverau +/-800 line patch to change the name from pivot :)07:59
acolesok, let's get that merged before bothering to push other patches.07:59
*** d0ugal has quit IRC08:01
mattoliverauacoles: I hope its right :P08:05
acolesmattoliverau: it is forwards progress, thanks for doing it08:06
mattoliverauacoles: seems I used the term pivot alot. I kept some instances, as a pivot being serching for a place in the data to split sounds right.. (in the doc glossary) but hit that last so maybe I didn't have to remove _all_ of them08:06
mattoliveraualso, not sure that made sense, /me might still be jetlagged :P08:07
acolesmattoliverau: did you mean to use shardrange (no spaces) like here https://review.openstack.org/#/c/505097/2/swift/common/db_replicator.py@37408:08
patchbotpatch 505097 - swift (feature/deep) - Stop using the term Pivot08:08
*** d0ugal has joined #openstack-swift08:09
mattoliverauI think whats suppose to read, the shard ranges stored in the container...08:11
acolesmattoliverau: ok no worries I will fix up as I review08:12
*** mvk has quit IRC08:21
*** mvk has joined #openstack-swift08:50
mattoliverauacoles: I was thinking maybe the main shard functions to test, misplassed objects, audit, meld and explode etc, for ease of testing could simply take a "job". Then when searching for jobs we just generate jobs and pass them to methods.. this would hopefully make it easier to write tests, remove dodginess like the self.node_idx and take us a step toward the concurrently or goal, in a sharer or replicator daemon.08:57
mattoliverau*concurrency goal08:59
acolesmattoliverau: yep, I was working on not hanging node_idx etc off self. That's going to need rebasing on the name change patch :)08:59
mattoliverauI think we talk about just providing a broker, which could work too, but the job could also contain other things, like node is etc09:00
mattoliverauCool09:00
acolesalso was looking at moving the static methods to be broker methods??09:00
acolesso you could instantiate a broker and ask it for it's shard points/ranges09:01
mattoliverauCool, in my concurrent sharder tests I've been using job dicts which work clean things up nice.09:01
mattoliverauYeah, good idea.09:01
acolesyeah that's a nice pattern too09:01
mattoliverauThe broker is shard aware, so makes sense09:01
acolesright, seems like broker already has some shard aware stuff09:01
*** joeljwright has joined #openstack-swift09:09
*** ChanServ sets mode: +v joeljwright09:09
*** mvk has quit IRC09:32
*** kei_yama has quit IRC09:32
*** mvk has joined #openstack-swift09:45
*** tovin07_ has quit IRC10:09
*** SkyRocknRoll_ has joined #openstack-swift10:15
*** skudlik has left #openstack-swift10:23
*** joeljwright has quit IRC10:26
*** joeljwright has joined #openstack-swift10:27
*** ChanServ sets mode: +v joeljwright10:27
*** SkyRocknRoll_ has quit IRC10:29
*** mingyu has quit IRC10:50
*** cshastri has quit IRC11:13
*** SkyRocknRoll has quit IRC11:29
*** ypbg has joined #openstack-swift11:30
*** mingyu has joined #openstack-swift11:34
*** abhitechie has joined #openstack-swift11:45
*** saint_ has joined #openstack-swift11:53
*** abhitechie has quit IRC11:57
*** abhitechie has joined #openstack-swift11:57
*** oshritf_ has joined #openstack-swift11:58
*** oshritf has quit IRC12:01
*** abhitechie has quit IRC12:10
*** abhitechie has joined #openstack-swift12:10
*** lucasxu has joined #openstack-swift12:15
acolesmattoliverau: fixed up and merged the name changes. I hope I correctly interpreted your intentions.12:18
*** MVenesio has joined #openstack-swift12:19
acolestimburke: rebased probe test on ^^ and merged12:19
acolestorgomatic: note there has been a significant churn on feature/deep to rename pivots to shard ranges12:19
acoleshmmm, this has merged https://review.openstack.org/#/c/504127/ but gerritbot is still not alerting for feature/deep12:20
patchbotpatch 504127 - openstack-infra/project-config - Update gerritbot branch alerts in #openstack-swift (MERGED)12:20
*** abhitechie has quit IRC12:26
*** catintheroof has joined #openstack-swift12:49
*** skudlik has joined #openstack-swift12:55
*** mat128 has joined #openstack-swift12:58
*** mabrams has quit IRC13:14
*** chlong has joined #openstack-swift13:19
*** _ix has joined #openstack-swift13:48
_ixHey, good morning folks. I had a quick question about swift and splitting larger objects. We take some fairly large compressed db backups, and as a rule, we compress and split them.13:49
_ixBut, it looks like swift does splitting natively. Is the split step that we do necessary?13:49
*** skudlik has left #openstack-swift13:50
_ixFurther, I don't know how it works exactly, but, if swift does the splitting, does that mean we don't have to download the segments, cat them together, and uncompress; instead, we just do a swift download for the object and it takes care of all of the stitching for us?13:50
tdasilva_ix: the answer is kind of twofold: the swift api (server side) has support for large file objects, but the client is responsible for splitting up the segments and uploading. Read here for more: https://docs.openstack.org/swift/latest/api/large_objects.html13:57
_ixThanks, tdasilva.13:58
tdasilva_ix: the swift *client*, has support for splitting the objects automatically for you, so that's what you might be seeing...13:58
tdasilva_ix: but you don't have to use the python swiftclient per se, you could be sending requests to the API in a different way, so it's really up to your client application how to send those segments to the server13:59
tdasilva_ix: see here for swift client upload options: https://docs.openstack.org/python-swiftclient/latest/cli/index.html#swift-upload14:00
*** SkyRocknRoll has joined #openstack-swift14:22
mwheckmannHas anyone seen a lot of "Too many open files" errors with the object-server after switching to "servers_per_port" config? i.e with the a port per disk in the ring.14:25
mwheckmannThis is happening to us under load testing since moving over to per disk ports.14:26
mwheckmannsystemd sets a 4k file descriptor limit on the object server.14:26
mwheckmannAt first glance, it seems to trigger a little bit quicker on nodes w/ less CPU cores available14:27
mwheckmannRunning 2.13.0 BTW14:27
tdasilvamwheckmann: some of the guys on the PST might have more ops experience running servers_per_port14:34
mwheckmanntdasilva: sorry, what do you mean by PST? The timezone?14:43
tdasilvamwheckmann: sorry, yeah, PST timezone. I'd ask the question again in a couple of hours14:43
mwheckmannwill do. thanks for the tip14:44
*** cbartz has quit IRC15:00
*** cbartz has joined #openstack-swift15:00
*** joeljwright has quit IRC15:00
*** joeljwright has joined #openstack-swift15:05
*** ChanServ sets mode: +v joeljwright15:05
*** gyee has joined #openstack-swift15:05
*** MVenesio has quit IRC15:30
*** chsc has joined #openstack-swift15:38
*** oshritf_ has quit IRC15:42
*** chsc has quit IRC15:45
*** joeljwright has quit IRC15:59
*** joeljwright has joined #openstack-swift16:02
*** ChanServ sets mode: +v joeljwright16:02
*** silor has joined #openstack-swift16:03
*** hamzy has quit IRC16:11
*** itlinux has joined #openstack-swift16:11
*** hseipp has quit IRC16:19
*** geaaru has quit IRC16:22
timburkegood morning16:28
*** mat128 has quit IRC16:32
acolestimburke: o/ ... got some sharding patches merged today16:36
*** joeljwright has quit IRC16:36
timburkeacoles: just taking a look at the dashboard :)16:37
timburkei see there are a few that need rebasing; i think i'll start there16:37
acolestimburke: inevitable merge conflicts with the big rename16:38
*** pcaruana has quit IRC16:38
timburkethanks for getting the probe test in. i keep carrying it around so i can do an integration test when i've got another patch i'm working on16:38
acolestimburke: right! I was about to cherry-pick it for same reason :) so just got it merged16:39
acolesuntil we get unit test coverage up it's the best check we have16:39
timburkeyup. still need to come up with some other edge cases for it... and think about whether the skip limit of 100 is too low16:40
acolesI added notes here for anyone wanting to run the sharding probe test https://etherpad.openstack.org/p/deep-containers16:41
acolesthe sample internal-client.conf does not have account_autocreate16:41
timburkeyeah... i ran into a bit of trouble with that yesterday, too. remind me: when did we introduce internal-client.conf? was that for EC stuff (so we expect accounts/containers to already exist)?16:43
acolestimburke: I asked in -infra about the gerritbot alerts for the new feature branches - sounds like they're working through gerrit upgrade fallout before taking in new config16:43
timburkeah, yeah, that seems fair16:43
*** chsc has joined #openstack-swift16:44
*** chsc has joined #openstack-swift16:44
acolesso the dashboard is the place to go, and trello16:44
*** chsc has quit IRC16:44
acolestimburke: internal-client.conf came in with a container sync change to use it16:45
timburkehuh. so i guess you can't sync to an account unless it already exists... i suppose that's reasonable...16:46
acolesI think it is only used for the get side of sync, was needed to support EC gets16:50
*** mat128 has joined #openstack-swift16:51
*** klrmn has joined #openstack-swift16:51
*** rcernin has quit IRC16:52
*** tesseract has quit IRC16:52
*** mvk has quit IRC17:01
*** cbartz has quit IRC17:06
acolestimburke: what should I be looking to review/merge tomorrow?17:17
acoleslisting middleware on master perhaps?17:17
timburkegood plan. https://review.openstack.org/#/c/503938/ would also be worht a peek once i've got it rebased17:18
patchbotpatch 503938 - swift (feature/deep) - Always use node_idx 0 as scanner17:18
acolestimburke: ok17:19
timburkei'm trying to remember if there was anything else we wanted to land on master to use for sharding...17:20
acolestimburke: might be worth getting this done on master in case we end up making changes that would conflict...but it needs more than review :/ https://review.openstack.org/#/c/448480/17:23
patchbotpatch 448480 - swift - DB replicator cleanup17:23
timburkewill take a look. https://review.openstack.org/#/c/441520/ would probably be good to have, too, but for feature/s3api17:24
patchbotpatch 441520 - swift - Use swift3's check_signature function17:24
timburkeoh! we should maybe think about https://review.openstack.org/#/c/464084/17:25
patchbotpatch 464084 - swift - Apply remote metadata in _handle_sync_response17:25
*** chsc has joined #openstack-swift17:25
*** chsc has joined #openstack-swift17:25
acolesk17:25
timburkeit does a two-way sync on the metadata before we start shipping object rows around, which seems like something we may want to start doing...17:27
mwheckmannHas anyone seen a lot of "Too many open files" errors with the object-server after switching to "servers_per_port" config? i.e with the a port per disk in the ring.17:46
mwheckmannThis is happening to us under load testing since moving over to per disk ports.17:46
mwheckmannI asked this morning (EST) but since there are more folks active in the channel now17:47
*** MVenesio has joined #openstack-swift17:49
mwheckmannlsof output shows the object server processes w/ a bunch of open pipes as well as some AF_ALG sockets. The latter seems odd since those are for the crypto API which we are not using.17:51
mwheckmannthis is on 2.13.0 BTW17:51
*** mvk has joined #openstack-swift17:53
*** ChubYann has joined #openstack-swift17:59
*** saint_ has quit IRC18:08
*** itlinux has quit IRC18:22
*** silor has quit IRC18:40
*** mingyu has quit IRC18:45
*** mingyu has joined #openstack-swift18:46
*** mingyu has quit IRC18:47
*** mingyu has joined #openstack-swift18:48
*** cschwede has quit IRC18:55
*** itlinux has joined #openstack-swift18:59
*** Sukhdev has joined #openstack-swift19:17
openstackgerritMerged openstack/swift master: Remove all post_as_copy related code and configes  https://review.openstack.org/50406919:19
*** lucasxu has quit IRC19:26
*** mat128 has quit IRC19:46
*** rcernin has joined #openstack-swift20:14
*** _ix has quit IRC20:18
-openstackstatus- NOTICE: Zuul and Gerrit are being restarted to address issues discovered with the Gerrit 2.13 upgrade. review.openstack.org will be inaccessible for a few minutes while we make these changes. Currently running jobs will be restarted for you once Zuul and Gerrit are running again.20:27
*** _ix has joined #openstack-swift20:28
*** Sukhdev has quit IRC20:41
*** catintheroof has quit IRC20:50
*** catintheroof has joined #openstack-swift20:50
*** catintheroof has quit IRC20:51
*** itlinux has quit IRC20:56
*** itlinux has joined #openstack-swift20:57
*** itlinux has quit IRC20:58
*** itlinux has joined #openstack-swift20:59
*** _ix has quit IRC21:21
*** itlinux has quit IRC21:21
*** itlinux has joined #openstack-swift21:25
*** klrmn1 has joined #openstack-swift21:29
*** klrmn has quit IRC21:31
*** jistr has quit IRC21:41
*** Sukhdev has joined #openstack-swift21:50
*** jistr has joined #openstack-swift22:01
*** Sukhdev has quit IRC22:12
*** MVenesio has quit IRC22:12
*** chlong has quit IRC22:21
-openstackstatus- NOTICE: Gerrit is being restarted to address some final issues, review.openstack.org will be inaccessible for a few minutes while we restart22:34
*** vint_bra has quit IRC22:45
*** catintheroof has joined #openstack-swift22:45
*** openstackgerrit has quit IRC22:47
*** itlinux has quit IRC22:49
*** rcernin has quit IRC23:11
mattoliverauMorning23:18
timburkegood morning mattoliverau23:18
timburkelots of sharding stuff in the gate23:19
timburkepoor -infra's been overworked with the zuul/gerrit issues :-(23:19
*** openstackgerrit has joined #openstack-swift23:19
openstackgerritTim Burke proposed openstack/swift feature/deep: Reduce the number of queries in get_shard_range  https://review.openstack.org/50543923:19
timburkeyay! we get our branch notifications now!23:19
*** hoonetorg has quit IRC23:23
*** kei_yama has joined #openstack-swift23:26
mattoliveraumwheckmann: in 2.15.0+ we new change the default eventlet threadpool size, because of the amount of threads servers_per_port where using. This might have something to do with your problem. see: https://github.com/openstack/swift/blob/2.15.0/swift/obj/server.py#L202-L22823:33
*** chsc has quit IRC23:33
mattoliverau*we changed the default..23:33
mattoliverautimburke: nice!23:34
timburkelooks like p 505321 is going to hit a conflict once p 505329 lands -- but with all the patches in flight, i'm inclined to let acoles handle the rebase after things have settled a bit23:34
patchbothttps://review.openstack.org/#/c/505321/ - swift (feature/deep) - remove node-id, node_idx and part instance vars in...23:34
patchbothttps://review.openstack.org/#/c/505329/ - swift (feature/deep) - Remove some instance vars from _one_shard_cycle23:34
mattoliverauyeah, I see a bunch of gerrit messages in backlog. as always I suspect some teathing issues with the new verison.23:35
mattoliverautimburke: wow, you both have been busy!23:35
*** hoonetorg has joined #openstack-swift23:36
clarkbemail sucks in the new version23:39
clarkbwe are trying to debug it23:39
mattoliveraubugger.23:39
timburkeclarkb: no worries man, thanks for all the hard work!23:39
*** ChanServ changes topic to "Queens PTG planning https://etherpad.openstack.org/p/swift-ptg-queens | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Review Dashboard: http://not.mn/reviews.html"23:41
-openstackstatus- NOTICE: Gerrit is once again part of normal puppet config management. Problems with Gerrit gitweb links and Zuul post jobs have been addressed. We currently cannot create new gerrit projects (fixes in progress) and email sending is slow (being debugged).23:41
mattoliverauI wish the 'f' button worked in the new version liked it worked a few versions ago (open the list of changed files).23:41
mattoliverautimburke: FYI I hope to get to some shard work (reviewing and dev) later today, but need to work on some suse tasks first :(23:42
timburkeno worries. acoles and i can bounce between each other for a while -- we'll be sure to raise up anything that seems like it really needs your attention23:44
mattoliverautimburke: thanks man23:44
*** Sukhdev has joined #openstack-swift23:46
openstackgerritKazuhiro MIYAHARA proposed openstack/swift master: Symlink implementation.  https://review.openstack.org/23216223:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!