| paladox | notmyname does memcached need to be on the swift servers if the proxy is not on them? | 00:09 |
|---|---|---|
| *** linkmark has quit IRC | 00:23 | |
| mattoliverau | oh morning again. I forgot to say anything here. Was too busy trying to think of something to submit for the Berlin summit. | 00:31 |
| mattoliverau | notmyname: thanks for the bug reports | 00:31 |
| openstackgerrit | Joshua Harlow proposed openstack/python-swiftclient master: Log exceptions received during retrying https://review.openstack.org/581921 | 00:51 |
| *** labster has left #openstack-swift | 01:01 | |
| *** itlinux has joined #openstack-swift | 01:36 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Add ability to generate a temporary URL with an IP range restriction https://review.openstack.org/581374 | 01:46 |
| openstackgerrit | Merged openstack/python-swiftclient master: Treat 404 as success when deleting segments https://review.openstack.org/538349 | 01:46 |
| clayg | https://www.irccloud.com/pastebin/dM2SwkHz/ | 01:50 |
| clayg | well that's kinda stupid... | 01:51 |
| clayg | ContextualVersionConflict: (ipaddress 1.0.16 (/usr/lib/python2.7/site-packages), Requirement.parse('ipaddress>=1.0.17'), set(['swift'])) | 01:51 |
| clayg | ^ cc tdasilva seeing some probetest cent7 failure about that there | 01:51 |
| clayg | just FYI | 01:52 |
| *** itlinux has quit IRC | 02:02 | |
| kota_ | clayg: I'm currently with you at p 581913 | 02:19 |
| patchbot | https://review.openstack.org/#/c/581913/ - swift - Include s3api schemas in sdists | 02:19 |
| clayg | heheh | 02:20 |
| kota_ | I can find the rng schema in my ubuntu distro. | 02:20 |
| * clayg shrugs | 02:20 | |
| clayg | packaging is always a cluster | 02:20 |
| kota_ | so I also don't know why it doesn't work with CentOS packaging. | 02:20 |
| *** yuxin_ has quit IRC | 02:20 | |
| clayg | i appreciate you confirming i'm not entirely crazy tho | 02:21 |
| kota_ | I should go your link [2] | 02:21 |
| *** yuxin_ has joined #openstack-swift | 02:21 | |
| kota_ | it's sort of "And creating “smart” built distributions, such as an RPM package or an executable installer for Windows, is far more convenient for users even if your distribution doesn’t include any extensions." things? | 02:23 |
| kota_ | interesting | 02:24 |
| openstackgerrit | Nguyen Hai proposed openstack/swift master: add lower-constraints job https://review.openstack.org/556255 | 02:24 |
| *** psachin has joined #openstack-swift | 02:45 | |
| openstackgerrit | Clay Gerrard proposed openstack/python-swiftclient master: Add more validation for ip_range args https://review.openstack.org/581906 | 02:57 |
| *** spsurya_ has joined #openstack-swift | 03:32 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Add unittest for slo_etag https://review.openstack.org/581943 | 03:51 |
| *** links has joined #openstack-swift | 05:03 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:17 | |
| *** SkyRocknRoll has quit IRC | 05:25 | |
| *** dr_gogeta86 has quit IRC | 05:26 | |
| openstackgerrit | Merged openstack/swift master: Include s3api schemas in sdists https://review.openstack.org/581913 | 05:30 |
| openstackgerrit | Merged openstack/swift master: Include SLO ETag in container updates https://review.openstack.org/337960 | 05:30 |
| *** d0ugal_ has joined #openstack-swift | 05:55 | |
| *** d0ugal has quit IRC | 05:56 | |
| kota_ | thanks clayg to look at the patch 337960. I was worried about i may be a blocker to get it merged. | 06:11 |
| patchbot | https://review.openstack.org/#/c/337960/ - swift - Include SLO ETag in container updates (MERGED) | 06:11 |
| clayg | kota_: i'd still love to get your opinon on the follow up - but I'm working (slowly) on getting my s3api setup | 06:12 |
| tdasilva | clayg: trying to catch up, where do you see probetest failures? | 06:15 |
| clayg | https://review.openstack.org/#/c/581913/ | 06:15 |
| patchbot | patch 581913 - swift - Include s3api schemas in sdists (MERGED) | 06:15 |
| *** armaan has joined #openstack-swift | 06:18 | |
| *** cshastri has joined #openstack-swift | 06:18 | |
| *** bkopilov has quit IRC | 06:30 | |
| *** hseipp has joined #openstack-swift | 06:41 | |
| *** hseipp has quit IRC | 06:45 | |
| *** hseipp has joined #openstack-swift | 06:45 | |
| *** hseipp has quit IRC | 06:57 | |
| *** bkopilov has joined #openstack-swift | 06:58 | |
| kota_ | clayg: alright, I'll do my best, perhaps, I could have time to look at... | 07:06 |
| clayg | s'ok if you don't I'll get to it eventually ;) | 07:07 |
| *** gkadam has joined #openstack-swift | 07:14 | |
| *** rcernin has quit IRC | 07:20 | |
| kota_ | clayg: much appreciated that you make it progressed XD | 07:29 |
| clayg | kota_: timburke: did ya'l see this one p 580333 | 07:37 |
| patchbot | https://review.openstack.org/#/c/580333/ - swift - HEAD to check existence before container PUT | 07:37 |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Fix misleading error msg if swift.conf unreadable https://review.openstack.org/581280 | 07:45 |
| kota_ | clayg: let me check on what p 580333 would solve, thanks for heads up | 07:56 |
| patchbot | https://review.openstack.org/#/c/580333/ - swift - HEAD to check existence before container PUT | 07:56 |
| kota_ | head up | 07:56 |
| kota_ | i have only one head. | 07:56 |
| kota_ | :/ | 07:56 |
| *** mikecmpbll has joined #openstack-swift | 08:01 | |
| *** d0ugal_ has quit IRC | 08:03 | |
| *** d0ugal has joined #openstack-swift | 08:03 | |
| *** d0ugal has quit IRC | 08:03 | |
| *** d0ugal has joined #openstack-swift | 08:03 | |
| kota_ | hmmm... i'm wondering how we should estimate container db heavy load (i.e. got LockTimeout for sqlite db) for the prod clusters. | 08:10 |
| kota_ | it sounds obviously unhealthy and then, 503 ServiceUnavailable is not so much bad status for us? | 08:10 |
| *** itlinux has joined #openstack-swift | 08:16 | |
| *** ccamacho has joined #openstack-swift | 08:17 | |
| kota_ | clayg: is there any idea how large overhead between HEAD container and PUT container? | 08:20 |
| *** hseipp has joined #openstack-swift | 08:21 | |
| kota_ | I'm now looking at the container-server code, then PUT container obviously update the timestamp and commit the change for the db but it looks like HEAD container also calling commit_puts_stale_ok() to get the container info that will make a commit to merge the pending file. | 08:21 |
| kota_ | if i'm not crazy on my eyes, changing PUT container to HEAD container is not so much effective to mitigate the load. | 08:22 |
| *** links has quit IRC | 08:24 | |
| *** mikecmpb_ has joined #openstack-swift | 08:24 | |
| *** mikecmpbll has quit IRC | 08:25 | |
| *** links has joined #openstack-swift | 08:26 | |
| *** mvk_ has quit IRC | 08:30 | |
| *** tesseract has joined #openstack-swift | 08:37 | |
| *** mvk_ has joined #openstack-swift | 08:54 | |
| acoles | good morning | 08:55 |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Fix misleading error msg if swift.conf unreadable https://review.openstack.org/581280 | 09:01 |
| *** cshastri_ has joined #openstack-swift | 09:03 | |
| *** cshastri has quit IRC | 09:06 | |
| *** links has quit IRC | 09:10 | |
| *** links has joined #openstack-swift | 09:12 | |
| *** mvk_ has quit IRC | 09:16 | |
| *** hoonetorg has quit IRC | 09:27 | |
| *** mvk_ has joined #openstack-swift | 09:28 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Check other params preserved when slo_etag is extracted https://review.openstack.org/582125 | 09:31 |
| *** hoonetorg has joined #openstack-swift | 09:39 | |
| *** mikecmpb_ has quit IRC | 09:50 | |
| *** mikecmpbll has joined #openstack-swift | 09:52 | |
| *** kei_yama has quit IRC | 11:27 | |
| *** armaan has quit IRC | 11:48 | |
| *** armaan has joined #openstack-swift | 11:49 | |
| *** itlinux has quit IRC | 12:01 | |
| *** armaan has quit IRC | 12:41 | |
| *** armaan has joined #openstack-swift | 12:41 | |
| *** zaitcev has joined #openstack-swift | 13:37 | |
| *** ChanServ sets mode: +v zaitcev | 13:37 | |
| *** psachin has quit IRC | 13:46 | |
| *** links has quit IRC | 13:50 | |
| *** psachin has joined #openstack-swift | 13:51 | |
| *** armaan_ has joined #openstack-swift | 14:01 | |
| *** armaan has quit IRC | 14:02 | |
| *** linkmark has joined #openstack-swift | 14:03 | |
| *** psachin has quit IRC | 14:05 | |
| *** mikecmpbll has quit IRC | 14:08 | |
| *** mikecmpbll has joined #openstack-swift | 14:09 | |
| *** ccamacho has quit IRC | 14:16 | |
| *** ccamacho has joined #openstack-swift | 14:21 | |
| clayg | acoles: good morning | 14:57 |
| acoles | clayg: o/ | 14:57 |
| acoles | clayg: I +A'd your patch but it failed test in gate https://review.openstack.org/581943 :( | 14:58 |
| patchbot | patch 581943 - swift - Add unittest for slo_etag | 14:58 |
| clayg | thanks for trying, i guess that test is flakey? | 14:58 |
| acoles | I think the json needs to be loaded to avoid key order variations in the serialized version | 14:59 |
| acoles | was going to fix it but got engrossed in PUT+POST | 14:59 |
| clayg | k, i'll square it | 14:59 |
| acoles | IIRC you're comparing two serialized versions | 14:59 |
| *** armaan_ has quit IRC | 15:01 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Add unittest for slo_etag https://review.openstack.org/581943 | 15:02 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Check other params preserved when slo_etag is extracted https://review.openstack.org/582125 | 15:02 |
| acoles | whoosh | 15:07 |
| *** cshastri_ has quit IRC | 15:23 | |
| *** ccamacho has quit IRC | 15:26 | |
| *** gyee has joined #openstack-swift | 15:30 | |
| *** gyee has quit IRC | 15:34 | |
| *** gkadam has quit IRC | 15:47 | |
| notmyname | good morning | 15:53 |
| *** bharath1234 has joined #openstack-swift | 16:03 | |
| bharath1234 | torgomatic, i am studying the unique as possible placement algorithm. I m reading the code in the get_more_nodes function which i believe is used to get the handoff nodes. I didnt get why you hashed the partition number and shifted it by the partition shift. The number of parts in my cluster is 1024 and the when we hash the partition number and shift , i get 192. Could you elaborate as on to why that was done? Thank you | 16:03 |
| *** bharath1234 has quit IRC | 16:04 | |
| openstackgerrit | John Dickinson proposed openstack/swift master: added docker test target tools https://review.openstack.org/577467 | 16:06 |
| *** gyee has joined #openstack-swift | 16:08 | |
| *** hseipp has quit IRC | 16:20 | |
| clayg | bharath1234 you man specifically: part = struct.unpack_from('>I', key)[0] >> self._part_shift ? | 16:21 |
| *** armaan has joined #openstack-swift | 16:22 | |
| clayg | I feel the easiest way to think of that step is just the modulo? basically you're just placing the key into the bucketspace - but maybe with a little fancy math. | 16:24 |
| *** itlinux has joined #openstack-swift | 16:25 | |
| clayg | oh... no your question is more specific - in get_more_nodes we're not hashing a name - we alreayd have a part - so why do the rehash? | 16:26 |
| *** spsurya_ has quit IRC | 16:26 | |
| clayg | part_hash = md5(str(part).encode('ascii')).digest() | 16:26 |
| clayg | ^ yeah idk, that looks kind of weird?! | 16:26 |
| *** mikecmpbll has quit IRC | 16:28 | |
| clayg | I think we started hashing the part here: https://review.openstack.org/#/c/23404/ | 16:32 |
| patchbot | patch 23404 - swift - Updated get_more_nodes algorithm (MERGED) | 16:32 |
| *** itlinux has quit IRC | 16:32 | |
| clayg | it'd be awesome to ask gholt why that might of been - but he would claim he doesn't remember | 16:33 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: PUT+POST: Detect older object server by not sending content-type https://review.openstack.org/582298 | 16:39 |
| clayg | timburke: do we have an example ~/.s3cfg for tempauth test:tester? | 16:39 |
| clayg | like on a saio? | 16:39 |
| clayg | https://docs.openstack.org/swift/latest/middleware.html#module-swift.common.middleware.s3api.s3api I guess | 16:40 |
| clayg | acoles: you're on fire! | 16:51 |
| *** armaan has quit IRC | 17:03 | |
| *** tesseract has quit IRC | 17:16 | |
| *** mikecmpbll has joined #openstack-swift | 17:21 | |
| notmyname | tdasilva: timburke: kota_: looking at pyeclib, I don't think there's anything there that needs a release. however, a libec release (just a x.x.1) might be good. that will get a patch with better crc32 | 17:31 |
| zaitcev | https://www.mail-archive.com/python-committers@python.org/msg05628.html | 17:53 |
| zaitcev | clayg: Without seeing the code, I think you are right to be suspicious. All invocations of str() carry a danger of producing "b'foo'" silently. I would say, extreme danger even. We really should aim to exterminate all str() and not think of it as a handy way to coerce to a native string. | 17:55 |
| *** mikecmpbll has quit IRC | 18:10 | |
| *** mikecmpbll has joined #openstack-swift | 18:10 | |
| *** mikecmpbll has quit IRC | 18:11 | |
| *** bkopilov has quit IRC | 18:18 | |
| *** mikecmpbll has joined #openstack-swift | 18:22 | |
| *** armaan has joined #openstack-swift | 18:24 | |
| *** armaan has quit IRC | 18:28 | |
| *** armaan has joined #openstack-swift | 18:29 | |
| *** armaan has quit IRC | 18:33 | |
| openstackgerrit | Merged openstack/swift master: Add unittest for slo_etag https://review.openstack.org/581943 | 18:48 |
| *** jistr has quit IRC | 18:50 | |
| *** mikecmpbll has quit IRC | 18:51 | |
| *** jistr has joined #openstack-swift | 19:25 | |
| *** mvk_ has quit IRC | 19:27 | |
| *** mvk_ has joined #openstack-swift | 19:56 | |
| paladox | Hi, does anyone know how i can balace the storage accross two nodes? | 21:25 |
| paladox | so that half is on one and the other on the other? | 21:25 |
| zaitcev | make them the same size | 21:26 |
| paladox | the file storage? | 21:26 |
| paladox | they are both 150gb :) | 21:26 |
| paladox | one is near to using 150gb | 21:26 |
| paladox | the other is using 33gb. | 21:27 |
| zaitcev | and the sums of device weights in the rings are the same for both nodes? | 21:27 |
| paladox | yep | 21:29 |
| paladox | 145 | 21:29 |
| zaitcev | Interesting. | 21:30 |
| zaitcev | I'd start by making sure that the cluster is healthy otherwise and that the replicators and expirers run normally (by looking at logs). | 21:30 |
| zaitcev | Then, I'd try to identify where all the data is | 21:31 |
| paladox | ah ok. | 21:31 |
| zaitcev | e.g., make sure nothing crazy is going on with respect to quarantine. | 21:31 |
| paladox | hmm ok. I did set concurrent to 0 for replication but that was yesturday and swift has been running for over a week. | 21:32 |
| paladox | it replicated 33gb i think but never deleted any copied data from the other node. | 21:33 |
| notmyname | paladox: what have you tried so far? | 21:34 |
| openstackgerrit | Merged openstack/swift master: Check other params preserved when slo_etag is extracted https://review.openstack.org/582125 | 21:35 |
| paladox | notmyname i've been forced to try and reduce load. So i had to set concurrent to 0. I have the replication systemd service up. But i have tryed leaving the swift replication to do it's thing. | 21:35 |
| paladox | but rsync kept failing i guess when it used alot of ram. | 21:35 |
| paladox | my config: https://github.com/miraheze/puppet/tree/master/modules/swift/templates | 21:35 |
| *** wer has quit IRC | 21:42 | |
| notmyname | paladox: have you run `swift-recon --all`? any issues reported there? | 21:42 |
| * paladox runs that | 21:42 | |
| paladox | notmyname it shows no errors | 21:43 |
| paladox | paste: https://phabricator.wikimedia.org/P7364 | 21:43 |
| notmyname | your disk usage isn't reported there. shows an error. also shows your replication hasn't run in 20 hours | 21:45 |
| paladox | notmyname hmm. The server is on a openvz so i guess that's why the disk usage won't be shown. and for replication i guess it was affected by changing concurrent to 0? | 21:46 |
| paladox | but even then it was replicating but not deleting | 21:46 |
| paladox | stuff it replicated. | 21:46 |
| notmyname | paladox: you could run `swift-object-replicator` directly from the command line against just one partition and see what happens. maybe that would show you any issues | 21:56 |
| paladox | ah | 21:56 |
| paladox | will try that! | 21:57 |
| paladox | thanks! | 21:57 |
| *** rcernin has joined #openstack-swift | 21:58 | |
| paladox | notmyname hi, i just set falloc to 11% of available storage. which swift1 would be affected as it has 4.1gb left. https://static.miraheze.org/traunstoanerwiki/thumb/a/ab/Bergham.jpg/180px-Bergham.jpg is now returning 503. | 22:11 |
| paladox | but that exists on swift1. | 22:11 |
| paladox | im guessing it's because account / container folders are out dated on swift2? | 22:11 |
| notmyname | why did you set fallocate reserve? | 22:13 |
| paladox | notmyname we were low on storage on swift1 and wanted all PUTs to fall over to swift2. | 22:13 |
| notmyname | remember that swift is not a "fill and spill" storage system | 22:15 |
| notmyname | did you run the replicator against just one partition? | 22:17 |
| paladox | notmyname going to run that now | 22:18 |
| paladox | notmyname could it be because i have swift account and swift container rings defined for both swift* | 22:20 |
| notmyname | when you get a 503 on a GET, what do the logs say? grep logs on both servers for the transaction id | 22:21 |
| notmyname | eg txcc762fcf0e8048fbba48b-005b47d3d7 | 22:21 |
| paladox | ok | 22:22 |
| paladox | notmyname from what i can see swift says it's a 404 but from the console log it is saying 503 | 22:28 |
| paladox | "GET /simfs/0/AUTH_admin/traunstoanerwiki-mw/b/b5/Thalham_02.jpg" 404 - "GET http://185.52.3.121:8080/v1/AUTH_admin/traunstoanerwiki-mw/b/b5/Thalham_02.jpg" "tx96406ad6a280437fa2ab0-005b47cf0e" "proxy-server 5004" 0.0004 "-" 25645 0 | 22:28 |
| paladox | swift-object-replicator object-server.conf -p 1 -v | 22:31 |
| zaitcev | Replicator is what actually deletes anything that's deleted. If you don't run it, you'll overflow your storage as all deleted objects accumulate. So it's imperative that you keep it operating. | 22:32 |
| zaitcev | This does not explain the asymmetry though. Unless it was running normally on the other node, but not on this one. | 22:32 |
| paladox | zaitcev is there a way to get it so that replication does not run frequently. as it causes very high load | 22:33 |
| paladox | when rsync runs | 22:33 |
| zaitcev | You can turn the reclaim time down and see if that help. | 22:33 |
| notmyname | zaitcev: for background, paladox has a 1-replica, 1 drive cluster. and he added a second drive on another server. replication has moved some data, but hasn't deleted anything from the first | 22:33 |
| zaitcev | Good lord | 22:34 |
| notmyname | yes :-) | 22:34 |
| paladox | swift-object-replicator object-server.conf -p 1 -v only shows "swift-object-replicator: Starting object replication pass." | 22:34 |
| paladox | notmyname zaitcev yeh, i wish i could have the higher specs (ssds more cores) but budget carn't afford that :) | 22:36 |
| paladox | we needed something so we could add more storage. | 22:36 |
| paladox | we were prevously on nfs | 22:36 |
| paladox | zaitcev notmyname i wonder if having container and account on mutiple servers is the problem? Dosen't that container where the objects are (on which server) | 22:37 |
| paladox | because the images are working itermittent | 22:37 |
| paladox | one minute they work and the next they are saying they do not exist. | 22:38 |
| notmyname | that would suggest one server is misconfigured. so you could try to hit each one separately. or look at your nginx config to see how it's balancing requests | 22:38 |
| paladox | notmyname i have nginx configured to send requests to the swift proxy which is on another server. | 22:39 |
| notmyname | IMO your tests should be directly to a proxy server to test it. when you see that working, then you move up a level. that will help you isolate where the issue may exist | 22:41 |
| paladox | notmyname it works on swift1 | 22:42 |
| paladox | swift2 dosen't | 22:42 |
| paladox | i see this: | 22:42 |
| notmyname | then swift2 is likely where your issue is | 22:42 |
| paladox | yeh | 22:42 |
| paladox | though shoulden't it try swift1, if that fails then swift2? | 22:43 |
| notmyname | no. stop. fix swift2. whatever is going there is likely why you're not getting the balance you need | 22:43 |
| paladox | notmyname hmm. | 22:44 |
| paladox | ok | 22:44 |
| paladox | but the objects should not be on swift2 though if they are on swift1 | 22:47 |
| mattoliverau | morning | 22:57 |
| zaitcev | Could someone take a look at https://review.openstack.org/579227 | 23:01 |
| patchbot | patch 579227 - swift - PUT+POST: break out putter specific test classes | 23:01 |
| zaitcev | I agreed with acoles about it, and made patch 427911 depend on it, but now it's _exceedingly_ inconvenient. | 23:02 |
| patchbot | https://review.openstack.org/#/c/427911/ - swift - Replace MIME with PUT+POST for EC and Encryption | 23:02 |
| zaitcev | I don't know how you guys deal with stacked patches | 23:03 |
| zaitcev | I mean sure... git rebase is slick. BUT what if I want to change anything? The only decent way I found is to try and commit something separately, and then git rebase -i, ask for sqaush. | 23:04 |
| zaitcev | Doable, but ewww | 23:04 |
| paladox | maybe im having the same issue as https://ask.openstack.org/en/question/111731/swift-return-404-when-get-some-objects-after-adding-new-hdds/ | 23:05 |
| *** kei_yama has joined #openstack-swift | 23:13 | |
| *** SPF|Cloud has joined #openstack-swift | 23:20 | |
| *** drewn3ss has quit IRC | 23:47 | |
| *** mikecmpbll has joined #openstack-swift | 23:50 | |
| *** mikecmpb_ has joined #openstack-swift | 23:56 | |
| *** mikecmpb_ has quit IRC | 23:58 | |
| *** mikecmpbll has quit IRC | 23:58 | |
| *** mikecmpbll has joined #openstack-swift | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
