| *** david-lyle is now known as david-lyle_afk | 00:31 | |
| *** tellesnobrega_ has quit IRC | 00:31 | |
| *** dmorita has joined #openstack-swift | 00:36 | |
| *** annegent_ has joined #openstack-swift | 00:40 | |
| *** annegent_ has quit IRC | 00:45 | |
| *** addnull has joined #openstack-swift | 00:45 | |
| *** kyles_ne has quit IRC | 00:58 | |
| *** addnull has quit IRC | 01:00 | |
| *** david-lyle_afk has quit IRC | 01:08 | |
| *** Viswanath has joined #openstack-swift | 01:09 | |
| *** Viswanath has quit IRC | 01:12 | |
| *** tellesnobrega_ has joined #openstack-swift | 01:14 | |
| *** addnull has joined #openstack-swift | 01:16 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Make error limits survive a ring reload https://review.openstack.org/134419 | 01:17 |
|---|---|---|
| *** tellesnobrega_ has quit IRC | 01:19 | |
| *** shri has quit IRC | 01:26 | |
| *** tellesnobrega_ has joined #openstack-swift | 01:28 | |
| *** annegent_ has joined #openstack-swift | 01:41 | |
| *** gyee has quit IRC | 01:43 | |
| *** nosnos has joined #openstack-swift | 01:43 | |
| *** annegent_ has quit IRC | 01:47 | |
| *** addnull has quit IRC | 01:51 | |
| *** annegent_ has joined #openstack-swift | 02:01 | |
| *** addnull has joined #openstack-swift | 02:01 | |
| *** addnull has quit IRC | 02:08 | |
| *** annegentle has quit IRC | 02:10 | |
| *** aswadr has joined #openstack-swift | 02:13 | |
| *** annegent_ has quit IRC | 02:13 | |
| *** tellesnobrega_ has quit IRC | 02:13 | |
| *** tellesnobrega_ has joined #openstack-swift | 02:16 | |
| *** lcurtis has joined #openstack-swift | 02:19 | |
| *** tsg has quit IRC | 02:23 | |
| *** tsg has joined #openstack-swift | 02:24 | |
| *** tellesnobrega_ has quit IRC | 02:28 | |
| *** tsg has quit IRC | 02:34 | |
| *** lcurtis has quit IRC | 02:34 | |
| *** haomaiwa_ has quit IRC | 02:45 | |
| *** lcurtis has joined #openstack-swift | 02:47 | |
| *** addnull has joined #openstack-swift | 02:50 | |
| *** haomaiwang has joined #openstack-swift | 03:01 | |
| *** annegentle has joined #openstack-swift | 03:11 | |
| *** annegentle has quit IRC | 03:16 | |
| *** lcurtis has quit IRC | 03:18 | |
| *** nosnos has quit IRC | 03:30 | |
| *** nosnos has joined #openstack-swift | 03:30 | |
| *** tellesnobrega_ has joined #openstack-swift | 03:34 | |
| *** nosnos has quit IRC | 03:35 | |
| *** addnull has quit IRC | 03:41 | |
| *** annegentle has joined #openstack-swift | 03:41 | |
| *** jd__ has quit IRC | 03:44 | |
| *** jd__ has joined #openstack-swift | 03:44 | |
| *** cebruns has quit IRC | 03:44 | |
| *** cebruns has joined #openstack-swift | 03:45 | |
| *** Anticimex has quit IRC | 03:45 | |
| *** d0ugal has quit IRC | 03:45 | |
| *** annegentle has quit IRC | 03:46 | |
| *** anteaya has quit IRC | 03:46 | |
| *** Anticimex has joined #openstack-swift | 03:47 | |
| *** d0ugal has joined #openstack-swift | 03:48 | |
| *** sungju has joined #openstack-swift | 03:48 | |
| *** d0ugal is now known as Guest32556 | 03:48 | |
| *** anteaya has joined #openstack-swift | 03:50 | |
| *** tellesnobrega_ has quit IRC | 04:02 | |
| *** Tao has joined #openstack-swift | 04:03 | |
| *** ppai has joined #openstack-swift | 04:05 | |
| *** sgowda has joined #openstack-swift | 04:18 | |
| *** nosnos has joined #openstack-swift | 04:27 | |
| *** Tao has quit IRC | 04:40 | |
| *** annegentle has joined #openstack-swift | 04:41 | |
| *** annegentle has quit IRC | 04:46 | |
| *** annegentle has joined #openstack-swift | 05:41 | |
| *** annegentle has quit IRC | 05:46 | |
| *** addnull has joined #openstack-swift | 05:47 | |
| *** exploreshaifali has joined #openstack-swift | 06:02 | |
| *** zaitcev has quit IRC | 06:03 | |
| *** nshaikh has joined #openstack-swift | 06:13 | |
| *** addnull has quit IRC | 06:18 | |
| *** hondak has quit IRC | 06:27 | |
| *** addnull has joined #openstack-swift | 06:28 | |
| *** annegentle has joined #openstack-swift | 06:41 | |
| *** addnull has quit IRC | 06:44 | |
| *** annegentle has quit IRC | 06:48 | |
| *** addnull has joined #openstack-swift | 06:54 | |
| *** sungju has quit IRC | 07:01 | |
| *** k4n0 has joined #openstack-swift | 07:08 | |
| *** addnull has quit IRC | 07:17 | |
| *** exploreshaifali has quit IRC | 07:22 | |
| *** aswadr has quit IRC | 07:30 | |
| *** addnull has joined #openstack-swift | 07:49 | |
| *** nosnos has quit IRC | 08:00 | |
| *** exploreshaifali has joined #openstack-swift | 08:00 | |
| *** nosnos has joined #openstack-swift | 08:03 | |
| *** jordanP has joined #openstack-swift | 08:30 | |
| *** geaaru has joined #openstack-swift | 08:32 | |
| *** addnull has quit IRC | 08:44 | |
| *** addnull has joined #openstack-swift | 08:45 | |
| *** acoles_away is now known as acoles | 09:03 | |
| *** leopoldj has joined #openstack-swift | 09:08 | |
| *** jistr has joined #openstack-swift | 09:19 | |
| *** exploreshaifali has quit IRC | 09:20 | |
| *** addnull has quit IRC | 09:24 | |
| *** aswadr has joined #openstack-swift | 09:37 | |
| *** joeljwright has joined #openstack-swift | 09:45 | |
| *** nellysmitt has joined #openstack-swift | 09:54 | |
| *** aix has joined #openstack-swift | 09:56 | |
| *** nellysmitt has quit IRC | 09:59 | |
| *** ppai has quit IRC | 10:03 | |
| *** addnull has joined #openstack-swift | 10:05 | |
| *** Guest32556 is now known as d0ugal | 10:14 | |
| *** addnull has quit IRC | 10:14 | |
| *** d0ugal is now known as Guest92455 | 10:14 | |
| zigo | clayg: I have built and upload packages for both liberasurecode & pyeclib, it's now in the FTP masters NEW queue, so it will be available "soon". | 10:15 |
| *** Guest92455 is now known as d0ugal | 10:16 | |
| *** d0ugal has quit IRC | 10:16 | |
| *** d0ugal has joined #openstack-swift | 10:16 | |
| *** ppai has joined #openstack-swift | 10:17 | |
| openstackgerrit | Alistair Coles proposed openstack/python-swiftclient: Fix misplaced check for None in SwiftUploadObject. https://review.openstack.org/133107 | 10:20 |
| *** foexle has joined #openstack-swift | 10:32 | |
| *** addnull has joined #openstack-swift | 10:38 | |
| *** nosnos has quit IRC | 10:43 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Make in process functional tests use sample proxy-server.conf https://review.openstack.org/127607 | 10:46 |
| *** addnull has quit IRC | 10:49 | |
| *** nosnos has joined #openstack-swift | 10:51 | |
| *** addnull has joined #openstack-swift | 10:56 | |
| *** ppai has quit IRC | 10:58 | |
| *** addnull has quit IRC | 11:00 | |
| *** haomaiwang has quit IRC | 11:03 | |
| *** exploreshaifali has joined #openstack-swift | 11:07 | |
| *** addnull has joined #openstack-swift | 11:08 | |
| *** ppai has joined #openstack-swift | 11:12 | |
| *** nellysmitt has joined #openstack-swift | 11:26 | |
| *** sgowda has quit IRC | 11:27 | |
| *** nellysmitt has quit IRC | 11:31 | |
| *** ppai has quit IRC | 11:44 | |
| openstackgerrit | Daisuke Morita proposed openstack/swift: Show the sum of every policy's amount in /recon/async https://review.openstack.org/125179 | 11:47 |
| *** mahatic has joined #openstack-swift | 11:47 | |
| *** addnull has quit IRC | 11:49 | |
| *** ppai has joined #openstack-swift | 11:58 | |
| *** ppai has quit IRC | 12:14 | |
| *** exploreshaifali has quit IRC | 12:24 | |
| *** mkollaro has joined #openstack-swift | 12:25 | |
| *** dmorita has quit IRC | 12:30 | |
| *** nosnos has quit IRC | 12:33 | |
| *** tellesnobrega_ has joined #openstack-swift | 12:39 | |
| *** cbaesema has quit IRC | 12:45 | |
| *** joeljwright1 has joined #openstack-swift | 12:48 | |
| *** joeljwright has quit IRC | 12:51 | |
| *** exploreshaifali has joined #openstack-swift | 12:54 | |
| *** dmsimard_away is now known as dmsimard | 12:58 | |
| *** nshaikh has quit IRC | 13:09 | |
| *** annegentle has joined #openstack-swift | 13:14 | |
| *** annegentle has quit IRC | 13:20 | |
| *** tellesnobrega_ has quit IRC | 13:28 | |
| *** dmsimard is now known as dmsimard_away | 13:30 | |
| *** aswadr has quit IRC | 13:40 | |
| *** jordanP has quit IRC | 13:47 | |
| *** jordanP has joined #openstack-swift | 13:48 | |
| *** lcurtis has joined #openstack-swift | 13:50 | |
| *** haomaiwang has joined #openstack-swift | 14:06 | |
| *** lpabon has joined #openstack-swift | 14:18 | |
| *** mkollaro has quit IRC | 14:19 | |
| *** wobbly1 has joined #openstack-swift | 14:20 | |
| *** dencaval has quit IRC | 14:21 | |
| *** wobbly1 has quit IRC | 14:24 | |
| *** tdasilva has joined #openstack-swift | 14:29 | |
| *** lcurtis has quit IRC | 14:29 | |
| *** cbaesema has joined #openstack-swift | 14:49 | |
| *** joe__ has quit IRC | 14:52 | |
| *** joe__ has joined #openstack-swift | 14:56 | |
| *** dmsimard_away is now known as dmsimard | 14:58 | |
| tdasilva | acoles, cschwede: thanks for the reviews :-) | 15:02 |
| acoles | tdasilva: welcome | 15:02 |
| joe__ | swifterdarrel: regarding my issue with the miss match hash, it turns out that one of the machines that I restarted replicated back, that means it is not a rouge object replicator :( | 15:06 |
| swifterdarrell | joe__: rats :( | 15:06 |
| *** leopoldj has quit IRC | 15:25 | |
| *** annegentle has joined #openstack-swift | 15:31 | |
| *** SkyRocknRoll has joined #openstack-swift | 15:32 | |
| *** rcrit has joined #openstack-swift | 15:39 | |
| rcrit | I'm working on getting tempest to work in an SSL-enabled environment against devstack, specifically using a tls-proxy | 15:40 |
| rcrit | I've got a failed test in swift. It is returning a 301 with an http address rather than https | 15:40 |
| rcrit | now swift itself is running in the clear, we have a TLS Proxy in front of it, so I can't query anything local to know what scheme to use | 15:41 |
| rcrit | I was wondering if anyone had any bright ideas | 15:41 |
| rcrit | e.g. is the service catalog handy to query, something like that? | 15:42 |
| *** mahatic has quit IRC | 15:42 | |
| rcrit | the test in question is tempest.api.object_storage.test_container_staticweb.StaticWebTest.test_web_index | 15:43 |
| *** annegentle has quit IRC | 15:59 | |
| *** mahatic has joined #openstack-swift | 15:59 | |
| joe__ | swifterdarrell: how can I handle this?.. you know I think the object replicator just copies everything missing from on the drive without checking the hash, do you mind looking at the code | 16:02 |
| swifterdarrell | joe__: what verison of swift are you running? | 16:03 |
| swifterdarrell | joe__: (not that I think the replicator's changed *that* much, but I might as well look at the same code you're running) | 16:03 |
| joe__ | 2.1 | 16:05 |
| joe__ | swifterdarrell: yes running 2.1 | 16:06 |
| swifterdarrell | joe__: k | 16:06 |
| *** mmcardle has joined #openstack-swift | 16:07 | |
| swifterdarrell | joe__: can you tell if there are any of these affected objects which are only on disk in the old/wrong locations? I.e fresh HEAD/GET requests 404 but you know they are on disk at old path? | 16:09 |
| swifterdarrell | joe__: ah, of course... the replicator operates on partitions and does not open up every object and inspect its name and cross-check that against its location on disk | 16:11 |
| swifterdarrell | joe__: that's be much more expensive, and is the sort of thing teh auditor does (and is rightfully quarantining these now-misplaced objects). However, the object-replicator will put them right back. | 16:12 |
| swifterdarrell | joe__: (cc cschwede) ok, I think we're back to: "Swift can't auto-correct this in a scalable manner, so you'll need a walker tool that inspects each object and if it's in the wrong place, copies it to the correct location instead of quarantining it" | 16:13 |
| swifterdarrell | joe__: which sounds to me like a slightly-modified version of the object-auditor that you run once on each node; that'll make sure all objects in wrong location are in teh cluster at teh right location | 16:14 |
| *** mmcardle has quit IRC | 16:14 | |
| joe__ | swifterdarrell: yeah exactly, it keeps coming back, of course in an small cluster it would just quarantine the file in all the nodes before they would replicate.. but in my case it replicates back before that | 16:14 |
| *** mmcardle has joined #openstack-swift | 16:15 | |
| swifterdarrell | joe__: do you still have the old prefix/suffix values that were in the incorrect swift.conf? | 16:15 |
| joe__ | swifterdarrell: maybe.. let me check | 16:15 |
| joe__ | swifterdarrell: I have an old value, but I am unsure that was it | 16:17 |
| swifterdarrell | joe__: actually, that's not necessary, I don't think... if you have one replica, all you need to know is the partition it mapped to with the old swift.conf and you have that in its filesystem location; then you can just lookup that same partition number in the object ring and calculate the other 2 locations that might have the other 2 replicas of teh object in-hand. | 16:17 |
| *** mmcardle has quit IRC | 16:17 | |
| swifterdarrell | joe__: okay, so here's what a correcting walker needs to do: when an object in wrong place is encountered, ensure its 3 replicas in the new/correct location are present, then delete the object from all 3 old locations | 16:19 |
| swifterdarrell | joe__: after all taht, you probably want to clean out the quarantine data since that'll just be wasting some disk space at that point | 16:19 |
| swifterdarrell | joe__: afaik, no such tool exists at this time | 16:20 |
| peluse | clayg, thanks for the review on https://review.openstack.org/#/c/134065/ I'm guessin you used both eyes :) Have to go teach some JA classes, will address when I get back... | 16:20 |
| joe__ | swifterdarrell: I checked that manually once, and I was able to find the same partition in the other node | 16:21 |
| swifterdarrell | joe__: btw, I don't think we want the object-auditor doing this sort of thing automatically, as it could put crazy/bad data into good locations which would be worse | 16:21 |
| *** annegentle has joined #openstack-swift | 16:25 | |
| openstackgerrit | Daniel Wakefield proposed openstack/python-swiftclient: Verify MD5 of uploaded objects. https://review.openstack.org/129254 | 16:25 |
| openstackgerrit | Daniel Wakefield proposed openstack/python-swiftclient: Verify MD5 of uploaded objects. https://review.openstack.org/129254 | 16:27 |
| joe__ | swifterdarrell: so from my previous paste http://paste.openstack.org/show/132411/ for example I checked the location in node1 (192.168.168.2.21) and then on node5 192.168.2.25 and I just switched the drive letter as it says in the metadata leaving the same partition number and it is the same object (checked by md5sum) | 16:28 |
| swifterdarrell | joe__: *nod* so that's confirmation that the old swift.conf prefix/suffix values are not necessary to find all locations of all replicas of a misplaced object | 16:29 |
| swifterdarrell | joe__: but you still need something to find all the misplaced objs, make sure they exist in the right place, and then very-close-together delete all 3 replicas which are in the wrong place | 16:30 |
| swifterdarrell | joe__: do you have a development or test cluster? | 16:30 |
| swifterdarrell | joe__: if so, I'd recommend replicating the problem on a smaller scale there, and use that cluster for tool development | 16:31 |
| *** foexle has quit IRC | 16:31 | |
| joe__ | swifterdarrell: thanks, yes we do have a dev environment, so you think from the auditor is the best to create the tool from? | 16:39 |
| *** gyee has joined #openstack-swift | 16:42 | |
| *** SkyRocknRoll has quit IRC | 16:52 | |
| *** exploreshaifali has quit IRC | 16:54 | |
| openstackgerrit | paul luse proposed openstack/swift: Add node/pair index patch back into feature/EC https://review.openstack.org/134065 | 16:56 |
| *** zaitcev has joined #openstack-swift | 17:06 | |
| *** ChanServ sets mode: +v zaitcev | 17:06 | |
| *** david-lyle has joined #openstack-swift | 17:10 | |
| *** kyles_ne has joined #openstack-swift | 17:10 | |
| *** annegentle has quit IRC | 17:15 | |
| *** geaaru has quit IRC | 17:22 | |
| *** annegentle has joined #openstack-swift | 17:25 | |
| *** X019 has quit IRC | 17:37 | |
| *** annegentle has quit IRC | 17:41 | |
| *** jistr has quit IRC | 18:07 | |
| *** leews_ has joined #openstack-swift | 18:16 | |
| *** leews has quit IRC | 18:17 | |
| *** jordanP has quit IRC | 18:41 | |
| *** lpabon has quit IRC | 19:05 | |
| *** rdaly2 has joined #openstack-swift | 19:07 | |
| *** acoles is now known as acoles_away | 19:09 | |
| *** nellysmitt has joined #openstack-swift | 19:10 | |
| *** tdasilva has quit IRC | 19:14 | |
| *** nellysmitt has quit IRC | 19:14 | |
| swifterdarrell | joe__: ya, I'd start w/the auditor... it's got the walk-every object code, it's detecting the mismatch and quarantining, and it has a "run-once" mode, iirc | 19:15 |
| *** mkollaro has joined #openstack-swift | 19:16 | |
| swifterdarrell | joe__: the missing bits are just to hack it to not read all bytes of file for md5sum (unnecessary for your problem) and have different behavior (only) when the obj name hashes to a different location than the filesystem location at which the object was found. | 19:16 |
| joe__ | swifterdarrell: thanks, my true problem is no so much identifying it but deleting it before it replicates again | 19:18 |
| swifterdarrell | joe__: *nod* a piece of code will be faster than you at getting all 3 replicas deleted | 19:19 |
| swifterdarrell | joe__: but of course, you only want to do that if you're sure the obj is also already in the right location | 19:19 |
| swifterdarrell | joe__: (or you could lose data) | 19:19 |
| joe__ | swifterdarrell: I agree, thanks | 19:20 |
| *** shri1 has joined #openstack-swift | 19:28 | |
| *** tdasilva has joined #openstack-swift | 19:29 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift-specs: Spec for at-rest encryption. https://review.openstack.org/123220 | 19:31 |
| *** exploreshaifali has joined #openstack-swift | 19:36 | |
| *** annegentle has joined #openstack-swift | 19:44 | |
| *** annegentle has quit IRC | 19:45 | |
| *** Guest33321 has joined #openstack-swift | 19:45 | |
| *** Guest33321 is now known as annegentle | 19:45 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift-specs: Spec for at-rest encryption. https://review.openstack.org/123220 | 19:53 |
| *** nellysmitt has joined #openstack-swift | 20:12 | |
| *** tdasilva has quit IRC | 20:25 | |
| *** Masa_ has joined #openstack-swift | 20:31 | |
| *** Masa_ has quit IRC | 20:33 | |
| *** foexle has joined #openstack-swift | 20:35 | |
| *** tdasilva has joined #openstack-swift | 20:36 | |
| *** mkollaro has quit IRC | 20:40 | |
| *** Nadeem has joined #openstack-swift | 21:00 | |
| *** Nadeem has quit IRC | 21:00 | |
| *** X019 has joined #openstack-swift | 21:06 | |
| *** fifieldt has quit IRC | 21:10 | |
| *** X019 has quit IRC | 21:12 | |
| *** fifieldt has joined #openstack-swift | 21:24 | |
| *** X019 has joined #openstack-swift | 21:31 | |
| *** foexle has quit IRC | 21:56 | |
| *** kyles_ne has quit IRC | 22:08 | |
| *** kyles_ne has joined #openstack-swift | 22:14 | |
| *** dmsimard is now known as dmsimard_away | 22:22 | |
| clayg | torgomatic: why are you trying to be helpful? it just throws everyone off. https://review.openstack.org/#/c/117193/ | 22:31 |
| *** rdaly2 has quit IRC | 22:39 | |
| *** rdaly2 has joined #openstack-swift | 22:40 | |
| *** Viswanath has joined #openstack-swift | 22:42 | |
| *** rdaly2 has quit IRC | 22:45 | |
| *** Viswanath has quit IRC | 22:45 | |
| *** foexle has joined #openstack-swift | 22:47 | |
| *** aix has quit IRC | 22:48 | |
| *** foexle has quit IRC | 22:58 | |
| *** exploreshaifali has quit IRC | 22:59 | |
| *** nomad411 has joined #openstack-swift | 23:12 | |
| *** nomad411 has quit IRC | 23:12 | |
| *** nomad411 has joined #openstack-swift | 23:16 | |
| *** kyles_ne has quit IRC | 23:20 | |
| *** nomad411 has quit IRC | 23:21 | |
| *** nomad411 has joined #openstack-swift | 23:22 | |
| *** gyee has quit IRC | 23:36 | |
| *** X019 has quit IRC | 23:42 | |
| *** mahatic has quit IRC | 23:43 | |
| *** occupant has joined #openstack-swift | 23:50 | |
| *** foexle has joined #openstack-swift | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!