| *** tovin07_ has joined #openstack-swift | 01:14 | |
| *** psachin has joined #openstack-swift | 01:19 | |
| *** vint_bra has quit IRC | 01:49 | |
| thurloat | howdy mattoliverau | 02:01 |
|---|---|---|
| *** Shatadru has joined #openstack-swift | 02:21 | |
| *** Shatadru is now known as Shatadru|coffee| | 02:33 | |
| mattoliverau | thurloat: o/ | 02:48 |
| thurloat | o7 | 02:48 |
| thurloat | I've heard of folks deleting all the hashes.pkl files to help clean up wonky tombstones, is that effective? | 02:51 |
| *** Shatadru|coffee| is now known as Shatadru | 03:05 | |
| *** guimaluf has quit IRC | 03:26 | |
| *** gkadam has joined #openstack-swift | 03:35 | |
| mattoliverau | thurloat: that's only if your still affected by the ts hanging around bug. Because they were missed until hashes was updated/recreated. But in either case only ts older then reclaim age in any case will be removed. Otherwise removing all the hashes would cause a bunch of iops as they all have to be recalculated. | 03:35 |
| mattoliverau | So yes, in a way.. but also depends on what you mean my wonky :) | 03:44 |
| *** links has joined #openstack-swift | 03:50 | |
| *** Shatadru is now known as Shatadru|afk | 04:04 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:09 | |
| *** mingyu has quit IRC | 04:12 | |
| *** mingyu has joined #openstack-swift | 04:13 | |
| *** mingyu has quit IRC | 04:17 | |
| *** Shatadru|afk is now known as Shatadru | 04:34 | |
| *** SkyRocknRoll has quit IRC | 04:44 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:48 | |
| *** Dinesh_Bhor has joined #openstack-swift | 04:48 | |
| *** cshastri has joined #openstack-swift | 05:36 | |
| *** spectr has joined #openstack-swift | 05:54 | |
| *** ChubYann has quit IRC | 06:01 | |
| *** mingyu has joined #openstack-swift | 06:08 | |
| *** klrmn has quit IRC | 06:27 | |
| openstackgerrit | HCLTech-SSW proposed openstack/swift master: Added the man page for container-sync-realms.conf https://review.openstack.org/510017 | 06:35 |
| *** rcernin has joined #openstack-swift | 06:40 | |
| *** hseipp has joined #openstack-swift | 06:44 | |
| *** skudlik has joined #openstack-swift | 07:03 | |
| *** pcaruana has joined #openstack-swift | 07:08 | |
| *** hoonetorg has quit IRC | 07:33 | |
| *** Shatadru is now known as Shatadru|brb | 07:34 | |
| *** tone_zrt has joined #openstack-swift | 07:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.openstack.org/510000 | 07:40 |
| *** skudlik has left #openstack-swift | 07:44 | |
| *** geaaru has joined #openstack-swift | 07:44 | |
| *** hoonetorg has joined #openstack-swift | 07:46 | |
| *** cbartz has joined #openstack-swift | 07:58 | |
| *** ianychoi has joined #openstack-swift | 07:59 | |
| *** oshritf has joined #openstack-swift | 08:17 | |
| *** tone_zrt has quit IRC | 08:23 | |
| *** tone_zrt has joined #openstack-swift | 08:24 | |
| *** ianychoi has quit IRC | 08:26 | |
| *** d0ugal_ has quit IRC | 08:31 | |
| *** d0ugal has joined #openstack-swift | 08:31 | |
| *** d0ugal has quit IRC | 08:31 | |
| *** d0ugal has joined #openstack-swift | 08:31 | |
| *** Shatadru|brb is now known as Shatadru | 08:39 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Added the man page for container-sync-realms.conf https://review.openstack.org/510017 | 08:57 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Add example to container-sync-realms.conf.5 man page https://review.openstack.org/510450 | 09:08 |
| *** psachin has quit IRC | 09:20 | |
| *** psachin has joined #openstack-swift | 09:23 | |
| *** mingyu has quit IRC | 09:39 | |
| *** mingyu has joined #openstack-swift | 09:39 | |
| *** mingyu has quit IRC | 09:42 | |
| *** mingyu has joined #openstack-swift | 09:42 | |
| *** mingyu has quit IRC | 09:47 | |
| *** mvk has quit IRC | 10:00 | |
| *** psachin has quit IRC | 10:05 | |
| *** mvk has joined #openstack-swift | 10:11 | |
| *** tovin07_ has quit IRC | 10:13 | |
| *** silor has joined #openstack-swift | 10:19 | |
| *** psachin has joined #openstack-swift | 10:20 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: test account autocreate listing format https://review.openstack.org/510520 | 10:34 |
| *** mjseger has joined #openstack-swift | 10:49 | |
| mjseger | mattoliverau: ping | 10:49 |
| mattoliverau | mjseger: pong | 10:52 |
| mjseger | how do you want to talk? | 10:54 |
| mjseger | ok, hang on | 10:54 |
| *** Shatadru is now known as Shatadru|Gone | 10:56 | |
| mjseger | I'm mjseger@gmail | 10:57 |
| mjseger | can you connect to 779-620-397? | 10:57 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: copy put-timestamp from original to sharded db https://review.openstack.org/510092 | 11:01 |
| mjseger | mattoliverau: having issues? do you want to invite me? | 11:02 |
| *** silor has quit IRC | 11:04 | |
| *** silor has joined #openstack-swift | 11:09 | |
| *** SkyRocknRoll has quit IRC | 11:32 | |
| thurloat | thanks for the info mattoliverau, the biggest penalty is iops while everything refigures itself though, right? this cluster is 1.13.x-ish so it's missing a bunch of the tombstone fixes | 11:44 |
| *** SkyRocknRoll has joined #openstack-swift | 11:49 | |
| openstackgerrit | Merged openstack/swift master: Stop clearing params for account_autocreate responses https://review.openstack.org/509594 | 12:08 |
| openstackgerrit | Merged openstack/swift master: Ignore all auditor_status_*.json files in reconstructor https://review.openstack.org/474801 | 12:08 |
| openstackgerrit | Merged openstack/swift master: swift-account-audit: compare each etag to the hash from container https://review.openstack.org/508000 | 12:09 |
| openstackgerrit | Merged openstack/swift master: Replace SOSO auth prefix in examples with more-standard AUTH https://review.openstack.org/508001 | 12:09 |
| *** hoonetorg has quit IRC | 12:23 | |
| *** gkadam has quit IRC | 12:35 | |
| *** chlong has quit IRC | 12:37 | |
| *** hoonetorg has joined #openstack-swift | 12:43 | |
| *** catintheroof has joined #openstack-swift | 12:43 | |
| *** tone_z has joined #openstack-swift | 12:46 | |
| *** tone_zrt has quit IRC | 12:49 | |
| *** jarbod has quit IRC | 12:56 | |
| *** oshritf has quit IRC | 13:19 | |
| *** vint_bra has joined #openstack-swift | 13:39 | |
| *** psachin has quit IRC | 13:48 | |
| *** links has quit IRC | 14:03 | |
| *** chlong has joined #openstack-swift | 14:07 | |
| *** mingyu has joined #openstack-swift | 14:16 | |
| *** skudlik has joined #openstack-swift | 14:18 | |
| *** skudlik has left #openstack-swift | 14:20 | |
| *** chlong_ has joined #openstack-swift | 14:21 | |
| *** spectr has quit IRC | 14:30 | |
| *** cshastri has quit IRC | 14:53 | |
| *** gyee has joined #openstack-swift | 14:55 | |
| *** links has joined #openstack-swift | 14:56 | |
| *** klrmn has joined #openstack-swift | 15:00 | |
| *** SkyRocknRoll has quit IRC | 15:06 | |
| *** mingyu has quit IRC | 15:09 | |
| *** mingyu has joined #openstack-swift | 15:10 | |
| *** mingyu has quit IRC | 15:14 | |
| *** chlong has quit IRC | 15:22 | |
| *** chlong_ has quit IRC | 15:22 | |
| *** cbartz has quit IRC | 15:28 | |
| *** mvk has quit IRC | 15:45 | |
| *** catintheroof has quit IRC | 15:48 | |
| *** skudlik has joined #openstack-swift | 16:05 | |
| *** oshritf has joined #openstack-swift | 16:06 | |
| *** klrmn has quit IRC | 16:08 | |
| *** mingyu has joined #openstack-swift | 16:10 | |
| *** mingyu has quit IRC | 16:15 | |
| *** chlong_ has joined #openstack-swift | 16:16 | |
| *** chlong has joined #openstack-swift | 16:16 | |
| *** skudlik has quit IRC | 16:17 | |
| *** skudlik has joined #openstack-swift | 16:18 | |
| *** mvk has joined #openstack-swift | 16:26 | |
| *** pcaruana has quit IRC | 16:32 | |
| *** oshritf has quit IRC | 16:34 | |
| timburke | good morning | 16:47 |
| *** rcernin has quit IRC | 16:49 | |
| *** oshritf has joined #openstack-swift | 16:51 | |
| *** oshritf has quit IRC | 16:54 | |
| *** skudlik has left #openstack-swift | 16:55 | |
| *** hseipp has quit IRC | 16:56 | |
| *** links has quit IRC | 16:58 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:59 | |
| *** klrmn has joined #openstack-swift | 17:02 | |
| thurloat | o7 timburke | 17:03 |
| thurloat | I think I'm going to write a book / horror non-fiction about what happens when your swift object disks get full | 17:04 |
| *** mingyu has joined #openstack-swift | 17:12 | |
| *** mingyu has quit IRC | 17:16 | |
| notmyname | good morning | 17:24 |
| thurloat | \o/ | 17:26 |
| *** geaaru has quit IRC | 17:30 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 17:43 | |
| *** SkyRocknRoll_ has quit IRC | 17:50 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 18:03 | |
| timburke | we should probably land https://review.openstack.org/#/c/464216/ -- it's got two +A'ed follow-ups :P | 18:09 |
| patchbot | patch 464216 - swift - domain_remap: be more careful about client-path ma... | 18:09 |
| *** pcaruana has joined #openstack-swift | 18:11 | |
| *** ejat has quit IRC | 18:38 | |
| *** ejat has joined #openstack-swift | 18:41 | |
| *** ejat has quit IRC | 18:41 | |
| *** ejat has joined #openstack-swift | 18:41 | |
| *** SkyRocknRoll_ has quit IRC | 19:02 | |
| *** SkyRocknRoll has quit IRC | 19:06 | |
| *** hseipp has joined #openstack-swift | 19:31 | |
| *** hseipp has quit IRC | 19:32 | |
| openstackgerrit | Merged openstack/swift feature/deep: copy put-timestamp from original to sharded db https://review.openstack.org/510092 | 19:39 |
| *** mat128 has joined #openstack-swift | 19:46 | |
| *** ChubYann has joined #openstack-swift | 20:02 | |
| *** silor has quit IRC | 20:13 | |
| *** pcaruana has quit IRC | 20:18 | |
| jungleboyj | notmyname: You around? | 20:31 |
| notmyname | jungleboyj: yes. what's up? | 20:33 |
| jungleboyj | notmyname: Hey, been given the opportunity to administer a cloud with Swift and I am having issues with it as the Glance backend. Not sure what I am doing wrong. | 20:36 |
| jungleboyj | notmyname: Thought it might be related to accidentally having servers in the list that didn't actually exist. So I removed them and rebalanced. | 20:36 |
| notmyname | in the list? what list? | 20:37 |
| jungleboyj | https://www.irccloud.com/pastebin/bbCerw6O/ | 20:37 |
| jungleboyj | When I try to add an image. | 20:37 |
| jungleboyj | The list of nodes from swift-ring-builder account.builder | 20:38 |
| notmyname | ok | 20:38 |
| notmyname | "in the ring" then | 20:38 |
| jungleboyj | notmyname: Right. This was a trippleO deploy and I expect to have the three controllers in the ring. | 20:40 |
| notmyname | jungleboyj: first up, let's use `curl` to test out the cluster (so we know exactly what requests are being made) | 20:40 |
| notmyname | can you GET the /healthcheck url? `curl -i http://your.swift/healthcheck` | 20:41 |
| jungleboyj | notmyname: Ok. | 20:41 |
| notmyname | do you get a 200 back? | 20:41 |
| jungleboyj | Trying here. | 20:48 |
| *** mingyu has joined #openstack-swift | 20:49 | |
| jungleboyj | notmyname: So I did curl -i http://172.24.0.12:6002/healthcheck and got a 200 back. | 20:50 |
| jungleboyj | That was one of the 3 nodes in the ring. | 20:50 |
| notmyname | are the other two ok? | 20:50 |
| jungleboyj | Yep, everyone is home. | 20:51 |
| notmyname | ok, good | 20:52 |
| notmyname | do you have an auth token for the swift cluster? you can get one pretty easily with `swift auth -v` | 20:53 |
| *** mingyu has quit IRC | 20:53 | |
| notmyname | after you have a token, make a container and upload an object | 20:54 |
| jungleboyj | Ok, I just got a token I think. | 20:55 |
| notmyname | `curl -i <swift endpoint>/new_container -XPUT -H "x-auth-token: foobar` | 20:55 |
| notmyname | `curl -i <swift endpoint>/new_container/new_object --data-binary 1234 -XPUT -H "x-auth-token: foobar` | 20:55 |
| notmyname | one of those will likely fail like the log line you're seeing when you save a glance image | 20:55 |
| notmyname | note the returned x-trans-id header that's returned in the error response, and grep all of your logs for that trans-id. then pastebin that for me | 20:56 |
| jungleboyj | When I did the 'swift auth -v' I jsut got beack 6 exports. I didn't see a token. | 20:56 |
| notmyname | should have gotten back a token and an endpoint | 20:57 |
| notmyname | I guess maybe more than one endpoint | 20:57 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: test account autocreate listing format https://review.openstack.org/510520 | 20:58 |
| jungleboyj | Interesting. I just got this: | 20:58 |
| jungleboyj | https://www.irccloud.com/pastebin/GqZGmT9H/ | 20:58 |
| notmyname | oh. umm... leave off the -v | 20:59 |
| jungleboyj | :-) Ah, now that looks like a Token. | 21:00 |
| jungleboyj | Hmmm, I get a bad request . 400 | 21:05 |
| notmyname | for which one? | 21:05 |
| jungleboyj | The -XPUT thought then it says 'Invlaid path: /jays_container' which was what I was trying to add. | 21:06 |
| * jungleboyj is going to embarrass myself with how little I know about Swift. | 21:07 | |
| notmyname | no worries | 21:07 |
| notmyname | when you got the token, you also got a storage endpoint | 21:07 |
| jungleboyj | Yes, OS_STORAGE_URL | 21:08 |
| notmyname | something like http://swift.is.awesome/v1/AUTH_jayisgreat/ | 21:08 |
| * jungleboyj laughs | 21:08 | |
| notmyname | so use that and append the container name | 21:08 |
| jungleboyj | Oh. Ok. | 21:08 |
| notmyname | so `curl -i http://swift.is.awesome/v1/AUTH_jayisgreat/newcontainerhere/ -XPUT` | 21:09 |
| jungleboyj | Sweet. That worked. | 21:10 |
| notmyname | nice. | 21:10 |
| jungleboyj | Now the next step. | 21:10 |
| notmyname | so a 201 response? | 21:10 |
| notmyname | right | 21:10 |
| notmyname | add an object | 21:10 |
| notmyname | append the object name to the thing above, still use -XPUT, and use --data-binary 1234 to send a 4 byte body of "1234" | 21:10 |
| jungleboyj | Ok, I got a 201 for that too. | 21:12 |
| notmyname | hmm.. interesting | 21:12 |
| jungleboyj | Yeah. | 21:13 |
| notmyname | so vary the object name a few times and see if you can get a 503 | 21:13 |
| jungleboyj | So add more objects to my container? | 21:13 |
| notmyname | yes | 21:13 |
| notmyname | but the good news so far is that the proxy servers are working, the account and container servers are working, and it seems that the object servers are working | 21:13 |
| notmyname | at first glance, seems like you have a functional cluster | 21:13 |
| jungleboyj | Hmmm, just made like 6 objects and they were all fine. | 21:14 |
| notmyname | what about reading an object | 21:15 |
| notmyname | given one of the object names you just added, use -XGET and drop the --data-binary parts | 21:16 |
| jungleboyj | Looking good. I got back what I put in. | 21:17 |
| notmyname | that's good | 21:18 |
| notmyname | you could grep the proxy logs for ' 503 ' to see if you can find the bad responses from earlier | 21:19 |
| openstackgerrit | Merged openstack/swift master: domain_remap: be more careful about client-path mangling https://review.openstack.org/464216 | 21:26 |
| jungleboyj | Strange, not seeing any 503's in the logs here. | 21:27 |
| jungleboyj | notmyname: So, the 503 is glance trying to talk to swift using swiftclient. Right? | 21:33 |
| openstackgerrit | Merged openstack/swift master: Improve domain_remap docs https://review.openstack.org/471712 | 21:34 |
| jungleboyj | Strange, now I was able to recreate the 503 from the command line. | 21:47 |
| jungleboyj | notmyname: I am still seeing rsync errors trying to sink to ring nodes that I removed. Could that be causing me issues? | 21:50 |
| *** chlong_ has quit IRC | 21:59 | |
| *** chlong has quit IRC | 21:59 | |
| jungleboyj | Maybe it is because the ring settings were only updated on one of the 3 systems? And perhaps I was getting lucky with loadbalancing? | 21:59 |
| notmyname | jungleboyj: sorry. I was called away for a bit | 22:05 |
| jungleboyj | No problem. I may have a beat on the problem. | 22:05 |
| notmyname | yeah, make sure you have the same rings on all the servers | 22:06 |
| jungleboyj | notmyname: Yep, just fixing that. | 22:06 |
| notmyname | jungleboyj: `swift-recon` can help you find many of these issues | 22:07 |
| openstackgerrit | Merged openstack/swift master: Added the man page for container-sync-realms.conf https://review.openstack.org/510017 | 22:09 |
| jungleboyj | notmyname: Before I was getting errors back when trying to do 'swift-recon container' . It would show the hosts it couldn't contact. | 22:13 |
| jungleboyj | notmyname: Now it just says it is starting recon but it still says 6 hosts when there should only be 3. | 22:14 |
| jungleboyj | notmyname: Is that because I need to also remove the bad IPs from the container.build and object.builder? | 22:15 |
| notmyname | it should be using the .ring.gz files, not the .builder files | 22:16 |
| notmyname | but assuming you deployed the .ring.gz fields after rebalancing, yes. remove the missing IPs | 22:17 |
| jungleboyj | So, let me take a step back. | 22:17 |
| jungleboyj | It appears there is a bug in TrippleO if you redeploy the environment it just appends IPs into the ring.gz files. Instead of replacing. So, that is what I am trying to recover from. | 22:19 |
| *** Sukhdev has joined #openstack-swift | 22:23 | |
| notmyname | ah | 22:25 |
| notmyname | might be interesting to ask cschwede if he's run in to that | 22:25 |
| *** vint_bra has quit IRC | 22:39 | |
| mattoliverau | morning | 22:43 |
| *** catintheroof has joined #openstack-swift | 22:47 | |
| *** Sukhdev has quit IRC | 22:51 | |
| timburke | mattoliverau: o/ | 23:04 |
| openstackgerrit | Tim Burke proposed openstack/swift feature/deep: Fix create_shard_ranges_table migration https://review.openstack.org/510707 | 23:10 |
| timburke | i shoulda figured those two changes might cause some troubles when combined... | 23:12 |
| timburke | also, i really should have included tests in https://review.openstack.org/#/c/508355/ | 23:12 |
| patchbot | patch 508355 - swift (feature/deep) - Add shard_ranges table on failed writes, not reads (MERGED) | 23:12 |
| jungleboyj | notmyname: Things appear to be working better now that I have cleaned up the IP mess. *fingers crossed* | 23:17 |
| notmyname | yay | 23:17 |
| notmyname | ls | 23:17 |
| *** kei_yama has joined #openstack-swift | 23:27 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: domain_remap: stop mangling client-provided paths https://review.openstack.org/464217 | 23:37 |
| torgomatic | I smile every time I'm looking at the SLO middleware and see our use of StreamingPile | 23:47 |
| timburke | torgomatic: you're welcome ;-) | 23:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!