| mattoliverau | clayg: yeah, Melboure is safe from fires. My house is still standing so that's nice :) | 00:05 |
|---|---|---|
| *** takamatsu has joined #openstack-swift | 00:16 | |
| *** takamatsu has quit IRC | 00:31 | |
| seongsoocho | morning ~ ! | 00:32 |
| seongsoocho | mattoliverau: I'm so happy you are safe. | 00:32 |
| mattoliverau | seongsoocho: thanks :) | 00:33 |
| mattoliverau | cool, well how about we meet on collins street and swanston intersection somewhere, we can wander down that direction. Flinders land and degraces street is near there which also should have nice laneway cafes. in case we decide to go somewhere else, but dumblings also sound good :) | 00:36 |
| mattoliverau | opps sorry, wrong window :P | 00:36 |
| mattoliverau | but tdasilva ^ :P | 00:36 |
| mattoliverau | but will take it back to private chat :P | 00:36 |
| tdasilva | :) | 00:37 |
| *** renich has quit IRC | 00:55 | |
| *** tkajinam has quit IRC | 02:34 | |
| *** mvkr has joined #openstack-swift | 02:49 | |
| openstackgerrit | zhufl proposed openstack/swift master: Fix the duplicated words issue like "the the " https://review.opendev.org/701127 | 03:02 |
| ktsuyuzaki | back from new year vacation. A happy new year Swifters. | 03:39 |
| *** tkajinam has joined #openstack-swift | 03:41 | |
| *** psachin has joined #openstack-swift | 03:41 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:33 | |
| *** tkajinam has quit IRC | 05:52 | |
| *** tkajinam has joined #openstack-swift | 05:56 | |
| *** threestrands has quit IRC | 05:58 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: Let developers/operators add watchers to object audit https://review.opendev.org/212824 | 06:03 |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: WIP: Add a reference implementation of audit watchers https://review.opendev.org/701139 | 06:03 |
| *** rcernin has quit IRC | 06:12 | |
| *** tkajinam has quit IRC | 06:52 | |
| *** tkajinam has joined #openstack-swift | 07:06 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 07:26 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: s3api: Implement object versioning API https://review.opendev.org/673682 | 07:37 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: Skip container-sync when versioning enabled https://review.opendev.org/698139 | 07:42 |
| *** cschwede has joined #openstack-swift | 07:55 | |
| *** ChanServ sets mode: +v cschwede | 07:55 | |
| alecuyer | ktsuyuzaki: happy new year! | 08:10 |
| ktsuyuzaki | alecuyer: happy new year! | 08:10 |
| *** ktsuyuzaki is now known as kota_ | 08:10 | |
| *** tesseract has joined #openstack-swift | 08:11 | |
| *** tkajinam has quit IRC | 08:19 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: Skip container-sync when versioning enabled https://review.opendev.org/698139 | 08:38 |
| *** rpittau|afk is now known as rpittau | 08:39 | |
| *** rdejoux has quit IRC | 09:06 | |
| *** rdejoux has joined #openstack-swift | 09:35 | |
| *** pcaruana has joined #openstack-swift | 09:54 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 10:02 |
| *** pcaruana has quit IRC | 10:11 | |
| *** pcaruana has joined #openstack-swift | 10:12 | |
| *** diablo_rojo has joined #openstack-swift | 15:00 | |
| *** diablo_rojo has quit IRC | 15:02 | |
| *** psachin has quit IRC | 15:37 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 16:14 |
| *** gyee has joined #openstack-swift | 16:15 | |
| clayg | well see how that goes | 16:15 |
| *** psachin has joined #openstack-swift | 16:27 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Make FakeSwift query param order agnostic https://review.opendev.org/701247 | 16:33 |
| *** rpittau is now known as rpittau|afk | 16:58 | |
| *** rdejoux has quit IRC | 17:16 | |
| *** CommaCrazy has joined #openstack-swift | 17:17 | |
| CommaCrazy | Hello ladies and gents, can anyone tell me what happens if one of the drives gets full but there is plenty of space on other drives. Will it rebalance by itself or do I need to edit the weight of the object drive in the object.builder and sync the ring? | 17:17 |
| CommaCrazy | Drive is healthy. Oh and Happy New Year to all <3 | 17:18 |
| DHE | if the weight is set wrong, that's a problem and should be corrected. otherwise it's PROBABLY caused by too many handoffs going to this disk and it hasn't bee able to clean itself out | 17:20 |
| DHE | is it 100% FULL, or just uncomfortably full relative to other disks | 17:21 |
| CommaCrazy | it is at 100% and it weight the same as all the other drives that are the same size so proper weight is set, | 17:26 |
| DHE | ..is this hypothetical or is it actually happening? | 17:28 |
| CommaCrazy | actually happening | 17:29 |
| DHE | ah | 17:29 |
| CommaCrazy | looks like it is trying to clear itself out but it is getting back to it's full self for a couple of hours now | 17:29 |
| DHE | so generally the cause would be that this disk is unable to connect to other hosts to validate that the data it is holding as a handoff has been successfully delivered and hence safe to delete | 17:30 |
| CommaCrazy | ahhhh got ya, gonna check the whole cluster for anything related to that. | 17:30 |
| CommaCrazy | Thank you DHE much appreciated. | 17:30 |
| DHE | unless you've done something especially wild like deleted a replication policy that still held data, or anything writing to /srv/node/$THISDISK that shouldn't be? | 17:31 |
| DHE | is there only 1 disk in this server? | 17:31 |
| DHE | there's something not right here. this should not affect exactly 1 disk in the whole cluster. for my communication issue theory to make sense you'd either have the whole machine suffering more, or you'd have more examples of disks with the same issue | 17:32 |
| CommaCrazy | Nope, the status quo from healthy state has been preserved so it has to be another server in the cluster that is causing the issue then. | 17:32 |
| CommaCrazy | nope it has many drives on on server. | 17:33 |
| *** evrardjp has quit IRC | 17:33 | |
| CommaCrazy | Will dig and make sense of it. Thank you for pointing me in the right direction :) | 17:33 |
| *** evrardjp has joined #openstack-swift | 17:33 | |
| DHE | maybe someone else can chime in. if a disk gets full, bad stuff can happen but there's no good reason for only 1 disk to fill up on its own and not drain on its own | 17:34 |
| clayg | agree, very strange for any problem in the cluster to only effect one disk - unless the problem is THE DISK (better chect disk utilization with iostat and /var/log/messgaes for disk errors) | 17:38 |
| clayg | if you have higly variable dispersion overall (like disks are between 20-90% full across the cluster) then it's more reasonable that you're only *noticing* the one outlier that managed to get THE MOST full | 17:39 |
| clayg | @CommaCrazy have you added capacity to the cluster before? How do you monitor your rebalances? | 17:39 |
| *** zaitcev has joined #openstack-swift | 17:47 | |
| *** ChanServ sets mode: +v zaitcev | 17:47 | |
| clayg | yay, I love the new account listing addition on object versions! 😍 @tdasilva | 17:47 |
| tdasilva | if only tests passed :/ | 17:47 |
| tdasilva | clayg: but i think p 701247 should be the way | 17:49 |
| patchbot | https://review.opendev.org/#/c/701247/ - swift - Make FakeSwift query param order agnostic - 1 patch set | 17:49 |
| clayg | @tdasilva maybe, those tests didn't pass either 🤣 | 17:49 |
| tdasilva | clayg: is it because of the pdb statement? | 17:50 |
| clayg | LOL | 17:55 |
| clayg | what did I do!! 😞 | 17:55 |
| clayg | sorry, i'm getting my saio off of SSL - my workspace is a mess | 17:56 |
| clayg | tdasilva: oic in the fakeswift fix - yeah i was gunna checkout the py3 failures on the object versions patch | 17:57 |
| timburke | ahahaha sorry clayg :P | 17:57 |
| timburke | happy new year kota_! | 17:58 |
| clayg | timburke: it'll be great! I do want to go ahead and get the cert installed - it seemed like you knew how to do that from your comments on the PR tho? | 17:58 |
| CommaCrazy | Bah found the issue. Looks like a custom setup for a container that is eating at a specific drive, that's all I can say :D much appreciate your time and response. Have a wonderful day! | 17:58 |
| *** CommaCrazy has quit IRC | 17:58 | |
| timburke | 👍 | 17:58 |
| clayg | CommaCrazy: container sharding is getting to be pretty awesome, try it out! | 17:59 |
| *** pcaruana has quit IRC | 18:07 | |
| *** psachin has quit IRC | 18:12 | |
| *** pcaruana has joined #openstack-swift | 18:16 | |
| *** pcaruana has quit IRC | 18:21 | |
| *** tesseract has quit IRC | 18:38 | |
| timburke | clayg, what do you think about putting encryption/keymaster in the noauth pipeline in https://github.com/swiftstack/vagrant-swift-all-in-one/ ? (maybe set to disabled?) | 19:02 |
| clayg | so the normalize_path plan is fine - but there is something up with those reconciler tests 🤷♂️ | 19:04 |
| clayg | timburke: idk, i think I use my noauth app differently than you do | 19:05 |
| clayg | timburke: because we recently put versioned writes in there too | 19:05 |
| clayg | it seems like you're moving it in the direction of "just like a regular pipeline but without auth" where as I used to use it as a way to peak at "how the storage servers really look like w/o having to bust open direct client" | 19:05 |
| clayg | but I think my use-case has other solutions ... so I can probably come around | 19:06 |
| timburke | yeah, that makes sense, too -- it's no real hardship for me to add it after bring up if i want it | 19:08 |
| timburke | that's why i asked :-) | 19:08 |
| timburke | i really *do* mainly use it like a regular proxy but without auth -- i've definitely come around to the notion that the best swiftclient is curl ;-) | 19:09 |
| timburke | clayg, in light of "how the storage servers really look like w/o having to bust open direct client", i think you'll *really like* https://github.com/swiftstack/vagrant-swift-all-in-one/pull/88 :D | 19:20 |
| *** lifeless has quit IRC | 19:24 | |
| *** lifeless has joined #openstack-swift | 19:25 | |
| openstackgerrit | Merged openstack/swift master: Fix the duplicated words issue like "the the " https://review.opendev.org/701127 | 19:50 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 19:57 |
| clayg | shit, i just squashed in the normalize stuff with new object versionisng - i should probably pull it out and rebase on top :smh: | 19:58 |
| *** ianychoi has quit IRC | 20:04 | |
| *** ianychoi has joined #openstack-swift | 20:19 | |
| clayg | ain't no party like a unittest party! | 22:13 |
| mattoliverau | morning | 22:16 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Make FakeSwift query param order agnostic https://review.opendev.org/701247 | 22:27 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 22:33 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: s3api: Implement object versioning API https://review.opendev.org/673682 | 22:34 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: s3api: Make multi-deletes async https://review.opendev.org/648263 | 22:35 |
| *** rcernin has joined #openstack-swift | 22:40 | |
| seongsoocho | morning | 22:42 |
| clayg | seongsoocho: good morning! | 22:43 |
| seongsoocho | o/ | 22:52 |
| *** tkajinam has joined #openstack-swift | 23:01 | |
| clayg | it's weird for me how some tests are listing containers and complaining about ' a10542c70fbef445c68bcd088cb1444cf3' being in the list unexpectedly - then other tests are throwing 404 about that same container not being there?! 'https://zuul.opendev.org/t/openstack/build/e7abc231dc1944db89d346c3ec23074b/console | 23:15 |
| clayg | seems racy | 23:15 |
| clayg | timburke: will p 701247 try and merge now or do I have to say recheck too? | 23:16 |
| patchbot | https://review.opendev.org/#/c/701247/ - swift - Make FakeSwift query param order agnostic - 2 patch sets | 23:16 |
| clayg | timburke: I did try out the no-op gatekeeper and honestly mostly liked it | 23:21 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!