| *** djhankb has quit IRC | 01:50 | |
| *** djhankb has joined #openstack-swift | 01:55 | |
| *** baojg has quit IRC | 02:20 | |
| *** baojg has joined #openstack-swift | 02:21 | |
| *** rcernin has quit IRC | 02:58 | |
| *** rcernin has joined #openstack-swift | 03:19 | |
| *** neonpastor has quit IRC | 04:00 | |
| *** rcernin has quit IRC | 04:12 | |
| *** rcernin has joined #openstack-swift | 04:14 | |
| *** m75abrams has joined #openstack-swift | 04:20 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** baojg has quit IRC | 04:37 | |
| *** baojg has joined #openstack-swift | 04:38 | |
| *** rcernin has quit IRC | 05:48 | |
| *** baojg has quit IRC | 05:53 | |
| *** baojg has joined #openstack-swift | 05:53 | |
| *** rcernin has joined #openstack-swift | 06:03 | |
| *** djhankb has quit IRC | 07:24 | |
| *** djhankb has joined #openstack-swift | 07:25 | |
| *** zigo has joined #openstack-swift | 07:46 | |
| *** rcernin has quit IRC | 07:51 | |
| zigo | timburke: Hi there! When reading your patch here: | 07:51 |
|---|---|---|
| zigo | https://opendev.org/openstack/swift/commit/7d429318ddb854a23cdecfe35721b1ecbe8bcccc | 07:51 |
| zigo | I am wondering what's the implication of the last sentence: | 07:51 |
| zigo | "When switching from Python 2 to Python 3, first upgrade Swift while on Python 2, then upgrade to Python 3." | 07:51 |
| zigo | How can one do that? That's not how distro packages are working... How should we do it? | 07:51 |
| *** baojg has quit IRC | 07:53 | |
| *** baojg has joined #openstack-swift | 07:54 | |
| *** aluria has quit IRC | 08:00 | |
| *** DHE has quit IRC | 08:00 | |
| *** irclogbot_3 has quit IRC | 08:02 | |
| *** aluria has joined #openstack-swift | 08:05 | |
| *** DHE has joined #openstack-swift | 08:05 | |
| *** irclogbot_1 has joined #openstack-swift | 08:08 | |
| -openstackstatus- NOTICE: due to a new release of setuptools (50.0.0), a lot of jobs are currently broken, please do not recheck blindly. see http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016905.html | 09:17 | |
| *** baojg has quit IRC | 09:39 | |
| *** baojg has joined #openstack-swift | 13:10 | |
| *** baojg has quit IRC | 13:49 | |
| *** baojg has joined #openstack-swift | 13:49 | |
| *** openstackgerrit has quit IRC | 14:37 | |
| *** TViernion has quit IRC | 14:37 | |
| *** josephillips has joined #openstack-swift | 14:40 | |
| *** TViernion has joined #openstack-swift | 14:43 | |
| *** josephillips has quit IRC | 15:05 | |
| *** m75abrams has quit IRC | 15:20 | |
| *** josephillips has joined #openstack-swift | 16:03 | |
| *** josephillips has quit IRC | 16:04 | |
| *** josephillips has joined #openstack-swift | 16:10 | |
| *** baojg has quit IRC | 16:11 | |
| *** baojg has joined #openstack-swift | 16:12 | |
| *** djhankb has quit IRC | 17:10 | |
| *** djhankb has joined #openstack-swift | 17:11 | |
| ormandj | on a rebalance, when does the corresponding purge happen from the source as stuff is shuffled around? let's say a reasonably full cluster, and you were adding a new node with a bunch of drives, not doing the step-stone approach as you just wanted to online capacity ASAP - does the reblanace have to fully complete before the corresponding 'emptying' happens on the nodes where the data migrated from? | 17:19 |
| ormandj | we're testing this in a dev cluster with a bunch of data, and it does not appear the drives are draining in the original servers, while the new server is most definitely filling up | 17:20 |
| ormandj | (just added the new server with full eventual weight to see what would happen) | 17:20 |
| timburke | zigo, that's part of why i'm working on backporting the fix -- plan is to have new tags on ussuri and train (at least; should i go back further?) and tell operators to upgrade to latest tag for their stable release before attempting a rolling upgrade that would change the major python version used for swift | 17:28 |
| timburke | if you can tolerate a stop-the-world upgrade, that's fine, too -- but there's currently no way to have new swift on py3 write down encryption metadata (for paths with any non-ascii characters) that will be readable on old swift running on py2 | 17:30 |
| timburke | ormandj, for each moved partition, the former-primary needs to get acks that data's durable on all three current-primaries before it'll be willing to delete the data. meanwhile, the former-and-current primaries will *also* want to push data to the new node | 17:35 |
| timburke | (with default configs) | 17:35 |
| timburke | you'll want to look at a couple config options to make your rebalances go faster (and free space off of disks more quickly): https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L287-L304 | 17:36 |
| timburke | handoffs_first tells those former-and-current primaries to cool their heels so the former-primary has more of a chance to replicate | 17:37 |
| timburke | handoff_delete tells the former-primary that it's OK to delete even when data is less than fully-replicated | 17:39 |
| timburke | i'd start with handoffs_first, see how well that gets things moving. if you've been running with a nearly-full (>90% avg fill?) for a while, you might need handoff_delete as well | 17:41 |
| timburke | of the two, handoff_delete is the more dangerous option, since you're willing sacrificing durability to free space faster | 17:42 |
| *** gyee has joined #openstack-swift | 17:43 | |
| timburke | handoffs_first is mainly about how we schedule work to be done, not how much work to do. there's a similar option for EC -- handoffs_only -- that *does* affect how much work we do in a given cycle; i put that somewhere in between. it's good to use on occasion, but after the expansion settles you'll want at least *some* time with it turned off to ensure you're still fully-durable | 17:47 |
| *** renich has joined #openstack-swift | 18:21 | |
| renich | Good $tod, swift-minded people! o/ | 18:23 |
| renich | I am trying to figure out SSL certs with swift. I am trying to use letsencrypt certs. At first, swift-proxy couldn't read them due to permissions, so I used a post-hook to put them at /etc/swift; with owner/group swift and permissions 400. It reads them now, I presume. The thing is, I am getting an empty response when I try: openstack container list | 18:23 |
| renich | Unable to establish connection to http://os.sof.cloudsigma.com:8080/v1/AUTH_8ac555e42913493e95808b305e628474: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) | 18:24 |
| renich | and I can't find anything on the logs. | 18:24 |
| renich | BTW, the config file I edited was /etc/swift/proxy-server.conf | 18:25 |
| renich | Just the cert_file, key_file settings. I am using the exact same endpoints and stuff. | 18:25 |
| renich | Just to add to the context, other openstack commands work fine so I am 70% sure this is a swift issue. | 18:26 |
| timburke | renich, looks like it's still trying to connect over http (not https) -- maybe you need to update the endpoint_url in your keystone catalog? | 18:27 |
| renich | timburke: so, I need to change the endpoint to https, then? | 18:27 |
| renich | OK | 18:27 |
| renich | let me try that | 18:27 |
| renich | I can use openstack endpoint set --url whatever some-id | 18:28 |
| renich | https://paste.centos.org/view/fabc7d3a | 18:31 |
| renich | Does that seem correct? | 18:31 |
| renich | got this error now: https://paste.centos.org/view/ac27c98c | 18:32 |
| timburke | renich, do you see anything in the proxy server logs? | 18:44 |
| renich | timburke: no. I am gonna turn log_level to DEBUG or something | 18:45 |
| renich | timburke: nothing in the logs; not even with DEBUG log level. I'll try to revert back to http | 18:55 |
| renich | It's strange because curl doesn't want to use port 8080 for https... | 18:58 |
| renich | root@keystone0:~# curl https://os.sof.cloudsigma.com:8080/v1/ | 18:58 |
| renich | curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to os.sof.cloudsigma.com:8080 | 18:58 |
| renich | maybe I should change the port or something? | 18:59 |
| renich | Or, maybe, keystone should be under https as well? | 19:00 |
| ormandj | timburke: awesome info tim, will look into that. | 19:21 |
| ormandj | thank you | 19:21 |
| ormandj | timburke: only at 75% fill or so, so no worries on that bit | 19:22 |
| ormandj | we just had to drop the number of replication workers from 1x drive to 1/8th drive count so the cluster didn't self-immolate, so it's going to take ~10 days to finish up based on the average data across drives. it was beating the cluster to death with one worker per disk. not sure if servers_per_port would help, since that's our next step once we've finished expanding | 19:25 |
| ormandj | the cluster was 500ing out the wazoo with the full 56 replication workers :) | 19:25 |
| ormandj | (even set ionice_class to idle, but seems to have made 0 difference - the rsync appears to pick a bunch of source dirs at once, so even though it's working as a single process, it drives iops up, and not sure ionice idle class handles scheduling that well in cfq | 19:26 |
| *** dsariel has joined #openstack-swift | 19:39 | |
| *** dsariel has quit IRC | 20:26 | |
| *** baojg has quit IRC | 21:18 | |
| *** baojg has joined #openstack-swift | 21:19 | |
| *** openstackgerrit has joined #openstack-swift | 22:18 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add sorting_method=none to get a consistent node order https://review.opendev.org/747310 | 22:18 |
| *** baojg has quit IRC | 22:46 | |
| *** baojg has joined #openstack-swift | 22:47 | |
| *** rcernin has joined #openstack-swift | 22:51 | |
| timburke | made a doodle for the PTG meeting slots: https://doodle.com/poll/ukx6r9mxugfn7sed | 23:29 |
| seongsoocho | done! | 23:36 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!