| opendevreview | Tim Burke proposed openstack/swift master: tests: Fix replicator test for py311 https://review.opendev.org/c/openstack/swift/+/886538 | 00:28 |
|---|---|---|
| opendevreview | Tim Burke proposed openstack/swift master: tests: Stop trying to mutate instantiated EntryPoints https://review.opendev.org/c/openstack/swift/+/886539 | 00:28 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: test under py311 https://review.opendev.org/c/openstack/swift/+/886541 | 00:28 |
| opendevreview | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/c/openstack/swift/+/886694 | 03:14 |
| opendevreview | Philippe SERAPHIN proposed openstack/swift master: Adjust column width https://review.opendev.org/c/openstack/swift/+/886802 | 04:46 |
| opendevreview | Jianjian Huo proposed openstack/swift master: proxy: add new metrics to account/container_info cache for skip/miss https://review.opendev.org/c/openstack/swift/+/885798 | 05:14 |
| opendevreview | Merged openstack/swift master: tests: Fix replicator test for py311 https://review.opendev.org/c/openstack/swift/+/886538 | 11:59 |
| reid_g | eventlet is at .25 (Ubuntu 20.04 standard version) and I see that Ubuntu Cloud Archive has .30 available | 13:22 |
| reid_g | The process is started by systemd with user & group = swift | 13:23 |
| reid_g | The config file also says user = swift | 13:24 |
| reid_g | Is there a setting in conf for keepalive? I don't see it in the docs | 13:25 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: remove client_chunk_size and skip_bytes from GetOrHeadHandler https://review.opendev.org/c/openstack/swift/+/886823 | 13:42 |
| reid_g | The issues happens with just the object-server from what I recall. | 13:57 |
| reid_g | It's odd that when the error occurs I don't see anything using the port in netstat||ss | 13:57 |
| reid_g | timburke, any other thoughts? | 14:32 |
| paladox | timburke: it also lowered the load + I/o as well! | 16:06 |
| paladox | Wasn’t expecting that | 16:06 |
| opendevreview | Tim Burke proposed openstack/swift master: py3: Quiet RemoteDisconnected tracebacks https://review.opendev.org/c/openstack/swift/+/886780 | 16:41 |
| timburke | reid_g, i don't think we have a setting for keepalive -- i just saw you pointing at the comment and wondered if you'd tried commenting out the next line or something. like i said, we should be using SO_REUSEPORT already, so i'm similarly confused. not seeing anything else using the port makes it even stranger :-/ | 16:56 |
| timburke | paladox, yeah, that's a client bug :-( glad to hear the situation's improving, though! the reduced i/o makes sense to me -- when a disk fills up, we start trying to over-replicate anything that should be there across the rest of the cluster, which can add a bunch of extra load | 16:58 |
| reid_g | timburke, I guess I'm not that great at querying the ports. I just found that they are sitting in Time-Wait state after shutting down the server. | 17:11 |
| timburke | but they should still be associated with that user, right? i would've thought we'd be fine to start listening again | 17:14 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: error limit the correct node when resuming GETs https://review.opendev.org/c/openstack/swift/+/886864 | 17:23 |
| reid_g | When I look at the info from `ss -ntape "( sport = :6000 )"` it doesn't have a user | 17:24 |
| reid_g | TIME-WAIT 0 0 10.40.196.24:6000 10.40.196.18:6200 timer:(timewait,25sec,0) ino:0 sk:58b | 17:24 |
| reid_g | New entries keep popping up from other object-server-replication servers. Each time a new connection appears, that new connection timer starts at 60s | 17:27 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Include filter with no _pipeline_final_app https://review.opendev.org/c/openstack/swift/+/886010 | 17:58 |
| reid_g | it looks like all of our dangling time-waits close out if we stop object-replicator and wait. Does the replicator use the source ports from the object-server? | 18:35 |
| paladox | timburke: oh I see | 18:36 |
| opendevreview | Tim Burke proposed openstack/swift master: obj: Close replicator/reconstructor connections https://review.opendev.org/c/openstack/swift/+/886868 | 19:09 |
| timburke | reid_g, ^^^ might help? i think we haven't been cleaning up our backend connections well enough :-/ | 19:09 |
| opendevreview | Merged openstack/swift master: Use format_exc, not format_stack https://review.opendev.org/c/openstack/swift/+/884433 | 19:12 |
| reid_g | /shrug. On the server where we were getting the port bind issue, I stopped the object-replicator and object-reconstructor so maybe? it's strange to me that the infon from netstat/ss doesn't show the object-replicator proc in the socket. | 19:22 |
| opendevreview | Merged openstack/swift master: tests: Stop trying to mutate instantiated EntryPoints https://review.opendev.org/c/openstack/swift/+/886539 | 20:12 |
| opendevreview | Tim Burke proposed openstack/swift master: Encode header in latin-1 with wsgi_to_bytes https://review.opendev.org/c/openstack/swift/+/884240 | 21:23 |
| opendevreview | Tim Burke proposed openstack/swift master: Add non-ascii meta to ssync probe test https://review.opendev.org/c/openstack/swift/+/884954 | 21:23 |
| opendevreview | Merged openstack/swift master: move test_GET_pipeline to BaseTestObjectController https://review.opendev.org/c/openstack/swift/+/883084 | 23:41 |
| opendevreview | Tim Burke proposed openstack/swift master: sharder: Fix completed metric emission https://review.opendev.org/c/openstack/swift/+/886883 | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!