| simondodsley | Is there a problem with review.opendev.org. Can't get my CI system to connect. Getting `Received disconnect from 199.204.45.33 port 29418:12: Too many concurrent connections (96) - max. allowed: 96` when try to get the gerrit version using SSH. | 16:06 |
|---|---|---|
| simondodsley | This is when trying to set up SoftwareFactory to build a third party CI | 16:07 |
| clarkb | simondodsley: you're hitting a built in connection limit failsafe | 16:11 |
| clarkb | the user you are using already has 96 connections over ssh and we aren't allowing you to make more | 16:11 |
| simondodsley | how is that possible? the user is only being run by the SoftwareFactory build commnad. Are these historical connections that are dead? | 16:12 |
| clarkb | sometimes this can occur if you have an aggressive firewall killing idle connections without sending appropriate reset/fin type packets back to the server | 16:12 |
| clarkb | maybe. I would start with a quick investigation on your end to see if you have a bunch of open ssh connections | 16:12 |
| clarkb | if not thne you may need to look into the firewall possibility | 16:12 |
| simondodsley | thanks - I'll check that out | 16:13 |
| fungi | simondodsley: if it helps, all 96 connections are from the same ip address, so it likely is stale/dead sockets | 16:46 |
| simondodsley | thanks fungi | 16:46 |
| simondodsley | struggling with softwarefactory - refusing to start | 16:47 |
| clarkb | fungi: did a change get pushed for the pbr testing thing yet? I feel like I had an idea over the weekend that I didn't write down and now its gone | 17:17 |
| clarkb | maybe it was to cave and make pbr a pep517 package then have working build isolation | 17:18 |
| clarkb | ok one tricky thing with that is that pbr currently relies on itself and relative code paths to install the hooks for setuptools | 17:21 |
| clarkb | I wonder if we can do requires . in the pyproject.toml but how do you do that without a chicken and egg problem | 17:22 |
| clarkb | https://github.com/pypa/setuptools/blob/main/pyproject.toml#L3-L4 maybe we do something like this | 17:22 |
| clarkb | oh wait no the pbr setup.py isn't a full blown pbr setup.py its only relying on a specific function which I think may continue to work with relative paths if we do pyproject.toml and just list setuptools | 17:23 |
| * clarkb tests this | 17:23 | |
| fungi | clarkb: i think where we ended was "we can just serialize the tests, they're fast anyway" | 17:26 |
| fungi | but no, i didn't push a change for that. timburke had already talked about doing that anyway, but not sure if he's still interested now that the change which was hitting those concurrency issues merged (though he has a couple more stacked up behind it too) | 17:27 |
| fungi | i agree that the root of the problem is that our "unit tests" for pbr can't reliably mock pip interactions, so we're stuck calling out to it and dealing with its side effects and incomplete solutions for concurrent processes | 17:28 |
| fungi | and just accepting that is less work that trying to fight against it | 17:29 |
| timburke | i can go ahead and push that up, see if it seems to make tests more reliable. i don't think the other patches were stacked, exactly; just blocked waiting on having a functional (py2) tempest-full | 17:29 |
| fungi | yeah, i didn't mean to imply there was a get relationship between them, just that you couldn't merge them until that other fix landed | 17:29 |
| fungi | s/get/git/ | 17:30 |
| clarkb | ok I did a quick test of switching pbr over to pyproject.toml and we still end up with a build/ dir. The problem there is that pbr relies on setuptools and setuptools is going to make the build/ dir | 17:31 |
| clarkb | for this reason we should proceed with the serial test runs | 17:31 |
| clarkb | also build isolation doesn't mean what you think it means apparently | 17:31 |
| fungi | yeah, "build isolation" means isolation from python packages installed in the calling environment, but using a venv | 17:32 |
| fungi | i had just hoped that it also might try not to step on the toes of other build processes happening at the same time | 17:33 |
| clarkb | I think we do serial tests or do a full copy of the source tree and then install out of that. Honestly serial seems simple and fast enough here | 17:33 |
| fungi | clearly that's out of scope | 17:33 |
| fungi | yep, i concur | 17:33 |
| opendevreview | Tim Burke proposed openstack/pbr master: Serialize tests https://review.opendev.org/c/openstack/pbr/+/906310 | 17:36 |
| fungi | the remaining changes should hopefully merge much fasted after that ^ | 17:38 |
| clarkb | looks like setuptools is also basically going to always delete these dirs. I really wonder if they would accept a tmpdir patch instead | 17:39 |
| clarkb | probably not since I think the code paths are deep in legacy stuff they probably want to fully refactor/remove rather than improve | 17:39 |
| fungi | yeah, the "cleaning" is probably responsible for the enoent/file-not-found errors | 17:39 |
| clarkb | yup | 17:40 |
| clarkb | its trying to delete a dir/file that was already cleaned up by another install process | 17:40 |
| opendevreview | Merged openstack/pbr master: Serialize tests https://review.opendev.org/c/openstack/pbr/+/906310 | 21:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!