| fungi | Clark[m]: oh, yep! that definitely does | 00:48 |
|---|---|---|
| tonyb | Interesting, I don't thinnk we saw 100% CPU, not the clock_gettime() in strace. Did we? | 01:01 |
| fungi | we saw haprocy consuming 100% of both processors | 01:03 |
| fungi | haproxy | 01:03 |
| fungi | you can see it on the cacti graphs | 01:03 |
| tonyb | Oh okay | 01:03 |
| fungi | tonyb: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=70926&rra_id=all | 01:05 |
| tonyb | Well there you go. I missed it | 01:06 |
| fungi | it shows up quite well on the weekly and monthly graphs. it happened for long enough that you can even see it briefly on the yearly | 01:06 |
| fungi | when i ran top on the server, it was clearly all the haproxy process | 01:07 |
| fungi | consuming about 97% of both cpus (anything that wasn't used by system basically) | 01:07 |
| tonyb | Ahh okay. Then that's is probably out issue | 01:09 |
| tonyb | Having said that, I think sticking with :lts is a good balance for us. | 01:10 |
| fungi | sure | 01:13 |
| frickler | oh, cool, all issues solved why I was sleeping. now I'm curious how the fix for 2395 will look like, but I must say I'm already very impressed by how haproxy people are handling their issues | 05:59 |
| SvenKieske | frickler: well the haproxy guy is also a very long time linux kernel maintainer (Willy Tarreau), so I guess he knows his stuff ;) | 09:13 |
| opendevreview | Merged opendev/irc-meetings master: Update Barbican project team meeting time https://review.opendev.org/c/opendev/irc-meetings/+/903976 | 12:14 |
| eandersson | frickler: for Senlin I have been actively maintaining the CI and general code base. I'll continue to do so. I unfortunately don't have the time to be PTL for the project, but outside of that the project should be healthy. | 13:36 |
| eandersson | I saw that it was being marked as inactive due to the lack of PTL. | 13:36 |
| eandersson | I can ask around as well to see if someone would be interested in stepping up taking over as PTL for Senlin and maybe delegating some tasks. | 13:38 |
| fungi | eandersson: that topic is probably better discussed in the #openstack-tc channel, for future reference | 13:39 |
| eandersson | I see. Yea I didn't know about that channel. | 13:40 |
| fungi | (openstack is sort of a "customer" or "user" of the opendev collaboratory, but opendev doesn't make policy decisions for openstack) | 13:40 |
| fungi | infra-root: since 903805 merged yesterday, i'm taking gitea-lb02 and zuul-lb01 out of the ansible disable list | 13:46 |
| fungi | and done | 13:46 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Re-add py38 to requirements-constraints job https://review.opendev.org/c/openstack/project-config/+/904222 | 15:59 |
| clarkb | I might have to declare bankruptcy on most things. I am curious to know if there is a tldr on the ems discussion | 16:55 |
| clarkb | if not I can find time to pull up the meeting logs. I'm just trying to be lazy :) | 16:56 |
| fungi | clarkb: no firm decision yet, but sounds like we'd prefer to see if the foundation can find budget to cover the higher cost for 2024 | 17:00 |
| clarkb | thanks | 17:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade to etherpad 1.9.5 https://review.opendev.org/c/opendev/system-config/+/904231 | 17:05 |
| clarkb | I don't think I have the ability to test and drive that to completion right now. But having a change up serves as a good reminder during a time period where I'm definitely going to need reminders for stuff | 17:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 17:19 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to 1.21.3 https://review.opendev.org/c/opendev/system-config/+/904233 | 17:19 |
| clarkb | I have rotated out the autohold for ^ | 17:20 |
| clarkb | again I'm not quite sure what state I'm in to drive things, but trying to keep a backlog somewhat fresh that I can come back to eventually | 17:20 |
| clayg | I'm getting an error like `error: remote unpack failed: error Missing tree cc67304289881ff49676288b8f555723f6f48cf7` trying to run git review - curious if any one has seen this before | 17:53 |
| clarkb | clayg: yes it is likely an issue between pack negotation in c git and j git | 17:54 |
| clarkb | clayg: there is a flag you can pass to git-review to work around it let me find it | 17:55 |
| clarkb | clayg: try `git-review --no-thin` this tells git under the surface to not try and optimize the negotation of packs and just do the whole thing | 17:56 |
| clarkb | its quite a bit more expensive so should only be used when you hit this particular problem and rely on the default of negotiation otherwise | 17:56 |
| clayg | yes that works | 17:56 |
| clayg | thank you | 17:56 |
| clarkb | you're welcome | 17:57 |
| clarkb | unfortuantely the origianl issue in launchpad for this is from like 2012 and upstream knows about it but says it is difficult to solve (for reasons I don't fully comprehend) | 17:58 |
| opendevreview | Matthew Thode proposed openstack/diskimage-builder master: gentoo: don't uninstall packages that aren't installed https://review.opendev.org/c/openstack/diskimage-builder/+/904236 | 18:57 |
| Clark[m] | https://158.69.66.126:3081/opendev/system-config is the newly held gitea node | 19:14 |
| *** blarnath is now known as d34dh0r53 | 19:24 | |
| *** elodilles is now known as elodilles_pto | 19:24 | |
| opendevreview | Matthew Thode proposed openstack/diskimage-builder master: gentoo: don't uninstall packages that aren't installed https://review.opendev.org/c/openstack/diskimage-builder/+/904236 | 21:35 |
| opendevreview | Matthew Thode proposed openstack/diskimage-builder master: gentoo: don't uninstall packages that aren't installed https://review.opendev.org/c/openstack/diskimage-builder/+/904236 | 21:38 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add san support to growvols https://review.opendev.org/c/openstack/diskimage-builder/+/903265 | 21:51 |
| tonyb | "it should be ok now for the 3.0-dev. It will be backported as usual with other fixes. Thanks for your help !". sounds like haproxy issue is fixed | 22:04 |
| opendevreview | Ruslan Aliev proposed openstack/project-config master: Add 2 new projects to the airship namespace (armada-go & operator) https://review.opendev.org/c/openstack/project-config/+/904239 | 22:47 |
| opendevreview | Ruslan Aliev proposed openstack/project-config master: Add 2 new projects to the airship namespace (armada-go & operator) https://review.opendev.org/c/openstack/project-config/+/904239 | 22:47 |
| opendevreview | Ruslan Aliev proposed openstack/project-config master: Add 2 new projects to the airship namespace (armada-go & operator) https://review.opendev.org/c/openstack/project-config/+/904239 | 22:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!