16:00:21 <gthiemonge> #startmeeting Octavia
16:00:21 <opendevmeet> Meeting started Wed Nov 15 16:00:21 2023 UTC and is due to finish in 60 minutes.  The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:21 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:21 <opendevmeet> The meeting name has been set to 'octavia'
16:00:28 <gthiemonge> o/
16:01:51 <oschwart> o/
16:03:20 <tweining> o/
16:04:05 <gthiemonge> #topic Announcements
16:04:12 <gthiemonge> * 2024.1 Caracal Release Schedule: Caracal-1
16:04:16 <gthiemonge> this week is Caracal-1
16:04:32 <gthiemonge> patches for new releases of octavia-lib and python-octaviaclient have been proposed
16:04:38 <gthiemonge> https://review.opendev.org/c/openstack/releases/+/900760
16:04:42 <opendevmeet> gthiemonge: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
16:04:45 <gthiemonge> oops
16:04:52 <gthiemonge> https://review.opendev.org/c/openstack/releases/+/900777
16:05:32 <gthiemonge> note: the fix in python-octaviaclient was already included in a bugfix release
16:06:30 <tweining> you mean the hsts "fix"
16:06:44 <gthiemonge> yep
16:08:26 <gthiemonge> #topic CI Status
16:08:37 <gthiemonge> FYI we are facing 2 recurring errors in the CI
16:08:52 <gthiemonge> a DB deadlock (due to the rework for sqlalchemy 2)
16:08:57 <gthiemonge> "Potential deadlock in check_quota_met during the first call of a tenant"
16:09:01 <gthiemonge> https://bugs.launchpad.net/octavia/+bug/2038798
16:09:12 <johnsom> o/
16:09:14 <gthiemonge> there are 2 patches that should fix it
16:09:20 <gthiemonge> https://review.opendev.org/c/openstack/octavia/+/899663
16:09:32 <gthiemonge> johnsom: \o
16:09:52 <gthiemonge> the 2nd error is: "greenlet.error: cannot switch to a different thread"
16:09:57 <gthiemonge> https://bugs.launchpad.net/octavia/+bug/2039346
16:10:16 <gthiemonge> there's a thread on the ML about eventlet
16:10:22 <gthiemonge> [all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries
16:10:43 <gthiemonge> if we drop eventlet, it will fix our issue :D
16:11:21 <johnsom> But that will be probably D or E release I would expect.
16:11:34 <johnsom> That is a hard turn for a big ship....
16:12:25 <gthiemonge> johnsom: so maybe we need to report this issue to oslo.messaging
16:12:54 <johnsom> Yeah, I think we might want to pursue fixing oslo messaging
16:13:00 <gthiemonge> johnsom: based on some queries I did in opensearch, it also impacts other projects
16:13:19 <johnsom> Yeah, there are definitely reports from other projects
16:13:28 <gthiemonge> recent reports?
16:15:15 <johnsom> I sent you some links, they start in 2021 and are still open on the eventlet side, but a recent change in oslo messaging seems to have increased it
16:15:24 <johnsom> #link https://github.com/eventlet/eventlet/issues/662
16:15:27 <johnsom> for example
16:15:42 <gthiemonge> oh yeah I saw some old reports but that specific issue was addressed
16:15:43 <johnsom> #link https://github.com/eventlet/eventlet/issues/432
16:16:18 <johnsom> Let me find the oslo messaging patch, give me a minute or two on that
16:16:28 <gthiemonge> becasue that issue reappeared in october 2023
16:17:48 <johnsom> There is this one too:
16:17:51 <johnsom> #link https://github.com/ChameleonCloud/zun/issues/15
16:19:10 <gthiemonge> ack
16:19:15 <johnsom> There is some chatter that this change make the occurrence more often:
16:19:17 <johnsom> #link https://github.com/openstack/oslo.messaging/commit/fd2381c723fe805b17aca1f80bfff4738fbe9628
16:20:42 <johnsom> I think the heartbeat_in_pthread setting should block all import and reference to eventlet in oslo messaging, which is not the case today.
16:20:54 <johnsom> #link https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_utils.py#L25
16:22:00 <gthiemonge> hmm ok
16:22:29 <gthiemonge> unfortunately I don't reproduce it in my env, I ran the tests during 12h, not a single occurrence of the bug
16:23:24 <johnsom> I think it triggers when rabbit has an issue, like a network interruption, but I don't have a reproducer either
16:24:57 <gthiemonge> ok, I will report it to the oslo.messaging launchpad, I'll share the link here
16:25:10 <gthiemonge> maybe my env is not really up-to-date
16:26:06 <johnsom> I saw it in the check gates
16:26:26 <gthiemonge> yeah many times
16:29:07 <gthiemonge> #topic Brief progress reports / bugs needing review
16:29:19 <gthiemonge> there's a new bug reported by skraynev:
16:29:20 <opendevreview> Tom Weininger proposed openstack/octavia master: Clarify certificate related config options  https://review.opendev.org/c/openstack/octavia/+/901017
16:29:26 <gthiemonge> https://bugs.launchpad.net/octavia/+bug/2043582
16:29:43 <gthiemonge> basically Octavia rejects/triggers an error when the user creates a listener with a SSL certificate that has an empty CN field
16:30:05 <johnsom> Yeah, we should go back to the standards and see if having no CN at all is valid. I think not.
16:30:38 <johnsom> Well, it's not CN really, it is the subject field. As there are alternatives to CN
16:31:49 <gthiemonge> https://security.stackexchange.com/questions/55414/is-the-common-name-mandatory-for-digital-certificates
16:32:16 <opendevreview> Tom Weininger proposed openstack/octavia master: Clarify certificate related config options  https://review.opendev.org/c/openstack/octavia/+/901017
16:33:08 <johnsom> It looks like there are just two RFCs to parse through
16:33:28 <johnsom> I can do some digging on this to confirm/deny
16:33:40 <gthiemonge> johnsom: thanks
16:33:46 <johnsom> That said, I have mentioned before, that cert_parser code is not good
16:35:01 <gthiemonge> yeah I remember that
16:35:43 <tweining> I've also been working on setting up certificates and https://review.opendev.org/c/openstack/octavia/+/901017 is the result of some of the trouble I had
16:36:11 <tweining> one mistake was probably that I looked at octavia's config options around certs instead of refering to the guide
16:36:46 <johnsom> Yeah, focus on the guide, it is the only source of truth. lol
16:37:19 <tweining> anyway, gthiemonge and I had a discussion about the agent_server_* options in https://review.opendev.org/c/openstack/octavia/+/901017/7/octavia/common/config.py
16:38:01 <tweining> it seems the path is already hardcoded in https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/tasks/compute_tasks.py#L195-L198 so changing the config option would break Octavia?!
16:38:42 <gthiemonge> IMHO it's purely internal, admins should not change the setting
16:39:47 <gthiemonge> we should even not mention it in the doc
16:39:50 <johnsom> Correct, the ONLY reason those are even in the config definition is we don't have a separate code base for the agent, so it shares the config infrastructure with the controller side. Those are not configurable by admins
16:40:12 <johnsom> We didn't until the docs were changed and the /etc/octavia.conf was removed.
16:40:26 <gthiemonge> :D
16:40:30 <gthiemonge> https://opendev.org/openstack/octavia/src/branch/master/octavia/opts.py#L30
16:40:33 <tweining> if it really cannot be changed without breaking octavia, we could simply remove it IMO
16:41:02 <gthiemonge> if we remove this line, they will not be included in the doc, but I don't know the other impacts of this change
16:41:45 <tweining> I guess we don't have to decide it now, but we can discuss it in my patch
16:41:58 <gthiemonge> yep
16:42:09 <tweining> the other question I have is about the guide https://docs.openstack.org/octavia/latest/admin/guides/certificates.html
16:42:10 <johnsom> We can break the agent out into it's own repository....
16:42:42 <gthiemonge> or split config.py into 2 files
16:43:01 <johnsom> oslo cfg is a global
16:43:09 <gthiemonge> argh
16:43:33 <tweining> there are the setting [certificates].ca_certificate and [haproxy_amphora].server_ca that get set to the same path. If it's the same setting, why are there two config options?
16:44:04 <johnsom> See, the conf that get created automatically and dropped in the config drive sets those. the agent has to share the cfg with the controllers, so it's listed here.
16:44:34 <johnsom> What line in the guide tweining?
16:44:53 <gthiemonge> tweining: in devstack it's not the same path
16:44:58 <tweining> Under "Configuring Octavia"
16:45:14 <tweining> point 2 and 4
16:45:30 <gthiemonge> https://opendev.org/openstack/octavia/src/branch/master/devstack/plugin.sh#L432-L437
16:46:10 <tweining> ah, ok. thanks.
16:47:01 <johnsom> Yeah, in dual CA (don't look at deployment tools like tripleo that do it completely wrong) those are different files
16:47:58 <johnsom> Oh, no, those are the same file, just used in different places
16:48:11 <johnsom> Ugh, I haven't looked at this in a bit.
16:48:24 <johnsom> The guide is right
16:48:58 <tweining> I guess it is fine. It's just something I noticed an wondered why it is the same path in the guide.
16:50:16 <johnsom> Right, I remember now, the haproxy_amphora server_ca may need to include intermediate certs, where the usage in [certificates] would not
16:50:34 <tweining> one other thing that surprised me a bit was that the defaults for the cert and key path is in https://opendev.org/openstack/octavia/src/branch/master/octavia/certificates/common/local.py /etc/ssl/certs and not /etc/octavia/certs
16:51:21 <tweining> but in my case I intend to override the default using the environment variable
16:52:19 <tweining> anyway, that is all from me for today I think.
16:52:21 <johnsom> I think those are only used (if they are) in tests
16:52:23 <gthiemonge> no idea on this
16:52:33 <QG> Hi ! i have a question about a patch i did : https://review.opendev.org/c/openstack/octavia/+/898803,
16:52:33 <QG> when I run the unit test alone, I don't reproduce the prb  ( octavia.tests.functional.api.v2.test_load_balancer.TestLoadBalancer.test_create_with_vip_subnet_fills_network )
16:53:02 <johnsom> snakeoil certs are created by the OS at install time. They are bogus self-signed (but valid if you ignore the self signed part)
16:53:30 <johnsom> My wonder is if those globals are even used anywhere
16:53:31 <gthiemonge> QG: you're talking about this error? https://zuul.opendev.org/t/openstack/build/e7e27ff8071140e5a142a5eadd4eee6f
16:54:02 <gthiemonge> QG: I'll take a look
16:54:09 <QG> gthiemonge : Yes !
16:54:10 <gthiemonge> there are some other weird errors
16:54:22 <gthiemonge> {"faultcode": "Client", "faultstring": "Invalid input for field/attribute vip_network_id. Value: \'<MagicMock name=\'get_network().id\' id=\'139946822340080\'>\'. Value should be UUID format", "debuginfo": null}'
16:54:44 <gthiemonge> I'm surprised that you don't have it in your nev
16:54:48 <gthiemonge> env
16:55:42 <gthiemonge> anyway, I'll run the tests tomorrow morning
16:55:54 <QG> gthiemonge: ok thanks !
16:56:08 <gthiemonge> folks, do you have any other topics for today?
16:56:11 <oschwart> I wanted to ask quick question about
16:56:13 <oschwart> https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/893066
16:56:23 <oschwart> if it is too late we can talk about it tomorrow
16:56:34 <gthiemonge> you have 4 min oschwart!
16:56:40 <oschwart> haha
16:56:42 <oschwart> :o
16:56:43 <oschwart> After the latest review, I removed the additional config option here, and after the check pipepline failed, I remembered why we had them in the first place :)
16:56:51 <oschwart> as stable branches don't have the noop certificate manager
16:57:08 <gthiemonge> right
16:57:29 <gthiemonge> I remember that
16:57:34 <oschwart> should I revert the patch to have the noop cert option too?
16:57:53 <johnsom> So either backport the noop cert manager or gate the tests on the API version
16:58:36 <gthiemonge> johnsom: you don't agree with this new config option?
16:59:01 <johnsom> As I commented, it's really duplicate
16:59:35 <gthiemonge> FYI I have another meeting in 1min
16:59:47 <oschwart> let's talk about it tomorrow then
16:59:49 <johnsom> It seems we could backport it
16:59:56 <gthiemonge> let's close the meeting, I'll check wiht oschwart
17:00:02 <tweining> +1
17:00:08 <gthiemonge> thank you!
17:00:08 <oschwart> thanks for the input all
17:00:12 <gthiemonge> #endmeeting