16:00:17 <bauzas> #startmeeting nova
16:00:17 <opendevmeet> Meeting started Tue Nov 21 16:00:17 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:17 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:17 <opendevmeet> The meeting name has been set to 'nova'
16:00:30 <bauzas> hey folks
16:00:34 <bauzas> aloha
16:00:42 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:01 <Uggla_> o/
16:01:13 <grandchild> hi
16:01:27 <bauzas> awaiting a few more people
16:01:42 <gibi> o/
16:01:43 <sean-k-mooney> o/
16:02:07 <elodilles> o/
16:03:03 <bauzas> okay, let's start
16:03:10 <bauzas> #topic Bugs (stuck/critical)
16:03:17 <bauzas> #info No Critical bug
16:03:24 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 37 new untriaged bugs (+5 since the last meeting)
16:03:28 <bauzas> #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:03:36 <bauzas> auniyal6: any bug you wanted to raise ?
16:04:19 <bauzas> looks like he's offline
16:04:25 <bauzas> I'll take the baton this week
16:04:31 <bauzas> #info bug baton is bauzas
16:04:37 <bauzas> moving on
16:04:40 <bauzas> #topic Gate status
16:04:45 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:04:52 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:04:59 <bauzas> we had a fun gate blocker this week
16:05:31 <sean-k-mooney> the tooz thing?
16:05:31 <bauzas> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/L72QU3SR2VFVYOFXVYH74V7HGMQ3YJRU/
16:05:39 <bauzas> yeah
16:05:55 <sean-k-mooney> you know we also use tooz in the ironic virt driver
16:06:00 <bauzas> anyway, we skipped the oslo tooz release
16:06:12 <sean-k-mooney> so its not just used by cinder
16:06:29 <bauzas> so now the tooz community needs to discuss how to support both etcd versions*
16:06:47 <bauzas> sean-k-mooney: sure but the gate was failing due to the cinder API issue
16:07:05 <sean-k-mooney> they could for now just swap to using hte memcached drier instead of ectd
16:07:07 <sean-k-mooney> in the gate
16:07:29 <bauzas> it's no longer a nova problem :)
16:07:42 <sean-k-mooney> :)
16:07:43 <bauzas> anyway
16:08:17 <bauzas> fwiw, now we have back a nova-emulation job issue https://zuul.openstack.org/builds?job_name=nova-emulation&project=openstack/nova
16:08:23 <bauzas> chateaulav: around ?
16:08:55 <bauzas> oh wait
16:08:59 <bauzas> this is not against master
16:10:14 <sean-k-mooney> ya so the emulation job i think is  running on one of the stable branches
16:10:17 <bauzas> anyway, we'll see how this goes next week
16:10:20 <sean-k-mooney> which it should not be
16:10:58 <bauzas> sean-k-mooney: well, it's checking for yoga
16:11:04 <sean-k-mooney> its in check on yoga but it should only be in perodic-weekly
16:11:14 <bauzas> so probably in the next month, shouldn't be a problem :p
16:11:17 <sean-k-mooney> right we just forgot to move it on that branch
16:11:31 <sean-k-mooney> sure
16:11:37 <bauzas> we'll see
16:11:39 <bauzas> moving on
16:12:04 <bauzas> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:12:06 <bauzas> #topic Release Planning
16:12:12 <bauzas> #link https://releases.openstack.org/caracal/schedule.html
16:12:16 <bauzas> #info Nova deadlines currently on review https://review.opendev.org/c/openstack/releases/+/877094
16:12:34 <bauzas> I just hope the release cores will accept it quickly
16:12:54 <sean-k-mooney> its alredy merged
16:13:12 <bauzas> wow
16:13:16 <bauzas> haven't seen it
16:13:18 <sean-k-mooney> was that the right link
16:13:22 <sean-k-mooney> that was for bobcat
16:13:28 <bauzas> doh
16:13:33 <bauzas> #undo
16:13:33 <opendevmeet> Removing item from minutes: #info Nova deadlines currently on review https://review.opendev.org/c/openstack/releases/+/877094
16:13:43 <sean-k-mooney> https://review.opendev.org/c/openstack/releases/+/901547
16:14:10 <bauzas> #info Nova deadlines currently on review https://review.opendev.org/c/openstack/releases/+/901547
16:14:18 <sean-k-mooney> it has a -1 for elod specfreeze is wrong
16:14:19 <bauzas> okay, just saw elodilles's question
16:14:26 <elodilles> o:)
16:14:29 <bauzas> I'll look at it
16:14:42 <sean-k-mooney> elodilles:++ sorry that should read from not for
16:14:51 <bauzas> and yeah you're right
16:15:11 <bauzas> elodilles: indeed was wrong, should be milestone-2 for spec freeze, my bad :)
16:15:20 <elodilles> np :)
16:15:28 <bauzas> I'll upload a new rev
16:15:39 <elodilles> thx
16:16:13 <bauzas> #info Caracal-2 (and spec freeze) milestone in 7 weeks
16:16:41 <bauzas> as a reminder, we'll have another spec review day
16:17:13 <bauzas> now, I have a question
16:17:41 <bauzas> we said necxt week that we could add a caracal-1 tag for nova
16:17:53 <bauzas> but then I looked at all the releases we did from yoga
16:18:01 <bauzas> and none of them have any milestone tag
16:18:10 <bauzas> hence my question
16:18:16 <bauzas> should we create a specific milestone for Nova and Placement?
16:18:19 <bauzas> I don't think so
16:18:21 <sean-k-mooney> we are not required to create them. we are just required to have 1 rlease before the final release
16:18:29 <sean-k-mooney> so the RC-1 release covers that
16:18:31 <bauzas> this is rc1
16:18:36 <bauzas> yeah
16:18:58 <sean-k-mooney> i dont personlly think milestone release make sense for nova
16:18:58 <bauzas> and I very quickly looked at cinder, they don't add a milestone too
16:19:03 <sean-k-mooney> or non lib projects
16:19:07 <bauzas> yup
16:19:11 <elodilles> no need for Caracal-1 (.b01) release (which is not rc1)
16:19:23 <elodilles> for nova & placement
16:19:27 <elodilles> imo
16:19:34 <bauzas> the only usecase would be for operators if they want to test milestones
16:19:52 <elodilles> (it's just a possibility, but we don't need)
16:19:54 <bauzas> but we haven't had any operators doing this from a long time
16:19:59 <bauzas> okay
16:20:01 <sean-k-mooney> they could just deploy a commit at that point
16:20:21 <sean-k-mooney> googing though the release process for that has very little return
16:20:36 <sean-k-mooney> we do it for libs since we do not install libs form git as the default
16:20:40 <bauzas> #action bauzas to not provide a caracal-1 tag actually (compared to what we said on the previous weekly meeting)
16:20:42 <sean-k-mooney> unlike service projects which we do
16:20:49 <bauzas> I'm fine
16:21:00 <bauzas> I just wanted to remove the action item that we agreed last week
16:21:01 <elodilles> +1
16:21:20 <bauzas> I don't know how many people read our meeting minutes but... :)
16:21:33 <bauzas> anyway, moving on
16:21:46 <bauzas> #topic Review priorities
16:21:51 <bauzas> #link https://etherpad.opendev.org/p/nova-caracal-status
16:21:53 <bauzas> hehe
16:22:21 <bauzas> so, folks who want to add some bugfixes can now add them in ^
16:22:41 <bauzas> we have a specific section
16:22:55 <bauzas> and cores, please look at this etherpad in order to know what to review
16:23:05 <gibi> aye sir :)
16:23:34 <bauzas> nay, I'm not a sir :)
16:23:43 <gibi> :)
16:23:43 <bauzas> :D
16:23:51 <bauzas> #topic Stable Branches
16:24:00 <bauzas> elodilles: please
16:24:02 <gibi> monsieur ? :)
16:24:07 <elodilles> :)
16:24:15 <elodilles> #info stable gates don't seem blocked
16:24:25 <elodilles> except now yoga with nova-emulation job :D
16:24:41 <elodilles> which is almost constantly failing as it looks...
16:24:44 <sean-k-mooney> do we want to nuke that
16:24:59 <bauzas> +1
16:25:03 <elodilles> is it needed for yoga?
16:25:04 <sean-k-mooney> we woudl not be running it on unmaintaed anyway
16:25:18 <sean-k-mooney> and if its brokekn im not sure it worth fixing
16:25:19 <bauzas> unless we want to unmaintan yoga
16:25:27 <bauzas> *now* I think
16:25:30 <sean-k-mooney> elodilles: it was ment to be in perodic-weekly
16:25:37 <sean-k-mooney> not in check
16:25:46 <sean-k-mooney> so while it shoudl work its not required
16:25:56 <elodilles> sean-k-mooney: ACK, i'll check and try to remove from the check pipeline then
16:26:25 <bauzas> +1
16:26:28 <elodilles> bauzas: we'll unmaintain yoga soon, but let's fix that for now if that is possible by eliminating that job o:)
16:27:25 <elodilles> it's also needed if we want a final release off yoga before it gets unmaintained
16:27:47 <elodilles> btw, stable releases:
16:27:51 <elodilles> #info stable release patches still open for review: https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova
16:28:17 <elodilles> do we still want to wait for patches?
16:28:30 <elodilles> a couple of stable bug fix have landed meanwhile
16:28:39 <bauzas> I think I clicked +2 on a yoga change
16:29:07 <sean-k-mooney> the one from tobias-urdin
16:29:08 <bauzas> elodilles: what's the deadline ?
16:29:10 <elodilles> (and that failed due to the nova-emulation)
16:29:12 <bauzas> sean-k-mooney: yup
16:29:24 <sean-k-mooney> we should include that in the final release
16:29:51 <elodilles> bauzas: no strict deadline planned, as we (community) are already late, so just ASAP :)
16:30:49 <sean-k-mooney> we can quickly update the jobs after the meeting and then recheck the patch
16:30:54 <elodilles> i mean the EM transition deadline was Nov 2nd, but meanwhile the process is changing and now we need to move to Unmaintained instead
16:31:02 <elodilles> sean-k-mooney: +1
16:31:24 <bauzas> okay
16:31:34 <elodilles> then the general message then:
16:31:35 <bauzas> cool enough
16:31:37 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:31:56 <elodilles> add here stable gate issues if you encounter one ^^^
16:32:09 <elodilles> and that's it i think
16:33:45 <bauzas> ++
16:33:52 <bauzas> moving on then
16:34:06 <bauzas> #topic Open discussion
16:34:09 <bauzas> there was one item
16:34:13 <bauzas> in the agenda
16:34:22 <bauzas> (fwiesel) vmwareapi driver deprecation/removal: SAP wants to  step up provide the resources to keep the driver maintained (CI, tests  coverage, code changes, ...?). what would be needed?
16:34:27 <fwiesel> Hi, that would be me.
16:34:35 <bauzas> yes indeed
16:35:13 <fwiesel> So, essentially me and my colleagues talked to our management, and we have the commitment from them to dedicate 1FTE just for upstream work.
16:35:55 <fwiesel> My understanding is, the most pressing issue would be to have a working stable CI worker for the vmware api, followed by cleanup work, test-coverage, etc...
16:36:10 <fwiesel> Is that correct?
16:36:52 <bauzas> we basically deprecated the virt driver by Bobcat
16:37:04 <sean-k-mooney> and have someone work basically fultime on maintining it
16:37:04 <bauzas> and we explained the reasons in the relnotes
16:37:50 <bauzas> https://docs.openstack.org/releasenotes/nova/2023.2.html#deprecation-notes
16:37:59 <sean-k-mooney> fwiesel: for context this would be the tird time that someone has commited to maintianing that ci and working upstream to maintian the vmware dirver
16:38:00 <bauzas> "The vmwareapi driver is marked as experimental and may be removed in a future release. The driver is not tested by the OpenStack project and does not have a clear maintainer."
16:39:11 <fwiesel> Well, I hope that we can step up to be the clear maintainer. And I hope we are not held responsible for the actions of others.
16:39:14 <sean-k-mooney> we coudl delay the removal for a short period fo time to have you setup the ci but honestly i would prefer to proceed witht he removal and had hoped it woudl have merged already
16:39:33 <bauzas> the problem here is that most of the times, someone pops up when we are about to delete a virt driver, asks us to not remove it, tells us we will have a 3rd-party CI for a long time, but then after a few months, disappears while the CI job has problems
16:39:56 <bauzas> this is really a matter of trusting people and companies
16:40:56 <bauzas> in the maintime, some users think they can use that virt driver and find some problems
16:40:57 <gibi> fwiesel: what woudl be your timeline setting up a the CI?
16:41:10 <opendevreview> Elod Illes proposed openstack/nova stable/yoga: Remove nova-emulation from check pipeline of Yoga  https://review.opendev.org/c/openstack/nova/+/901604
16:41:19 <bauzas> this is why we now say those kind of drivers are experimental
16:42:09 <grandchild> hi, also from SAP here, and we understand the lack of trust. not much we can say to that now, we'll have to show our work first.
16:42:36 <grandchild> the issue is exactly the time frame for us to set up a CI while you (understandably) go ahead with removal.
16:42:44 <fwiesel> Sure, trust must be earned, so we can't expect you do that right away. And I can't expect the driver not to be removed considering current timelines. But hopefully we can find a way forward to re-integrate it then later under some better circumstances.
16:43:06 <bauzas> fwiesel: grandchild: thanks for replying
16:43:11 <bauzas> so, I have a question
16:43:28 <bauzas> say you need some time for running a 3rd-party CI
16:43:36 <bauzas> and you need to reply to gibi on that
16:43:41 <bauzas> then,
16:44:06 <bauzas> how would you be able to provide solutions for the tests that would have problems ?
16:44:15 <bauzas> do you know about the vmwareapi driver?
16:44:41 <bauzas> running a job is possible, making it fiable is more difficult
16:44:44 <grandchild> if you ask about our familiarity with the vmware driver code, i'd say it's fairly hih
16:44:46 <fwiesel> gibi: It is hard for us yet to give a timeline on the CI as we are yet unclear about how much effort it will be for us.
16:44:56 <bauzas> s/fiable/reliable
16:45:03 <gibi> fwiesel: I see
16:45:51 <fwiesel> We maintain a fork here: https://github.com/sapcc/nova
16:46:30 <fwiesel> Mostly because we try to work around issues of scalability in vmware which are probably not of interest to the general community.
16:46:32 <bauzas> fwiesel: maybe if you could tell us how long you would need to find the needed efforts, this would be nice
16:46:47 <bauzas> have you already run tempest against your fork ?
16:47:38 <fwiesel> bauzas: Yes, we run tempest and unit tests. We have disable some tempest tests, I can check which ones.
16:48:35 <sean-k-mooney> fwiesel: what do run them with is it devstack?
16:48:49 <fwiesel> fwiesel: On the effort estimate, I can come back to you in a week.
16:48:51 <bauzas> fwiesel: that's good to hear, that's a first thing
16:48:52 <grandchild> we have a dedicated qa environment
16:49:06 <fwiesel> sean-k-mooney: We run it in qa environments on k8s
16:49:07 <sean-k-mooney> fwiesel: the tird party ci would have to deploy the gerrit chagne under review and report back with tempest test using the vmware driver
16:49:46 <sean-k-mooney> you would not need to use devstack you could use your k8s install tool provided it uses the nova commit form the review
16:49:46 <bauzas> fwiesel: do you already have ideas on how many runs you need to do for nova if you add a 3rd party CI ?
16:49:55 <bauzas> fwiesel: because this can be large
16:50:37 <sean-k-mooney> bauzas: its much less then it used ot be but yes its non trivial
16:50:44 <bauzas> https://zuul.openstack.org/builds?project=openstack%2Fnova&branch=master&pipeline=check&skip=0
16:50:45 <fwiesel> sean-k-mooney: The main concern is here more that we have to run arbitrary code, so we want it a bit more isolated.
16:51:16 <sean-k-mooney> fwiesel: yep form past expeince runing a third party ci for nova/neutron
16:51:24 <sean-k-mooney> geting it approval to run it is non trivial
16:51:30 <bauzas> fwiesel: grandchild: you can see in the above link how many builds we run
16:51:38 <bauzas> for the 'check' pipeline
16:52:33 <bauzas> and we can't unfortunately only test changes that modify the virt driver
16:52:40 <grandchild> (zuul link throws 500 in fetching the jobs ;) )
16:53:17 <bauzas> because we could regress on vmwareapi by a change not related to the driver
16:53:22 <grandchild> but i'd wager, like fwiesel said, isolation is more of an issue than load.
16:53:35 <grandchild> bauzas: sure, we'd run the whole pipeline, we are aware of that
16:53:41 <bauzas> for example, say someone modifies the virt API and forgets to modify vmwareapi module
16:54:06 <bauzas> grandchild: cool, I just wanted to be aware
16:54:07 <sean-k-mooney> grandchild: well you dont need to run the full pipeline
16:54:21 <sean-k-mooney> you only need 1 job that runs on all changes to a subet of direcotries
16:54:52 <sean-k-mooney> ideally any change the touches the compute and  virt modules
16:54:54 <bauzas> sean-k-mooney: that's my point
16:54:58 <sean-k-mooney> and ideally som others
16:54:58 <grandchild> sean-k-mooney: sorry if the terminology is off -- i meant the check for the whole codebase
16:55:13 <bauzas> sean-k-mooney: I don't really know which modules they should only check
16:55:47 <bauzas> but we could try to run something small first
16:55:48 <sean-k-mooney> well ideally every change but at least every chagne to the compute and virt modules
16:55:55 <bauzas> without removing the experimental tag
16:56:09 <bauzas> and see how this goes first
16:56:23 <bauzas> before adding more checks to the 3rd party pipeline
16:56:37 <sean-k-mooney> sof feature freeze is feb 29th
16:56:50 <sean-k-mooney> i had wanted the removals to happen before m1
16:57:07 <sean-k-mooney> i could be convinced to wait till around m2 for ci or very early febuary
16:57:16 <grandchild> that sounds fair actually, 3 months to get something on its feet. might be achievable.
16:57:31 <bauzas> here is what I can propose
16:57:40 <grandchild> but like fwiesel said, a week for an estimate from us.
16:57:42 <gibi> (if we remove the the driver from master the driver is be on the stable branches)
16:57:42 <fwiesel> We could also turn it the other way around. You remove it, and we re-add it, when you are convinced it is fine.
16:57:49 <bauzas> grandchild: fwiesel: would you be able to attend our meeting on a weekly basis ?
16:57:57 <grandchild> bauzas: definitely
16:58:07 <bauzas> okay, here is my proposal then
16:58:09 <fwiesel> fwiesel: Yes
16:58:19 <fwiesel> bauzas: Yes
16:58:24 <bauzas> we could defer the removal until caracal-2
16:58:50 <bauzas> and every week, I'd add a topic in our meeting about your 3rd party CI
16:59:07 <bauzas> you could then tell us where you are and what you need
16:59:51 <bauzas> and if before end of December, we can't really see anything coming, then we would remove the virt driver
17:00:04 <fwiesel> Fair enough.
17:00:05 <grandchild> that sounds fine to me, fwiesel?
17:00:10 <grandchild> k
17:00:19 <bauzas> any other opinions but me ?
17:00:32 <gibi> sound OK to me too
17:00:33 <bauzas> I'm just a code herder, I'm not a L
17:00:44 <sean-k-mooney> i can defer writing the remaining patches until after milestone 2
17:00:53 <sean-k-mooney> i have the patch to remove the driver already done
17:01:13 <sean-k-mooney> but i can wait for the api/object changes until we see if the ci turns up by milestone 2
17:01:18 <bauzas> okay, then I can write the action item
17:01:35 <bauzas> sean-k-mooney: I haven't checked, but surely the main patch isn't merged yet, right ?
17:01:44 <sean-k-mooney> ill -w the removal patch for now
17:01:49 <sean-k-mooney> its not but i wanted it to be
17:01:54 <bauzas> okay, sounds a plan
17:01:58 <sean-k-mooney> i was goign to ask for review today
17:02:09 <bauzas> and we'll see on a weekly basis how this goes
17:02:16 <bauzas> lemme write the action
17:02:53 <bauzas> #agreed we defer the vmwareapi driver removal until caracal-2
17:03:29 <bauzas> #agreed fwiesel and/or grandchild will report on a weekly basis about their efforts on running a 3rd party CI for vmwareapi
17:03:53 <bauzas> #action bauzas to create a meeting topic in order to discuss the efforts
17:03:55 <grandchild> ftr: jkulik is not in the office today, but also in the mix from us :)
17:04:03 <bauzas> sure
17:04:20 <bauzas> sean-k-mooney: I appreciate your understanding about the deferral
17:04:35 <sean-k-mooney> its ok ill ask ye to review other patches instead :P
17:04:38 <grandchild> sean-k-mooney: we do too! thanks :)
17:04:38 <bauzas> grandchild: fwiesel: I aslo appreciate your efforts in trying to bond with us
17:04:53 <bauzas> let it be going
17:05:02 <bauzas> and we'll see during the next weeks how things go
17:05:12 <bauzas> we're waaaay overlate
17:05:15 <sean-k-mooney> before we finish the meeting
17:05:20 <sean-k-mooney> i did have a topic for this week
17:05:22 <bauzas> shoot, but quick
17:05:25 <sean-k-mooney> https://review.opendev.org/c/openstack/nova/+/899381
17:05:36 <sean-k-mooney> basically does that need a specless blueprint or bug
17:06:02 <sean-k-mooney> tl;dr libvirt can now tell use how many sev instnace we can create
17:06:09 <sean-k-mooney> we have a optional config option to do that
17:06:25 <sean-k-mooney> this patch will just auto detect the amount and report it
17:06:43 <sean-k-mooney> it was deferd form the orgianl spec because we need libvirt to be modifed to enable this
17:06:45 <bauzas> I'm fine with approving it as specless but telling them to create a spec if during the implementation reviews we find some design concerns
17:07:02 <sean-k-mooney> well its actully more or less ready to merge
17:07:18 <sean-k-mooney> i have one nit and wanted to answer this question before +2ing it
17:07:21 <bauzas> given the change is already there, it's fine to review it now and ask for a spec if really needed
17:07:26 <sean-k-mooney> and stephenfin previously +2'd it
17:07:31 <bauzas> they wouldn't be impacted by the spec freeze
17:07:35 <bauzas> which is later
17:07:51 <bauzas> any controversial opinions on it being specless ?
17:08:15 <bauzas> looks not
17:08:29 <bauzas> we'll need tkajinam to create a blueprint
17:08:33 <sean-k-mooney> ok lets ask  tkajinam to file a specless blueprint for this then and review it as normal once done
17:08:38 <sean-k-mooney> ya
17:08:42 <sean-k-mooney> ok that was it
17:08:45 <gibi> look OK to me
17:08:52 <bauzas> #agreed https://review.opendev.org/c/openstack/nova/+/899381 could be specless
17:09:21 <bauzas> #action tkajinam to create a blueprint and report it to bauzas so that he can late-approve it given above ^$
17:09:33 <bauzas> okay, we're definitely overlate
17:09:40 <bauzas> really sorry about that
17:09:42 <bauzas> thanks all
17:09:46 <bauzas> #endmeeting