16:00:11 <gibi> #startmeeting nova
16:00:16 <openstack> Meeting started Thu Apr  2 16:00:11 2020 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:19 <openstack> The meeting name has been set to 'nova'
16:01:12 <artom> o/ (sorta - it's also lunch time, so I'm feeding the kiddos)
16:01:13 <bauzas> \o
16:01:17 <gmann> o/
16:01:21 <bauzas> (it's beer time but I can wait)
16:01:27 <_erlon_> \O
16:01:30 <gibi> o/
16:01:37 <elod> o/
16:01:57 <dansmith> o/
16:02:02 <gibi> bauzas: I understand your pain as we are in the same timezone so I will be quick
16:02:11 <gibi> #topic Last meeting
16:02:20 <gibi> #link Minutes from last meeting: http://eavesdrop.openstack.org/meetings/nova/2020/nova.2020-03-26-16.00.log.html
16:02:28 <gibi> Plans for a virtual PTG is still up on the ML #link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013555.html
16:02:36 <gibi> Two company rule are less strict now as per #link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013553.html
16:02:43 <gibi> We have two new cores lyarwood and gmann. Welcome! \o/
16:03:00 <gibi> anything we have to discuss about these topics?
16:03:04 <gmann> thanks everyone.
16:04:12 <gibi> #topic Bugs (stuck/critical)
16:04:19 <gibi> No Critical bugs
16:04:30 <gibi> somebody added the following to the wiki without a nick
16:04:36 <gibi> OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause [1]
16:04:45 <bauzas> gibi: FWIW, there was a meeting from the Foundation about any plans for the Virtual PTG
16:04:48 <rambo_li> yeah ,it is me
16:04:55 <bauzas> but i can discuss this during the open discussion
16:05:10 <gibi> bauzas: ack, let's do discuss that in the open
16:05:17 <rambo_li> https://bugs.launchpad.net/neutron/+bug/1815989
16:05:19 <openstack> Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney)
16:05:19 <rambo_li> How can we solve this bug thoroughly?And it has serious impact on the OpenStack environment.
16:05:20 <gibi> rambo_li: floor is yours
16:05:54 <gibi> rambo_li, sean-k-mooney: do you see a need for a change in nova?
16:07:11 <sean-k-mooney> gibi: i have not looked at this is some time but if i remeber correctly this is related to the use of multple port binding and some nova neutron interactions
16:07:16 <rambo_li> I want to know if there is  someone following up, such an sean
16:07:57 <sean-k-mooney> https://review.opendev.org/#/c/602432/
16:08:08 <sean-k-mooney> i think that is require among other things to make it work
16:08:14 <sean-k-mooney> well to fix the issue
16:08:31 <sean-k-mooney> the proablem is as long as libvirt is pluging the interface we can not fix this
16:09:10 <sean-k-mooney> so we need to delegate pluging in all cases to os-vif to fix it which that does but we need a neutron fix as well before that can proceed
16:09:20 <sean-k-mooney> which is https://review.opendev.org/#/c/640258
16:09:45 <sean-k-mooney> i have not been working on this since last cycle
16:09:58 <gibi> sean-k-mooney: thanks
16:10:17 <gibi> rambo_li: do you or somebody on your side can help with pushing the patches forward?
16:11:21 <gibi> rambo_li: I also suggest to ping ralonsoh from neutron about the neutron side of the fix
16:11:58 <sean-k-mooney> clip notes version is libvirt add the port to ovs, then there is a short race between neutorn wiring it up and qemu sending the RARP
16:12:03 <rambo_li> I agree the ning yao in this comment .so how can we solve it thoroughly. Is that possible to activate the port binding before the vm shutting down on the source host and vm being running on the destination host?
16:12:18 <sean-k-mooney> rambo_li: no wont work
16:12:26 <sean-k-mooney> rambo_li: libvirt deletes the prot and recreates it
16:12:30 <ralonsoh> I'll sync up with sean-k-mooney after the meeting
16:12:38 <sean-k-mooney> which mean that neutron needs to set it up again
16:12:44 <sean-k-mooney> but yes we can talk after
16:13:13 <rambo_li> yeah , we can talk it after
16:13:39 <gibi> ralonsoh, sean-k-mooney, rambo_li: thanks. let me know if I can help somehow
16:14:00 <gibi> still about bugs
16:14:01 <gibi> #link 108 new untriaged bugs (+6 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
16:14:09 <gibi> so we are over 100 again :/
16:14:29 <gibi> still I haven't had time to go and triage some
16:14:37 <gibi> anyhow moving forward
16:14:42 <gibi> #topic Gate status
16:14:46 <gibi> melwitt, lyarwood, gmann: how is the gate doing recently?
16:15:22 <gibi> I filed a functinal test stability bug and pushed a fix recently https://bugs.launchpad.net/nova/+bug/1870385
16:15:24 <openstack> Launchpad bug 1870385 in OpenStack Compute (nova) "AcceleratorServerTest.test_create_server_with_error fails intermittently" [Medium,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)
16:16:15 <gmann> i did not see any blocking but i was busy on policy things to finish so not aware if any bug is happening frequently (but not blocking).
16:16:46 <gibi> gmann: thanks
16:16:57 <gibi> #topic Release Planning
16:17:02 <gibi> This week we have "Final release for non-client libraries"
16:17:13 <gibi> I think we are ready. os-vif did not need a new release since 2.0.0
16:17:19 <gibi> I talked with tetsuro, he handles os-resource-classes and os-traits
16:17:39 <gibi> One week until Milestone 3
16:17:54 <gibi> which means Feature freeze is next week
16:17:59 <gibi> from the 30 approved BPs for Ussuri 10 have already been implemented(+2 since last week) \o/
16:18:04 <gibi> There are 9 open BPs still targeted to ussuri-3 #link https://launchpad.net/nova/+milestone/ussuri-3
16:18:23 <gibi> any question about the FF?
16:18:48 <artom> Any differences/plans around FFEs this cycle?
16:19:01 <gibi> I don't think there is any difference
16:19:02 <artom> Or ask as usual for things that almost made it and are worht it?
16:19:07 <gmann> gibi: 9th April is FF right?
16:19:15 <sean-k-mooney> gibi: the cybrog blueprint is more or less complete although there are some follow up patches that would still be good to merge
16:20:00 <bauzas> I wouldn't argue for my single side, but I think that in this period, an exceptional FFE period could help a bit
16:20:02 <gibi> artom: yes. post an FFe mail to the ML if you need one
16:20:15 <gibi> gmann: yes, Apr 9 EOB if the FF
16:20:23 <artom> gibi, not me personally, I got nothing this cycle - but I'm aware of at least 1 that'll probably happen, so wanted to make sure
16:20:24 <gmann> ok, thanks
16:20:25 <bauzas> one week left indeed
16:21:09 <gibi> sean-k-mooney: I see some doc patch open but that can go in after FF
16:21:21 <gibi> sean-k-mooney: do you want to push the evac fix for Ussuri?
16:21:42 <sean-k-mooney> am if i can get it done before FF it would be nice to have
16:21:56 <sean-k-mooney> so we have at least one move operation but lets cross that bridge next week
16:22:07 <gibi> sean-k-mooney: sure, if it is ready I think me and dansmith can push it through
16:22:36 <sean-k-mooney> i need to finish refactoring it and testing it but i hope to have a version for ye to review tomrrow
16:23:05 <gibi> bauzas: we have to produce RC1 on the week of 20th of April so we have 3 weeks between FF and RC1
16:23:19 <bauzas> I know
16:23:27 <gibi> so I don't see too much possibility to extend FF (except the usual FFe request)
16:23:32 <bauzas> and I know it will stretch things if we go with exceptions
16:23:58 <bauzas> I'd have appreciated if the TC would have changed a bit of release window, but that's not the case
16:24:07 <bauzas> anyway, not a big case on my side like I said
16:24:32 <gibi> OK, anything else about FF?
16:25:06 <gibi> M3 also means Final release for client libraries
16:25:14 <gibi> we have novaclient patches open to support the newly merged microversions, 2.83 and 2.84
16:25:22 <gibi> #link https://review.opendev.org/#/c/713089
16:25:28 <gibi> #link https://review.opendev.org/#/c/714561
16:25:37 <gibi> I left feedback on 2.83
16:25:42 <artom> Have we started requiring osc patches as well, btw?
16:25:47 <gibi> 2.84 is trivial just queued up behind 2.83
16:25:53 <artom> Otherwise the novaclient-osc gap will just get larger and larger
16:25:57 <gibi> artom: good question
16:26:01 <artom> I guess we can save that for open discussion?
16:26:19 <gibi> artom: I have nothing else now for the release topic so lets discuss osc
16:26:36 <bauzas> gibi: okay, I'll review those novaclient patches urgentlyu
16:26:53 <gibi> honestly I don't know if we have a written rule for requiring osc patches
16:26:59 <bauzas> we don't (yet)
16:27:14 <artom> Or I guess osdk? It's the new official thing, right?
16:27:35 <gibi> artom: I feel you know more about osc and sdk than me :)
16:27:46 <artom> gibi, if I do, it's only barely ;)
16:28:03 <artom> FWIW, I think we should make it compulsory, like we do for novaclient
16:28:06 <bauzas> it's just some companies recommend this tool in their documentation but not novaclient
16:28:10 <gibi> anyhow time is sort until the last osc release for ussuri so let's try to make a decision for V cycle about such requirement
16:28:22 <artom> We can ask mordred for the right place to put the new code, but we can't continue just adding stuff to novaclient
16:28:39 <artom> Otherwise, as I said, the gap will grow and grow and grow
16:29:05 <artom> But yeah, V is a good time to start :)
16:29:11 <gibi> artom: if we add the bindig code to novaclient then osc can consume that still?
16:29:31 <bauzas> we discussed it at a couple of PTGs
16:29:35 <bauzas> the main issue wasn't the design
16:29:36 <artom> gibi, not sure - IIUC it still needs to be wired up
16:29:48 <sean-k-mooney> gibi: it could but its kind of going againt the dirction of osc swapping to the sdk
16:29:56 <bauzas> the main issue was that nobody was signing off for it
16:30:07 <gibi> OK, so this is a topic for the PTG
16:30:19 <artom> sean-k-mooney, yeah, I wasn't sure of what the new hotness was
16:30:37 <artom> Substitute "official openstack client program" for osc :)
16:30:42 <gibi> artom, bauzas, sean-k-mooney: does some of you can add the topic to the etherpad with as many background as you have?
16:30:54 <bauzas> I can, my tab is open
16:30:55 <artom> gibi, I brought it up, I'll do it
16:31:00 <bauzas> oh cool
16:31:02 <gibi> https://etherpad.openstack.org/p/nova-victoria-ptg
16:31:03 <artom> bauzas, ok, unjinx! It's all yours
16:31:06 <artom> Dammit, too late!
16:31:08 <artom> ;)
16:31:39 <gmann> FYI. Virtual PTG planning meeting is on April 6th and 7th one.
16:31:48 <gmann> about dates and all
16:32:11 <bauzas> gmann: I followed one this afternoon my time
16:32:20 <bauzas> there are notes already
16:32:24 <gmann> bauzas: great
16:32:30 <bauzas> but I promised to discuss it during the opens
16:32:34 <gibi> yepp
16:32:38 <gibi> so moving forward
16:32:40 <gibi> #topic Stable Branches
16:32:41 <gmann> +1
16:32:44 <gibi> I see fairly short review backlogs on both train and stein
16:32:50 <gibi> lyarwood is off this week. elod: anything newsworthy on stable?
16:33:05 <elod> nothing special,
16:33:18 <elod> short backlogs as you wrote
16:34:06 <gibi> thanks
16:34:08 <gibi> #topic Sub/related team Highlights
16:34:15 <gibi> Placement (tetsuro)
16:34:43 <gibi> API (gmann)
16:35:00 <gibi> gmann: anything besides the progressing policy work?
16:35:04 <gmann> pushing the policy stuff in speedy way
16:35:20 <gmann> other than that nothing specific to mention.
16:35:25 <gibi> gmann: thanks
16:35:37 <gibi> #topic Stuck Reviews
16:35:41 <gibi> nothing on the agenda
16:36:03 <gibi> do you have anything to discuss here?
16:36:52 <gibi> #topic Open discussion
16:36:54 <gibi> New Scheduler Affinity/Anti-affinity Filter (erlon)
16:37:02 <gibi> _erlon_: your turn
16:37:21 <_erlon_> hey guys
16:38:02 <_erlon_> so, we are planning to add a Affinity/Anti-affinity filter to Nova, and want to get some feedback
16:38:22 <artom> Different from what we already have?
16:38:32 <_erlon_> artom: yes
16:38:52 <sean-k-mooney> _erlon_: what is the goal and how does it differ form the existing filter/weigher we have for hard and soft affintiy
16:39:04 <bauzas> I think it could be discussed in a spec
16:39:29 <_erlon_> artom: so, the use case is that, we can want to create a bunch of VMs and want Nova to spread them  across multiple AZs
16:39:39 <sean-k-mooney> well it will need one but i was hoping for a clip notes version asn tehre are issues with palacement when it comes to affinity and anti afinity
16:39:55 <_erlon_> bauzas: that was my next question, if we needed that to be on a spec
16:40:18 <dansmith> that would go against our current AZ behavior, outside even the control of a filter, IMHO
16:40:26 <_erlon_> sean-k-mooney: clip note version?
16:40:27 <bauzas> what dansmith said
16:40:28 <dansmith> that'd be the job for an external tool, IMHO
16:40:39 <bauzas> but again, I'd prefer to discuss this in a spec
16:40:46 <sean-k-mooney> _erlon_: you answered it you want to do AZ anti affintiy
16:41:00 <_erlon_> dansmith: what outside the ontrol of a filter? the host group affinity does exacly that
16:41:04 <sean-k-mooney> clip notes version is the short version :)
16:41:21 <_erlon_> got it
16:41:39 <sean-k-mooney> _erlon_: well for a start it would only work if you did not have an az in the boot request.
16:42:12 <dansmith> sean-k-mooney: but then we assign the default AZ if you don't,
16:42:13 <bauzas> I feel they want host aggregate anti-affinity
16:42:14 <_erlon_> sean-k-mooney: yes, i think that should be a limitation
16:42:16 <sean-k-mooney> _erlon_: this is specvicly for nova multi create api right e.g. boot 5 servers with one request
16:42:18 <dansmith> so the filter would need to override that behavior I think
16:42:40 <bauzas> and not server anti-affinity
16:43:06 <bauzas> but either way, drafting it on the fly seems difficult as we at least need to understand what problem we have to solve
16:43:08 <sean-k-mooney> dansmith: which would be too late if we have dont the az->host aggrate to placement aggreaet mapping already
16:43:31 <bauzas> _erlon_: do you feel okay with writing a spec and mainly explaining *what* you want to solve ?
16:43:38 <sean-k-mooney> _erlon_: have you filed a blueprint to track this feature?
16:43:51 <bauzas> _erlon_: forget first about the solution and explain what's missing for your needs
16:44:00 <bauzas> in the nova toolket
16:44:01 <_erlon_> sean-k-mooney: not necessarily, we tryied using the same group affinity for hosts. So, we create a group, set its affinity, and all the isntances created in that group will land/or not in the same or different AZ
16:44:04 <gibi> bauzas:  +1
16:44:19 <bauzas> and maybe we could come up with a solution for you that wouldn't require some new implementation work
16:44:31 <dansmith> sean-k-mooney: I'd have to look at the order of where that gets set and assumed, but I'm skeptical about doing this inside nova in general
16:44:36 <bauzas> _erlon_: in general, that's what happens with filters
16:44:50 <_erlon_> bauzas: so, are you faminilar witth the host affinity groups?
16:44:59 <bauzas> most of the times, the current state is flexible enough to address lots of needs
16:45:15 <gibi> _erlon_: as bauzas suggested, please write a short spec focusing on the use case and let's continue the discussion in that spec.
16:45:15 <bauzas> _erlon_: spec up, and then ping me
16:45:23 <sean-k-mooney> dansmith: ya i was more thinking it might have to be a pre request filter at a minium as a normal filter would be too late.
16:45:33 <_erlon_> gibi: bauzas: sounds good
16:45:35 <sean-k-mooney> still intersted in understand the usecase in anycase
16:45:52 <gibi> OK, going back to virtual PTG plannign
16:45:52 <_erlon_> the other questions
16:45:57 <gibi> nvm
16:46:00 <gibi> _erlon_: yes?
16:46:02 <bauzas> _erlon_: and if you already have code up, good news! you can just use your own filters out-of-tree without loosing time to convince ourselves :p
16:46:04 <_erlon_> I didnt see any spec freeze in the nova schedule
16:46:10 <_erlon_> do you guys have it?
16:46:20 <sean-k-mooney> its noramlly milestone 2
16:46:23 <gibi> _erlon_: Ussuri spec freeze are in the past
16:46:29 <sean-k-mooney> although we have change it in the past to be milestone 1
16:46:32 <bauzas> we no longer do spec freeze
16:46:45 <bauzas> but yeah m-2
16:46:50 <bauzas> actually, yeah we do
16:47:02 <gibi> V cycle schedule #link https://releases.openstack.org/victoria/schedule.html
16:47:03 <_erlon_> ok
16:47:25 <gibi> M2 is end of july
16:47:28 <sean-k-mooney> _erlon_: you have about 2-4 months before the deadline
16:47:39 <sean-k-mooney> ya
16:47:40 <_erlon_> sean-k-mooney: ok
16:47:43 <bauzas> again, focus on the problem
16:47:58 <bauzas> explain your usecase
16:48:18 <_erlon_> about using it of tree, we have hit problems in the past as we have to manually alter the distro package
16:48:31 <_erlon_> bauzas: sure, makes sense
16:48:44 <sean-k-mooney> _erlon_: im guesin you are using AZs to model fault domains and you want to magage the scudling accordingly to acchive ha
16:49:00 <bauzas> _erlon_: FWIW, you can just package your single filter separately, it'll work fine
16:49:05 <bauzas> no need to alter anything
16:49:09 <sean-k-mooney> but i think we can likely move on for now if there are other topics
16:49:13 <bauzas> ++
16:49:30 <gibi> let's continue this on #openstack-nova as I'd like to go back to the virtual PTG thing for a bit
16:49:44 <gibi> bauzas: so you already heard some info from the fundation today
16:50:01 <bauzas> no precise info, it was a discussion
16:50:10 <_erlon_> bauzas: we only manage to make nova to see it by changing the eggs
16:50:11 <bauzas> and I raised the concerns I had
16:50:28 <bauzas> _erlon_: let's continue this over -nova please
16:50:32 <gibi> bauzas: cool. I will try to read up on that discussion and see how it relates to my current plans
16:50:44 <_erlon_> but anywasy, Ill publish a spec later guys. thanks a lot for now
16:50:50 <gibi> _erlon_: thanks
16:51:01 <bauzas> #link https://etherpad.openstack.org/p/Virtual_PTG_Planning
16:51:18 <bauzas> tl;dr: the PTG would be on 2 weeks (or more)
16:52:01 <bauzas> we could potentially have two distinct periods, one for cross-project discussion, the other being the project team discussions
16:52:09 <bauzas> overlapping sessions potentially
16:52:40 <bauzas> and maybe large time slots in the day to allow contributors from different timezones to join
16:52:54 <bauzas> but the main thing is : there will be two more sessions to attend
16:53:01 <bauzas> (if you bear using Zoom)
16:53:11 <bauzas> dates are on the etherpad
16:53:19 <gibi> April 6th at 17 UTC
16:53:23 <bauzas> so, like any other business, if you care, you dare join
16:53:24 <gibi> April 7th at 2 UTC
16:53:34 <gibi> bauzas: thanks for the summary
16:53:36 <bauzas> that's it from me
16:53:43 <sean-k-mooney> ideally i think we should try to organsise our own seesion slots within that period to avoid overlap were we can
16:54:08 <sean-k-mooney> not that i expect we would all attend every session. we dont even do that in person
16:54:20 <sean-k-mooney> but jsut to avoid overlap where it not needed
16:54:27 <gibi> sean-k-mooney: yeah, I feel organizing a real time session per topic for important topics are easier than have an umbrella slot
16:54:37 <bauzas> sean-k-mooney: ideally I'd appreciate the Foundation to back us
16:54:55 <gibi> bauzas: back us with what?
16:55:04 <bauzas> by providing mission letters to our management, asking them to not bear us during the timeslots
16:55:14 <gibi> bauzas: I see
16:55:15 <sean-k-mooney> well i dont think its very different then how we normally manage our time at the ptg
16:55:22 <bauzas> that's why defining some specific period of time could help
16:55:29 <bauzas> sean-k-mooney: yes and no
16:55:49 <bauzas> not paying attention to the internal channels during the usual PTG definitely helps
16:55:51 <gibi> bauzas: good point. I have to block out all my downstream meeting for that two weeks
16:56:08 <gibi> which will be fun :)
16:56:25 <bauzas> gibi: that's why the Foundation wants the dates to be well communicated
16:56:27 <bauzas> and understood
16:56:54 <sean-k-mooney> if we can arrange some buckets of time that suite the majority of peopel, with foundation help if possibel we can then schdule within those slots as needed.
16:57:03 <artom> bauzas, tbf, I don't think RH management will be a problem :) Other companies I have no idea about
16:57:23 <sean-k-mooney> ya that true
16:57:36 <sean-k-mooney> if i was at the ptg i would not be attending the meetings that week anyway
16:58:06 <gibi> anything else? we have 2 minutes
16:58:06 <bauzas> artom: sure, but I don't speak here for my single company :)
16:58:24 <bauzas> although some escalations could occur...
16:58:53 <artom> bauzas, right, but that's implicitly understood to be top priority at all times
16:59:01 <artom> Anyways
16:59:14 <artom> FWIW, RH hosted a community event around running virtual meetups
16:59:21 <artom> I took some notes, with some maybe interesting ideas
16:59:24 <artom> I'll send that to the ML
16:59:35 <gibi> artom: thanks!
16:59:46 <gibi> ... 20 seconds
16:59:52 <sean-k-mooney> im not really planning to attend the virtual ptg planning metting
17:00:01 <sean-k-mooney> artom bauzas can you update us next week
17:00:02 <gibi> thank your for joining. o/
17:00:09 <gibi> #endmeeting