14:00:40 <whoami-rajat> #startmeeting cinder
14:00:40 <opendevmeet> Meeting started Wed Aug 24 14:00:40 2022 UTC and is due to finish in 60 minutes.  The chair is whoami-rajat. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:40 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:40 <opendevmeet> The meeting name has been set to 'cinder'
14:00:48 <geguileo> hi! o/
14:00:51 <eharney> hi
14:00:55 <whoami-rajat> #topic roll call
14:01:33 <jungleboyj> o/ kind of ... multi-tasking.
14:01:37 <rosmaita> o/
14:01:46 <enriquetaso> hi
14:01:56 <felipe_rodrigues> hi
14:02:01 <HappyStacker> hi
14:02:35 <whoami-rajat> #link https://etherpad.openstack.org/p/cinder-zed-meetings
14:02:46 <HelenaDantas[m]> o/
14:03:02 <luizsantos[m]> o/
14:03:04 <raffaelacunha[m]> o/
14:04:28 <whoami-rajat> good turnout today
14:04:30 <whoami-rajat> let's get started
14:04:35 <whoami-rajat> #topic announcements
14:04:49 <whoami-rajat> i think the first two announcements are by rosmaita , would you like to do it?
14:05:42 <rosmaita> sure
14:05:51 <whoami-rajat> great
14:06:15 <rosmaita> as agreed last week, we proposed to EOL cinderlib on stable/train and stable/ussuri
14:06:29 <rosmaita> #link https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030013.html
14:06:47 <rosmaita> the release team is OK with the patches, but have put a hold to give people time to respond
14:06:56 <rosmaita> haven't heard anything in a week so far
14:07:26 <rosmaita> what will happen is that they will tag the branches with 'train-eol' or 'ussuri-eol'
14:07:31 <rosmaita> and then delete the branches
14:07:40 <rosmaita> so if you need them, you can always checkout the tag
14:07:51 <rosmaita> next item
14:08:12 <rosmaita> we agreed at the midcycle that third-party CI should run on all os-brick changes
14:08:25 <rosmaita> so there's an announcement to the ML:
14:08:36 <rosmaita> #link https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030014.html
14:08:49 <rosmaita> no responses to that one either
14:09:08 <rosmaita> hopefully the driver maintainers have seen it
14:09:15 <whoami-rajat> to all vendors in the meeting ^
14:09:30 <rosmaita> yes, we should point out to any new people
14:09:38 <geguileo> rosmaita: I realized something we'll have to merge and backport in Cinder for cinderlib...
14:10:04 <geguileo> rosmaita: because afaik the primary user of cinderlib is oVirt to attach RBD volumes, and with the removal of the keyring, we broke it
14:10:20 <geguileo> rosmaita: I have a patch to fix this in Cinder (will only work with Cinderlib)
14:10:26 <geguileo> so no security issue for Cinder
14:10:32 <geguileo> https://review.opendev.org/c/openstack/cinder/+/846054
14:10:36 <rosmaita> i thought we had merged that, or am i thinking of something else?
14:10:44 <geguileo> it's now failing on mypy, but it should be ok otherwise
14:10:59 <geguileo> rosmaita: I thought we had already merged it as well
14:11:16 <geguileo> rosmaita: but it turns out we didn't, so iirc right now cinderlib is broken for RBD
14:11:40 <geguileo> rosmaita: oooooh, wait, wait, we merged that
14:11:59 <geguileo> rosmaita: this is a second patch, for when the config is not under /etc/ceph (which is usually the case in cinderlib)
14:12:06 <rosmaita> ah, ok
14:12:17 <geguileo> anyway, just wanted to give the fyi on that one
14:12:39 <geguileo> we can continue, and once CI says it's ok I'll go around with the usual begging for reviews
14:12:42 <rosmaita> guess we need to prioritize getting cinderlib master open for zed development
14:12:46 <rosmaita> ok
14:12:58 <rosmaita> final thing i wanted to say to driver maintainers
14:13:03 <rosmaita> you may know this already
14:13:35 <rosmaita> but when we send stuff to the openstack-discuss mailing list about the cinder project, we put '[cinder]' in the subject line so you can filter easily
14:13:58 <rosmaita> for stuff that impacts driver maintainers, we also put '[cinder][drivers]' in the subject line
14:14:06 <rosmaita> so you should *really* pay attention to those
14:14:10 <whoami-rajat> rosmaita, would be good to get a current status on cinderlib Zed development (maybe later), I'm kind of not updated there
14:14:32 <rosmaita> ok, let's see if there's time during open discussion
14:14:40 <whoami-rajat> ack
14:14:51 <rosmaita> (i am not completely updated myself, tbh)
14:15:27 <rosmaita> though now i remember, so let's talk later
14:15:33 <whoami-rajat> I'm sure more updated than me !
14:15:38 <geguileo> whoami-rajat: the tldr; is that it's broken
14:15:38 <whoami-rajat> sounds good
14:16:03 <whoami-rajat> geguileo, :( so it's in the state i last remember
14:16:17 <geguileo> whoami-rajat: yeah, I've been a bit busy with the unit tests mess
14:16:30 <rosmaita> busy indeed!
14:16:33 * whoami-rajat will pay more attention to cinderlib after M-3
14:17:01 <whoami-rajat> don't forget quotas!
14:17:17 <geguileo> whoami-rajat: oh I haven't!!
14:17:37 <whoami-rajat> :D
14:17:41 <rosmaita> geguileo has nightmares about quotas
14:17:42 <geguileo> (I have nightmares with it now)
14:17:47 <rosmaita> :)
14:17:50 <geguileo> rosmaita: you know me so well!!
14:18:27 <whoami-rajat> so we can discuss about cinderlib if we get time after the topics
14:18:58 <whoami-rajat> rosmaita, anything else from the first two announcements?
14:19:06 <whoami-rajat> s/from/for
14:20:18 <rosmaita> nope, i think that's all
14:20:44 <whoami-rajat> ok, thanks for the announcements
14:20:47 <whoami-rajat> next, os-brick release this week
14:20:58 <whoami-rajat> we've the os-brick release deadline tomorrow
14:21:11 <whoami-rajat> I've created an etherpad from which most of the changes have merged
14:21:16 <whoami-rajat> #link https://etherpad.opendev.org/p/zed-os-brick-patches
14:21:26 <whoami-rajat> I've left a -1 on the release patch for the meantime
14:21:46 <whoami-rajat> not all the patches are release critical but good to have them merged
14:22:08 <geguileo> this one would be good to have merged: https://review.opendev.org/c/openstack/os-brick/+/854281
14:22:29 <geguileo> that together with a cinder patch can prevent unit tests not returning to the shell
14:22:51 <whoami-rajat> geguileo, yep, i was reviewing it but the meeting started, will finish it after the meeting
14:22:56 <whoami-rajat> that sounds great!
14:22:59 <geguileo> thanks
14:23:00 <rosmaita> we should probably merge it, we're supposed to be supporting py 3.9, and that seems to be where it happens
14:23:16 <geguileo> rosmaita: this is a different blocking  ;-)
14:23:28 <geguileo> there are 2 kinds of blockings    rofl
14:23:42 <geguileo> because, why not!
14:24:06 <geguileo> this is a weird interaction of the garbage collector calling the close method of the wrapper
14:24:09 <rosmaita> geguileo is laughing because otherwise he would be crying
14:24:26 <geguileo> rosmaita: yup
14:25:06 <whoami-rajat> there is also a topic dedicated to UTs by geguileo later during the meeting
14:25:42 <whoami-rajat> next, Antelope PTG Registration
14:26:06 <whoami-rajat> by this time everyone might be aware that the Antelope PTG in october is going to be virtual
14:26:25 <whoami-rajat> but some people might not have registered before thinking it's in person and forgot about it later
14:26:40 <whoami-rajat> so for a reminder, please register for the upcoming PTG, it's virtual and free
14:26:41 <whoami-rajat> #link https://www.eventbrite.com/e/project-teams-gathering-october-2022-tickets-374118026087
14:27:55 <whoami-rajat> that's all for announcements
14:28:15 <whoami-rajat> let's move on to topics
14:28:28 <whoami-rajat> #topic Our Unit Tests are a broken clock
14:28:31 <whoami-rajat> nice heading
14:28:34 <whoami-rajat> geguileo, that's you
14:28:44 <geguileo> thanks
14:28:59 <geguileo> so basically it's by pure luck that our unit tests run successfully
14:29:07 <geguileo> we have all kicks of madness going on there
14:29:21 <geguileo> and it's easy to see by running tox -epy39 -- --random
14:29:40 <geguileo> if you can run that 3 times without a failure, you should play the lottery, because your luck is out of this world
14:30:02 <whoami-rajat> lol
14:30:06 <geguileo> besides the 10 to 15 seconds tests we have that abishop is working on
14:30:35 <geguileo> I have found that our test runs may just get stuck
14:30:48 <abishop> I think my patches to eliminate timed delays all merged
14:31:09 <geguileo> abishop: didn't know if you had more you were working on or not, thanks
14:31:21 <geguileo> today I hit the blocking issue at the gate https://zuul.opendev.org/t/openstack/build/73706405b8cb472982f4f09b056d3d09
14:31:31 <geguileo> that's on py38
14:31:40 <geguileo> and it shows as the CI job timing out
14:32:06 <geguileo> I have a whole series of patches (16 or so) fixing unit test issues
14:32:13 <geguileo> links on the etherpad
14:32:41 <geguileo> I just want to point out 4 things here for reviewers and coders
14:33:26 <geguileo> - Look at the results of the unit tests jobs when new UTs are being added to confirm there are no >1sec new test
14:34:05 <geguileo> - Make sure that data in 'ddt.data(...)' are static values, otherwise those tests are not executed
14:34:19 <geguileo> example: https://review.opendev.org/c/openstack/cinder/+/854262)
14:34:22 <enriquetaso> oh
14:35:05 <geguileo> - Use self.mock_object and self.patch and NEVER, EVER replace things with something.another = mock.Mock()
14:35:25 <geguileo> Because when that happens on a library you'll break tests that come afterwards
14:36:10 <geguileo> - Preferrably don't use global variables (or class attributes) with data that is going to be used in the tests unless they are not going to be modified
14:36:34 <geguileo> otherwise even if you usually use copy and deepcopy, you will eventually forget somewhere and make your tests flaky
14:36:43 <geguileo> as success will depend on the order in which they are run
14:36:59 <geguileo> A lot of examples of these last 2 in the patches I have proposed
14:37:23 <geguileo> One of the patches proposes adding  --random to how we run stestr
14:37:33 <geguileo> to try to detect these sooner
14:38:22 <geguileo> oh, and for anyone that uses 'stestr run --load-list XYZ' to reproduce a gate job tests in the same order
14:38:30 <geguileo> they should know that that is currently broken in stestr
14:39:14 <geguileo> I have proposed a patch to fix it: https://github.com/mtreinish/stestr/pull/331
14:39:50 <geguileo> and that's all I wanted to complain about
14:40:33 <whoami-rajat> there are a series of patches on the etherpad that geguileo mentioned fixes our UTs so please review
14:40:37 <rosmaita> i don't think it counts as complaining when you have posted all those patches fixing stuff
14:41:14 <whoami-rajat> rosmaita, +1
14:41:21 <whoami-rajat> thanks for all the work on this geguileo !
14:41:41 <geguileo> fingers cross that those are the only big issues for a while
14:42:11 <geguileo> s/cross/crossed
14:43:06 <tosky> geguileo: just one question: stestr is pinned in requirements.git for each release
14:43:34 <geguileo> tosky: I don't know WHEN they broke that feature
14:43:54 <tosky> so what does it mean for older releases? I guess the unit test fixes are still valid to be backported, but the older branches won't benefit much
14:44:01 <tosky> or at least partially
14:44:17 <geguileo> in my opinion for older branches I wouldn't backport the fixes
14:44:25 <geguileo> unless they are problematic for another backport
14:44:26 <tosky> uhm, the "when it broke" is a good question, I hope stestr maintainers could help
14:44:42 <geguileo> tosky: well, the feature that broke is not related to all those fixes
14:44:57 <geguileo> tosky: I think I wasn't clear on what is broken in stestr
14:45:17 <geguileo> so when you run tox for unit tests, or when it runs at the gate and it fails
14:45:23 <geguileo> but you run it locally and it doesn't
14:45:34 <geguileo> it's usually because tests are executed in a different order
14:45:56 <geguileo> so it's a problem with some test that is not mocking things properly and breaks later tests
14:46:10 <geguileo> the way to reproduce this locally is not download the subunit file from the gate
14:46:21 <geguileo> see the worker number that has failed in the logs
14:46:35 <geguileo> extract that worker tests into a file
14:46:38 <tosky> oh, but then wouldn't it make sense to backport them then?
14:46:43 <geguileo> and tell stestr to load that file and run it in that order
14:47:05 <geguileo> and that is the part that's broken in stestr, that it doesn't preserve the order of the list
14:47:15 <geguileo> so you can't reproduce the issues at the gate
14:47:32 <geguileo> so it doesn't affect our gates
14:47:40 <geguileo> (the stestr bug)
14:47:51 <geguileo> only our avility to reproduce the issues to fix things
14:48:15 <eharney> i suspect we'll want to backport some, like https://review.opendev.org/c/openstack/cinder/+/854259/
14:48:38 <geguileo> eharney: yeah, probably the generic 3 generic ones
14:48:39 <eharney> many of them probably could be ignored for backport until obviously needed, as they've sat and run in the gates for years, mostly working
14:48:42 <geguileo> (first 3)
14:48:54 <geguileo> eharney: +1
14:49:05 <geguileo> if it's not broken, don't touch it  ;-)
14:52:19 <geguileo> if there's nothing else, we can continue or finish
14:52:40 <whoami-rajat> we've open discussion :)
14:52:44 <whoami-rajat> so that's all the topics had for today, let's move to open discussion
14:52:48 <whoami-rajat> #topic open discussion
14:53:03 <whoami-rajat> one thing i wanted to mention, we've the feature freeze next week so make sure to review the feature patches
14:53:11 <whoami-rajat> #link https://etherpad.opendev.org/p/cinder-zed-features
14:54:16 <whoami-rajat> rosmaita, would you like to provide an update on cinderlib or we can discuss that later?
14:55:00 <rosmaita> real quick, here's what's going on
14:55:23 <rosmaita> current cinderlib master is running gate CI against cinder and os-brick stable/yoga
14:55:51 <rosmaita> the patch to open it to zed development will run it against cinder/os-brick master
14:55:54 <rosmaita> and that is failing
14:56:04 <rosmaita> mostly due to database changes, i believe
14:56:56 <rosmaita> so if you propose a patch to master right now, i think the CI will be green
14:57:06 <rosmaita> but that's not a "real" green CI
14:58:25 <rosmaita> sorry, got distracted for a minute
14:58:56 <rosmaita> this is the patch opening cinderlib for zed development
14:58:59 <rosmaita> #link https://review.opendev.org/c/openstack/cinderlib/+/848846
15:00:13 <rosmaita> anyway, we could merge the rbd keyring fix to master now and backport it as necessary
15:00:31 <rosmaita> it would just be a little weird
15:00:32 <geguileo> rosmaita: oh, that's a cinder patch
15:00:42 <geguileo> but it's for cinderlib
15:00:43 <rosmaita> oh, then there's no problem wiht that
15:00:46 <geguileo> yup
15:00:59 <rosmaita> ok, good, because i wasn't liking what i suggested there
15:01:08 <rosmaita> can we delete the last 10 lines from teh meeting log?
15:01:48 <whoami-rajat> doesn't look doable but we can end meeting quickly
15:02:18 <whoami-rajat> we're already out of time
15:02:22 <rosmaita> yes, please do, before i say anything else stupid
15:02:34 <whoami-rajat> thanks for the update rosmaita
15:02:37 <whoami-rajat> and thanks everyone for attending
15:02:44 <whoami-rajat> #endmeeting