14:00:13 <rosmaita> #startmeeting cinder
14:00:13 <opendevmeet> Meeting started Wed Mar 23 14:00:13 2022 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:13 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:13 <opendevmeet> The meeting name has been set to 'cinder'
14:00:19 <rosmaita> #topic roll call
14:00:22 <whoami-rajat> Hi
14:00:31 <simondodsley> Hi
14:00:48 <fabiooliveira> o/
14:02:00 <eharney> hi
14:02:21 <rosmaita> hello everyone
14:02:23 <jungleboyj> o/
14:02:30 <rosmaita> #link https://etherpad.opendev.org/p/cinder-yoga-meetings
14:02:36 <rosmaita> let's get started
14:02:42 <rosmaita> #topic announcements
14:02:45 <enriquetaso> hi
14:03:04 <rosmaita> European Summer Time starts 27 March, so this meeting may be 1 hour later for you next week
14:03:16 <rosmaita> and, next week is the final Yoga cycle meeting!
14:03:29 <andrebeltrami[m]> hello
14:03:31 <rosmaita> and, as the last meeting of the month, it will be in video
14:03:49 <rosmaita> usual connection info will be on the agenda etherpad
14:04:02 <rosmaita> the week after that is the Zed (virtual) PTG
14:04:16 <rosmaita> don't forget to register for the PTG if you haven't yet
14:04:23 <rosmaita> info is on the planning etherpad:
14:04:26 <rosmaita> #link https://etherpad.opendev.org/p/zed-ptg-cinder-planning
14:04:47 <rosmaita> there are a lot of topics proposed, so whoami-rajat will have his hands full doing the planning!
14:05:09 <whoami-rajat> isn't this meeting as per UTC timing? I never faced a time change during DST or not
14:05:39 <whoami-rajat> oh or is it just for US Europe region
14:05:41 <rosmaita> whoami-rajat: yes, we keep it in UTC, so it's only a relative change for some people
14:05:53 <whoami-rajat> ack got it
14:06:00 <rosmaita> yeah, i just wanted to give a heads up to Europe region folks
14:06:22 <rosmaita> tbh, i am not sure if European Summer Time is observed everywhere in Europe or not
14:06:23 <whoami-rajat> re:  whoami-rajat will have his hands full doing the planning! -- but that shouldn't stop you from adding topics!
14:06:48 <whoami-rajat> ok
14:06:56 <enriquetaso> whoami-rajat++
14:07:38 <rosmaita> I am thinking that if next week's agenda is light (like today's is), we can do a cycle retrospective at next week's meeting
14:07:49 <rosmaita> or not
14:08:19 <rosmaita> i will think about it, if anyone has an opinion on that, let me know
14:08:28 <whoami-rajat> sounds like a good idea
14:08:31 <rosmaita> ok, final announcement:
14:08:44 <rosmaita> Final Yoga Release Candidate must be cut *tomorrow*
14:08:47 <jungleboyj> Never a bad idea.
14:09:25 <rosmaita> ok, i will put retrospective on the agenda
14:09:47 <rosmaita> i will probably resurrect the retrospective surveys i posted a few cycles ago
14:09:54 <rosmaita> to prime the pump, as it were
14:10:25 <rosmaita> ok, so let's talk about the final yoga release candidate
14:10:33 <rosmaita> #topic final yoga release candidate
14:10:43 <rosmaita> #link https://etherpad.opendev.org/p/cinder-yoga-release-critical-changes
14:11:23 <rosmaita> ok, the first item is the prohibit managing to an encrypted volume type
14:11:34 <rosmaita> jungleboyj: could you renew your +2 on https://review.opendev.org/c/openstack/cinder/+/834478 ?
14:11:53 <rosmaita> i had a typo in the commit message that was corrected and wiped out your +2
14:11:54 <jungleboyj> Looking.
14:12:16 <rosmaita> and we need another stable core to look at https://review.opendev.org/c/openstack/cinder/+/834478
14:12:44 <rosmaita> the cgroups v1 -> v2 work is delayed
14:12:55 <rosmaita> so i will renew the "known issues" about that one
14:13:06 <jungleboyj> +2 from me renewed.
14:13:18 <rosmaita> thanks!
14:13:48 <rosmaita> at this point, i think the throttling for image downloads is unusable because it depends on cgroups v1, which probably is unsupported in most current operating system versions
14:13:54 <eharney> i'll look at 834478
14:14:03 <rosmaita> thanks eric
14:14:27 <rosmaita> so we will need to merge the fix early (like, in the next 2 weeks or so), and cut a new Yoga release
14:14:35 <rosmaita> after backporting it, of course
14:14:50 <rosmaita> and it will have to be backported to Xena where it was promised to be fixed
14:15:30 <rosmaita> next: nfs encryption fix ... will have to be backported when it's ready in master
14:16:24 <rosmaita> the ceph backup-restore issue: there was a request to make sure it was tested, but I'm not sure anyone had the bandwidth to do it
14:16:30 <rosmaita> #link https://review.opendev.org/c/openstack/cinder/+/750782/
14:17:09 <rosmaita> has anyone had a chance to test it?
14:17:24 <enriquetaso> don't we have a tempest test to try this?
14:18:13 <rosmaita> enriquetaso: i am not sure ... i assume not, because otherwise it would have broken our ceph ci
14:18:28 <eharney> a tempest test that would back up a ceph volume and restore it to iscsi?  seems unlikely since we don't usually have much that goes between different backends
14:19:01 <rosmaita> i think eharney is correct, we are passing because we probably test ceph-to-ceph, which doesn't show the problem
14:19:49 <enriquetaso> oh cool
14:19:50 <eharney> also needs testing around encrypted volumes, to be thorough
14:19:53 <rosmaita> i guess it's ok to hold off on this fix since we are already committing to an early Yoga release for cgroups
14:21:02 <rosmaita> ok, we will not include it tomorrow, but will prioritize working on it over the next 2 weeks
14:21:19 <rosmaita> final issue (not on the etherpad yet)
14:21:23 <rosmaita> is geguileo here?
14:21:40 <rosmaita> i see that he filed a big stack of bugs over NVMeOF
14:21:50 <rosmaita> in enriquetaso's bug report this week
14:22:12 <rosmaita> looks like that connector is broken for encryption (among other things)
14:22:32 <rosmaita> so I will need to write a "known issues" note about that
14:22:46 <rosmaita> i will follow up with geguileo
14:22:54 <tosky> we have tests that check the migration between different backends (if configured), but I doubt we have tests that configures the source and destination backend for a backup
14:23:17 <rosmaita> tosky: thanks, i was thinking of your matrix CI job
14:23:19 <tosky> we would need a multibackend job also (I think enriquetaso  was working on such a job iirc)
14:23:45 <enriquetaso> I think I proposed something but I'm not sure if it's okay https://review.opendev.org/c/openstack/cinder/+/784815
14:24:18 <tosky> or that multibackend job, yes, it could run also an hypotetical test which performs cross-backend backup and restore
14:24:21 <rosmaita> ok, we have something to look at for this
14:24:35 <rosmaita> thanks enriquetaso and tosky
14:25:02 <rosmaita> i am thinking that we have a lot of combinations that could be tested
14:25:34 <rosmaita> we could run them on periodic jobs in zuul so that we don't consume too many resources on each patch
14:25:51 <rosmaita> or only run them on patches that touch the backup service
14:26:21 <rosmaita> or something ... anyway, my point is that it's worth putting this into tempest so that the testing is available
14:26:42 <enriquetaso> ++
14:27:11 <whoami-rajat> I think neutron has modified their CI jobs quite a bit to run them on only the code affecting them, although i haven't checked or have a deeper insight on it
14:27:54 <rosmaita> that's good to know
14:28:09 <rosmaita> i guess we can deal with it when people complain that we are doing too much testing
14:28:11 <rosmaita> :)
14:28:54 <whoami-rajat> :)
14:29:17 <rosmaita> ok, so other than https://review.opendev.org/c/openstack/cinder/+/834478 , it looks like the only outstanding stuff for final RC is release notes
14:29:28 <rosmaita> so i will work on those today and ping people to review
14:29:50 <jungleboyj> ++
14:29:54 <rosmaita> i guess i should also put up a release patch with a -1 so the release team knows we intend to do an RC-2
14:30:06 <rosmaita> i'll do that right after the meeting
14:30:27 <rosmaita> #action rosmaita put up a release patch with a -1 so the release team knows we intend to do an RC-2
14:30:38 <rosmaita> #action rosmaita "known issues" for yoga release
14:31:04 <rosmaita> ok, is there anything else brewing?  anyone working on an absolutely critical bug?
14:33:11 <rosmaita> guess ont
14:33:17 <rosmaita> *not
14:33:39 <rosmaita> #topic open discussion
14:33:47 <enriquetaso> I have a bug that it's not critical but I'd like to mention it here
14:33:50 <rosmaita> ok
14:34:03 <enriquetaso> #link https://bugs.launchpad.net/cinder/+bug/1965847
14:34:10 <enriquetaso> During exporting and exporting a backup record the original backup gets deleted.
14:34:17 <rosmaita> i just left a comment on that one
14:34:30 <rosmaita> https://bugs.launchpad.net/cinder/+bug/1965847/comments/1
14:34:56 <rosmaita> that's definitely a bad bug
14:35:11 <enriquetaso> oh, cool
14:35:14 <enriquetaso> thanks rosmaita++
14:35:20 <rosmaita> don't think it's release critical (though i am happy to consider other opinions)
14:35:27 <rosmaita> fix looks pretty straightforward
14:36:31 <rosmaita> i'm not sure we want to jam through a fix and backport it in the next 24 hours, though
14:36:49 <enriquetaso> i don't think so
14:36:58 <rosmaita> so the question is, should we mention this as a known issue?
14:37:22 <jungleboyj> Yuck.  I think it would be a good idea to mention.
14:37:52 <eharney> yeah, seems worthwhile to document
14:38:15 <rosmaita> guess it won't hurt ... though in that situation, presumably if you make the call to import a second time, it should work because now the record is deleted
14:38:43 <rosmaita> ok, i will add it to knows issues
14:39:14 <rosmaita> in the meantime, it's worth looking at where this happens in the code from an engineering perspective
14:39:39 <rosmaita> nested try statements, where the outer one catches Exception ... it is a code smell
14:41:01 <whoami-rajat> I've a query regarding one multiattach test i proposed, I'm doing a dd on the block device from /dev/urandom but when trying to read it, it fails
14:41:10 <whoami-rajat> if anyone has time to take a look on it https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/791921
14:41:56 <rosmaita> interesting
14:42:03 <eharney> fails when reading the data from the block device to verify it?
14:42:28 <whoami-rajat> eharney, yes
14:43:09 <whoami-rajat> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 2: invalid start byte
14:43:11 <whoami-rajat> this is the error
14:44:00 <eharney> probably trying to convert bytes to str in a place it shouldn't
14:44:11 <rosmaita> maybe you want to explicitly read in binary mode?
14:44:38 <rosmaita> what eharney said .. make sure you are reading bytes and not strings
14:45:14 <rosmaita> that will definitely be a nice test to have
14:45:16 <whoami-rajat> yeah i think tempest is decoding the data somewhere it shouldn't, i will look for an option to skip it/read bytes
14:45:24 <eharney> i suspect it's the ssh transport
14:45:25 <whoami-rajat> thanks rosmaita eharney for the help
14:45:41 <eharney> maybe instead of shuffling the data blocks over ssh, just pipe the read into sha1sum instead and check that
14:46:00 <eharney> (i'd have to look deeper to see if that's really the issue)\
14:47:06 <whoami-rajat> i guess you're right, the ssh module in tempest is doing the decode
14:47:06 <whoami-rajat> File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 225, in exec_command
14:47:07 <whoami-rajat> out_data = out_data.decode(encoding)
14:47:44 <whoami-rajat> here's the test failure if you would like to see the whole traceback
14:47:44 <whoami-rajat> https://fe21a6496ae4e9bbcecb-5ff3dddcf26e50a416e8b904201353d9.ssl.cf1.rackcdn.com/791921/7/check/cinder-tempest-plugin-lvm-multiattach/35a57de/testr_results.html
14:48:26 <eharney> yeah it's probably only setup to interact w/ text output and not binary data, so using a checksum seems like a good fix
14:48:43 <whoami-rajat> ack, will try that, thanks
14:49:40 <rosmaita> that was productive! anything else for open discussion?
14:52:46 <rosmaita> easy docs patch for someone to review: https://review.opendev.org/c/openstack/cinder/+/834875
14:56:19 <jungleboyj> rosmaita:  I will look at that now.
14:56:47 <rosmaita> thanks!
14:56:57 <rosmaita> ok, guess that's all for today
14:57:05 <rosmaita> thanks everyone! see you next week
14:57:18 <enriquetaso> thanks\o
14:57:36 <whoami-rajat> thanks!
14:57:38 <jungleboyj> Thank you!
14:58:22 <rosmaita> #endmeeting