14:01:45 <whoami-rajat> #startmeeting cinder
14:01:45 <opendevmeet> Meeting started Wed Jan 24 14:01:45 2024 UTC and is due to finish in 60 minutes.  The chair is whoami-rajat. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:45 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:45 <opendevmeet> The meeting name has been set to 'cinder'
14:01:48 <whoami-rajat> #topic roll call
14:01:52 <rosmaita> o/
14:01:53 <jungleboyj> o/
14:01:54 <simondodsley> o/
14:02:08 <akawai> o/
14:02:17 <inori> o/
14:02:42 <gireesh> hi
14:03:14 <Saikumar> o/
14:03:14 <msaravan> Hi
14:03:25 <whoami-rajat> #link https://etherpad.opendev.org/p/cinder-caracal-meetings
14:03:52 <crohmann> o/
14:04:57 <whoami-rajat> let's get started
14:05:02 <whoami-rajat> #topic announcements
14:05:04 <whoami-rajat> first, Dalmatian release schedule
14:05:10 <whoami-rajat> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/KS2WP242E4EQZ2GANROQMS4OHPQQ3ZJZ/
14:05:19 <whoami-rajat> #link https://review.opendev.org/c/openstack/releases/+/906050
14:06:27 <jayaanand> o/
14:06:36 <whoami-rajat> the D release schedule is proposed, If you are interested you can take a look and plan accordingly
14:06:56 <whoami-rajat> next, RDO at CentOS Connect and FOSDEM
14:07:05 <whoami-rajat> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/JVBVRUVZDEGQYSIBWVWQI4CBOYKHNQ3U/
14:07:16 <whoami-rajat> Opportunities to learn more about RDO in centos connect and FOSDEM in Brussels, BE
14:07:20 <whoami-rajat> centos connect: January 31st and February 1st
14:07:21 <whoami-rajat> FOSDEM: February 2nd and 3rd
14:07:46 <whoami-rajat> this is in person event so people who are attending can keep a look out for RDO related sessions if you are eager to learn more about it
14:08:46 <whoami-rajat> finally we have some upcoming milestones
14:08:47 <whoami-rajat> Non-client library freeze:     February 22nd, 2024 (R-6 week)
14:08:47 <whoami-rajat> Client library freeze:         February 29th, 2024 (R-5 week)
14:08:47 <whoami-rajat> Caracal-3 milestone:           February 29th, 2024 (R-5 week)
14:08:47 <whoami-rajat> 2024.1 Caracal final release:  April 3rd, 2024
14:08:47 <whoami-rajat> 2024.2 'D' virtual PTG ( https://openinfra.dev/ptg/ ): April 8th-12th, 2024
14:10:14 <whoami-rajat> that's all for announcements
14:10:21 <whoami-rajat> there are some cinder related questions on the ML
14:10:28 <whoami-rajat> which to my understanding were answered
14:11:03 <whoami-rajat> but good to see people raising deployment specific issues and other members helping them out
14:12:42 <rosmaita> \o/
14:13:01 <whoami-rajat> anyone has anything else for announcements ?
14:16:08 <whoami-rajat> looks like not
14:16:15 <whoami-rajat> let's get to the topics
14:16:29 <whoami-rajat> #topic Ceph caps for Cinder / Glance
14:16:42 <whoami-rajat> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5SVYDYOXWMG4TJKWEA6BFMPZZGC3Q5CS/
14:17:38 <whoami-rajat> I've been busy for the past couple of days but i will make a note to reply to it today
14:17:51 <whoami-rajat> and please ping me tomorrow if i don't
14:17:54 <whoami-rajat> sorry for the delay here
14:18:16 <crohmann> thanks a bunch whoami-rajat! I will do the patches, but I'd rather have the proper caps down before I even push anything
14:18:46 <whoami-rajat> I can check and provide the info about cinder caps, but do we have info from glance and nova team about their caps?
14:19:06 <whoami-rajat> or you are planning to just do a cinder specific doc effort
14:19:25 <crohmann> no. I'd rather have them all down. For operators it's just "a list of users + caps"
14:20:32 <whoami-rajat> okay, i will add a reply from cinder side on it (and the team can correct me if something is not accurate)
14:20:55 <crohmann> Could you maybe drag someone from Glance / Nova into the conversation on the ML to have this done all in one go?
14:21:24 <crohmann> not one reply ... I mean in one effort / a few days.
14:22:38 <rosmaita> crohmann: i will put something on the glance agenda, though i think the meeting may be cancelled this week
14:22:40 <whoami-rajat> I can mention to ask the teams for their input and the benefit it will have on operators
14:23:38 <rosmaita> crohmann: it would probably help to raise a bug and add all appropriate projects to it
14:24:06 <crohmann> I will do that then. whoami-rajat do you want to hold off your reply until there is a bug then or use the ML firsT?
14:24:17 <whoami-rajat> crohmann, sure, let me know once it's filed
14:24:38 <crohmann> k.
14:25:02 <whoami-rajat> thanks crohmann for keeping on top of it, anything else on this topic?
14:25:50 <crohmann> not frome me.
14:26:07 <whoami-rajat> okay, so we can move to the next topic
14:26:10 <whoami-rajat> #topic Any update on the performance issues with cinder-backup and the chunked driver (e.g. S3)
14:26:17 <whoami-rajat> #link https://bugs.launchpad.net/cinder/+bug/1918119
14:27:11 <whoami-rajat> crohmann, that's you again
14:27:13 <crohmann> This is me again asking for a follow up / current state.
14:27:35 <whoami-rajat> i don't like to ping zaitcev again and again but only he can answer it
14:28:06 <crohmann> I'd like to know if this is treated as a real issue (S3 not being usably fast for production).
14:28:25 <rosmaita> well, it's not not being treated as a real issue
14:28:56 <rosmaita> pete is working on a few different backup items, and one of them has caused some slowdown due to database differences
14:29:22 <rosmaita> so it's still on his radar, just probably not much progress this week
14:29:51 <whoami-rajat> I haven't heard a lot of people use S3 for backups, most of the user survey answers mentions Ceph but happy to be corrected about it
14:29:53 <rosmaita> he's in utc-6 time zone
14:29:56 <crohmann> I did not mean to put this on him personally. The question was more of general nature about getting cinder-backup using the chunked driver to a different performance level
14:30:35 <crohmann> whoami-rajat: Yeah. But if you look at the referenced but, Cern wanted to use S3, but it's simply too slow due to implementation inefficiencies.
14:30:41 <crohmann> s/but/bug/
14:30:52 <crohmann> They even spoke about this in their talk ;-)
14:31:22 <rosmaita> crohmann: slow because of the chunked driver, or slow on the S3 side?
14:31:23 <crohmann> And it's not just "S3", but all non-rbd drivers that base on the chunked driver
14:32:20 <crohmann> Since rbd is just a "rbd export | rbd import" with no data hitting Python at all it's  a whole different ball game
14:33:17 <crohmann> the "chunked driver" on the other hand does actively read "chunks" from the source volume and mangles them until they end up being uploaded to some (remote) storage. Be it S3, GCP or others
14:33:25 <crohmann> s/GCP/GCS/
14:34:13 <crohmann> see https://bugs.launchpad.net/cinder/+bug/1918119 for some of the analysis we did to narrow down where the time is spent
14:34:25 <whoami-rajat> so the improvement is not specific to S3 but any driver inheriting from chunked driver, this makes the effort more generalized
14:34:52 <crohmann> Yes whoami-rajat
14:35:27 <zaitcev> The performance of the chunkdriver is in my QC goals and literally is my only over-arching bug. So it's either that or my neck.
14:35:31 <crohmann> The S3 driver is simply the last bit of the processing (the upload bit) to remote storage. The heavy-lifing was already done to the data at that point
14:36:04 <rosmaita> sounds like zaitcev is motivated to work on this!
14:36:29 <crohmann> Yeah. I did not mean to pull on the grass to make it grow any faster (as they say in German).
14:37:03 <rosmaita> i have not heard that expression before, but i like it!
14:37:31 <crohmann> In any case - zaitcev please let me know if you require any more input, testing, ideas ...
14:37:43 <crohmann> That's all I wanted to discuss regarding this topic.
14:37:43 <zaitcev> crohmann: sure, thanks
14:37:57 <simondodsley> new Cinder motto - we don't pull on the grass...
14:38:03 <zaitcev> Speaking of backup, we need to drum up +2 for https://review.opendev.org/c/openstack/cinder/+/886584
14:38:07 <jungleboyj> LOL
14:38:56 <crohmann> (Yeah. I put that one and some more cinder-backup related quick fixed in the review list of this meeting)
14:39:06 <zaitcev> oh
14:39:22 <rosmaita> i will look at 886584 today
14:40:15 <whoami-rajat> i think i reviewed it earlier, will take a look at the update
14:40:44 <whoami-rajat> okay, anything else on this topic?
14:42:32 <whoami-rajat> i will take the silence as no, let's move on to the next one then
14:42:37 <whoami-rajat> #topic some CI updates
14:42:39 <whoami-rajat> rosmaita, that's you
14:42:45 <rosmaita> hello
14:43:12 <rosmaita> current word on the street is that CI jobs have having odd timeouts
14:43:30 <rosmaita> i guess one theory is that swapping is causing slowdowns
14:43:55 <rosmaita> so, a patch merged recently to enable zswap on devstack runs
14:44:08 <rosmaita> #link https://review.opendev.org/c/openstack/devstack/+/890693
14:44:35 <rosmaita> (that patch had a typo in the flag name, so there's a followup in the gate now: https://review.opendev.org/c/openstack/devstack/+/906504 )
14:44:50 <rosmaita> i think the plan is that nova will enable this on a few jobs to see what happens
14:45:11 <rosmaita> i don't know if we want to get in on that action or not?
14:45:34 <rosmaita> there are some limitations noted on the original patch's commit message
14:46:03 <whoami-rajat> if it can make the situation worse, then maybe better to wait for nova to publish their findings
14:46:05 <rosmaita> anyway, just wanted to bring this up in case anyone is interested in this
14:46:20 <whoami-rajat> if it can only improve the situation, i don't have any issues
14:46:42 <whoami-rajat> but I need to do a bit of reading on the concepts used here
14:46:47 <simondodsley> seems to me zswap could only make things better
14:47:39 <rosmaita> well, it's a bit weird because we're using RAM to store swap pages that are swapped out because of lack of RAM, it's just that they're compressed
14:48:05 <rosmaita> so i think it's going to depend on what the pages are like, i.e., how compressible they are
14:48:13 <rosmaita> but maybe i am being too naive
14:48:19 <simondodsley> the nice thing about ZSWAP is that it can dynamically shrink when not needed, unlike SWAP
14:48:52 <whoami-rajat> zswap basically trades CPU cycles
14:48:53 <whoami-rajat> for potentially reduced swap I/O.  This trade-off can also result in a
14:48:53 <whoami-rajat> significant performance improvement if reads from the compressed cache are
14:48:53 <whoami-rajat> faster than reads from a swap device.
14:49:24 <whoami-rajat> that's what i got from the docs
14:49:39 <rosmaita> it will be interesting to see what happens
14:50:01 <whoami-rajat> i don't know how to determine the performance of reading from compressed cache
14:50:04 <whoami-rajat> but let's see
14:50:25 <rosmaita> clarkb has said that a lot of time is wasted in swap preparation (writing out zeros), so this should be faster
14:51:55 <rosmaita> anyway, that's all from me
14:52:56 <whoami-rajat> thanks for bringing this up rosmaita , we can keep an eye on it for any improvements in the gate situation (we surely need it)
14:53:17 <whoami-rajat> that's all the topics we had for today
14:53:32 <whoami-rajat> let's move to open discussion
14:53:36 <whoami-rajat> #topic open discussion
14:54:02 <inori> Hi, I'd like to highlight a patch I've been working on: https://review.opendev.org/c/openstack/cinder/+/896077
14:54:44 <inori> It recieved a +2 from whoami-rajat last week, I'm now seeking additional reviews from other core reviewers to merge it. Your input would be highly valuable.
14:56:31 <inori> So could you please take a moment to review the patch at your earliest convenience?
14:57:38 <whoami-rajat> it's a Fujitsu feature sitting for a long time, good to get that in
14:57:43 <whoami-rajat> s/for/from
14:58:19 <rosmaita> i'll take a look, though i share zaitcev's surprise that CLI gives you a speedup
14:58:36 <rosmaita> but if that's what Fujitsu is seeing in their testing, what the heck
14:58:42 <inori> Thx for your time.
14:58:58 <rosmaita> we value empirical evidence in the cinder project
15:00:12 <zaitcev> I'll look at it again.
15:00:35 <inori> thank you zaitcev
15:00:48 <whoami-rajat> i didn't see any performance numbers on the patch but if Fujitsu claims it for their driver, i can believe that
15:00:53 <whoami-rajat> okay we are out of time
15:01:01 <whoami-rajat> please take a look at the review request section
15:01:06 <whoami-rajat> thanks everyone for joining
15:01:07 <whoami-rajat> #endmeeting