16:00:39 <jungleboyj> #startmeeting cinder
16:00:40 <openstack> Meeting started Wed Jan 10 16:00:39 2018 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:43 <openstack> The meeting name has been set to 'cinder'
16:00:54 <jungleboyj> Courtesy ping:  jungleboyj DuncanT diablo_rojo, diablo_rojo_phon, rajinir tbarron xyang xyang1 e0ne gouthamr thingee erlon patrickeast tommylikehu eharney geguileo smcginnis lhx_ lhx__ aspiers jgriffith moshele hwalsh felipemonteiro
16:01:01 <geguileo> hi!  o/
16:01:02 <tommylikehu> hi
16:01:04 <lhx_> hi all
16:01:08 <pots> o/
16:01:10 <eharney> hi
16:01:10 <erlon> hey
16:01:11 <xyang> hi
16:01:36 <amito> hi
16:01:37 <jungleboyj> Hello all.  Do we have an smcginnis in the house?
16:02:01 <tommylikehu> searching
16:02:12 <jungleboyj> Processing .... Processing.
16:02:33 <erlon> nop
16:03:03 <jungleboyj> Ok, well we will need to get started then as my flight boards in 30 minutes.  :-)
16:03:16 <jungleboyj> I think this should be a short meeting though.
16:03:28 <jungleboyj> #topic announcements
16:03:29 <Swanson> hello
16:03:43 <geguileo> jungleboyj: you had to say it... it will last forever now  };-)
16:03:59 <jungleboyj> So, first, Happy New Year to everyone.  Hope everyone had a great holiday.
16:04:02 <jungleboyj> geguileo:  :-p
16:04:38 * patrickeast wanders in late
16:04:41 <jungleboyj> Wanted to make everyone aware that I released new versions of os-brick at the request of StorPool and Nova.
16:04:59 <e0ne> hi
16:05:01 <jungleboyj> I also did releases for stable/pike and stable/ocata to get some fixes out there.
16:05:41 <jungleboyj> I need to take a look I think cinderclient might need one too.
16:05:47 <jungleboyj> Will look before the end of the week here.
16:06:34 <jungleboyj> Moving on from announcements ...
16:06:38 <e0ne> jungleboyj: I had some issues with cinderclient and keystoneauth today. I'll check it once more tomorroe and let you know
16:06:46 <jungleboyj> #topic Rocky PTG
16:06:53 <jungleboyj> e0ne:  Cool.  Let me know.
16:07:06 * e0ne still uses cinderclient CLI
16:07:08 <jungleboyj> So, I know have the planning etherpad for the PTG put together:
16:07:26 <jungleboyj> #link https://etherpad.openstack.org/p/cinder-ptg-rocky
16:07:42 <jungleboyj> If you are planning to attend, please add you name out there and start adding topics.
16:08:04 <jungleboyj> I will organize the times and everything as we get closer to the PTG.
16:08:33 <jungleboyj> Also, we had talked in Denver about doing T-Shirts with the Cinder Logo.  Are people still interested in doing that?
16:08:34 <geguileo> e0ne: were your issues related to this bug? https://bugs.launchpad.net/python-cinderclient/+bug/1695054
16:08:35 <openstack> Launchpad bug 1695054 in python-cinderclient "AttributeError: 'Namespace' object has no attribute 'os_domain_id'" [High,Fix released]
16:08:56 <e0ne> geguileo: yes, thanks for the link!
16:09:06 <e0ne> jungleboyj: +1
16:09:06 <jungleboyj> e0ne:  Ah, I have seen that as well.
16:09:06 * smcginnis stumbles in from a dentist appointment
16:09:14 <geguileo> e0ne: np, solution is unset OS_AUTH_TYPE
16:09:18 <jungleboyj> smcginnis: Excellent!
16:09:29 <jungleboyj> #chair smcginnis
16:09:30 <openstack> Current chairs: jungleboyj smcginnis
16:09:44 <jungleboyj> smcginnis:  Adding you in case I need to run.
16:09:50 <smcginnis> ++
16:10:05 <jungleboyj> So, e0ne  is interested in doing shirts.  Anyone else?
16:10:10 <amito> +1
16:10:25 <smcginnis> Shirts would be cool!
16:10:28 <e0ne> jungleboyj: and stickers with old logo, please!
16:10:39 <jungleboyj> The Cinder block one?
16:10:45 <jungleboyj> e0ne:
16:10:48 <e0ne> yes
16:11:08 <jungleboyj> e0ne:  Ok, not sure I have many of those left.  Will look.  If not I will work with smcginnis  to get more.
16:11:17 <smcginnis> A cinder block dropping on a horse's head? Maybe a little too dark? :)
16:11:27 <e0ne> :)
16:11:30 <geguileo> lol
16:11:35 <jungleboyj> I will look into shirts and see if I can have something designed by next week's meeting based on the notes from Denver.
16:11:38 * smcginnis was channelling his inner Swanson
16:12:06 <Swanson> smcginnis, What? Was looking for a Marathon Man dentist scene...
16:12:15 <jungleboyj> Guessing that we will need some time to get the shirts made up.
16:13:17 <jungleboyj> #action jungleboyj  to get more Cinder Block stickers and to propose a t-shirt design.
16:13:45 <jungleboyj> #topic Merged Specs without Reviews Associated
16:14:10 <jungleboyj> So, I just did a quick pass through the etherpad today and we have at least 3 specs that I don't see code reviews associated with.
16:14:19 <jungleboyj> 1) Transfer snapshots with volumes
16:14:19 <smcginnis> Which ones?
16:14:29 <jungleboyj> 2) Update cheesecake promotion spec
16:14:36 <jungleboyj> 3) Provisioning improvements
16:14:49 <jungleboyj> I swear I had seen code on 1, didn't have time to find it.
16:15:11 <jungleboyj> For 2) is there going to be code changes associated or was that more just documentation?
16:15:13 <erlon> jungleboyj: Im working on 3)
16:15:27 <erlon> should post the code in the last minute :)
16:15:34 <jungleboyj> erlon:  Ok, I thought that was the case.  Do you have anything you can link in the etherpad?
16:15:49 <erlon> jungleboyj: not yet
16:16:04 <jungleboyj> #link https://etherpad.openstack.org/p/cinder-spec-review-tracking
16:16:17 <jungleboyj> Ok, let me know when you do have that up as we are running short on time.
16:16:22 <erlon> jungleboyj: Ill gather thing here and post there
16:16:32 <jungleboyj> erlon:  Thank you.
16:17:01 <smcginnis> Keep in mind, non-client lib freeze is next week, so if there are any features that require os-brick changes, they better be quick ones.
16:17:10 <erlon> jungleboyj: proposal freeze is 15 right?
16:17:41 <jungleboyj> smcginnis:  ^^
16:18:15 <e0ne> #link https://releases.openstack.org/queens/schedule.html
16:18:20 <smcginnis> jungleboyj: I can't remember, what did you set spec freeze to?
16:18:48 <smcginnis> Ah, yep, next week.
16:18:51 <erlon> it says 15-19, so 19 I assume :p
16:18:51 <jungleboyj> e0ne:  Thank you.  Looks like it is the end of next week.
16:18:51 <e0ne> smcginnis: https://releases.openstack.org/queens/schedule.html#q-cinder-spec-freeze
16:19:04 <erlon> jungleboyj: good
16:19:06 <smcginnis> But a bit tricky with the lib freezes, so gotta keep that in mind.
16:20:01 <jungleboyj> I am going to spend a bit more time going through that etherpad tomorrow and will reach out to people with work pending.
16:20:02 <e0ne> smcginnis: do we have any cinder or nova  features depended on os-brick under development now?
16:20:30 <smcginnis> e0ne: None that I am aware of, but something we need to watch out for.
16:20:38 <jungleboyj> e0ne:  I don't know of any either.
16:20:43 <jungleboyj> Will do.
16:20:45 <smcginnis> Same with anything needing python-cinderclient updates. That's the following week.
16:21:16 <jungleboyj> So, I will follow up on 1 and 2 as I am not seeing anything for those merged specs either.
16:21:21 <e0ne> we've got one more week for cinderclient release
16:21:26 <jungleboyj> Guess we can always back them out of Queens.
16:21:37 <smcginnis> We should plan on reverting or moving them to rocky if we don't see code soon.
16:21:47 <e0ne> smcginnis: +1
16:21:49 <jungleboyj> smcginnis:  ++ agreed.
16:22:07 <tommylikehu> +1
16:22:26 <erlon> smcginnis: soon you men the proposal freeze?
16:22:38 <e0ne> I'll propose my backups-related patches early in Rocky. I won't have a time to finish them in Queens but any feedback on current proposals are welcome
16:22:57 <tommylikehu> e0ne:  sure
16:23:22 <smcginnis> erlon: I just mean if there are approved specs but we don't have code for them yet, or if it looks like there will not be enough time to get code changes merged, we need to move them out of queens so it doesn't confuse anyone expecting to see those changes in the release.
16:23:52 <jungleboyj> Ok.  So, I am going to keep watching this closely this week and we can touch base in next week's meeting.
16:24:07 <jungleboyj> Anything else or can we move on to the last topic?
16:24:23 <erlon> smcginnis: ok, it makes sense
16:24:30 <smcginnis> jungleboyj: Should be good to move on.
16:24:40 <jungleboyj> #topic RBD driver performance issues:
16:24:45 <jungleboyj> mnaser:  You here?
16:24:50 <mnaser> hola!
16:24:57 <smcginnis> I think eharney has some background too.
16:25:05 <jungleboyj> Hola.  Como estas?
16:25:12 <mnaser> sorry that was my limit of spanish :)
16:25:22 <smcginnis> :)
16:25:22 <mnaser> so one of the things that we've ran into is very poor performance with the stats collection for cinder-volume with rbd
16:25:51 <geguileo> mnaser: which release?
16:25:52 <jungleboyj> Ok.
16:25:57 <geguileo> mnaser: because we've fixed that
16:26:00 <mnaser> geguileo: pike (and i believe the issue is in master)
16:26:12 <mnaser> geguileo: right, the fix still has a lot of performance issues which i wanted to dive into
16:26:15 <geguileo> mnaser: I believe it's no longer in master, since my patches landed
16:26:30 <mnaser> the code right now has this loop which goes over all the rbd volumes of a pool.  in some cases, the pool might contain non cinder volumes
16:26:59 <geguileo> mnaser: that's not a good deployment idea (sharing the pool)
16:27:02 <mnaser> so for example, we might have a pool with 1600 rbd volumes, only 800 which are cinder volumes, yet it does iterate and gets the allocated size for all of them
16:27:11 <erlon> mnaser: is that related to listing all volumes in every get stat?
16:27:20 <geguileo> mnaser: yes, because it needs to report that information
16:27:27 <jungleboyj> Yeah, not a good idea to share pools.
16:27:35 <geguileo> mnaser: otherwise it cannot report the information to Cinder
16:27:36 <mnaser> (its a legacy mistake that's hard to fix at this point)
16:27:49 <mnaser> but regardless, imagine we had 1600 cinder volumes in a single rbd poool
16:27:58 <erlon> mnaser: how long are we talking about? to list all volumes?
16:27:59 <geguileo> mnaser: well, we cannot change the driver to ignore those or we would be breaking the cinder contract
16:28:11 <mnaser> it lists all volumes and then gets the sizes of all volumes, but the interesting thing here is that the number it gets really is pretty much the db size of the volume
16:28:35 <mnaser> erlon: cpu rates are out of the roof and the loop blocks the main process from what it seems like
16:28:38 <mnaser> so the agent reports down
16:28:40 <geguileo> mnaser: but there are volumes that are not in the DB, and we need to return the total of all volumes (cinder and not cinder)
16:28:47 <mnaser> and so volumes cannot be scheduled anymore
16:28:55 <erlon> mnaser: hmm
16:29:21 <mnaser> geguileo: right, but based on your logic -- there shouldn't be any other volumes, so we can use the db number, and if there is, its a deployer issue.  i'm not opposed to the loop, i just think that its currently blocking the agent from reporting up
16:29:21 <geguileo> mnaser: with just 1600 volumes?  r:-???
16:29:27 <erlon> geguileo: isnt a way to cache that?
16:29:30 <mnaser> and so the agent goes down
16:29:36 <erlon> geguileo: how acurate it needs to be?
16:29:49 <geguileo> mnaser: No, a good deployment shouldn't share the pool, but if you share we have to report them all
16:30:01 <mnaser> i mean, i did a small loop the other day and it took quite sometime.  (manual bash for loop with rbd info)
16:30:22 <mnaser> now to imagine that this runs every x seconds it ends up making the agent report down
16:30:25 <erlon> geguileo: +1
16:30:37 <geguileo> mnaser: the question was how long it took for cinder to get the data not running in bash (as they are different)
16:30:45 <jungleboyj> Could it be made configurable?
16:31:01 <geguileo> jungleboyj: No, or we go back to drivers not reporting the right data
16:31:08 <mnaser> my doubts are solidifed with the fact that the cinder-volume agent for the pool with low # of volumes would report up
16:31:13 <geguileo> jungleboyj: we just landed a spec making it very clear what needs to be reported
16:31:15 <mnaser> maybe we need to take the reporting into a seperate thread?
16:31:23 <geguileo> jungleboyj: and making it optional would make this driver non compliant
16:31:44 <jungleboyj> Oh, it is tied into that. Ok.
16:31:56 <mnaser> i dont want to pull out code that does the job properly and correctly
16:31:59 <erlon> mnaser: I think it is a separate thread currently, soft though
16:32:05 <jungleboyj> geguileo: you are right
16:32:07 <geguileo> mnaser: we should already be doing that
16:32:22 <geguileo> mnaser: using an eventlet proxy (though if we are not doing that it could be the problem)
16:32:38 <mnaser> hmm, so there must be something that's causing it to lock up and report the agent as down
16:32:55 <geguileo> mnaser: yup, looks like it
16:33:01 <mnaser> so in my checking
16:33:09 <geguileo> mnaser: I'll check it in my env and see if I see the same issue
16:33:18 <smcginnis> Can we get someone from Red Hat to work with mnaser to debug this further. Doesn't sound like something we can solve in the meeting.
16:33:20 <geguileo> mnaser: so 1600 volumes with only half belonging to Cinder, right?
16:33:37 <jungleboyj> smcginnis:  ++
16:33:39 <geguileo> smcginnis: Yup, that's what I was getting at, I'll try to reproduce
16:33:43 <mnaser> geguileo: yes, but i guess you can do 1600 volumes all cinder and i assume the same issue will occur
16:33:47 <smcginnis> geguileo: Cool, thanks.
16:33:55 <geguileo> mnaser: OK, will try to reproduce
16:34:07 <mnaser> geguileo: https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L492 fyi this is the call that when switched to =0 the agent would go up and have no problems
16:34:23 <jungleboyj> geguileo: thank you for taking that.
16:34:35 <mnaser> so it's within the _get_usage_info().  i'm always around on irc so please feel free to ping me anytime to further discuss this so we can improve the performance of the rbd driver.  thank you everyone :)
16:35:04 <geguileo> mnaser: will ping you as soon as I verify it
16:35:09 <smcginnis> Thanks mnaser and geguileo.
16:35:48 <jungleboyj> #action geguileo  to help reproducing and debug
16:36:23 <jungleboyj> So is that all for that topic?
16:36:30 <smcginnis> Nothing more from me.
16:37:01 <jungleboyj> #topic open discussion
16:37:18 <jungleboyj> Any other topics that people need to cover?
16:38:03 <jungleboyj> *crickets*
16:38:10 <chandankumar> I have just refactored cinder tempest plugin
16:38:11 <chandankumar> https://review.openstack.org/529824
16:38:40 <chandankumar> https://review.openstack.org/531206 -> zuulv3 based job before removing bundled intree tempest plugin
16:38:45 <chandankumar> please have a look,
16:39:06 <jungleboyj> Cool. They can take a look at that.
16:39:08 <smcginnis> Oh yeah, we need to get the in-tree tests removed.
16:39:12 <chandankumar> one more thing do we want to publish cinder tempest plugin on pypi and want to doa  tag release
16:39:41 <chandankumar> if we do a tag release, it will be easier for RDO packages to consume and maintain easily
16:40:25 <jungleboyj> smcginnis: was it you that had a patch out there for removal? I hadn't approved it cuz it wasn't clear that everything had been moved out of tree.
16:40:48 <smcginnis> jungleboyj: It was, but your comment was confusing so I asked for more info. :)
16:41:15 <jungleboyj> Okay. I will follow up on that.
16:41:25 <eharney> chandankumar: is the cinder-tempest-plugin repo going to have branches?
16:41:33 <chandankumar> eharney: nope
16:41:48 <smcginnis> I guess if we are following tempest, that means we follow that policy.
16:41:54 <chandankumar> eharney: it is brnachless in nature
16:41:56 <eharney> just curious
16:41:58 <smcginnis> I don't really like that policy though.
16:42:25 <eharney> also, an unrelated FYI: the driverfixes/newton and ocata branches are functioning again, and we've started landing patches
16:42:37 <smcginnis> Thanks for sorting that out eharney!
16:42:46 <tommylikehu> eharney:  thanks
16:42:55 <jungleboyj> eharney: yay! Thank you for fixing that  a
16:43:00 <e0ne> eharney: great!
16:43:16 <smcginnis> Just need some unit tests fixed in driverfixes/mitaka now.
16:44:17 <smcginnis> Anything else for today?
16:44:52 <jungleboyj> So, what is the word on the tempest plugin? We tag it ot not
16:45:45 <smcginnis> Sure :)
16:45:57 <chandankumar> jungleboyj: thanks :-)
16:46:00 <jungleboyj> smcginnis: ok, that is clear.
16:46:17 <jungleboyj> Yeah it with three release?
16:46:32 <smcginnis> ?
16:46:38 <jungleboyj> *tag it with the release?
16:46:46 <jungleboyj> :-p
16:47:03 <chandankumar> i mean a tag which will work with Queens one
16:47:05 <smcginnis> Ah, yep. When a release is done it gets tagged. And just need to set it to publish to pypi.
16:47:23 <jungleboyj> Ok. Makes sense.
16:48:01 <jungleboyj> Ok. Anything else?
16:48:48 <Swanson> No.
16:48:49 <jungleboyj> Guess not.
16:49:06 <jungleboyj> thanks team. Talk to you all next week.
16:49:12 <smcginnis> Safe travels jungleboyj
16:49:18 <Swanson> Toodles.
16:49:32 <jungleboyj> smcginnis: thank you. Will be good to be back home.
16:49:43 <jungleboyj> #endmeeting