16:00:27 #startmeeting Cinder 16:00:27 Meeting started Wed Jun 21 16:00:27 2017 UTC and is due to finish in 60 minutes. The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:32 The meeting name has been set to 'cinder' 16:00:43 ping dulek duncant eharney geguileo winston-d e0ne jungleboyj jgriffith thingee smcginnis hemna xyang1 tbarron scottda erlon rhedlind jbernard _alastor_ bluex karthikp_ patrickeast dongwenjuan JaniceLee cFouts Thelo vivekd adrianofr mtanino karlamrhein diablo_rojo jay.xu jgregor lhx_ baumann rajinir wilson-l reduxio wanghao thrawn01 chris_morrell watanabe.isao,tommylikehu mdovgal ildikov wxy viks ketonne abishop 16:00:43 sivn 16:00:48 hi 16:00:48 Hi 16:00:56 hi! 16:01:00 hi 16:01:01 hi 16:01:07 Sean has asked me to cover today. :-) 16:01:13 hello 16:01:14 Hey 16:01:37 Hello 16:01:45 * smcginnis is not really here, will catch up later 16:01:49 hi 16:01:53 hi 16:02:13 smcginnis: Is more here than expected. ;-) 16:02:35 Ok, so it looks like we have a good group here to start with. 16:02:45 #topic Announcements 16:03:21 The usual reminder to please take a look at the review priority for the release: https://etherpad.openstack.org/p/cinder-spec-review-tracking 16:03:25 #link https://etherpad.openstack.org/p/cinder-spec-review-tracking 16:04:04 smcginnis: and I went through the reviews from last month's bug smash and worked to get as many things merged as possible. 16:04:35 jungleboyj, did our tagging bugs ahead of time help them out? 16:05:12 Also went through all the spec tracking items and got them updated. 16:05:20 diablo_rojo: I think that worked well. 16:05:23 hi 16:05:54 diablo_rojo: I think it is a practice we should continue to do. It looked like that was used as a starting point and then some other items were added. 16:05:59 Cool, we will have to keep that in mind for tagging bugs. Might even want to consider a monthly bugsmash day like other projects do. I 16:06:07 Last time I checked there were about 19 bugs that were addressed and closed. 16:06:08 jungleboyj, jinx 16:06:23 diablo_rojo: Jinx. ;-) 16:06:51 I have been trying to coordinate with smcginnis once a month to go through and clean up bugs, etc. So, yeah, we are on the same page there. 16:07:27 I think people have been doing a good job of looking at reviews from what I have seen. Thank you. 16:08:07 Also, one other announcement, I marked all the proposed drivers that missed the Pike deadline with a -2 just so we don't accidentally merge anything there. 16:08:14 hi 16:08:25 * jungleboyj wasn't very popular for that. 16:08:34 @!h 16:08:34 jungleboyj (/ .□.) ︵╰(゜Д゜)╯︵ /(.□. ) 16:08:51 So, I think that was all I had for announcements. 16:09:16 #topic Add new config option in volume driver class 16:09:22 guyk: You here? 16:09:33 jungleboyj: yes 16:09:45 Take it away. 16:10:44 this idea was proposed by Mike Perez in one of my patch. It is to create and use generic variable in the volume driver. 16:11:16 what kind of generic variable? r:-?? 16:11:17 And replace current and different config options used in the drivers 16:11:30 guyk, what's the patch? 16:11:42 so consolidation of configuration variables 16:11:42 geguileo: set the ip address and the port of the volume backend 16:11:52 lhx_: https://review.openstack.org/#/c/457426/3 16:11:59 thx :) 16:12:33 Oh yes, we have had this discussion before. 16:12:41 in general, I like this idea 16:12:46 #link https://review.openstack.org/#/c/457426/3 16:12:55 wouldn't it be better to support a list 16:13:08 geguileo +1 16:13:19 geguileo: +1 16:13:27 address list? 16:13:39 problem is that each address could have a different port 16:13:47 unlikely, but possible 16:14:12 geguileo: So that is concern with the specific patch. 16:14:18 guyk just to be clear, we do already have this option FWIW 16:14:19 san_ip 16:14:28 just some don't use it 16:14:32 I think we want to first address consolidation? 16:14:37 jgriffith: ++ 16:14:45 I'm sort of leary about introducing yet another "common" config option that people choose to not use 16:15:13 jgriffith: yes some of the drivers don't use it because their backend doesn't use the san protocol so they rename it 16:15:16 if we do want a better name that's totally cool with me, but we should deprecate the old one and make it clear we expect it to be used 16:15:29 guyk there's no such thing as a 'san' protocol 16:15:38 guyk san just means it's an external device 16:15:38 jgriffith: +1 16:15:45 we have some "common" config options in san.py that some drivers have been using: https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/san/san.py#L42 16:15:52 guyk and that you're accessing it via API on the 'san_ip' address 16:16:47 jgriffith: yes, made a mistake 16:17:31 In other words, it's the name game IMO; and if people want to play the name game that's fine by me; but be very clear that we already have a generic config option, this change should be a proposal to rename the existing san_ip 16:17:46 we'll still have the problem we have now of 1/2 the devices not using it though :) 16:18:04 jgriffith: I agree that we have 2 topics: name game and adoption 16:18:20 For adoption we could go through deprecating all individual configuration options 16:18:27 to force drivers to update and consolidate 16:18:41 though it may not be a very nice thing to do :- 16:18:43 ( 16:18:48 geguileo yeah 16:19:00 geguileo there's a broader effort there if we wanted to consider it 16:19:23 geguileo there's pretty extensive duplication of config options and exception classes... 16:19:34 anyway; that's a bit of a rat-hole probably :) 16:19:37 I think consolidation is a good idea 16:19:52 and geguileo yes, it's sort of not nice to do probably 16:20:02 but I'm not a nice person.. so... there's that :) 16:20:07 lol 16:20:23 It would be best if we could consolidate all/most of the common options 16:20:30 back to guyk 's patch though 16:20:34 but then we would really need to make sure they are adopted 16:20:49 there is a broader question, but what do we want to do with his proposal to start with? 16:20:57 geguileo +1 16:21:49 bueller.... bueller.... 16:21:50 geguileo: I agree but that is going to require a large commitment and planning to implement it. 16:21:57 should we decide if we want to go with the deprecation of other options? 16:22:14 geguileo, do it in waves? A few at a time? 16:22:18 because as I see it, without adoption this will be another san_ip 16:22:27 geguileo: Agreed. 16:22:27 diablo_rojo the problem is that never seems to work for us :( 16:22:34 jgriffith: ++ 16:22:39 diablo_rojo we set out to do things that way rather often and never finish 16:22:42 diablo_rojo: yeah, but the only way to force it is if we deprecate the options 16:22:45 jgriffith, touche 16:22:46 I think it would need to be all at once. 16:22:50 diablo_rojo: and then remove them 16:23:01 diablo_rojo although it is an optimal approach (when it works) 16:23:16 So deprecating few at a time is not a good idea? 16:23:19 keep in mind you have to deal with upgrades too 16:23:21 it's migration from individual cfg opts to common opts instead of adoption 16:23:31 diablo_rojo: I would go with deprecating them all in one patch 16:23:31 How many approx need to be deprecated? 16:23:38 geguileo, that works too :) 16:23:38 adoptions sounds like you had a choice of not to 16:23:46 winston-d good point! 16:24:01 but it seems not adopting is not allowed here. 16:24:01 geguileo: ++ 16:24:20 so here's an excercise for folks that are interested... generate a current sample.conf 16:24:23 winston-d: if we want this to be really usefull yeah, it wouldn't be optional 16:24:37 geguileo: Agreed. 16:25:06 geguileo, +1 16:25:21 so we agree to force the migration to common config options? 16:25:26 Does the upgrade mean that currently installed cinder.conf files will need to be updated manually? 16:25:27 jgriffith: I think I know where you are going. 16:25:38 jungleboyj :) 16:25:43 cFouts: I think it might. 16:25:44 geguileo, yes? 16:25:54 geguileo: i agree. 16:25:56 cFouts yup, that's the problem 16:26:03 to support backward compatibility, the old options will have to be supported for many releases 16:26:05 yeah, it is 16:26:09 geguileo: I don't want to go that far yet. 16:26:31 Let me make a proposal before we charge forward. 16:26:33 xyang2: yeah, :( 16:26:47 I think we need to do what jgriffith started to propose. 16:27:02 Generate a config file and take inventory of the options. 16:27:13 so we'll end up with more options than we wanted 16:27:14 See how many options this may actually impact. 16:27:18 cfg opts is also considered as user-facing interface 16:27:37 Once we have an idea of the impact we can make a plan going forward. 16:27:45 winston-d: +1 16:28:05 Deprecate the old ones. Do a release accordingly. 16:28:12 Document the upgrade path to help people. 16:28:14 this metting , today it's running... 16:28:34 Thoughts on this proposal? 16:28:55 jungleboyj: do you think people read document?:) 16:29:10 jungleboyj: you'll get a support call first:) 16:29:16 xyang2: the will once something will fail 16:29:26 so all san options will be deprecate too? 16:29:31 xyang2: True enough. 16:29:46 lhx_: That scares me. A lot of people use that. 16:31:04 Well, that stopped the discussion. 16:31:17 Everyone is looking through the sample config? 16:31:55 jungleboyj :) 16:32:01 jungleboyj, I think people lost interest when it started seeming less imminent 16:32:06 Lol :) 16:32:11 * jungleboyj is suddenly very lonely 16:32:18 so I think if this is an approach we want to take the starting point would be to use what we have first 16:32:39 jgriffith: Can you say more about that? 16:32:43 ie consolidate on the existing things like san_ip etc, find as many as we can without introducing new ones 16:32:53 jgriffith: ++ 16:32:55 yesterday I tried 'tox -egenconfig' on master branch tip, it's not working. :( but probably that's just me. 16:33:09 if we just go in and introduce a whole new set of "new" options we're just making the problem worse 16:33:09 winston-d: :-( Yikes. 16:33:20 jgriffith: Totally agree. 16:33:25 even if it's temporary.. and remember temporary is like at least 2 releases 16:33:48 jgriffith: +1 16:34:08 is there a user survey of new options? not sure user care about the new ones. 16:34:17 :( 16:34:31 winston-d I just ran it succesfully.. so maybe refresh on your tox env? 16:34:58 lhx_: my guess is user wants the existing options they've already configured to continue to work:) 16:34:59 Look.. here's the thing; our config file is 4,859 lines!!!! 16:35:01 That's stupid 16:35:13 Agreed 16:35:15 diablo_rojo: This can't be an imminent thing as it directly impacts users. 16:35:19 jgriffith: yup, i should try with new virtual env. 16:35:22 Need to be careful here. 16:35:25 lhx_: if an existing option stops working, they will get upset 16:35:27 granted that's all the blank-lines and comments, but still 16:35:34 it's not exactly consumable 16:35:51 Right. 16:36:13 jungleboyj, I wasn't saying go for it :) I just mean people are less interested when its not a threat. 16:36:27 * jgriffith hands folks a copy of War and Peach and says.. "here's your example, all the answers are in there" 16:36:36 good luck 16:36:37 jgriffith, lol 16:36:48 I would like to know how much consolidation will change that. 16:36:49 haha... "Peach" 16:36:51 'May the odds be ever in your favor' 16:36:54 xyang2, haha, +1 16:36:56 jgriffith: *Laughing* 16:37:07 I love War and Peach 16:37:43 So we agree we have a problem here. 16:37:56 * jungleboyj wonders if there is support group for config option addiction. 16:38:45 o/ My name's Lance and it's been 187 days since I've added a configuration option to my project 16:39:03 Hi Lance ... 16:39:21 o/ My name's Jay and I cried while writing a config file once. 16:39:56 haha 16:39:57 grep -v '^$\|^\s*\#' /etc/cinder/cinder.conf 16:40:11 that's not profanity it does something useful 16:40:22 Anyway, so, it has gotten quiet again. Do we have anyone who would be willing to take the config file and get an idea of how many options could be consolidated to use existing options. 16:40:50 bswartz thanks for that regex foo 16:41:17 I think that would be the first step and then re-address discussion in the next meeting. 16:41:23 *crickets* 16:41:33 I can put it on my backlog to take a look. 16:41:57 diablo_rojo: Would be good to have someone familiar with config options do it. 16:42:02 :D 16:42:40 Will you have a chance to at least get an idea of how many duplicates there are before next week? 16:43:36 Maybe, I can do an introductory foray to get a general idea of a lot- little, etc. 16:43:49 diablo_rojo: Ok, cool. 16:44:02 #agreed We need to look at our crazy list of config options. 16:44:05 I will see if I can get a more exact number, but I suspect it will be tedious and time consuming as hell. 16:44:25 diablo_rojo you're suspicion is correct :) 16:44:30 #action diablo_rojo will try to get an idea of how many options could possibly be consolidated to existing items 16:44:38 which is why we have the sprawl we have 16:44:55 #agreed We should start by using existing options rather than adding new ones. 16:45:12 Those agreements and action look good? 16:45:40 jgriffith, hence my unease at accepting the task lol 16:45:52 jungleboyj, yeah looks fine to me 16:46:07 #action Jay to add to agenda for an update next week. 16:46:14 diablo_rojo: I usually ask driver author to use options in san.py, but sometimes they argue that they need to use a different option in their driver for some reason. Just FYI 16:46:19 diablo_rojo too late, it's been registered in IRC.... you're DOOOOMED 16:46:20 :) 16:46:40 With a number/impact in mind we can put together a plan. 16:46:42 15 mins reminder 16:46:48 I guess cinder only cfg opts are only < 50% of all 4,859 lines? 16:46:51 jgriffith, thanks for the vote of confidence ;) 16:46:52 diablo_rojo: You are good at those details. 16:47:07 jungleboyj, not sure if that is a compliment or not lol 16:47:12 Ok. Lets wrap this up for this meeting. 16:47:13 xyang2, noted :) 16:47:16 diablo_rojo: :-) 16:47:18 many shall be library introduced opts. 16:47:38 guyk: Sorry, we kind of hijacked things here. 16:47:39 diablo_rojo I have confidence in you, I just feel bad for you :) 16:48:04 jgriffith, can't be any worse than standing in front of thousands of people and a demo not working ;) 16:48:06 guyk: So, I think we want to hold on your patch at the moment and look at this before going forward. That ok? 16:48:09 jungleboyj: it is ok 16:48:22 Ouch ... 16:48:27 guyk: Ok. Thanks. 16:49:15 #topic Dynamic Reconfiguration 16:49:21 diablo_rojo: More config discussion. 16:49:29 Indeed. 16:50:00 It appears that there is a nontrivial section of Cinder cores that think this is a bad idea- especially for multinode setups 16:50:16 And I am gravitating towards that school of thought 16:50:19 diablo_rojo: could you please share a link to the patch? 16:50:30 e0ne, sure one sec 16:50:38 is there a short summary of the reason it's tough for multinode/HA? 16:51:04 eharney: ++ 16:51:05 #link https://review.openstack.org/#/c/446132/ 16:51:18 diablo_rojo: thanks 16:51:27 Sad to see the discussion go this way. 16:51:28 eharney, basically it gets messy fast because processes don't end all at the same time. 16:51:46 eharney: synchronziation between multi-nodes is hard? 16:51:48 bswartz: Manila supports this, doesn't it? 16:51:57 * e0ne still thinks that SIGHUP signal will be always a good potion 16:52:07 ? 16:52:35 s/potion/option 16:52:36 e0ne, agreed, especially if CInder actually inherits from oslo's services and its handled how it looks to be written. 16:52:41 jungleboyj: not yet, we want it 16:52:56 bswartz: Ah, ok. I had been told you have it. 16:53:00 e0ne: potion is better, we need magic here. 16:53:02 * diablo_rojo votes to wait till Manila has solved it like we do with most our problems ;) 16:53:04 diablo_rojo: afaik current SIGHUP will shutdown everything in Cinder afaik 16:53:08 jungleboyj: maybe we inherited it? 16:53:15 winston-d: :) 16:53:21 or maybe I'm less informed than you >_< 16:53:25 bswartz: it's not in cinder yet:) 16:53:45 * jungleboyj is confused. I though the point of this work was to get SIGHUP working. 16:53:49 geguileo: yes, but we can implement graceful shutdown if it's not implemented yet 16:53:58 bswartz: I doubt it. 16:54:04 e0ne: we have graceful shutdown implemented there 16:54:16 bswartz: I remember we talked about this in some manila meetings, referencing cinder 16:54:24 e0ne: but it still needs to shutdown, which may take a loooong time 16:54:49 geguileo: it's another issue we should care about 16:55:09 the long time? 16:55:34 so what's the core reason that we can't implement this? it's hard? 16:56:14 winston-d: diablo_rojo I'm trying to understand what is different in the multi-node case than when one shuts down service on a single node, changes config, and brings it up while other nodes are running. 16:56:24 eharney: That is my fear. 16:56:24 winston-d: diablo_rojo are we contrasting this with: 16:56:40 shut down service on all nodes, update config, start on all nodes? 16:56:57 tbarron: Yeah, I don't understand why that is more dangerous that what we currently do. 16:57:16 eharney, well yes, but from looking through the code, it looks like it should work already. Some people have said it works for them and others have said it does not. I personally haven't been able to figure out why that is. 16:57:16 that last is probably safest, but ... 16:57:23 just trying to understand the issue 16:58:08 and some config options would be safe, whereas some may have problems w/o synchronization across nodes I guess. 16:58:12 If we are talking about using SIGHUP to make cinder restart itself and reload the new config options 16:58:17 that's been working for a long time 16:58:47 Just shuts down for me. 16:58:49 geguileo, right, that should work but some people say it doesn't? 16:58:51 diablo_rojo: I think that had to do with whether you were running Cinder in Devstack/screen vs. using systemd 16:59:05 smcginnis: on which env? devstack? r:-?? 16:59:11 devstack tests don't count :) 16:59:22 smcginnis: In devstack ? 16:59:32 smcginnis: did you send SIGHUP to the parent process or to all of them? 16:59:34 jungleboyj, that was part of the confusion, but I thought some people also ran it with their own drivers too 16:59:51 geguileo: I thought I tried on actual deployment, but can't remember for sure now. Parent process. 16:59:57 for multinode unless you have a better multinode devstack than I've been able to do 17:00:04 geguileo, smcginnis: SIGHUP doesn't work on devstack with screen 17:00:06 there was some difference in how services behave when running on a terminal (i.e. in screen) that i looked into a while back, so yeah, devstack tests may not behave like normal services 17:00:17 Anyway, we have run out of time. 17:00:20 smcginnis: I tried it while preparing for my Barcelona presentation and it worked for me back then 17:00:30 screen is dead for devstck anyways 17:00:34 I think we need to finish this discussion. 17:00:42 e0ne: devstack doesn't use devstack anymore, right? 17:00:43 geguileo: OK, could be why I was having issues with this too: https://review.openstack.org/#/c/464028/ 17:00:46 Times up. 17:00:54 screen is dead, long live screen! 17:00:57 jungleboyj, yes, cause we have had this part of the discussion several times and made no progress :) 17:00:58 but there's a good question about incompatability across nodes 17:01:05 geguileo: yes, it switched to systemd by default 17:01:05 Can we get people to verify what they see working or not and we cna wrap up discussion next week? 17:01:24 Yes please 17:01:31 i think we need to do another round on the spec... 17:01:33 diablo_rojo: Ok. We will go that way. 17:01:38 eharney, :P 17:01:48 smcginnis: that's something I haven't explored but could be interesting 17:01:48 across three apis running concurrently, or if an option affects the rpc between services 17:02:06 #action Cores to try using sighup this week and then we will decide what to do in next week's meeting. 17:02:31 Ok, we need to free up the meeting room. 17:02:32 * tbarron watches jungleboyj SIGTERM this meeting 17:02:35 tbarron: got disconnected just now, will catch up with you in cinder 17:02:40 Thanks for the good discussion! 17:02:43 SIGKILL! 17:02:44 see you next week! 17:02:52 #endmeeting