18:02:39 <hub_cap> #startmeeting trove 18:02:39 <openstack> Meeting started Wed Mar 19 18:02:39 2014 UTC and is due to finish in 60 minutes. The chair is hub_cap. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:43 <openstack> The meeting name has been set to 'trove' 18:02:55 <imsplitbit> o/ 18:02:56 <cp16net> haylo 18:03:05 <SlickNik> o/ 18:03:05 <hub_cap> so looks like the first thing on the agenda is the refactoring datastore options thing 18:03:07 <kevinconway> 7o7 18:03:07 <pdmars> o/ 18:03:18 <cp16net> action items? 18:03:19 <hub_cap> #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting 18:03:23 <grapex> o/ 18:03:55 <hub_cap> #link 18:03:59 <hub_cap> lol stupid paste 18:04:07 <hub_cap> #link http://eavesdrop.openstack.org/meetinags/trove/2014/trove.2014-03-12-17.59.txt 18:04:08 <mat-lowery> o/ 18:04:17 <vgnbkr> o/ 18:04:22 <juice> o/ 18:04:26 <k-pom> \o 18:04:29 <doug_shelley66> o/ 18:04:40 <hub_cap> ok so im thinking next time, dotn worru about saying yer here 18:04:42 <kevinconway> hub_cap: 404 on that link 18:04:44 <hub_cap> i trust everyone is here 18:04:49 <glucas> o/ 18:05:09 <hub_cap> http://eavesdrop.openstack.org/meetings/trove/2014/trove.2014-03-12-17.59.txt 18:05:14 <juice> vipul is not here 18:05:21 <hub_cap> juice: then when i say, hey vipul 18:05:24 <hub_cap> and he doesnt respond 18:05:27 <hub_cap> ill know hes not here :) 18:05:28 <grapex> hub_cap: But roll call is my favorite part! 18:05:38 <kevinconway> it also logs us as there in the meeting notes 18:05:47 <SnowDust> grapex: present SIR ! 18:05:50 <SnowDust> :D 18:05:53 <hub_cap> kevinconway: im not sure thats necessary 18:06:06 <kevinconway> i get paid per lines in the meeting log 18:06:09 <hub_cap> anyhoo lets talk about it in open discuss 18:06:14 <hub_cap> kevinconway: lol exactly 18:06:18 <SnowDust> kevinconway :D 18:06:21 <hub_cap> so lets start w/ 18:06:27 <pdmars> hub_cap: i didn't see this topic on the agenda 18:06:33 <hub_cap> #link refactoring-datastore-options 18:06:40 <hub_cap> pdmars: hence my open discuss comment ;) 18:06:49 <hub_cap> #undo 18:06:50 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x38eddd0> 18:06:59 <hub_cap> #topic refactoring-datastore-options 18:07:10 <hub_cap> #link 18:07:15 <hub_cap> ARGGGGGGGG PASTE 18:07:25 <SlickNik> #link https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg 18:07:37 <hub_cap> thx SlickNik 18:07:40 <SnowDust> https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg 18:07:44 <SnowDust> thanks .. 18:07:50 <cweid> hub_cap: my linux box never has this problems. 18:08:04 <SlickNik> So we discussed this at the bp-review on Mon. 18:08:15 <hub_cap> cweid: my linux box is about 15 min old, so i havent fixed my keybindings :) 18:08:35 <hub_cap> SlickNik: yes but we didnt come to a conclusion about work starting 18:08:43 <cweid> ohhhhh welcome to the cool kids team =) 18:08:53 <SlickNik> SnowDust: The main concern was that the current design / implementation allows for a complete override of the datastore config values. 18:08:59 <hub_cap> cweid: your mind is terrible, ive been on linux for 6+mo 18:09:24 <cweid> oh my bad. 18:09:27 <SnowDust> SlickNik: as the override is from the config file or from the implementation of datastore( external package) 18:09:36 <SnowDust> this should be a "Safe" option 18:09:43 <SnowDust> but i am open to discuss the consequence 18:09:49 <hub_cap> SlickNik: are we suer thats the case? 18:09:51 <kevinconway> hub_cap: that's almost 7 months 18:10:06 <SlickNik> The proposal was to have the design extend the default datastore configs instead of override them completely. 18:10:08 <hub_cap> cuz teh code, if i grok correctly, only uses the options if they are in a optgrp 18:10:44 <hub_cap> kevinconway: +:=<2wks 18:10:58 <SnowDust> SlickNik, until an ADMIN overrided using conf ( whcih he can even do in the current code) 18:11:01 <SnowDust> no override happens 18:11:14 <SnowDust> in current code also 18:11:32 <hub_cap> so this code doesnt split the conifgs out in any way 18:11:33 <SnowDust> we may define [mysql] / [percona]/[redis]/ in conf files 18:11:46 <SnowDust> to override datastore defaults for the datastore 18:11:55 <hub_cap> it just moves the in memory ones to different files, but until the datastores use the optgroups 18:12:02 <hub_cap> they will be ignored 18:12:23 <hub_cap> iirc u need to do a cfg.mysql.mount_point to access the [mysql]...mount_point=blah 18:12:38 <hub_cap> so in its current state im not sure what this code does :) 18:13:24 <SnowDust> hub_cap: SlickNik: the BP is just to enhance the component nature of databases 18:13:48 <SnowDust> my idea was to separate datastore specific config from the cfg.py to their own packages 18:14:06 <SnowDust> as we had the design of datastore Managers being a class loaded using config 18:14:25 <SnowDust> so its safe to load the configurations from the module itself .. from which the Managers are loaded 18:14:29 <kevinconway> SnowDust: would that include the ability to override a global option but only in one specific datastore (like the mount_point hub_cap mentioned)? 18:14:45 <SlickNik> SnowDust: The concern is that you'd be able to override the config values for a datastore type so that you don't load the confs for say cassandra, but the guestagent manager still had a cassandra implementation. 18:14:45 <SlickNik> 18:14:48 <SnowDust> kevinconway: just like current code the nature remains the same for override 18:14:49 <hub_cap> kevinconway: some code would have to be ther for that 18:15:13 <juice> snowdust: I like the modular nature/i.e. reduced clutter but on the flip side I need to piece together two files to see the complete picture 18:15:16 <kevinconway> hub_cap: right, i'm just trying to keep up with what this BP is for 18:15:37 <juice> snowdust: not sure it is worth the additional complexiyt 18:15:39 <kevinconway> juice: is it two files or just two sections in one file? 18:15:52 <SnowDust> juice: its worth is for keeping datastore implementations external 18:15:58 <hub_cap> so kevinconway i wrote this for cinders multi backend 18:16:00 <juice> kevinconway: two files as it is proposed 18:16:01 <hub_cap> https://github.com/openstack/cinder/blob/master/cinder/volume/configuration.py 18:16:11 <hub_cap> it lets you specify an optgroup, and if not, it uses the defaults 18:16:32 <hub_cap> so u just do configuration.mount_point, and if it can find the "mysql" mount_point, it uses 18:16:50 <hub_cap> kevinconway: iirc ;) 18:17:00 <SnowDust> SlickNik: i didnot get the last one from u 18:17:43 <SnowDust> SlickNik: as u saw cfg.py ( code snippet in the BP) loads Cassandra by default ( with its well defined defaults) 18:18:02 <SnowDust> but override is user choice and thats why we have config variables ment 18:18:24 <SlickNik> SnowDust: https://github.com/openstack/trove/blob/fba8cabea326527bacdeca56760a97e14cdcc18f/trove/guestagent/dbaas.py#L34-L51 lets you specify datastore managers. 18:18:38 <SnowDust> SlickNik : right 18:18:53 <hub_cap> ok so looking at this, SlickNik SnowDust 18:19:05 <SnowDust> hub_cap : thanks .. 18:19:12 <hub_cap> does it override the existing opts instead of definig them _only_ in an optgroup? 18:19:23 <hub_cap> if thats the case this is a BIG nono 18:19:35 <SnowDust> its does not override any existing ones .. 18:19:36 <hub_cap> we will not, ever, overwrite opts in memory that were defined 18:19:56 <hub_cap> ok so how do i use the opt in [mysql] mount_point SnowDust ? 18:20:12 <SnowDust> confs just exist in ONE place .. their place of existance has been reworked .. from cfg.py ( which is a tied down approach) to the datastore module itself 18:20:24 <SnowDust> hub_cap 18:20:28 <SnowDust> as per the cfg.py code 18:20:38 <SnowDust> the datastore_options dictOpt 18:21:10 <SlickNik> SnowDust: the default managers are always loaded; similarly the default configs should also be always loaded. 18:21:12 <SnowDust> is a mapping between datastore_manager and the datastore_manager.implementation.module.entrypoint 18:21:28 <hub_cap> SnowDust: ok, so how do i get that option 18:21:50 <SnowDust> after that is done .. it used oslo.config's method .. import_group 18:21:59 <hub_cap> cuz what yer doing is using a cfg optgroup, and your code would have to look like cfg.mysql.mount_point right? 18:22:06 <SnowDust> import_group reads the module entry point to import the opts .. 18:22:21 <SnowDust> yeah hub_cap .. end of that 18:22:35 <SnowDust> its CONF.mysql.mount_point .. just as rigth now 18:22:49 <hub_cap> so you plan on changing that in all the datastores, correct? 18:22:59 <SnowDust> hub_cap right 18:23:08 <SnowDust> even which are external to the trove code 18:23:12 <hub_cap> ok so what happens when i wnat to use the default? 18:23:25 <hub_cap> SnowDust: thats not true at all, those are all over the trove guest code and other places 18:23:40 <hub_cap> if u look at a datastore impl it needs to use conf values, right? 18:23:49 <hub_cap> and those conf values are not changed 18:23:56 <SnowDust> hub_cap they are also loaded in cfg.py 18:24:00 <hub_cap> but there is also no backwards compatibility 18:24:04 <SnowDust> so .. default are available all the time 18:24:07 <hub_cap> how do i, SnowDust , which isthe question we are asking 18:24:12 <hub_cap> right but i have to code it for every one 18:24:13 <hub_cap> right? 18:24:13 <SnowDust> when ever we import from trove.common import cfg 18:24:30 <hub_cap> if not cfg.mysql.mount_point use cfg.mount_point 18:24:38 <SnowDust> we have done that only in current cfg.py for all the datastores 18:24:42 <hub_cap> if not cfg.#{datastore}.mount_point use cfg.mount_point 18:24:44 <SnowDust> there is not common defaults now .. 18:24:59 <SnowDust> hub_cap .. there is no cfg.mount_point now 18:25:03 <SnowDust> default have been removed 18:25:15 <hub_cap> ok so maybe mount 18:25:18 <hub_cap> point is a bad example 18:25:19 <SnowDust> i pushed a patchset on that question only .. then were suggested to go for this BP 18:25:48 <SnowDust> hub_cap .. let me share the abandoned patchset.. which wanted to restore the cfg.mount_point but was turned off 18:25:52 <hub_cap> no 18:25:57 <hub_cap> lets use backup_strategy 18:26:33 <hub_cap> what i think we want to see, is the ability to use these values but not put a bunch of if'stmts for each "default" vs a datastore specific 18:27:01 <hub_cap> i suggest u look @ the configuration object in cinder 18:27:20 <hub_cap> it handles it pretty transparently, but can likely be updated to be better :) 18:27:41 <hub_cap> kevinconway: SlickNik et al do yall feel better about it assuming that 18:27:58 <SlickNik> #link https://github.com/openstack/cinder/blob/master/cinder/volume/configuration.py 18:28:00 <hub_cap> * if we dont find a datastore specific, we use the one in the [DEFAULT] group 18:28:11 <SnowDust> hub_cap .. :) 18:28:24 <SnowDust> they all turned off .. the fallback 18:28:31 <SnowDust> amcrn, r u there? 18:29:08 <SlickNik> SnowDust: He had to step out. 18:29:11 <esp> hub_cap: I think there was some question regarding if we still consider mysql as the default datastore impl 18:29:24 <kevinconway> ok so we could provide datastore specific overrides/special entries and then fall back to default configs when they aren't present? 18:29:43 <kevinconway> would that allow for config options in mongo that don't exist in mysql? 18:29:56 <kevinconway> or do they all *have* to exist in DEFAULT? 18:29:56 <hub_cap> sure kevinconway 18:30:01 <esp> it probably makes sense to have a default datastore 18:30:08 <hub_cap> if they dont exist in default then we fail 18:31:15 <grapex> hub_cap: Wait, are you saying we can't add datastore specific config options? 18:31:21 <esp> hub_cap: meaning if you don't specify a required config for your datastore we fail? 18:31:28 <hub_cap> no not at all 18:31:31 <hub_cap> this is how configs work 18:31:34 <hub_cap> 1) in memory 18:31:36 <grapex> hub_cap: Ok, just checking 18:31:44 <hub_cap> 2) overrides from disk if avail 18:31:48 <hub_cap> 3) failures 18:32:07 <hub_cap> opt groups get funky since u have to identify them 18:32:17 <hub_cap> oslo wont "Default" for u 18:32:42 <hub_cap> if u specify cfg.somegrp.someval, if someval is in default, it wont pull it 18:33:08 <hub_cap> itll look for [somegrp]...someval=blah, and if it doesnt find, itll use whats in memory for that value, if any 18:33:15 <cp16net> sounds like something we could put a bug in oslo for? 18:33:16 <hub_cap> only for that memory in that optgroup 18:33:22 <hub_cap> its not a bug 18:33:30 <hub_cap> its the design of using different optgroups 18:33:38 <grapex> hub_cap: Ok- I thought you were saying oslo could be smart enough to use someval from default is cfg.somegrp.someval wadn't found 18:33:40 <grapex> However 18:33:43 <SnowDust> hub_cap : so the current BP implements the defaults 18:33:53 <hub_cap> we could do cfg.DEFAULT if we wanted :) 18:33:53 <SnowDust> i said when cfg is imported 18:33:57 <cp16net> ok 18:34:03 <grapex> you could have code in a datastore that checked to see if the group specific default was not None, and if it was try the default one 18:34:10 <SnowDust> the load_datastore_opts() 18:34:13 <SnowDust> function is called 18:34:20 <SnowDust> which loads the mapped datastores 18:34:25 <hub_cap> grapex: or just put it in a wrapper object, like configuration object 18:34:35 <grapex> hub_cap: Yeah 18:34:38 <hub_cap> u pass in None, or the optgroup, like 'cassandra' 18:34:42 <hub_cap> and then just do configuration.blah 18:34:47 <hub_cap> instead of cfg.blah 18:35:29 <SlickNik> So not getting caught up in the implementation details, and coming back to the bp. 18:35:38 <hub_cap> yes :) 18:35:59 <hub_cap> i think its ok to move the options to config groups for datastores 18:36:10 <hub_cap> assumign we can interact w/ them in a easy way ;) 18:36:14 <SlickNik> I think that the bp does simplify some of the datastore config value loading. 18:36:52 <hub_cap> id like to see a 2nd patchset w/ options actually updated before i can merge p1 :) 18:37:26 <hub_cap> as in a dependent patch set that shows the code updating, say, mysql, to use the optgrp 18:38:36 <SnowDust> hub_cap int_tests just show that what options are being selected for mysql 18:38:48 <hub_cap> but as it stands, im ok w/ this 18:39:52 <SlickNik> Same here, I'm okay with the bp. 18:40:05 <SlickNik> The implementation details might need some tweaking. 18:40:17 <grapex> Ok- sorry 18:40:29 <grapex> this all seems simple but I feel like I'm fuzy on some of the details we're talking about 18:40:37 <SlickNik> And I think it'd be easier to make some of these comments in gerrit. 18:40:45 <grapex> SlickNik: +1000 18:40:47 <hub_cap> yea after looking more at the code, it looks like we are already calling cfg.get(datastore_mgr).tcp_ports 18:40:52 <hub_cap> thats the only place we are doing it 18:40:52 <grapex> just to be sure, none of us want to load all of the datastore options in the common/cfg.py right? 18:41:18 <grapex> robert just pointed out on the last line here, it loads all of the config options for everything which seems a bit much: https://review.openstack.org/#/c/80061/5/trove/common/cfg.py 18:41:48 <SnowDust> grapex: yes 18:41:51 <robertmy_> all other openstack projects do not do this 18:41:55 <SnowDust> grapex: i suggested in last discussion 18:41:57 <hub_cap> robertmy_: ? 18:41:59 <SnowDust> this may be further trimmed 18:42:06 <SnowDust> if we want to do it from conf files only 18:42:08 <hub_cap> everything exists in a big global cfg 18:42:09 <robertmy_> they use the option groups in the file 18:42:17 <robertmy_> hub_cap: no 18:42:18 <grapex> SnowDust: Sorry, I guess we didn't all see that. 18:42:21 <hub_cap> robertmy_: tahts what he says this is doing 18:42:23 <SnowDust> so that only the one which is being enabled gets loaded 18:42:32 <hub_cap> robertmy_: this is a change then :) 18:42:58 <grapex> hub_cap: I don't get the reasoning 18:43:05 <grapex> why force all the config options to load when cfg.py is parsed? 18:43:14 <hub_cap> well wait up, i need robertmyers to explain 18:43:17 <robertmyers> let me find an example 18:43:34 <hub_cap> plz do, cuz last i thought, everyon grabs cfg.CONF 18:43:45 <hub_cap> and its parsed on load (thats how nova used to work) 18:43:48 <hub_cap> again, *used to* 18:44:07 <hub_cap> ok so maybe we need to move this convo out of here 18:44:09 <hub_cap> and move on 18:44:27 <grapex> hub_cap: https://github.com/openstack/nova/blob/master/nova/api/auth.py#L46 18:44:35 <robertmyers> https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/san/san.py 18:44:57 <robertmyers> so you see the option groups are defined at the point they are used 18:45:00 <hub_cap> right its still a global CONF 18:45:02 <grapex> That's what robert_my means- auth_opts are simply defined next to where they are used, and no special code exists to parse them as the root cfg.py is created 18:45:22 <hub_cap> oh u mean just the loading.. yea that can be taken out of the global cfg stuff 18:45:32 <hub_cap> there is no need for them to be loaded before their impls are loaded 18:45:42 <robertmyers> hub_cap: it is a global config file, but not a global object 18:45:54 <hub_cap> cfg.CONF ?? 18:46:11 <hub_cap> im pretty sure that using CONF everywhere defines a global obj right? 18:46:20 <robertmyers> well, it all reads/extends cfg.CONF 18:46:24 <hub_cap> my python-speak may be wrong 18:46:36 <hub_cap> everything uses CONF right? 18:46:38 <robertmyers> technically yes 18:46:54 <hub_cap> ok i think we are speaking the same thing 18:46:59 <robertmyers> but, it is lazy loaded 18:46:59 <hub_cap> i agree we dont need to "preload" Them 18:47:04 <robertmyers> ok 18:47:12 <hub_cap> the point of cfg is to allow u to define groups in the files they are used (or imported) 18:47:14 * grapex wipes sweat off his brow 18:47:18 <kevinconway> i think we should change the conf files to YAML 18:47:30 <hub_cap> kevinconway: if i only had /kick privs 18:47:31 <robertmyers> xml 18:47:35 <grapex> kevinconway: https://www.youtube.com/watch?v=4DgbUBoxa48 18:47:48 <hub_cap> so lets remove the preload stuff SnowDust 18:47:51 <hub_cap> and just let the app load it 18:48:05 <pdmars> lol 18:48:13 <hub_cap> and id like to see you update an impl to use more than just whats defined now in the optgropu, which is tcp_ports 18:48:26 <hub_cap> unless thats done 18:49:02 <hub_cap> we are already doing that in places 18:49:07 <hub_cap> so then we dont need to see that done 18:49:24 <hub_cap> so just removing the autoload will satisfy everyone, right? 18:49:39 <hub_cap> and we can make a different bp for the "fallback" to default 18:49:43 <grapex> hub_cap: aye 18:49:58 <SnowDust> hub_cap u mean i should not "REGISTER" the groups ? 18:50:06 <SlickNik> +1 on the lazy loading. I have a couple other comments, I'll add them to gerrit :) 18:50:11 <hub_cap> SnowDust: u register the opts/group _in_ the file 18:50:15 <robertmyers> +1 18:50:16 <hub_cap> whih i thought u were doing already 18:50:57 <hub_cap> https://review.openstack.org/#/c/80061/5/trove/guestagent/datastore/cassandra/options.py 18:51:06 <hub_cap> see you are already registering the group and opts 18:51:21 <hub_cap> so if u just completely remove all the code from cfg, itll work 18:51:37 <kevinconway> hub_cap: all the code?!? 18:51:42 <hub_cap> ALLL 18:51:44 <SnowDust> i dont get 18:51:53 <hub_cap> SnowDust: offline 18:51:57 <hub_cap> lets move on we have 9 min heh 18:52:07 <SnowDust> until u register a group 18:52:07 <SnowDust> how do u use the group options ? 18:52:12 <hub_cap> SnowDust: offline 18:52:14 <hub_cap> lets move on we have 9 min heh 18:52:27 <hub_cap> #topic remove mockito 18:52:28 <SnowDust> hub_cap i do register them in the options.py of datastore module 18:52:28 <SnowDust> yes 18:52:34 <hub_cap> #link https://blueprints.launchpad.net/trove/+spec/remove-mockito 18:52:37 <SnowDust> but we need to import the group in cfg 18:52:41 <hub_cap> so whos doing this? 18:52:45 <SnowDust> so that cfg.CONF 18:52:45 <SnowDust> can then be used to call them 18:52:45 <hub_cap> SnowDust: offline 18:52:50 <hub_cap> SnowDust: offline!!!! 18:52:51 <SnowDust> using CONF.mysql.root_on_create etc 18:52:51 <SnowDust> hub_cap ? 18:52:53 <hub_cap> SnowDust: offline!!!! 18:52:57 <SnowDust> hub_cap sure ! 18:52:57 <SnowDust> hub_cap : sure ! 18:53:09 <SnowDust> thanks ALL :) 18:53:23 <hub_cap> SnowDust: u will have to reiterate Q offline plz 18:53:26 <hub_cap> SlickNik: is this u? 18:53:32 <SlickNik> yup, I added this 18:53:34 <hub_cap> whos tackling removing mockito? 18:53:51 <juice> i can lend a hand 18:54:01 <SlickNik> I looked into this yesterday, probably gonna be juice and me. 18:54:50 <hub_cap> cool, anything to add to it? 18:54:52 <SlickNik> Lots of mockito in the tests, probably gonna take 2-3 days to get it done. 18:54:56 <hub_cap> yea for sure 18:55:05 <esp> SlickNik: is there any plan to remove mox? 18:55:25 <juice> esp: I guess mox is still in the mix 18:55:28 <SlickNik> esp: right now it's not an issue since mox exists in the global_requirements. 18:55:29 <robertmyers> do we use mox? 18:55:31 <juice> but we shouldn't do it 18:55:48 <juice> robertmyers: I think there was some mox in the trove code 18:55:56 <esp> robertmyers: I think there is some unit tests in python-troveclient that may use mox 18:55:57 <SlickNik> robertmyers: I haven't seen it in many places; but I haven't looked too hard. 18:56:13 <esp> is the one from google I believe 18:56:24 <robertmyers> let not use it 18:56:26 <esp> *it's 18:56:28 <SlickNik> Timeline wise, I think we might have to do this before the icehouse cut. 18:56:31 <esp> robertmyers: +1 18:56:45 <hub_cap> yes SlickNik lets focus on mockito 18:56:50 <hub_cap> and make esp do the mox stuff 18:56:53 <hub_cap> ;) 18:56:58 <SlickNik> esp: thanks! :) 18:56:59 <esp> hehe 18:57:02 <esp> np 18:57:06 <SlickNik> that's all I had. 18:58:11 <juice> i have to run - talk to you guys later 18:58:43 <SlickNik> juice: I'll catch you offline to chat about mockito. 18:58:45 <SlickNik> thanks! 18:59:13 <hub_cap> ok so the last topic 18:59:14 <hub_cap> no time 18:59:18 <esp> lol 18:59:20 <hub_cap> who put it on? 18:59:30 <hub_cap> we can move to #openstack-trove to discuss 18:59:35 <hub_cap> anyone? 18:59:43 <SnowDust> yep 19:00:06 <hub_cap> ok so if anyone wants to discuss container vs join, goto the channel 19:00:09 <hub_cap> #endmeeting