18:01:16 #startmeeting #trove 18:01:17 5hello 18:01:17 Meeting started Wed Dec 4 18:01:16 2013 UTC and is due to finish in 60 minutes. The chair is hub_cap. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:19 hello 18:01:20 /o/ 18:01:21 o/ 18:01:21 The meeting name has been set to '_trove' 18:01:23 o/ 18:01:23 \o\ 18:01:24 o/ 18:01:27 \o/ 18:01:28 o/ 18:01:28 \o 18:01:29 o/ 18:01:33 o7 18:01:33 o/ 18:01:37 o/ 18:01:38 o/ 18:01:40 #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting 18:01:40 here 18:02:10 o/ 18:02:27 whazzap 2 all 18:02:37 #topic separate guest agent 18:02:41 denis_makogon: go 18:02:41 )))))))) 18:03:05 #link https://github.com/denismakogon/trove-guestagent 18:03:09 o/ 18:03:32 last few days i was working on extracting guestcode into it's own repo 18:03:49 manually i ran all testcases 18:03:49 awesome denis_makogon 18:03:51 denis_makogon: nice! 18:03:59 looks good 18:04:04 i mean create/backup/restore/resize 18:04:26 denis_makogon: when we really go about creating the guest, we can use a git subtree to keep the history 18:04:29 for now i've got problems with few tests which are using DB connection 18:04:50 o/ 18:05:14 hub_cap: +1 18:05:27 +1 subtree 18:05:31 +1.5 18:05:31 ))))) 18:05:37 +1 yeah let's preserve that history 18:05:38 so, question is: do we plan to integrate guestcode ? 18:05:46 hub_cap: Yeah, that history goes back a year and half- let's do the subtree thing. 18:05:50 denis_makogon: yes, it should be merged with trove repo 18:05:53 another thing denis_makogon. Can you split off the git subtree, rather than have one big initial commit? This preserves git history... 18:06:25 denis_makogon: but first separate it, then merge it back in 18:06:30 SlickNik, i would try 18:06:32 ok cool. this brings up an interesting question, if i want to make changes to my guest, and i edit my local code, how does that code get into the instances i provision? 18:06:36 im worried less about tests, cuz the openstack pypi stuff is set up to handle this, but for day to day devrlopment 18:07:12 maybe git clone/pull ? 18:07:24 denis_makogon: its necessary 18:07:36 plz figure out how to do it and use that as the repo 18:07:41 i guess you'd have to set up your local repo as the authority 18:07:50 hub_cap: Do you mean when you're developing? Doesn't the rsync trip pull that in? Just point it somewhere else. 18:08:24 Seems like this is yet another thing that will live in /opt/stack 18:08:43 add it to devstack? 18:08:55 grapex: fair enough, but the "delivery" part is what hes talking about 18:09:01 and rsync will pull it from there; devstack will clone it if it doesn't exist, and people will make it a shared folder if they're using a VM 18:09:05 datsun_F40PH: when will vagrant be updated for this? 18:09:19 grapex, yes, i was playing with redstack and guest 18:09:24 kevinconway: when someone submits a pull request and i feel like merging it 18:10:22 #action denis_makogon: use git subtree for guestcode 18:11:05 denis_makogon: it looks like you made changes to agent after moving it out? 18:11:20 ok so what do we have to make decisions on here? 18:11:24 kevinconway, clean-up 18:11:35 was it needed to move the agent? 18:11:50 +1 that concern 18:12:01 do not do any cleanup unless its necessary 18:12:07 it's great to clean it, just make sure to keep the original branch unchanged 18:12:07 kevinconway, i was trying to keep agent code up-to-date wit trove repo 18:12:12 we can clean up in a second commit 18:12:30 ok, i will revert all my own changes 18:12:53 denis_makogon: when you do the subtree merge and restore history you can branch off master and re-apply your changes 18:12:56 no big deal 18:12:59 hub_cap, question it that how to keep guest code up-to-date 18:13:00 denis_makogon: don't take matches to it, lean on git to help you maintain your work 18:13:27 ok 18:13:29 thanks 18:13:46 moving on? 18:13:54 yes 18:14:23 #topic configuration parameter deathmatch 18:14:28 cp16net: denis_makogon go go go 18:14:29 #link https://gist.github.com/denismakogon/7789831 18:14:40 lol 18:15:02 i described all problems with current implementation of parameters configuration 18:15:11 * imsplitbit tosses some rusty hatchets into the ring 18:15:15 i'm on team "send the whole file and keep the GA dumb" 18:15:20 #link https://review.openstack.org/#/c/53168/ 18:15:24 I like keeping the guest agent dumb too. 18:15:31 heh denis_makogon next time put this on the meeting agenda so we can view it before hand 18:15:41 sorry 18:15:47 forgot 18:16:18 guest dumbness defined by database 18:16:57 +1 to keeping the guest-agent dumb. 18:17:14 Still reading the gist. Might need clarification on a couple of points. 18:17:17 denis_makogon: can you explain that? 18:17:28 can someone clarify what "keep the guest agent dumb" means in terms of action-items as compared to what's in the current review? 18:17:30 denis_makogon: - do not call configuration groups - overrides, it too mysql specific; 18:17:34 i'm not fully groking this gist 18:17:41 amcrn: It means having the server create the full and send it over 18:17:43 ya ir confused a bit 18:17:44 what do you mean? 18:17:51 amcrn: the alternate would mean the GA gets to figure out templating and filling in the blanks and locations and such 18:17:56 amcrn: Versus sending over only what to change in db specific terms, and having the guest apply changes to the file 18:18:26 grapex, Amazon RDS doing merging 18:18:53 as i said, only mysql could start with more then one conf file 18:18:58 out of box 18:19:05 not true 18:19:09 redis can 18:19:18 i've researched 18:19:24 but I'd love to hear more about servers that cannot 18:19:28 prof ? 18:19:29 redis can use more than one conf file. 18:19:40 it? 18:19:50 grapex: if it's moved to the server, will it still consult the ga for the source-of-truth, therefore requiring multiple roundtrips? 18:19:53 denis_makogon: True they do, however they don't support multiple datastores either. 18:19:56 imsplitbit, mongo, cassandra, postgresql 18:20:12 so the payload that comes from the server includes a whole new config or just a partial? 18:20:29 grapex: i.e. ask the ga what the default values were, then merge the change-set, then push the entire set 18:20:38 i vote for only part 18:20:51 amcrn: Not sure- last I checked, there's no round trip. 18:20:53 denis_makogon: postgresql and redis can do a conf.d w/ 1 line in their configs 18:20:56 cp16net: Can you confirm? 18:21:12 well now there isn't, but if it's moved from ga to server, what will it be 18:21:16 i think just providing a "get the current config" call would suffice 18:21:36 let someone smarter than six lines of awk do the merging work 18:21:58 datsun_F40PH: +1 18:22:02 datsun_F40PH, that's the problem, because there could be special config changes to make DB available for any hosts 18:22:07 no merge 18:22:12 push the entire config down 18:22:12 denis_makogon: details? 18:22:17 denis_makogon: in that case you employ a DBA to make those changes 18:22:18 robertmyers: did you just volunteer yourself? :) 18:22:22 there is no need to "merge" 18:22:26 SlickNik: sure 18:22:30 are we talking about database config like a my.cnf or guest agent config trove-guestagent.conf 18:22:32 amcrn: The server is assumed to be the source of truth currently. 18:22:42 vipul: my.cnf 18:22:51 vipul: my.cnf 18:22:54 my.cnf 18:22:56 k thanks 18:23:00 MY.CNF 18:23:03 :p 18:23:04 just in case u didnt get that vipul 18:23:13 other than thinning down the guestagent, what is this buying us? 18:23:28 the power to tune 18:23:42 suppose, i need to set bind address to eth0 inet addr 18:23:57 amcrn: A better question is what's it costing us? 18:24:01 but server didn't knew, that guest doing such thing 18:24:25 denis_makogon: why is the guest changing the config? 18:24:33 denis_makogon: Would you do that via the API? 18:24:34 denis_makogon: it would be the same process as when the guest first wrote the config 18:24:36 it must just be me, i'm having a hard time summarizing the positive net benefit of doing this (vs. not doing this) 18:24:37 simple example https://review.openstack.org/#/c/51884/19/trove/guestagent/datastore/cassandra/manager.py 18:25:38 amcrn: you're not alone, I'm trying to think through the exact same thing. 18:25:55 i'm getting a little lost in this convo 18:26:19 cp16net: +1 18:26:23 ok amcrn re-summarize 18:26:24 current design works only with databases which could start with more then one conf file 18:26:50 if there is no net gain/loss, then we dont change anything until we need it <--- general rule 18:26:52 so you are saying that trove would not work with DBs which cannot do such thing ? 18:26:58 amcrn: I think it makes the guest API much simpler. 18:27:20 im agree with dmakogon, will better to send array with K:V parameters through rpc, and render config in manager class 18:27:42 ashestakov_: why is that better? 18:27:54 denis_makogon: Would it though? It seems like if the db had only one conf file, you could just overwrite that file every time 18:27:54 i can send a string or a dict.. what is the difference? 18:28:09 Unless the idea is there is some secret stuff on the guest machine you don't want to overwrite 18:28:09 +1 to grapex's comment, that's where i'm lost 18:28:14 hub_cap: for DBs which dont support multiple configs, manager can render all settings to one file 18:28:31 its more flexible then send renderd configs 18:28:56 why not just send one file then? 18:29:03 the whole config 18:29:04 or one file.. 18:29:06 suppose trove creates database, why do i need reconfigure it to make it available for r/w operations ? 18:29:14 ashestakov_: I disagree, rendered configs could be generated in nearly any way- there's nothing that's unflexible about it except that it replaces the entirety of what's on the guest machine 18:29:16 except then the rendering occurs in the guest and that cost more memory wise, and u need the templates in the guest 18:30:02 oh, i think i get the gist of what denis_makogon is saying. example: if you have a datastore that doesn't support multiple confs, say you apply some changes with configuration "a". if the user sends in configuration "b" as an update, it's not straightforward to ascertain what the original/virgin/default conf was. 18:30:27 maine point: need to render config depends on manager class, for mysql can has my.cnf and conf.d/my.cnf 18:30:29 unless the datastore has a operation to say "print-defaults" of course 18:30:33 amcrn: u can regen the "default" from the config template w/ the updates passed in 18:30:35 so then it's up to each impl's GA to figure out what "all the configs" means 18:30:37 amcrn: So when you first do those changes with configuration "a", who or what does that? 18:30:43 that's part of documentation isn't it? 18:30:43 but for mongo we can have only /etc/mongodb.conf 18:30:57 maybe it's a list of different file locations and their respective contents 18:31:09 ashestakov_: why couldn't guest agent just write new config to /etc/mongodb.conf? 18:31:12 we have the default conf from rendering the conf again 18:31:17 hub_cap: today you can't, because only the overrides are stored in the database 18:31:22 ok i think we need to move forward and push this to after the meeting or the ML 18:31:24 I'm wondering if we shouldn't really hash this out more on the ML 18:31:29 it's 12:30 18:31:31 thats nothing more than whats happening in the create instance 18:31:37 or halfway through the meeting 18:31:43 kevinconway: coz, we have template with common settings and user defined settings 18:31:46 amcrn: but the config.template is stored in the taskmgr too 18:31:58 hub_cap: oh yeah, whoops ;) 18:32:04 so why dont we just regen each config.template + diff'd parameters and send that file down 18:32:16 so if all the overrides are in the db, couldn't you just send out a default conf + _all_ overrides to the guest agent. 18:32:20 single file should be ok 18:32:23 That way it gets a + b 18:32:27 i think SlickNik and i said the same thing 18:32:31 kevinconway: sure we can render all settings in one file, but this should be done for all datastores 18:32:39 that approach generally makes sense to me 18:32:58 hub_cap, why not to send dict with options and let manager decide how to apply it ? 18:33:04 ashestakov_: this seems like only an issue of how to get the new config to the agent. the agent can do what it wants with it. 18:33:20 what do they have to decide? 18:33:26 denis_makogon: In addition to making the guest smart, sending a dict down means you assume that everything will be in some format that will be easy to modify using a dictionary 18:33:32 if, always, we send down 1 file, what decision is to be made? 18:33:37 So it's a different assumption 18:33:41 +1 hub_cap grapex 18:33:53 ok but srsly 18:33:55 It sounds like we need to keep this more flexible 18:34:00 moving on. lets punt to after meeting 18:34:03 let's move on. 18:34:06 +1 18:34:07 good topic :) 18:34:07 ok 18:34:08 so we should send down a file- but maybe we should allow the guest agent to apply it differently 18:34:08 even if you decide that we'll only send one file we can't stop an intrepid tuner from doing that themselves 18:34:09 def 18:34:12 and then what do we do 18:34:12 hub_cap: can we do ML and not post-meeting? 18:34:28 grapex, that it what i say 18:34:28 +1 ML 18:34:30 +1 18:34:33 +1 ML 18:34:34 but i guess once you enable root all bets are off anyway 18:34:36 ++ML 18:34:39 this is a big discussion 18:34:48 #topic compat client changes 18:34:51 datsun_F40PH: Enabling root is like spitting in the face of our precious API! 18:35:01 grapex: and yet still allowed 18:35:04 but i would not prefer to use raw string, vote for dict 18:35:10 so there was a review that made changes to the compat client for pep 18:35:28 #link https://review.openstack.org/#/c/54900/ 18:35:42 i've updated review 18:35:44 today 18:36:02 well i dont think weve made a decision on what to do denis_makogon 18:36:08 i took into accout all comments (alph. sort, etc) 18:36:24 did u take into account why i -2'd it? :) 18:36:36 which is why we are talking about compat.client )) 18:37:01 so i -2'd it cuz we said compat.client would not be updated (well i thought so) 18:37:10 but grapex has made a great point, that it is used 18:37:18 its used in all our integration tests 18:37:20 hub_cap, -2 only for discuss 18:37:34 for this discussion denis_makogon 18:37:44 I talked to our boss and we'll probably get someone around Rax to update the int tests to use the newer stuff soon, and will also figure out a way to add the XML support into the newer code paths in an OpenStacky way. 18:37:45 yes =) 18:37:51 Sometime this month hopefully. 18:38:07 we still need the compat stuff to be able to do management api stuff 18:38:18 grapex: ok cool. vipul good point 18:38:24 i only do PEP8 checks 18:38:24 so until that's sorted out it seems fair to keep it up to date 18:38:30 im ok w/ altering this since its still in use 18:38:32 good point, vipul 18:38:42 hub_cap vipul: Holy crap, there's no mgmt api stuff in the new CLI? 18:38:45 altering = modifying the code 18:38:49 since it is still used today 18:38:51 grapex: you must not use it :p 18:38:54 grapex: yes, that's the case. 18:38:57 vipul: lol 18:38:58 GUILTY 18:39:01 lol 18:39:15 grapex: not yet.. the v1 code is ther its just not in the cli 18:39:19 who needs management anyway 18:39:20 so u can still program to it 18:39:27 Well we started using it a few weeks back, but only for non-mgmt stuff 18:39:28 u just cant $trove blah.... 18:39:35 I figured it was general auth related shaningans 18:39:45 Ok, then we'll try to add that in as well 18:40:03 thx grapex. 18:40:07 if that gets sorted out.. we should just nuke the compat code 18:40:14 vipul: correct 18:40:18 vipul: Agreed 18:40:40 we can put it as a branch so that it is still around for someone to use it 18:40:49 and NUKE it 18:40:55 lol 18:40:58 vipul / hub_cap ++ 18:41:05 hub_cap: that is what Pypi versions are for 18:41:33 robertmyers: fair enough, we can push one more update (itll tag the code in gh anyway) 18:41:38 sure 18:41:47 ok so, decision is that rax will clean up the mess i made 18:41:53 hah 18:42:01 seems fitting to me 18:42:02 more like hub_cap will clean up the mess you made 18:42:07 >:C 18:42:23 #action datsun_F40PH to clean up the mess hub_cap made 18:42:27 #undo 18:42:30 hub_cap: Lol 18:42:31 lol 18:42:32 i put a space 18:42:58 wow i just refershed the wiki and its spinning... not loading 18:43:01 whats the next topic? 18:43:04 ah its up 18:43:06 yup 18:43:13 #topic granular user privs 18:43:16 datsun_F40PH: daz u 18:43:17 yes 18:43:20 i'd like to do it 18:43:24 go 18:43:29 ahead 18:43:29 and if nobody argues with my bp i'll do it like i've proposed 18:43:36 link it plz 18:43:43 #link https://blueprints.launchpad.net/trove/+spec/user-privilege-control 18:43:52 there you go 18:44:07 * denis_makogon fast as bolt 18:44:24 well this is a good sign 18:44:40 so currently the user we create has grant all 18:44:49 why couldn't they do this via mysql 18:45:00 why couldn't they create users or dbs via mysql 18:45:10 vipul, +1 18:45:17 fair enough 18:45:31 i caution against re implemetnign more mysql specific things in our api 18:45:42 well this would be buried in mysql for now 18:45:44 is this not an extrention? 18:45:45 vipul: who says its MySQL only 18:45:51 but do we really need such functionality? i mean users, dbs ? 18:46:07 well i don't mean mysql specific.. i should have said things you can do via the database protocol itself 18:46:14 denis_makogon: this is an extension 18:46:18 did somebody say users?!? 18:46:26 kevinconway: shhh 18:46:27 i thought it came from Amazon RDS 18:46:30 oh heck 18:46:54 apparently it's been requested by users of our api at rax and i'd just like to implement it at least for mysql 18:47:25 since, cards on the table, i work at rax 18:47:26 Seems to me that it would be different depending on the datastore_type. So seems like a datastore-specific extension. 18:47:34 of course it would 18:47:40 ask redis about users 18:47:50 NO USERS!!! 18:47:50 c: 18:47:51 SlickNik, +1 18:47:55 :0 18:47:57 :) 18:48:12 =[E] 18:48:15 so i seek to implement this as a mysql extension primarily 18:48:28 i thought we were a provisioning api -- not a data api -- this is kinda in the middle 18:48:33 there is no reason to argue about this 18:48:35 since this is an extension, I am not sure that we need to have an argument over if it is needed or not 18:48:38 weve talked about it a billion times 18:48:41 this is an extension 18:48:49 its not a core api 18:48:50 moving on 18:48:51 datsun_F40PH: can you add this enhancement, but limit it to the user/databases extensions vs. adding it in instance-create? 18:49:15 and kevinconway, no users 18:49:15 agreed with vipul, Data API is for Dynamo/Simple DB 18:49:25 amcrn: yeah i'm not a fan of the all-in-one instance create personally 18:49:31 excellente' 18:49:52 amcrn: all-in-one instance create will probably not be in v2 18:49:53 so i'll only be messing with user calls 18:50:00 we would also need a way of limiting which privileges are 'assignable' 18:50:03 hub_cap: hooray! 18:50:17 the goal is not to duplicate the MySQL protocol, I get the arguments about creating a data API 18:50:18 amcrn: unless someone can do it well, generic, not ugly 18:50:29 :) 18:50:35 ok core team 18:50:41 do we approve this? im perfectly fine w/ it 18:50:51 but for basic functionality around users and databases, there is value in exposing via the API as these are very common tasks when setting up the DB 18:50:52 sure what the heck 18:50:58 ill approve the BP if no one has concerns 18:51:00 i welcome discussion and suggestions in the appropriate places 18:51:03 HAHAHAAHAHAH vipul 18:51:10 that's confidence! 18:51:14 i would rather we debate how to implement it 18:51:20 but I feel like most people don't care :) 18:51:36 unfortunately we don't have a good way to turn off extensions 18:51:38 so there's that 18:51:44 vipul: fix it ;) 18:51:47 vipul: we can talk about permission masks 18:52:00 datsun_F40PH: ok cool 18:52:18 if u want to turn off all the extensons, just change the path.. likely if u dont want users extension, u wont want root or databases either :P 18:52:20 hub_cap: yea will have to 18:52:28 i could take care of extensions 18:52:32 yea i totaly agree its crap 18:52:33 hub_cap: we don't want to turn off all 18:52:49 denis_makogon: ok thats a good idea. its a big upgrade 18:52:55 i want to see it done in multiple patchsets though 18:53:04 1000 lines of code in 1 patchset is too painful :) 18:53:07 so i'll begin to work on this 18:53:09 +1 18:53:18 hub_cap, ok 18:53:23 datsun_F40PH: make sure it mirrors the way that nova/cinder/etc do it 18:53:26 denis_makogon: ^ ^ 18:53:29 whoops datsun_F40PH 18:53:34 ok lets move on then? 18:53:40 hub_cap: i'll consider it but my hands may be tied 18:53:48 denis_makogon: there should already be a blueprint fyi 18:53:49 i'll do a bp for that 18:53:59 thanks denis_makogon 18:54:05 datsun_F40PH: your stuff will be ok... its not what denis is working on 18:54:07 refactoring for better pluggability ? 18:54:07 since we have a contract to uphold 18:54:25 those was mine 18:54:32 oh i see now 18:54:37 ok moving on then? 18:54:40 please 18:54:40 yes 18:54:49 #topic datastores fast follow 18:54:50 amcrn: go go go 18:54:54 #link https://blueprints.launchpad.net/trove/+spec/datastore-type-version-followon 18:55:11 gist: there were a couple conversations around some enhancements to the datastores implementation 18:55:22 this is an attempt to catalog them 18:55:40