20:02:15 #startmeeting trove 20:02:16 Meeting started Wed Sep 4 20:02:15 2013 UTC and is due to finish in 60 minutes. The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:02:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:02:19 The meeting name has been set to 'trove' 20:02:22 ;/ 20:02:23 o/ 20:02:25 ugh 20:02:29 what is the topic ? 20:02:31 Hello all 20:02:33 hi 20:02:37 hola! 20:02:45 Agenda is at https://wiki.openstack.org/wiki/Meetings/TroveMeeting#Agenda_for_the_next_meeting 20:02:59 707 20:03:06 Please update it if you have a topic you want to discuss. 20:03:11 o/ 20:03:21 already 20:03:22 o/ 20:03:26 #topic Update to Action items 20:03:39 #link http://eavesdrop.openstack.org/meetings/trove/2013/trove.2013-08-28-20.02.html 20:03:54 isviridov: hi 20:04:04 hi, everybody 20:04:10 hi 20:04:11 hub_cap to find out what happens w/ feature based reviews that land after FF 20:04:13 hai 20:04:15 hi 20:04:26 hub_cap is currently flying 20:04:34 in an aeroplane 20:04:44 Yeah. He told me to cover for this one. 20:04:49 trove, guys, let's talk about it)) 20:05:26 So after FF the branch remains frozen until we cut our first release candidate (RC1) 20:05:41 yeah, that is how it works) 20:05:47 Which should happen in about a week or so. 20:06:01 friday - H3 20:06:10 next friday RC 20:06:22 Once we have RC1 cut, we open the branch again for icehouse submissions. 20:07:16 any questions? 20:07:31 okay, moving on... 20:07:31 works for me 20:07:43 SlickNik to leave current patch as is and investigate adding trove role to default devstack users in another patch. 20:08:06 So that's what most of the other projects are doing. 20:08:17 So I will be adding a follow up patch once the devstack patch lands. 20:08:33 #link https://review.openstack.org/#/c/38169/ 20:09:25 datsun180b: you're up. 20:09:36 trove-conductor update. 20:09:42 Working on submitting my wip for trove-conductor now. 20:10:11 It's not complete, but I have a daemon listening to a queue, and a guestagent that speaks to a queue 20:10:49 Sweet. Looking forward to the review. 20:11:04 any updates about Conductor ? 20:11:06 Well, it's not complete. I need to hand it off, as I'll be gone all of next week 20:11:20 Hence a WIP review 20:11:21 hello sorry i'm late 20:11:22 datsun180b: I'll try to pick that up (then I may pawn if off later though) 20:11:26 * grapex laughs sinisterly 20:11:49 Fine by me, it'll be in good hands 20:11:50 any blocking stuff for Conductor ? 20:12:03 heh, gotcha datsun180b 20:12:19 dmakogon_: mostly configuration woes as i'm green to rabbit 20:12:41 that's all for me, you'll see the reviews when i submit them 20:13:00 With Rabbit it always seems like nothing works, until you get the last config tweaked perfectly 20:13:00 Thanks datsun180b. Sounds good, let's move on… 20:13:19 datsun180b: ok 20:13:46 datsun180b, any link to code? 20:13:55 Okay, next: team needs to discuss and come up with a consistent JSON notation across API 20:14:04 datsun180b: could you paste any code or specification of Conductor ? 20:14:04 datsun180b: code plox? 20:14:39 in the review guys 20:14:40 dmakogon_ /isviridov: datsun180b's in the process of posting his WIP review. 20:14:46 you'll seeit all 20:15:02 ok 20:15:24 So we still need to discuss having a consistent JSON notation. 20:15:34 And haven't had a chance to do it so far. 20:15:40 SlickNik: consistent JSON spec for the REST API? 20:15:45 about JSON: lets formalize notation and present it OpenStack community 20:15:55 present to 20:16:00 i thought hub_cap already did that 20:16:25 dmakogon_: i think that's been done in the past, but gone nowhere.. no harm in trying again i suppose 20:16:26 grapex: yes. right now we have some variable using under_scores and others using camelCase... 20:16:43 SlickNik: hub_cap made a wiki article on the next version of the API 20:16:46 what was the decision? 20:16:49 So the idea was we should formalize a consistent spec for Trove. 20:16:51 one feature of the wishlist was to have consistent JSON stuff 20:16:56 vipul: lets formalize JSON notation for trove and use it like standard 20:16:59 At least for the next version of the API. 20:17:05 I would suggest camelCase, as well as w3c uses it http://www.w3schools.com/json/ 20:17:09 i'm pretty sure hub_cap was for the underscored notation 20:17:20 it appears as the general rule is to follow python notation more so than any generic standard 20:17:30 vipul: agree with isviridov 20:17:32 this being the case, we should use underscores 20:18:08 it seems like other openstack projects are divided and inconsistent regarding this as well. 20:18:13 Which sucks. 20:18:14 isviridov: w3schools is not w3 20:18:32 #link https://wiki.openstack.org/wiki/Trove/NextAPI 20:18:41 i'd love to see general consensus across openstack programs ideally. 20:18:57 kevinconway, but they really follows w3c 20:19:25 does one lend itself to dynamic binding more than the other? 20:19:38 i don't believe so, since it's all dicts 20:19:51 +1 on under_score. also, I agree w/ Jay's comments on the current JSON schema (which is not mentioned in that aforementioned wiki) 20:19:55 that is, if I were to run our api through several code generator or object binding frameworks which one works more consistently? 20:20:43 I don't think we need to arrive at a consensus this very moment. I'd like to read up on it a bit more and revisit this when hub_cap is around as well. 20:20:49 +1 arborism 20:20:58 and +1 jay's comment 20:21:09 what comment? 20:21:28 oh recall a email thread... 20:21:45 what is the next topic ? 20:22:13 #topic MongoDB support 20:22:13 We're still doing action items, dmakogon_ 20:22:25 #link http://lists.openstack.org/pipermail/openstack/2013-August/000850.html 20:22:32 thanks vipul 20:22:43 (thanks for the assist on that link vipul, you beat me ;) ) 20:23:24 okay let's move on to the next action item 20:23:34 cp16net add the database model to the scheduled_task wiki 20:23:44 i've yet to do that... 20:24:00 i have it on my list 20:24:19 okay, do you want to re-action it? 20:24:23 yes plz 20:25:11 #action cp16net add the database model to the scheduled_task wiki 20:25:20 ty vipul 20:25:26 thanks vipul 20:25:32 #action team still needs to discuss and come up with a consistent JSON notation across API (hub_cap / grapex / vipul / SlickNik) 20:25:43 Okay that's all for action items. 20:25:46 Let's move on 20:25:49 SlickNik: Ok, who has a coin to flip? :) 20:25:52 souds good 20:26:05 grapex: likely going to come to that lol 20:26:05 #topic Update on h3/rc1/icehouse 20:26:20 grapex: http://www.random.org/coins/ 20:26:25 touched on that earlier. 20:26:29 Nothing more to add. 20:26:48 #topic MongoDB support https://blueprints.launchpad.net/trove/+spec/mongodb-support (needs approvement) 20:27:12 dmakogon_: that's you I believe. 20:27:13 woah MongoDB eh 20:27:30 trove should support MongoDB is popular enough 20:27:32 or isviridov 20:27:43 MongoDB is pretty common DB in NoSQL world, would like to see it in trove 20:28:05 and mongo has very interesting data storing schema 20:28:05 need your approve 20:28:18 isn't a mongodb blueprint jumping the gun considering the clustering api isn't finalized, region support isn't finalized, parameter groups aren't finalized, etc? 20:28:48 I guess you could do a single instance mongo 20:28:51 not sure how useful that will be 20:28:53 no more than adding any other datastores 20:28:58 arborism, you are right, it is one more reason to think about cluster-api better 20:29:14 dmakogon_: I think we need some more info in the blueprint, about the idea and design. 20:29:20 looking at support for it is a great set of data points to help shape all those features 20:29:40 demorris: good point 20:29:56 SlickNik: we'll do that 20:29:58 we run the risk of under delivering on those features if we don't include it in the discussions, so I seem them as able to run in parallel 20:30:02 since Trove is approved to support no-sql, I don't see why this would be an issue 20:30:25 just would be a single instance thing until clusters are fully baked 20:30:33 vupil: yes, why not ? 20:30:51 good point 20:31:12 Rather than having * number of blueprints of what other sql or no sql services we want to support why don't we work on making it a trivial task to add in new db engines.. 20:31:36 for now we could tweak trove for provisioning single instance of anything 20:31:48 cweid: +1 20:31:55 cweid: I think some of those things only get easier when you actually attempt to add another datastore 20:32:06 Correct 20:32:11 cweid: http://www.codinghorror.com/blog/2013/07/rule-of-three.html 20:32:11 but right now we have 3 pending. 20:32:22 dmakogon_: I'm not sure what you mean by "tweak" trove. Trove already supports multiple managers for the guest agent. 20:32:24 after cluster api will be finished, we'll try to do specification for mongo cluster 20:32:27 cweid/vipul: agreed, but the existing clustering api mail thread is already discussing 3+ datastores 20:32:59 i was just pointing out that some final consensus is needed there first, before a super-detailed mongodb spec is written 20:33:24 cweid: Well I think for now we do blue prints on new data stores, and as they come in, things in Trove should naturally become easier to reuse and more flexible (unless we screw up) 20:33:33 Sure, it's not just about clustering, i think there is room for improvement in guest agent and taskmanger for example.. 20:33:46 grapex: +1 20:33:53 arborism, with analizing other dbs we are getting cleare vision of cluster api also 20:34:00 correct, yes. 20:34:10 isviridov +1 20:34:27 there is demand of HA cluster, not just set of engines 20:34:41 vipul: I agree with you. I'd like to take those improvements to trove, but as part of being able to support "n" implementations, not specifically MongoDB, for example... 20:35:31 SlickNik: sure, but we have no other implementation of another engine happening in Public Trove.. (at least i have not seen much code) 20:35:50 I know we're talking about implementing Redis, and we are doing Vertica, etc.. 20:36:08 but there isn't much refactoring, pluggability work going on currently to support those 20:36:44 couldn't we just make db engines some kind of extension that we plug in? 20:36:53 since it seems like there's quite a bit of interest in this, i'd ask that people who haven't read that mail thread do so, because it's trying to address all of the topics being brought up, at least from a generic api perspective. I'd love to see comments about guest/taskmanager refactorings as well. 20:36:57 vipul: yeah i agree and that leads to me to thinkin not much *needs* to change now 20:37:13 vipul: here is Trove sertup to support Redis. https://github.com/cweidenkeller/trove 20:37:15 that work is already in progress, types/version blueprint is a step in that direction 20:37:24 vipul: As people do new db types, I think we should gate them on making sure the code is refactored enough that they don't stick out like a sore thumb or create issues, naturally putting pressure on keeping the code clean and making it more pluggable as the implementation count increases. 20:37:42 arborism: link to thread? 20:37:44 cweid: thanks 20:37:45 I'm totally good with this, but I'd like to see more details on what changes are needed added to the blueprint. Things like "We need to support X for better multi-engine support in trove", and not necessarily "We need to support Y because MongoDB" 20:37:54 grapex: +1 20:38:40 yup agree with grapex as well. 20:38:45 SlickNik: agreed, let's use the Mongo BP as a place to do this 20:38:58 kevinconway: http://lists.openstack.org/pipermail/openstack/2013-August/000719.html 20:39:18 SlickNik: agree, this is step one to that - https://wiki.openstack.org/wiki/Trove/trove-versions-types 20:39:18 specifically the gist I provided, talking about how to generically support cassandra + mongo + mysql + regions 20:39:29 cweid, cp16net: lookign at https://github.com/cweidenkeller/trove/blob/master/trove/guestagent/manager/redis.py -- i see a lot of duplication between that and mysql.py 20:39:42 seems like that would be a perfect opportunity to create some sort of an interface.. 20:39:44 right now sure you can make whatever run in Trove, but you can't actually run multiple DB engines and types on the same install 20:40:40 demorris: Maybe we do need to do some bare minimum work to make running multiple DB engines (aka service_types) at once possible- and just make which ones are enabled configurable. Then we should all agree to be a bit lax with these new db types- 20:40:57 +1 grapex 20:41:16 grapex, +1 20:41:38 i agree we need to be able to (dis)(en)able service types 20:41:47 agreed grapex. 20:41:48 and let code for them get checked in without being 100% done. I don't like that, but there will need to be changes to make Trove more flexible while the Mongo and Redis guys add there work, so we'll need to gate a bit less than usual (i.e. features for these types won't have to work 100% before a commit) 20:42:39 sure, grapex. I'd welcome changes that really make it pluggable 20:42:42 * grapex wishes he'd spelled "there" "their" 20:42:58 grapex: forty lashes 20:43:41 to be continued? 20:43:59 vipul: So the issue is, will these changes be led by the veterans or the people making new DB types? I think the pressures of what everyone's boss wants them to do will mean the new DB implementers may need to make it pluggable, but the friction is they may not always know the greatest way to do that since they're a bit new to the project. 20:44:17 Looking like one to me. 20:45:07 So with the bp in question though; I'd like to see a bit more definition in what this entails and how much of the trove-core code needs to change for it. 20:45:09 grapex: sure, it could be the vets.. I doubt we'd be abel to refactor everythign necessary for anothre service type 20:45:10 Plus, we don't really always know what it would take to add Redis or Mongo since we're not adding them ourselves, so I guess my point is we should allow some limited Redis and Mongo code to get in early on. 20:45:36 And lax our standards to jump start the conversation- just for a bit, and just so long as nothing that's working today breaks. 20:45:57 grapex: i'm cool with that 20:46:22 So I'd love for it to drive other changes in trove-core to make it more pluggable. 20:46:29 it's gonna have to be a combination of us going in there and refactoring, and also code review suggestions 20:46:47 vipul: I think so. 20:47:29 time check :) 20:47:32 So what I'm hearing is approve the bp for an initial stab at adding mongo 20:47:36 Whoops. 20:47:42 i thinks we've done with Mongo BP 20:48:03 okay move on for now... 20:48:11 to be continued. 20:48:28 #topic virgo based guest-agent (https://github.com/racker/virgo, https://github.com/racker/virgo-base) 20:48:47 who want to tell us about this client > 20:48:49 ? 20:48:57 Who added this to the agenda? 20:49:23 skip 20:49:28 hmm…okay. 20:49:30 moving on. 20:49:33 was in open discussion 20:49:38 why not discuss it 20:49:49 who knows about it 20:50:05 demorris: who's the right person to discuss it? 20:50:06 i heard it for the first time yesterday 20:50:09 anyone have something to tell? discuss ? 20:50:10 i tried to research virgo today, where is server part for it? 20:50:31 I think it would be just the agent 20:50:36 its just an agent 20:50:38 it was mentioned as on of possible implementation for guest-agent, any info about it? 20:50:42 to where it reporting? 20:50:42 Virgo is the guest agent used by Rackspace monitoring 20:51:15 it is a lightweight C-based agent with extensible Lua scripting for agent logic 20:51:16 can it receive rpc calls? 20:51:22 If whoever raised it isn't here to talk about it I think we should move on- its basically an agent framework of sorts written on top of luvit, which is NodeJS for Lua 20:51:37 I think this discussion might be best suited for #openstack-trove after the meeting (since it's more of a knowledge sharing one) 20:51:44 let us move on 20:51:57 #topic trove refactoring: guestagent main issues (dmakogon) 20:52:11 my point 20:52:18 GA issues 20:52:19 Weaken trove.guestagent.manager.mysql & .mysql_service dependency on trove.instance.models -- actually only small set of constants and one simple persistent model is required, not the whole bunch of code 20:52:31 Same with trove.guestagent.backup vs trove.backup.models 20:52:46 trove.guestagent.strategies should not use trove.common.remote. The latter pulls lots of stuff but only trivial one-line call is actually needed 20:53:08 Check trove.guestagent.manager.mysql.mysql_service dependency on trove.extensions.mysql.models (may be not so easy/important) 20:53:10 dmakogon_: nice! with conductor, the models dependencies are gone 20:53:20 vipul: thanks 20:53:41 dmakogon_: There's also another bp out to separate the guest agent into it's own repo / package to decouple it further. 20:53:47 So this is all goodness. 20:54:07 And if you opened a bug and fixed some of these issues, I'm sure no-one here would object :) 20:54:18 please add these to that bp, just to track them 20:54:25 SlickNick: we don't wont to extract GA to separate repo 20:55:01 SlickNik: we want to break dependencies 20:55:14 this is the main goal 20:55:20 dmakogon_: If it got its own repo it would become skinny quick. 20:55:22 dmakogon_: at some point, for packaging pusposes we should also consider moving it to a spearate repo 20:55:26 I know the two goals can be made seperate 20:55:33 but one would help the other. 20:55:40 grapex: +1 20:55:41 I'm totally fine with tackling the two separately. 20:55:53 But they're related. 20:55:58 vipul: i thinks, not 20:56:26 we should break deps and keep project together 20:56:30 what about common code for both projects? 20:56:37 oslo? 20:56:43 dmakogon_: that is not allowed 20:56:55 constants, so on 20:56:55 only one setup.py per git repo 20:56:56 there is common agent code in oslo 20:56:56 dmakogon_: I would like fewer repos myself, but alas, tis not the way of OpenStack. 20:57:20 ok we need to revist this i think 20:57:26 this seems like a tangent. 20:57:52 3 mins.. 20:57:57 we could put each daemon in it's own repo! 20:57:58 Let's revisit this and keep going 20:58:12 seems better to keep it in one repo, or api should be defined 20:58:14 #topic versions / service_types 20:58:28 its mine 20:58:38 lets see this one again https://gist.github.com/andreyshestakov/b1f1b06fd4aef18011ea 20:58:40 take it away, ashestakov 20:58:43 also affects cluster api 20:59:28 ashestakov: why separate the service_type_id and verstion_id? 20:59:35 shoudln't they be grouped (on instance create) 20:59:46 vipul: we disscussed this on previos meeting 21:00:22 ok missed that then 21:00:25 version_id is optional, you should specify it only you have multiple active versions 21:00:39 there another issue 21:00:40 tehy could still be grouped, since they are related things 21:01:17 in that spec, once we eliminated service_type as it is now (eg, mysql and percona), we should add same parameter 21:01:23 vipul: I agree, it optimizes the case with no multiple versions but it less organized if you do have them. 21:02:11 ashestakov: how about talking about it in openstack-trove later 21:02:26 vipul: after this meeting? 21:02:55 soem folks might be leaving 21:03:00 yup, let's talk about it immediately after this meeting. 21:03:07 ok we can do that too 21:03:15 if it's a meeting time you should probably reschedule it for next time as well 21:03:24 Or we can pick another time if folks are leaving. 21:03:31 time = item 21:03:39 Let's move on for now 21:03:55 #topic guest_agent service registry (dict_opt vs single opt) 21:04:02 who's got this one? 21:04:08 So I'll comment 21:04:18 please do 21:04:26 Is this about the guest config file having both a dictionary of managers, and a key to which of those managers to use? 21:04:37 yes 21:05:02 I don't get the utility of it, as it appears the dictionary isn't used out of the bin script 21:05:05 Maybe I missed something? 21:05:25 I think the idea is if we want to suppot a single package of guest agent that support >1 service types 21:05:32 how do we dynamically load the 'manager' 21:05:42 based on the service type of the instance 21:05:54 I guess my issue is the manager could be as easily loaded by specifying the class name 21:06:05 as a single conf entry? 21:06:08 Yes 21:06:10 yes you could do that.. 21:06:17 means you have different confs for different service types 21:06:25 i guess there is a bit my dynamicism if you go with dict 21:06:33 because the same conf can be resued 21:06:37 +1 vipul on that 21:07:01 +1 vipul 21:07:02 How? To change which manager is loaded, you have to still change the conf 21:07:04 vipul: i thinks we could do dynamic module deliveries 21:07:17 the conf would contain all possible managers 21:07:25 vipul: yes 21:07:32 servie_type (hopefully in the future) is something that will be passed into guest_info 21:07:35 than we do dymamic load 21:07:43 Ah 21:07:45 grapex: I think the idea would be to have all managers in the conf, and the pick the manager based on the service_type (in the rpc call?) 21:07:56 vipul: backup/restore strategies still using mysql impl 21:07:58 rpc call would be too late 21:08:01 SlickNik: Ok- I thought that maybe that was the plan. 21:08:38 yea, we wnat to gradually configure the backup / restore strategies as well 21:08:49 this is step #1 to getting the right manager loaded 21:08:54 Ok, I feel it could be YAGNI but I'm ok if we're all on board with making use of the dictionary very soon 21:09:05 about automated backups 21:09:23 Automated backup design: 1. Define limits for backups per tenant. 2. Define storing strategy. 3. Define timing strategy. 4. Define cluster backup strategy than 1-3 for cluster 21:09:29 Yeah, this is the same thing with backup / restore. 21:10:06 what do you say ? 21:10:09 dmakogon_: You might want to talk to cp16net about that as feedback for him. 21:10:26 I'm fine with the dict_opt 21:10:26 SlickNic: ok 21:10:45 anything else to add, vipul? 21:10:54 no i'm good... 21:10:55 about registry, cool with dict 21:11:03 Okay, #topic open discussion 21:11:05 datsun180b: Conductor code? 21:11:13 #topic Open Discussion 21:11:14 Just one closing note - I did a bit of JSON API comparison - of 13 sites: 8 use _ (underscore); 3 use camelCase; 1 uses - (dash); and the last one was all lowercase 21:11:35 juice: nice! 21:11:42 kevinconway: pay attention, both reviews are up 21:12:01 we shoudl look at things like jackson too, see if they expect a certain type 21:12:09 my guess is probably not 21:12:14 Good info juice. I'm leaning towards _ too. (mostly because it's what we already use in the majority of situations) 21:12:24 jackson you can specify what ever field name you like 21:12:49 @XMLElement("my_dogs_name") or something like that 21:12:53 Okay, I think we're good with the meeting. Let's take further discussion to #openstack-trove 21:13:04 Thanks all, and sorry for the OT. 21:13:09 #endmeeting