18:02:01 #startmeeting trove_bp_review 18:02:03 Meeting started Mon Apr 28 18:02:01 2014 UTC and is due to finish in 60 minutes. The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:06 The meeting name has been set to 'trove_bp_review' 18:02:13 Sorry had an extra '#' in there. 18:02:31 \o/ 18:02:34 щ. 18:02:37 o/ 18:02:40 o/ 18:02:40 o/ 18:02:43 o/ 18:02:51 |o| gooooooool 18:02:59 cp16net, lol 18:03:03 ;) 18:03:06 o/ 18:03:10 #link https://wiki.openstack.org/wiki/Meetings/TroveBPMeeting 18:03:11 ;) 18:03:30 o/ 18:03:56 #topic Percona support 18:04:00 o/ 18:04:01 \o 18:04:06 mattgriffin: Around? 18:04:39 seems to be online 18:04:58 hub_cap, looks like your proposal for design session will be very useful 18:05:12 single implementation to rule them all 18:05:14 Let's give him a minute, if not we can table the discussion it for later. 18:05:25 agreed 18:05:33 looks like mattgriffin's proposal here is to upgrade the percona components 18:05:55 juice, it seems like yes 18:05:58 SlickNik, hey... sorry. OTP... doh meeting! 18:06:00 appears that it will benefit the backup since there is direct integration with swift/s3 yes? 18:06:07 can we talk about my BPs at the end of the meeting? 18:06:14 juice: Yes, and I had some questions around backward compat. 18:06:21 very risky mattgriffin :) 18:06:36 move on? 18:06:38 mattgriffin: okay. But no guarantees the other discussions won't be a black hole. 18:06:38 mattgriffin, go-go-go with your BPs 18:06:39 Yup. 18:06:57 Let's come back to this. 18:07:12 #topic Percona support 18:07:23 oops 18:07:23 vertica support, maybe? 18:07:28 lolz 18:07:33 #topic Support for Vertica 18:07:34 <_shal_kh> Yeah 18:07:34 amrith, yup 18:07:39 ok 18:07:45 yogeshmehra: go for it 18:07:49 yup 18:08:06 This is for adding vertica as a datastore in trove 18:08:19 what's the justification for it ? 18:08:40 simple googling: Vertica has 600 customers and has been around since 05... 18:08:44 it offers the devs another db to work within trove 18:08:48 denis_makagon: choices 18:08:57 what's the justification for mongo/cassandra/anything 18:09:13 denis_makogon: :) sry was afk 18:09:14 denis_makogon, huh? i don't understand the point you seem to be making 18:09:16 vipul, customer envolvement 18:09:26 What's the emoticon for a stick figure rushing in late while raising his hand? 18:09:30 denis_makagon: to be clear this is a product that HP owns and has a vested interest in offering in dbaas 18:09:37 justification :- it's a big data DB 18:09:47 well.. 18:09:49 is there a free version for testing? 18:09:53 thats my biggest gripe 18:09:54 there are numerous justifications 18:09:54 yes 18:09:57 hub_cap, ++ 18:09:58 :-) 18:10:01 if there is no version we can use for validation / verification 18:10:04 https://my.vertica.com/community/ 18:10:05 hub_cap, yes 18:10:08 So, I'm fine with this given: a. The trove API makes sense for vertica (i.e. including the backup / restore parts), and b. We will have some way of testing this. 18:10:13 then im a-ok w/ it 18:10:14 the best one is to offer the trove users another option.. 18:10:16 ++ by me 18:10:22 lets add it in 18:10:29 Also it adds flexibility to Trove 18:10:34 yogeshmehra: does vertica need to be installed as a cluster? 18:10:35 so, a question for core 18:10:41 nopes 18:10:47 as we add new data stores 18:10:52 the cluster part comes with teh trove cluster support 18:10:53 some of them are going to need special options 18:10:56 vipul: i dont want to register tho... 18:10:59 in the case of vertica, cluster is one of them 18:11:01 it moves with trove 18:11:15 does it make sense to have a notion of "core" datastores and "beta" datastores 18:11:23 cp16net: yogeshmehra: there is a apt repo somewhere that doesn't require registration? 18:11:33 amrith, we can't, all datastores are equal 18:11:48 vipul:the community edition is free and is available from vertica 18:11:48 amrith: I think we shouldn't restrict what datastores run, but we should think before putting in features that no one can easily try or test 18:11:51 amrith, we cannot promote or have favorite 18:12:08 denis_makogon: yes we can 18:12:09 grapex, agreed 18:12:09 without registering yogeshmehra? 18:12:14 because all code won't be equally stable and mature 18:12:15 if 99% of us work on mysql, mysql is fav 18:12:31 hub_cap, its from customer perspective 18:12:35 denis_makogon : you answered the best justification 18:12:36 if hp wants to spend a billion dollars and 200 developers on trove+veritca 18:12:40 one time registration is required but the CE can be taken and dropped to apt 18:12:48 amrith: this sounds like a documentation issue 18:12:48 then trove+vertica will be the best supported :) 18:13:00 juice, not true 18:13:11 amrith: thats how other projects do it 18:13:21 what will adding new datastores do to testing time? 18:13:21 i ++ juices idea, thats the whole point of the support matrix 18:13:33 amrith: on the product page we can just state the level of testing/stability 18:13:39 hmm in the vertica docs it doesnt say anything about an apt-repo to install it 18:13:42 heh amrith ask nova if they test anything besides libvirt ;) 18:13:44 it is *also* a documentation issue 18:13:50 amrith: hub_cap correct me if I'm wrong here: Right now our stance is that the other datastores (other than mysql) are 'experimental'. 18:13:54 and ask cinder how many backends they test 18:14:00 hub_cap: ++ 18:14:11 hub_cap, heh, nova is a good example =) 18:14:13 SlickNik, if that's the case, you have (already) addressed my issue 18:14:17 we do however need to start working on running tests against the various data stores..I think we have a private one that runs for percona 18:14:27 SlickNik: that was the case, but im not sure we can say tha tanynmore 18:14:28 I suggest we document and move on as juice says 18:14:30 :) LOL on testing consensus 18:14:33 its more of they have different levels of implementation 18:14:38 plz see the support matrix hehe 18:14:43 raise glass of juice to toast: +1 18:15:03 amrith: hopefully not prune juice 18:15:12 juice: i think thats the general openstack consensus, private tests for other impls 18:15:17 any other questions for yogeshmehra :) 18:15:18 #link https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix 18:15:22 juice, I agree 18:15:24 I think we need to discuss this at the Wednesday's meeting. 18:15:24 but we might need to confirm w/ some peoples at the OS summit 18:15:32 juice it's most general canberry 18:15:40 I'll put an agenda item in. 18:15:46 Is vertica impl just for testing, or intended for production? 18:15:49 the only thing i'd say is.. make the binary available for folks running Trove and we're good. If someone decides to run Trove with Vertica on their laptop.. the image should build.. it should just work for them 18:15:50 That'll give us some time to think about it as well. 18:15:52 so, back to the BP 18:15:57 SlickNik: ++ inv mordred and sdague and co plz 18:15:59 are we all ok with the BP? 18:16:05 hub_cap: Will do. 18:16:13 vgnbkr: for productiona s well.. 18:16:23 ++ on the bp 18:16:24 there's example of Sahara project 18:16:28 but it is more for devs to taste and enjoy 18:16:31 first 18:16:38 #startvote Add vertica implementation? yes, no 18:16:38 Does it make sense to use vertica on a single instance, or did you have something else in mind? 18:16:38 Begin voting on: Add vertica implementation? Valid vote options are yes, no. 18:16:39 Vote using '#vote OPTION'. Only your last vote counts. 18:16:50 #vote yes 18:16:52 yes 18:16:55 boo 18:16:58 #vote yes 18:16:58 Mirantis has putted hbase/hadoop packages into the our out repo and it's 100% available 18:17:01 #vote yes 18:17:06 #vote yes 18:17:07 #vote yes 18:17:09 #vote yes 18:17:13 #vote yes 18:17:26 #vote yes 18:17:27 final tally, 8 yes, 1 boo 18:17:36 Okay enough data points. 18:17:40 cool 18:17:41 #endvote 18:17:42 Voted on "Add vertica implementation?" Results are 18:17:43 yes (8): SlickNik, amrith, denis_makogon, grapex, juice, cp16net, vipul, SnowDust 18:17:50 vipul: yes + boo :-) 18:17:54 lol 18:17:56 denis_makogon: what is the take away/point you just made? 18:17:59 Let's move on. 18:18:06 juice, after meeting 18:18:07 ok let's move on 18:18:23 thanks guys 18:18:29 #topic Resource management driver 18:18:36 it's mine 18:18:40 At current state Heat support in Trove is nothing else than experimental. Trove should be able to fully support Heat as resource management driver. 18:18:54 Why is it so important? Because Trove should not do what it does now (cloud service orchestration is not the part of the OS Database Program). Trove should delegate all tasks to Cloud Orchestration Service (Heat). 18:19:32 Why is it needed? The first answer is: To split-out two completely different resource management engines. Nova/Cinder/Neutron engine etc. called �NATIVES� and Heat engine called �ORCHESTRATOR�. As you can all know they cannot work together, because they are acting with resources in their own manners. But both engines are sharing more than enough common code inside the Trove. 18:19:42 are u just copy/pasting from the bp denis_makogon ? 18:19:43 so are we talking about refactoring the "use_heat" code path.. to be plugin driven? 18:19:52 hub_cap, i got some notes 18:19:58 vipul, yes 18:20:01 ok those notes should be on bp :) 18:20:08 and there is no need for a plugin 18:20:11 we should use heat and heat only 18:20:14 hub_cap, it's on the 2 ML's 18:20:16 and deprecate the old way 18:20:24 and then remove the old way 18:20:28 just like weve talked about since day one 18:20:31 hub_cap, heat is not 100% ready 18:20:36 then make it 100% ready 18:20:37 denis_makogon: ++ 18:20:43 im not saying we do i tnow 18:20:47 hub_cap: ++ 18:20:51 im saying instead of a silly plugin approach 18:20:52 fix heat 18:20:53 it doesn't support volume resizes 18:20:54 deprecate old 18:21:04 then add that to heat 18:21:13 making heat a resident trovian... 18:21:16 Well here's the thing- I think we're already at the point where we need an interface in Task Manager we can switch out for different strategies 18:21:18 how would a plugin model help that? 18:21:29 So, there are three valid options: 18:21:35 use NATIVES if there's no available Heat; 18:21:35 We can say "don't waste time on that, fix heat" but fixing heat is a much larger effort than refactoring our code to read better. 18:21:39 Do we need a plugin approach if the goal is for heat to be the only way of doing things? 18:21:39 use ORCHESTRATOR to work with Heat only; 18:21:53 grapex: no one is touching that code :) 18:22:04 Also... I'd prefer we not use "plugins" but just a simple strategy pattern as we do in dozens of other places in Trove 18:22:05 SlickNik, of course we need 18:22:14 if there are defeciencies in heat, we should fix them 18:22:15 current code looks strange and messy 18:22:22 denis_makogon: Don't just say of course, please list the reasons. 18:22:23 rather than do 70% heat, 30% trove homegrown 18:22:42 just like the other projects, we need to use heat or dont use it 18:22:48 denis_makogon: why not enlist the improvements required in the heat integration and work them out 18:22:54 we cant use heat for some thinkgs, then use non heat, and make heat out of sync 18:23:02 yogeshmehra, it's already done 18:23:05 if w edont have volume resize in heat 18:23:06 and we resize 18:23:11 then our heat volume size is wrong 18:23:20 denis_makogon: then second step, to fix them :-) 18:23:21 so when u call get info what do we use for the volume? 18:23:31 So the takeaway I'm getting from this is that we need to work with devs on the heat team to fix heat to make sure it can support all our use cases. 18:23:31 itll get _more_ confusing 18:23:35 as hub_cap said, the long-term goal is to have heat as the only option, but since the existing path will continue to exist for awhile (as deprecated), there's no reason to add yet another path to workaround heat "deficiencies" considering the deprecated path will still exist. 18:23:35 hub_cap: I agree with what you're saying long term 18:23:37 yes SlickNik 18:23:52 but I think denis_makogon's want here is to refactor some of the code- 18:23:54 grapex: and short term, no one use heat till the long term is done :) 18:23:59 all i say, resource provisioning should be done through heat 18:24:09 id rather see us 1) fix heat, 2) refactor our code 18:24:11 grapex, that's the first step 18:24:12 I guess I'm with him on this- though maybe we'll need to talk more on the particulars of how it looks. 18:24:20 than 1) refacotr our code, 2) fix heat 3) refactor _again_ 18:24:29 grapex, in parallel implement required use cases in heat 18:24:32 +1 hub_cap 18:24:37 1) dont touch anything 2) fix heat 3) refactor once 18:24:37 3d step - re-use it in trive 18:24:38 grapex: I think the question is prioritization; i.e. what do we do first. 18:24:44 This is why we can't have nice things. :) 18:24:48 hub_cap, no refactoring after 18:24:49 amcrn: +1 i don't really care to refactor our code that we're goign to deprecate, but i also don't see getting out of the two code paths anytime soon 18:24:50 lol grapex 18:24:59 denis_makogon: thats bs there will def be refactoring after 18:25:04 once u add volume resize 18:25:06 hub_cap, no 18:25:09 u will need to refactor the code to use it 18:25:11 boom, refactor 18:25:21 hub_cap, you would need just to write new method - nothing else 18:25:22 once u add proper flavor resize 18:25:23 same 18:25:27 hub_cap +1 on approach 18:25:27 thats a refactor denis_makogon 18:25:36 hub_cap, that's feature coverage 18:25:38 if resource provisioning can be done using heat why have another strategy for provisioning 18:25:38 It's like living next to a creek bed everyone dumps garbage into; we keep saying "that land is scheduled for development so don't bother." I think we should at least be open to the idea- though I don't know if it should be an entire blueprint 18:25:43 if u move from using A to B and not chanign how the api looks its a refactor 18:25:47 rewriting existing code 18:26:04 denis_makogon: Are you looking to make the heat changes needed for trove? 18:26:13 denis_makogon: why don't you fix the missing things in Heat and then we can revisit this 18:26:15 SlickNik, yes 18:26:30 vipul: ++ 18:26:31 SlickNik, i filed and delegated several BPs in Heat to my college 18:26:40 denis_makogon: why not just do that work? 18:26:40 vipul, fix the missing things or identify them? 18:26:48 if u want it done so bad, do it! :) 18:26:52 yogeshmehra : how is yor bp different from denis_makogon s BP being discussed 18:26:58 amrith: i am assuming they have been identified.. 18:27:09 vipul, ok; thx. 18:27:21 vipul, I wasn't making that assumption. 18:27:24 SnowDust: denis_makogon wants to abstract the exact orchestration/resoruce-provisioning out 18:27:28 i hope :) amrith 18:27:30 and use heat just as an implementation 18:27:46 my BP was about makinng complete heat/trove integration 18:27:49 current heat support is nothing else than experimental 18:28:02 its not complete, true 18:28:11 denis_makogon: i think what most folks are saying here is that when Heat has the features we need, then rip out the old code and go with a pure Heat timplmentation 18:28:18 that's for no benefits if orchestration will b thru heat 18:28:18 until then, don't change the code 18:28:36 juice, what about backward compatibility ?? 18:28:49 denis_makogon: the existing path will still exist as hub_cap stated, it will just be deprecated. 18:28:56 we can't just rip out use of native services 18:29:05 denis_makogon: i think we need to go incremental on this and need to see how it goes along the clustering also 18:29:08 right, no one will use heat until it works for the use cases, if not we wil have to do data migrations every time we add "somehting" in heat 18:29:09 Okay, I think the consensus is that we should revisit this bp only once denis_makogon has the gaps in heat which are needed by trove (resize, et al) fixed. 18:29:15 denis_makogon: think about this 18:29:26 youve done all of the "core" stuff, now someoen adds volume resize 18:29:33 well all your existing voumes are out of sync 18:29:37 so u have to write a migration in heat as well 18:29:44 what about denis_makogon's question on backward compatibility 18:29:46 on top of changing what trove is referencing 18:30:08 hub_cap, heat would not do migration 18:30:16 if you have some instances that are currently running that were provisioned the old way, how would the new heat only approach work 18:30:21 hub_cap, that's why Trove needs think about migration 18:30:30 hub_cap, it's only our problem 18:30:45 we'd migrate them juice by creating whatever's needed in Heat 18:30:45 its only our problem if we do it yoru way 18:30:47 juice, let me show 18:30:51 I think we said there could be a migration script one day that would populate heat’s db (backfill) 18:30:51 its no ones problem if u fix heat first 18:31:07 ;) 18:31:08 juice: That's something we need to think about before we deprecate the old way. 18:31:19 juice, https://github.com/denismakogon/trove/blob/bp/resource-manager-interface/trove/taskmanager/resources/migration.py 18:31:22 does anyone else feel like the horse is beaten to death? i want to -2 this 18:31:34 and say the prereq is to make sure heat works for our use cases 18:31:41 and encourage denis_makogon to fix heat to work for us 18:32:00 Okay, time for vote 18:32:04 hub_cap: +1 18:32:06 hub_cap, i'm not working at heat, tasks were deletegated 18:32:07 -2 18:32:12 -1 18:32:26 #vote no 18:32:37 #startvote Resource management driver? yes, no, postpone_after_heat 18:32:38 Begin voting on: Resource management driver? Valid vote options are yes, no, postpone_after_heat. 18:32:39 Vote using '#vote OPTION'. Only your last vote counts. 18:32:41 denis_makogon: and they are being worked on now by people? u cant take them? like someone is actively coding on them? 18:32:43 I'm for this blueprint being drastically reduced in scope, but as is it's too big 18:32:45 #vote no 18:32:46 #vote no 18:32:52 #vote no 18:32:54 #vote no 18:33:04 #vote no 18:33:05 hub_cap, yes 18:33:14 #vote postpone_after_heat 18:33:17 #vote no 18:33:19 then ask them if u can help :) 18:33:26 #vote postpone_after_heat 18:33:29 #vote yes 18:33:31 * hub_cap changes his vote 18:33:36 lol juice your vote is yes now 18:33:49 u cant make a dependent vote 18:33:56 I saw the post pone after heat :) 18:34:00 i know ;) 18:34:07 #vote no 18:34:08 hub_cap, tasks already being spreaded, i can only wait 18:34:22 anyone else? 18:34:33 #endvote 18:34:33 Voted on "Resource management driver?" Results are 18:34:34 postpone_after_heat (2): SlickNik, hub_cap 18:34:35 no (5): amcrn, juice, cp16net, vipul, SnowDust 18:34:38 denis_makogon: plz send me the blueprints 18:34:44 so i can talk to the people about it 18:34:57 and make sure that they are all being worked on _immediately_ 18:35:10 hub_cap, search in your mailbox, i sent at least 2+ emails 18:35:24 with [Trove] tag 18:35:35 denis_makogon: It would be good if you could drive that work and make sure that all the scenarios needed for trove are covered in heat. 18:35:48 Okay, moving on. 18:35:49 denis_makogon: and the have the heat bps in them ? 18:35:56 SlickNik, thanks what i'm dling 18:35:58 *doing 18:36:03 hub_cap, yues 18:36:07 wonderful thx 18:36:26 #topic Managed Instances 18:36:35 so we talked a few weeks back about locking down nova vms which are managed by Trove. The first proposal was to have Trove Admin own the Instances in Nova. That was shot down. I went back to the nova team to discuss 18:36:52 phil day proposed using the lock/unlock feature in Nova (wow!) 18:37:13 juice: lock / unlock? 18:37:20 the basic concept is that Trove Admin will put a lock on instances preventing users from compromising the integrity of the vm 18:37:23 wow! 18:37:30 grapex - yes 18:37:34 denis_makogon: you do realize youre assigned to one of the bps right? 18:37:39 https://blueprints.launchpad.net/heat/+spec/handle-update-for-security-groups 18:37:43 basically there is a user level lock and an admin level lock 18:37:46 assignee denis m 18:37:52 so u can do that work :) 18:37:52 juice, what does a "lock" do? in reality? 18:37:59 hub_cap, yes, i haven't update it yet 18:38:07 well u should do that work! 18:38:08 amrith: it just puts a flag in the db 18:38:14 ¡mom 18:38:20 nova checks this flag on just about every modification call 18:38:34 so we just need to send lock=true for it? 18:38:40 juice: will it hide it from users when doing a nova list 18:38:41 ? 18:38:41 juice, and it will lock down ssh and things as well? or is that at a different level of protection? 18:38:46 if the instance is locked, it rejects the call (assuming that your role is lower than the lock holder) 18:38:59 esmute: no 18:39:26 amrith: no but the user cannot ssh into the instance because their key does not exist on the server 18:39:36 juice, thx 18:39:47 unfortunately security groups are still modifiable but just about everything else is locked down 18:39:52 if this won't protect security-groups, won't protect volumes, and won't protect swift objects (backups), i'd be hesitant to vote yes for this if the scope of work is high. 18:40:01 Trove Admin will be the only thing that can delete these 18:40:17 Should this (locking) be an optional feature? 18:40:24 it will protect volumes since there is not much you can do to a volume that is already attache 18:40:34 and you can't detach the volume from a locked server 18:40:38 juice: it'll protect a detach? oh, neat. 18:40:46 i take back my comment :) 18:40:49 vgnbkr: It is being proposed as optional. 18:40:52 vgnbkr: it will be optional 18:41:10 just a flag that say's "lock the instances" or some such 18:41:10 juice: does neutron support any locking? 18:41:28 vipul: not that I am aware of 18:41:32 Sorry, I looked through the bp and didn't see it being optional. OK, thanks. 18:41:41 vipul: Nope, I haven't seen anything in neutron corresponding to this. 18:41:43 vipul: let me look at my api printout sheet :) 18:41:52 so when we manage networks, secgroups in neutron, will that cause this to be disjoint? 18:42:07 meaning, i can delete a network from underneath the lock 18:42:29 vipul: can you give me an example of where you see some disjointedness 18:42:32 or is that ok? since they own the network 18:42:35 vipul: neutron won't let you delete a network that's in use. 18:42:44 if I recall correctly. 18:42:45 the network they own they better have access to 18:42:51 vipul: i think it works like volume-attached.. 18:42:58