20:02:41 <ttx> #startmeeting tc
20:02:42 <openstack> Meeting started Tue Apr 30 20:02:41 2013 UTC.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:02:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:02:45 <openstack> The meeting name has been set to 'tc'
20:03:02 <ttx> Agenda @ https://wiki.openstack.org/wiki/Governance/TechnicalCommittee
20:03:13 <ttx> Please all welcome shardy who will be Heat PTL for Havana
20:03:20 <markmc> welcome shardy :)
20:03:22 <mikal> Heya!
20:03:26 <jgriffith> hey shardy
20:03:33 <gabrielhurley> welcome and congrats!
20:03:40 <shardy> thanks all :)
20:03:50 <ttx> #topic RedDwarf Application for Incubation - introduction discussion
20:03:57 <ttx> RedDwarf folks filed for formal incubation at:
20:04:02 <ttx> #link https://wiki.openstack.org/wiki/ReddwarfAppliesForIncubation
20:04:12 <ttx> We'd like to kick off the discussion at this meeting and ideally make a decision by next week meeting
20:04:16 <markmcclain> hi
20:04:24 <ttx> So the idea for today is to raise the concerns we have and the questions we'd like answered before next week
20:04:25 <hub_cap> hello and thx for the consideration
20:04:35 <ttx> hub_cap: care to summarize the project so far and why you think it's ready for incubation ?
20:04:42 <hub_cap> sure thang
20:05:13 <hub_cap> the project contains the basics for a relational database as a service, with 2 implementations being tested and dev'd on, percona and oracle mysql
20:05:31 <hub_cap> at present the api does not preclude us from tackling postgres, or nosql solutions
20:05:52 <hub_cap> its got instance creation (which will eventually be instrumented by Heat), and user / schema creation
20:06:01 <hub_cap> as well as resizes, root password enabling
20:06:15 <hub_cap> quotas, limits, standard OpenStack stuffs
20:06:29 <hub_cap> all the features can be found at
20:06:45 <hub_cap> #link https://github.com/stackforge/database-api/blob/master/openstack-database-api/src/markdown/database-api-v1.md
20:06:57 <hub_cap> so why do we feel we are ready?
20:06:59 <hub_cap> a few things
20:07:12 <hub_cap> weve been following, and interested in being incubated for a while now
20:07:35 <hub_cap> we have support from > 1 company, and we are getting more support from vendors who are interested in getting into openstack from a db perspective
20:08:00 <hub_cap> weve been following all the guidelines, using stackforge, gating, running ci tests as gates as well as pep / unit
20:08:15 <hub_cap> keeping inline w/ blueprints / bugs, and the release schedule OpenStack follows
20:08:27 <hub_cap> #link https://launchpad.net/reddwarf
20:08:37 <hub_cap> so we feel like we are at critical mass, effectively
20:08:49 <hub_cap> and would like the help and support from OpenStack community
20:09:02 <hub_cap> ot make this a world class Database as a Service project
20:09:15 * hub_cap gets off soapbox
20:09:34 <ttx> OK, questions time
20:09:43 <ttx> The most obvious one is the need to rename the code name of the project into something more... hmm less... pre-existing
20:10:00 <ttx> but I'm pretty sure there are other questions
20:10:07 <russellb> yeah, here's one
20:10:09 <hub_cap> less preexisting than a 'cool star' ;)
20:10:14 <russellb> You've mentioned that RD has been deployed in production already. How will this affect openness to significant changes?  I suspect that entering incubation will greatly increase visbility of the project, and people interested in providing input.  Is that something you have concerns about?
20:10:26 <markwash> #link http://www.imdb.com/title/tt0094535/
20:10:35 <hub_cap> aww markwash :P
20:10:47 <hub_cap> russellb: at rax we are well aware of what this will do to the product, from a openness perspective
20:10:59 <jcooley> ditto for HP.
20:11:02 <hub_cap> we are embracing the change as a good thing, and i am devoted 100% to the public
20:11:23 <hub_cap> my title @ rax is community dev lead, i focus on the product in the open. period.
20:11:29 <shardy> hub_cap: A lot of instance-specific stuff seems to have been defined in the API, do you expect that to remain, e.g after you abstract the instance behind a heat template?
20:11:54 <shardy> Seems like implementation has bled through rather than being a pure DB API abstraction
20:11:55 <markmc> on the name, fwiw - I don't see it "creating confusion in the market place" - but IANAL, and we should get real advice
20:12:10 <mordred> I actually agree with markmc
20:12:20 <hub_cap> shardy: i think thats a good question. we havent begun to look @ how heat will help us, and if it changes our api, then thats something we have to roll with
20:12:20 <russellb> yeah, i was surprised to see security group management in your API, for example.  seems like a duplicate of something that lives in other apis
20:12:27 <annegentle> hub_cap: have you reached out to mirantis about Savanna (or have they come to you?)
20:12:30 <hub_cap> there is some mysql specifics, and id love to take those out
20:12:39 <hub_cap> annegentle: not at present
20:12:59 <hub_cap> russellb: the security groups is a pass thru, so people dont have to go to nova for some of the functionality
20:13:00 <shardy> russellb: agree
20:13:03 <vipul> the security groups impl is there to abstract away the VM from the end user, but also allow them to manage some things
20:13:06 <russellb> one of the incubation questions is "how likely is the architecture to change drastically?" or something to that effect ... this question about heat seems to be high risk in this area
20:13:27 <ttx> mordred/markmc: well, we can certainly push it to the Foundation and see how that rolls, before manding a name change.
20:13:31 <hub_cap> russellb: its possible that some of the backend will change but we will strive to make this as painles as possible to deployers and users
20:13:35 <russellb> hub_cap: yeah but i don't agree that passthrough is a solution, only a temporary thing if needed
20:13:37 <markmc> agree on the issue of future major architecture changes re heat
20:14:00 <hub_cap> russellb: thats fine wrt secgroups
20:14:07 <gabrielhurley> Personally I see huge value in Red Dwarf, and everything that was discussed in the email thread on RD's incubation was spot on. But this vote comes down to "are we now openly admitting that OpenStack will accept IaaS++ projects into Integrated?" We're already moving that way, but if we go there for real it's opening the doors for many others. This isn't really a question to anyone; it's merely what I'm grappling with in
20:14:09 <mordred> I'm not sure I agree wrt heat
20:14:11 <russellb> i think we should stick to the heat topic, that's a much bigger issue
20:14:12 <markmc> it'd be nice to at least understand how re-architecting to use heat will change things
20:14:41 <markmc> mordred, agree that it should use heat? or agree we should know in advance how it will change things?
20:14:49 <hub_cap> so i see heat as a way, under the covers to help us create instances, and clusters
20:14:50 <grapex> russelb: We've already had to change a lot of architecture in the past (back when we were a fork of Nova well over a year ago and changed to be a stand-alone project making REST calls), and we managed to do it while maintaining our API.
20:14:53 <mordred> I think that heat is a new wrinkle for many of the projects that they'll need to deal with and part of the overall project evolving
20:14:53 <ttx> russellb: yes
20:15:09 <gabrielhurley> I'd like to see the Heat work as a condition of successful graduation
20:15:12 <shardy> I think exposing details of how the DB is deployed via the API is wrong, e.g in our RDS nested stack resource implementation all of that is hidden
20:15:15 <jd__> gabrielhurley: I think having Swift is already answering the question of "are we going IasS+" :-)
20:15:27 <shardy> users shouldn't care about those details IMO
20:15:39 <russellb> shardy: agreed
20:15:44 <ttx> The only potential barrier to incubation that I can see would be "shouldn't it be rearchitected to use  Heat prior to be accepted for incubation"
20:15:45 <mordred> I agree with jd__
20:15:49 <hub_cap> shardy: right. thast why they will create a "instance" and we will use heat to plug it all in
20:16:05 <dolphm> hub_cap: regarding mysql specifics, has there at least been a proof of concept to manage nosql? i don't want to see us hit an openstack release with a blocker for something we're claiming future support for
20:16:06 <hub_cap> i dont see that significantly altering our api
20:16:21 <hub_cap> dolphm: im working on a redis POC now
20:16:26 <ttx> personally, I think part of the incubation process is to create the incentive to align with other integrated projects
20:16:37 <shardy> hub_cap: I think the whole usage of the word "instance" wrt providing a database service needs to be reconsidered
20:16:40 <russellb> ttx: that's interesting perspective
20:16:55 <russellb> calling it "instance" if != "instance" in nova is very confusing
20:16:55 <gabrielhurley> one wrinkle: if red warf uses Heat under the hood (which we all want) then we've moved Heat into a dependency tree whereas it was independently optionaly before...
20:16:56 <shardy> imho of course ;)
20:16:57 <hub_cap> shardy: thats fine, we went back and forth for a LONG time to try to come up w/ something that made sense to us
20:17:03 <ttx> so that question could be used against RD for graduation, not for incubatuion ?
20:17:20 <hub_cap> ttx: i like that :)
20:17:26 <markmc> gabrielhurley, reddwarf will be optional :)
20:17:35 <russellb> i still think it's worth considering the heat issue now vs later
20:17:39 <gabrielhurley> markmc: yeah, that's about the only solution I can see
20:17:40 <annegentle> I'm trying to decide if it's preservationist or conservationist to require use of Heat API for integration? Does it preserve resources or actually require more?
20:17:47 <russellb> because what if someone looks into this and decides it's easier to just start over to do a heat based approach?
20:17:52 <annegentle> "it" means the heat req.
20:18:15 <hub_cap> i dont think it alters the api annegentle. things that would alter the api woudl be deciding to call it something diff than instances moreso...
20:18:25 <annegentle> russellb: words are difficult to preserve :)
20:18:26 <shardy> annegentle: It's just about avoiding huge duplication of effort IMO
20:18:26 <hub_cap> we are going to use heat behind the scenes
20:18:27 <jd__> ttx: makes sense to me
20:18:28 <markmc> annegentle, architectural sanity; avoiding having two things in the architecture that overlap
20:18:29 <demorris> to me, Heat integration would be contained within the implementation and does not mean that it would bleed through and cause the API to change...
20:18:41 <markmc> hub_cap, great :)
20:18:54 <markwash> demorris: +1
20:19:01 <gabrielhurley> demorris: that's what I'd expect too
20:19:01 <russellb> this is more about blessing an API, though ...
20:19:03 <annegentle> hub_cap: demorris: okay.
20:19:15 <russellb> err, more than blessing an API
20:19:16 <markmc> the "using heat won't change the API" seems a bit hand-wavey to me, though
20:19:28 <jcooley> dmorris: also +1, the only concern may be around terminology. but has Heat settled on terminology yet or is that still in flux?
20:19:30 <markwash> I want to imagine that Heat would be on the implementation rather than api side of RD
20:19:31 <shardy> demorris: that would be true if the API didn't already expose details of the implementation
20:19:33 <markmc> i.e. is that based on looking at how you'd use heat, or a vague understanding of what heat is?
20:19:37 <hub_cap> sure, but does the history of incubated projects mean that they all have solid, complete apis, before becoming incubated?
20:19:39 <annegentle> yeah my okay was more like oh-kayyy.
20:19:58 <annegentle> markmc: ^^ yeah
20:20:00 <hub_cap> i think the process to becoming integrated will be to solidify the api around openstack
20:20:16 <hub_cap> yall didnt have input before we were incubated cuz we wre just loosely affiliated
20:20:23 <hub_cap> now we are asking for input
20:20:25 <jcooley> markmc: the hand-wavvy part is that we may need additional parameters, but the operations are the same... right?  CRUD model.
20:20:25 <demorris> shardy, am curious in what contexts we are showing impl details in the API
20:20:27 <russellb> question is, is RD as it is today actually where you would start for something like this based on heat, or would you start over?
20:20:46 <hub_cap> and during this process we will be _fixing_ this stuff :)
20:20:49 <russellb> unless someone has seriously looked into this, i don't think anyone can answer that
20:20:50 <hub_cap> russellb: i see no need to start over
20:20:56 <russellb> do you know that?
20:21:02 <hub_cap> i cant predict the future ;)
20:21:11 <markmc> well, would e.g. the taskmanager service be required if you were using heat ?
20:21:11 <hub_cap> nor have i looked into it deeply
20:21:15 <russellb> but someone can spend some in depth time considering what this would look like
20:21:21 <shardy> demorris: security groups and rules, instance flavors, instance-orientated terminology
20:21:22 <markmc> or would the api talk directly to heat and hand off
20:21:28 <mordred> well, I think since rax and hp have this in front of customers, I doubt they'll start completely from scratch
20:21:38 <hub_cap> markmc: task mgr also takes care of long running db tasks
20:21:42 <jcooley> markmc: yes, it would.
20:21:44 <hub_cap> backups, restores, replication etc...
20:21:57 <markmc> ah, ok
20:22:00 <jd__> changing or building a new API doesn't sound like something bad as long as it's well handled, which is a criteria to judge for accepting a project to core, not really the API itself, IMHO
20:22:01 <annegentle> I think RD has flexed along the way and proven flexible, but my concern is about centralized resource sharing (docs, qa, etc.) It's my usual line of questioning.
20:22:08 <russellb> btw, i'm disappointed that we have to consider the fact that companies put this into production before integrating with the community in our decision process
20:22:09 <jcooley> markmc: we still want to factor the front end API tier from the backend.
20:22:11 <markmc> but taskmanager might not be required for instance management tasks?
20:22:33 <demorris> shardy: I guess I was thinking you were meaning we were exposing DB specific implementation, which I think is what matters in this case
20:22:34 <hub_cap> markmc: my guess is "probably not?" wiht a ?
20:22:43 <hub_cap> we can remove those code paths :D
20:22:45 <ttx> russellb: do we ?
20:23:00 <russellb> ttx: well it has been brought up multiple times in this conversation
20:23:00 <mordred> russellb: I am as well, but I see this as a path to getting some of that back under control personally
20:23:01 <shardy> one interesting chicken/egg thing is we'd like to use RD instead of our nested-stack RDS implementation
20:23:10 <mordred> ttx: I think it is a salient data point
20:23:16 <jcooley> markmc: no, anything that could cause latency on the front-end would want to be moved to the taskmanager/backend.
20:23:24 <russellb> mordred: yeah, guess so ... but I also want to make it clear that this was the wrong order of operations.
20:23:32 <mordred> russellb: ++
20:23:37 <markmc> shardy, is that really a problem ? seems like it would work fine
20:24:05 <shardy> markmc: probably not, was just thinking of the bi-directional dependencies
20:24:13 <mordred> shardy: I had the same question - but I haven't wrapped my head around it fully yet - but couldn't heat call rd which then calls heat ?
20:24:28 <gabrielhurley> I don't see any reason why it couldn't
20:24:39 <russellb> yeah, would really like someone to do some in depth investigation on how you'd design something like this to work with heat, we're doing a lot of guessing on it
20:24:41 <ttx> maybe you can work on some clear Heat integration plan that you would follow if you were to be incubated, so that we can judge how realistic it sounds
20:24:46 <shardy> mordred: Yep, that would work, quite similar to our existing nested stack mechanism but defined via RD
20:24:50 <hub_cap> russellb: ++
20:24:51 <russellb> ttx: ++
20:24:58 <markmc> heh, mordred of OpenStack on Openstack claiming *this* is hard to get his head around :)
20:25:03 <hub_cap> we can do that ttx
20:25:09 <mordred> markmc: hehe
20:25:18 <gabrielhurley> lol
20:25:21 <ttx> that would be great, seems to be one of the most pressing concerns at this point
20:25:34 <ttx> anything else RD needs to work on ?
20:25:37 <hub_cap> agreed
20:25:45 <gabrielhurley> did we get any answers around docs/ci/etc?
20:25:45 <markmc> thanks hub_cap and co
20:25:50 <hub_cap> welcome
20:25:53 <gabrielhurley> (did I just miss that?)
20:25:55 <markmc> for the record, you guys seem to have a tonne right
20:26:07 <demorris> apologies, but I need some help here. I am still struggling with why a Heat plan, or use of Heat needs to be a pre-condition for incubation or graduation..
20:26:09 <hub_cap> gabrielhurley: i dont think we have talked bout it
20:26:09 <markmc> also for the record, I did a quick license check - all code is apache licensed
20:26:10 <dolphm> given that rax / hp have products based on this, how flexible is the api at this point?
20:26:19 <ttx> #info RD should investigate and document transition to use Heat if it were to be accepted in incubation
20:26:26 <markmc> and deps check - nothing new, except factory_boy which looks fine
20:26:30 <russellb> yeah, love the usage of openstack processes
20:26:37 <gabrielhurley> hub_cap: gothca. annegentle asked about it but it fell through the cracks.
20:26:37 <russellb> also happy with the scope/fit question
20:26:41 <annegentle> hub_cap: so you linked to a markdown API doc but I thought you were working on API docs that are more like the rest of OpenStack API docs?
20:26:41 <mordred> they're doing a good job on CI on stackforge, and are currently running third-party integration tests triggered by our gerrit
20:26:42 <markmc> hub_cap, do you intend to keep the rsdns client in tree?
20:26:45 <demorris> to me, it is a matter of implementation on how we orchestrate the creation of database resources, clusters, etc...
20:27:02 <annegentle> I've seen their stackforge work also and it's good.
20:27:06 <russellb> demorris: yes but the implementation *is* an important question that we do consider as criteria.
20:27:07 <hub_cap> dolphm: i say its flexible. worst case we can version and go
20:27:19 <russellb> demorris: it's not something to be dismissed as trivial and unimportant.
20:27:22 <mordred> we've got hooks for them in devstack/devstack-gate and know how to hook them in if they become incubated
20:27:28 <hub_cap> annegentle: we are, mike from rax has some that hes almost done with
20:27:35 <demorris> russellb: agree, not trying to dismiss it
20:27:35 <vipul> hub_cap, dolphm: yep, we can always version if drastic changes are needed
20:27:39 <annegentle> hub_cap: one suggestion is to put a link to your github codebase on the wiki application page
20:27:41 <ttx> demorris: we want to avoid a project incubated realizing it needs to be completely rewritten to be integrated with other projects
20:27:42 <hub_cap> i pushed that out to give people an idea of whats going on
20:27:48 <demorris> struggling to keep up with all the conversations
20:27:52 <hub_cap> annegentle: good call ill do that now
20:27:54 <ttx> demorris: so investigating that in advnace sounds snae
20:27:54 <hub_cap> demorris: lol
20:27:55 <dolphm> hub_cap: so, maintain backwards compatibility on further changes to v1?
20:27:57 <ttx> *sane
20:27:58 <jcooley> russellb: its flexible enough that rax uses straight mysql and hp uses percona.
20:28:12 <russellb> jcooley: *nod*
20:28:20 <markmc> hub_cap, do you intend to keep the rsdns client in tree?
20:28:23 <hub_cap> dolphm: ya i think so
20:28:26 <demorris> ttx: agree, I think some things got lost in translation for me
20:28:32 <annegentle> hub_cap: your bug list seems fairly triaged, are you using test suites now?
20:28:48 <hub_cap> markmc: not if we can use a client that moniker has built for us :D
20:28:50 <dolphm> hub_cap: i'm asking because i'm noticing what i'm hoping are bugs on the spec doc, rather than impl :)
20:29:14 <markmc> hub_cap, well, it seems odd for a project to carry something like that IMHO - would be good to move it out
20:29:17 <markwash> I'd love to understand why we are pushing Heat for RD as well. . is it that RD is currently duplicating a lot of Heat functionality?
20:29:28 <gabrielhurley> markwash: yes
20:29:30 <markmc> hub_cap, what in-guest distros do you currently support/use ?
20:29:30 <hub_cap> annegentle: im not sure what u mean wrt test suites
20:29:39 <annegentle> hub_cap: how do you test now?
20:29:50 <shardy> markwash: yes, orchestration basically ;)
20:29:59 <hub_cap> ahh, so we have unit and integration tests, the integration is done via jenkins that hp has put out
20:30:00 <markmc> markwash, yes, dupication of instance management and the like - RD would use Heat as an implementation detail
20:30:06 <markwash> so, is there a concern that this duplication is divisive?
20:30:13 * hub_cap pulls the next question off the stack
20:30:25 <mordred> hub_cap: you guys still have proboscis in the tree? or are you totally on to testr yet?
20:30:33 <russellb> hub_cap: can always come back and read scrollback to catch all the points raised
20:30:45 * hub_cap will have to russellb ;)
20:30:46 <grapex> mordred: The plan is to make proboscis run on testr so we can migrate.
20:30:47 * markwash will definitely look more into the Heat question for himself, so we can move on
20:31:02 <mordred> grapex: that hurts my head
20:31:14 <hub_cap> dolphm: we dont have any bugs in the spec doc persay at present
20:31:23 <grapex> mordred: Rather than drop the suite complete. However once we make it run testr, it should run everything fairly trivially the way other projects do and fit into the existing CI systems.
20:31:33 <mordred> grapex: great
20:31:40 <hub_cap> markmc: we support deb based currently, ubuntu/debian
20:31:45 <ttx> hub_cap: what's the timeframe for that Heat adoption analysis ? trying to see if it's worth it keeping the incubation decision on the agenda for next week
20:31:55 <markmc> hub_cap, any plans for supporting other distros? any problems anticipated?
20:31:58 <hub_cap> i dont think its out of the question to say 1 wk
20:32:11 <ttx> hub_cap: post to openstack-dev when done
20:32:11 <hub_cap> markmc: woudl love to support rpm based, no problems there
20:32:15 <hub_cap> ttx: roger
20:32:48 <hub_cap> ok i _think_ i got everyones Qs... if u still have one fire or ping me
20:32:56 <markmc> interestingly, we didn't discuss the scope of the project
20:32:58 <russellb> s/u/you/
20:33:07 * markmc thinks the scope is fine FWIW
20:33:14 <russellb> markmc: +1
20:33:23 <ttx> Fine discussing it if that's an issue for anyone
20:33:32 <mordred> ++
20:33:35 <ttx> otherwise we'll just go to the next topic
20:33:41 <mikal> Works for me
20:33:45 <markwash> I'm still a tiny bit concerned
20:33:55 <hub_cap> thx for your consideration peoples
20:34:06 <ttx> markwash: I don't like to see you concerned.
20:34:13 <demorris> hub_cap: +1
20:34:13 <hub_cap> markwash: we can chat about it if you'd like offline? or online if need be :D
20:34:15 <markwash> because I would see the inclusiveness of its scope a prereq, but so far it seems like mostly promises and plans
20:34:24 <markwash> all of which seem lovely
20:34:30 <hub_cap> heh
20:34:37 <gabrielhurley> markwash: what, you don't trust them? ;-)
20:34:48 <markwash> well, I don't trust anybody forward looking, so :-)
20:34:49 <hub_cap> ya markwash?
20:34:54 <hub_cap> :P
20:34:55 <jgriffith> haha... we have to start somewhere
20:34:56 <ttx> markwash: I definitely like the idea to be more database-agnostic... but so far it's just a plan, true
20:35:37 <markwash> but to me, honestly, if it fits the AWS bill of what DBaaS means, its probably right for our system
20:35:38 <jd__> it's also something that can be proved during incubation
20:35:42 <ttx> Maybe they could also prepare a rationale as to why their architecture will also support other-SQL and NoSQL ?
20:35:44 <mordred> agree on more agnostic. otoh, adding more backends is a "if someone cares" thing
20:36:01 <mordred> same as virt drivers
20:36:06 <ttx> without forcing us to individually look into the code and figure it out
20:36:07 <gabrielhurley> yeah, wouldn't want to mandate more backends only to have them be abandoned after the POC
20:36:09 <mordred> we only added hyperv because someone showed up who wanted it
20:36:14 <russellb> speaking of AWS, i noticed that they have nosql as a separate service instead of all in one
20:36:15 <demorris> ttx: we have blueprints for the API that discuss how the API will be DB agnostic
20:36:16 <dolphm> ttx: that's why i'd like to see a proof of concept of a nosql impl, rather than more wording :)
20:36:28 <markwash> russellb: oh interesting
20:36:39 <russellb> we should consider that before we just assume it would be a part of this
20:36:40 <dolphm> russellb: interesting, do you know why?
20:36:45 <mikal> I like the idea of having fewer services
20:36:47 <russellb> no, was hoping someone else did :)
20:36:50 <ttx> dolphm: that sounds a bit harsh to mandate for the incubation decision. maybe something we could require for graduation ?
20:36:51 <dolphm> hub_cap: ?
20:36:54 <russellb> but the APIs are quite different
20:36:58 <markwash> I think I'd settle for some convincing proof that the api is compatible with other likely backends (Cassandra, say)
20:37:01 <mordred> ttx: ++
20:37:03 <russellb> the nosql aws service seems to have a lot more ... nosql-isms
20:37:15 <hub_cap> that'd be my guess russellb, different apis...
20:37:21 <dolphm> ttx: agree
20:37:23 <mordred> I don't personally think that reddwarf needs to support nosql
20:37:25 <annegentle> I think scope is an important part of this whole process (incubation, graduation, core etc)
20:37:28 * markmc is fine with the idea that nosql could be a separate service
20:37:28 <russellb> yeah, but are different apis what we think is ideal?
20:37:31 <russellb> or not?
20:37:33 <markmc> if needs be
20:37:38 <mordred> because I don't think that nosql and sql things really share much in the way of operational semantics
20:37:43 <ttx> At this point I want them to explain why it should be agnostic. Then we can put them against their words in the following months to prove it
20:37:52 <mordred> same as how swift and cinder are different
20:37:53 <markwash> mordred: I'd be fine with that, so long as other agree that there might be room for a simliar NoSQL DBaaS project in OpenStack
20:37:55 <mordred> because they are DIFFERENT
20:38:05 <mordred> markwash: I'd be totally open to that
20:38:11 <hub_cap> we might find that it doesnt fit, then we focus more on relational :)
20:38:12 <markmc> markwash, yes, there would be room
20:38:13 <russellb> same here, i think, based on what i've read so far
20:38:25 <markwash> well, cool
20:38:32 <markwash> that might seem more sane to me anyway
20:38:34 <demorris> markwash: yes the topic around a distributed key-value store service similar to DynamoDB should be had
20:38:41 <ttx> it shoudl AT LEAST be SQL-db-agnostic
20:38:52 <russellb> ttx: +1 :)
20:39:01 <hub_cap> ttx: def. weve strived for that so far.. maybe i do a postgres impl instead of redis/cassandra?
20:39:15 <mordred> btw - I would like to congratulate the reddwarf team for not following the aws footsteps in forking mysql to do their work
20:39:17 <russellb> yes, i think postgres may be a better next step
20:39:17 <hub_cap> to prove its db agnosting
20:39:28 <markmcclain> ttx:	+1
20:39:39 <mordred> or drizzle :)
20:39:50 * mordred shuts up
20:39:51 <russellb> sqlite!
20:39:51 <russellb> wait
20:39:52 <russellb> :-p
20:39:57 <cp16net> lol
20:39:59 <hub_cap> ok so for next week, ill do the following, 1) have a postgres POC, 2) have diagrams / wiki for heat design in reddwarf
20:40:21 <hub_cap> are there other outstanding things i should bring ot the table?
20:40:23 <markwash> #action markwash research the differences between AWS DBaaS offerings for sql-like and nosql-like
20:40:26 <markwash> just for my own notes
20:40:43 <ttx> Personally I'd be fine with you explaining why the code supports postgres rather than a POC, but a POC may actually be faster ;)
20:40:59 * jd__ votes postgres
20:41:01 <hub_cap> i wonder markwash, do they not expose a "backend" to the serivce, like we would do w/ mongo or cassandra
20:41:11 <demorris> markwash: would love to share my thoughts with you on this topic if you are up for it
20:41:21 <hub_cap> so their api has to have doc retrieval / storage etc
20:41:23 <ttx> OK, unless there are still people concerned, I'd like to switch to next topic
20:41:26 <demorris> have studied this area in depth
20:41:30 <markwash> demorris: cool
20:41:33 <markwash> ttx: go for it
20:41:38 <hub_cap> thank you all again
20:41:41 <ttx> #topic Discussion: What is our goal with OpenStack WRT API behavior and implementation
20:41:44 <ttx> jgriffith: Care to introduce this one ?
20:41:58 <jgriffith> ttx: sure
20:42:16 <jgriffith> So the issue is a question of API and expected behaviors
20:42:35 <jgriffith> My take has been that the end user should not know/care what the backend storage is for the most part
20:42:51 <jgriffith> That means, when I say "cinder create-snapshot" I get a snapshot
20:42:58 <jgriffith> for example
20:43:07 <jcooley> markwash: AWS has developed the offerings with different teams and different, optimized hardware environments... :)
20:43:18 <ttx> I read the thread... I think our goal is not to let storage backend vendors differentiate with unique features. Our goal is to provide a set of basic API calls that reflects our view of "block storage as a service"... and have drivers for the backends which support that set
20:43:34 <jgriffith> Well...
20:43:47 <jgriffith> My proposal is that the base/refernce implementation is just that
20:43:53 <mikal> Also, we need to remember that some deployments might have more than one vendor in them, and those need to feel consistent.
20:43:54 <jgriffith> If you can't meet that bar you are out
20:44:06 <jgriffith> If you want to exceed functionality that's fine
20:44:13 <jgriffith> mikal: +1000000
20:44:20 <markwash> jgriffith: does the base/reference impl just clone on snapshot?
20:44:24 <jgriffith> mikal: So that's my entire argument
20:44:30 <markmcclain> remember this also not unique to storage.. networking has this issue too with API compliance
20:44:33 <jgriffith> markwash: it can do whatever you want to implement
20:44:43 <ttx> jgriffith: I'm actually not sure of the value to openstack of allowing differentioation in extensions, at least for the block storage project...
20:44:44 <jgriffith> markwash: I don't care what the implementation looks like
20:44:46 <jd__> jgriffith: how are you sure the reference implementation can cover all cases that another vendor might offer?
20:44:49 <mikal> We see a similar problem with multiple hypervisors in nova too
20:44:51 <jgriffith> markwash: all I care about is the end result
20:44:58 <jgriffith> jd__: it can't
20:44:59 <ttx> jgriffith: but I can be convinced otherwise I guess
20:45:15 <jgriffith> ttx: well... yes
20:45:25 <jgriffith> ttx: so there are mechanism via things like volume-types
20:45:28 <dolphm> keystone has this concern as well, considering how likely it is for someone to write their own identity driver / auth plugin / etc
20:45:29 <jgriffith> ttx: that covers this nicely
20:45:30 <jgriffith> IMO
20:45:31 <jd__> jgriffith: that sounds like a problem to me somehow :)
20:45:36 <jgriffith> jd__: which?
20:45:54 <jgriffith> jd__: ie what sounds like a problem?
20:46:05 <markwash> jgriffith: rather than defining an api with a base/ref impl, I'd rather define it in terms of an api with independent testing. .
20:46:11 <jd__> jgriffith: leveling down all drivers to your base implementation features?
20:46:22 <markwash> jgriffith: the significance being, I might be willing to cut corners for something that is "just" the base impl
20:46:24 <jgriffith> jd__: markwash ok timeout
20:46:29 <markwash> sorry
20:46:30 <jgriffith> you're going the opposite direction
20:46:36 * markwash pauses
20:46:39 <jd__> :)
20:46:41 <jgriffith> I'm stating jus the opposite!
20:46:41 <gabrielhurley> I still say that the "core API" should not by definition include everything that the reference implementation supports.
20:46:52 <jgriffith> There are vendors that can't do features that the base offeres
20:46:52 <gabrielhurley> core API needs to look across implementations
20:46:54 <jgriffith> offers
20:47:01 <jgriffith> and they're suggesting they shouldn't have to
20:47:07 <jgriffith> I'm stating that's not an option
20:47:10 <ttx> jgriffith: +1
20:47:13 <annegentle> what gabrielhurley said about core API is the way I see the current state of affairs on Compute
20:47:20 <jgriffith> annegentle: gabrielhurley I agree
20:47:22 <ttx> that sounds like the bare minimum to me
20:47:39 <jgriffith> gabrielhurley: in fact that's my entire proposal in a nut shell really
20:47:43 <gabrielhurley> yep
20:47:49 <jd__> jgriffith: ok, you mean raising NotImplemented isn't an option? so it's better to not have this driver basically?
20:47:54 <jgriffith> jd__: exactly
20:48:08 <jgriffith> jd__: LVM isn't exactly a super high bar to begin with
20:48:13 <mikal> But only for the "core api" right? Drivers can implement functionality over and above that?
20:48:14 <gabrielhurley> my only addendum to that is that any capability that's not core needs to be "discoverable".
20:48:23 <markwash> high bar or not, its a bottom-up description
20:48:26 <jgriffith> mikal: over and above is fine
20:48:29 <gabrielhurley> e.g. you can query /extensions and find out what's supported beyond the "core"
20:48:30 <mikal> How do I know what's core or not?
20:48:36 <jgriffith> mikal: as long as it doesn't change the API
20:48:42 <mordred> I think that the core API needs to be suported by everything
20:48:46 <jgriffith> mikal: I'll publish it
20:48:48 <jd__> jgriffith: fair enough, I think the real question is where the bar is, this can be different for each case
20:48:51 <jgriffith> mikal: or just look at the API today
20:48:53 <mordred> or else it's a TERRIBLE user exprience
20:48:59 <mikal> Agreed
20:49:08 <gabrielhurley> mordred: you say that as if this isn't already the case...
20:49:09 <ttx> mordred: yes, what jgriffith proposes sounds like the bare minimum to me
20:49:10 <jgriffith> mordred: so that's what I wanted mind share on
20:49:12 <mikal> I think we implicitly do this in nova already to be honest
20:49:13 <markmcclain> mordred: agreed
20:49:18 <jgriffith> mikal: not ture
20:49:19 <jgriffith> true
20:49:22 <mikal> i.e. all drivers can launch a VM, not all can do console output
20:49:23 <jgriffith> we're fixing it in nova
20:49:37 <jgriffith> in the past IIRc there was a divergence
20:49:39 <jd__> that's an interesting question even for Ceilometer since we have a few drivers lacking behind the API and not offering all features for now
20:49:51 <mordred> and I don't think we should artificially keep a feature oout of core just becaus a vendor might no be able to keep up
20:49:52 <ttx> does anyone disagree with jgriffith that drivers should implement the core set of API features or else be rejected ?
20:49:56 <markwash> jgriffith: is this about raising NotImplemented for a given function, or raising NotImplemented for a given function depending on the inputs?
20:50:01 <mordred> if it's a thing that's a fundametal feature
20:50:08 <mordred> ttx: yes
20:50:16 <jd__> ttx: we agree since we kind-of decided the same thing during the summit for Ceilometer
20:50:17 <dolphm> ttx: yes
20:50:20 <mordred> ttx: otherwise they are pointless drivers
20:50:21 <jgriffith> markwash: it's about you MUST implement the API
20:50:29 <jgriffith> I don't give a rats butt how you do it
20:50:30 <dolphm> the identity api has to be able to raise 501 or 403 as an expected behavior, depending on driver / driver config
20:50:31 <ttx> mordred: so you don't disagree with jgriffith
20:50:55 <ttx> looks like veryone agrees
20:50:58 <mordred> ttx: I find it highly unlikely that I disagree with anyone
20:51:03 <jgriffith> mordred: haha!!!
20:51:10 <markwash> a little lost actually
20:51:10 <hub_cap> mordred: i disagree w/ that statement
20:51:10 <ttx> except my keyboard which I really need to change
20:51:30 <jgriffith> markwash: I can discuss after if you like?
20:51:31 <ttx> let me summarize
20:51:36 <jgriffith> or anyone else for that matter
20:51:46 <markwash> It sounds like you're definition of API is implicitly defined such that NotImplemented is an invalid result
20:51:50 <ttx> #info block storage drivers should implement the core set of API features or else be rejected
20:51:57 <jgriffith> ttx: bingo
20:52:11 <markwash> but I could define a core API where it was acceptable, and that would completely undermine this discussion
20:52:22 <jgriffith> markwash: ?
20:52:30 <mordred> markwash: what would you accomplish by doing that?
20:52:45 <ttx> markwash: is that a rhetorical question ?
20:52:46 <mordred> and what would the user experience be?
20:53:05 <markwash> ttx: yes, somewhat rhetorical
20:53:06 <jgriffith> markwash: I wasn't suggesting a philosophical discussion, I'm just proposing that things like Horizon work
20:53:16 <markwash> but I"m trying to get down to the specific case in question
20:53:17 <gabrielhurley> markwash: invalid for something that is "core", which should be a smaller subset of "essential" functionality. e.g. core are things that you would not reasonably build a service without (yes, that's still subject to interpretation, but at some point it has to be).
20:53:20 <jgriffith> and that and end user get's the expected results regardless of what backend is in use
20:53:22 <markmc> so, here's a thought - our core API could expose what optional features are available
20:53:31 <markmc> build the concept of capabilities into the core API
20:53:32 <jgriffith> markwash: horrid idea
20:53:44 <russellb> markmc: yeah i think gabrielhurley has been begging for that :)
20:53:49 <gabrielhurley> markmc: I think I've been saying that for 2 years... ;-)
20:53:49 <jgriffith> markwash: if you're suggesting the dynamic API reading?
20:53:50 <markmc> rather than only having optional features as extensions
20:54:04 <jgriffith> markwash: perhaps I'm misunderstanding.. sorry
20:54:15 <markwash> I think I'm on you all's side here, just trying to understand and consider a specific case
20:54:16 <ttx> markmc: that's orthogonal to the discussion of requiring core APi featyures to be implemented in drivers
20:54:37 <gabrielhurley> the flipside is the possibility that *nothing* is "core" and *everything* is a capability that must be discoverable. but that makes for really messy clients and wonky UX's.
20:54:47 <markmc> ttx, no, it's not - if the core API supported optional feature, drivers would not have to implement optional core features
20:54:48 <ttx> markmc: you're talking about extra API features discovery, right ?
20:54:53 * jgriffith thinks he's regretting bringing this up :)
20:55:28 <ttx> markmc: oh, I see
20:55:39 <jgriffith> So here's the thing, the cleanest way for what markmc is proposing (I think)
20:55:42 <jgriffith> is via extensions...
20:55:45 <markmc> jgriffith, well, would it be ridiculous for snapshots to be optional and for it to be discoverable via the API that a volume isn't snapshotable
20:55:57 <ttx> markmc: I still think what jgriffith wants mindshare on is a good starting point
20:55:57 <jgriffith> reason being there can be multiple backends that have different capabilities
20:55:59 <jd__> gabrielhurley: sounds beautiful *and* scary :)
20:56:10 <markmc> jgriffith, notice I say the capability is discoverable *per volume*
20:56:11 <dolphm> i think consistent API behavior across drivers is much more important than whether a feature is wholly available or not -- a single call should mean the same thing to two different drivers, if both implement it
20:56:16 <markmc> jgriffith, that's not an extension
20:56:20 <jgriffith> markmc: indeed and that's what's scary
20:56:25 <markwash> dolphm: +1
20:56:28 <jgriffith> markmc: that volume can be anywhere on any backend
20:56:28 <markmc> jgriffith, why?
20:56:40 <jgriffith> and the end user has to get all the capabilities to know what he/she can/can't do
20:56:46 <markwash> dolphm: I think that's my angle as well, much better put when you said it
20:56:58 <jgriffith> and not only that, most SP's don't want their customers knowing the details
20:56:59 <gabrielhurley> dolphm, markwash: that's the world of "everything must be discoverable", 'cuz otherwise I can't build a UI for that.
20:57:09 <markmc> jgriffith, I'd say e.g. gabrielhurley would be perfectly happy if he could query per-volume whether a snapshot button should be enabled
20:57:15 <dolphm> gabrielhurley: agree
20:57:25 <jgriffith> markmc: ok... well for the record
20:57:29 <markwash> gabrielhurley: that sounds like a sensible requirement
20:57:32 <jgriffith> I violently object to that idea
20:57:33 <ttx> Running out of time... I think this discussion can safely move to the ML, as I'm not sure we need a cross-project TC call on this
20:57:45 <markmc> jgriffith, wonderful :) the violence really helps :)
20:57:48 <jgriffith> I think that completely kills the user experience
20:57:51 <gabrielhurley> I'd rather have what jgriffith proposes, but honestly I'll take either. What I can't take it the current state of affairs which is neither. :-D
20:57:55 <markwash> we need drinks for this :-)
20:58:00 <jgriffith> markmc: I didn't mean that literally
20:58:17 <gabrielhurley> markwash: I will happily debate this over drinks tomorrow ;-)
20:58:24 <mordred> I would like for there to be a sane set of things that I don't have to query that I can count on to work
20:58:35 <gabrielhurley> mordred: +1
20:58:40 <mordred> for things that may or may not work, I'd like to know how to know if they do or don't
20:58:45 <ttx> jgriffith: I think what you propose is sane. Mark's option is sane too... but in the end whoever does it gets to decide ?
20:58:49 <markwash> mordred: +1, but if not possible somehow, what dolph said
20:58:58 <mordred> but really, I want those things to be things I'm likely to not care about in the general case
20:59:01 <jgriffith> ttx: markmc works for me
20:59:06 <gabrielhurley> ttx: I think that's why we *do* need a TC decision on this. lol.
20:59:26 <ttx> last minute question...
20:59:28 <jgriffith> I guess I'm not understanding the debate here
20:59:34 <jgriffith> ttx: sorry.. go ahead
20:59:36 <ttx> shardy: you had a question about whether adding AutoScaling to Heat was a formal extension of scope that the TC should give its opinion on
20:59:46 <ttx> quick answer from TC members ?
20:59:57 <shardy> ttx: shall I just ping the ML since we're out of time?
20:59:57 <markmc> think it's worth the TC discussing at least
20:59:58 <gabrielhurley> +1 on adding it
21:00:07 * markmc had always thought auto-scaling would be a separate service
21:00:18 <markmc> but maybe that's just because aws does it that way
21:00:20 <ttx> shardy: push to the ML and we'll discuss it next time
21:00:33 <shardy> ttx: will do
21:00:37 <mikal> Juju does it in the same service IIRC
21:00:42 <mikal> Which makes sense to me
21:00:46 <ttx> or consensus will just be reached on the ML, who knows
21:00:58 <markmc> the general idea of the TC discussing the widening of scope of projects seems sane, anyway
21:01:05 <markwash> +1
21:01:07 <gabrielhurley> +1
21:01:10 <ttx> #endmeeting