15:00:26 <jimbaker> #startmeeting craton
15:00:27 <openstack> Meeting started Mon Feb 20 15:00:26 2017 UTC and is due to finish in 60 minutes.  The chair is jimbaker. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:29 <jimbaker> #chair sigmavirus sulo jimbaker thomasem
15:00:31 <openstack> The meeting name has been set to 'craton'
15:00:32 <openstack> Current chairs: jimbaker sigmavirus sulo thomasem
15:00:45 <jimbaker> #link https://etherpad.openstack.org/p/craton-meetings
15:00:56 <jimbaker> #topic Roll Call
15:00:59 <jimbaker> o/
15:01:32 <thomasem> o/
15:02:29 <git-harry> hi
15:02:47 <farid> o/
15:03:47 <thomasem> #topic Action Items from Last Meeting
15:03:55 <thomasem> #link http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-02-16-17.02.html
15:04:15 * thomasem thomasem to write BP regarding deployment as a starting point for iterating on a suggested deployment model
15:04:36 <thomasem> Didn't get to it given other priorities. Carrying forward.
15:04:43 <thomasem> #action ACTION: thomasem to write BP regarding deployment as a starting point for iterating on a suggested deployment model
15:04:49 <thomasem> #undo
15:04:50 <openstack> Removing item from minutes: #action ACTION: thomasem to write BP regarding deployment as a starting point for iterating on a suggested deployment model
15:04:57 <thomasem> #action thomasem to write BP regarding deployment as a starting point for iterating on a suggested deployment model
15:05:00 <thomasem> There we go
15:05:08 * thomasem jimbaker to map Dusty's requirements to work or existing features of Craton, especially with respect to short-term deliverable (~2 weeks remaining)
15:05:46 <jimbaker> thomasem, still working on this action item
15:05:58 <thomasem> #action jimbaker to map Dusty's requirements to work or existing features of Craton, especially with respect to short-term deliverable (~2 weeks remaining)
15:06:05 * thomasem thomasem to review Dusty and Bjorn's stories/use-cases and add notes on concerns or questions
15:06:37 <thomasem> I finished up going through Björn's this morning and added several questions/concerns
15:07:00 <thomasem> Will review Dusty's stuff as well and update tomorrow.
15:07:16 <jimbaker> +1
15:07:17 <thomasem> #action thomasem to review Dusty stories and add notes on concerns or questions
15:07:26 * thomasem sigmavirus to finish up testing on cli
15:07:49 <thomasem> I don't think sigmavirus is here, so we need to carry this forward, probably.
15:08:05 <jimbaker> agreed. we can get back to it if sigmavirus joins us
15:08:09 <thomasem> #action sigmavirus to finish up testing on cli
15:08:10 <thomasem> Sounds good
15:08:18 <thomasem> #topic Stand Up
15:08:24 <thomasem> #info each team member briefly describes what they are working on this week, and describes blockers (if there are any)
15:08:58 <thomasem> #topic Stand Up :: jimbaker
15:09:59 <jimbaker> variables support in client/CLI; refactor alembic for foreign key constrain setup; reviews
15:10:01 <jimbaker> DONE
15:10:34 <thomasem> #topic Stand Up :: thomasem
15:10:36 <jimbaker> fwiw, there will be more than that
15:11:37 <thomasem> clouds resource (wiring it up through other resources, like how region was); two bugs related to projects resource links; use-cases/stories discussions for short-term deliverable; reviews
15:11:42 <thomasem> done
15:11:57 <thomasem> #topic Stand Up :: git-harry
15:12:43 <git-harry> adding /v1/devices endpoint for listing devices
15:12:48 <git-harry> done
15:13:59 <thomasem> #topic Stand Up :: farid
15:16:30 <farid> Gathered sample inventories to start loading into Craton, need to look into parsing/loading pieces. Client was buggy (seems to have been fixed? tb tested)
15:16:46 <thomasem> +1
15:16:48 <thomasem> excellent
15:16:56 <farid> o/
15:17:01 <jimbaker> farid, thanks. we will work through those bugs
15:17:12 <farid> good deal, will try to see what else I come across as well
15:17:14 <farid> done
15:17:21 <thomasem> woot
15:17:22 <thomasem> #topic This Week's Priorities
15:17:44 <jimbaker> thomasem, i think farid just highlighted this week's priorities
15:18:00 <thomasem> Lol, is there an enumeration of the bugs?
15:18:02 <jimbaker> get inventories into craton; find and fix bugs that prevent such usage
15:18:19 <thomasem> Gotcha
15:18:32 <git-harry> Is someone actively working on the client to fix the pagination bug(s)?
15:18:34 <jimbaker> thomasem, those would be our critical and high bugs
15:18:43 <jimbaker> git-harry, pagination hopefully is now fixed in client
15:18:54 <jimbaker> but this is what sigmavirus worked on
15:19:17 <jimbaker> i haven't pushed the envelope on pagination, but the simple case seems to work now
15:19:19 <git-harry> excellant
15:19:34 <jimbaker> eg hosts = list(craton.hosts.list())
15:19:47 <jimbaker> to be honest, i find that sort of awkward
15:20:15 <git-harry> generator?
15:20:16 <thomasem> If that's a generator, why not exhaust it while printing?
15:20:25 <jimbaker> yeah, it's now a generator
15:20:28 <jimbaker> thomasem, all valid
15:20:54 <jimbaker> it's more that it's craton.hosts.list(various-filter-thingies)
15:20:58 <thomasem> I guess because filtering and such.
15:21:07 <jimbaker> which returns the generator
15:21:13 <jimbaker> i think if it's called `list`
15:21:13 <thomasem> Yeah
15:21:18 <jimbaker> it should return a LIST
15:21:34 <thomasem> Okay, what about craton.hosts.all()? :D
15:21:49 <jimbaker> thomasem, could be a nice convenience iterator
15:22:17 <thomasem> Just seems to sort of defeat the purpose if we put it all in memory like that. Of course, unless we're really at scale, will be a non-issue.
15:22:46 <thomasem> Talking about drop in the well until we get to pretty high scales.
15:22:50 <jimbaker> thomasem, i was going to add, a new convenience iterator for doing careful paginated DOS
15:22:55 <jimbaker> ;)
15:23:03 <thomasem> Hahahaha
15:23:13 <thomasem> I suppose that would happen anyway with the list() if it's paginated.
15:23:18 <git-harry> maybe add a bug and we can look at it in a few weeks
15:23:22 <thomasem> Sure
15:23:29 <jimbaker> yeah, i don't think it matters very much. but i don't think we need `all`
15:23:50 <thomasem> I was offering that up as an alternative since list() might confuse when it returns a generator.
15:23:52 <jimbaker> what i would suggest is just have the `list` method return a `list` object
15:24:18 <jimbaker> and we can bikeshed on what to call the other method
15:24:39 <thomasem> Yes. jimbaker, you want to add a bug about that?
15:24:46 <jimbaker> thomasem, yeah, i will do that
15:24:47 <thomasem> Or shall I?
15:24:51 <thomasem> awesome thanks!
15:25:15 <thomasem> #action jimbaker add bug regarding list() method returning generator instead of list object in client library
15:25:27 <jimbaker> +1
15:26:04 <thomasem> Alright, enumerating priorities:
15:26:26 <thomasem> 1. Critical bugs
15:27:00 <thomasem> No way... I can't link to the "bugs" page?
15:27:17 <thomasem> nvm
15:27:18 <thomasem> Got it
15:27:19 <jimbaker> we can start with https://bugs.launchpad.net/python-cratonclient
15:27:30 <thomasem> #link https://bugs.launchpad.net/python-cratonclient
15:27:31 <jimbaker> and filter on it accrodingly
15:27:35 <thomasem> #link https://bugs.launchpad.net/craton
15:28:37 <thomasem> Should we make those project link bugs a higher priority since pagination is basically broken otherwise?
15:28:41 <thomasem> for projects
15:28:56 <thomasem> And how does that compare to, say, clouds
15:28:57 <jimbaker> thomasem, +1
15:29:31 <jimbaker> every resource collection should support pagination
15:29:50 <thomasem> Right, so we'd probably put fixing that above clouds, since that's not an existing resource in master yet... correct?
15:30:35 <git-harry> If they're both required, does it matter?
15:30:45 <jimbaker> agreed with git-harry - they are of equal priority
15:30:50 <thomasem> It does if it means CLI bugs can be fixed sooner.
15:31:04 <thomasem> But, I get your point.
15:31:14 <thomasem> Questioning it as a matter of dependency for others, is all.
15:31:20 <jimbaker> ok
15:31:21 <thomasem> Beyond that, I can do whatever order as long as they get fixed.
15:31:47 <thomasem> Anyway, I will focus on clouds, then, since I'm already under the hood on that anyway.
15:32:14 <jimbaker> thomasem, agreed
15:32:44 <thomasem> Okay, so priorities are basically
15:32:46 <jimbaker> from the CLI, it's the same abstract machinery to support. still need to robustly verify in our integration testing (no matter how formalized at this point)
15:33:24 <thomasem> 1. python-cratonclient critical bugs now and those that come up throughout the week (As a result of us trying to get data imported into Craton deployment)
15:33:36 <thomasem> 2. critical bugs on craton API itself
15:34:03 <thomasem> Pretty much all the critical things. We need to finish up that /devices spec so git-harry isn't blocked.
15:35:02 <thomasem> git-harry: even so, do you feel you have enough agreement on that spec to get started on that work?
15:35:04 <jimbaker> agreed at /v1/devices collection support
15:35:34 <git-harry> thomasem: I've made a start already. I will adjust what I've done if the spec changes.
15:35:42 <thomasem> awesome, sounds good!
15:35:44 <jimbaker> there seems to be some differences re how much the underlying modeling in the sqlalchemy should be exposed, but i assume minor
15:35:49 <jimbaker> like device_type vs sub_type
15:35:51 <thomasem> I'll do another review of that by EoD.
15:36:28 <jimbaker> fwiw, i think if we go with some name changes like this, we should push into the sqlalchemy model
15:36:58 <git-harry> that's what I've done
15:37:04 <jimbaker> this is because at some point, we will want to support arbitrary sql usage of our model. and having two names for the same thing, this is not good
15:37:54 <jimbaker> thomasem, this gets back into a conversation we might have had (tell me if i'm misremembring)
15:38:26 <jimbaker> re sqlalchemy and object graphs; vs just sqlalchemy and doing interesting queries
15:38:58 <thomasem> We touched on it a bit, but I don't remember us going into great detail.
15:39:00 <jimbaker> fortunately sqlalchemy supports both approaches
15:39:15 <thomasem> You're referring to my dislike of ORMs? :)
15:39:29 <jimbaker> thomasem, no worries, i just bring it up because it's in bjorn's reqs, that's all
15:39:36 <jimbaker> in terms of being able to do analytics
15:39:50 <jimbaker> thomasem, re ORMs - i hope you will like in the end what we have done here :)
15:40:12 <thomasem> Lol, it's all good. It's always "it depends". I'm hesitant to truly say I dislike them as a blanket statement.
15:40:15 <jimbaker> i'm hoping we have the following: cake + eat it too
15:40:41 <thomasem> If done well and conscientiously I'm sure it'll be fine.
15:41:02 <thomasem> Anyway, I'll avoid going down that rabbit hole.
15:41:17 <jimbaker> fwiw, i usually look a the model from just the mysql client when i'm debugging things
15:41:31 <thomasem> Yeah, so... if we're expecting them to do analytics and such directly against the DB, hmmm..
15:41:33 <thomasem> So we really want that?
15:41:39 <thomasem> s/So/Do/
15:41:41 <jimbaker> i think so
15:42:06 <thomasem> Why? Kind of looking behind the curtain a bit and limits the versioning control that an API provides.
15:42:12 <jimbaker> and i don't think it's going to be a bad thing either, since we can always view things from a table perspective in SA
15:42:18 <thomasem> They're going to write things against a changing schema.
15:42:31 <thomasem> Mmmm, it is a bad thing when it comes to expectations of support.
15:42:34 <jimbaker> thomasem, oh sure. unless we write this stuff too
15:43:05 <thomasem> Classic way to get yourself in that sort of bind is setting the expectation that it's okay to query against the DB and ignore the API.
15:43:25 <thomasem> What do you mean? We write it... for them?
15:43:34 <thomasem> Isn't that basically what the API is for?
15:43:50 <jimbaker> yes, we can add it to our API :)
15:44:33 <git-harry> Sorry, I'm losing track a bit, is this something we need to add in the next two weeks.
15:44:47 <jimbaker> git-harry, no, it is not
15:44:51 <thomasem> I don't know, actually. Again, the use-cases as they map to short-term requirements are vague at best.
15:44:54 <jimbaker> it is relevant to a meeting i have in 15 min
15:44:56 <thomasem> Okay, good to know.
15:45:08 <jimbaker> so i'm just putting out a position there
15:45:08 <git-harry> Yeah, I'll be at that
15:45:43 <thomasem> Okay. Well, I would really like to discuss this reporting stuff when it does become relevant, because I'm going to plant my feet about people talking directly to the DB rather than going through the API.
15:45:45 <jimbaker> thomasem, i think the relevant contract is the SA model
15:46:04 <git-harry> but I thought that meeting was only to confirm immediate priorities
15:46:17 <thomasem> git-harry: I hope so
15:46:27 <jimbaker> always good to be prepared
15:46:51 <thomasem> jimbaker: you can't version that as easily as you can an API...
15:47:09 <thomasem> The data model will become a nightmare if we have to keep around old representations of the data.
15:47:25 <thomasem> Because somebody wrote a report against it and they don't have time to change it.
15:47:26 <jimbaker> thomasem, i believe that's the whole point of alembic and other migration work
15:47:42 <thomasem> I don't think that's what folks had in mind, exactly.
15:47:50 <thomasem> But, I'd be interested to hear your grounds for that reasoning.
15:48:04 <jimbaker> but sure, if you cannot update your report, you are on shaky grounds
15:48:10 <thomasem> That WILL happen.
15:48:45 <git-harry> Is there anything else we need to cover off before the end of this meeting? Otherwise I'm going to take back 10.
15:48:45 <thomasem> And it's insane to expect everyone who wrote some reporting thing against our schema to update every time we need to change the schema.
15:49:03 <thomasem> git-harry: nope. I think we have our priorities, and they will become clearer.
15:49:07 <thomasem> #topic Open Discussion
15:49:17 <jimbaker> then if we assume that reporting will be out of sync, we need to bring this into the API. some reports then become more supported than others
15:49:21 <thomasem> Okay, now jimbaker and I can nerd knife fight
15:49:31 <jimbaker> thomasem, oh sure
15:49:48 <thomasem> jimbaker I think all reports ought to query the API, and that's it.
15:49:49 <jimbaker> i do dislike opaque databases
15:50:03 <thomasem> Anything going behind the API is going to cause a maintenance problem.
15:50:17 <thomasem> Because we don't have as much flexibility and power in versioning the DB as we do in versioning the API.
15:50:40 <thomasem> Alembic and other migration tools are meant to go from version to version, not necessarily support multiple versions simultaneously.
15:50:50 <thomasem> Unless I'm misunderstanding something?
15:50:53 <jimbaker> alembic does NOT support multiple versions
15:51:25 <jimbaker> any thought that your ad hoc reporting could survive a versioning change, that's obviously wrong
15:51:26 <thomasem> Exactly. You're at a version, and that's it. So, if folks are writing reporting and tooling against our schema and we need to go to another version, we risk breaking their things, and that's going to put people up in arms and hamstring our progress.
15:51:51 <thomasem> okay, so, given that then, wouldn't it be better to not set the expectation that it's okay to use our DB directly?
15:52:42 <thomasem> It'd be better to set a clear expectation (and this is definitely once we actually have versioning implemented in the API, too) that our API is what one ought to use for querying for reporting and other tooling?
15:52:46 <jimbaker> so two opinions then. i believe in making it feasible to do certain things, but understanding there is a risk involved
15:52:47 <farid> where did this direct db querying issue come from ?
15:52:57 <jimbaker> farid, this is about ad hoc reporting
15:53:12 <thomasem> I believe in protecting the customer from themselves.
15:53:23 <jimbaker> one option is: they have to build all of their representations of the inventory from what they can query from the api
15:53:35 <thomasem> and providing a clean UX to support their needs in a flexible, safe way.
15:54:14 <thomasem> Another consideration - allowing direct access to the DB opens you up to operational problems
15:54:24 <jimbaker> the other is, there's an underlying clean relational database
15:54:27 <thomasem> We put quotas and rate limits and other things in place to protect Craton from malicious or mistaken users
15:54:50 <farid> jimbaker: was ad-hoc reporting in the UC document ?
15:54:55 <jimbaker> farid, yes
15:55:13 <farid> which $?
15:55:16 <farid> sorry, #? :)
15:55:27 <farid> talking about 8?
15:55:42 <jimbaker> UC2
15:55:51 <jimbaker> and it's not initially required per bjorn
15:55:53 <farid> got it thanks
15:56:05 <thomasem> That UC looks to be satisfied by Craton API?
15:56:26 <jimbaker> thomasem, depends on how much we bake into our API :)
15:56:50 <thomasem> Okay, if they need more, we need to allow more from our API, in my opinion. But, allowing direct access to the DB sends the wrong impression.
15:57:00 <thomasem> They may come to rely on it as it is.
15:57:31 <thomasem> Even if you tell customers "this is subject to change at any time," they'll ignore you and expect it to stay that way.
15:57:34 <jimbaker> thomasem, i certainly know where you are coming from
15:57:57 <jimbaker> anyway, time to wrap up this meeting
15:58:02 <thomasem> Yeppers
15:58:04 <thomasem> #endmeeting