14:01:46 <nikhil> #startmeeting glance_artifacts_sub_team
14:01:52 <openstack> Meeting started Mon Oct  5 14:01:46 2015 UTC and is due to finish in 60 minutes.  The chair is nikhil. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:55 <ativelkov> o/
14:01:55 <openstack> The meeting name has been set to 'glance_artifacts_sub_team'
14:01:56 <nikhil> #chair ativelkov
14:01:57 <openstack> Current chairs: ativelkov nikhil
14:02:15 <ativelkov> Hi folks
14:02:52 <nikhil> #topic agenda
14:03:06 <nikhil> ativelkov: there's no specific agenda today
14:03:10 <ativelkov> We don't have much of agenda today (as usual), but I have a topic to discuss
14:03:14 <mfedosin> o/
14:03:17 <mfedosin> hey
14:03:18 <nikhil> so, I think we should just set topics as per the discussion items
14:03:27 <nikhil> ativelkov: cool
14:03:37 <ativelkov> #topic v3 api interoperability
14:03:58 <ativelkov> Thanks to jokke_ for giving me some hints on this last week
14:05:40 <ativelkov> As far as I understood, there are major problems with our v3 API proposal, as it as kind of a dynamic API (i.e. it is plugin-based), and defCore has some serious objections against such solutions, as the clouds with plugin-based APIs are actually not interoperable
14:06:32 <ativelkov> This means that we are unable to have a public API which depends on some plugins to be enabled
14:06:45 <nikhil> It depends on who can plugin into the code struct
14:07:11 <ativelkov> In my understanding it's the deployer who installs the plugins
14:07:13 <nikhil> if we can add plugins to glance for extensibility for supporting diff data assets then I think we can debate on it
14:07:29 <nikhil> for deployer scenario we would have issues
14:08:02 <mfedosin> I think the problem is plugin can implement its own apis and data schemes
14:08:51 <ativelkov> mfedosin: the overall API structure remains the same, but particular data schemas - yes, they differ from plugin to plugin
14:09:05 <mfedosin> and we have to normalize them to be more identical
14:09:45 <ativelkov> nikhil: so, do you mean that we may "bundle" some of the plugins as part of the glance distribution, so it remains kind of static from defCore's point of view?
14:10:29 <mfedosin> ativelkov, and no... plugin can: first, add its own custom logic (which is new api calls); and second, reimplement the existing methods
14:11:01 <ativelkov> mfedosin: re-implementation is not an issue, as it does not affect API
14:11:26 <ativelkov> mfedosin: what do you mean by "new API calls"?
14:11:48 <ativelkov> actions?
14:11:58 <mfedosin> like 'action/import'
14:12:00 <mfedosin> yes
14:12:15 <ativelkov> Ok, so this needs to be discussed more
14:13:04 <mfedosin> and plugin can re-implement, for example, 'list' call and the way it returns data
14:13:23 <nikhil> ativelkov: I think people think about defcore in a weird way possibly as things aren't very clear
14:13:54 <nikhil> ativelkov: DefCore wants a standard way for some major actions for a particular service to be interop
14:13:56 <ativelkov> nikhil: right, and I don't know how to make it more clear
14:14:18 <mfedosin> in Images plugin v1 we will have 'extra_properties' in response, but in Images v2 - custom properties will be the part of base Image
14:14:27 <nikhil> if plugin give more of a optional way to do things that are cloud specific then it's not a big deal
14:15:22 <nikhil> best example is the tasks.. that API isn't going away and the optional flows that can be done still remain cloud specific. Just that a standard minimum viable use case needs to be clarified by a interop API for upload.
14:15:46 <nikhil> So, in case of artifacts we should have some standardization of the APIs for say CRUD
14:15:47 <ativelkov> Well, from my point of view, the most important thing is that the plugins are defined by particular openstack components which are going to use glance v3 as their backend to store artifacts
14:16:13 <nikhil> all other misc operations like import (Advanced create, version changes etc) can be plugin specific)
14:16:22 <ativelkov> nikhil: cool
14:17:01 <ativelkov> so, we may say that the plugins define component specific API which is kind of private for the cloud, i.e. it affects only the component which is going to use that plugin
14:17:30 <nikhil> I think we can start with that
14:17:36 <ativelkov> OK.
14:17:51 <ativelkov> But this rises another question
14:18:06 <nikhil> ativelkov: how far are we to the plan for customizing data schema?
14:18:14 <nikhil> ativelkov: oops, you go first
14:18:37 <ativelkov> So, if we have a "common CRUD" API
14:18:58 <mfedosin> how can we list 'all' artifacts? :)
14:19:10 <ativelkov> yup
14:19:23 <ativelkov> so, say, we have a GET /v3/artifacts/all call
14:19:32 <mfedosin> if they are stored in different tables
14:19:44 <ativelkov> which is supposed to return some generic list of artifacts, including Images
14:20:21 <ativelkov> In the etherpad I've shared earlier we supposed to be able to redefine the database access layers on a per-plugin basis
14:20:48 <ativelkov> But this would not allow us to have a single access point to all the artifacts
14:21:03 <nikhil> curious why not?
14:21:31 <ativelkov> how would you list (with marker-based pagination) objects from different DB tables?
14:22:49 <ativelkov> Even a single get-by-id is tricky, but at least you may iterate all the enabled plugins, until you get an artifact, which is inefficient, but at least doable
14:23:18 <nikhil> ativelkov: um, I am not seeing how complex of a pagination you are thinking. However, for a standard multi-marker (one marker per artifact type situation) we can just make calls to the underlying tables and return as a part of single call.
14:23:38 <nikhil> ativelkov: yes, the case that you describe is pretty inefficient
14:24:11 <nikhil> ativelkov: so, I think the cloud would have to use some sort of rate limiting proxies to expose such calls publicly
14:24:35 <nikhil> ativelkov: and I think for discovery we rather offload the work to searchlight
14:24:42 <ativelkov> "one marker per artifact type" - this looks a bit dirty, as it leaks the abstraction through the API
14:25:08 <ativelkov> what do you mean by discovery?
14:25:15 <nikhil> that was the main motivation behind that project -- even in images once your table grows beyond  a point discovery becomes pretty slow
14:25:29 <ativelkov> you mean complex searches?
14:25:44 <mfedosin> nikhil, there is a question, is it hard to migrate current image tables in db to be more artifactized? i.e. move min_ram and etc. to properties
14:25:44 <nikhil> ativelkov: the 'all' call comes in "discovery of artifacts" concept
14:26:37 <nikhil> mfedosin: I think so yes. We would have to implement some fine grained CRUD restrictions on artifacts properties to keep images immutable prop as so.
14:26:55 <ativelkov> Well, the problem is that searchlite is an optional component, and we cannot expect all the clouds to have it
14:27:42 <mfedosin> hmm, in that case many tough problems are easy solvable...
14:28:08 <ativelkov> so, we need some solution which will provide at least limited listing capabilities in a regular glance v3 api, without doing elastic search
14:28:28 <nikhil> ativelkov: SO, I think performance is in general a problem in OpenStack if we focus too hard on interop; on different levels. You can see yourself on the upload case.
14:28:59 <ativelkov> yup
14:29:23 <ativelkov> Seems like we are out of time here
14:29:23 <nikhil> ativelkov: what I am thinking is --  a sophasticated artifacts deployment (large images tables incl, metadef incl) would strongly consider deploying searchlight
14:29:38 <nikhil> ok, thanks. we can continue somewhere else if we want.
14:29:55 <ativelkov> yes, let's go to #openstack-glance
14:30:00 <mfedosin> thanks
14:30:26 <ativelkov> #endmeeting