14:01:02 <nikhil_k> #startmeeting Glance
14:01:03 <openstack> Meeting started Thu May 28 14:01:02 2015 UTC and is due to finish in 60 minutes.  The chair is nikhil_k. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:06 <openstack> The meeting name has been set to 'glance'
14:01:08 <flaper87> o/
14:01:09 <kragniz> o/
14:01:24 <rosmaita> o/
14:01:38 <bpoulos> o/
14:02:27 <nikhil_k> Thanks, looks like we've decent turnout soon after the summit.
14:02:36 <nikhil_k> We have a short agenda #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:02:45 <abhishekk> 0/
14:02:53 * sigmavirus24 apologizes for being late
14:03:09 <nikhil_k> Thanks for joining guys
14:03:11 <nikhil_k> Let's get started
14:03:14 <nikhil_k> #topic Request from Docker team
14:03:46 <nikhil_k> We had a request from Docker team prior to summit to collaborate on #link https://etherpad.openstack.org/p/liberty-fishbowl-magnum-project-ideas
14:04:10 <nikhil_k> More discussion at #link https://etherpad.openstack.org/p/YVR-ops-containers
14:04:19 <ativelkov> o/
14:04:20 <mfedosin> o/
14:04:33 <TravT_> o/
14:04:45 <ivasilevskaya> o/
14:05:02 <nikhil_k> Cool, more people coming in.
14:05:03 <flaper87> I couldn't attend that session but I'll catch up
14:05:12 <nikhil_k> Thanks flaper87
14:05:44 <nikhil_k> #action everyone: Please go through the links and please share your thoughts related to Glance. If you have concerns/suggestions please bring them up at our meetings.
14:06:05 <nikhil_k> Moving on
14:06:09 <nikhil_k> #topic Domain Model
14:06:15 <nikhil_k> Looks like this was from flaper87
14:06:18 <GB21> hello!
14:06:27 * nikhil_k hands over the mic
14:06:38 <flaper87> yeah
14:06:48 <flaper87> so, it's more like a general note and gathering feedback
14:06:50 <jokke_> o/
14:07:12 <flaper87> I'm not a huge fan of the domain model we have but I do understand that planning a change now might be too crazy
14:07:32 <flaper87> BUT, we do have some areas where we're introducing serious race scenarios that would break HA for glance-api
14:07:32 <ativelkov> well, at least we may consider dropping it for v3 if we decide to
14:07:39 <flaper87> ativelkov: ++
14:07:58 <jokke_> I don't think it's never too late to get rid of that, but ativelkov please ;)
14:08:05 <ativelkov> what's the connection of domain with races?
14:08:11 <flaper87> so, I'd like to get consensus on allowing exceptions were we skip the domain model and just do saves to the database
14:08:13 <mfedosin> domain is too much from Java world I think
14:08:13 <nikhil_k> ativelkov: flaper87 : Can we collaborate on ideas and suggestions on an etherpad?
14:08:21 <flaper87> yes
14:08:23 <flaper87> but wait
14:08:26 <flaper87> let me anser ativelkov
14:08:30 * flaper87 gets a link
14:08:58 <flaper87> #link https://github.com/openstack/glance/commit/b000c85b7fabbe944b4df3ab57ff73883328f40d#diff-dffa5011f2dd4af675a4075b5c33d4adR250
14:09:02 <mfedosin> btw, do you need an article about organization of domain model?
14:09:23 <mfedosin> with examples and pictures
14:09:32 <flaper87> That deactive method gets the image status, checks with an if, sets it to something else and never saves the new value
14:09:40 * jokke_ thinks that if we get rid of eventlet that would be good timing to refactor the domain model as well if we agree to do so
14:09:42 <flaper87> instead it just  goes ahead and lets the rest of the domain calls to happen
14:09:48 <ativelkov> ah, got it
14:09:52 <TravT_> mfedosin: do you have that link?
14:09:54 <flaper87> That kind of operations should happen atomically
14:10:01 <ativelkov> Agree
14:10:13 <flaper87> my proposal for now is, that in cases like this we should just call a function from the db_api and do the update in place
14:10:14 <mfedosin> TravT_, yes, but ut's in russian
14:10:30 <ativelkov> mfedosin: lol :)
14:10:47 <flaper87> I guess doing that meanwhile we get rid of it should be fine
14:10:53 * mfedosin needs some time to translate
14:11:05 <flaper87> it's  not like we're breaking it, just skiping the chain for the sake of safety
14:11:12 <ativelkov> as far as I understand DM is mostly about the separation of concerns
14:11:15 <flaper87> it'll end up in more writes, I guess but mmh
14:11:24 <flaper87> ativelkov: yeah
14:11:32 <flaper87> thoughts ?
14:11:32 <ativelkov> but probably it does not work due to these races
14:11:47 <nikhil_k> Sounds good. Update in place can make things messy if we do not refactor. For example, authZ for admin or not.
14:11:49 <ativelkov> I think I know another example
14:11:49 <jokke_> flaper87: sounds like decent workaround, but I hope we don't need to do too many of those or it gets really messy
14:11:55 <flaper87> I'm happy to do these changes myself but I'd like to avoid suprises on reviews
14:12:00 <flaper87> jokke_: agreed
14:12:08 <ativelkov> the quota thing should be racy as well in this case
14:12:15 <nikhil_k> yeah
14:12:17 <flaper87> we've several cases like this
14:12:19 <flaper87> :(
14:12:36 <flaper87> that's all I have
14:12:42 <flaper87> thanks for listening
14:12:48 * flaper87 hands tea toe veryone \_/?
14:12:49 <nikhil_k> Domain model was introduced in 2012-2013 exactly for such reasons
14:13:11 <jokke_> nikhil_k: to create race conditions? ;p
14:13:22 <ativelkov> so, all these things should happen in transactions, and the reads of that transactions should be done with "select for update"
14:13:27 <nikhil_k> We have expanded our feature domains a bt broad since, so it/s kinda ironic that this does not scale well.
14:13:48 <flaper87> ativelkov: indeed
14:13:58 <nikhil_k> I see
14:14:04 <nikhil_k> that has another tradeoff though
14:14:04 <flaper87> ultimatedly, we should get rid of the DM
14:14:17 <flaper87> but for now, i think this is a good solution to fix these issues
14:14:27 <nikhil_k> If you have multiple select for updates you will have a livelock
14:14:27 <flaper87> with not such a huge impact in performance
14:14:32 <flaper87> Last famous words
14:14:34 <ativelkov> And the current v2 implementation is conceptually wrong: at first it reads the objects from DB (and closes the transaction afterwards), builds domain objects, runs some layers of logic - and then flushes the changes into DB in another transaction
14:15:00 <ativelkov> nikhil_k: yes, livelock
14:15:11 * flaper87 welcomes ativelkov to his hell :D
14:15:17 <ativelkov> and also Galero have issues with select for update
14:15:23 <ativelkov> Galera*
14:15:29 <nikhil_k> hmm, I hear they are fixing it
14:15:45 <nikhil_k> May be we get to know more on how
14:16:06 <sigmavirus24> Yeah we can probably talk to the galera team about that
14:16:13 <flaper87> indeed
14:16:22 <flaper87> I'd like to take small steps towards the final fix
14:16:24 <ativelkov> Yup, but at least for now Galera is likely to throw DeadLockExceptions under heavy loads if the select for update is used
14:16:27 <jokke_> I can ping the guys if needed
14:16:28 <rosmaita> so what's the general idea here, all status changes should be persisted immediately to the DB?  Any other image fields we need to do this for?
14:16:32 <flaper87> first fix the immediate races, then we can do refactors
14:16:37 <jokke_> but galera only solution is not good enough
14:17:03 <flaper87> rosmaita: all ops that require cmp+swap
14:17:26 <nikhil_k> umm, flaper87 I think he may mean some properties too
14:17:53 <ativelkov> yup, quota checks, various kinds of uniqueness checks etc
14:17:56 <nikhil_k> for example the dynamic update to some files like prop protection, (may be) conf
14:18:05 <rosmaita> just trying to guess how many LOC this will impact
14:18:14 <nikhil_k> rosmaita: me too!
14:18:15 <ativelkov> jokke_: that's not "galera only", that's about "galera compliant"
14:18:19 <nikhil_k> thanks for bringing that up
14:18:23 * flaper87 is trying to think about that too
14:18:29 <flaper87> no idea, tbh.
14:18:40 <flaper87> are you going to propose to refactor right away?
14:18:41 <ativelkov> "select for update" is a generic SQL thing, but specifically on Galera it may fail.
14:18:55 <nikhil_k> flaper87: how about we share our thoughts on etherpad to see how much of traction is there for a change and everyone opinions are in a single place?
14:18:56 <jokke_> ativelkov: sure ... just ensuring that we're not trying to solve it with single db vendor specific fix ;)
14:18:57 <rosmaita> flaper87: that's what i'm wondering
14:19:16 <nikhil_k> #link https://etherpad.openstack.org/p/liberty-glance-domain-model
14:19:18 <flaper87> nikhil_k: rosmaita might be worth putting some extra thought on this
14:19:28 <rosmaita> we will have to improve test coverage a lot first, i think
14:19:56 <flaper87> yup
14:20:16 * ativelkov thinks about writing v3 without the domain at all
14:20:31 * flaper87 would be in favor, FWIW
14:20:51 * jokke_ too
14:21:21 <mfedosin> I have nothing against domain model actually, but then you have to suggest better solution
14:21:27 <nikhil_k> #info everyone please share thoughts on DM refactoring and cases when we need and do not need the same: https://etherpad.openstack.org/p/liberty-glance-domain-model
14:21:29 <ativelkov> When we were doing artifacts, domain caused more problems than it solved
14:21:58 <nikhil_k> The etherpad link has been added to our liberty etherpad: #link https://etherpad.openstack.org/p/liberty-glance
14:22:35 <flaper87> ativelkov: did it solve *any* problem? #RandomRant
14:22:48 <ativelkov> flaper87: well, at least we could do things one-by-one
14:23:09 <mclaren> I was a bit late, are we talking about refactoring v2 (as well as our v3 work)?
14:23:18 <nikhil_k> heh
14:23:32 * flaper87 pictures mclaren calling 911
14:23:34 <jokke_> mclaren: just the guts of it ;)
14:23:40 <nikhil_k> mclaren: We are just ranting about Domain Model and hoping Mark is not reading this
14:23:49 <ativelkov> and separate the work between developers: mfedosin was doing DB while ivasilevskaya worked with store: as these were different layers they almost didn't interfere
14:24:12 <flaper87> ativelkov: no no no, don't say anything in favor of it PLEASE!
14:24:14 <flaper87> :P
14:24:20 <nikhil_k> Please everyone collaborate else, we may soon have another divided team that we can't afford this cycle
14:24:23 <flaper87> jokes apart, those are good points
14:24:43 <flaper87> So, etherpad, brainstorm, come back with a plan!
14:24:46 * flaper87 drops mic
14:24:50 <ativelkov> agreed
14:24:55 * jokke_ prepares to fork glance twisted without DM ;)
14:25:10 <nikhil_k> or write it in GO
14:25:11 <nikhil_k> ?
14:25:16 * flaper87 read that as: prepare to fork glance and make it use twisted
14:25:27 <flaper87> RUSTLANG!
14:25:30 <kragniz> let's just do the whole thing in haskell
14:25:32 * flaper87 does the rust dance
14:25:49 * flaper87 pushes kragniz down the hill
14:25:51 <ivasilevskaya> mm.. lisp will also do
14:26:07 * flaper87 drinks ivasilevskaya's tea
14:26:23 <flaper87> ok, really, we can change topic now :D
14:26:28 <flaper87> I have nothing else to say
14:26:32 <flaper87> in case you didn't notice it
14:26:32 * nikhil_k laughs at his evil plan to distract everyone from DM and make mclaren a bit happy
14:26:34 <flaper87> :P
14:27:06 * flaper87 pictures mclaren throwing his pager out of the window into the sea
14:27:14 <mclaren> lol
14:27:17 <flaper87> (inside a box with some weight in it)
14:27:22 <nikhil_k> Thanks all for the thoughts on an important topic
14:27:30 <nikhil_k> Moving on
14:27:33 <jokke_> flaper87: the sea is bit far away, mut I see it rolling on the parking lot ;)
14:28:07 <nikhil_k> #topic Search project update
14:28:14 <TravT_> hey everybody. thank you all for the engaging session at the summit and all the discussion we've had on catalog index service this past year.
14:28:21 <sigmavirus24> flaper87: rust
14:28:27 <TravT_> we've started the process of launching it as a new project. To start with, we've now got a meeting scheduled for 15:00 on Thursdays on #openstack-meeting-4
14:28:29 <nikhil_k> #link https://etherpad.openstack.org/p/search-team-meeting-agenda
14:28:33 <flaper87> jokke_: don't doubt mclaren's strenght
14:28:45 <kragniz> TravT_: so just after this one?
14:28:52 <sigmavirus24> kragniz: yes
14:28:59 <flaper87> TravT_: w0000h000000
14:29:02 <sigmavirus24> TravT_: I'll be there
14:29:04 <TravT_> yep
14:29:14 <TravT_> very conveniently turned out to be open
14:29:27 <sigmavirus24> TravT_: that's probably our fault ;)
14:29:29 <TravT_> and got it approved in the schedule list by ttx last night
14:29:41 <TravT_> i'd like to invite any of you that are interested in helping to move this forward to join in!
14:29:51 <ativelkov> Cool, do you start today?
14:29:54 <TravT_> yep
14:29:58 <TravT_> today is the first meeting
14:30:22 <TravT_> main topics will be around the basics of getting everything setup
14:30:33 <TravT_> including a name
14:30:44 <TravT_> which we have to clear trademark review.
14:31:25 <sigmavirus24> Yeah "Searchlight" might conflict with Fox's Searchlight movie studio
14:31:52 <TravT_> yeah, i'm not clear on trademark law.  my google lawyer skills aren't too effective
14:32:37 * ativelkov remembers three renamings of the project which is currently known as "Murano" - exactly due to copyright issues
14:32:53 <mfedosin> and Sahara too ;)
14:32:54 <TravT_> ahh, i didn't remember that ativelkov
14:33:00 <TravT_> i remembered quantum
14:33:07 <kragniz> ativelkov: copyright shouldn't come into it
14:33:21 <TravT_> and flaper87 renamed macaroni ;)
14:33:30 <ativelkov> kragniz: ehm, I mean trademark. It always confuses me
14:33:31 <mfedosin> Sahara was Savanna previously
14:33:35 <mclaren> http://martinfowler.com/bliki/TwoHardThings.html
14:33:37 <sigmavirus24> TravT_: because people were getting fat while working on it ;)
14:33:42 <TravT_> lol
14:33:44 <sigmavirus24> == mclaren
14:33:47 <kragniz> ativelkov: yeah, trademark != copyright
14:34:11 <sigmavirus24> Trademark law and Copyright law are two similar yet disjoint things that are baffling to everyone including those who write the laws
14:34:45 <TravT_> so, this will be one of the opening topics in the next meeting.  would love for you to stick around for it.
14:35:24 <TravT_> https://etherpad.openstack.org/p/search-team-meeting-agenda
14:35:42 <TravT_> that's all I have at the moment. :)
14:35:43 <sabari> * thinks Searchlight should have a cool logo like the bat signal * ;)
14:35:59 <jokke_> ++
14:36:02 <nikhil_k> nice imagination sabari
14:36:13 <nikhil_k> I always get confused with OSX' spotlight
14:36:13 <TravT_> yeah, anybody have photoshop skills?
14:36:20 <kragniz> whatever the name is, it needs to be easy to come up with a cool logo
14:36:25 * kragniz glares at glance
14:36:30 <sabari> lol
14:36:37 <TravT_> did you guys get a glance patch at the summit?
14:36:45 <kragniz> TravT_: inkscape is better for logos
14:36:48 <ativelkov> we still need a logo for Glance
14:36:59 <jokke_> we have really cool one ;)
14:37:04 <TravT_> kragniz: sounds like you are volunteering
14:37:23 <kragniz> TravT_: come up with a name first!
14:37:31 <jokke_> TravT_: you can have Glance's old ... even before we get new one ;)
14:37:34 <TravT_> i'm hoping searchlight sticks... but we'll see
14:39:22 <nikhil_k> Well now that we have a path forward for CIS, I wanted to thank everyone for the reviews and the contribution made to Glance over the past few months. We now have a way for discovering Data Assets incl. metadefs efficiently.
14:40:17 <nikhil_k> The vision and mission of Glance will now be partially supported by better searching techniques. It started by TravT_'s brilliant idea of metadefs #link http://specs.openstack.org/openstack/glance-specs/specs/juno/metadata-schema-catalog.html
14:40:44 <nikhil_k> And I don't want to miss out on the rest of the team that helped contribute ideas, code, reviews etc
14:41:06 <TravT_> Yes, thanks everybody!
14:41:54 <nikhil_k> If we have enough data in Glance that people can use to discover and analyze to make magic happen in the cloud, the IAAS industry should rock!
14:42:38 <nikhil_k> Only yesterday, we recieved a message from the Auto Scaling team at the Rack; they may have a decent use case for indexing DB.
14:43:20 <TravT_> mclaren and kragniz had the swift PTL approach them asking for indexing...
14:43:32 <nikhil_k> They do need this for instances though and my feeling is that others too so, it makes sense to broaden the scope and help support the search service startng with Glance followed by other important services.
14:44:02 <rosmaita> are there other important services? :)
14:44:07 <nikhil_k> :)
14:44:17 <nikhil_k> At least, now we have another one
14:44:43 <jokke_> Designate guys were interested as well
14:45:50 <sigmavirus24> rosmaita: says the image service product person ;)
14:46:03 <nikhil_k> Also one very important that things that happend in Glance in Kilo. We have some good improvements done in the notifications side due to the priority need of CIS. So, thanks everyone who made this happen; we had a win-win case there.
14:46:43 <mclaren> off topic, but I've a spec up if anyone has some free review cycles: https://review.openstack.org/#/c/170564
14:48:40 <nikhil_k> Last but not the least, I think other clouds would be happy to see this service successful as we had some excellent feedback from sigmavirus24 around his discussion with other ops.
14:49:12 <nikhil_k> That was it from my end.
14:49:23 <nikhil_k> Moving on as we are a bit short of time..
14:49:27 <nikhil_k> #topic V3 post-summit follow-up
14:49:40 <ativelkov> That was mine
14:49:50 <nikhil_k> Thanks, please take the mic
14:50:12 <ativelkov> So, we actually have two patches pending on review, and one has to be updated based on the summit results
14:50:49 <ativelkov> i.e. we need to rename v0.1 to v3 and put it back to glance-api process
14:51:03 <ativelkov> but before we do it I want to discuss one more API-related things
14:51:43 <ativelkov> in the proposed artifact API we had a common URI section for all the artifacts: i.e. we have /v0.1/artifacts/some_type/...
14:52:18 <ativelkov> So, if we now consider it as v3 API: do we still want to have the "artifacts" section or should we drop it?
14:53:05 <ativelkov> i.e. what is better: /v3/artifacts/images, /v3/artifacts/heat-templates etc or /v3/images, /v3/heat-templates etc?
14:53:20 <mfedosin> I think we should get rid of 'artifacts' section
14:53:24 <sigmavirus24> Hm
14:53:30 <ativelkov> https://etherpad.openstack.org/p/glance_v3_API
14:53:32 <mclaren> could there be a future case where we have a resourse which isn't an artifact?
14:53:33 <ativelkov> #link https://etherpad.openstack.org/p/glance_v3_API
14:53:35 <sigmavirus24> That's a good question, and probably a better question for the API-WG
14:53:37 <nikhil_k> I personally like the second one, for resons of a consistent v3 API
14:53:40 <ativelkov> mclaren: we have such cases
14:53:44 <nikhil_k> == sigmavirus24
14:53:50 <ativelkov> for example, there are schemas
14:53:55 <sigmavirus24> Also /v3/images, /v3/templates fwiw
14:53:57 <ativelkov> and there are tasks
14:54:00 <nikhil_k> mclaren: example
14:54:00 <sigmavirus24> not heat-templates (please)
14:54:01 <nikhil_k> ?
14:54:21 <ativelkov> sigmavirus24: the actual name of type-specific section is up to the plugin developers
14:54:45 <sigmavirus24> ativelkov: sure, just saying, templates is a nicer route ;)
14:54:50 <nikhil_k> ativelkov: However, we will keep a published list of templates right?
14:55:09 <rosmaita> ativelkov: how will discovery work? i.e., how do you find out what artifacts are available?
14:55:12 <sigmavirus24> nikhil_k: confusing that with apps.o.o?
14:55:21 <sigmavirus24> rosmaita: good point
14:55:29 <mclaren> if there are non-artifact resources then possibly putting things that have the common artifact behaviour under /artifacts potentially makes sense
14:55:36 <sigmavirus24> /v3/types might be hard to have as an extension if that's how you discover types
14:55:48 <ativelkov> so, the etherpad I've linked has some pros and cons mentioned
14:55:51 <sigmavirus24> I see where mclaren is coming from
14:56:17 <mclaren> and a get of /artifacts potentially lists all artifacts
14:56:27 <rosmaita> mclaren: +1
14:56:29 <ativelkov> mclaren: yes, this makes sence
14:56:34 <sigmavirus24> Yeah I think I'm with mclaren
14:56:43 <sigmavirus24> Especially since tasks probably can't be considered a plugin type
14:56:48 <nikhil_k> ativelkov: your proposal says that /artifacts will have separate endpoints
14:56:54 <sigmavirus24> And someone may want to distribute task artifacts (task files) via /v3/artifacts/tasks
14:57:00 <nikhil_k> how will that discovery proposal work in such case?
14:57:06 <ativelkov> nikhil_k: separate keystone endpoints, yes
14:57:24 <nikhil_k> Then there won't really be discovery on Glance, rather all on keystone
14:58:06 <ativelkov> nikhil_k: yes, this may cover discoverability issue
14:58:23 <nikhil_k> Having it under /v3/<artifact_type> and another call mydomain.org/v3/artifacts-resources can enable discovery
14:58:58 <nikhil_k> naming can be different
14:59:01 <rosmaita> ativelkov: you are thinking a separate service catalog entry for every single artifact-type?
14:59:12 <nikhil_k> yep
14:59:21 <nikhil_k> BTW, we are almost out of time.
14:59:32 <sigmavirus24> I'm not sure about that
14:59:32 <ativelkov> rosmaita: yes, as this was the feedback from def-core according to what I understood from flaper87
14:59:38 <nikhil_k> We should continue the discussion on #link https://etherpad.openstack.org/p/glance_v3_API
14:59:40 <stevelle> shouldn't types discovery happen at /v3/ then
14:59:43 <sigmavirus24> == nikhil_k
14:59:54 * flaper87 was a bit distracted
14:59:54 <ativelkov> yup, any feedback is welcome
15:00:18 <nikhil_k> Sorry ativelkov , we can continue this in th next meeting
15:00:25 <ativelkov> BTW, we may have deeper discussion with API WG - I just need some quick decision now so we can proceed with the commits
15:00:27 <nikhil_k> Heads up: glance_store release coming out soon
15:00:34 <flaper87> w000t
15:00:45 <nikhil_k> ativelkov: please keep us in the loop
15:00:51 <nikhil_k> Thanks all!
15:00:53 <kragniz> nikhil_k: thanks
15:00:56 <nikhil_k> #endmeeting