15:01:22 <TravT> #startmeeting openstack search
15:01:22 <openstack> Meeting started Thu May 12 15:01:22 2016 UTC and is due to finish in 60 minutes.  The chair is TravT. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:25 <openstack> The meeting name has been set to 'openstack_search'
15:01:35 <TravT> o/
15:01:42 <yingjun> o/
15:01:44 <rosmaita> o/
15:01:51 <lei-zh> o/
15:02:03 <sjmc7> ello
15:02:17 <TravT> hello everybody!
15:02:40 <TravT> so meeting agenda is here
15:02:41 <TravT> https://etherpad.openstack.org/p/search-team-meeting-agenda
15:02:46 <TravT> please add anything you see fit
15:02:48 <lakshmiS_> o/
15:03:08 <TravT> i'm going to skip the first topic and come back to it, because i don't see tyr yet
15:03:44 <TravT> #topic versioned notifications
15:04:06 <TravT> So, we brought it up a bit last week
15:04:09 <TravT> http://lists.openstack.org/pipermail/openstack-dev/2016-May/093962.html
15:04:30 <TravT> in the context of the nova v2 cells announcement that they were going to look into searchlight for multi-cell deployments
15:04:50 <TravT> i spent some time going through the specs
15:04:56 <TravT> and chatted a bit with sjmc7
15:05:16 <sjmc7> i was honored
15:05:19 <TravT> i'll just share a few thoughts and then maybe yingjun also had some time
15:05:25 <TravT> to look over things
15:05:56 <TravT> anyway, it seems that versioned notifications will certainly help with being able to tell when notification payloads change
15:06:06 <TravT> but they don't necessarily guarantee we'll get all the data
15:06:12 <sjmc7> well, that depends what they decide to do :)
15:06:18 <sjmc7> or have they made a decision?
15:06:45 <TravT> spec is still unchanged
15:06:50 <TravT> still has a few -1's
15:06:57 <TravT> i went to what I thought would be the meeting time this week
15:07:01 <TravT> but nobody appeared
15:07:07 <TravT> so maybe they are still doing in every two weeks
15:07:44 <sjmc7> so there actually is a chance it’ll make things worse than now if they become just a delta
15:08:09 <TravT> sjmc7 do you want to explain that a bit for everybody here
15:08:35 <sjmc7> right now notification payloads are mostly identical indepdenent of the event
15:08:49 <sjmc7> and they contain a (mostly) full representation of the instance
15:09:12 <sjmc7> there is some talk of moving to sending payloads that are deltas of what the event changed
15:09:41 <sjmc7> since elasticsearch has no ‘update’ mechanism per se, it’s a bit more work for us to do that (though not insurmountable)
15:09:51 <sjmc7> but it increases the risk of dangerous race conditions
15:10:12 <sjmc7> because we can no longer ignore notifications older than the record we have stored
15:10:35 <sjmc7> right now it’s moot since we hit nova’s API but we really need to stop doing that if possible
15:11:29 <sjmc7> i’m done explaining
15:11:30 <TravT> sjmc7: i'm not sure exactly where is the best place to leave those comments, but maybe you could just add them to this spec in the comments so they are captured somewhere
15:11:31 <TravT> https://review.openstack.org/#/c/286675/
15:11:46 <sjmc7> yeah, will do
15:12:00 <sjmc7> it’s not clear what the motiviation for making the versioned change is (i.e. the specific use cases)
15:12:14 <sjmc7> so we’ll see how much influence we have
15:12:45 <TravT> so, the main motivation as i understand it is so that consumers know when the payload changes
15:12:57 <TravT> eg. version 1.0 of notification to version 1.1 to 2.0
15:13:12 <sjmc7> which consumers though?
15:13:35 <sjmc7> i get the overall reasoning, i’m just not sure what drives specific decisions
15:13:40 <TravT> that is unclear and is part of the spirit of the notes i left on the spec
15:13:43 <sjmc7> but i guess we’ll find out
15:14:03 <sjmc7> the advantage of it all though is that they can potentially add info to payloads
15:14:12 <TravT> yes.
15:14:17 <sjmc7> which might let us stop mashing at nova’s api
15:14:25 <TravT> so, sjmc7 and i were thinking that we should do the following in newton
15:14:46 <sjmc7> interestingly, one counter argument was that it would actually be better if services could query nova’s api more cheaply and do exactly what we do now
15:15:10 <TravT> 1) update our callbacks and mapping to nova to request the latest version
15:15:25 <TravT> 2) https://blueprints.launchpad.net/searchlight/+spec/nova-optionally-disable-some-events
15:16:13 <TravT> 3) keep working on versioned notification with the nova team as consumers, but since we can't count on them 100% do 1) & 2)
15:16:40 <TravT> thoughts? (hope we didn't suck the oxygen out of the room)
15:17:14 <sjmc7> breathe!
15:17:26 <rosmaita> i got nothin'
15:17:53 <TravT> rosmaita coffee running low today? ;)
15:18:25 <rosmaita> TravT: just one large this morning, guess that's the problem
15:18:41 * rosmaita needs another coffee with extra oxygen
15:18:52 <TravT> well, i know we just threw a lot of info out there
15:19:28 <TravT> we can come back to this if others have more thoughts.
15:19:44 <TravT> #topic backport 0.2.1
15:19:57 <TravT> still wanting to finish out items so we can tag stable/mitaka
15:20:15 <TravT> definite ones are bugs relates to ES 2.0
15:20:35 <TravT> i believe this is the last one for the service: https://review.openstack.org/#/c/311834/
15:21:00 <sjmc7> gonna look at it again today. it was more complicated than i initially thought
15:21:21 <TravT> okay
15:21:30 <TravT> can you also cherry pick this one to stable/mitaka: https://review.openstack.org/#/c/307504/
15:21:34 <TravT> ?
15:22:01 <sjmc7> yeah. looks like there’s a merge conflict but i’ll do it after this
15:22:11 <TravT> sjmc7: i think backporting the nova spec could be a follow on backport
15:22:16 <TravT> https://review.openstack.org/#/c/307504/
15:22:18 <TravT> oops
15:22:31 <TravT> https://blueprints.launchpad.net/searchlight/+spec/nova-optionally-disable-some-events
15:22:37 <TravT> for a different release tag
15:22:45 <sjmc7> yeah, that’s fine. we’re not really meant to backport features, though this one’s kind of a bug-ture
15:23:10 <TravT> yeah, it could perhaps be considered a scalability bug...
15:23:42 <TravT> ok, still not tyr or david-lyle it looks like, so will come back to the searchlight ui topic
15:23:59 <TravT> looks like rosmaita topic is next
15:24:01 <david-lyle> TravT: I'm here
15:24:08 <TravT> hey david-lyle
15:24:08 <david-lyle> just reading along
15:24:16 <TravT> okay
15:24:20 <TravT> #topic upcoming changes to glance that will affect the searchlight plugin
15:24:24 <TravT> rosmaita: ^
15:24:28 <rosmaita> ok
15:24:41 <rosmaita> the change is that there will be more image 'visibility' values
15:25:03 <sjmc7> we like a challenge :)
15:25:15 <rosmaita> and the community scheme will allow people to create quasi-public images
15:25:19 <rosmaita> anyone can use them
15:25:29 <rosmaita> but they only appear in your image-list if you "bookmark" them
15:25:45 <rosmaita> by using the member-list thing we already use for 'shared' images
15:25:56 <rosmaita> and 'shared' will become an actual visibility
15:26:16 <rosmaita> and after that, there's a push for what i'm calling 'protected' images
15:26:25 <rosmaita> (though there will be bikeshedding over the name)
15:26:58 <rosmaita> these will be images that appear in the default image-list of descendant tenants in the keystone hierarchical tenancy scam
15:27:06 <rosmaita> i mean 'scheme'
15:27:10 <sjmc7> hahahaha
15:27:19 <TravT> lol
15:27:22 <sjmc7> tip your waiter!
15:27:33 <rosmaita> so, it occurs to me that that is a lot of logic to be duplicating into the searchlight plugin
15:27:39 <sjmc7> yeah :(
15:27:57 <david-lyle> still don't understand why HMT won't die
15:27:57 <rosmaita> and also, i wonder whether there is any thing we can do in glance to make this easier for indexing
15:27:59 <sjmc7> as long as we know what it is towards the end of a release, it’s not a huge deal
15:28:01 <TravT> community concept didn't initially freak me out (maybe it should), but that second one scares me
15:28:14 <sjmc7> the big concern is that we are never more permissive than glance
15:28:25 <rosmaita> right
15:28:38 <rosmaita> anyway, i wanted to put this on people's radar
15:28:47 <sjmc7> yeah, thanks. are there specs/bps for this?
15:28:53 <TravT> https://review.openstack.org/#/c/271019/
15:28:58 <rosmaita> on the agenda
15:29:09 <TravT> so rosmaita, shared will actually be a value now?
15:29:11 <sjmc7> ah! splendid
15:29:27 <rosmaita> TravT: yes
15:29:59 <TravT> and all deployments will get this by default?
15:30:04 <TravT> for v2?
15:30:25 <rosmaita> well, it will be in newton (if the spec is approved)
15:30:40 * rosmaita has 50 windows open and can't find the relevant ones
15:30:54 <TravT> community sounds pretty easy
15:31:17 <rosmaita> here's the community spec, close to approval: https://review.openstack.org/#/c/271019/
15:31:30 <TravT> the diff between get by ID actually is a new use case...
15:31:33 <TravT> :-S
15:32:15 <rosmaita> the multitenancy stuff is discussed here, though it's preliminary: https://etherpad.openstack.org/p/newton-image-visibility-changes
15:32:28 <TravT> basically, community is "if i know the ID of it, I can get it directly".  "If i'm added as a member to it, I can list it"
15:32:34 <TravT> rosmaita: ? ^
15:32:51 <rosmaita> TravT: yes, plus you can discover community images
15:33:03 <rosmaita> GET v2/images?visibility=community
15:33:03 <TravT> how do you discover them?
15:33:59 <rosmaita> and the listing is subject to the member_status==accepted, just like shared images
15:35:03 <rosmaita> anyway, just wanted to request a quick look-over if y'all have some free time, just to point out if we're proposing anything insane
15:35:43 <TravT> thanks!
15:35:57 <sjmc7> as long as we can translate the api to filters we’re good
15:36:03 <TravT> what's the probability of all this making it into newton?
15:36:17 <TravT> from a glance perspective
15:36:34 <TravT> rosmaita would you mind opening a searchlight BP for each of these?
15:37:11 <rosmaita> you make it sound so easy
15:37:27 <TravT> :)
15:37:28 <TravT> https://blueprints.launchpad.net/searchlight/+addspec
15:37:32 <TravT> ;)
15:37:45 <rosmaita> sorry, that comment was addressed to sjmc7
15:37:54 <rosmaita> but sure, i can open a bp
15:37:56 <sjmc7> :)
15:37:58 <rosmaita> for each
15:38:21 <rosmaita> community should make it into newton
15:38:24 <TravT> everything is easy for sjmc7... at least he makes it look that way
15:38:47 <sjmc7> i’m actually juggling plates right now
15:39:11 <TravT> okay, next topic
15:39:17 <TravT> was hoping tyr might show up
15:39:21 <rosmaita> nikhil has 'image sharing changes' as a priority for newton, that will def include the community spec, possibly the hierarchical
15:39:37 <rosmaita> (sorry, i have severe typing lag today)
15:39:58 <nikhil> ++
15:40:23 <TravT> okay cool
15:40:38 <TravT> rosmaita all done with this topic for now?
15:40:47 <rosmaita> yep
15:41:05 <TravT> #topic Searchlight UI as top level Dashboard instead of panel
15:41:18 * sjmc7 puts on helmet
15:41:30 <TravT> tyr isn't here, but we have matt-borland and david-lyle
15:41:36 <matt-borland> o/
15:41:40 <TravT> i took a screenshot http://imgur.com/a/p7dmB
15:41:59 <TravT> basically right now, the SL ui is a panel under the "Project" dashboard
15:42:12 <TravT> tyr has a patch up to move it into its own dashboard
15:42:21 <TravT> https://review.openstack.org/#/c/313666/
15:42:52 <TravT> right now, you can already check an option to search across projects from the SL dashboard (if you are admin)
15:43:18 <TravT> and with yingjun's latest patch merging (hypervisors), SL now also has data that is admin only and only shown to admins
15:43:55 <TravT> david-lyle had some comments on the patch above^
15:44:17 * david-lyle looking for trouble
15:44:48 <TravT> so, just wanted to open this up for any additional discussion
15:44:55 <david-lyle> I just wanted to point out that mixing admin and owner roles can be confusing to users as well as complicate the action invocation
15:44:56 <sjmc7> i’m not a huge fan of it
15:45:19 <sjmc7> from a consistency standpoint i think it makes more sense to make it available from Project and Admin
15:45:38 <sjmc7> for pretty much the same reasons as david-lyle
15:46:10 <matt-borland> sjmc7, does that basically mean that scope is limited to project when within 'project', and across projects when in 'admin'?
15:46:13 <sjmc7> i think the usage is likely to be quite different in those two contexts
15:46:17 <sjmc7> yeah
15:46:42 <sjmc7> it would also allow us to potentially define changes in behavior in the two contexts
15:46:45 <david-lyle> an admin very rarely wants to do more than observe unless cleaning up a problem
15:47:02 <david-lyle> where as an admin acting on their own project will make changes freely
15:47:19 <matt-borland> I think that although it seems strange from a dev's perspective, 2 searches makes more sense from an implementation/use perspective
15:47:28 <david-lyle> the line of ownership gets blurry
15:48:01 <matt-borland> also is becomes really unclear how/when you can perform actions on the project side
15:48:09 <TravT> so, david-lyle we have also talked about top-nav search (a searchbar from top nav).  if a user clicked into that, wouldn't that face a similar problem.
15:48:11 <matt-borland> (when you lump everything together across projects)
15:48:20 <sjmc7> two searches also allow us to e.g. change default columns without it looking weird
15:48:38 <sjmc7> the top search wouldn’t have actions in the same way, at least in the incarnation we showed in vancouver
15:48:44 <david-lyle> TravT: yes and no, depending on whether you expose the actions or redirect
15:48:53 <TravT> but would you link them into projects or into admin
15:48:58 <david-lyle> yes
15:49:23 <david-lyle> there is some UX needed to make that choice clear
15:49:35 <sjmc7> off the top of my head i’d defauit top search to all but make it clear in results which ones were in your current project
15:49:46 <david-lyle> sjmc7: ++
15:50:06 <sjmc7> being in the left-nav causes a lot of the inconsistency for me
15:50:17 * matt-borland is in another meeting, ping if needed
15:50:22 <TravT> so, here's a couple of items to consider either way
15:50:28 <sjmc7> it feels like it fits more neatly in the existing dashboards to me
15:50:55 <TravT> the available plugins will need to expose context
15:51:13 <TravT> right now, if you come in with an admin token and list plugins, you just get them all
15:51:33 <TravT> so if the ui remains in project and gets an admin one, then project should not show them
15:51:37 <sjmc7> yeah, there is a BP to implement ‘admin only’ plugins whatever that means
15:51:54 <sjmc7> i agree that’s currently a bit weird. although it’s weird however this discussion goes
15:52:02 <TravT> there also is another alternative to think about if we go with a global search
15:52:32 <TravT> maybe SL could provide data on each item as to why you got it?
15:52:50 <david-lyle> another option is change how the actions are invoked
15:52:52 <TravT> the only way I know of doing this would be to to do both a project scoped search and if admin also doing admin scoped search
15:53:10 <TravT> and then mark the admin only ones as such
15:53:10 <david-lyle> if there is a choice of admin vs owner, make the user explicitly select that
15:53:24 <david-lyle> could still be in a separate dash
15:53:33 <david-lyle> I just think the UX needs more thought
15:53:47 <sjmc7> yeah, i’m not gonna argue it to death
15:53:53 * david-lyle trails the discussion a bit
15:54:01 <sjmc7> but this discussion might’ve been better with some mockups beforehand rather than a full review
15:54:02 <TravT> the actions are definitely problematic
15:54:28 <david-lyle> 2 things to be separate dash
15:54:41 <david-lyle> 1) clear indicator of ownership vs not
15:54:58 <david-lyle> 2) very explicit user choice as to scope of the action
15:55:28 <david-lyle> again fixable, but it's not just as trivial as throwing the views into a new dash
15:57:09 <TravT> the scope of actions seems rather tricky... with a lot of possible implications.
15:57:23 <david-lyle> yes
15:57:35 <david-lyle> and batch actions become a nightmare of scope
15:57:45 <david-lyle> not sure whether those are in plan
15:58:06 <sjmc7> ooo!
15:59:01 <TravT> okay, so, we are low on time, but i'd like to continue this discussion later if possible.
15:59:07 <sjmc7> in the thunderdome!
15:59:11 <TravT> i think we need a little matrix of scenarios
16:00:05 <TravT> in any case, i think we need to give more info on plugins
16:00:15 <TravT> or filter by project or something
16:00:20 <TravT> but we are out of time for this meeting
16:00:26 <TravT> thanks everybody
16:00:28 <matt-borland> good discussion, good points!
16:00:36 <TravT> #endmeeting