15:00:50 <TravT_> #startmeeting openstack search
15:00:51 <openstack> Meeting started Thu Mar  3 15:00:50 2016 UTC and is due to finish in 60 minutes.  The chair is TravT_. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:55 <openstack> The meeting name has been set to 'openstack_search'
15:01:25 <TravT_> o/
15:01:36 <nikhil> o/
15:01:42 <yingjun> o/
15:01:53 <lei-zh> o/
15:02:18 <TravT_> sjmc7: are you there?
15:02:35 <TravT_> hi lakshmiS
15:02:54 <sjmc7> hi
15:02:57 <TravT_> how is everybody today?
15:04:02 <sjmc7> sleepy
15:04:11 <lakshmiS> hi travis. more or less same
15:04:17 <rosmaita> o/
15:04:25 <rosmaita> late
15:04:34 <TravT_> yeah, i'm a bit sleepy as well... couldn't help getting up at 1 am and working a bit
15:04:49 <TravT_> anyway, i know it is late ofr lei-zh and yingjun
15:04:55 <TravT_> so, agenda is here https://etherpad.openstack.org/p/search-team-meeting-agenda
15:05:06 <TravT_> add anything you see fit
15:05:24 <TravT_> #topic mitaka 3
15:05:54 <TravT_> The searchlight client has a release!
15:06:03 <TravT_> https://pypi.python.org/pypi/python-searchlightclient/0.2.0
15:06:35 <TravT_> on the downside, I followed old instructions to do it
15:06:43 <TravT_> and bypassed the new route
15:07:04 <sjmc7> any port in a storm
15:07:09 <TravT_> but chatted with dhellmann
15:07:15 <TravT_> and he said it was okay
15:07:24 <TravT_> and to go ahead and put up the normal review
15:07:54 <TravT_> So I put that up:     Release management: https://review.openstack.org/#/c/287439/
15:08:11 <TravT_> I also updated the patch adding SL client to global requirements
15:08:25 <TravT_> https://review.openstack.org/#/c/268394/
15:08:50 <TravT_> once that lands, we can fix our devstack script.
15:08:54 <sjmc7> yay
15:09:04 <TravT_> but whatever the case. great work to all in getting that out!
15:09:26 <lei-zh> cool
15:09:59 <TravT_> so, we also need to tag mitaka 3 for the service
15:10:46 <TravT_> i believe we should see if we can land the neutron plugin and swift plugin today
15:10:54 <TravT_> i've looked through both
15:11:18 <TravT_> but reviews on those two will be especially appreciated
15:11:28 <yingjun> and i uploaded a new patch to fix the README as Travis found: https://review.openstack.org/#/c/287555/
15:11:39 <TravT_> yingjun: thanks!
15:12:18 <lakshmiS> the swift plugin is functionally complete with some patches for review comments coming up today
15:12:34 <TravT_> yeah, let's talk swift for a minute.
15:12:38 <TravT_> #topic swift
15:12:55 <TravT_> the swift team is having a hackathon this week
15:13:12 <TravT_> sjmc7 sent a request to get our need for notifications onto their agenda
15:13:23 <TravT_> turns out a number of people wanted that topic
15:13:42 * notmyname lurks
15:13:49 <TravT_> so yesterday sjmc7 and i dialed in and had a lively discussion with the team.
15:13:53 <TravT_> notmyname: hello
15:14:15 <TravT_> notmyname: i'll give my impression and then you can correct me
15:14:29 <notmyname> I'm listening to one person while reviewing code for another and also in here. yes, please carry on with your summary :-)
15:14:51 <TravT_> search is very interesting to many people involved with swift
15:15:10 <TravT_> and the need to index data is pretty well agreed on
15:15:17 <TravT_> but, doing notifications via traditional middleware and rabbitmq was worrisome to the team.
15:15:36 <TravT_> so a lot of discussion went around about how to best accomplish this
15:16:27 <TravT_> and the agreement seemed to be that we'd like to try an approach that allows swift to call some searchlight library code where they essentially directly send updates through to searchlight / elasticsearch
15:16:56 <TravT_> and the swift team would look at various options to integrate it at the most appropriate layer in swift
15:17:35 <notmyname> yes, that sounds all correct to me
15:17:51 <TravT_> cool
15:18:17 <notmyname> does that make sense to everyone else? I can answer any questions you have about it
15:18:43 <sjmc7> i am in favor of removing moving parts, but i’d like to speak to our ceilometer team about issues they had with rabbit
15:18:54 <notmyname> and we (swift) have one request of you (searchlight): some example code on how to add something to the index
15:18:55 <lakshmiS> i guess the question would be - is that the only option?
15:19:04 <notmyname> that == no rabbit?
15:19:10 <lakshmiS> yes
15:19:54 <notmyname> we've had at least 2 medium to large production clusters that have specifically rejected using rabbit to transport notifications from swift. seems the problems are mostly with the receivers keeping up
15:20:00 <sjmc7> notmyname: we can (will) point you at the current code, though it’d obviously need some refactoring to be plugged into other projects
15:20:34 <sjmc7> ah, yes, rmq doesn’t deal well with that
15:20:38 <notmyname> and other deployers who are exploring search integration as well and are looking for more efficient ways to do index updates
15:21:08 <sjmc7> yeah, doing it directly is certainly more efficient
15:21:28 <lakshmiS> yeah in that case direct updates to searchlight will more or less resemble the plugin code in searchlight
15:21:29 <sjmc7> handling errors (which we spoke about) is potentially complicated
15:21:33 <sjmc7> but we can deal with that
15:22:07 <notmyname> yeah, and the thought is basically "we're just dumping stuff on a queue so that it can go to an indexer and put it on its own work queue. can we just skip that first part?"
15:22:25 <TravT_> notmyname: we were thinking we could get a bit of that code refactored to support this usage concept as well before sending it to you.
15:22:39 <notmyname> cool
15:22:44 <sjmc7> right. although the argument in favor of queues is that in theory they can handle burst high loads
15:22:45 <TravT_> notmyname: question for you
15:23:02 <TravT_> if SL or ES is down, does swift have built in mechanisms for retry
15:23:15 <TravT_> or would any client code of SL need to handle that?
15:23:20 <notmyname> softlayer or espana?
15:23:36 <notmyname> I don't know what SL or ES is
15:23:52 <TravT_> SL = searchlight.  ES = elasticsearch
15:24:30 <notmyname> but yeah. if you heard on the phone, our conversation was that *anything* we do has to be best-effort. swift will need to retry if it doesn't get a success from the other side
15:24:34 <notmyname> ok :)
15:24:56 <TravT_> notmyname
15:25:02 <TravT_> yeah, that's what we were wondering
15:25:35 <TravT_> and i was thinking we could potentially use messaging as backup
15:25:44 <notmyname> retrying and algorithms that still make progress is a common pattern in swift. although this code isn't yet written (so technically, no, we don't yest have built in retries for it), I wouldn't expect it to land without that functionality. it's the only way to scale
15:26:07 <TravT_> ok.
15:26:22 <TravT_> well, i think this is great progress
15:26:34 <TravT_> and we're very interested in trying out this direct pattern
15:26:45 <notmyname> no, I don't think we'd want to see messaging as a backup. messaging (even if it scaled) is viewed as the optimization that can fail and itself needs a backup. message queues arent' reliable
15:26:59 <lakshmiS> :)
15:28:01 <TravT_> btw, for everybody here.  Monday night i put together a brief demo of POC swift object search in the new horizon searchlight panel https://youtu.be/-Io8uaNLrVc?vq=hd720
15:28:09 <notmyname> cool
15:28:38 <nikhil> that's awesome!
15:29:29 <TravT_> ok, so anything else to talk about re: swift for today?
15:29:36 <lakshmiS> so for mitaka, SL swift plugin will be tagged as Experimental until notification middlware lands(direct call from swift in future)
15:29:59 <TravT_> right. we did the same last release for glance metadefs.
15:30:05 <TravT_> very clearly documented.
15:30:11 <TravT_> disabled by default
15:30:28 <lakshmiS> yes in devstack and docs
15:31:03 <lakshmiS> will pull sjmc code and push it to SL repo today
15:31:27 <lakshmiS> which is a swift patch currently
15:33:37 <TravT_> ok, on to next
15:34:15 <TravT_> #topic feature freeze exceptions
15:34:45 <TravT_> it looks like zero downtime reindexing will need it.  as well as cinder plugin
15:34:56 <TravT_> both have had several rounds of patches
15:35:06 <TravT_> and are close
15:35:49 <sjmc7> i’m working on cinder today
15:36:08 <TravT_> lei-zh what is the status on the zaqar forwarder?
15:37:15 <lei-zh> I'll rebase it and some feedback is welcome
15:37:27 <yingjun> any link for the cinder one?
15:37:49 <lei-zh> should I extract the data enhancement from the es indexing plugins
15:38:00 <sjmc7> yingjun: https://review.openstack.org/#/c/280273/
15:38:02 <TravT_> yingjun: 280273
15:38:20 <yingjun> ok, thanks
15:40:25 <TravT_> ok
15:40:41 <TravT_> so, let's all attack reviews.
15:40:55 <TravT_> is there ability / interest in a live review day next Tuesday?
15:41:01 <sjmc7> sure
15:41:31 <lakshmiS> what time?
15:41:48 <TravT_> I can do pretty much all morning
15:42:02 <TravT_> so what time works for you?
15:42:10 <TravT_> i know this is late for yingjun
15:42:13 <rosmaita> TravT_: global openstack hackdays are 7-9
15:42:19 <TravT_> ah, right
15:42:25 <lakshmiS> is that pst?
15:42:32 <TravT_> march 7 - 9
15:42:36 <lakshmiS> oh
15:43:00 <TravT_> #topic hackdays
15:43:14 <TravT_> rosmaita: nikhil: how are these different than just doing my every day job?
15:43:40 <nikhil> not much, for most active folks
15:43:56 <nikhil> it will be a lot of active folks all over the world
15:44:01 <rosmaita> big difference for me
15:44:03 <nikhil> you may see a surge of bugs taken
15:44:27 <nikhil> possibly and hopefully a ton for you to review
15:44:55 <nikhil> foundation and team wants it to help make a stable and successful release
15:45:06 <nikhil> so the side-effects are accordingly
15:45:20 * nikhil done
15:45:36 <TravT_> ok. well, it sounds very interesting.
15:45:43 <TravT_> last week you had a link
15:45:46 <TravT_> can you share again?
15:46:15 <nikhil> https://twitter.com/nikhilskomawar/status/704518613026230273
15:46:27 <nikhil> TravT_: oh, which link?
15:46:41 <nikhil> I got that one for lakshmiS (scrolling back logs)
15:47:27 <TravT_> oh looks like we'd want to add candidate bugs to here: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka-Bugs
15:47:27 <nikhil> I got that twitter as I am hoping there was be some conversations around it
15:47:36 <lakshmiS> cool
15:47:38 <nikhil> some folks said they had some ..
15:47:47 <nikhil> meh :|
15:47:56 <nikhil> TravT_: correct
15:48:10 <TravT_> so we can put up new bugs for fixing and bug reviews
15:48:12 <TravT_> ?
15:48:34 <nikhil> TravT_: yes please
15:49:02 <TravT_> well, maybe we should just do that right now
15:49:17 <nikhil> s/was/will/
15:49:27 <TravT_> https://bugs.launchpad.net/searchlight
15:49:50 <TravT_> any on there we would like to see if we can get attention on from bug day?
15:50:26 <sjmc7> 1538349 has a patch up but is tricky to test
15:51:13 <sjmc7> it’s sort of tough figuring out how much effort it would be for someone to come in cold and be able to work on things
15:51:18 <TravT_> yeah, i know
15:51:26 <nikhil> oh!
15:51:34 <nikhil> I've seen that in glance before
15:51:38 <sjmc7> yeah :)
15:51:43 <sjmc7> we’rte using the same code :)
15:51:48 <TravT_> but if somebody has interest, would love to enable them.
15:51:56 <nikhil> haha
15:52:01 <nikhil> ++
15:52:20 <nikhil> I think it will help
15:52:24 <sjmc7> maybe some of the e-s 2.0 compatibility stuff
15:52:32 <nikhil> we can give prize in the NYC center who solves it!!! :D
15:52:41 <sjmc7> that’s not urgent for this release but we will need in N
15:52:47 <nikhil> muhaha
15:52:55 <sjmc7> i think we’ll find more bugs next week as we start testing more
15:52:59 <TravT_> And the "Solved ALL THE THINGS" award goes to....
15:53:09 <sjmc7> last release you found a bunch from working on the horizon panel as we got up to the summit
15:53:24 <TravT_> maybe we should have a bug for each of the sets of missing functional tests.
15:53:32 <sjmc7> yeah..
15:53:40 <nikhil> TravT_: https://bugs.launchpad.net/searchlight/+bug/1526856
15:53:41 <openstack> Launchpad bug 1526856 in OpenStack Search (Searchlight) "Refactor plugin unit test code" [Medium,New]
15:53:54 <nikhil> I think someone would def like that one
15:54:50 <TravT_> sjmc7 you okay with that?
15:54:58 <TravT_> let's open a bug for each group of functional tests
15:55:09 <sjmc7> sure
15:55:13 <TravT_> maybe we should open a bug for functional tests needing to run on zuul
15:55:21 <TravT_> that'd be a great one to get solved
15:55:25 <sjmc7> yeah.
15:55:30 <TravT_> it is currently a BP
15:55:37 <TravT_> but i'd be happy to switch it to a bug
15:55:49 <sjmc7> getting e-s installed on the test nodes is i think the main blocker
15:55:56 <nikhil> you can link bugs to BP
15:56:07 <TravT_> ok, I'll open that one and add to the etherpad
15:56:15 <nikhil> that way all of the tests are congregated
15:57:10 <TravT_> lakshmiS sjmc7 any chance you could open a bug for each set of functional tests missing?
15:57:18 <TravT_> then add to the bug smash etherpad?
15:57:22 <lakshmiS> sure
15:57:58 <TravT_> cool, thanks
15:59:46 <TravT_> ok will add more later, but i opened the zull one
15:59:47 <TravT_> https://bugs.launchpad.net/searchlight/+bug/1552767
15:59:48 <openstack> Launchpad bug 1552767 in OpenStack Search (Searchlight) "Enable running of functional tests in zuul" [Undecided,New]
16:00:01 <TravT_> it looks like we are out of time for today
16:00:07 <TravT_> thanks everybody
16:00:09 <TravT_> #endmeeting