Tuesday, 2015-08-25

TravTif you are here, please add +1 to this: https://review.openstack.org/#/c/212092/00:59
TravTwe need it merged so fresh devstack installs work.00:59
sjmc7it's been merged01:32
TravToh, good. my bugging them worked.01:35
*** lakshmiS has joined #openstack-searchlight05:21
*** lakshmiS has quit IRC05:35
*** GB21 has joined #openstack-searchlight07:09
*** GB21_ has joined #openstack-searchlight07:09
*** GB21_ has quit IRC07:11
*** lakshmiS has joined #openstack-searchlight07:21
*** GB21_ has joined #openstack-searchlight07:49
*** GB21 has quit IRC07:53
*** GB21_ is now known as GB2109:00
*** GB21 has quit IRC09:14
*** GB21 has joined #openstack-searchlight09:24
*** sigmavirus24_awa is now known as sigmavirus2413:00
david-lyleTravT: when you're around ping me re: roles13:29
* david-lyle says as he walks afk for a few13:29
david-lyle;)13:29
*** lakshmiS has quit IRC13:30
TravTdavid-lyle: i'm here, but my computer is experiencing some issues this morning.  so, you'll probably see my drop in and out of IRC a few times as I restart.13:58
TravTbbiab13:58
*** TravT has quit IRC13:59
*** TravT has joined #openstack-searchlight14:04
david-lyleTravT: so assume I'm illiterate and couldn't read your email, what was you question about roles?14:09
david-lyleyou are using openstackclient to verify roles?14:09
TravThey.14:11
TravTdavid-lyle14:11
TravTi did a fresh devstack last night14:12
TravTand i was expecting to be able to see via openstack client that the searchlight user has the admin role on the service project14:12
TravTbut it doesn't seem to list them.14:12
TravTi'll try a rest request here.14:12
david-lylerestacking with master14:31
david-lyleTravT: there is a tool that helps visualize all these role assignments in a visual manner. You might have core status on it?14:32
TravTsounds interesting. are you saying it has a GUI. maybe even a web interface?14:33
david-lylesomething like that14:34
david-lyle:D14:34
david-lyleor when in doubt just mysql14:34
TravTok, i used this "gui" tool you mentioned14:35
TravTand it is showing searchlight admin in service project.14:35
david-lylethat's correct14:35
TravTso, maybe something weird with openstack client14:35
TravTi'm getting paste deploy errors again.14:36
* TravT changed topics14:36
david-lylebecause of bug 968696 admin in service is admin for all as long as your token is scoped to service14:37
openstackbug 968696 in Cinder ""admin"-ness not properly scoped" [Undecided,In progress] https://launchpad.net/bugs/968696 - Assigned to Brent Roskos (broskos)14:37
TravTahh, the famous bug14:38
*** lakshmiS has joined #openstack-searchlight14:38
david-lylethat's the one14:39
* david-lyle back in a minute, but here15:00
TravTCourtesy Searchlight review session reminder in #openstack-searchlight: lakshmiS, nikhil_k, rosmaita, sigmavirus24, TravT, krykowski, david-lyle, wokuma, sjmc715:01
* sigmavirus24 waves15:02
lakshmiSo/15:02
sigmavirus24Wayne's not here it seems15:02
TravTno, i think that's okay though15:02
TravTI think others will trickle in15:04
TravTbut let me share out the reviews link15:04
TravTReviews link: http://tinyurl.com/searchlight-reviews15:04
TravTThese four are all high / critical / essential15:05
TravTAdd 'sort' parameter, Fix RBAC filters, Enable configurable index names, Add nova instances plugin.15:05
TravTI've already gone through the first 3, but maybe lakshmiS / sigmavirus24 could look at them?15:06
TravTI'm in the middle of testing the latest nova patchset15:06
lakshmiSHave been busy with functional test issues but I already tested the two of them so will go ahead and +2 those two15:07
TravTlakshmiS, the elasticsearch devstack package was broken, but it should be fixed now15:09
TravTFYI15:09
TravThttps://review.openstack.org/#/c/212092/15:09
TravTceilometer team found it and submitted fix, but it wasn't getting love15:09
TravTso i started bugging cores on devstack15:09
sigmavirus24sorry, not sorry that I -1'd something with a +215:10
lakshmiSTravT: thx for the link15:10
TravTsigmavirus24, which one?15:11
david-lylesort15:12
lakshmiShttps://review.openstack.org/#/c/206268/15:12
TravTdon't be sorry about -1.  that's why we have multiple reviewers.15:12
sigmavirus24sorry, jumping between a video call and reviews15:13
TravTno worries...15:13
sigmavirus24I don't know if we want to let ES handle that kind of junk15:13
sigmavirus24But it seems like we would do well by our users to error early since we're already type checking15:13
nikhil_ka bit late, sorry. was on a urgent call15:15
sjmc7hi, apologies for being late15:15
TravThey sjmc7 see sigmavirus24's comment on https://review.openstack.org/#/c/206268/15:15
openstackgerritMerged openstack/searchlight: Enable configurable index names  https://review.openstack.org/20767215:15
TravTi think it makes sense15:15
sjmc7yeah, i will take a look15:17
sjmc7there will be things we can't catch, and we should also look at how we handle errors from e-s15:17
rosmaitaTravT: sorry i'm late ... anything simple for me to look at?15:20
rosmaitaor complicated, if you don't need a quick response15:21
TravThi rosmaita:15:21
TravTHere's all the outstanding reviews:15:21
TravThttp://tinyurl.com/searchlight-reviews15:21
rosmaitaty15:21
TravTi do have a question on nova for you15:22
rosmaitasure15:22
TravTIn this file:15:22
TravThttps://review.openstack.org/#/c/198852/15/searchlight/elasticsearch/plugins/nova/servers.py15:22
TravTline 3115:22
TravTfor sjmc7 too15:22
nikhil_kTravT: thanks for the dashboard link, I reviewed a couple easy ones that had jenkins +115:22
TravTis hostId really not sensitive?15:23
TravTor should we be filtering it as well.15:23
nikhil_ksjmc7: or TravT : does anyone want me to rebase https://review.openstack.org/#/c/212563/ ?15:23
sjmc7TravT i looked into it...15:23
nikhil_kor someone already taking care of that?15:23
sjmc7it looks like hostId is a hash of project+host name15:23
sjmc7and is judged not sensitive. it's also probably not very useful, so i'm fine redacting it15:23
rosmaitasjmc7: +1 -- it's obscured15:23
TravTok.15:24
rosmaitaactually, it is useful15:24
rosmaitayou can tell if you're on the same host or not15:24
TravTsjmc7, maybe you could make your inline comment have that info15:24
sjmc7ok15:24
TravTnikhil_k: up to sjmc7, but I'd be fine with you rebasing that one15:24
rosmaitabut that's just you, you can't use someone else's hostId to tell if you're on the same host as them to conduct a denial of resources attack15:24
sjmc7rebase away, nikhil_k15:25
rosmaitabut you can tell if your instances are same host or not, which sometimes you care about15:25
nikhil_ksjmc7: aight!15:25
TravTsjmc7: would you actually mind making the nova patch depend on that RBAC patch?15:26
sjmc7nope, i'll do that15:27
TravTsjmc7: lakshmiS:15:27
TravThttps://review.openstack.org/#/c/198852/15/searchlight/elasticsearch/plugins/nova/servers_notification_handler.py15:27
TravTline 9715:27
lakshmiSTravT: why the dependency15:27
TravTlakshmiS: because i test all these different combinations, including making instances in different projects15:28
TravTand it would be nice to have that rbac one in front of it.15:28
lakshmiSbut the RBAC patch only does glance15:29
TravToh, let me see.15:29
TravTyou are right15:29
TravTnevermind: sjmc715:30
TravTso, back to15:30
TravThttps://review.openstack.org/#/c/198852/15/searchlight/elasticsearch/plugins/nova/servers_notification_handler.py15:30
TravTline 9715:30
TravTin glance images, we moved from doc as upsert15:30
TravTbut i think i found a different error that perhaps one of you could triage15:31
TravThttps://bugs.launchpad.net/searchlight/+bug/148678115:31
openstackLaunchpad bug 1486781 in OpenStack Search (Searchlight) "Glance images not searchable after update notification" [Undecided,New]15:31
openstackgerritSteve McLellan proposed openstack/searchlight: Add 'sort' parameter  https://review.openstack.org/20626815:31
TravTsigmavirus24: ^15:31
sjmc7TravT - the nova patch implements RBAC in the same way as that patch already, so that one doesn't need a change15:32
TravTsjmc7: i didn't get all the way through nova testing.15:33
TravTbut since you have to update it again, i'm going to hold off.15:33
TravTi did all the testing on previous patches and it worked.15:33
TravTand you addressed most my comments.  several of them we agreed should be addressed as future patch sets.15:34
TravTbut i will re-test once others look through it and you address my comments / their comments.15:34
sjmc7ok, taking a look at the nova one now15:34
TravTi'll revalidate the sort parameter now with you latest change.15:35
TravTs/you/your15:35
TravTsjmc7: as a process, would you mind putting a reply of done to comments you address in previous patches (e.g. the sigmarvirus24 comment).  It is helpful to see when going back through reviews.  I added some on your nova patch yesterday.15:37
sjmc7yeah, i try to. i will be more diligent15:38
sjmc7so TravT - we're not doing doc_as_upsert at the moment, pending that other bug?15:40
openstackgerritNikhil Komawar proposed openstack/searchlight: Allow indexing of single type or index  https://review.openstack.org/21075915:40
TravTsjmc7: i'm not sure what our best solution is.15:41
sjmc7ok. i'll keep the nova patch consistent with what's there15:41
nikhil_kTravT: sjmc7 lakshmiS : fixing merge on this too https://review.openstack.org/#/c/212563/ :P15:41
sjmc7i haven't had a chance to look at it15:41
lakshmiSas per document both update and index does replace the existing doc15:41
TravTnikhil_k, cool. Thanks!15:41
TravTlakshmiS: index wasn't seeming to make the doc searchable.15:42
TravTor I had a bad configuration.15:42
TravTi can retest as well, but if you could try it out.15:42
lakshmiSTravT: what i was trying to say is the new document using "update" and "index" results in exactly same new verion of doc. but anyway let me test it15:42
sjmc7no, that's not necessarily true15:43
sjmc7update will leave properties around even if they no longer exist15:43
sigmavirus24TravT: sjmc7 left another -115:43
TravTis there another complementary doc_as_upsert.15:43
TravTto remove properties?15:43
sjmc7index15:44
lakshmiSsjmc7: that's right but the process of deleting the old doc, and getting the content inserted in the new doc should be same15:44
TravTthat's what it is using15:44
TravTbut at least in my tests on glance, search wasn't working immediately15:44
TravTthere is a refresh option15:44
TravTbut that didn't seem to change things, and from docs isn't necessarily recommended.15:45
TravTthanks sigmavirus24:15:45
TravTmaybe you can give the nova patch a once over?15:45
sjmc7no, refresh must be used with care15:45
TravTyeah, perf impact15:45
TravTdavid-lyle: rosmaita: maybe you can look at this one.  Not too big.15:48
TravThttps://review.openstack.org/#/c/20022515:48
david-lyleTravT: looking15:54
sjmc7nikhil_k, you said you're rebase 212563?15:56
nikhil_ksjmc7: yes, doing that now15:57
sjmc7ok. i'll hold off the nova one since it's dependent15:57
sjmc7this is fun, it feels like those group projects in college you left until the night before it was due15:57
TravTsjmc7: funny enough, i think this style of collaboration is how i've done most things in openstack.15:59
TravTnot the procrastination part, but the everybody pitching in15:59
sjmc7well.. :)16:00
sjmc7partly the procrastination :)16:01
TravTreminds me, i did exchange email with thierry on project planning with liberty. have added that to thursday meeting agenda.16:02
TravTlakshmiS: i'm seeing the index searchability thing still on latest devstack.16:06
TravThave you been able to reproduce?16:06
lakshmiSi will have to switch to my linux boot after this meeting to test. going through docs, there seems to be an auto refresh of 1 sec by default16:07
sjmc7yeah, that sounds familiar16:08
TravTwell, it is much longer than 1 second. i could screenshare to show you what i'm doing and seeing.16:08
sjmc7i remember when writing tests in the past we put delays of 1-2 seconds16:08
sjmc7that sounds like a different issue16:08
sjmc7the only time i've noticed delays is under VERY high i/o or when doing artificial tests (PUT/GET immediately following one another)16:09
sjmc7incidentally, GET by id should always work16:09
sjmc7so you might want to test that16:09
lakshmiSok let me test it. will be right back16:09
*** lakshmiS has quit IRC16:09
TravTit'll return the result if you don't filter at all16:09
sjmc7ah, ok. then it's definitely something else :)16:10
TravTit is when you filter (by name for example) that it doesn't return16:10
openstackgerritMerged openstack/searchlight: Clear cached os-client auth on unauthorized  https://review.openstack.org/20022516:10
TravTbut then if I do a fresh searchlight-manage index sync16:10
sjmc7might be tokenized vs untokenized16:10
TravTsame query works16:10
sjmc7hmm. that's a bit concerning16:10
sjmc7i'll try and take a look later16:10
TravTso, let's eliminate Travis mistakes and have you look over my shoulder later.16:11
sjmc7i doubt it's a refresh thing16:11
sjmc7:)16:11
sjmc7ok16:11
TravTi'll try a few other ways as well.16:11
openstackgerritNikhil Komawar proposed openstack/searchlight: Fix RBAC filters  https://review.openstack.org/21256316:12
nikhil_kTravT: this need very minor changes. can we merge it and follow up with another one or can someone make the changes you know? https://review.openstack.org/#/c/197909/16:17
nikhil_k<tests are also passing?16:18
nikhil_k>16:18
TravThmm... let me see.16:18
TravTi haven't actually tried it out.16:18
TravTactually line 63 in this file16:19
TravThttps://review.openstack.org/#/c/197909/2/contrib/vagrant/Vagrantfile16:19
TravTmakes me think it wouldn't work16:19
TravTnikhil_k, maybe you could just co-author it and make the fixes?16:20
sjmc7sigmavirus24's got his approvin' stick out now!16:25
sjmc7i'll rebase and push up the nova patch once 212563 lands16:25
*** lakshmiS has joined #openstack-searchlight16:27
david-lyleso I've been looking at https://review.openstack.org/#/c/200225/4 which had me look at https://github.com/openstack/searchlight/blob/master/searchlight/elasticsearch/plugins/openstack_clients.py16:28
sjmc7before you continue, i did mention i wanted people to look at this a while ago, but continue16:29
david-lylemy question is: is the intent of https://github.com/openstack/searchlight/blob/master/searchlight/elasticsearch/plugins/openstack_clients.py for integrating with python-openstackclient16:29
david-lyle?16:29
sjmc7oh. no16:29
sjmc7it's to provide APIs to the plugins16:29
sjmc7the reason it's not IN each plugin16:29
nikhil_kTravT: hmm, lemme think about it. May be sometime today..16:30
sjmc7is that at the time, nova needed glance16:30
david-lyledef register_cli_opts() ?16:30
nikhil_kI will ping here if/once I start working on it16:30
sjmc7it now doesn't, so could be self contained if we never have interdependencies16:30
TravTnikhil_k, i just went back through the vagrant patch and saw a few other things.  left more comments.16:30
lakshmiSTravT:   new image and update image are available in search results immediately16:30
nikhil_kawesome, thanks TravT16:30
TravTlakshmiS: are you filtering?16:30
sjmc7oh, that... that was to register the credentials settings david-lyle16:30
TravTlakhsmiS: I see them in match all, but not in filters.16:31
lakshmiSno. just listing all results. any specific filter your tried?16:31
TravTlet me give example.16:31
TravTi created an image with a name of listener-image16:31
TravTnotification received16:32
david-lylesjmc7: confusing collection of functions to me, but ok, just trying to wrap my head around the intent16:32
TravTand is in unfiltered results16:32
TravTIf I do a query like this:16:32
TravT{16:32
TravT    "index": "glance",16:32
TravT    "type": "image",16:32
TravT    "query": {16:32
TravT      "query_string": {16:32
sjmc7it evolved a bit, i'm sure it could do with improvement16:32
TravT          "fuzziness": "AUTO",16:32
TravT          "query": "name:listener-image"16:32
TravT      }16:32
TravT    }16:32
TravT}16:32
TravTno results16:32
TravTif i re-run searchlight-manage index sync16:32
sjmc7david-lyle - there may well be a better way to do it16:32
lakshmiSok let me try that16:32
TravTand re-execute above, i get the result16:32
david-lylesjmc7: I'll keep walking through it, just for my own edification16:34
TravTnikhil_k: in my comments on vagrant patch, i put in what the glance, nova, neutron conf should be in devstack.16:34
TravTi think a separate patch could be to just add a note to the glance rst about the glance conf setting for devstack16:35
TravTe.g., just put a little devstack section and say:16:35
TravTadd the following to devstack local.conf for notifcations to work:16:36
TravT[[post-config|$GLANCE_API_CONF]]16:36
TravT[DEFAULT]16:36
TravTnotification_topics = notifications, searchlight_indexer16:36
TravTwithout that, people have to manually restart g-api after devstack is done.16:36
sjmc7sure, david-lyle. call out anything looks fishy16:36
sjmc7i agree with that TravT, it's a pain16:37
lakshmiSTravT:  that query works for me16:38
TravTokay... that's very odd.16:38
TravTis your elasticsearch installed from the devstack package?16:38
lakshmiSnope16:38
TravTmaybe bug or config diff?16:39
lakshmiStime to compare the conf files16:39
TravTwould you be able to do a hangout with me, so I could show you.  Maybe I'm making a mistake.16:40
lakshmiSsure16:40
TravThttps://plus.google.com/hangouts/_/gwnwvxgab57pompc63selrfvzua16:40
TravTbut just a minute.16:41
TravTi have a meeting in a few minutes that I want to try to get out of16:41
lakshmiSok16:41
ekarlsohey guys16:53
ekarlsoyou about ?16:53
nikhil_kTravT: oops, comp restart... thanks for the details, makes sense!16:53
ekarlsohttps://review.openstack.org/#/c/199099/ < the comment there regarding records16:54
ekarlsoanyone wanna help me work that out ?16:54
openstackgerritSteve McLellan proposed openstack/searchlight: Add nova instances plugin  https://review.openstack.org/19885216:56
sjmc7ekarlso sure. the comment from graham?16:56
mugsieekarlso: sorry, meant to ping you on that16:57
sjmc7speak of the devil16:57
mugsieo/16:58
ekarlsothe records thing..16:58
ekarlsomugsie: ..16:58
mugsiejust definie it as an object16:58
mugsie(i wouold like to have searchlight take the structiured data at some point)16:59
ekarlsomugsie: meaning ?16:59
sjmc7https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-nested-type.html17:00
sjmc7you can leave properties blank if you don't know what they are17:00
ekarlsokk17:01
mugsieekarlso: you also need to rename that file17:02
sjmc7if you know you're going to have arrays of objects, i'd strongly consider nested, although because they become separate documents there are some querying weirdnesses17:02
sjmc7if you ever want to be able to do record.a = 1 AND record.b = 2, nested is the only way to do it17:03
ekarlsowhat u think mugsie ?17:03
mugsienested17:03
TravTlakshmiS: here's the local.conf we're talking about in hangout17:06
TravThttp://paste.openstack.org/show/427232/17:06
lakshmiSthx17:07
TravTprobably you'll want to comment out a couple things like designate17:07
TravTand ceilometer17:07
TravTjust for speed17:07
TravTup to you17:07
TravTfor designate, you have to install the plugin first17:07
TravTso, probably comment that out17:07
openstackgerritMerged openstack/searchlight: Fix RBAC filters  https://review.openstack.org/21256317:19
mugsiesjmc7: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-array-type.html17:22
mugsiei thought you said there was no array typer17:22
*** lakshmiS has quit IRC17:23
sjmc7there isn't :)17:25
sjmc7it's saying e-s knows how to map lists you send it17:26
sjmc7you can't define explicit arrays/lists17:26
sjmc7the simplest thing is to not worry about this until you need to. you could decide not to explicitly map records at all and let E-S handle it17:27
sjmc7the only slight concern i have with what we spoke about is that if you have the possibility of two records with the same fields but different types, you may get problems17:28
sjmc7for instance, 'record': {'a': 1}    and later   'record': {'a': 'something else'}17:28
*** _sigmavirus24 has joined #openstack-searchlight18:17
*** sigmavirus24 has quit IRC18:17
*** _sigmavirus24 is now known as sigmavirus2418:20
*** sigmavirus24 has joined #openstack-searchlight18:20
openstackgerritSteve McLellan proposed openstack/searchlight: More efficient result filtering  https://review.openstack.org/20768218:24
*** GB21 has quit IRC19:14
*** rosmaita is now known as brianr-afk19:34
*** TravT has quit IRC19:58
*** TravT has joined #openstack-searchlight20:01
openstackgerritSteve McLellan proposed openstack/searchlight: Add 'sort' parameter  https://review.openstack.org/20626820:30
*** brianr-afk is now known as rosmaita21:42
ekarlsoyou guys wanna test out a patch ?22:22
*** bitblt has joined #openstack-searchlight23:28
*** bitblt has quit IRC23:28

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!