17:59:50 <stevemar> #startmeeting keystone
17:59:51 <openstack> Meeting started Tue Feb  7 17:59:50 2017 UTC and is due to finish in 60 minutes.  The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:59:54 <openstack> The meeting name has been set to 'keystone'
17:59:57 <gagehugo> o/
18:00:00 <browne> o/
18:00:18 <lamt> o/
18:00:22 <samueldmq> howdy
18:00:23 <samueldmq> o/
18:00:24 <morgan> hm.
18:00:30 <stevemar> agenda is at https://etherpad.openstack.org/p/keystone-weekly-meeting
18:00:31 <rderose> o/
18:00:32 <stevemar> #link https://etherpad.openstack.org/p/keystone-weekly-meeting
18:00:39 * morgan goes back to sleep. :P
18:00:57 * stevemar gives morgan a pillow
18:01:08 <stevemar> may as well get comfy
18:01:31 <stevemar> not much was added to the agenda !
18:01:39 <stevemar> hopefully we can make this short :)
18:01:59 <rodrigods> ++
18:02:44 <stevemar> okay, lets get going
18:02:46 <stevemar> #topic announcements
18:02:55 <stevemar> we're in RC week
18:03:26 <stevemar> next week is the final RC candidates and the week after we're out the door!
18:03:43 <stevemar> hope we don't get any upgrade bugs :(
18:04:00 <stevemar> also, PTG in 2 weeks, yay!
18:04:13 <ayoung> Oyez
18:04:25 <stevemar> in case anyone missed it, stable/ocata branch was created, we can work on pike goals now
18:05:16 <stevemar> and lastly, you'll all have a new PTL in ~6 hours
18:05:51 <stevemar> #topic functional/integration tests prior merging features
18:05:53 <stevemar> rodrigods: ^
18:05:57 <stevemar> you're up boss
18:06:01 <rodrigods> cool
18:06:12 <samueldmq> stevemar: lbragstad \o/
18:06:31 <rodrigods> this is a request and hopefully you will all agree :)
18:06:32 <crinkle> o/
18:06:56 <rodrigods> has been a while that i'm implementing integration tests for keystone and it is not uncommon to find bugs in features that are there for some time
18:07:14 <rodrigods> you can see examples in the agenda
18:07:37 <ayoung> Still need a mechanism for LDAP and Federation.
18:07:48 <rodrigods> in Barcelona, me and stevemar discussed about requiring a functional/integration test submitted prior approving new features
18:07:53 <morgan> I disagree, mostly to be contrary... and only in the meeting...and only when I am pre-coffee. and only when I am not being serious. :P
18:07:56 <rodrigods> ayoung, right... federation we already have
18:08:35 <ayoung> Ah, good...SAML via Shib?
18:08:40 <rodrigods> ayoung, yes
18:08:45 <morgan> rodrigods: a lot of cases, functional was not possible before. FTR. this is not unreasonable now.
18:08:57 <rodrigods> morgan, it is totally doable as of today
18:09:05 <morgan> exactly.
18:09:24 <stevemar> we made functional tests required in OSC
18:09:27 <rodrigods> what the team think about this?
18:09:34 <rodrigods> i mean... this is not very hard
18:09:40 <morgan> eh. depends on the feature
18:09:47 <rodrigods> and can help *a lot* in the quality of stuff we delivery
18:09:57 <rodrigods> deliver*
18:10:08 <ayoung> So...say the RBAC thing...something that really needs a remote service to test...are we going to do an echo server thing for functional tests, to do a complete round trip with middelware too?
18:10:12 <morgan> but honestly, most of our recent "features" aren't features like federation/ldap
18:10:53 <rodrigods> morgan, ++
18:11:07 <morgan> so, I am hesitant to make it blanket. PCI DSS functional tests are really already covered... but not in the integration manner you are speaking, nor would they be
18:11:11 <morgan> for example.
18:11:11 <rodrigods> i can always help as well
18:11:31 <rodrigods> morgan, it is, for some settings
18:11:34 <rodrigods> i've implemented myself
18:11:41 <rodrigods> (PCI-DSS)
18:12:17 <rodrigods> morgan, https://review.openstack.org/#/c/378624/
18:12:19 <morgan> not in the way you're describing. most of the code is Keystone specific and unit test covered
18:12:30 <morgan> so, let's step back
18:12:51 <rodrigods> functional/integration tests are not supposed to be exhaustive
18:13:01 <rodrigods> exhaustion is done in unit testing
18:13:24 <morgan> let's do a clear delineation of functional vs unit and outline further where we are breaking our tests up before we require a new type of test for features
18:13:36 <ayoung> rodrigods, exhaustion is done trying to get a patch through review
18:13:57 <rodrigods> ayoung, not always, but usually
18:14:01 <morgan> right. now we still have a total mishmash and mismatched and impossible to follow test suite that is not clearly defined what the test types should be
18:14:02 <dstanek> i'm trying to pay attention here as much as i can (in another meeting), but if we require them what is the criteria to say they are done
18:14:19 <dstanek> if they are not exhaustive...is it that at least one is included?
18:14:20 <rodrigods> morgan, functional is tempest-like, unit is everything else that lands inside our /tests folder
18:14:40 <rodrigods> dstanek, something that shows in a high level the feature
18:14:44 <rodrigods> an API call
18:14:45 <morgan> right. explain to me how to pick what test to Implement where.
18:14:57 <morgan> we have tests in unit that should be functional
18:15:03 <rodrigods> unit is everything, all edge cases (including negative tests)
18:15:06 <dstanek> rodrigods: so only happy path is required? and for every API call?
18:15:06 <morgan> we have never cleaned up our tests.
18:15:15 <rodrigods> dstanek, basically
18:15:40 <dstanek> if we do mandate this we have to have clear guidelines...otherwise reviewers won't know what to do or won't be consistent
18:15:43 <morgan> so.. I am -1 on this. we havent done basic work to fix our current tests to.conform or identify what should be moved over/addressed.
18:15:57 <rodrigods> the guidelines are basic what tempest requires
18:16:04 <morgan> we are so inconsistent we can't mandate a. we test type right now
18:16:07 <stevemar> on the topic of "fixing our current tests"
18:16:15 <stevemar> this was proposed to the TC as a goal: https://review.openstack.org/#/c/369749/
18:16:18 <morgan> reviews will be inconsistent.
18:16:30 <dstanek> rodrigods: by guidelines i don't mean how to write tests, but something that tells us what tests to write
18:16:32 <stevemar> we would need to create a keystone-tempest repo
18:16:33 <rodrigods> example of tempest-like tests: https://review.openstack.org/#/c/425927/
18:16:41 <rodrigods> https://review.openstack.org/#/c/426249/
18:17:08 <rodrigods> dstanek, this is what i meant... tempest doesn't accept negative tests anymore, for example
18:17:19 <rodrigods> so a happy API path should be enough for most features
18:17:21 <morgan> I'm still -1 on this for the above stated reasons.
18:17:37 <rodrigods> morgan, it is not necessary to move over old tests
18:17:49 <rodrigods> just to require a simple functional tests for new stuff
18:17:53 <dstanek> i don't think we should enforce this yet. we should encourage it and see how that goes
18:18:07 <rodrigods> this has lots of consequences
18:18:09 <stevemar> dstanek: that might be the right call
18:18:11 <rodrigods> we frequently break APIs
18:18:23 <rodrigods> as an example
18:18:53 <morgan> mandating new test types when we don't even know what tests we have should be moved over, and there is no clear way to say move these tests (identify the ones, don't do it, backlog it) AND not having g a clearly written case, is not going f to help
18:18:56 <rodrigods> and the fact that find bugs is not uncommon, is a red flag for me
18:19:06 <dstanek> maybe we can push back a little on reviews and see if we can get them to include the tests
18:19:08 <morgan> "like this test *point*", -2
18:19:26 <rodrigods> morgan, i guess i can write a spec?
18:19:49 <stevemar> rodrigods: bugs are always going to happen
18:19:59 <stevemar> rodrigods: no need for a spec IMO
18:20:04 <morgan> clear descriptor, identity tests that are incorrect in our suit, specification, and documentation on the right way, +1, and more.convo on making it mandatory
18:20:19 <morgan> a/identity/identify
18:20:20 <stevemar> rodrigods: i think the issue people have is that they don't want to slow down an already slow process
18:20:24 <rodrigods> stevemar, although bugs are always going to happen, they can get fix prior the feature is merged
18:20:33 <rodrigods> instead of finding issues later, and having to backport stuff
18:20:57 <stevemar> i think we should encourage new APIs to include functional tests
18:21:04 <stevemar> not make it required tho
18:21:05 <rderose> stevemar: ++
18:21:10 <stevemar> i don't know how to add new tests
18:21:15 <stevemar> i assume i'm not alone?
18:21:25 <rderose> you're not
18:21:32 <stevemar> haven't looked into it much at all
18:21:41 <morgan> so, what I am asking for is a clear description on what is wanted, documentation on what we are expecting (encouraged), identifications of tests to move over (backlog work)
18:21:43 <rodrigods> so we just found a problem that can be solved :)
18:21:52 <morgan> then we can discuss making it mandatory.
18:21:56 <rodrigods> lbragstad already told me about our functional tests
18:22:02 <rodrigods> there is no doc around them
18:22:03 <samueldmq> morgan: ++
18:22:14 <morgan> but for now, like I said -1 or -2 based on this convo.
18:22:41 <stevemar> start off with a doc that says how to add a test?
18:22:45 <morgan> filling in the requests I just made before we make it mandatory +2
18:22:49 <morgan> :)
18:22:53 <stevemar> http://docs.openstack.org/developer/keystone/devref/development_best_practices.html#testing-keystone
18:23:11 <rodrigods> stevemar, cool
18:23:22 <rodrigods> will add a section about our tempest plugin
18:23:25 <rodrigods> devstack plugin
18:23:35 <topol> o/
18:23:42 <samueldmq> looks like everybody agrees it's good to have those tests, it's just about the way we get to/enforce that
18:23:49 <stevemar> samueldmq: yep
18:23:54 <morgan> I would expect the mandatory part to be for the Q cycle, but highly encouraged, documentation, and identified tests to convert for pike
18:23:57 <morgan> doc first.
18:24:00 <rodrigods> and how to add new tests to tempest
18:24:01 <rodrigods> tempest repo ^
18:24:07 <dstanek> samueldmq: ++
18:24:49 <rodrigods> ok, so i think i'm done here
18:25:02 <rodrigods> with the topic, i mean :)
18:25:08 * morgan is also done and wants coffee. :p
18:25:12 <morgan> ^_^
18:25:32 <morgan> ∆caffiene
18:25:48 <stevemar> rodrigods: i look forward to adding new tests myself in P :)
18:25:52 * topol wondering if morgan is using his french press
18:25:54 <stevemar> #topic open discussion
18:26:03 <rodrigods> stevemar, ++
18:26:07 <morgan> topol: not yet, soon.
18:26:11 <stevemar> anyone have anything they want to talk about?
18:26:22 <rodrigods> who submitted talks to the next summit?
18:26:28 <rodrigods> submitted one about federation, again
18:26:52 <stevemar> apparently the deadline was extended :D
18:26:57 <morgan> Someone was doing pci-dss. make sure to talk with rderose .
18:27:06 * lbragstad sets https://etherpad.openstack.org/p/keystone-pike-ptg on the desk
18:27:10 <topol> rodrigods I have a talk and a panel submitted on Interop and a talk on OpenStack Kube integration
18:27:11 <morgan> if not collaborations already.
18:27:33 <rodrigods> topol, cool! mine is a panel too (not a talk)
18:27:34 <stevemar> i haven't submitted anything yet, i should probably do something...
18:27:44 <rderose> morgan: ++
18:27:50 <rodrigods> stevemar, you are usually invited :)
18:27:57 <topol> stevemar I think you missed the deadline
18:27:57 <lbragstad> oh! how'd https://etherpad.openstack.org/p/keystone-pike-ptg get here?! just a reminder to make sure folks fill it out if they have feedback ;)
18:27:58 <dstanek> what's the new deadline?
18:28:10 <morgan> topol: doesn't like me, he never wants to collaborate on a talk with me :P (I would prob say no.. but...I should be asked) :P (j/k) ^_^
18:28:16 <rodrigods> lol lbragstad
18:28:17 <morgan> the 8th?
18:28:17 <stevemar> topol: they extended it
18:28:26 <samueldmq> morgan: it's myself and rderose doing pci-dss
18:28:26 <topol> stevemar til when???
18:28:34 <morgan> yep, the 8th
18:28:35 <dstanek> neat, maybe i'll come up with something then
18:28:47 <stevemar> topol: https://twitter.com/OpenStack/status/829018983009357824
18:28:54 * topol groan more internal abstracts to review #wantmylifeback :-0
18:28:57 <stevemar> Feb 8, 11:59pm PT
18:29:03 <gagehugo> I'll have to see the pci-dss talk then if I go
18:29:09 <stevemar> topol: can i skip the internal review?
18:29:14 <topol> yes
18:29:20 <stevemar> \o/
18:29:21 <topol> just submit something please
18:29:26 <rodrigods> let's submit a talk: "requiring integration tests for new features" :P
18:29:28 <stevemar> just trying to save you from doing work
18:29:44 <topol> yay
18:29:50 <lbragstad> now that we've been performance testing data - we should do a performance talk
18:30:04 <lbragstad> I have the data and would be willing to tag team it with someone
18:30:30 <dstanek> lbragstad: to report on how we've been doing?
18:31:06 * topol always fun doing a talk with Morgan... Sad our paths have diverged to diff topics of late
18:31:24 <topol> drinking with him in a nice consolation prize. hope that happens
18:31:55 <lbragstad> dstanek well - we have a bunch of data
18:33:18 <lbragstad> could just start asking questions like "what did it help us accomplish?" "was it effective?" "has our performance improved or degraded?" "how can we expand on what we've built?" etc...
18:33:19 <dstanek> lbragstad: i may be interested...let's talk offline?
18:33:28 <lbragstad> dstanek ++
18:33:57 <lbragstad> i would assume we'd be able to come up with enough things to make an interesting talk
18:35:09 <rodrigods> dstanek, lbragstad me too
18:35:16 <stevemar> well if no one else has anything to discuss we can finish this off early
18:35:26 <rodrigods> not to be a presenter, but to help with it in general
18:35:40 <lbragstad> rodrigods ++
18:35:46 * stevemar will celebrate chairing his last meeting as ptl with a big cup of coffee
18:35:51 <dstanek> rodrigods: ++
18:36:00 <rodrigods> stevemar, s/coffee/beer
18:36:00 <lbragstad> stevemar you're ending your last meeting as PTL early?!
18:36:04 <dstanek> stevemar: seems like we can call it a day
18:36:07 <topol> stevemar CONGRATS. You earned that coffee!!
18:36:15 <stevemar> lbragstad: i don't see a better reason to end it early :D
18:36:15 <lbragstad> topol ++
18:36:16 <rodrigods> thanks for everything stevemar
18:36:30 <lbragstad> stevemar thanks for all the hard work, sir
18:36:48 <stevemar> looking forward to a change of pace :)
18:37:04 * topol I'll save the I remember when Steve first started on OpenStack stories for the PTG
18:37:07 <lbragstad> stevemar are you excited to write code again?
18:37:16 <lbragstad> topol ++
18:37:30 <lbragstad> topol I remember the first call I had with you two on OpenStack stuff ;)
18:37:43 <stevemar> lbragstad: haha, that's right, you taught us!
18:37:44 <topol> lbragstad, good times
18:38:01 <stevemar> lbragstad: my code will be terrible, it'll receive many -1's
18:38:27 <lbragstad> stevemar topol ++
18:38:44 <stevemar> alright, thanks folks!
18:38:47 <stevemar> #endmeeting