16:01:04 #startmeeting defcore 16:01:05 Meeting started Wed Apr 6 16:01:04 2016 UTC and is due to finish in 60 minutes. The chair is markvoelker. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:09 The meeting name has been set to 'defcore' 16:01:13 o/ 16:01:16 #chair hogepodge 16:01:17 o/ 16:01:18 Current chairs: hogepodge markvoelker 16:01:18 o/ 16:01:22 #chair eglute 16:01:22 Current chairs: eglute hogepodge markvoelker 16:01:25 o/ 16:01:30 o/ 16:01:34 o/ (in another meeting, but here!) 16:01:45 #link https://etherpad.openstack.org/p/DefCoreRing.18 Today's Agenda 16:02:49 Please take a quick look through the agenda and make sure we're not missing anything folks 16:03:05 #topic Straw Man Proposal for DefCore Owning DefCore Tests in Tempest 16:03:18 #link https://review.openstack.org/#/c/301879/ Strawman proposal 16:03:50 hogepodge posted this yesterday so I'll let him speak to it. =) 16:04:21 As a general note, you are all (as usual) encouraged to please provide your input in Gerrit. 16:04:41 The blueprint has a couple of intents. The first is to solidify the relationship with the QA team by marking defcore tests, approved by the board, as important. 16:04:42 hogepodge: around? 16:04:44 doh 16:04:47 yes you are =) 16:05:25 We've had friction in the past about test names changing, functionality changing, and playing a bit of catch up and writing tooling to work around the changes. 16:06:37 I'm not sure it's a part of the proposal that would pass, especially using a defcore named tags. This is in part because tempest doesn't necessarily want outside influence on the project like this, but they're not 100% opposed to it either. Particularly with regards to tags that indicate a property 16:06:53 do defcore-required may be bad? interop may be good? 16:07:06 It's starting a dialogue there, which I feel is important. 16:07:11 ++ 16:07:35 Secondly it's encoding some of the requirements we talked about at the previous meeting for a spec that defines "interop" test 16:07:35 hogepodge:: ++, dialog is good 16:08:10 I tried to write down the minimal things we could reach consensus on, but is a work in progress. I would expect that an official spec would be part of the defcore repository. 16:08:46 hogepodge i do not recall us having consensus on tagging 16:09:06 Again, it's trying to encode a relationship, saying that we take responsibility for maintaining, along with qa, what an interop test looks like. 16:09:10 hogepodge: Does it really encode though? It's a decorator, but it doesn't enforce anything directly 16:09:36 dwalleck: it puts meaning in code, backed by a spec. 16:10:06 But it does start a dialog. It does add a burden to Tempest reviewers to understand all the things that interop means (single user, etch) 16:10:14 finally, there is talk of forking tempest. I am strongly opposed to that, and personally feel it is harmful to both our aims as a standard body and to the community at large 16:10:24 we do not have the specs yet. this is jumping ahead and making assumptions what the spec would look like 16:10:40 * docaedo arrives a little late to lurk :) 16:11:19 eglute: we do have an informal spec. For example, tests are not admin. That is a requirement. They must be api, that is also a requirement. 16:12:19 hogepodge: But that's a spec for what attributes a Tempest test should have to be an interop test, which is different from defining an actual spec for capabilities 16:12:35 hogepodge we agreed to work on formal specs first before proceeding further. otherwise we will run into issues again where the test dictates what the spec should be, and not the other way around 16:13:01 The second being the one I thought we had been talking about at the midcycle, but if I'm mistaken, that's okay :-) 16:13:54 the one we talked about at the midcycle? 16:14:08 i agree with dwalleck here, we need to start with defcore specs, not tempest specs 16:15:51 not sure how much time I can commit to this, but I can have a go at trying to start a spec 16:16:06 I see the tagging as useful regardless, though 16:16:10 like an orthogonal thing 16:16:22 gema that would be good. perhaps you could work with dwalleck since he already started on some analysis 16:16:24 gema: that would be great 16:16:42 eglute: yep, that was going to be my first task, catch up with dwalleck :D 16:16:52 ++ 16:17:01 #action gema and dwalleck to start working on a defcore spec 16:17:08 dwalleck: o/ 16:17:09 gema: Absolutely, I'd be glad to contribute 16:17:17 ! 16:17:20 ok, we'll start this week and see how it goes :D 16:17:24 eglute: I would very much like to be involved with what a spec looks like too. As the person who has worked most closely with qa, I'd like us to maintain a good relationship on that. 16:17:43 hogepodge i dont think anyone will be excluded! 16:17:44 hogepodge: how about I start a conversation on our mailing tomorrow 16:17:47 and we make it collaborative 16:18:12 then I put the result together on some sort of commit 16:18:18 But which spec are we talking about? The spec for what defines a capability, or the specs of what makes a good interop test? 16:18:30 I thought the spec for defcore 16:18:36 dwalleck: We have Board-approved definitions of Capabilities 16:18:45 test spec, that is 16:18:48 dwalleck: Spec for what makes a good interop test 16:19:01 VanL, right 16:19:22 We have a couple principles that everyone agreed on, IIRC: 16:19:24 Gotcha, wanted to make sure we were using the same terminology 16:19:27 1. No admin 16:19:46 2. (From Chris:) Two paid accounts not needed. 16:19:52 3. Atomic 16:20:18 I think those were the only ones that we got to before we moved on 16:20:25 I would also suggest: 16:20:36 4. available for at least 3 releases 16:20:42 4. The test matches the public documentation 16:20:46 VanL: I object to you using language like "that everyone agreed on". It over simplifies and may not accurately represent my views. 16:21:06 hogepodge: I am going to go back to the midcycle notes tomorrow and summarize our discussion on this 16:21:13 then frame the conversation 16:21:16 VanL for example, 3. Atomic, may differ from dwalleck's 16:21:23 Rockyg: Yes, but that is a capabilities requirement, not a interop spec. 16:21:28 One suggestion: let's be careful to not overlap with the Board-approved Criteria we already have. 16:21:36 #link http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst Criteria 16:21:50 or 2, when I say two tenant isolated accounts I mean two tenant isolated accounts, not paid accounts. 16:21:59 We don't want to essentially have two sources of truth about those. 16:22:14 +1 markvoelker 16:22:15 hogepodge: ++ ... Atomic is the reason causing the issues we have with today's test set 16:22:19 hogepodge: that's a fair example. Then we need to drill down into those bits and talk them through and come to a community understanding 16:22:43 hogepodge: Please don't personalize this. I am trying to honestly represent and summarize what I understood, and if the summary isn't accurate, then by all means chime in 16:22:46 dwalleck: yes. I agree 16:23:05 VanL: no worries, fortunately for everyone my memory is bad, so I will go back and read the notes 16:23:08 x) 16:23:16 I was mostly trying to quote from /summarize the notes 16:24:08 My opinions are my own, but as someone who works in the community and has invested time, I have to share my thoughts, which doesn't mean anyone has to agree 16:24:22 dwalleck: ++ 16:24:35 gema: hogepodge: dwalleck: do you guys have what you need to proceed on drafting something up? 16:24:45 I spent a great deal of time working on Tempest and then later on CloudCafe, so I have some strong opinions about tests :) 16:24:48 markvoelker: I think so 16:24:51 Yes 16:25:00 dwalleck: me too 16:25:03 we'll have fun :D 16:25:05 FYI: https://etherpad.openstack.org/p/DefCoreSpring2016MidCycle 16:25:24 Gemma: yup! 16:25:34 Also: https://etherpad.openstack.org/p/DefCoreSpecProposals 16:25:51 Great. I'll look forward to reviewing it then. =) We're 25 minutes in and have several other agenda items--ready to move on? 16:26:17 markvoelker +1 16:26:24 ++ 16:26:27 Yup 16:26:32 #topic Capabilities with no tests (or only admin tests) 16:26:49 #link https://review.openstack.org/#/c/299491/ 16:26:58 #link https://review.openstack.org/#/c/300608/ 16:27:13 I'd like to start capturing all of those and make a list 16:27:16 of tests to be created 16:27:36 I have a couple on keystone as well 16:27:52 We've identified some advisory tests which turn out to require admin creds. We can't use those in Guidelines, so they'll have to go. 16:28:08 The biggest question is what to do about it, because the underlying features genereally don't require admin 16:28:13 So, two things: 16:28:28 1.) I'm going to spend some time looking for other tests of the same capabilities that don't require admin 16:28:56 2.) I think we should have a chat with the project teams/PTL's/QA teams about potentially changing those tests to not need admin 16:29:52 Those are sort of the natural outcomes...the bigger question is what to with the tests in the meantime since it's uncertain how quickly we could get the tests modified (if that's even realistic...there may be good reasons for using admin) 16:30:04 https://review.openstack.org/#/c/300608/ is an easy one to review and merge... because the capability affected still have some tests in it after the admin tests are flagged or removed.. 16:30:19 Obviously we've got a few months before the Guideline comes up for approval here, so there's potential for us to get things done before then. 16:31:08 Failing that, we could consider leaving the relevant Capabilities in advisory state for another cycle so we don't have to restart the clock by dropping them and then having them go through another advisory cycle 16:31:23 thank you catherineD for calling those tests out. i agree with markvoelker that we need better tests, but also agree that no idea how long this will take 16:31:55 How much real-world vetting have the advisory capabilities had? Finding admin-requiring tests makes me nervous that they have not been properly vetted by enough people out in the community 16:32:01 i think leaving them advisory would be ok, provided we start working on getting better tests somehow 16:32:11 eglute: from a cursory look at a few of them, I think it may be realistic to improve some of these before the Guideline goes up the board, for what it's worth. 16:32:25 VanL: I have verified all the tests in the advisory section 16:32:36 and thus found the one with admin requires 16:32:42 VanL: the scoring for them is in the scoring sheet. They all stacked up to criteria like widely deployed. 16:33:03 markvoelker i would be really hesitant to make them required if we are going to change the tests 16:33:11 markvoelker: qa has a history of helping sort the issues out, maybe not in time for a guideline though. They could stay advisory 16:33:23 Thera are also some test records in RefStack that tests the entire API Tempest test that we can use to verify 16:33:40 eglute: why so? 16:33:55 markvoelker: Sure, but we run into the problem of the *capability* being widely deployed, but the tests to show a capability not being representative or otherwise having problems. 16:33:56 markvoelker: otherwise they would be immediately flagged, with the action to be to fix or remove. That seems an odd way to start, but it's definitely a workable solution 16:33:59 oh, you mean make the required and then change the tests? Yeah, definitely. 16:34:08 I'm saying we change the tests before we make them required. 16:34:32 Remember that the next Guideline won't get approved for several months yet. So there's a pretty good window of time here. 16:34:37 markvoelker: yes, I think that's better (or write new tests if admin is required) 16:34:40 markvoelker: we do have a patch today to move advisory to required 16:34:40 But if a test defines the capability, how can we add a test that we plan on immediately refactoring? 16:34:43 markvoelker people that are/could be testing agains this as advisory might be blindsided by new tests. in theory, new tests should test the same stuff, but in reality they might not be exactly the same 16:35:28 at the mimimum we should flagged those admin tests .... if not removed 16:35:35 It gives you a placeholder for intent, but I'm not sure what other value it provides 16:35:37 eglute: I see the concern. I think in most cases though, all we'd need to do is modify the existing tests to use ordinary credentials rather than admin ones. E.g. we're not fundamentally changing what's being tested. 16:35:38 i think keeping them as advisory is a good compromise 16:36:05 ahh, admin part. Missed that, never mind :) 16:36:06 eglute: keeping the capability but flagged or removed the admin tests ? 16:36:16 catherineD: keep in mind that even if the patch moving them to required lands, that doesn't actually mean anything until the Board votes on the Guideline in August 16:36:24 keeping the capability as advisory rahter than making it required 16:36:34 catherineD ^^ 16:36:38 catherineD: until then it's just a draft. But I'd be fine with dropping those tests from moving right now just for clarity. 16:37:16 markvoelker: agreed but if may make the required in next.json and the advisory in 2016.01 mis-match 16:37:42 markvoelker: agree for dropping ... I think it is cleaner 16:37:45 markvoelker: eglute: catherineD: it's pretty easy to tell if you're changing essential functionality. I saw this in the multi-tenant tests. It was clear multi-tenancy was not required on all but two tests. 16:38:01 and send a good message the defcore does not require admin 16:38:12 Ok, so I think what I'm hearing here is: 16:39:03 #action markvoelker submit new patchset for https://review.openstack.org/#/c/294287/ to leave the admin-requiring tests as advisory 16:39:12 +1 16:39:24 #action markvoelker (and anyone else interested) look for alternative tests for the same capabilities 16:39:30 +1 16:39:58 #action markvoelker (and anyone else interested) investigate the feasibility of modifying the tests to not need admin 16:40:02 markvoelker: do we leave the admin tests or remove the test but leave the capability? 16:40:40 we really should not have any admin tests now that we identify them 16:40:41 catherineD: I'm planning to leave them for the moment since it seems likely we're going to try to get some of them changed to not need admin. If that doesn't happen, we have several more months to drop them 16:41:18 that sounds like a good plan to me 16:41:44 But if folks feel strongly one way or the other, I'm not opposed to dropping and re-adding them. Just seems to make a bit more work for later. 16:42:04 * markvoelker looks at clock 16:42:19 i am ok with dropping as well. the only thing i would be opposed to is making these tests required for 2016.08 16:42:28 Ok, so I'll go that route and if anyone has strong opinions about dropping them now or not then voice them in gerrit and we'll iterate there. =) 16:42:39 Move on? 16:42:41 thank you markvoelker! 16:42:42 yes 16:42:58 #topic RefStack update to use Tempest tag 10.0.0 as default 16:43:06 catherineD: the floor is yours 16:43:13 thx 16:43:46 so we will update RefStack to tempest tag 10.0.0 which is the tag for Mitaka 16:44:08 currently RefStack still running with a Sept 11, 2015 tempest vestionb 16:44:17 it is kind of old ... 16:44:40 catherineD do you foresee any issues with the new tag? 16:45:42 eglute: so we have taken care of the test names change by https://review.openstack.org/#/c/297993/ ... 16:46:06 ok 16:46:19 there are also some change in tempest conf that the user have to pay attention to 16:46:54 I know that a lot of the user is using master copy of tempest .... those users will not be affected 16:47:04 tempest is moving to a model where credentials and resources are defined outside of tempest.conf. It's a good change, but one that may have short term impact 16:47:05 cathrineD: this looks like a pretty routine update to me. So, seems fine. I'll see if I can run that version through my lab, but I don't foresee any real issues. 16:47:45 hogepodge: Yep, saw that too. Perhaps something to note in our instructions. 16:47:46 question is for DefCore to identify new tests to add ... should defcore also use the latest test list? 16:48:17 catherineD: do you mean on aliasing? 16:49:03 alias is for the existing defcore test ... but if defcore going to add new tests ... defcore should use update to date test list 16:49:35 catherineD: hogepodge: that's covered here: https://review.openstack.org/#/c/294287/ 16:49:37 oops 16:49:44 I mean http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/schema/1.4.rst#n85 16:50:16 So basically for a new Guideline (next.json) we use the current name of the test as it appears in Tempest. 16:50:30 If there's an older name that's been changed, we move that to the alias field in next.json 16:50:55 for the new tests .. if defcore use new test lists for its tests we will avode alias 16:50:57 I think the same principle applies to new tests we're adding too 16:51:05 avode --> avoid 16:51:38 new tests here means the test name that has never appeared in defcore json 16:51:57 catherineD: Well, usually, probably so. But possibly not always. Because if someone's using an older version of Tempest it might have an older test name. 16:52:17 So conceivably we may need to add an alias even though we haven't required that test in the past. 16:52:32 I suspect that'll be a pretty rare case in practice though 16:52:33 if someone is testing a new OpenStack release they should not use old json 16:53:20 Well, we don't bound what version of tempest is used. So it's not so much using old json as it is using an older Tempest (which might be necessary due to bugs, etc). 16:53:32 for example for Mitaka the require tests would jump from 100+ to 200+ because of the advisory tests 16:53:39 because defcore covers several releases, they might be using older json 16:53:52 if certify for Mitaka they must use 2016.08 16:54:27 agree .. for older release the tests will be covered by aliases 16:54:54 but the newly defcore added tests does not need to be burdened with aliases 16:55:01 true 16:55:15 cathrineD: correct, they need to use 2016.08. That Guideline might have a test called "ABC", which is what Tempest calls it today. But Tempest might have changed it to "ABC" from "ZYX" the week before we added it to the Guideline. 16:55:44 So if someone is using an older version of Tempest, they'll see "ABC" skipped because it's not found. They might therefore reasonably request an alias be added. 16:55:51 markvoelker: in that case it will cover by aliases becayse the test already exists in defcore 16:56:26 think about the tests that are not in any defcore json today 16:56:44 those are the tests that we should use from new test list 16:57:00 anyway back to refstack update default tempest 16:57:17 I guess it is a go ? 16:57:18 three minutes 16:57:27 catherineD: go from me 16:57:28 yes, catherineD lets go ahead with new version 16:57:35 want to make sure people look at gema's identity capabilities patch 16:57:41 yes please 16:57:47 no need to take time here, though :D 16:57:49 #topic keystone scoring 16:58:02 everyone, please review gema's patch 16:58:02 thank you ... yea after all that tempest ins 6 month old ...as old as the openstack release :-) 16:58:09 #link https://review.openstack.org/#/c/297847/ Keystone scoring patch (new rev) 16:58:24 thank you catherineD! 16:58:34 and thank you gema, I will review the patch today 16:58:40 eglute: thanks! 16:58:47 if any questions, lets take them to the defcore channel 16:59:13 With that we're about out of time. We skipped the advisory->required topic, but we actually already discussed it a bit earlier and I have an AI to submit a new patchset on it 16:59:21 Thanks everyone! 16:59:24 thanks! 16:59:25 #endmeeting