15:00:27 #startmeeting DefCore Flag.15 15:00:28 Meeting started Wed Sep 16 15:00:27 2015 UTC and is due to finish in 60 minutes. The chair is zehicle. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:32 #link https://etherpad.openstack.org/p/DefCoreFlag.15 15:00:33 The meeting name has been set to 'defcore_flag_15' 15:00:42 o/ 15:00:44 o/ 15:00:46 #chair eglute hogepodge markvoelker 15:00:47 Current chairs: eglute hogepodge markvoelker zehicle 15:00:50 o/ 15:00:54 roll call please 15:00:54 o/ 15:01:31 please review the agenda - we have a lot of patches, so I'd like to make sure we have time for discussion if needed 15:02:31 o/ 15:02:37 Hello from Colorado 15:02:59 hello hogepodge! 15:03:15 :) 15:03:18 zehicle i think agenda looks good, full! first topic? 15:03:21 o/ 15:03:31 o/ 15:03:35 #topic scoring weights 15:03:49 markvoelker, I have not done anything. you? 15:04:03 zehicle: I have a couple of things brewing. 15:04:24 I haven't put patches up yet b/c I've been spending time on scoring, but will put up a couple of RFC patches this week 15:04:34 based on discussions, I think that we'll need to figure out balance w/ future direction weight 15:04:50 or have a policy for consideration that we can treat objectively 15:04:58 Going through some of the existing scoring patches has been useful on that front, as it's given me the chance to play with weights and see if they match up with gut feel. 15:05:43 zehicle: Yeah, I'll include some ideas around future direction (forward looking) and widely deployed (backward looking). 15:06:28 #action markvoelker to propose some RFC's for changing weights this week 15:06:36 +1 15:06:39 +1 15:06:40 zehicle: in the interest of time, move on to next agenda item? 15:06:48 #meeting year.next.hson 15:07:11 next.json this makes sense to me. 15:07:15 markvoelker made a good point with that i think 15:07:29 i am ok with making that change! 15:07:49 eglute: please do +1 or +2 it in gerrit then. =) 15:07:52 I'm +2 on it (literally) 15:07:53 so it would be just "next.json" correct? 15:08:11 yes 15:08:22 that's my preference 15:08:42 ok, there are several patches that talk about year. which one should go first? 15:08:42 I have a question related to how we defining capability .. 15:09:30 eglute: they can both go in. 223243 is against 2015B and will have to get Board approval before we can implement it. 15:09:44 ok, thank you markvoelker 15:09:49 So in the meantime, 223234 sticks with current processes and is safe to land. 15:09:50 * zehicle thinks that hogepodge and I want to skip the require vendors at this point. no action 15:10:05 we can revisit it 15:10:05 catherine__: sure, what's the question? 15:10:10 I think we can make the name change for now 15:10:20 so far, we define capabilities based on the tests on tempest ...while working on the Heat scoring , a few members of the Heat core team t 15:10:22 and update 2015B to reflect it 15:10:34 it's not a controversial change to the process. we should be OK 15:10:41 just like schema changes 15:10:57 think that the tests covered are very limited ... so they would like first to define capabilities 15:11:25 so some of the capabilities may not have tests associaged with it ... how do we deal with this situation? 15:11:38 catherine__: without tests we really couldn't enforce that vendors provide those capabilities in the same way. They need to add tests. 15:11:38 o/ 15:11:38 markvoelker, I think it's ok to jump to next.json if we've got 100% consensus on it. 15:11:52 catherine__: ++ 15:12:22 #meeting using Designated Sections to enforce API (214756) 15:12:24 #link https://review.openstack.org/#/c/214756/ 15:12:49 the team thinks that we should list out the capabilities first and then pull in or create tests for them .. ofcourse we can score them low 15:12:55 uintil there are tests associated with it 15:13:22 morgan feels very strongly about that patch, that apis are fundamentally important and need to be discoverable, and forcing 403 should be explicit policy 15:13:33 I'm always a fan of starting with a spec 15:13:37 I tend to agree with him, and think that enforcing with tests is too easy to slip up on. 15:13:38 catherine__: Without tests I'm not even quite sure what we'd be scoring...e.g. I can't look at a test and see "oh, they're basically saying the XXX API must work". 15:13:50 but right now the score matrix do have column defining whether tests exist .. 15:13:50 * zehicle sees what markvoelker did w/ 2016.next as bridge until 2015B lands. OK 15:14:43 I agree with the intent. just want to make sure that we're using the right mechanism to enforce 15:14:48 whoa, too many topics at once folks. =) Could we finish with catherine__'s question for a moment before we go on to the keystone thing? 15:14:59 sorry, thought we'd jumped 15:15:15 what's catherine__ question? 15:15:38 capability needs a test. If they want the capability, write the test. 15:16:07 catherine__: I think tests are pretty ingrained in all our processes at this point. For example: https://github.com/openstack/defcore/blob/master/doc/source/process/CoreDefinition.rst 15:16:37 We have more developers interestd in writing needed tests. Maybe a way to specify tests that we think are important? 15:17:06 Like Swift the Heat team thinks they have a rich set of in-tree tests .. but not in Tempest 15:17:13 catherine__: So I think what they want to do is drawn up a spec for what tests they want to add to Tempest rather than try to score something in DefCore for which no tests exist. 15:17:23 But if the tests weren't designed with defining a spec in mind, how can we say for sure it really defines a capability? I like the idea of the Heat team adding some additional tests that they think clearly define their capabilities 15:17:31 catherine__: Ah, so there are in-tree tests? Ok, slightly different answer then... 15:17:48 +1 dwalleck 15:18:04 yes there are in-tree tests .. 15:18:07 That would put them in the same boat as Swift where we need those tests to be runnable by Tempest via plugin so RefStack can consume them 15:18:43 catherine__: make sense? 15:19:22 (for those who don't know what I'm talking about, see comments from midcycle in https://review.openstack.org/#/c/164865/ ) 15:19:27 that bring us to another topic ... do we requre all test entry from tempest our RefStack needs a plugin itself? 15:19:35 does anyone know what the status of the tempest plugin is? 15:19:46 mtreinish would like to see those tests moved over to tempest where feasible, but it's not a strict requirement now that we can use tempest plugins 15:20:09 eglute: for swift, in progress. 15:20:22 catherine__: My recollection from the midcycle is that we wouldn't necessarily rule out using something other than Tempest as an entry point... 15:20:33 plugins are going to be a big topic in tokyo, so it's something that's moving 15:20:35 catherine__: ...but there weren't really good reasons not do use Tempest 15:20:36 there is also talk about bringing in the EC2 test by Rsndy Bias team ... would tempest willing to bring those in? 15:20:57 catherine__: ....and it would make refstack a lot more complicated and put a burden on people trying to run tests in lots of different ways 15:21:06 catherine__: So tempest made the most sense. 15:21:09 markvoelker, I remember the opposite. I thought we'd decided to only consider Tempest runnable 15:21:40 * zehicle looks for reference in doc 15:21:59 zehicle: plugins to tempest are loaded when tempest is started, so produce the output in the same run if the plugin is available in the load path 15:22:43 I prefer the tempest path because it make RefStack easy to consume ... just want to confirm on our direction going forward .. 15:22:52 "#DECISION > even if we bringin in "non-Tempest" tests, they MUST still be gate tests" 15:22:55 catherine__: As for EC2, you'd have to ask the Tempest folks. Personally I guess I don't see why those wouldn't be runnable via plugin too, so could live in-tree for the ec2 api if temepst doesn't want them in tree. 15:23:29 zehicle: yes to that also. so swift tests are gate for swift project 15:23:35 so, plug-ins would work 15:23:36 The QA team perspctive is they are planning to be the repository for th tests DefCore use. 15:24:06 Ok, so sounds like we have an answer here... 15:24:21 catherine__: Anything else, or shall we move on to keystone? 15:24:35 that is it thx .. 15:24:54 But, I think it would be good if DefCore could broadcast intentions for interop by targeting capabilities as prospective core 15:25:29 rockyg, ?? 15:25:41 rockyg: we already do that with advisory status 15:25:41 could we log our agreement here before go on to the next topic 15:25:52 catherine__: good idea 15:25:59 what's the statement you want us to vote on? 15:26:17 Not really. We say these are capabilities with tests we want, not capabilitis we want 15:26:47 ah, rockyg. I'm not sure we're in a position to drive new capabilities. 15:26:52 That we agree for RefStack not to develop plug-in for tests... it will use tempest as the path for testing 15:26:54 We only release list of tests. What we also need are list of capabilities that don't have tests. 15:27:22 Like what catherine__ said. zzero on tests, but identified as core on scoring. 15:27:38 How do we let everyone knows it meets our requirements except tests? 15:27:39 zehicle: +1, that TC territory 15:27:42 catherine__: +1 15:27:55 and Product Group 15:28:15 catherine__: I'm not sure that was the agreement. I thought swift as a plugin was a thing we wanted to go forward on. Am I wrong about that? 15:28:16 The capabilities are already there, just not the tests. 15:28:34 catherine__, I'm confused now. I thought that plug-ins were ok 15:28:43 hogepodge, right, but Refstack doesn't do the dev for plug-ins 15:28:44 it is not just for swift ...the next one is Heat ... 15:28:45 as long as the tests could run from the Tempest framework 15:29:01 I'm reading that as "refstack will use tempest, which will have a plugin to run in-tree tests, so refstack doesn't need to develop it's own plugin" 15:29:21 markvoelker, ++ 15:29:21 (which I think is fine) 15:29:22 matkvoelker: +2 15:29:23 ah, it's about Refstack! ok 15:29:28 markvoelker: ok, fine with that. I'll take the action to move the plugin code that was submitted there to a more appropriate place 15:30:03 #vote DefCore requires that all considered tests be in-tree and runnable under Tempest (using project developed plug-ins are acceptable) 15:30:25 zehicle: amend to add "and gate tests for project" 15:30:26 I think that is (should be) a Hacking entry 15:30:34 #endvote 15:30:48 #vote yes 15:31:01 hold on... I did not start vote right 15:31:18 you need to giv options? 15:31:50 #startvote DefCore requires that all considered tests be in-tree and runnable under Tempest as gate tests for projects (using project developed plug-ins are acceptable) 15:31:51 Unable to parse vote topic and options. 15:32:07 I don't know the vote syntax.... hold on 15:32:14 can someone phrase the question? DefCore requires that all considered tests be in-tree and runnable under Tempest (using project developed plug-ins are acceptable) 15:32:25 zehicle: unless somebody objects, could we just use #agree? =) 15:32:28 th options are th yes, no, etc 15:32:54 How about anyone opposed to the proposal? If not, then we'll put it in as agreed 15:33:03 #startvote DefCore requires that all considered tests be in-tree gate testa and runnable under Tempest (using project developed plug-ins are acceptable)? yes, no 15:33:04 Begin voting on: DefCore requires that all considered tests be in-tree gate testa and runnable under Tempest (using project developed plug-ins are acceptable)? Valid vote options are yes, no. 15:33:06 Vote using '#vote OPTION'. Only your last vote counts. 15:33:18 lol, yes 15:33:18 #vote yes 15:33:25 #vote yes 15:33:26 #vote yes 15:33:33 #vote yes 15:33:46 #vote yes 15:34:14 notification of closing vote 15:34:20 #vote yes 15:34:50 #endvote 15:34:51 Voted on "DefCore requires that all considered tests be in-tree gate testa and runnable under Tempest (using project developed plug-ins are acceptable)?" Results are 15:34:52 yes (6): rockyg, tim_o, markvoelker, eglute, hogepodge, zehicle 15:35:29 #vote yes 15:35:33 catherineD|2, you missed the vote. it passed 15:35:35 * markvoelker provides a quick time check: 35 past the hour, 25m remaining 15:35:47 next topic? 15:35:47 sorry lost connection ... 15:35:59 thank YOU!!! 15:36:27 On to https://review.openstack.org/#/c/214756/ now? 15:36:34 #topic schema schange 15:36:46 let's do schema quickly first 15:37:16 #link https://review.openstack.org/#/c/223250/ 15:37:16 sure 15:37:45 So, the patch is fairly self-explanatory and these changes actually wouldn't break anythign for RefStack that I can tell. 15:37:48 It's mostly just tidying up 15:38:12 But if there are any other schema changes people have noticed we need, it would be good to propose those at some point too. This gets the ball rolling. 15:38:27 ok, we can wait a week for other changes 15:38:31 thank you markvoelker for getting it started 15:38:52 zehicle: I'd actually say let's merge this if people are ok with it and folks can submit further patches to the file. 15:38:57 it lgtm. We can apply to next.json after approval, but it's a good chance to really give the schema some thought, so agree with egle about letting it bake for a week. 15:39:05 All this does is start a definition, we don't have to move next Guideline to 1.4 immediately 15:39:18 ok 15:39:27 I'm ok to merge 15:40:00 done 15:40:00 cool, merging it! 15:40:10 i am too slow! 15:40:12 That was a speedy topic. =) Next? 15:40:20 * markvoelker notes 20m remaining 15:41:04 #topic all apis must exist in designated 15:41:11 #link https://review.openstack.org/#/c/214756/ 15:41:46 * zehicle notes that our next meeting (!6) is voice interactive and all about capability scoring 15:41:56 i agree that apis must exist and be implemented properly, but i dont know that next.json is the right place 15:42:06 So on this one I think so far everyone agrees with the general intent, but there's a lot of disagreement that designated sections are the right way to accomplish it 15:42:29 So, this is very interesting. major influencers in dev are becoming aware of the usefulness of Defcore and this is one of the outcomes of the awareness. 15:42:33 E.g. designated sections are pointers to parts of the OpenStack code rather than a decree about server behavior 15:42:35 does anyone want to speak in favor of DS as the approach to enforce? 15:42:55 rockyg, :) 15:43:01 It becomes an enforceable part of the standard whether the test exists or not. It sends a clear message about the importance of api discoverability. I understand the dissent on the issue, though. 15:43:15 zehicle: me 15:43:26 I think this really is saying that it doesn't matter if you implemented, if you return the interop friendly 403 15:43:33 cool, I wanted someone to be able to explain the position 15:43:51 a few things, the optional apis are never going to have capabilities associated with them, since they are optional. 15:43:57 so tests won't be enforced anyway 15:43:58 because I consider the DS to be difficult to enforce 15:44:32 shouldn't that be part of technical review? 15:44:46 zehicle: what do you mean? 15:44:48 I'd expect that would be a coding issue on how the calls are structure? 15:44:55 This really is to get all projects and all implementers to use the same code response to non-accessible code within the projects 15:45:15 It's about discoverability and consistency. 15:45:20 I think that if we have 100% of the require capabilities doing this in a tested way 15:45:27 hogepodge: Hmm. Not so sure about that. If we had a test (or battery of tests) that tested that you got a proper response from the server whether a features was admin-disabled or not, we could require that test as a Capability 15:45:38 then the optional ones would be highly motivated to follow suit 15:45:47 This also means that anyone could write tests to verify behavior whether the api code exists or not 15:45:54 E.g. "keystone-server-correct-response" 15:45:59 doesn't keystone provide a list of available APIs? 15:46:05 markvoelker, +1 15:46:13 markvoelker: we then expand our capabilities to negative capabilities across the entire api, which adds confusion 15:46:27 for RefStackwill not know t hat there is a 403 returned ... RefStack only notes that it did not pass 15:46:29 I have a lot of trouble believing something is that important to the API but fundamentally not API testable 15:46:45 hogepodge: I'm not sure that's any less confusing than making a statement about it in designated sections 15:46:48 Less so, actually 15:47:06 A 403 is a proper response 15:47:11 So the tst should pass 15:47:54 catherineDl2: refstack wouldn't have to know anything special. A Tempest test would be ridden that calls API's and checks that it either got a 200-series (it worked, capability supported) or a 403 (administratively disabled). 15:47:58 But, we might want a way to identify that 403 was the response 15:48:01 IF so, the test passes. Otherwise it fails. 15:48:08 rockyg: Thx .. will double check on how subunit process 403 ... 15:48:08 if 403 response is important to interop then we need a way to validate compliance 15:48:13 So from RefStack's point of view, it's just another test passing or faiing 15:48:39 so if not implemented, but returns 403, is a pass, then a regular "feature works" pass is also a pass 15:48:50 403 is very important for interop. That way, code can be written to deal with any 403 response 15:48:57 markvoelker: RefStack right now take whatever status defined by subunit processing ... we can take a close look on that ... 15:49:11 it's saying that the api properly implements the response codes, not which apis are actually implemented. Only that anything administratively disabled sends the proper response (403) or is enabled. 15:49:19 so, we are saying that some features can be disabled that that's ok for DefCore interop 15:49:20 rockyg i agree that it is important, but 403 is actually different from having that feature available for interop 15:49:29 if the are not disabled, we expect them to work 15:49:45 zehicle: yes, we're trying to say that some features can be disabled, but they need to send a proper response if they are. 15:50:08 is the problem that we don't want to require the features if the are enabled? 15:50:10 zehicle: some clouds use the load balancer to obscure error responses in the name of "security" 15:50:22 I saw the thread 15:50:26 hogepodge: exactly. The test passes if it receives either a 200 (feature implemented) or a 403 (admin disabled) but not if, say, it times out or gets back some other response from the vendor's API gateway 15:50:27 eglute, right. That's why the designated section requirement. If Defcore requires the capability, but it's turned off, you get the 403, but the code "designated" 15:50:52 i think they mean for all apis to be responsive in some way 15:51:06 eglute: responsive in an actionable way 15:51:06 not just defcore capability required 15:51:16 hogepodge, + 15:51:30 using DS for that is a mess. since it's pretty easy for people to "have the code" 15:51:37 So, this review faces both the dev community and defcore 15:52:13 i think this should be listed separetely somewhere 15:52:23 and maybe worded differently 15:52:32 especially since you are going to be using an API test to validate it anyway 15:53:04 I'm willing to cede the point. I do want to go on record that APIs need to behave in a predicatable and actionable way to support interoperability. I see the problems with using the designated section. 15:53:05 zehicle, how do we handle inaccessible functionality that is "required" for Defcore, but not available to "user" (it's there for admin)? 15:53:18 hogepodge, +1 15:53:39 rockyg, we don't require admin functions for that reason 15:53:41 hogepodge: +1, I would love to see this general idea implemented (as tests =p) across all projects 15:53:45 should it be in https://github.com/openstack/defcore/blob/master/doc/source/process/2015B.rst 15:53:46 ? 15:53:49 rockyg: it's a fail. defcore tests are required to be non-admin. If there's a policy change to the api by the vendor it's not interoperable. 15:53:49 I think 403 is a good step 15:54:01 it allows the technical community to create an option in tests 15:54:13 for APIs that are not always enabled. 15:54:23 I'll look at writing a test for it to add as a capability 15:54:25 that allows us to bring in new capabilities without forcing everyone to add them 15:54:42 but if they DO add them then we can ensure they are consistent 15:55:06 my one hope is that we'd have refstack able to tell us if the pass is "PASS worked" or "PASS 403" 15:55:15 I think that may be too much to ask for now 15:55:19 zehicle, ++ 15:55:29 because I really want to know the data of 403 vs enabled 15:55:51 really, 403, enabled, not installed, not working. all four are valid data to collevt 15:55:52 OK, 5 minutes folks...anything else we need to cover today? 15:55:56 This is an example where IRC is not as effective as voice to get through this discussion... 15:56:01 #meeting next meeting agenda 15:56:07 #topic next meeting agenda 15:56:23 At 14, we talked about making 16 also be interactive for scoring 15:56:28 I think we should stick w/ that plan 15:56:32 +1 15:56:41 +1 15:56:42 I've already setup the meeting invite and call in 15:56:42 works for me 15:56:51 #link https://etherpad.openstack.org/p/DefCoreFlag.16 15:56:52 ++ 15:57:14 everyone should make sure the links and order are up to date 15:57:21 Yup. 15:57:31 #topic open discussion 15:57:33 Very good discussions about glance and neuton yesterday. 15:57:46 I'm going to send the scoring to the openstackdev mailing lest for community comment. 15:57:59 In particular, neutron, keystone, glance, and cinder scoring sheets. 15:58:03 dhellmann, really did a good job w/ Glance issues 15:58:15 I want to take advantage of that conversational momentum. 15:58:19 hogepodge, thanks +1 15:58:22 Unless there are objections. 15:58:24 And he's planning to be more Defcore inclusive in Mitaka 15:58:29 would be great to have participation 15:58:35 +1 15:58:37 Look for major them of Mitaka to be interop 15:58:45 please remind everyone that we are in DRAFTING phase 15:58:46 hogepodge: just noticed the network scoring patch needs rebasing thanks to one of those merges. Will do that now before you send it out 15:59:09 ok, it will be later today. it's a travel day, and I need to hit the road as soon as this meeting ends 15:59:11 is networking ready to be merged otherwise? 15:59:28 I'll do it later today from the airport once I have settle time, or on the plane if there's wifi 15:59:43 eglute: IMO yes, but as the submitter I can't +1 it. =) Folks should review it. 15:59:55 thanks markvoelker 16:00:16 thanks everyone! great discussion today 16:00:16 eglute, you can merge that one :) 16:00:23 will do! 16:00:24 good meeting! 16:00:29 #endmeeting