16:00:03 #startmeeting defcore 16:00:03 Meeting started Wed Jun 15 16:00:03 2016 UTC and is due to finish in 60 minutes. The chair is markvoelker. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:06 The meeting name has been set to 'defcore' 16:00:09 #chair hogepodge 16:00:15 Current chairs: hogepodge markvoelker 16:00:19 o/ 16:00:20 o/ 16:00:24 o/ 16:00:39 o/ 16:00:42 0/ 16:00:43 #link https://etherpad.openstack.org/p/DefCoreLunar.7 today's agenda 16:01:59 So, lots to talk about today I think folks. I want to make sure we get through at least the first three items on the agenda as they're a bit time-sensitive. 16:02:04 Let's dive right in. 16:02:06 o/ 16:02:20 #topic Additional properties on API responses 16:02:36 #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/097349.html Thread on the -dev ML 16:02:36 Has everyone had a chance to read that thread? 16:02:45 o/ 16:03:01 diagonally 16:03:02 I read through early yesterday evening. Seems to be a hot topic 16:03:10 * markvoelker has, and has replied to it a few times 16:03:19 Some vendors have been caught with putting additional properties into Nova API responses. 16:03:25 Just wanted to say that if you read through the comments on the review that turned off the ability, there was concern then of an incompatible change that was ignored 16:03:40 I will be replying later today. 16:03:59 I'd like to give them more time to either a) remove the extra properties or b) work with upstream to get those properties (or something like it) microversioned 16:04:16 * shamail sneaks in 16:04:19 Also, some additiional properties were grandfathered in, but unless you put in a ptch for yours, they weren't considered 16:04:21 IMO, the tests serves the purpose of bring out interops issues. We should not change the test. From Powered Logo point of view, I think we should flag the tests until 2017.01 16:04:40 o/ 16:04:40 Ideally, this would be accepted by upstream qa as a temporary feature to tempest, and by this working group as an acceptable practice. 16:04:43 I think there is a fundamental misunderstanding at play in that thread: None of what Tempest does, or what Nova does, or what the TC does, defines what defcore is. The specific implementation may involve a change to tempest, but tempest devs *do not* get a veto (or fiat) on how defcore-required tests evolve. 16:05:00 that is the purpose of RefStack that we want to collect data... if we change the tests the data really does not mean much 16:05:08 catherineD: My sticking point still is that we're not changing the test. If there was a test for API contracts, that would make sense. That's not what happened though 16:05:18 VanL, ++ 16:05:35 Trenish's comments about "This is the way we do it, so screw it" is fundamentally wrongheaded 16:05:37 dwalleck, ++ 16:05:39 If there were actual tests for API response contracts, we could be talking about flagging or making those tests advisory 16:05:44 dwalleck: the proposal is to comment out the response checking that is changing the test to me 16:05:46 catherineD: we would flag a whole wealth of compute tests, making the guideline meaningless. This is a syntax vs functional change 16:05:47 o/ 16:05:56 hogepodge: so be it 16:06:11 +1 catherineD 16:06:14 Also, Matt's comment of "I'd never consider Juno" when that's where the problem is 16:06:25 the the same thing as mask out checking the fundamental of API response compliant 16:06:29 there's still meaning behind saying "this api performs this action" 16:06:41 that is the #1 enemy of interops 16:06:41 I also think that this is essentially an undeclared capability 16:06:42 catherineD: Not necessarily. This is all the work I had to do to fix my issues, which was 3 changes https://github.com/dwalleck/tempest/commit/53e82a55d2b39a48ef7e50c9b900823105b6a164 16:07:08 catherineD, they changed the original test to encforce the checking 16:07:18 At the start of Kilo cycle 16:07:21 dwalleck: that fixes everything? 16:07:22 There was a defined capability "Add stuff to the API" that was, for obvious reasons, not tested 16:07:28 dwalleck: that's really useful to know 16:07:47 hogepodge: for Rackspace for extra properties? Yes 16:07:53 This was acceptable and encouraged - just until it was not 16:07:58 I would agree if we add a test to specificly test for the response 16:08:09 to hightlight the issue areas 16:08:33 What I would do is propose a new test that tests *only* strict API checking 16:08:51 Add that test as an independent test of an independent capability 16:08:52 Rockyg: so the change is to correct the test that is why I think we should flag the tests 16:09:07 VanL: ++++ 16:09:09 And then turn off that capability for all defcorish tests 16:09:19 VanL: +++ 16:09:20 Also, catherineD, the checking is in the code itself starting in either Kilo or sometime later. So, it's code enforced, not test enforced 16:10:02 Rockyg: my thinking is we should not mask out the issue that we discover .... 16:10:06 catherineD: The problem is that since the property checking isn't it's own test, you would have to flag every test. To me it's a separation of concerns issue, but I know that's not the opinion of the QA folks 16:10:13 Not mask, but not fail. 16:10:23 I also think that we need to make another decision - probably a discussion for midcycle - as to whether defcore's purpose is to put a ceiling on functionality or a floor 16:10:35 I agree with VanL: 's proposal that we should still the the response behavior 16:11:06 I think that in general, defcore should present a floor - a base layer of agreed-upon capability 16:11:09 If you look to Sean's email, it*will* be handled in code for Newton. But our tests are generally used for older releases 16:11:10 Masking out the issue is not why we want to spend all our time and energy for interop test 16:11:14 And I think that should be strictly enforced 16:11:40 but I think that we should, as defcore, allow vendors to add stuff to the API, or to responses. 16:11:43 Upstream has been very clear that additional responses and extensions that aren't upstream are out. I feel obligated to fall in line with that eventually. I'm asking for extra time for vendors to either remove the extra attributes or work with upstream to release new microversions 16:11:52 VanL: + 16:11:58 I think we all think of DefCore here as "floor" 16:12:02 Maybe there is a namespacing issue - vendor-specific stuff under /vendor/ 16:12:29 hogepodge: ++ 16:12:30 Also, I've been told by python-openstack client that strict response checking is on the feature list for their client, so this is going to be the norm in the future rather than the exception 16:12:31 hogepodge, ++ 16:12:44 hogepodge: by not testing the response we won't know which vendors still have the issue 16:12:59 My goal here is to raise the issue transparently to the community and buy time. 16:13:02 hogepodge: As soon as there is an upstream way for vendors to fix issues and add capabilities. 16:13:26 catherineD: we will know exactly with calls are returning extra data, my proposal makes it a requirement to report it 16:13:31 I fundamentally do not agree with "the fastest any vendor can go is at the speed of the community" 16:13:39 If this is the norm going forward and we don't change Tempest, is there a way we could make the DefCore guideline specifically saying that API response checking is a required thing? 16:13:55 hogepodge: who will report and how? 16:13:58 hogepodge, until the enforcement is in the code, I don't think we can fail the vendors who take advantage of the laxness 16:13:58 I *do* agree that any vendor-specific functionality should be explicitly marked, so as to not affect interop 16:14:13 VanL: this is why, at a minimum, I also thing we need to start using a branch of Tempest and back-porting changes, so we have a known working instance at time of guideline approval 16:14:22 s/think/thing 16:14:51 hogepodge, ++ 16:14:55 I'd just like to make sure the requirement is captured in some way that can be referenced, since you can't currently point to a test right now that defines that requirement 16:15:04 catherineD: vendors will report to the foundation at time of applying for the trademark, and the APIs that vary will be listed in their marketplace entry 16:15:07 hogepodge: +1, at a minimum. We will need to fork sooner or later. Friendly fork, but fork 16:15:42 DefCore does not work from trunk, it works on releases that *aren't* changing so neither should he test/test frameworks 16:16:15 dwalleck, hogepodge: What about adding a new "adheres to strict API checking" test, and adding that as advisory? 16:16:36 VanL: ++ 16:16:38 And turning it off in all other tests (per dwalleck's changes)? 16:16:38 I wouldn't think a fork would be required if you can find a tempest tag or release the stick with 16:16:45 dwalleck, it's in a design spec. Change in code functionality. They changed the tests first to prevent regression in new code being added 16:16:51 * docaedo forgot to say hi earlier 16:17:44 docaedo, problem is, if you go for a tag, you don't get any of the new tests or bugfixes 16:17:44 Rockyg: as a interop testing group. Arer't we glad that they change the tests to expose this? 16:17:48 VanL: I really don't know. I don't disagree with the response being checked inline, I just want more time to let that be optional. 16:17:50 docaedo: The difference is in the purpose for testing. Gate testing (avoid regression) is fundamentally different from defcore testing (assure baseline interop) 16:18:27 catherineD, yes, but not without fixing the code, too. 16:18:54 VanL and hogepodge: By hogepodge's original email, if we could make those "additionalProperties" fields configurable, which shouldn't be difficult, then we could really handle this problem without impacting Tempest further 16:19:02 hogepodge: /me waves back at old "atomic" discussions 16:19:07 Rockyg: fixing the code means do not allow extension? 16:19:24 VanL and Rockyg understood, and I assume "fork tempest" does not intend to diverge long term from tempest - but please continue, did not mean to distract from the conversation at hand 16:19:27 The issue is that dev )QA dev) thinks that all of vendors, no matter what release they put out there or when, should meet all new test standrds 16:20:31 no problem, docaedo ;-) We like developers to know our pain and help solve it 16:20:34 We could write tests specifically for API compat and require those, while still using the rest of Tempest without the need to go our own route 16:20:42 Question for this group: Is there anyone *here in defcore* that thinks we should have strict API checking *right now*? 16:20:45 Rockyg: well, if we assume tempest works at guideline release, we should branch it at guideline release then cherry pick bug fixes. Then for a given guideline we know that we won't be caught by changes 16:20:47 dwalleck: +1 16:21:08 VanL: In six months, yes. 16:21:11 hogepodge, ++ Like in the old days :-) 16:21:13 With the added benefit that folks not totally API compliant would be able to use Tempest at all for functional testing 16:21:13 I see that there are different thoughts about its applicability later 16:21:54 VanL: I believe in strict API testing for interop purpose and we should enforce or expose those who do not ... but does not have to fail them for trademark purpose 16:21:59 VanL, when it's in the code, and only for those releases that have it and going forward 16:22:26 catherineD: +1 16:22:32 markvoelker: in interest of time, do you have any suggestions on how we should proceed? 16:22:40 Rockyg: that is why we should not fail them for trademark purpose 16:22:50 we're not going to reach consensus this week 16:23:05 (if ever ;-P ) 16:23:09 hogepodge: agreed. So, a couple of things: 16:23:39 hogepodge: I can accept strict API checking when/iff there is an allowed extension mechanism that doesn't gum up interop. 16:24:13 1.) There are a lot of opinions being expressed here that I don't think have made it out to the bigger audience on the ML's yet, so I'd encourage you to chime in there. 16:24:26 This is an important topic for the entire ecosystem. 16:24:37 catherineD, I agee we need to know. I also think that we need to somehow get devs to understand that the change in contract going forward (strict checking) is a now and forward conract that can't be applied (but can be exposed) for artifacts that existed before the contract change 16:24:42 markvoelker: ++ 16:25:15 markvoelker, ++ 16:25:20 2.) hogepodge, I think we need to create formal proposal of some sort that folks can vote on and that lays out where changes would be made (in tempest, refstack, or elsewhere) 16:25:45 markvoelker: "folks can vote on" - which folks? 16:26:02 Rockyg: ++ ... that is the reason why we have the RefStack to collect data for statistical purpose eventually 16:26:11 VanL: DefCore needs to decide if we're going to flag tests or not, and if we're going to add additional separate capabilties in the future or not 16:26:18 That is one important point: We listen to and take input from other groups, but they don't get a vote 16:26:29 VanL: As always, I'm interested in hearing feedback from others. 16:26:34 oh, and markvoelker, could you change you nick so I don' have to type 2/3 of your name before I get a unique match on IRC?) 16:27:18 3.) I imagine this is the sort of thing the Board is going to be interested in since we are ultimately a Board body. 16:27:50 There is a Board meeting coming up later this month, and I think we should put in an agenda item to raise awareness even if we don't have a clear plan of action yet. 16:28:02 ++ and I think that is what hogepodge said in the original post. 16:28:03 markvoelker: for 2) I will write up a proposal, which will include some alternatives based on what happens in this group and upstream 16:28:14 markvoelker: I think that this is the sort of thing that should go in the defcore test spec that gema has been working on - we could propose that to the board and get guidance 16:28:30 markvoelker and hogepodge: does it make sense to hold off on the voting until after midcycle? This topic seems perfect for discussion at the event. 16:28:30 VanL, ++ 16:28:31 And that could be *defcore 16:28:37 's guidance 16:28:38 VanL: ++ 16:29:31 shamail: I think we should get a proposal put together sooner rather than later. When it lands is going to depend on the discussion. 16:30:24 Also, expext discussion in TC this week or next 16:30:30 Sounds good, thanks. 16:30:31 shamail: keep in mind that this is time sensitive, we have vendors looking to renew or get a new trademark now or soon 16:30:54 shamail: too long may be too late, but we also can't rush it. 16:30:59 hogepodge: how many tests are we talking about? 16:31:02 shamail: basically, a whole lot of no fun 16:31:12 catherineD: from dwalleck's patch, three 16:31:23 catherineD: no, not tests, APIs 16:31:27 catherineD: I don't know how many tests 16:31:27 hogepodge: thx 16:31:57 catherineD: dwalleck and Rockyg would have better data than I do 16:32:00 hogepodge: I can give you some more stats if that would be helpful 16:32:06 dwalleck: yes please 16:32:13 Rockyg: Could you take a look at dwalleck's patch - does that work for Huawei? 16:32:29 please. I don't get stats from our team, just confused whining 16:32:37 #link https://github.com/dwalleck/tempest/commit/53e82a55d2b39a48ef7e50c9b900823105b6a164 16:32:47 Oops. did I say that in my IRC voice? 16:33:17 I would be interesting to know whether those vendors are in fact interop with others at the "core" level 16:33:24 I'll pass it by our guys and request a quick response. Should be able to get it for tomorrow. 16:33:27 Heh :) I like the idea of having a proposal to review... The discussion could carry over if necessary to midcycle (maybe even a short-term versus long-term position discussion if we resolve this actual topic) 16:33:47 dwalleck, is that patch everything my team might need? 16:34:26 Rockyg: That depends on where your extra properties are. This squashes checking on servers, flavors, and quotas responses 16:34:35 catherineD: I'm working under the presumption that the additional properties would not impact any applications, and can be safely ignored. I wouldn't be advocating for this if I thought otherwise 16:34:40 catherineD, my understanding is our team added the extra response to handle network stuff they neede that didn't exist in Neutron 16:35:08 Yours are probably different, but if you send me a gist with what your responses look like, I might be able to point you in the right direction 16:35:43 dwalleck, Thanks. My guys are pretty good. I should be able to get the list of non-pass from them, too. 16:35:46 Rockyg: I'm also working under the assumption that these additional properties will be removed or integrated upstream 16:35:51 catherineD: If you use the common rule (from other similar standards) to "Ignore what you don't understand," then, yes, interop works 16:35:55 (in future product releases) 16:36:15 Can we call time on this to hit the other topics? We ok with that? 16:36:40 ++ 16:36:49 ++ 16:37:04 They know their stuff, just not community process, and hogepodge so am I. Move to next OpenStack release for them will solve lots of problems and reduce where they neede to do extra stuff. 16:37:08 It may perhaps be a good idea to prove that point out a bit...personally I haven't found a lot of commonly used toolkits that barf on extra attributes, but it may be worth putting some time into testing ones we know are widely used. 16:37:47 (the user survey has some data on popular sdk's and such) 16:37:52 markvoelker: That would actually be a very helpful thing we could point to 16:38:12 ++ to more dat and to moving on 16:38:43 Ok, 20 minutes to go and we really need to hit a couple of the other agenda items today. 16:39:03 #topic Midcycle planning 16:39:07 VanL: markvoelker: so the common rule (best practice) is for vendors to " ignore what you don't understand" for interop puprpose 16:39:49 catherineD: Yes, although the real target of that advice is clients 16:39:57 #link https://etherpad.openstack.org/p/DefCoreSummer2016Sprint Midcycle planning etherpad 16:40:24 So from the preliminary info we got there, it looks like the week of August 1 works well for most folks 16:41:03 We also have 4 volunteers to host. 2 in Silicon Valley, 1 in Texas, 1 in the UK. 16:42:58 We need to firm this up, and the sooner the better. I think the -2/-1/+1/+2 format we used for dates actually worked pretty well...should we do that for locations too? 16:43:13 sure 16:43:13 Or would folks prefer I send out a Doodle poll? Or...? 16:43:17 I'm not saying this, but I'm just saying, our guys haven't figured out sponsoring meetings yet, so it could take me some time to make it happen. I'd like to go through the steps with them so they understand the requirements, especially planning and costs, but likely can't act quickly enough. 16:43:53 Rockyg: so, we may need to withdraw Huawei as a possible site then? 16:44:19 if vmware is an option, the valley would be covered anyway 16:44:54 hogepodge: True. And I can confirm VMware is definitely an option. 16:45:01 We have a couple venues at our campus here, depending on how many people we have (size is not a problem, just different room sets fit different sizes) 16:45:06 markvoelker, I'd say, it would be best to this round. We made the gesture, now I have to get them to understand both the cost of doing it aand the cost of lost opportunity for not. 16:45:25 Rockyg: ok. Thanks for the interest, and for clarifying. 16:45:43 Yeah. It's an education process. 16:46:17 Ok, with that in mind I'll send out mail this week asking folks to fill in -2/-1/+1/+2 on the pad and we'll finalize at next week's meeting. Sound good? 16:46:28 +1 16:46:29 Thanks all for understanding. Hopeuflly ready for either next cycle or next time valley is an option 16:47:11 Great. Ok, there's also a topic list on the pad there, so let's try to make sure we get all prospective topics listed too 16:47:13 Can you leave us on, but grayed out or something? I want them to see the process with us in it, but not winning. 16:47:15 (I'll add that to the note) 16:47:32 Actually, most of the folks here voting for something else would do it. 16:47:54 Rockyg: so you want us to -1 you? 16:48:05 Plus VMware has a *much* nicer campus and better food. 16:48:07 #action markvoelker will send email to ML this week to finalize dates/places on the etherpad 16:48:18 gema, yup. 16:48:26 Rockyg: you got it :) 16:48:41 Anything else on midcycle planning we need to hit today? 16:49:16 Moving on... 16:49:26 #topic Upcoming Board Meeting 16:49:56 There's a BoD meeting at the end of the month. I put a couple items in the pad that we may want to bring up. Anything else? 16:50:31 The spec 16:51:09 looks good. 16:51:17 VanL: noted. 16:51:53 Uh, is there a report on how many and types of vendors that have passe, and which guidelines, etc? 16:52:10 It would be great to start getting some numbers into the minutes 16:52:59 Rockyg: no, although that data is available in the marketplace and I think we've occasionally provided a rollup in the past. 16:53:53 I think we should schedule a numbers report to the for a specific time in our cycle 16:54:01 to the boar. 16:54:06 board. 16:54:15 Rockyg: reasonable. Maybe part of the major issues report? 16:54:19 But not part of this agenda, so future item 16:54:32 markvoelker, ++ 16:54:41 Ok, moving on if nothing further for the Board... 16:54:47 #topic Documentation generation update 16:54:52 hogepodge: take it away 16:55:24 Rockyg: markvoelker: When RefStack finishes vendor/product registation .. this data would be obtained easily 16:55:29 Can everyone look at the documentation updates? It's a work in progress, but I want to get those published to openstack docs 16:55:42 #link https://review.openstack.org/#/c/329727/ 16:55:47 The first step is cleaning things up 16:55:51 can do 16:56:06 The second is adding a post-gate job that publishes 16:56:12 catherineD: duly noted 16:56:20 I want to add the lexicon, clean things up, but that's a start 16:56:44 hogepodge: thanks for this. I'll take a look. 16:56:45 plus, if I'm missing anything, please let me know 16:56:55 #action everyone please review https://review.openstack.org/#/c/329727/ 16:57:48 Anything else on this? We're down to a couple of minutes 16:57:49 doing that now... 16:57:50 hogepodge: now that we move schema to JSON format ... we seem like loosing the comments 16:58:34 The comments has been very helpful ... should we still keep the rst file for comments 16:58:44 catherineD: I added an rst file, so we can annotate there 16:59:09 catherineD: change log and imports the schema, we can do an unofficial mock up with comments too https://review.openstack.org/#/c/329727/1/doc/source/schema/1.5.rst 16:59:26 cool 16:59:56 Ok, I think we're out of time. Thanks folks--and again, I'd love to see some of you chime in on the broader ML discussion. See you soon! 17:00:04 #endmeeting