16:00:45 #startmeeting defcore 16:00:45 Minutes: http://eavesdrop.openstack.org/meetings/monasca/2016/monasca.2016-03-02-15.00.html 16:00:45 bye 16:00:46 Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2016/monasca.2016-03-02-15.00.txt 16:00:47 Log: http://eavesdrop.openstack.org/meetings/monasca/2016/monasca.2016-03-02-15.00.log.html 16:00:48 Meeting started Wed Mar 2 16:00:45 2016 UTC and is due to finish in 60 minutes. The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:53 The meeting name has been set to 'defcore' 16:00:55 o/ 16:00:58 o/ 16:01:04 o/ 16:01:05 o/ 16:01:05 #chair markvoelker 16:01:06 Current chairs: eglute markvoelker 16:01:19 #topic agenda 16:01:22 #link https://etherpad.openstack.org/p/DefCoreRing.14 16:01:33 Hello Everyone! please review the agenda 16:01:34 o/ 16:02:02 #topic Feedback from running DefCore tests in CI 16:02:11 dwalleck, the floor is yours! 16:02:14 o/ 16:02:24 eglute: Thanks! 16:03:00 So I've been working with folks inside Rackspace for awhile now on continuous DefCore testing. It's been....fun :) 16:03:45 But there were some common themes that have come up that might be of value to a larger audience 16:04:12 o/ 16:04:50 The majority of the feedback was in some way centered on Tempest results 16:05:49 People really, really don't like that when you run the subset of DefCore tests, that the pass/fail numbers don't add up to the total of tests to be run for DefCore 16:06:05 dwalleck: why? 16:06:22 o/ 16:06:31 because of the tempest.conf 16:06:35 you can configure tests out 16:06:45 purp: If there's a failure in a test fixture, that counts as one failure rather than the number of tests to be run in the class 16:07:38 dwalleck: ... thus the numbers don't sum, leaving you feeling empty and cold. Got it. 16:07:46 purp: Exactyl 16:07:56 that does sound like a lot of fun... 16:08:17 So after some manual auditing, we wrote a quick additional subunit parser to check the list of tests executed against the one used to run the tests 16:08:28 https://github.com/arithx/subunit_verify 16:08:42 This is a hack, but it gave people warm fuzzies :) 16:08:43 dwalleck: we have room in the defcore repository for tools like that 16:08:59 #link https://github.com/arithx/subunit_verify parser to check the list of tests executed against the one used to run the tests 16:09:09 hogepodge: That would be great! 16:09:24 +1 hodgepodge 16:09:27 dwalleck: DefCore pass/fail shoukd be checked against the DefCore test list and not from pass/fail based on subunit file. 16:10:15 I also know that if I had product teams push their results to RefStack, alot of that would be sorted out. However, having a dozen product teams sending results to RefStack would be a challenge to understand 16:10:18 My opinion is that the repository can house tools to make testing life easier. The defcore chairs have the final say on what goes, though. refstack may be a more appropriate place for some tools also. 16:11:01 I had never thought about how to use RefStack properly if you have multiple entities from one provider trying to use it all at once 16:11:05 i am ok with having additional tools! 16:11:50 dwalleck: you might be pleased to note that the RefStack folks have thought about that. =) There are some specs out for review on vendor/product relationships that catherineD can probably point you to. 16:11:51 hogepodge: Totally agree. I figured the "where" could be sorted out as long as the "why" made sense 16:12:04 A tools repo to improve life is a great start. Over time, key functionality from random tools should be migrated into RefStack or a similar all in one client though. :-) 16:12:31 ^re-useable not reliable 16:12:39 o/ 16:13:12 markvoelker: That's good to hear :) I'll have to ping catherineD|2 next week with some of the specifics 16:14:22 The last bit of feedback, bluntly, is that people just don't seem to understand Tempest test failures, i.e. what part of capability is actually broken when a test fails 16:14:49 This has given me gray hairs over the last 6 months :) 16:15:10 i thought you said it was fun? :) 16:15:35 To give you an example, our public cloud has a number of failures due to certain extra properties being returned in API responses 16:15:43 Maybe dwalleck is aiming for Silver Fox look, so the grey hairs are fun? 16:15:44 fun like being in the middle of a hurricane... 16:15:53 so given this feedback, what do you think the solution would be? use the tool you worked on for people that want warm fuzzies? 16:16:31 This causes a large number of test failures, but all with the same root cause. I had to write some aggregator tooling for results so that people didn't think the sky was falling :) 16:16:37 also, anyone else testing regularly have similar (or different) feedback? 16:17:05 Sounds like maybe you'd also like to see some improvements to the test failure messages ("test failed because stuff I don't know about was detected in the JSON respnonse")? 16:17:06 But even with that, I had a hard time explaining that the entire capability (create server, etc) was not broken, but just an aspect of it 16:17:35 mvoelker: Exactly. That would be a place to start 16:17:46 Part of what I'm seeing is that the Tempest tests as-is are not quite atomic enough. It would be nice if a failure was self-explanitory 16:18:09 dwalleck: Failure due to extra info response is the purpose if interops test right? 16:18:37 extra info won't necessarily break people's apps, lack of info would 16:18:38 I could debate that. Failure is fine. Failure shoudl be explicit though 16:18:46 catherineD|2: yes, according to the test authors/TC 16:19:02 gema: +1 16:19:02 catherineD|2: True, but me delivering a result to a product manager saying "the create capability is broken" isn't specific enough, and causes panic 16:19:06 catherineD|2: but I think the larger point was that it was difficult to tell that that's why the test failed 16:19:30 And is misleading because the functionality of the capability worked 16:19:53 dwalleck I think that's the key 16:20:00 I don't disagree that it should be a failure. I would argue that we could make aspects of it more atomic 16:20:12 agree that debugging on failure is not easy 16:20:25 I'd say that a test which fails due to extra info in a response is either too fragile or should fail because it's specifically testing to prove the info isn't there. 16:20:39 * purp is contrary 16:20:41 purp: +1 16:20:43 purp: my point as well 16:21:03 part of what dwalleck is talking about when he says the tests need to be more atomic. :-) 16:21:04 Either you're specifically testing to see that the info is excluded, or your test is too fragile and breaking for poor reasons. 16:21:09 purp: The challenge is that the response checking is built into the Tempest clients, and therefore into every test as an implicit check 16:21:10 Yeah. Tempest was not designed to return info on option by option basis 16:21:28 ^what dwalleck just said 16:21:33 hogepodge what have you been hearing from people that are testing/certifying? 16:21:34 An explicit check for response conformity, etc, would be much easier to understand 16:21:51 in current implementation *any* test will fail if there are extra params in the repsonse, regardless of whether or not the functionality actually works 16:22:05 eglute: ++. I'm really interested to hear if others have had similar thoughts 16:22:13 Okay, so sadly, then, I'd say the Tempest parser is either too strict for purpose (not reflective of real world) or we're expecting to enforce a response interface that is not allowed to include any extra info. 16:22:21 I'd view the latter as a poor design choice. 16:22:30 +1 16:22:32 Configuration is always a problem. Pointing to the official tempest configuration guide solves a lot of problems. 16:22:33 +1 purp 16:22:43 Part of the problem is that there is no "standards" for writing api tests (or any others) for that matter, other than it tests something the devs care about. 16:23:03 purp: SammyD: Yeah, that's the explicit intent of the authors. I can point you to the discussion on this a few months ago when that change was made to Tempest, but let's do that offline so as not to distract from the larger point: atomicity and better failure messages 16:23:09 Usually vendors get permanently stuck when they've made downstream modifications or are backing their block storage with ceph. 16:23:26 mvoelker: agreed, thanks. :-) 16:23:28 hogepodge: But what about interpreting results? Have you had much feedback in that area? 16:23:45 So, unless/until there are more standards as to what comprises the test, or how the test should respond/report, we'll see this crop up. 16:23:56 markvoelker: before we offline, want to understand. I believe you just said that strict parsing is an intentional design choice. 16:24:18 They get confused about what needs to be run. Oftentimes I'll get support tickets on required but flagged tests (the terminology is not helpful) 16:24:43 purp: correct. A decision was made that adding extra data to API repsonses is no bueno and that vendors should not change the API. 16:24:43 I didn't mean to soapbox this long, but I was hoping this would be helpful feedback :) I'll be in Austin next week and I can show the specifics and some of the other tooling I've put together 16:25:00 when fixtures fail, it's usually trackable to common problems 16:25:20 markvoelker: I'm sad to hear that. Okay. 16:25:36 dwalleck: This is good feedback! Would you consider writing up a short summary in the etherpad (or somewhere else) so we can refer back to it? 16:25:37 eglute markvoelker: do we have a session for this in Austin? 16:25:43 I can dig up the mail thread about that, given enough time. I can post to ML> 16:25:52 we have a session to talk about tests in general, yes 16:25:57 markvoelker: Absolutely! I'll do that 16:26:11 purp: The API response thing? No, it was a decision made by nova-core IIRC. 16:26:39 and/or other projects 16:26:49 Feedback is always good. We shouldn't consider the standard or our policies as inflexible (although as the person administering the program I do need to enforce the it) 16:26:52 markvoelker: Sorry, meant the tests and responses. 16:27:10 Yeah. more strict validation of expected return info. 16:27:35 mvoelker: I don't agree or disagree with the paramater checking, just that it is not explicit when it fails due to the way it is implemented. :-) 16:27:38 purp: Oh! Sort of. There's an hour on the tentative schedule to talk about the possibility that Tempest isn't meeting all our needs 16:27:39 that is to say, we learn from what we have and evolve the standards and guideline 16:28:02 SammyD: yep, I hear ya. =) 16:28:29 SammyD: file a bug against the test? What was the response? 16:28:30 I propose that we move the rest of this discussion to the midcycle. Thanks dwalleck, would be great if you could do a write up, as markvoelker suggested 16:29:07 rockyg: Couldn't file a bug against the test, there is no test for the API responses. It's a global default validation in the client. :-( 16:29:17 I think some of this dovetails into the topic we currently have at 2:30 on Day 2... 16:29:32 #link https://etherpad.openstack.org/p/DefCoreSpring2016MidCycle First pass at agenda for midcylce 16:29:33 Ah. Yup. I feel your pain. 16:29:33 eglute: no problem, thanks! 16:29:36 right, lets talk about the midcycle 16:29:43 #topic midcycle 16:29:57 Huawei next cycle will hit same problem. 16:30:29 if you have not responded yet regarding your restaurant preferences, please do so today before 2PM CST: http://doodle.com/poll/x3n9yvdws9miyrck 16:31:05 Also, please take a look at the agenda: https://etherpad.openstack.org/p/DefCoreSpring2016MidCycle 16:31:23 thanks markvoelker for preliminary schedule setup 16:31:33 So, regarding the agenda: 16:31:56 You'll notice it's pretty packed. =) I made a first pass at it including all the topics that were in the suggested topics list from the pad 16:32:00 Would like to swap the 9:30a and 10a sessions on the first day. 16:32:11 But I think there are probably some candidates for pruning in there so we can free up more time for longer discussions 16:32:30 You'll notice a couple of themes for each day 16:32:41 right, i think the test discussion is strategic and we need to have it on day 1 16:32:49 Those are sort of arbitrary, just a way to bundle together related topics 16:32:51 and i am guessing it will take more than 1 hour 16:32:57 right, thanks markvoelker 16:33:27 Egle and I were thinking of swapping the Outcomes and Tests sessions so we can tackle Tests on Day 1 16:33:58 And also probably dropping the Publishing DefCore Information and Docs session since it really just seems like a to-do list item we can tackle in 5 minutes 16:34:10 Any objections to either of those things? 16:34:39 works for me 16:34:44 +1 16:35:08 #action markvoelker to swap Outcomes and Tests sessions, drop Publishing session 16:35:24 Ok, any other items that look like targets for pruning? 16:36:22 might need to move user feedback to the next day based on Edgar Magana's availability 16:36:51 eglute: I was just about to come to that and tackle purp's request in one fell swoop =) 16:37:08 :) 16:37:25 also, I am worried that some of these discussions will take longer than we have the time 16:37:30 Ok, before we get there: if anyone else has suggestions for stuff you don't think is urgent enough to warrant time at the midcycle, drop me a line and we'll discuss 16:38:12 So, purp: I actually originally had those swapped, but one of those sessions will have Edgar Magana from the User Committee joining us remotely from California (where it'll be pretty early) 16:38:16 markvoelker: please allow time for RefStack discusssion (one hour) under Refstack demo 16:38:33 And actually we just heard from him that he'll need a later timeslot anyway, so stand by while we tinker with that some more 16:38:38 markvoelker understood, and I think that session precedes any rational discussion of operating model 16:39:10 purp: Sure, I think we can work that out. 16:39:16 i would like to move Scoring: Promoting Advisory Capabilities discussion to before actual scoring 16:39:25 markvoelker cool. Otherwise, I'm happy to roll with whatever. 16:40:11 eglute: That's easily done, the scoring discussions can really be in almost any order unless they require remote attendance from anyone. 16:40:26 thanks markvoelker 16:40:46 #action markvoelker Move Promoting Advisory Capabilities to earlier in the day 16:40:47 hogepodge are there any updates on the licensing changes? 16:41:33 eglute: I can check in again. It's with the lawyers and exec staff right now 16:41:46 #action markvoelker rework Value From DefCore session to accommodate remote attendee 16:41:50 thanks hogepodge, we can always cover it during the regular meeting 16:42:25 Any other shuffling folks are interested in after looking at the first pass? 16:43:09 If not, I'll rework the agenda with the feedback you've all given so far and send out an update to the ML this afternoon for further iteration 16:43:14 thanks markvoelker 16:43:23 One more big important thing.... 16:43:52 also, everyone, please note that the schedule starts at 8:30 rather than my original invite at 9:00 AM. does 8:30 work for everyone? 16:43:52 It was a little hard to tell who added some topics to the suggested topics list, so you'll notice that some of the sessions on the strawman agenda don't have a designated leader 16:44:10 +1 eglute 16:44:21 If you added a topic to the list and want to lead it and are not listed as the leader in the agenda, please add yourself! 16:44:31 +1 16:44:36 argh. I get in 1am that morning, but Ill make. 16:44:37 For those that don't pick up a leader, I'll either handle them myself or split them up with Egle. 16:45:16 markvoelker: +1 16:45:20 Regarding the 8:30 thing: please note that 8:30-9 is just introductions, agenda review, and coffee 16:45:30 So if you're a little late, that's probably not the end of the world 16:45:48 nobody wants to be late for coffee with such an agenda 16:46:11 any other people have concerns about starting at 8:30 AM? i don't like mornings in general, but that's a different issue :D 16:46:13 * markvoelker just subtly put brussen on the hook for making sure there is copious coffee, but he's a smart cookie and probably already thought of that 16:46:26 :D 16:46:56 any other comments regarding midcycle planning? 16:47:12 eglute: two quick ones 16:47:49 We have an official dinner planned...anyone interested in informal dinner/drinks gatherings Monday evening or Wednesday evening (for those who get in early/leave later) 16:48:22 IF so, please drop your name in the DefCoreRing.14 etherpad and if there's enough interest we'll try to coordinate 16:49:00 eglute: ok, done now. =) 16:49:14 I'm always down for drink....err food :) 16:49:28 #topic refstack 16:49:32 Heh. I have drinks plans in South Austin Monday, would welcome dinner. 16:49:34 catherineD|2, go ahead! 16:49:39 eglute: thx 16:50:06 purp -- put your name in https://etherpad.openstack.org/p/DefCoreRing.14 line 38! 16:50:16 The first topic is about prevent uploading of duplicate test results to RefStack 16:50:48 #link RefStack bug https://bugs.launchpad.net/refstack/+bug/1498159 16:50:49 Launchpad bug 1498159 in refstack "refstack-client should help user to avoid test duplication" [Medium,In progress] - Assigned to David Liu (lzbj) 16:51:50 it seems like the only way to ensure that duplicate test results are not uploaded to RefStack to enforce testing done with RefStack client and upload the result immediately after the test is done 16:52:31 vs today ... test results can be uploaded afterward 16:53:05 i think it is ok that the tests can be updated later 16:53:17 i prefer to have the flexibility 16:53:42 Hmm...wonder if this could be solved by computing a SHA on the test results + logs or something? 16:53:44 And I wouldn't assume that they're always run from an internet-connected machine. 16:53:53 * markvoelker is thinking out loud which is often dangerous 16:54:04 markvoelker: I was thinking UUID identifier in the beginning of test run 16:54:05 purp: ++ 16:54:05 eglute: since we allow test can be uploaded later we have no good ways to ensure that duplicate test results uploaded to RefStack 16:54:25 * purp assumes ++ was for internet-connection-optional 16:54:30 purp: that would have to be built into tempest though if we don't require that refstack-client be used, right? 16:54:46 markvoelker it would, but would help overall debuggability 16:54:53 (if that's a word) 16:55:17 This is just one aspect of the bigger issues of test integrity 16:55:33 markvoelker catherineD|2 perhaps both: UUID for human readability, and a SHA on the upload to point out duplicates. 16:55:39 purp: Yeah, I was thinking a SHA on the results/logs gives a unique identifier that's independent of the test runner 16:55:56 E.g. you're essentially creating a unique identifier at the end rather than at the start 16:56:18 catherineD|2, ++ But, we've been operatibg ob trust since the beginning. 16:56:19 markvoelker: purp: subunit file or RefStack JSON file is a text file ... it is easy to change 16:56:31 markvoelker fair, and I think both would be useful. 16:56:44 4 min warning 16:56:55 * markvoelker glances at the clock and wonders if we should take the brainstorming over to #openstack-defcore so Catherine can talk about other stuff 16:56:59 it the SHA is created at refstack-client ... the "malicious" user can really recreated the SHA 16:57:18 I think this can be a topic at the midcyle 16:57:34 thanks catherineD|2! 16:57:46 next move on to the second topic of RefStack 16:57:52 i will be in the defcore irc after this as well 16:57:55 So, from my perspective, we shouldn't worry too much with what tempest is as the moment and work to build in better accountability through enhancing/adding checks to tempest in future. 16:58:16 RefStack vendor registration question in http://lists.openstack.org/pipermail/defcore-committee/2016-February/001035.html 16:58:23 we are out of time... if you can stay a bit longer, please move to the #openstack-defcore 16:58:38 thanks everyone, see you next week! 16:58:42 hogepodge: do we allow "Vendors registering with RefStack not being in OpenStack MarketPlace" 16:58:45 #endmeeting