16:00:47 <eglute> #startmeeting defcore
16:00:48 <openstack> Meeting started Wed Jan 20 16:00:47 2016 UTC and is due to finish in 60 minutes.  The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:52 <openstack> The meeting name has been set to 'defcore'
16:01:00 <markvoelker_> o/
16:01:05 <hogepodge> o/
16:01:07 <eglute> #topic agenda
16:01:08 <eglute> #link https://etherpad.openstack.org/p/DefCoreRing.9
16:01:12 <eglute> Hello Everyone!
16:01:20 <brunssen> Hello
16:01:37 <eglute> Please review the agenda, and add/update as needed
16:02:15 <eglute> If you are here for the DefCore meeting, please wave your hand o/
16:02:46 <eglute> #topic 2016.01 guideline
16:02:48 <brunssen> o/
16:02:52 <dwalleck> o/
16:02:54 <blogan_> o/
16:03:16 <eglute> The first item on the agenda is the 2016.01 guideline
16:03:31 <eglute> #link https://review.openstack.org/#/c/239830/
16:03:55 <eglute> we need to get board approval for it during the next board meeting, which is on Tuesday afternoon
16:04:04 <catherineD> o/
16:04:20 <markvoelker_> This needs a rebase b/c of https://review.openstack.org/#/c/264339/, right?
16:05:19 <eglute> yes, the changes need to make it into the latest guideline. I am not sure why gerrit didnt give a merge conflict for https://review.openstack.org/#/c/239830/
16:05:29 * markvoelker_ rbases it
16:05:46 <eglute> thank you markvoelker_ ! awesome as always
16:06:07 <markvoelker_> eglute: it just hasn't been rechecked since the other change merged.  Should be fine now but I'll keep an eye on it.  Move on unless there's further discussion?
16:06:20 <eglute> besides that change, are there any other outstanding ones? i will be reviewing it later this week
16:06:21 <eglute> yes
16:06:36 <w00tburger> anyoen know if there is there such a thing for hyper V which is that of "guest tools" in environments such as vmware and virtualbox?
16:06:37 <eglute> #chair markvoelker_ hogepodge
16:06:38 <openstack> Current chairs: eglute hogepodge markvoelker_
16:06:46 <eglute> #topic midcycle
16:07:17 <purp_too> o/
16:08:05 <eglute> w00tburger i think you should try a different channel
16:08:32 <eglute> regarding midcycle, IBM has kindly agreed to host us again. they have great offices in Austin
16:09:01 <markvoelker_> excellent.  Thanks brunssen et al.
16:09:10 <brunssen> @eglute, Yes, we plan to host.  I will work to make sure we can get the same space as we had last time.
16:09:19 <eglute> yes, really appreciate it :) thank you brunssen
16:09:25 <brunssen> Do we have dates.
16:09:31 <eglute> yes, march 8-9
16:09:57 <brunssen> OK, I will put in the request for the rooms.  Do we have an idea of how many people?
16:10:31 <eglute> we have 11 right now, I am guessing maximum 15
16:10:53 <hogepodge> eglute: rolling back to previous topic quickly, this test http://git.openstack.org/cgit/openstack/defcore/tree/2016.01.json#n1036 needs to be removed from the standard. QA is not going to fix the bug and will drop it
16:10:59 <brunssen> OK, that will be easy to manage.
16:10:59 <hogepodge> that's all
16:11:11 <eglute> thank you brunssen!
16:11:33 <brunssen> You are welcome
16:11:39 <eglute> hogepodge can you submit PR?
16:12:23 * markvoelker_ can if hogepodge is busy, I have a small window before my next meeting
16:12:24 <w00tburger> cripes, ive been booted out of 4 other channels. seems no one knows lol
16:12:47 <eglute> w00tburger this is a meeting channel, try just #openstack
16:12:52 <w00tburger> ty
16:13:08 <eglute> markvoelker_ would appreciate it!
16:13:08 <dwalleck> hogepodge: That's bizzare. That test works, just not guaranteed a soft reboot on devstack
16:13:49 <eglute> we did have it flagged, so there must have been an issue
16:13:50 <hogepodge> dwalleck: it's been flagged for a while, that bug won't be fixed, and "it's not testing what we think it's testing"
16:14:51 <dwalleck> hogepodge: Only in devstack does in not. In a full environment, which anyone would have for defcore, it works correctly. But that's a different discussion for me to have with QA :)
16:15:27 <catherineD> dwalleck: the issue is no real  checking that the VM is actually soft reboot .... it can be hard reboot
16:15:48 <eglute> right... we can take a look at adding another test for soft reboot for next cycle
16:16:00 <dwalleck> catherineD: There is. The instance actions API will tell you what action was taken
16:16:17 <dwalleck> Any soft reboot test would rejected for the same reasoning you've mentioned
16:17:05 <dwalleck> sorry, soapbox put away :-)
16:17:33 <eglute> dwalleck do you think there is a better test instead of that one for soft reboot?
16:18:18 * markvoelker_ has been typing up a patch during this discussion: https://review.openstack.org/270294
16:18:35 <dwalleck> eglute:  The issue is that if the hypervisor used in the test environment doesn't support soft reboot, it quietly falls back to hard reboot
16:18:56 * eglute thinkgs markvoelker_ has superpowers. https://review.openstack.org/#/c/270294/
16:19:08 <dwalleck> So essentially any soft reboot test would stumble through this
16:19:14 <hogepodge> good spot for disussion.
16:19:19 <eglute> dwalleck ok, so sounds like a bigger issue
16:19:29 <hogepodge> Sorry to hijack, and thanks dwalleck markvoelker_ and catherineD
16:19:45 <dwalleck> hogepodge: no problem!
16:19:51 <eglute> #topic 2016.01 guideline
16:20:14 <eglute> we can set the topic back. I rather have a good guideline that try to fix it later :)
16:20:17 <catherineD> eglute: the issue is are we ready to take the test non-tempest test
16:20:43 <eglute> catherineD what do you mean?
16:21:37 <catherineD> the soft-reboot test is skipped in Tempest ... are we willing to unskipped it in DefCore test?
16:22:04 <eglute> no... if it is skipped in tempest, we should not have it as required in DefCore
16:22:27 <catherineD> eglute: ++ thx
16:22:39 <hogepodge> It's been a flagged test since the very beginning. We've talked about how flags need to be removed for some reason, or the test removed. I'm not willing to un-flag a permanent skip test
16:22:59 <eglute> hogepodge i agree with you.... thank you for catching this
16:23:25 <eglute> after the meeting i will review https://review.openstack.org/#/c/270294/
16:23:30 <eglute> everyone please do the same
16:23:51 <eglute> any other concerns with 2016.01?
16:24:00 * markvoelker_ notes that needs to be removed from .next to and will do that shortly
16:24:08 <eglute> thank you markvoelker_
16:25:14 <eglute> if anyone notices any other issues with 2016.01, please let us know, either in defcore channel or email
16:25:23 * markvoelker_ submits patchset 2
16:25:31 <eglute> #topic Introduce data driven testing
16:26:06 <eglute> hogepodge can you give a brief overview of the issue? i put down your notes in the agenda
16:26:19 <eglute> #link https://review.openstack.org/#/c/223953/
16:27:40 <hogepodge> QA is looking at updating their tests
16:27:53 <hogepodge> using a library that can automate what was formerly done in for loops
16:28:10 <hogepodge> it can simplify the actual test code, make it easier to extend to edge cases
16:28:39 <hogepodge> at the cost of automatically generating a whole new set of tests with mangled names based on the source test name
16:29:07 <hogepodge> Some of the interop tests would be impacted by this change, which means we would have to figure out how to identify and check for the tests.
16:29:49 <hogepodge> catherineD says it's technically possible to do this in refstack, but QA was worried about the user interface issues that could come from this, particularly in identifying which tests had run and debugging those tests
16:30:03 <dwalleck> hogepodge: Is your key concern the generated name or the fact that those tests would not have a unique id?
16:30:32 <purp> imho, this strengthens the argument for separating DefCore/RefStack testing from the mainline testing.
16:30:32 <hogepodge> a big concern for me is how do we know what test names should be generated so we can check for all of them, and how do we account for this in potential updates to tempest that change the data range parameters
16:31:15 <eglute> also, this would break the existing guidelines since right now we dont pin to a tempest version
16:31:17 <hogepodge> dwalleck: my main concern is how an outside observer would know what all the generated tests should be
16:31:24 <purp> hogepodge +1. If they constrain the test generation to deterministic naming/ids, I suspect things get bery complex.
16:31:40 <dwalleck> There are data driven testing approaches that let you create sane, deterministic names. I also think by tagging each data set with an id, you could get exactly what we have now
16:31:51 <dwalleck> It's more work, but doable
16:32:35 <hogepodge> It's mostly just a process and tooling problem, but one we need to be aware of. QA isn't even sure if they want to use this approach.
16:32:42 <hogepodge> mtreinish could offer more insight
16:32:44 <eglute> anyone know what the timeline for this blueprint would be? https://review.openstack.org/#/c/223953/
16:32:53 <purp> dwalleck I understand that, and am more concerned about whether the effort saved in generating for loop'd tests merits the additional complexity
16:33:18 <hogepodge> purp: that's the question on everybody's mind
16:33:21 <catherineD> hogepodge: for the interface concern , RefStack would just handle it similar to the aliases
16:33:21 <eglute> if it is pretty far out, we could have this as a midcycle topic
16:33:42 <dwalleck> purp: It depends on the use case. To be honest, in the PR hodgepodge is referencing, it's saving some code but not a lot
16:33:52 <hogepodge> catherineD: we only send passed test results, how would refstack know if a vendor passed _all_ the tests associated with an id?
16:34:30 <catherineD> hogepodge: in that case we will have to send the full test method new name with parameters ..
16:34:31 <dwalleck> But data driven testing has been a large part of my large scale tests for OpenStack
16:34:54 <dwalleck> I think the concept is worth addressing
16:35:25 <eglute> I agree.. the question is how fast we need to address it
16:35:28 <catherineD> hogepodge: the passing status testing will done at the server side .... on the test side all pass tests will be collected ...
16:35:30 <eglute> topic for midcycle?
16:35:49 <dwalleck> eglute: ++ to talking about it at midcycle
16:36:04 <purp> hogepodge dwalleck makes sense. Also ++ for midcycle
16:36:20 <eglute> #action eglute to add https://review.openstack.org/#/c/223953/ and related discussion to the midcycle agenda
16:36:22 <dwalleck> I think this is much easier discussed with examples
16:36:33 <eglute> #topic RefStack requirements
16:36:41 <eglute> #link https://docs.google.com/document/d/1s_dAIuluztlCC6AZ-WO4_FR2CLje1QyS6kbMxlFHaMk/edit
16:38:18 <eglute> catherineD i have left a few comments there, mostly about using "guidelines" terminology. Alexandre Levine has responded to them
16:38:18 <catherineD> eglute: I just put a link of question list on the agenda ... could we discuss here?
16:39:07 <eglute> #link https://etherpad.openstack.org/p/mitaka-refstack-requirements
16:39:28 <catherineD> eglute: yup I see ... that is the reason I bring this requirement to the attention of DefCore...
16:40:00 <mtreinish> purp: right, that's the debate which is happening on the review right now :)
16:40:02 <catherineD> the requirement suggests to include non-DefCore guidelines
16:40:28 <eglute> my main concern is that guidelines gets confusing. I get what you are trying to do, I think, but then it sounds in the requirements that eveyvendor testing would also be creating own guidelines
16:40:42 <purp> mtreinish am reading to catch up, and definitely see that. I'll catch up with you separately on this.
16:41:19 <eglute> catherineD perhaps just language changes, and using association of guidelines and vendors in some way and guidelines ownership as separate
16:41:36 <catherineD> agree ... also even that we allow non-DefCore guidelines (criteria) .. do we want to host these features on the refstack.openstack.org site?
16:42:10 <catherineD> eglute: I think this is a good topic for mid-cycle meeting
16:42:24 <eglute> midcycle works for me!
16:42:53 <eglute> other comments on refstack requirements that people want to discuss here rather than in etherpad or google doc?
16:43:10 <catherineD> Meanwhile there are other aspect that RefStack can implement now ... those are vendor registration related ..
16:43:32 <eglute> catherineD I agree! lots of good stuff
16:43:48 <catherineD> could we quickly go through the 4 questions in  https://etherpad.openstack.org/p/mitaka-refstack-requirements
16:44:15 <eglute> what kind of audit do you have in mind?
16:44:27 <eglute> oh, user info.
16:44:38 <hogepodge> eglute: I like that vendors can have their own guidelines. I also would like DefCore to have special dispensation in that world.
16:44:45 <eglute> i think if user is logged in, it should be logged, even if it is private logs
16:45:13 <eglute> hogepodge i dont disagree with that, the issue is how requirements are worded it is rather confusing
16:45:21 <hogepodge> for audit I'd like to have tempest version and testrepository file
16:45:45 <catherineD> for auditing, we wont be able to implement a complete audit feature ..
16:46:04 <eglute> catherineD how about what hogepodge is suggesting?
16:46:17 <hogepodge> We are adding language to the license agreement to say "we would want the right, but not the obligation, to audit their results.  This way it’s there but we don’t have a compulsory obligation to do it."
16:46:48 <catherineD> currently we have none ... I was thinking we at least have to log the last action and the name of the user who perform the action ...
16:46:50 <eglute> hogepodge i like that
16:47:01 <eglute> catherineD that would be good start
16:47:21 <catherineD> I just  want to check that DefCore wants us to at least start with that ...
16:47:31 <catherineD> vs we have none at the moment
16:47:45 <hogepodge> this would means stating outright that vendors absolutely need to capture the .testrepository file associated with the run.
16:48:33 <markvoelker_> catherineD: I think taking a step in that direction now and implementing more incrementally is fine
16:48:37 <catherineD> What hogepodge: is referring to is the testing aspect auditting which we have not discussed .. and can table to do that
16:49:04 <dwalleck> hogepodge: That sounds reasonable
16:49:43 <eglute> for the sake of time, lets put additional comments in https://etherpad.openstack.org/p/mitaka-refstack-requirements
16:49:48 <catherineD> markvoelker_: a few member in RefStack thinks that we do not need that at this phase ... personally, I think we should ... I just want to confirm with DefCore that this is what we should start with
16:50:05 <markvoelker_> catherineD: I'm ++ on it. =)
16:50:08 <eglute> catherineD i think some logging would be really good
16:50:23 <catherineD> eglute: Ok... we can also discuss over in #openstack-defcore
16:50:40 <eglute> thank you catherineD
16:50:56 <eglute> #topic Follow up on multi-tenant testing
16:51:08 <eglute> hogepodge have you had a change to look at https://review.openstack.org/#/c/253138/
16:51:10 <hogepodge> I haven't done my follow-up homework for that.
16:51:14 <catherineD> RefStack team  appreciates any comments on  https://etherpad.openstack.org/p/mitaka-refstack-requirements
16:51:27 <eglute> is this an issue you want  to save for midcycle?
16:52:00 <hogepodge> yeah, that would be good. I'm about to be spottily available for a week and a half because of travel obligations.
16:52:14 <eglute> cool, that will work!
16:52:29 <eglute> #action eglute to add https://review.openstack.org/#/c/253138/ to midcycle
16:52:35 <hogepodge> (I'll be in Austin starting tomorrow and this weekend if anyone wants to get lunch or dinner with me)
16:52:38 <eglute> #topic NIA Interoperability Results for NFV
16:52:45 <eglute> I am not sure who added this topic?
16:52:48 <hogepodge> I added that
16:52:55 <eglute> ah ok
16:53:14 <hogepodge> NFV is one of the big growth areas where interop can have an important impact
16:53:33 <hogepodge> The report is worth looking over to see the interop challenges they faced.
16:53:56 <hogepodge> I can post to the mailing list too, but I wanted to make the committee aware of it for review.
16:54:17 <eglute> are there any major takeaways you can quickly share?
16:54:21 <markvoelker_> hogepodge: I was just looking over that report this morning as well.  Sort of speaks to the flavors idea we've discussed before.
16:54:25 <eglute> but yes, mailing list would also be good
16:54:51 <hogepodge> The short of it is they have to tune NFV installations for the flavor of OpenStack that's installed
16:56:18 <eglute> they dont seem to mention defcore
16:56:27 <eglute> do you think defcore would help them
16:57:21 <hogepodge> If I recall correctly I think they mention it in a roundabout way. Not by name though.
16:57:37 <eglute> right, the search didnt bring it up
16:57:49 <markvoelker_> eglute: Well, something like DefCore might conceivably help.  However they're looking at some fairly specific areas that might not work well with general compute clouds, for example
16:57:56 <markvoelker_> And have concerns beyond just which API's are available
16:57:59 <markvoelker_> etc etc etc
16:58:07 <markvoelker_> So, it's more nuanced.
16:58:16 <eglute> thanks markvoelker_! i need to read this report :)
16:58:21 <hogepodge> yeah, they're looking at hardware level tuning at some points
16:58:44 <eglute> hardware specific is hard
16:59:00 <eglute> defcore does not address hardware at all
16:59:01 <markvoelker_> hogepodge: Yeah, I think one takeaway here is that it's one more voice toward the idea that interoperability standards should go further than the API.
16:59:10 <markvoelker_> (which also came up during the TC discussion recently)
16:59:32 * markvoelker_ looks at the clock and sees we're about out of time
16:59:51 <hogepodge> I thought cloud was supposed to make the hardware irrelevant ;-)
16:59:55 <eglute> we are out of time. thank you everyone! I will be available in #openstack-defcore or directly
17:00:10 <eglute> metal cloud is complicated!
17:00:22 <eglute> thanks everyone!
17:00:24 <eglute> #endmeeting