01:02:37 <markvoelker> #startmeeting DefCore
01:02:38 <openstack> Meeting started Thu Jun 11 01:02:37 2015 UTC and is due to finish in 60 minutes.  The chair is markvoelker. Information about MeetBot at http://wiki.debian.org/MeetBot.
01:02:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
01:02:41 <openstack> The meeting name has been set to 'defcore'
01:02:48 <purp> o/
01:02:52 <markvoelker> #chair eglute
01:02:53 <openstack> Current chairs: eglute markvoelker
01:02:56 <hughhalf> .
01:02:56 <zehicle> o/
01:03:04 <markvoelker> #chair zehicle
01:03:04 <openstack> Current chairs: eglute markvoelker zehicle
01:03:05 <eglute> thanks, it didnt like me then.
01:03:33 <zehicle> #link https://etherpad.openstack.org/p/DefCoreFlag.3
01:03:34 <markvoelker> no problem....I'll turn it over to you to drive now. =)
01:03:40 <eglute> thank you markvoelker
01:03:49 <zehicle> roll call pleasde
01:04:11 <eglute> o/
01:04:12 <zehicle> please indicate your time zone also, I'd like to make sure that we're accomplishing our objective
01:04:16 <zehicle> o/  Central
01:04:20 <eglute> o/ CST
01:04:23 <markvoelker> o/ EDT
01:04:34 <dwalleck> o/ CST
01:04:38 <hughhalf> o/ AEST - GMT+10 (Canberra Australia)
01:04:45 <hogepodge> o/ PDT
01:05:03 <purp> PDT
01:05:13 <purp> o/ PDT
01:05:27 <eglute> someone emailed the list and said this time does not work for him, he is in GMT +12
01:05:35 <zehicle> yy, that's Robert
01:05:44 <eglute> y
01:06:04 <zehicle> ok, I'm thinking to follow the agenda on the ether pad
01:06:27 <zehicle> please make additions/changes there and we'll incorporate it
01:06:38 <eglute> sounds good
01:06:42 <zehicle> #topic mid-cycle meeting
01:07:44 <zehicle> I'd like to have use close (or have a plan to close) the location & timing
01:07:51 <zehicle> do we have quorum to do that here?
01:07:54 <purp> Checked in with mtreinish; he's looking at late July/early August. Trying to sync with infra midcycle
01:08:27 <zehicle> We've got some block out times: OSCON & OpenStack BoD
01:08:32 <eglute> zehicle regarding quorum, we had some people last week that said they would travel, some are not here right now
01:08:38 <zehicle> yy
01:08:55 <zehicle> I suspect that we cannot close it tonight.  if so, can we get it down to two choices?
01:09:04 <zehicle> maybe 3
01:09:11 <purp> Seems reasonable.
01:09:20 <eglute> my concern with waiting too long is that then we are at timeline July 27 (Summit - 3 months)
01:09:24 * zehicle gets his calendar out
01:09:53 <eglute> BoD is on July 28th (Tuesday), oscon week before that
01:10:13 <purp> Week of July 6?
01:10:18 <purp> Too early?
01:10:19 <zehicle> I'm on vacation th week after
01:10:26 <zehicle> and traveling the week of July 6
01:10:47 <zehicle> So, 7/13 week
01:10:57 <zehicle> Could we plan it like this:
01:11:02 <zehicle> 1) SJC week of 7/13
01:11:15 <zehicle> 2) PDX week of 7/20
01:11:24 <zehicle> 3) ATX week of 7/27
01:11:36 <zehicle> with a target dates of Wed-Thurs
01:11:40 <eglute> works for me.
01:11:48 <purp> I can't make 13 Jul - 13 Aug. But we knew that.
01:12:01 <eglute> so doodle to the mailing list with the dates?
01:12:11 <zehicle> that'
01:12:31 <eglute> #action eglute send out doodle to the mailing list with date options
01:13:14 <zehicle> #topic v1.3 schema
01:13:40 <zehicle> so we merged the v1.3 schema with some discussion still potentially open
01:14:02 <eglute> yes, i think there were some things that were missed.
01:14:14 <zehicle> markvoelker, were you OK w/ the latest HACKING or did we jump the gun?
01:14:23 <eglute> lots of comments on  https://review.openstack.org/#/c/185158/, so if something was missed, please submit a new patch
01:14:41 <hogepodge> #link  https://review.openstack.org/#/c/185158/
01:14:52 <markvoelker> So, I think the thing we may have overlooked was the bit where we removed the ability to drop tests completely...
01:15:08 <eglute> i think there was at least one thing missed, need to find the new patch that was created that called it out
01:15:10 <eglute> yeah
01:15:27 <zehicle> It looks like we've started the discussion on the new patch
01:15:28 <eglute> #link https://review.openstack.org/#/c/189961/
01:15:39 <eglute> ^^ new patch
01:15:40 <markvoelker> I wasn't sure if that was intentional or not though
01:16:02 <eglute> not intentional!
01:16:04 <hogepodge> yup, we need to be able to drop tests from next. I had proposed a change to do that, but it didn't make it in.
01:16:31 <hogepodge> which is part of the point of next, to resolve outstanding flags.
01:16:32 <markvoelker> Ok, so that should be relatively easy to correct
01:16:58 <markvoelker> Shall I submit a patch for it, or does someone already have one in the works?
01:17:01 <eglute> the previous patch ended up with too many comments and i could not tell that something was still not resolved. so yes, we need to fix it
01:17:25 <hogepodge> markvoelker: you should feel free to do that
01:17:39 <eglute> #action markvoelker submit a patch to fix process for removing tests
01:17:42 <hogepodge> I don't have one in the works that's ready to go
01:17:43 <zehicle> I'd like to make sure that the patch w/ the flags stays about flags
01:17:50 <eglute> +1
01:17:58 <zehicle> do we have tests that need to be removed?  where did they go?
01:18:13 <markvoelker> zehicle: Yes, see https://review.openstack.org/#/c/189961/
01:18:15 <hogepodge> zehicle: yes, there are tests that are hypervisor specific
01:18:43 <zehicle> soundn't we keep the tests but flag them?
01:18:54 <hogepodge> and this https://review.openstack.org/#/c/189979/
01:19:04 <hogepodge> In next? No.
01:19:05 <zehicle> I thought that was the purpose of the flags?
01:19:09 <eglute> i think the hypervisor one should be dropped, based on the discussions
01:19:13 <zehicle> I'm worried about having tests in the system that don't show up
01:19:27 <markvoelker> zehicle: these are tests for capabilities that never should have been DefCore in the first place b/c they fail Core Criteria
01:19:38 <zehicle> +1 on that
01:19:39 <markvoelker> So we could keep them...forever...
01:19:45 <zehicle> except that they would be flags
01:19:45 <hogepodge> My understanding is that a required test is something we want, and that the actions on a flag are to fix or remove.
01:19:50 <markvoelker> But that would just create clutter and we'd have to re-evaluate them every cycle
01:19:59 <markvoelker> Whcih seems like a waste of resources
01:20:04 <zehicle> not if the flag reason was clear
01:20:11 <zehicle> OR... could we put them into a capability?
01:20:25 <zehicle> I'm worried that we're going to have them pop up every cycle for review
01:20:35 <eglute> in hypervisor case, it would not meet the common use case
01:20:38 <zehicle> I'd rather make a decision about them one time and let it ride
01:20:45 <hogepodge> confusing to end users. I've been telling vendors that a flag means a capability is required, but has a problematic test.
01:21:03 <zehicle> that still seems like a "compute-hypervisor-vendor-specific" capability
01:21:12 <markvoelker> zehicle: so we had recently decided that we were going to re-evaluate all flagged tests at the beginning of every cycle.
01:21:12 <zehicle> then the cap is NOT required
01:21:13 <hogepodge> zehicle: if that's the case I misunderstood the purpose then
01:21:13 <purp> This seems to point up the need for multiple flag types. No?
01:21:25 <zehicle> I'
01:21:27 <markvoelker> So if we just leave them in the flag list indefinitely we're going to be re-evaling them indefinitely
01:21:40 <markvoelker> And that list is going to get longer and longer
01:21:42 <zehicle> I'm happy to discuss.... my understanding was that we'd eventually list _every_ test
01:21:43 <hogepodge> zehicle: I thought that a capability couldn't be removed in a cycle, but could in the next.
01:21:58 <zehicle> hogepodge, so we can flag them and move them
01:22:10 <hogepodge> zehicle: but it shouldn't be in the required capability list
01:22:21 <markvoelker> zehicle: I don't think it's feasible to list every test.  That's a moving target.
01:22:25 <dwalleck> Does it make sense to have capabilities that are driver specific? As in, if you are using a certain driver, within that context the expected behaviors are <things>
01:22:28 <hogepodge> zehicle: moving I'm ok with too
01:22:31 <markvoelker> Especially when you consider non-tempest tests
01:22:58 <zehicle> markvoelker, eventually, it would be good to have them all tracked somehow since we want the data from all of them
01:23:23 <zehicle> dwalleck, we can have all sorts of capabilities.  it's helpful to users to map functionality
01:23:24 <markvoelker> zehicle: I'm saying that eventually will never come though....the tests are moving much faster than we are.
01:23:27 <zehicle> we just don't have to require them
01:23:36 <hogepodge> we have a mechanism for listing all of the tests, it's the repository the test comes from
01:24:10 <eglute> is this a topic we should leave for mid-cycle?
01:24:16 <zehicle> I'm OK w/ this suggestion - I have some concerns based on earlier postures
01:24:49 <zehicle> if we're thinking it's not a big deal then I'm OK to remove
01:24:58 <zehicle> since we have a lot of tests that are not covered
01:25:09 <zehicle> I just want to make sure that we don't keep trying to re-add them
01:25:10 <eglute> i think for now, i vote +1 to remove
01:25:30 <markvoelker> So, how about this: I'll just propose the patch and we can discuss it there.  I'm happy to be overruled if folks think differently
01:25:44 <purp> markvoelker: +1
01:25:48 <eglute> works for me
01:25:50 <markvoelker> (in fact, I thought I had been overruled when the schema change with the procedure change landed! =p)
01:26:01 <markvoelker> #action markvoelker to propose patch for test removal
01:26:06 <zehicle> no, we wanted to split the discussion
01:26:16 <zehicle> that's why we want to make sure to cover it here
01:26:32 <zehicle> because we suspected that the non v1.3 issues had not been resolved
01:26:40 <eglute> +1 on splitting discussions in patches as appropriate. otherwise important issues get lost
01:27:02 <zehicle> based on discussion here, it seems like v1.3 is OK
01:27:09 <markvoelker> I'll get that patch up tomorrow morning so we can get moving on it and have decision soonish so Chris's patches can move forward or be refactored
01:27:16 <zehicle> and we're aware that some of the issues in that patch need further discussion
01:27:24 <eglute> yes
01:27:32 <zehicle> mission accomplished
01:27:53 <zehicle> other discussion on v1.3 patch and related skeltons unearthed?
01:28:19 * zehicle thinks we unblocked a lot of good discussion around flagging
01:28:20 <hogepodge> need a proper validating schema?
01:28:20 <eglute> the skeletons might be hiding in the next topic
01:28:32 <hogepodge> Also, gate job to ensure json is parsable?
01:28:37 <hogepodge> and correct?
01:28:42 <zehicle> #topic capabilities subdivisions
01:28:54 <zehicle> hogepodge, +1
01:29:12 <zehicle> getting more critical w/ each addition
01:29:53 <zehicle> Van was not going to make this meeting
01:30:01 <zehicle> I had two items related
01:30:10 <eglute> right, i think we skipped this topic last week as well
01:30:12 <zehicle> 1) who/when can we get the capabilities subsets done
01:30:15 <zehicle> ?
01:30:21 <zehicle> does someone have that list?
01:30:39 <hogepodge> It's in a google sheet that I'm sure I can dig up a link for given enough time.
01:30:55 <zehicle> that has the correct subdivisions?
01:31:02 <zehicle> hogepodge, will doing that break all your patches?
01:31:02 <hogepodge> Van and Catherine have been the primary contributors, and both aren't here (Catherine should be back next week)
01:31:14 <eglute> #link https://docs.google.com/spreadsheets/d/15Fkt2n95WPPe7CYH-0WIKz3dhMeW3FW2gl5aSlamEeY/edit#gid=6
01:31:26 <eglute> right, i think Van will be as well
01:31:43 <hogepodge> zehicle: I'm not worried about the patches, they can be rewritten if the schema updates. I wanted to reclassify the tests for discussion, and figured those patches would be long lived and require rebasing
01:32:19 <zehicle> ok
01:32:48 <eglute> so skip this topic for now?
01:32:50 <zehicle> so, can we get that subdivision done?
01:33:15 <hogepodge> It's probably a matter of automation based on the list, so yes?
01:33:15 <zehicle> I'm worried about it causing downstream work if we delay
01:33:33 <zehicle> if you've got the material in that format
01:33:44 <zehicle> it's really not that much data to tweak
01:34:01 <zehicle> I'm thinking about 30 minutes tops
01:34:13 <zehicle> so, likely less effort than automation
01:34:39 <zehicle> the cap list is not the question - it's the test membership
01:34:49 <zehicle> that's what was missing from the earlier patch
01:35:00 <eglute> hogepodge is this something you can work on with Van and Catherine?
01:35:12 <hogepodge> I can
01:35:21 <eglute> thank you hogepodge
01:35:44 <zehicle> ok, I see the list now
01:35:51 <eglute> #action hogepodge work with Van and Catherine to subdivide capabilities
01:36:41 <zehicle> depending on the flags, they may actually be able to merge cleanly
01:37:10 <zehicle> the second part was just the we had a discussion about how to score capabilities in IRC
01:37:17 <zehicle> in the #openstack-defcore channel
01:38:01 <zehicle> the short version is that Van suggested a way to vote that may work
01:38:11 <zehicle> we'll need to try it sometime soon
01:38:18 <eglute> if everyone is looking at the spreadsheet, voting should work
01:38:19 <markvoelker> zehicle: got a link or a date when that happened?  Was probably while I was on PTO and I'd love to read...
01:38:32 <zehicle> hogepodge, are there any new capabilities under consideration?
01:38:38 <markvoelker> I can dig in eavesdrop if not
01:38:42 <eglute> markvoelker i think that happened over the phone?
01:38:50 <hogepodge> zehicle: not that I know of immediately
01:38:51 <zehicle> it was IRC
01:39:05 <zehicle> yesterday I think - around 4pm Central
01:39:32 <markvoelker> Ok, I'll dig and post a link if I can turn it up before end of topic
01:39:33 <zehicle> was a big burst of activity including patches around that time
01:39:34 <markvoelker> thanks
01:39:45 <eglute> i missed then as well
01:39:48 <dwalleck_> I'd be glad to help with capabilities as well. I should have the bandwidth
01:39:54 <zehicle> wanted to make sure that people were are we'd dicussed it - we'll need to formally doc it before we try it
01:40:00 <eglute> thank you dwalleck_ !
01:40:22 <zehicle> dwalleck_ and hogepodge I think you've got all the data you need to subdivide from the spreadsheet
01:40:39 <hogepodge> markvoelker: https://etherpad.openstack.org/p/defcore_scoring is a log of that conversation
01:40:45 <markvoelker> #link http://eavesdrop.openstack.org/irclogs/%23openstack-defcore/%23openstack-defcore.2015-06-09.log.html#t2015-06-09T20:04:03
01:40:52 <hogepodge> markvoelker: or that too :-)
01:41:00 <markvoelker> #link https://etherpad.openstack.org/p/defcore_scoring
01:41:04 <markvoelker> Thanks hogepodge
01:41:11 <purp> Thanks hogepodge
01:41:13 <zehicle> next topic....
01:41:20 <zehicle> #review hacking file (flag tests)
01:41:24 <zehicle> #topic review hacking file (flag tests)
01:41:42 <zehicle> 20 minutes remaining and this could be the rest....
01:41:45 <zehicle> other topics first?
01:42:22 * zehicle reserves last few minutes to review the choice of time
01:42:33 <eglute> i think we need to resolve this... dont think other topics will be short either
01:42:49 <purp> This is the one I missed last week, yes?
01:43:09 <eglute> yes i think so
01:43:11 <zehicle> purp, it's a long runniung thread
01:43:20 <eglute> there is a patch somewhere, let me find it
01:43:22 * purp is jealous of the stereo zehicles.
01:43:26 <markvoelker> To be clear, are we talking about this? https://review.openstack.org/#/c/188661
01:43:39 <zehicle> yy
01:43:43 <zehicle> #link https://review.openstack.org/#/c/188661/
01:43:43 <markvoelker> #link https://review.openstack.org/#/c/188661
01:43:57 <eglute> yes, thanks markvoelker
01:44:10 <markvoelker> So IMHO there's not that much that's really controversial in the patch.  I made a few suggestions in the review.
01:44:18 <zehicle> oh, can we cover #link https://review.openstack.org/#/c/182105/ first?
01:44:20 <markvoelker> The big question for me is: is the list intended to be exhaustive?
01:44:44 <markvoelker> zehicle: sure
01:44:45 <zehicle> markvoelker, I think so.  that's a good catch
01:44:53 <zehicle> ok, back to the 2015A patch for a minute
01:45:08 <markvoelker> #link https://review.openstack.org/#/c/182105/
01:45:20 <hogepodge> Oh, that's mine.
01:45:31 <zehicle> I wanted to discuss my objections and make sure that we were on the same page
01:45:47 <zehicle> yy, you were adding flag details to the 2015A process
01:46:07 <zehicle> to an extent, we did not add those details there on purpose
01:46:16 <zehicle> since they were left to DefCore to manage
01:46:33 <zehicle> I'm open to discussion that they should be part of the broader process
01:46:50 <zehicle> but it would have to be 2015B in that case and go back to the board
01:47:13 <markvoelker> Yeah, I think this one predates some of the recent changes to HACKING that came in with schema 1.3.
01:47:25 <markvoelker> Much of this is now covered there
01:47:40 <zehicle> you are correct markvoelker
01:47:48 <markvoelker> E.g. rule D307 covers what is required when flagging a test
01:47:53 <hogepodge> I'd like something to make flagging a harder thing to do
01:48:09 <hogepodge> if that's covered in hacking, fine.
01:48:24 <zehicle> that was the goal of getting written down in hacking
01:48:41 * zehicle hacking is not the most obvious name for rules of engagement
01:48:53 <markvoelker> hogepodge: D307 basically says "everything in the current schema's flagging section is required", so I think we're good there
01:49:09 <markvoelker> I'm happy to obtain different opinions though. =)
01:49:14 <hogepodge> markvoelker: I can't find what's in the schema though. I don't think it's documented
01:49:33 <markvoelker> hogepodge: https://github.com/openstack/defcore/tree/master/schema
01:49:53 <hogepodge> markvoelker: ah, ok
01:49:53 <zehicle> hogepodge, I moved it so we could track versions
01:50:08 <zehicle> since I was changing a lot of comments when I was fixing the readme
01:50:51 <zehicle> I just wanted to resolve if we felt there was a process change beyond hacking required
01:51:46 <hogepodge> hacking needs more guidance along the lines of what I wrote, but is that part of what markvoelker is going to send up as part of his action item?
01:52:19 <markvoelker> hogepodge: guidance about when a flag can be removed, or...?
01:52:22 <zehicle> I have no issues adding the rules
01:52:27 <zehicle> my concern was about where
01:52:47 <zehicle> 2015B would be the right place if we wanted it at the board level
01:53:14 <zehicle> Hacking is right if we think we can hold the line
01:53:27 <eglute> some flagging guidance should be in 2015B i think
01:53:32 <eglute> but maybe not as detailed
01:53:40 <markvoelker> zehicle: I'm of the opinion that this level of detail isn't of interest to the Board.  But then, I'm not on the Board. =)
01:53:49 <hogepodge> plus info about removing tests in next (or not)
01:53:50 <zehicle> +1
01:54:13 <zehicle> there are some items that the process leaves to defcore
01:54:24 <dwalleck_> hackings have always been a good "how do I work within this project" place to me
01:54:25 <markvoelker> hogepodge: I'll definitely hit the removal of tests in .next in my patch
01:54:26 <eglute> so sounds like everyone in favor of when to flag and what, when to remove to put hacking doc?
01:54:26 <purp> Agree.
01:54:36 <zehicle> test membership and flagging are both in that category
01:54:54 <eglute> should we create a separate flagging process doc?
01:55:04 <zehicle> I
01:55:12 <zehicle> I'm ok w/ them in hacking - since it's central
01:55:17 <eglute> ok
01:55:23 <zehicle> let's collect them there and see if we need more
01:55:23 <markvoelker> eglute: I thought about that, but came around to kind of liking it in HACKING
01:55:35 <hogepodge> five minute warning
01:55:47 <zehicle> as long as we all agree with the location, the name of the file works for me
01:55:48 <markvoelker> Basically b/c I suspect the flagging process is an area where people not routinely involved in DefCore will want to interact with us
01:55:55 <eglute> markvoelker ok, that works for me if we have consensus. most important to have them somewhere :)
01:56:04 <markvoelker> and HACKING is, as dwalleck_ pointed out, where to look for info about how to interact with a project
01:56:17 <zehicle> easy enough to xlink in the readme
01:56:29 <eglute> works for me then
01:56:34 <zehicle> ok, my topic on that patch took most of the time.  sorry
01:56:39 <purp> +1 HACKING
01:56:53 <zehicle> markvoelker, +1 on adding a "list is comprehensive" statement
01:57:10 <zehicle> we don't want flag to be allowed because we did not think of a reason to block them
01:57:25 <zehicle> if a new reason surfaces then we can discuss
01:57:35 <zehicle> except for pixie dust
01:57:43 <zehicle> not up for discussion
01:58:10 <eglute> flag neutron: because of pixie dust.
01:58:14 <zehicle> #topic is this time working?
01:58:31 <zehicle> just a check
01:58:32 <markvoelker> zehicle: I'm also curious about the difference b/t D401 and D402.  Didn't really seem to be one?
01:58:45 <purp> Hard for PDT ... family dinner hour.
01:59:09 <zehicle> did we get additional audience based on the time?
01:59:16 <eglute> no, less people
01:59:17 <purp> markvoelker: I read D401 as "inadequately tests" and D402 as "test fails inappropriately"
01:59:20 <markvoelker> time is ok for me...note that we have an AI to re-eval this in ~a month to see if it's working
01:59:27 <zehicle> dwalleck_, you are our rep from alternate time zone
01:59:36 <eglute> dwalleck_ is in CST
01:59:40 <dwalleck_> yup
01:59:42 <zehicle> oh, sorry
01:59:52 <purp> That's pretty alternate. I've been there.
01:59:59 <dwalleck_> Though the nice thing about this time is that it's not possible to conflict with anything else :)
02:00:00 <eglute> we do have someone from Australia
02:00:07 <zehicle> who was our
02:00:13 <hogepodge> hughhalf: is in australian time zone
02:00:14 <dwalleck_> well, work conflict
02:00:17 <markvoelker> purp: hrm.  I guess I read both as "the test is borked". =)
02:00:23 <zehicle> it was hughhalf
02:00:29 <eglute> hughhalf what do you think?
02:00:42 <hogepodge> AEST - GMT+10 (Canberra Australia)
02:00:43 <hughhalf> Sorry, stepped away, one mo please
02:00:55 <zehicle> I think you've answered the question
02:01:10 <hughhalf> Doorbell rang! :)
02:01:15 <purp> Heh.
02:01:16 <eglute> ok, we will try couple more times and then re-evaluate i think!
02:01:20 <zehicle> eglute, +1
02:01:27 <hogepodge> any meetings following this? please tell us to vacate if so.
02:01:27 <eglute> we agreed to give it a month
02:01:28 <hughhalf> So yes, this time is ok for me
02:01:29 <zehicle> just wanted to test it at the end
02:01:35 <zehicle> not suggesting a change
02:01:55 * hughhalf nods
02:02:00 <purp> #action purp will ping Robert to see what times work better (due before end of June)
02:02:02 <zehicle> we're done
02:02:12 <eglute> thank you everyone!
02:02:16 <markvoelker> zehicle: did you want me to propose a new patchset for https://review.openstack.org/#/c/188661/3/HACKING.rst or did you want to?
02:02:24 <zehicle> any addition discussion, we'll be on regular channel
02:02:24 * hughhalf notest that fwiw a time that will work for Robert will likely work for most Oz folk too.
02:02:25 <hughhalf> thanks all
02:02:37 <zehicle> markvoelker, go ahead
02:02:41 <zehicle> I was just getting it started
02:02:48 <zehicle> #endmeeting