19:00:12 <catherineD> #startmeeting refstack
19:00:12 <openstack> Meeting started Tue Jun 28 19:00:12 2016 UTC and is due to finish in 60 minutes.  The chair is catherineD. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:15 <openstack> The meeting name has been set to 'refstack'
19:01:08 <pvaneck> o/
19:02:20 <sslypushenko> o/
19:02:28 <rockyg> o/
19:02:35 <catherineD> hello everyone
19:02:44 <catherineD> #link meeting agenda and notes, please feel free to add items https://etherpad.openstack.org/p/refstack-meeting-16-06-28
19:03:54 <catherineD> #topic Test results ownership
19:04:18 <catherineD> #link     Ownership of the test results  (  https://blueprints.launchpad.net/refstack/+spec/test-results-ownership  )
19:04:37 <catherineD> #link     Mark a test record as �used for OpenStack certification� (  https://blueprints.launchpad.net/refstack/+spec/certification-test-record )
19:05:28 <catherineD> #link     Specification to associate test results to products. (  https://review.openstack.org/#/c/332260/ )
19:06:04 <catherineD> There is a lot of discussion in https://review.openstack.org/#/c/332260/
19:06:33 <catherineD> could you please take a look
19:08:32 <catherineD> we can discuss here or you can just add you comment to the blueprint or the patch ... I know it is a lot to read at one time
19:10:25 <sslypushenko> catherineD:  I'd say really *a lot*
19:10:40 <catherineD> sslypushenko: I know ... let's review off line?
19:11:02 <sslypushenko> maybe we can move on now and return later
19:11:09 <rockyg> works for me.....
19:11:11 <catherineD> ok
19:11:35 <rockyg> let's make sure hogepodge comments on this, though.
19:11:36 <hogepodge> o/
19:11:41 <catherineD> the agenda is kind of light today
19:11:46 <sslypushenko> or even schedule some mid week discussion in #refstack channel
19:11:48 <rockyg> And maybe markvoelker
19:11:54 <andrey-mp> o/
19:11:57 <catherineD> hogepodge: hi
19:12:01 <hogepodge> I have a topic for open discussion too
19:12:21 <catherineD> hogepodge: could you add that to the agenda https://etherpad.openstack.org/p/refstack-meeting-16-06-28
19:13:55 <sslypushenko> hogepodge:  o/
19:14:05 <catherineD> schedule a midweek meeting seems like a good idea ... I really want to discuss the ownership transfer topic so that we proceed to coding ... I want to get feedback with live demo at defcore meeting
19:14:26 <catherineD> How is everyone's schedule on Thursday?
19:15:01 <rockyg> I'm off Thursday.
19:15:11 <catherineD> oh
19:15:19 <rockyg> And Friday.
19:15:29 <catherineD> how about Wed?  which is tomorrow
19:16:02 <rockyg> works for me
19:16:40 <catherineD> hogepodge: sslypushenko:  pvaneck: andrey-mp: how about you?
19:17:02 <pvaneck> whenever really
19:17:23 <hogepodge> I'm generally free this week, especially after 12:30 tomorrow (PST) or Thursday
19:18:02 <catherineD> how about meet at #refstack at 19:30 UTC on Wednesday?  Please +1 or -1
19:18:20 <catherineD> Wednesday June 29, 2016
19:18:27 <andrey-mp> wednesday and thursday works for me
19:18:35 <rockyg> =1
19:18:43 <sslypushenko> catherineD:  +1 for Wednesday
19:18:49 <rockyg> +1
19:18:57 <andrey-mp> what time?
19:19:10 <andrey-mp> 19:00 UTC ?
19:19:17 <andrey-mp> I
19:19:21 <catherineD> 19:30 which is like at this time tomorrow
19:19:28 <andrey-mp> I'll ask Alex what day he prefer
19:19:42 <catherineD> hogepodge: is only available after 12:30 PST
19:19:44 <hogepodge> Can't do that time
19:20:02 <hogepodge> I could swing 20:00
19:20:20 <hogepodge> I mean 18:00
19:20:20 <catherineD> hogepodge: what is the best time for you tomorrow?
19:20:53 <catherineD> let's vote again to meet at 18:00 UTC on Wed (June 29) at #refstack
19:20:54 <hogepodge> 18:00 UTC, or after 22:00 utc are ideal for tomorrow
19:21:25 <andrey-mp> after 22:00 UTC we will sleep )
19:22:11 <sslypushenko> 18:00 UTC works for me
19:23:38 <catherineD> I take that everyone is OK for us to meet 18:00 UTC tomorrow
19:23:50 <catherineD> andrey-mp: could you please let Alex know
19:24:03 <pvaneck> +1 works for me
19:24:32 <catherineD> OK I will send an invitation ... thx everyone
19:24:42 <catherineD> moving on
19:24:58 <catherineD> #topic Pending reviews
19:26:29 <catherineD> #link     Replace run_tempest.sh script with ostestr command  (  https://review.openstack.org/#/c/329691/ )
19:26:52 <catherineD> Luz just made an other update ... please review
19:26:55 <andrey-mp> catherineD: Alex can meet at 17:00 UTC and don't know about 18:00
19:27:58 <catherineD> hogepodge: does 17:00 works for you?
19:28:37 <hogepodge> No, I have a WG meeting
19:30:46 <catherineD> andrey-mp: Please check with Alex....the other time slot is 22 which I think is late for andrey-mp: sslypushenko: and Alex
19:31:31 <catherineD> so we will meet at 18:00 tomorrow ...
19:31:40 <catherineD> anyother question?
19:31:57 <catherineD> moving on ..
19:32:08 <catherineD> #topic Open discussion
19:32:35 <catherineD> let's discuss item 3.2 Optionally upload subunit files for Foundation evaluation
19:32:58 <sslypushenko> yeap
19:33:10 <hogepodge> It's looking like at some point we're going to need a way to get subunit files
19:33:53 <sslypushenko> hogepodge:  that is sounds strange... a bit
19:34:00 <hogepodge> Generally for special cases (like the waiver program we're looking at in DefCore) or when disputes arise.
19:34:12 <hogepodge> json files are way to easy to forge
19:34:31 <hogepodge> and we may need to run scripts on some results to parse information
19:34:44 <hogepodge> subunit shouldn't be publicly available though
19:34:55 <hogepodge> but having a record in one place would be nice
19:35:11 <hogepodge> I can do without the feature, but I'd prefer centralized storage of test results
19:36:39 <catherineD> The reason from the beginning is privacy concern ..
19:37:14 <sslypushenko> hogepodge:  subunit files can contain some privacy data
19:37:20 <catherineD> the subunit file may contain credential
19:37:36 <hogepodge> not any more, tempest and qa has worked hard to remove that
19:37:49 <hogepodge> mtreinish: do credentials leak into subunit files?
19:38:24 <catherineD> also at the very beginning .. the thought is to only focusing on pass test ..
19:38:28 <mtreinish> hogepodge: they should not, the only thing in subunit attachments are the logs, stdout, and stderr. If we leak passwords into the logs that's a critical bug
19:38:48 <catherineD> we do not need to know about fail tests especially the fail reason
19:39:03 <sslypushenko> hogepodge:  you need to convince not only us, but all potential RefStack users
19:39:08 <hogepodge> catherineD: that's changing, it doesn't need to be required, but optional would be nice
19:39:48 <hogepodge> sslypushenko: if vendors want to pass defcore with special cases or appeal disputes to their results, that's convincing enough
19:40:13 <catherineD> sslypushenko: +1 ... I think the submitter needs to know
19:40:24 <hogepodge> catherineD: it would be optional
19:40:51 <sslypushenko> hogepodge:  sounds reasonable, though
19:41:14 <hogepodge> I can ask for them on the side, it's just better to have information in one place if possible.
19:41:28 <catherineD> hogepodge: RefStack so far save very little confidential data ... we save no credential or any private data ..
19:42:14 <catherineD> once the system involve saving private data .. wouldn't the server become a different class of server ... that need more stringing IT policy?
19:42:24 <rockyg> So, we'd need a ui with confirmation for the subunit uploads.  You have to answer yes after selecting the upload...
19:42:28 <hogepodge> if a user configures their testing correctly the only place credentials should appear is accounts.yaml, which I definitely don't want
19:42:46 <hogepodge> catherineD: but consider that if I'm testing a cloud, I may want to save the output of my own test results for reference
19:43:05 <hogepodge> catherineD: and that's the case right now as I test public clouds independently
19:43:49 <catherineD> hogepodge: you do ... I am just taking about the responsibility of the party who store private data ...the server now becomes a different IT class server with different IT policy to abide to ..
19:43:58 <sslypushenko> hogepodge:  I'm ok with optional upload of raw subunit data...
19:44:10 <catherineD> at least that is how it work in our company
19:44:30 <sslypushenko> but we need to provide a possibility to user for review uploaded data
19:45:10 <sslypushenko> it is not a piece of cake to look thought subunit results
19:45:22 <rockyg> yeah, what catherine says is about corporate compliance and legal responsibilities
19:45:29 <catherineD> sslypushenko: once there is a way to upload data it does no matter whether it is optinal or not
19:45:38 <catherineD> rockyg: +1 exactly
19:45:56 <mtreinish> you do realize there is nothing in subunit that's not in the log file or the console output
19:46:08 <rockyg> This sounds like a midcycle agenda item.  And maybe with defcore, too.
19:46:27 <sslypushenko> rockyg: +!
19:46:48 <catherineD> mtreinish: that is right .. that is why refstack does not save log or console or subunit file ,,, just a list of test case names
19:46:58 <catherineD> rockyg: +1
19:47:45 <rockyg> But discoverin the deeper issues before midcycle is also very good.
19:47:48 <catherineD> refstack does not need to know which tests fail and the reason of failures ... we only need to know the good news which is which tests pass
19:48:08 <sslypushenko> hogepodge:  one more question... subunit files can be also forged, what your thoughts about it?
19:48:21 <hogepodge> catherineD: but that is likely going to change, especially with this additional properties business
19:48:30 <hogepodge> we need a way to evaluate why a failure happened
19:48:30 <mtreinish> sure, but that doesn't mean you're leaking anything confidential, that would be a serious bug in tempest. It just means you've never looked
19:49:04 <hogepodge> The board is favoring the waiver process, and to make it workable I need subunit data
19:50:29 <sslypushenko> got it... but what the pros of subunit on json results?
19:50:50 <hogepodge> sslypushenko: if a test fails because if additional properties on the response, that can be parsed out and evaluated
19:50:55 <hogepodge> sslypushenko: for example
19:51:34 <hogepodge> sslypushenko: also, we had a company passing the test earlier with a pass on a permanently skipped test. subunit would have given me an opportunity to evaluate if they actually ran the test (with a modified tempest) or cheated
19:52:39 <sslypushenko> I can not agree with the last statement
19:53:15 <sslypushenko> subunit tests can be also a fake
19:53:31 <catherineD> hogepodge: cheating is an issue that we discuss earlier ... and we agree that it is the customers of the vendors who will findout ..
19:53:55 <rockyg> So, this raises all sorts of legal compliance issues for protection of client confidential data
19:54:05 <hogepodge> there's nothing confidential
19:54:07 <sslypushenko> it is a bit complicated then to edit json... but it is not a good solution on the long run
19:54:14 <rockyg> subunit is confidential.
19:54:25 <hogepodge> rockyg: it's self reported, private to everyone else except the foundation, and required for administration of the program
19:54:46 <hogepodge> rockyg: why?
19:55:19 <hogepodge> rockyg: if I need it to verify trademark compliance, it's well within the rights of the foundation to demand it, as well as access to the software in question
19:55:27 <hogepodge> rockyg: or we just deny the trademark application
19:55:31 <rockyg> Because it *does* contain specific operational data.  This could be considered a company secret and handing it over might be voluntary, but custodial responsibilities change
19:55:32 <catherineD> hogepodge: so foundation will be responsible if the data on the server is leak?
19:55:42 <hogepodge> it's a matter of making the job easier and maintaining a record
19:56:18 <sslypushenko> in the perfect world, I'd like to see in RefStack a way to have some how signed test results
19:56:39 <hogepodge> catherineD: we're responsible for keeping confidential data confidential. test results for a publicly administered trademark program are hardly confidential, especially when they are reproducible by end users
19:56:41 <rockyg> One of the original reasons, beside credential leakage, we parsed subunit was so that we ensured against receiving any data a company might consider secret
19:57:09 <catherineD> rockyg: ++
19:57:15 <hogepodge> rockyg: the moment you allow someone to run tempest against your cloud all secrecy behind test results are gone
19:57:21 <hogepodge> open is open
19:58:12 <rockyg> not true.  User is not admin.  An admin running user tests may provide a different log experience than a user running the same tests
19:58:49 * catherineD looking at the clokd only 2 mins to go
19:59:14 <catherineD> I think this is a good topic for mid-cycle...
19:59:40 <catherineD> hogepodge: thanks for bringing the topic for discussion ...
19:59:57 <rockyg> This will certainly make midcycle interesting....
20:00:16 <catherineD> we need to end the meetying now .. thanks everyone !
20:00:21 <catherineD> #endmeeting