16:00:24 <eglute> #startmeeting defcore
16:00:32 <openstack> Meeting started Wed Nov 11 16:00:24 2015 UTC and is due to finish in 60 minutes.  The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:36 <openstack> The meeting name has been set to 'defcore'
16:00:55 <eglute> Good morning everyone!
16:01:13 <eglute> #topic roll call
16:01:19 <eglute> raise your hand if you are here
16:01:25 <dwalleck> o/
16:01:49 <eglute> hi dwalleck! looks like it is just the two of us here so far
16:02:49 <eglute> i think i sent so many updated calendar invites, people gave up on me
16:03:16 <Guest78752> o/
16:03:18 <eglute> #agenda https://etherpad.openstack.org/p/DefCoreRing.2
16:03:48 <dwalleck> yeah, last I saw was 10:30
16:03:56 <eglute> really?
16:03:58 <eglute> hm
16:04:20 <eglute> Guest78752 what was the latest invite that you saw
16:04:28 <eglute> hogepodge, are you around
16:05:01 <hogepodge> o/
16:05:11 <Guest78752> Guest78752 is Catherine ... Hello ... I saw one with 10:00 am
16:05:26 <eglute> Thanks Catherine!
16:05:39 * VanL thought this meeting was in openstack-meeting-2
16:05:57 <eglute> Hi VanL... I sent out multiple invites, many calendar fails
16:07:16 <eglute> I hope that this time and place works for everyone. i think we still need to update the official calendar, will do so later
16:07:50 <eglute> next quick item- if you have not yet reviewed Schema migration patches, please do so
16:07:56 <eglute> #link https://review.openstack.org/#/c/240939/
16:08:04 <eglute> #link https://review.openstack.org/#/c/240942/
16:08:41 <eglute> also, is everyone ok with the aliases in the schema?
16:08:48 <eglute> #link https://review.openstack.org/#/c/238224/
16:09:22 <dwalleck> I have to peek through those, haven't seen them yet
16:09:37 <eglute> thanks dwalleck
16:09:46 <eglute> hogepodge are you ok with aliases
16:09:51 <hogepodge> yes
16:10:18 <eglute> thanks- please review when you have a chance
16:10:24 <eglute> #topic tests
16:10:33 <eglute> hogepodge i think you added all the test agenda items
16:10:41 <hogepodge> Yes
16:10:51 <eglute> #chair hogepodge
16:10:52 <openstack> Current chairs: eglute hogepodge
16:11:09 <hogepodge> I've been trying to run the defcore tests against a public cloud as an end user might.
16:11:24 <hogepodge> One paid account, configuring tempest with the limitations that presents.
16:11:25 <eglute> which clouds have you tried
16:11:35 <kbaikov> o/
16:11:38 <hogepodge> So far just Dreamhost.
16:11:46 <rockyg> o/
16:12:34 <hogepodge> I wanted to see how many I could get passing without two user accounts and the limited resources from a basic account (I would up upgrading from free to paid so I could get two nodes).
16:13:00 <hogepodge> I could pass 30 of 112 compute tests, which isn't great.
16:13:11 <eglute> against which guideline?
16:13:26 <hogepodge> Some because of resource constraints (not enough nodes available to me)
16:13:30 <hogepodge> 2015.07
16:13:48 <dwalleck> The weird part is that I don't think any of the selected tests actually require multiple accounts, but Tempest wants them
16:13:52 <eglute> which guideline is dreamhost certified
16:13:59 <hogepodge> many because I didn't have multiple accounts
16:14:01 <hogepodge> eglute: they aren't
16:15:24 <eglute> dwalleck raises a good point about tempest
16:15:26 <hogepodge> It would be nice if we had a test suite that met one of our original goals: allowing users to test clouds independently. For dreamhost, I could easily spend $100 out of pocket right now just to get an accurate sense of how well they pass defcore tests.
16:15:43 <dwalleck> The only tests that should need multiple accounts are ones that test authorization (can I access a resource owned by someone else)
16:16:28 <hogepodge> dwalleck: I don't think tests like that have anything to do with ineroperability. They are security tests, which are important, but not what we're testing.
16:16:51 <hogepodge> I don't know if anyone things the same though.
16:16:54 <dwalleck> Hmm, then maybe those tests should be removed?
16:17:15 <dwalleck> or flagged
16:17:28 <eglute> so, we have at least two different issues: tempest requires multiple accounts when they should not be necessary and the amount of resources required to test
16:17:30 <mfisher_ora> out of curiosity, why are you running defcore against a cloud you don't have control over?
16:17:49 <hogepodge> I don't want to go removing tests without replacements or refocusing of the capabilities.
16:18:22 <hogepodge> mfisher_ora: one of the goals of defcore is for users to be able to independently verify any cloud they want to run against.
16:18:36 <eglute> hogepodge what would like to see happen? what would be the ideal situation?
16:19:00 <hogepodge> so, I've done something similar against the aptira private cloud. I did have tenant separated accounts and they were able to pass powered compute 2014.04.
16:19:20 <mfisher_ora> it shouldn't be defcore's fault that it can't run against a certain cloud because you don't want to / can't have multiple accounts on said cloud
16:19:27 <hogepodge> eglute: rewriting or replacement of tests that require too many resources
16:19:56 <eglute> so we should start with identifying those tests?
16:20:07 <hogepodge> mfisher_ora: I would argue that it's a bad standard.
16:20:27 <dwalleck> I'd be okay with helping identify what resources needed are by which tests
16:20:28 <mfisher_ora> I'm still not understanding why you would want defcore to run against 'any cloud'
16:20:30 <hogepodge> mfisher_ora: all I should need to know for interoperability is an endpoint and credentials, because as an app developer that is what I have to work with
16:21:02 <dwalleck> There's a bigger issue with resources that was talked about at the summit with the QA group
16:21:11 <rockyg> Identifying resources and tagging the tests with that info would be valuable, I would think
16:21:40 <eglute> mfisher_ora: the point of defcore is interoperability between different openstack clouds. In theory, all openstack clouds should exhibit the same behavior.
16:21:54 <rockyg> But this is just like the tests that don't need admin yet still require it
16:21:55 <hogepodge> mfisher_ora: https://github.com/openstack/defcore/blob/master/doc/source/process/CoreDefinition.rst
16:22:01 <hogepodge> "Tests can be remotely or self-administered"
16:22:41 <mfisher_ora> I'll gab with you more about it later rather than holding this up with questions
16:22:51 <rockyg> Also, with the security tests, I think they should be there, but maybe we need a second set of tests for security.  Different focus,  but important
16:23:03 <rockyg> Or group the security tests togehter.
16:23:31 <dwalleck> rockyg: There is a set of tests the security working group is working on
16:24:37 <rockyg> excellent!
16:25:19 <hogepodge> part of what I'm doing right now is figuring out which tests fail with more constraints, why, and if they can be replaced with something that tests the same capability but without the added assumptions
16:26:46 <rockyg> eglute, I think the foundation needs to give Chris a raise.  He's going far and beyond what any manager would have thought to have him do :-)
16:27:26 <eglute> hogepodge would you rather have a raise or an extra person to help with this? :D I wish I could give you and instant raise :D
16:27:52 <hogepodge> I have a couple of months. I also want to make sure it's a direction we want the committee to go in.
16:28:15 <hogepodge> Might be worthwhile holding off on more discussion for the next week. I think lots are absent because of the time change.
16:28:41 <eglute> Mark sent me an email he won't be able to attend today
16:28:47 <hogepodge> People to help would be nice. I've been given time to work upstream on this. I'm just gathering information right now.
16:28:51 <rockyg> I just want to say, this time is *much* better for me.  I have some coffee in me now!
16:29:10 <eglute> I think we all agree that tempest tests need improvement for defcore
16:29:25 <eglute> starting with one account would be good
16:30:30 <zehicle> running late
16:30:32 <eglute> hogepodge what would it take to get the defcore tests to work with one account
16:30:35 <eglute> hello zehicle
16:30:44 <SammyD> +1 improved tempest tests for defcore... :-)
16:31:16 <eglute> SammyD dwalleck is that something that you can help with?
16:31:25 <zehicle> hey
16:31:26 <hogepodge> eglute: I need to upgrade my account to allow more nodes. I also need to rewrite tests that require more than one account.
16:31:35 <rockyg> Also, categorizing the tests, as hogepodge is doing will help a lot, too.  Break the larger set into subsets
16:31:42 <dwalleck> eglute: Absolutely
16:31:49 <eglute> Would it help you to test on Rackspace Public cloud?
16:32:01 <rockyg> +1
16:32:20 <hogepodge> eglute: it would be nice to get more reference points. I'm planning on signing up for more clouds to see how far I can get with them.
16:32:36 <eglute> hogepodge I will work with you offline for accounts
16:33:07 <hogepodge> ok
16:33:14 <dwalleck> hogepodge: And I can help you with any issues if you run into them
16:33:16 <eglute> also, can you work with dwalleck and rockyg to see how they can get involved?
16:33:25 <hogepodge> thanks
16:33:38 <eglute> with helping with the tests that is
16:33:41 <dwalleck> I'm running continuous testing against our public cloud, so I've made it through the bumpy spots
16:33:56 <eglute> i think having better defcore tests will help everyone
16:34:08 <rockyg> Another advantage of clumping tests be capabilities could be that they would run faster.  Don't have to initialize capabilities that aren't going to be used until the appropriate test set
16:34:24 <SammyD> We certainly can help with the digging in and understanding what is working for what reason. I could see about dedicating some of our capacity this cycle to directly updating tempest tests for defcore. :-)
16:34:46 <eglute> thank you SammyD!
16:35:11 <eglute> #action SammyD, dwalleck, rockyg to help hogepodge with defcore tempests issues
16:35:44 <eglute> hogepodge you also have on agenda new interop tests
16:36:00 <eglute> do you have specific tests in mind that need to be written, or?
16:36:21 <hogepodge> no, that was it on interop
16:36:36 <hogepodge> the next item is about information sent to refstack
16:36:54 <rockyg> Something possibly worth looking at are the capabilities we would *want* to include but only have one test for
16:36:56 <eglute> #topic regstack results
16:37:16 <eglute> rockyg agree... can someone start a wish list for tests?
16:37:26 <catherineD> So currently on pass tests are sending to RefStack per blue print https://blueprints.launchpad.net/refstack/+spec/pass-only-uploads
16:37:44 <hogepodge> rockyg: yeah, we need to write some new tests for the caps that have only one. more upstream work
16:38:00 <rockyg> eglute, we've got a good start here.  Classification and categorization tags
16:38:40 <rockyg> hogepodge, tht would make markvoelker very happy
16:38:46 <SammyD> +1 on the wishlist...if I have something I can refer to I can try to work a certain amount of time throughout the cycle to grab the next test in the list for instance.. :-)
16:38:53 <eglute> i think we have two different topics going, sorry Catherine
16:39:14 <rockyg> OK.  once the next etherpad is created, I can start dumping some ideas
16:39:15 <hogepodge> to the topic, some folks, qa in particular, would love for refstack to be able to store subunit data, but as a committee we have stated only anonymized passing results are sent
16:39:17 <eglute> #action SammyD will work on wish list for tests with rockyg
16:39:34 <hogepodge> Guest78752 can speak more to that
16:39:54 <hogepodge> catherineD: that is
16:40:36 <catherineD> So refstack-client (which run on the client side) would sanitize the data from subunit format (which includes all test result information of pass/fail/skip .. log info) to a JSON with only pass tests and then upload to RefStack server
16:41:19 <catherineD> Upload only pass tests was implemented per blueprint https://blueprints.launchpad.net/refstack/+spec/pass-only-uploads
16:42:18 <SammyD> I only volunteered to *use* the wishlist to knock things out. I can help out some on building out the list too though. :-)
16:43:07 <eglute> #action SammyD will use tests wish list to implement tests
16:43:08 <catherineD> Current suggestion from the last RefStack IRC meeting is to send subunit data file to RefStack server.
16:45:20 <eglute> catherineD are you asking for all the data?
16:45:30 <eglute> or just for passing tests?
16:45:55 <catherineD> right now refstack-client only uploads pass data ...
16:46:09 <rockyg> My question would be is how many refstack users want this "feature"
16:46:23 <eglute> are you looking for failed test results as well? or?
16:46:56 <catherineD> personally I do not think failed tests would help DefCore ... only pass tests
16:46:59 <rockyg> We put the limits in place to protect both the vendor *and* the test runner.  Failed tests leak data into subunit
16:47:19 <mfisher_ora> well if you have failed tests posted with the reasons, and the reasons are consistent, makes for a good indicator to flag said tests
16:48:06 <eglute> catherineD what does subunit data include for passing tests?
16:48:38 <catherineD> subunit data includes all data (pass/fail/skip/log ..) log may include sensitive data ...
16:49:13 <dwalleck> stack traces and other fun data :-) It's useful in terms of testing
16:49:18 <catherineD> that is why currently we sanitize the data at the client side before sending to RefStack server ...
16:49:20 <eglute> so refstack would like all of that data uploaded?
16:49:57 <catherineD> no I don't see how  RefStack would use all data at this point ....
16:50:04 <rockyg> Maybe the client could just save the full results to a directory, or let the user name the results, including save path?
16:50:36 <rockyg> On the test runner's machine/network that is
16:50:37 <catherineD> besides subunit data file can be huge ...
16:50:58 <eglute> catherineD can you tell us what you would like to see happen?
16:51:02 <rockyg> especially with failure traces ;-)
16:51:11 <catherineD> rockyg: exactly ... all sensitive and large data should be handled at the client sid
16:51:45 <catherineD> DefCore/RefStack won't analyze fail reason ... the most we can do is statitical data anlaysis ...
16:51:59 <rockyg> I don't think refstack should change what goes to the server, but we could allow the client to have save options locally
16:52:25 <catherineD> rockyg: yup that is what being implemented right now ....
16:53:18 <catherineD> eglute: I think for the short term (next 1 to 2 releases ) sending passing data makes the most sense ...
16:53:24 <rockyg> what are the reasons given for wanting to save the files to the server?
16:54:39 <rockyg> catherineD, agree.  DefCore could revisit after  data from 2016.1 starts coming in, or there is more impetus behind a change.
16:54:39 <catherineD> the disadvantage is we will have to delay using QA subunut2sql utility .. but currently since we only parse for pass data the parsing task is relatively simple ...
16:55:06 <catherineD> rockyg: the main reason is to use subunit2sql util at this time
16:56:08 <rockyg> So, could the client use it only on client side, without passing the data on?
16:56:24 <catherineD> rockyg: yes
16:56:29 <rockyg> Let the user dump into their own csv file or db?
16:56:53 <rockyg> Then the user gets what the want/need, but don't push sensitive data
16:57:02 <catherineD> rockyg: sorry no ... client side can not use subunit2sql because the database (sql part) resides in the server side
16:57:43 <eglute> so for passing tests, is there sensitive info in subunit tests?
16:57:56 <dwalleck> Well, the subunit2sql call is just made on the stream that comes out of the runner. It would be a change to the refstack client
16:57:59 <catherineD> subunit2sql tool needs both the subunit file and the database
16:58:08 <rockyg> eglute, not supposed to be
16:58:16 <dwalleck> I'm currently dumping all my defcore test results with subunit2sql
16:58:41 <catherineD> eglute: no there is no sensitive data in the JSON of passing tests that refstack-client creates
16:59:20 <catherineD> dwalleck: do you dump it to a database?
16:59:32 <dwalleck> catherineD: Yes
16:59:43 <eglute> catherineD can you write up an email to the ML with what you are asking and have people comment/provide feedback?
16:59:55 <catherineD> and you have access to the database
17:00:15 <dwalleck> Yes, it's a database that I run. It's standalone
17:00:17 <rockyg> eglute, I think we need to clairfy that catherineD is not asking for this, but that some users are
17:01:19 <rockyg> And that what dwalleck does is maybe a clientside tool for users that would be separate
17:01:23 <zehicle> back online
17:01:30 <zehicle> been lurking
17:01:43 <rockyg> You missed it. zehicle!  You own all the action items!
17:01:44 <catherineD> dwalleck: exactly, you need both subunit file and database at your side ...
17:01:47 <eglute> ok, we are out of time, we need action on this
17:01:59 <eglute> catherineD can you write up an email with proposal?
17:02:04 <dwalleck> sorry, bouncing my attention
17:02:20 <zehicle> cool!
17:02:37 <zehicle> I thought we started :30 ago
17:02:40 <catherineD> eglute: I am not propose for any change at this time ... but I can document the request
17:02:42 <rockyg> But do we need action, other than adding it to the agenda?  We need more data on the user "wants" and whether we need to change refstack and client to meet them
17:02:51 <eglute> zehicle no, 1 hour ago. many calendar fails
17:03:00 <rockyg> zehicle, me too;-)  It would be a much better time.
17:03:01 <zehicle> ah, I could have made it at 10!
17:03:06 <catherineD> eglute: can we revisit this next week. ...
17:03:20 <catherineD> zehicle: has the background of why we only send pass data
17:03:20 <eglute> catherineD sure thing, lets re-visit next week
17:03:25 <zehicle> I'm out next week
17:03:31 <rockyg> catherineD, the etherpad is already created.  I'll toss it onto the agenda section
17:03:45 <zehicle> yy, pass data only was a board request
17:03:50 <catherineD> rockyg: thx ...
17:04:02 <zehicle> we did not want to expose "vendor broken things"
17:04:14 <eglute> right, i am fine with passing only data being sent, just was not clear on what catherineD wanted. we will re-visit
17:04:42 <zehicle> the idea being that failed tests = broken.
17:04:52 <eglute> thank you everyone! next week the meeting is at 10:00 CST, same as (almost) always
17:04:53 <zehicle> we did not want vendors to only run the required tests
17:05:05 <catherineD> eglute: I need the confirmation that DefCore wants RefStack server to accept pass data only
17:05:05 <rockyg> I really don't think it's what catherineD wants, just something that some users have requested
17:05:17 <zehicle> and that was the compromise to encourage maximizing the number of extra tests
17:05:25 <eglute> catherineD yes, pass only data for refstack
17:05:38 <zehicle> failed tests don't really give you much information
17:05:39 <catherineD> eglute: THANK YOU!!!
17:05:55 <zehicle> from an interop perspective
17:06:10 <rockyg> I think we can provide the users the request through a separate, clientside tool like dwalleck uses
17:06:33 <rockyg> And, it could even be in a contrib dir
17:07:21 <rockyg> so, do we need this on next week's agenda?  I think we are in agreement that we keep what goes to the server as is.
17:07:54 <eglute> yes, i think we are good. unless catherineD wants us to revisit
17:07:55 <catherineD> rockyg: client side is OK ..
17:08:12 <eglute> #agreed to send only pass only data to refstack
17:08:14 <rockyg> I'll remove it from agenda for 3.
17:08:19 <eglute> thanks everyone!
17:08:23 <eglute> #endmeeting