Wednesday, 2015-11-11

*** jmckind has joined #openstack-defcore00:22
*** breitz has quit IRC01:49
*** dwalleck has joined #openstack-defcore01:51
*** edmondsw has quit IRC02:04
*** jmckind has quit IRC02:10
*** dwalleck has quit IRC02:14
*** MarkAtwood has joined #openstack-defcore02:37
*** MarkAtwood has quit IRC02:54
*** MarkAtwood has joined #openstack-defcore03:47
*** hogepodge has quit IRC04:35
*** dwalleck has joined #openstack-defcore04:54
*** dwalleck has quit IRC07:05
*** catherineD has quit IRC07:43
*** catherineD has joined #openstack-defcore07:44
*** alex_klimov has joined #openstack-defcore08:21
*** alex_klimov has quit IRC09:56
*** alex_klimov has joined #openstack-defcore10:34
*** markvoelker has quit IRC10:37
*** MarkAtwood has quit IRC11:01
*** markvoelker has joined #openstack-defcore11:37
*** markvoelker has quit IRC11:42
*** MarkAtwood has joined #openstack-defcore11:47
*** MarkAtwood has quit IRC11:51
*** openstackgerrit has quit IRC12:31
*** openstackgerrit has joined #openstack-defcore12:31
*** markvoelker has joined #openstack-defcore12:53
*** markvoelker has quit IRC12:58
*** edmondsw has joined #openstack-defcore13:31
*** MarkAtwood has joined #openstack-defcore13:35
*** MarkAtwood has quit IRC13:40
*** markvoelker has joined #openstack-defcore13:54
*** markvoelker has quit IRC13:58
*** markvoelker has joined #openstack-defcore14:15
*** mfisher_ora has joined #openstack-defcore14:40
*** breitz has joined #openstack-defcore15:03
*** breitz has quit IRC15:18
*** breitz1 has joined #openstack-defcore15:18
*** breitz1 is now known as breitz15:22
*** MarkAtwood has joined #openstack-defcore15:23
*** MarkAtwood has quit IRC15:28
*** catherineD has left #openstack-defcore15:57
*** catherineD has joined #openstack-defcore15:58
*** catherine has joined #openstack-defcore15:59
*** catherine is now known as Guest7875215:59
eglute#startmeeting defcore16:00
openstackMeeting started Wed Nov 11 16:00:24 2015 UTC and is due to finish in 60 minutes.  The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
openstackThe meeting name has been set to 'defcore'16:00
egluteGood morning everyone!16:00
eglute#topic roll call16:01
*** dwalleck has joined #openstack-defcore16:01
egluteraise your hand if you are here16:01
dwallecko/16:01
eglutehi dwalleck! looks like it is just the two of us here so far16:01
*** SammyD has joined #openstack-defcore16:02
*** openstackgerrit has quit IRC16:02
*** openstackgerrit has joined #openstack-defcore16:02
eglutei think i sent so many updated calendar invites, people gave up on me16:02
Guest78752o/16:03
eglute#agenda https://etherpad.openstack.org/p/DefCoreRing.216:03
*** hogepodge has joined #openstack-defcore16:03
dwalleckyeah, last I saw was 10:3016:03
eglutereally?16:03
eglutehm16:03
egluteGuest78752 what was the latest invite that you saw16:04
eglutehogepodge, are you around16:04
hogepodgeo/16:05
Guest78752Guest78752 is Catherine ... Hello ... I saw one with 10:00 am16:05
*** VanL has joined #openstack-defcore16:05
egluteThanks Catherine!16:05
* VanL thought this meeting was in openstack-meeting-216:05
egluteHi VanL... I sent out multiple invites, many calendar fails16:05
*** jasonsb_ has joined #openstack-defcore16:07
egluteI hope that this time and place works for everyone. i think we still need to update the official calendar, will do so later16:07
eglutenext quick item- if you have not yet reviewed Schema migration patches, please do so16:07
eglute#link https://review.openstack.org/#/c/240939/16:07
eglute#link https://review.openstack.org/#/c/240942/16:08
eglutealso, is everyone ok with the aliases in the schema?16:08
eglute#link https://review.openstack.org/#/c/238224/16:08
dwalleckI have to peek through those, haven't seen them yet16:09
eglutethanks dwalleck16:09
eglutehogepodge are you ok with aliases16:09
hogepodgeyes16:09
eglutethanks- please review when you have a chance16:10
*** pbredenb1 has joined #openstack-defcore16:10
eglute#topic tests16:10
eglutehogepodge i think you added all the test agenda items16:10
hogepodgeYes16:10
eglute#chair hogepodge16:10
openstackCurrent chairs: eglute hogepodge16:10
hogepodgeI've been trying to run the defcore tests against a public cloud as an end user might.16:11
*** rockyg has joined #openstack-defcore16:11
hogepodgeOne paid account, configuring tempest with the limitations that presents.16:11
eglutewhich clouds have you tried16:11
kbaikovo/16:11
hogepodgeSo far just Dreamhost.16:11
rockygo/16:11
*** blogan has joined #openstack-defcore16:12
hogepodgeI wanted to see how many I could get passing without two user accounts and the limited resources from a basic account (I would up upgrading from free to paid so I could get two nodes).16:12
hogepodgeI could pass 30 of 112 compute tests, which isn't great.16:13
egluteagainst which guideline?16:13
*** jasonsb has joined #openstack-defcore16:13
hogepodgeSome because of resource constraints (not enough nodes available to me)16:13
hogepodge2015.0716:13
dwalleckThe weird part is that I don't think any of the selected tests actually require multiple accounts, but Tempest wants them16:13
eglutewhich guideline is dreamhost certified16:13
hogepodgemany because I didn't have multiple accounts16:13
hogepodgeeglute: they aren't16:14
eglutedwalleck raises a good point about tempest16:15
hogepodgeIt would be nice if we had a test suite that met one of our original goals: allowing users to test clouds independently. For dreamhost, I could easily spend $100 out of pocket right now just to get an accurate sense of how well they pass defcore tests.16:15
dwalleckThe only tests that should need multiple accounts are ones that test authorization (can I access a resource owned by someone else)16:15
hogepodgedwalleck: I don't think tests like that have anything to do with ineroperability. They are security tests, which are important, but not what we're testing.16:16
hogepodgeI don't know if anyone things the same though.16:16
dwalleckHmm, then maybe those tests should be removed?16:16
*** jasonsb_ has quit IRC16:17
dwalleckor flagged16:17
egluteso, we have at least two different issues: tempest requires multiple accounts when they should not be necessary and the amount of resources required to test16:17
mfisher_oraout of curiosity, why are you running defcore against a cloud you don't have control over?16:17
hogepodgeI don't want to go removing tests without replacements or refocusing of the capabilities.16:17
hogepodgemfisher_ora: one of the goals of defcore is for users to be able to independently verify any cloud they want to run against.16:18
eglutehogepodge what would like to see happen? what would be the ideal situation?16:18
hogepodgeso, I've done something similar against the aptira private cloud. I did have tenant separated accounts and they were able to pass powered compute 2014.04.16:19
mfisher_orait shouldn't be defcore's fault that it can't run against a certain cloud because you don't want to / can't have multiple accounts on said cloud16:19
hogepodgeeglute: rewriting or replacement of tests that require too many resources16:19
egluteso we should start with identifying those tests?16:19
hogepodgemfisher_ora: I would argue that it's a bad standard.16:20
dwalleckI'd be okay with helping identify what resources needed are by which tests16:20
mfisher_oraI'm still not understanding why you would want defcore to run against 'any cloud'16:20
hogepodgemfisher_ora: all I should need to know for interoperability is an endpoint and credentials, because as an app developer that is what I have to work with16:20
dwalleckThere's a bigger issue with resources that was talked about at the summit with the QA group16:21
rockygIdentifying resources and tagging the tests with that info would be valuable, I would think16:21
eglutemfisher_ora: the point of defcore is interoperability between different openstack clouds. In theory, all openstack clouds should exhibit the same behavior.16:21
rockygBut this is just like the tests that don't need admin yet still require it16:21
hogepodgemfisher_ora: https://github.com/openstack/defcore/blob/master/doc/source/process/CoreDefinition.rst16:21
hogepodge"Tests can be remotely or self-administered"16:22
mfisher_oraI'll gab with you more about it later rather than holding this up with questions16:22
rockygAlso, with the security tests, I think they should be there, but maybe we need a second set of tests for security.  Different focus,  but important16:22
rockygOr group the security tests togehter.16:23
dwalleckrockyg: There is a set of tests the security working group is working on16:23
rockygexcellent!16:24
hogepodgepart of what I'm doing right now is figuring out which tests fail with more constraints, why, and if they can be replaced with something that tests the same capability but without the added assumptions16:25
rockygeglute, I think the foundation needs to give Chris a raise.  He's going far and beyond what any manager would have thought to have him do :-)16:26
eglutehogepodge would you rather have a raise or an extra person to help with this? :D I wish I could give you and instant raise :D16:27
*** alex_klimov has quit IRC16:27
hogepodgeI have a couple of months. I also want to make sure it's a direction we want the committee to go in.16:27
*** zehicle has joined #openstack-defcore16:28
hogepodgeMight be worthwhile holding off on more discussion for the next week. I think lots are absent because of the time change.16:28
egluteMark sent me an email he won't be able to attend today16:28
hogepodgePeople to help would be nice. I've been given time to work upstream on this. I'm just gathering information right now.16:28
rockygI just want to say, this time is *much* better for me.  I have some coffee in me now!16:28
egluteI think we all agree that tempest tests need improvement for defcore16:29
eglutestarting with one account would be good16:29
*** Guest78752 has quit IRC16:30
zehiclerunning late16:30
eglutehogepodge what would it take to get the defcore tests to work with one account16:30
eglutehello zehicle16:30
SammyD+1 improved tempest tests for defcore... :-)16:30
egluteSammyD dwalleck is that something that you can help with?16:31
zehiclehey16:31
hogepodgeeglute: I need to upgrade my account to allow more nodes. I also need to rewrite tests that require more than one account.16:31
rockygAlso, categorizing the tests, as hogepodge is doing will help a lot, too.  Break the larger set into subsets16:31
dwalleckeglute: Absolutely16:31
egluteWould it help you to test on Rackspace Public cloud?16:31
rockyg+116:32
hogepodgeeglute: it would be nice to get more reference points. I'm planning on signing up for more clouds to see how far I can get with them.16:32
eglutehogepodge I will work with you offline for accounts16:32
hogepodgeok16:33
dwalleck hogepodge: And I can help you with any issues if you run into them16:33
eglutealso, can you work with dwalleck and rockyg to see how they can get involved?16:33
hogepodgethanks16:33
eglutewith helping with the tests that is16:33
dwalleckI'm running continuous testing against our public cloud, so I've made it through the bumpy spots16:33
eglutei think having better defcore tests will help everyone16:33
rockygAnother advantage of clumping tests be capabilities could be that they would run faster.  Don't have to initialize capabilities that aren't going to be used until the appropriate test set16:34
SammyDWe certainly can help with the digging in and understanding what is working for what reason. I could see about dedicating some of our capacity this cycle to directly updating tempest tests for defcore. :-)16:34
eglutethank you SammyD!16:34
eglute#action SammyD, dwalleck, rockyg to help hogepodge with defcore tempests issues16:35
eglutehogepodge you also have on agenda new interop tests16:35
eglutedo you have specific tests in mind that need to be written, or?16:36
hogepodgeno, that was it on interop16:36
hogepodgethe next item is about information sent to refstack16:36
rockygSomething possibly worth looking at are the capabilities we would *want* to include but only have one test for16:36
eglute#topic regstack results16:36
egluterockyg agree... can someone start a wish list for tests?16:37
catherineDSo currently on pass tests are sending to RefStack per blue print https://blueprints.launchpad.net/refstack/+spec/pass-only-uploads16:37
hogepodgerockyg: yeah, we need to write some new tests for the caps that have only one. more upstream work16:37
rockygeglute, we've got a good start here.  Classification and categorization tags16:38
rockyghogepodge, tht would make markvoelker very happy16:38
SammyD+1 on the wishlist...if I have something I can refer to I can try to work a certain amount of time throughout the cycle to grab the next test in the list for instance.. :-)16:38
eglutei think we have two different topics going, sorry Catherine16:38
rockygOK.  once the next etherpad is created, I can start dumping some ideas16:39
hogepodgeto the topic, some folks, qa in particular, would love for refstack to be able to store subunit data, but as a committee we have stated only anonymized passing results are sent16:39
eglute#action SammyD will work on wish list for tests with rockyg16:39
hogepodgeGuest78752 can speak more to that16:39
hogepodgecatherineD: that is16:39
catherineDSo refstack-client (which run on the client side) would sanitize the data from subunit format (which includes all test result information of pass/fail/skip .. log info) to a JSON with only pass tests and then upload to RefStack server16:40
catherineDUpload only pass tests was implemented per blueprint https://blueprints.launchpad.net/refstack/+spec/pass-only-uploads16:41
SammyDI only volunteered to *use* the wishlist to knock things out. I can help out some on building out the list too though. :-)16:42
eglute#action SammyD will use tests wish list to implement tests16:43
catherineDCurrent suggestion from the last RefStack IRC meeting is to send subunit data file to RefStack server.16:43
eglutecatherineD are you asking for all the data?16:45
egluteor just for passing tests?16:45
catherineDright now refstack-client only uploads pass data ...16:45
rockygMy question would be is how many refstack users want this "feature"16:46
egluteare you looking for failed test results as well? or?16:46
catherineDpersonally I do not think failed tests would help DefCore ... only pass tests16:46
rockygWe put the limits in place to protect both the vendor *and* the test runner.  Failed tests leak data into subunit16:46
mfisher_orawell if you have failed tests posted with the reasons, and the reasons are consistent, makes for a good indicator to flag said tests16:47
eglutecatherineD what does subunit data include for passing tests?16:48
catherineDsubunit data includes all data (pass/fail/skip/log ..) log may include sensitive data ...16:48
dwalleckstack traces and other fun data :-) It's useful in terms of testing16:49
catherineDthat is why currently we sanitize the data at the client side before sending to RefStack server ...16:49
egluteso refstack would like all of that data uploaded?16:49
catherineDno I don't see how  RefStack would use all data at this point ....16:49
rockygMaybe the client could just save the full results to a directory, or let the user name the results, including save path?16:50
rockygOn the test runner's machine/network that is16:50
catherineDbesides subunit data file can be huge ...16:50
eglutecatherineD can you tell us what you would like to see happen?16:50
rockygespecially with failure traces ;-)16:51
catherineDrockyg: exactly ... all sensitive and large data should be handled at the client sid16:51
catherineDDefCore/RefStack won't analyze fail reason ... the most we can do is statitical data anlaysis ...16:51
rockygI don't think refstack should change what goes to the server, but we could allow the client to have save options locally16:51
catherineDrockyg: yup that is what being implemented right now ....16:52
catherineDeglute: I think for the short term (next 1 to 2 releases ) sending passing data makes the most sense ...16:53
rockygwhat are the reasons given for wanting to save the files to the server?16:53
rockygcatherineD, agree.  DefCore could revisit after  data from 2016.1 starts coming in, or there is more impetus behind a change.16:54
catherineDthe disadvantage is we will have to delay using QA subunut2sql utility .. but currently since we only parse for pass data the parsing task is relatively simple ...16:54
catherineDrockyg: the main reason is to use subunit2sql util at this time16:55
rockygSo, could the client use it only on client side, without passing the data on?16:56
catherineDrockyg: yes16:56
rockygLet the user dump into their own csv file or db?16:56
rockygThen the user gets what the want/need, but don't push sensitive data16:56
catherineDrockyg: sorry no ... client side can not use subunit2sql because the database (sql part) resides in the server side16:57
*** MarkAtwood has joined #openstack-defcore16:57
egluteso for passing tests, is there sensitive info in subunit tests?16:57
dwalleckWell, the subunit2sql call is just made on the stream that comes out of the runner. It would be a change to the refstack client16:57
catherineDsubunit2sql tool needs both the subunit file and the database16:57
rockygeglute, not supposed to be16:58
dwalleckI'm currently dumping all my defcore test results with subunit2sql16:58
catherineDeglute: no there is no sensitive data in the JSON of passing tests that refstack-client creates16:58
catherineDdwalleck: do you dump it to a database?16:59
dwalleckcatherineD: Yes16:59
eglutecatherineD can you write up an email to the ML with what you are asking and have people comment/provide feedback?16:59
catherineDand you have access to the database16:59
dwalleckYes, it's a database that I run. It's standalone17:00
rockygeglute, I think we need to clairfy that catherineD is not asking for this, but that some users are17:00
rockygAnd that what dwalleck does is maybe a clientside tool for users that would be separate17:01
zehicleback online17:01
zehiclebeen lurking17:01
rockygYou missed it. zehicle!  You own all the action items!17:01
catherineDdwalleck: exactly, you need both subunit file and database at your side ...17:01
egluteok, we are out of time, we need action on this17:01
eglutecatherineD can you write up an email with proposal?17:01
dwallecksorry, bouncing my attention17:02
zehiclecool!17:02
zehicleI thought we started :30 ago17:02
catherineDeglute: I am not propose for any change at this time ... but I can document the request17:02
rockygBut do we need action, other than adding it to the agenda?  We need more data on the user "wants" and whether we need to change refstack and client to meet them17:02
eglutezehicle no, 1 hour ago. many calendar fails17:02
rockygzehicle, me too;-)  It would be a much better time.17:03
zehicleah, I could have made it at 10!17:03
catherineDeglute: can we revisit this next week. ...17:03
catherineDzehicle: has the background of why we only send pass data17:03
eglutecatherineD sure thing, lets re-visit next week17:03
zehicleI'm out next week17:03
*** dwalleck has quit IRC17:03
rockygcatherineD, the etherpad is already created.  I'll toss it onto the agenda section17:03
zehicleyy, pass data only was a board request17:03
catherineDrockyg: thx ...17:03
zehiclewe did not want to expose "vendor broken things"17:04
egluteright, i am fine with passing only data being sent, just was not clear on what catherineD wanted. we will re-visit17:04
zehiclethe idea being that failed tests = broken.17:04
eglutethank you everyone! next week the meeting is at 10:00 CST, same as (almost) always17:04
zehiclewe did not want vendors to only run the required tests17:04
catherineDeglute: I need the confirmation that DefCore wants RefStack server to accept pass data only17:05
rockygI really don't think it's what catherineD wants, just something that some users have requested17:05
zehicleand that was the compromise to encourage maximizing the number of extra tests17:05
eglutecatherineD yes, pass only data for refstack17:05
zehiclefailed tests don't really give you much information17:05
catherineDeglute: THANK YOU!!!17:05
zehiclefrom an interop perspective17:05
rockygI think we can provide the users the request through a separate, clientside tool like dwalleck uses17:06
rockygAnd, it could even be in a contrib dir17:06
rockygso, do we need this on next week's agenda?  I think we are in agreement that we keep what goes to the server as is.17:07
egluteyes, i think we are good. unless catherineD wants us to revisit17:07
catherineDrockyg: client side is OK ..17:07
eglute#agreed to send only pass only data to refstack17:08
rockygI'll remove it from agenda for 3.17:08
eglutethanks everyone!17:08
eglute#endmeeting17:08
openstackMeeting ended Wed Nov 11 17:08:23 2015 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:08
openstackMinutes:        http://eavesdrop.openstack.org/meetings/defcore/2015/defcore.2015-11-11-16.00.html17:08
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/defcore/2015/defcore.2015-11-11-16.00.txt17:08
openstackLog:            http://eavesdrop.openstack.org/meetings/defcore/2015/defcore.2015-11-11-16.00.log.html17:08
catherineDeglute: thanks ... decision made .. no need to re-visit17:08
egluteexcellent, thank you catherineD!17:08
rockygThanks!  and could we pretty please make the time 10:30 CST/8:30PST???17:08
*** pbredenb1 has left #openstack-defcore17:09
eglutei think we need to vote on that17:09
eglutei will send an email17:09
catherineDeglute: Thank you!17:09
rockygThanks!17:10
egluteEveyone: http://doodle.com/poll/suifaab6i5rxk2p817:16
rockygThanks!  Voted.17:18
*** zehicle has quit IRC17:26
*** markvoelker has quit IRC17:35
*** MarkAtwood has quit IRC18:01
*** dalees has joined #openstack-defcore18:21
*** markvoelker has joined #openstack-defcore18:31
*** MarkAtwood has joined #openstack-defcore18:34
openstackgerritMerged openstack/defcore: Migrate 2015.07 to Schema 1.4  https://review.openstack.org/24094218:38
*** rockyg has quit IRC18:39
openstackgerritMerged openstack/defcore: Migrate next and 2016.01 to Schema 1.4  https://review.openstack.org/24093918:39
*** MarkAtwood has quit IRC18:44
*** MarkAtwood has joined #openstack-defcore18:45
*** MarkAtwo_ has joined #openstack-defcore19:19
*** MarkAtwood has quit IRC19:19
*** MarkAtwo_ has quit IRC19:22
*** MarkAtwood has joined #openstack-defcore19:33
daleeshey, would those two merges above have just broken the refstack website?19:37
daleesI'm not seeing any data in 'corresponding openstack releases' for 2015.07 or 2016.01 anymore19:37
daleeson any community results under https://refstack.openstack.org/#/community_results19:38
catherineDdalees: confirm that refstack was not updated to handle schema version 1.4 yet .... we are working on a patch ...19:47
daleescatherineD, thanks!19:47
*** alex_klimov has joined #openstack-defcore20:09
*** MarkAtwood has quit IRC20:51
*** MarkAtwood has joined #openstack-defcore21:06
*** mfisher_ora has quit IRC22:02
*** alex_klimov has quit IRC22:08
*** edmondsw has quit IRC22:23
catherineDdalees: pvaneck  just pushed a patch https://review.openstack.org/#/c/244295/  waiting for an other core reviewer to review before we merge ...23:21

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!