15:01:10 <scottda> #startmeeting cinder-testing
15:01:11 <smcginnis> scottda: I think so.
15:01:11 <openstack> Meeting started Wed Aug  3 15:01:10 2016 UTC and is due to finish in 60 minutes.  The chair is scottda. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:14 <gouthamr> hey
15:01:15 <openstack> The meeting name has been set to 'cinder_testing'
15:01:29 <_alastor_> hey
15:02:30 <scottda> I'd hoped to go through all the patches listed in the etherpad and update stauts, etc.....I'll do that today
15:03:58 <e0ne> I'll propose patches with fake drivers refactoring to remove duplication, devstack integration later this week
15:04:14 <scottda> We'd discussed some stuff at the mid-cycle. Notes are here: https://etherpad.openstack.org/p/newton-cinder-midcycle-day2
15:04:24 <e0ne> also, I'm going to install only cinder+keystone for the functional job
15:04:30 <scottda> e0ne: +1
15:04:39 <e0ne> I'm going to send mail to openstack-dev once pathces will be ready for review
15:05:03 <eharney> keystone for the functional job?
15:05:27 <e0ne> eharney: TBH, I didn't try to run cinder w/o keystone yet
15:05:46 <eharney> that doesn't sound like it fits with what we defined functional tests as two weeks ago
15:05:50 <e0ne> eharney: it worked few years ago, but we need to check and fix it if needed
15:06:20 <smcginnis> eharney: Right
15:06:29 <e0ne> eharney: I agree that usage of 'noauth' will be better for functional tests
15:07:14 <eharney> at the midcycle, the agreement was that functional tests will work in an environment identical to unit tests today
15:07:18 <DuncanT> noauth is totally broken in cinder
15:07:20 <scottda> e0ne: Is it much work to test 'noauth' before moving on with this?
15:07:44 <e0ne> scottda: I didn't try to test it yet:(
15:07:48 <scottda> DuncanT: Well that answers my question...
15:07:55 <DuncanT> The easiest way forward with that would be to write a fake keystone middleware that just returns constant values
15:08:06 <smcginnis> eharney: My understanding as well. Functional should be extended unit tests.
15:08:30 <DuncanT> Functional tests will have a real db, right?
15:08:31 <e0ne> smcginnis, eharney: I propose to do it step-by-step
15:08:33 <eharney> that was already a departure from the previous definition of functional... now it looks like we're drifting elsewhere
15:08:40 <e0ne> DuncanT: +1
15:09:52 <e0ne> eharney: IMO, it's better to have keystone for the beginning and and functional tests with fake drivers rather than fixing noauth or fake middleware few months more
15:10:47 <eharney> for it to be "better" i think we need to define what these tests are supposed to do... which we did, i thought
15:11:13 <openstackgerrit> Merged openstack/cinder: Size in tintri driver should be converted to integer  https://review.openstack.org/325025
15:11:18 <smcginnis> We did. And there is no keystone as part of it.
15:11:42 <e0ne> #link https://github.com/openstack/cinder/blob/master/doc/source/devref/testing.rst#functional-tests
15:11:59 <smcginnis> That does need to be updated to be more specific.
15:12:21 <e0ne> smcginnis, eharney: what about rabbitmq?
15:12:29 <eharney> this is where we came up with the idea of "small-scale integration" which is an environment with Cinder+rabbit+db+Keystone and nothing else
15:12:38 <geguileo> This was said in the meeting (etherpad):
15:12:40 <geguileo> The original idea was to just have, cider, rabbit, and mysql
15:12:42 <geguileo> Anything more than that should go into tempest
15:12:52 <e0ne> geguileo: +1
15:13:02 <eharney> "original"
15:13:46 <geguileo> I'm ok to add Keytone for a little bit as long as it's clear that it should go away
15:14:03 <geguileo> Although I think it would be better to get it right from the start
15:14:11 <eharney> i don't think it's clear at all at the moment
15:14:40 <Swanson> xyang1, thanks for the review
15:14:47 <smcginnis> No keystone. These should basically just be like unit testing, but the scope is to test more than a "unit".
15:14:52 <xyang1> Swanson: np
15:14:56 <patrickeast> IMO if we add it in now it's harder to take out later... And more than likely won't happen
15:15:01 <scottda> If noone has time for the keystone work ATM, it pushes changes to functional tests out. Maybe no next release?
15:15:02 <patrickeast> smcginnis: +1
15:15:06 <smcginnis> I would even say no mysql and rabbitmq. Those don't make sense in that context.
15:15:18 <eharney> smcginnis: right
15:15:32 <Swanson> Next time it will be 2000 1 line patches.
15:15:33 <geguileo> I may be getting a little lost here
15:15:41 <geguileo> Then what are we going to test?
15:15:58 <geguileo> Are we going to fake all DB access and messaging?
15:15:59 <e0ne> scottda: I'll take a look on it
15:16:07 <eharney> the idea was to have unit tests be unit tests and huge run-through-ten-modules tests be in functional tests
15:16:14 <e0ne> scottda: I mean how to run cinder w/o keystone
15:16:15 <smcginnis> Anything outside of Cinder should be mocked.
15:16:28 <eharney> but they are essentially the same test setup, just with different styles of tests
15:16:28 <smcginnis> Just like we do for unit testing - anything outside the unit we are testing should be mocked.
15:16:35 <e0ne> smcginnis: are you talking about unit tests?
15:16:38 <xyang1> Swanson: that will bump yiur stats!
15:16:48 <smcginnis> e0ne: Functional testing.
15:16:51 <geguileo> smcginnis: So we won't be testing in functional tests anything that goes from API to Scheduler and then to the volume, right?
15:16:59 <smcginnis> geguileo: Correct.
15:17:15 <smcginnis> Although it can go from the API level through, but not a running deployment.
15:17:18 <e0ne> smcginnis: :(
15:17:19 <smcginnis> Just calling the methods.
15:17:44 <e0ne> smcginnis: it makes functional tests not useful, IMO
15:17:55 <smcginnis> e0ne: It makes them extremely useful.
15:17:58 <e0ne> smcginnis: it becases the same as unit + DB
15:18:13 <smcginnis> And give us a way to get out of the mess that our current unit tests are.
15:18:26 <eharney> so i posted a question on the etherpad about this issue as well
15:18:33 <smcginnis> Unit tests should actually be unit tests - but right now they are not.
15:18:39 <eharney> i think having the current (midcycle) functional tests be a separate job is problematic
15:18:56 <nikeshm> smcginnis: are yu familiar with Traceback in http://54.209.116.144/19/349019/13/check/kaminario-dsvm-tempest-full-iscsi/f871021/logs/screen-c-vol.txt.gz
15:19:00 <smcginnis> The next level up will allow us to test interaction between different modules.
15:19:07 <nikeshm> i m getting today
15:19:11 <scottda> eharney: Yeah, the py3 issue makes sense as a problem/blocker
15:19:16 <nikeshm> this in iscsi
15:19:20 <nikeshm> job
15:19:29 <nikeshm> when running
15:19:31 <eharney> if we're doing this version of functional tests, they should probably be run in the same jobs that run the unit tests
15:19:40 <smcginnis> eharney: I don't think so.
15:20:08 <eharney> smcginnis: there are at least two downsides to having them in a separate job, i'm not sure what the upsides are
15:20:33 <nikeshm> any one familiar with Traceback in http://54.209.116.144/19/349019/13/check/kaminario-dsvm-tempest-full-iscsi/f871021/logs/screen-c-vol.txt.gz
15:20:37 <smcginnis> It's a different scope of testing. Unit tests should be our basic level of tests run.
15:21:02 <smcginnis> Functional expands on that, but is good to separate so they are not run for basic validation.
15:21:07 <eharney> sure, but to do this right, we need py27 and py3 functional jobs
15:21:15 <smcginnis> That's true.
15:21:32 <eharney> and i'd like to be able to see coverage stats for both combined, but i guess that can be sorted out in tox.ini somehow outside of how we do jobs
15:21:48 <jgriffith> nikeshm: looks like privsep stuff :(
15:22:21 <eharney> so i guess we'll just make a functional-py3 job then?
15:23:16 <e0ne> I'm afraid to ask, but...
15:23:48 <e0ne> if we are so interested in functional-py3 jib, why nobody don't talk about tempest-py3 job?
15:23:56 <jgriffith> e0ne: +1
15:24:05 <eharney> sounds like a good idea to me
15:24:07 <smcginnis> Maybe "integration" testing would be a better name for these. Unit testing for units of code, integration for the interaction between these units, functional then defined as run against a running instance,
15:24:14 <nikeshm> jgriffith: any solution to avoid that in our CI
15:24:14 <smcginnis> e0ne: DOn't we have that already?
15:24:18 <geguileo> nikeshm: I think the problem is that scsi_transport_fc module is not loaded in the system
15:24:32 <e0ne> smcginnis: maybe in an experimental queue only
15:24:53 <eharney> smcginnis: i think that's backwards from normal terminology which is part of what jgriffith was trying to fix
15:24:54 <e0ne> integration between modules sounds interesting
15:25:05 <jgriffith> geguileo: indeed, but I can't figure out if it failed to load because of privsep error or just wasn't included?
15:25:10 <e0ne> but ingetration between different cinder components should be done too
15:25:13 <dulek_> haypo was talking that py3 tempest is another step in his conversion efforts.
15:25:15 <Swanson> smcginnis, integration tests are for testing against a running instance.
15:25:19 <geguileo> jgriffith: True
15:25:40 <smcginnis> Swanson: It seems most folks have different definitions of integration vs testing.
15:25:45 <smcginnis> * vs functional.
15:25:50 <jgriffith> sigh.. here we go again
15:25:50 <e0ne> I won't hollywar ot bukeshed on what is func. testing
15:25:59 <jgriffith> e0ne: +1
15:26:03 <geguileo> e0ne: I agree, but maybe those should be integration tests to differentiate them from functional
15:26:08 <e0ne> I'll propose my patches and send links  to the mailing list
15:26:18 <smcginnis> But I thought we were being consistent with what other projects were calling the types of testing.
15:26:21 <jgriffith> Call it other-testing AFAIAC this is rather silly
15:26:32 <eharney> well... "Tempest - The OpenStack Integration Test Suite" already exists, so let's not redefine it to something else
15:26:37 <smcginnis> jgriffith: +a
15:26:39 <jgriffith> smcginnis: yes, that was the whole point and the hour long discussion in Ft Collins
15:26:49 <xyang1> e0ne: jgriffith have we decided on the fake driver name yet
15:26:50 <smcginnis> As long as we differentiate the type of testing for one vs the other.
15:26:51 <jgriffith> smcginnis: which for some reason we've decided we need to talk about and rhash again
15:27:13 <smcginnis> jgriffith: I agree. I thought we had it all set. But apparently not.
15:27:26 <geguileo> eharney: Sure, but it also says: "This is a set of integration tests to be run against a live OpenStack cluster."
15:27:29 <Swanson> the jgriffith memorial test suite.
15:27:32 <e0ne> xyang1, jgriffith: AFAIR, we desided to use FakeLoggingDriver and FakeGateDriver names
15:27:45 <smcginnis> geguileo: Stale comments in the etherpad I believe.
15:27:48 <xyang1> e0ne: ok
15:28:01 <smcginnis> https://www.youtube.com/channel/UCJ8Koy4gsISMy0qW3CWZmaQ
15:28:14 <geguileo> smcginnis: Not really, it's from here http://docs.openstack.org/developer/tempest/overview.html
15:28:15 <e0ne> e.g. python-heatclient funcitonal tests http://logs.openstack.org/79/345379/12/check/gate-heatclient-dsvm-functional/14d3a7e/
15:28:19 <e0ne> #link http://logs.openstack.org/79/345379/12/check/gate-heatclient-dsvm-functional/14d3a7e/
15:28:36 <geguileo> eharney: And we are talking about integration tests where we should be able to have an error generator, right?
15:28:40 <scottda> smcginnis: How about you post a devref patch that states what you/we think are the definitions, we review and, once merged, we take that as the definitions?
15:28:45 <e0ne> I like how Heat team implemented funcitonal tests
15:28:55 <geguileo> scottda: +1
15:28:56 <eharney> e0ne: what do those tests do?
15:29:03 <nikeshm> geguileo: jgriffith: do we need that module in iscsi driver testing
15:29:14 <smcginnis> geguileo: OK, then back to the four levels we defined: unit tests, functional tests, integration tests, tempest.
15:29:23 <e0ne> eharney: they setup minimal devstack with heat components only
15:29:29 <jgriffith> scottda: I'll post it, but frankly you were there in the meeting and I specifically asked you if you understood and agreed.  You did
15:29:30 <geguileo> smcginnis: I'm fine with that
15:29:40 <eharney> e0ne: this is what i was proposing as "small scale integration"
15:30:05 <jgriffith> eharney: then propose that, don't stop progress on something else that somebody has started (ie e0ne )
15:30:07 <scottda> jgriffith: It's not about agreement/disagreement, it's about getting it into the devref so we all know what the defs are.
15:30:15 <eharney> jgriffith: i'm not stopping progress on anything
15:30:20 <jgriffith> scottda: fine, devref patch coming up
15:30:26 <jgriffith> eharney: ok
15:30:33 <jgriffith> eharney: sorry
15:30:39 <eharney> according to what we defined at the midcycle, the functional test environment is done and just needs tests, not more components added
15:30:49 <jgriffith> eharney: +1
15:31:16 <smcginnis> eharney: +1
15:31:18 <geguileo> nikeshm: In the logs I see "Fetching connector for FibreChannelConnector get_connector_properties" before the error
15:31:40 <e0ne> and we will have 10 functional, 3 "small scale integration" tests in the end of O release
15:32:09 <jgriffith> scottda: can you clarify what's missing from the devref for me?
15:32:20 <e0ne> I don't want to spray attention on too many of test types
15:32:23 <geguileo> nikeshm: http://54.209.116.144/19/349019/13/check/kaminario-dsvm-tempest-full-iscsi/f871021/logs/screen-c-vol.txt.gz#_2016-08-03_14_49_56_929
15:32:42 <scottda> https://www.irccloud.com/pastebin/4qBAq2bp/
15:32:54 <jgriffith> scottda: http://docs.openstack.org/developer/cinder/devref/testing.html
15:33:01 <scottda> Do we agree that functional tests run with a database?
15:33:11 <smcginnis> That last section should be removed, IMO.
15:33:14 <scottda> I thought that was contentious?
15:33:24 <scottda> smcginnis: Ok, so that needs changing
15:33:27 <xyang1> e0ne: do you have any patch that moves the fake driver?  or are we all set using the existing LoggingDriver to write the functional tests?
15:33:27 <jgriffith> smcginnis: the non-cinder services part?
15:33:35 <smcginnis> And we should add Integration Tests with a description of the "small scale integration" work being done now.
15:33:55 <smcginnis> jgriffith: The "database present and may start Cinder services to accept requests" I think.
15:33:58 <e0ne> xyang1: I don't know now
15:34:04 <jgriffith> ok
15:34:30 <scottda> And messageQ will run with "small scale integration", right?
15:35:39 <scottda> We also have nothing in the devref to differentiate between in-tree tempest and upstream
15:35:59 <eharney> in-tree tempest vs upstream tempest is not a very interesting distinction IMO
15:36:05 <xyang1> e0ne: ok, I'll start with the existing LoggingDriver and will change if needed
15:36:10 <eharney> they're tests run in the same environments, it's just where the code lives
15:36:38 <scottda> eharney: So how to people know where the tests should go?
15:36:47 <e0ne> xyang1: ok. I'll ping you if i have any update on fake frivers
15:36:53 <eharney> scottda: yeah, true, we do need to document that
15:37:05 <xyang1> e0ne: ok, thanks
15:38:07 <scottda> I don't recall coming to a decision at the mid-cycle around how to decide in-tree vs. out-of-tree for tempest?
15:38:42 <eharney> my working assumption is that in-tree is for things that we want to test that is outside of the scope of what tempest wants
15:39:02 <eharney> (what tempest wants as a project etc)
15:39:07 <jgriffith> smcginnis: scottda https://gist.github.com/j-griffith/063574aac688d9a383bf6fe4a50f00c9
15:39:17 <scottda> eharney: Yes, that seems correct. But I think we need to codify that and put that in the devref.
15:39:18 <jgriffith> does that work for the Functional section at least?
15:40:15 <scottda> jgriffith: Doesn't that mean we need messageQ running ?
15:40:25 <smcginnis> jgriffith: I would think even less than that. It wouldn't necessarily be all the way from API to driver. Could just be between two modules.
15:42:42 <jgriffith> smcginnis: ok, but just to be clear, there's no mechanism in there right now to fake those sorts of things out
15:43:02 <jgriffith> smcginnis: it's indeed "functional" in that it uses a client and send REAL API cmds
15:43:12 <jgriffith> s/send/sends/
15:43:33 <smcginnis> jgriffith: I think that's why it needs to state a smaller scope. Just unit test style tests that validate code paths between more than just a unit of code.
15:43:38 <e0ne> jgriffith: +1
15:43:50 <smcginnis> They almost fit "integration" testing in my definition more than "functional".
15:44:02 <jgriffith> I'm no longer interested in trying to drive this to concensus or to try and *force* my opinion or the methodology used in other OpenStack projects
15:44:05 <e0ne> smcginnis: you've just described most of our "unit
15:44:11 <e0ne> " tests
15:44:16 <eharney> e0ne: which is why we want to move a lot of unit tests to functional tests
15:44:21 <smcginnis> e0ne: Right, which is  aproblem.
15:44:36 <smcginnis> A lot of our unit tests right now are not unit tests.
15:44:38 <e0ne> jgriffith: +1 :(
15:45:07 <jgriffith> rather than bike-shed on this why don't people write/propose tests and see how things go?
15:45:26 <dulek_> jgriffith: +1
15:45:29 <scottda> Because then we'll keep having this conversation...
15:45:36 <smcginnis> jgriffith: I think we need at least a general consensus so it's not chaos. But we can try that.
15:45:40 <jgriffith> scottda: not if you don't keep bringing it up
15:45:41 <hemna> mornin
15:45:49 <scottda> Why not just draw the lines somewhere, write it down (in the devref) and be done with it.
15:46:03 <jgriffith> I did and we've done nothing but argue about it :)
15:46:04 <eharney> scottda: i think we have a good idea of what to propose in the devref now, seems like a good idea
15:46:19 <scottda> That's why I said, smcginnis put up a patch with the definitions and be done with it.
15:46:28 <smcginnis> jgriffith: I can try proposing something to the devref if you'd prefer. Then we can bikeshed on that. :)
15:46:35 <jgriffith> Ok
15:46:42 <scottda> or jgriffith or whomever. I don't care what the defs are, just define them once and for all
15:46:47 <jgriffith> smcginnis: as long as we have something to bikeshed about :)
15:46:51 <geguileo> smcginnis: +1
15:47:09 <jgriffith> smcginnis: I'll pass the torch to you, because frankly I can't figure out how to appease all of this
15:47:56 <smcginnis> ;0
15:48:58 <openstackgerrit> Walter A. Boring IV (hemna) proposed openstack/cinder: CI: Add CI_WIKI_NAME to all drivers  https://review.openstack.org/348002
15:49:00 <scottda> #action smcginnis Will put up a devref patch with definitions for unit, functional, small scale integrations, in-tree vs. out-of-tree tempest
15:49:00 <scottda> smcginnis: OK?
15:49:10 <smcginnis> scottda: +1
15:50:06 <scottda> Anyone have anything else today? Or shall be break before the Next Exciting Meeting?
15:50:50 <scottda> #endmeeting