| *** adjohn has quit IRC | 00:03 | |
| *** adjohn has joined #openstack-meeting | 00:04 | |
| *** AlanClark has quit IRC | 00:15 | |
| *** salv-orlando has quit IRC | 00:19 | |
| *** nati2 has quit IRC | 00:19 | |
| *** Guest36549 has quit IRC | 00:20 | |
| *** ayoung has joined #openstack-meeting | 00:47 | |
| *** dendro-afk is now known as dendrobates | 00:53 | |
| *** bcwaldon has quit IRC | 01:02 | |
| *** heckj has joined #openstack-meeting | 01:03 | |
| *** gyee has quit IRC | 01:04 | |
| *** bcwaldon has joined #openstack-meeting | 01:08 | |
| *** dcramer_ has joined #openstack-meeting | 01:18 | |
| *** sleepsonthefloo has quit IRC | 01:21 | |
| *** bengrue has quit IRC | 01:29 | |
| *** reed has quit IRC | 01:38 | |
| *** jakedahn has quit IRC | 01:40 | |
| *** jakedahn has joined #openstack-meeting | 01:40 | |
| *** bhall has quit IRC | 01:44 | |
| *** novas0x2a|laptop has quit IRC | 01:47 | |
| *** novas0x2a|laptop has joined #openstack-meeting | 01:49 | |
| *** davlap has quit IRC | 01:49 | |
| *** dragondm has quit IRC | 01:55 | |
| *** edgarmagana has quit IRC | 01:57 | |
| *** dcramer_ has quit IRC | 01:58 | |
| *** jdurgin has quit IRC | 02:02 | |
| *** ohnoimdead has quit IRC | 02:07 | |
| *** jakedahn has quit IRC | 02:09 | |
| *** adjohn has quit IRC | 02:11 | |
| *** oubiwann1 has quit IRC | 02:12 | |
| *** dcramer_ has joined #openstack-meeting | 02:14 | |
| *** novas0x2a|laptop has quit IRC | 02:38 | |
| *** dendrobates is now known as dendro-afk | 02:38 | |
| *** dendro-afk is now known as dendrobates | 02:39 | |
| *** dragondm has joined #openstack-meeting | 02:50 | |
| *** jakedahn has joined #openstack-meeting | 02:51 | |
| *** deshantm_ has quit IRC | 03:41 | |
| *** adjohn has joined #openstack-meeting | 04:12 | |
| *** bcwaldon has quit IRC | 04:14 | |
| *** dolphm has joined #openstack-meeting | 04:15 | |
| *** adjohn has quit IRC | 04:24 | |
| *** adjohn has joined #openstack-meeting | 04:24 | |
| *** bengrue has joined #openstack-meeting | 04:26 | |
| *** adjohn has quit IRC | 04:26 | |
| *** dcramer_ has quit IRC | 04:59 | |
| *** dragondm has quit IRC | 05:08 | |
| *** dolphm has quit IRC | 05:10 | |
| *** jakedahn has quit IRC | 05:16 | |
| *** scottsanchez has quit IRC | 05:25 | |
| *** scottsanchez has joined #openstack-meeting | 06:38 | |
| *** bengrue has quit IRC | 07:04 | |
| *** hggdh has joined #openstack-meeting | 08:26 | |
| *** zul has joined #openstack-meeting | 08:31 | |
| *** carlp has quit IRC | 08:39 | |
| *** carlp has joined #openstack-meeting | 08:40 | |
| *** zul has quit IRC | 08:40 | |
| *** zul has joined #openstack-meeting | 08:44 | |
| *** nevrax has joined #openstack-meeting | 08:44 | |
| *** darraghb has joined #openstack-meeting | 09:45 | |
| *** derekh has joined #openstack-meeting | 10:10 | |
| *** scottsanchez has quit IRC | 10:28 | |
| *** mikal has joined #openstack-meeting | 11:02 | |
| *** GheRivero has quit IRC | 11:11 | |
| *** zul has quit IRC | 11:59 | |
| *** hggdh has quit IRC | 12:01 | |
| *** derekh has quit IRC | 12:20 | |
| *** derekh has joined #openstack-meeting | 12:29 | |
| *** zigo has joined #openstack-meeting | 12:56 | |
| *** dolphm has joined #openstack-meeting | 13:00 | |
| *** zul has joined #openstack-meeting | 13:07 | |
| *** hggdh has joined #openstack-meeting | 13:12 | |
| *** rkukura has joined #openstack-meeting | 13:23 | |
| *** dolphm has quit IRC | 13:59 | |
| *** dprince has joined #openstack-meeting | 14:03 | |
| *** deshantm has joined #openstack-meeting | 14:14 | |
| *** mdomsch has joined #openstack-meeting | 14:41 | |
| *** mattray has joined #openstack-meeting | 14:47 | |
| *** nevrax has quit IRC | 14:55 | |
| *** deshantm_ has joined #openstack-meeting | 15:09 | |
| *** deshantm has quit IRC | 15:11 | |
| *** deshantm_ is now known as deshantm | 15:11 | |
| *** zigo has quit IRC | 15:11 | |
| *** dolphm has joined #openstack-meeting | 15:12 | |
| *** dolphm has quit IRC | 15:15 | |
| *** dolphm has joined #openstack-meeting | 15:32 | |
| *** yamahata_ has joined #openstack-meeting | 15:32 | |
| *** rnirmal has joined #openstack-meeting | 15:49 | |
| *** wwkeyboard has left #openstack-meeting | 15:57 | |
| *** adjohn has joined #openstack-meeting | 15:59 | |
| *** bcwaldon has joined #openstack-meeting | 16:08 | |
| *** mahmoh has joined #openstack-meeting | 16:11 | |
| *** hggdh has quit IRC | 16:15 | |
| *** yehudasa has quit IRC | 16:16 | |
| *** carlp has quit IRC | 16:16 | |
| *** hggdh has joined #openstack-meeting | 16:18 | |
| *** adjohn has quit IRC | 16:19 | |
| *** dendrobates is now known as dendro-afk | 16:20 | |
| *** danwent has joined #openstack-meeting | 16:22 | |
| *** adjohn has joined #openstack-meeting | 16:23 | |
| *** adjohn has quit IRC | 16:23 | |
| *** dolphm has quit IRC | 16:25 | |
| *** dragondm has joined #openstack-meeting | 16:26 | |
| *** dendro-afk is now known as dendrobates | 16:27 | |
| *** Ravikumar_hp has joined #openstack-meeting | 16:36 | |
| *** dwalleck has joined #openstack-meeting | 16:36 | |
| *** dwalleck_ has joined #openstack-meeting | 16:37 | |
| *** dwalleck has quit IRC | 16:41 | |
| *** nati has joined #openstack-meeting | 16:51 | |
| *** donaldngo_hp has joined #openstack-meeting | 16:56 | |
| *** rickl has joined #openstack-meeting | 16:57 | |
| *** zul has quit IRC | 16:59 | |
| nati | Hi all! | 17:00 |
|---|---|---|
| AntoniHP | Hi | 17:00 |
| jaypipes | nati: hi! :) | 17:00 |
| dwalleck_ | Hello! | 17:00 |
| jaypipes | AntoniHP: g'day | 17:00 |
| nati | jaypipes: U 2! | 17:00 |
| donaldngo_hp | hello | 17:00 |
| jaypipes | donaldngo_hp: morning | 17:01 |
| Ravikumar_hp | good morning Jay | 17:01 |
| *** rnirmal has quit IRC | 17:01 | |
| *** oubiwann has joined #openstack-meeting | 17:01 | |
| jaypipes | we'll start in a couple minutes... | 17:02 |
| nati | k | 17:02 |
| jaypipes | Ravikumar_hp: is Nayna around? | 17:05 |
| jaypipes | #startmeeting | 17:05 |
| openstack | Meeting started Wed Jan 11 17:05:19 2012 UTC. The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:05 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic. | 17:05 |
| jaypipes | OK, so there are two separate topics I'd like to discuss first: | 17:06 |
| Ravikumar_hp | nayna is another meeting | 17:06 |
| jaypipes | 1) Test case style | 17:06 |
| *** renuka has joined #openstack-meeting | 17:06 | |
| jaypipes | 2) Using code from novaclient for basic REST client | 17:06 |
| jaypipes | both topics based on email from AntoniHP today... | 17:06 |
| *** dwalleck has joined #openstack-meeting | 17:07 | |
| *** dwalleck_ has quit IRC | 17:07 | |
| jaypipes | #topic Test case style | 17:07 |
| *** openstack changes topic to "Test case style" | 17:07 | |
| jaypipes | so, has everyone seen AntoniHP's email from this morning? | 17:08 |
| *** heckj has joined #openstack-meeting | 17:08 | |
| AntoniHP | here is link https://lists.launchpad.net/openstack-qa-team/msg00023.html | 17:08 |
| jaypipes | there are a number of points AntoniHP raises, and I think we should discuss them here now in order. AntoniHP, ok with you? | 17:09 |
| Ravikumar_hp | I have not seen . may be not sent to the gruo email | 17:09 |
| dwalleck | It went to the group email list, which seems a bit flaky for some reason | 17:09 |
| nati | [Openstack-qa-team] Implementing tests ? | 17:09 |
| jaypipes | Ravikumar_hp: yeah, I did not get notified of it either... see link above. | 17:10 |
| jaypipes | #link https://lists.launchpad.net/openstack-qa-team/msg00023.html | 17:10 |
| Ravikumar_hp | ok | 17:10 |
| AntoniHP | I think yes, we can discuss it here, or if more time is needed discussus it on the end of meeting? | 17:10 |
| *** carlp has joined #openstack-meeting | 17:10 | |
| *** yehudasa has joined #openstack-meeting | 17:10 | |
| jaypipes | AntoniHP: so, let's start with 1) Dependability of test case on each other | 17:10 |
| heckj | AntoniHP has a good point - there is a NOSE test driver that alleviates some of that concern however: http://pypi.python.org/pypi/proboscis/1.0 | 17:12 |
| heckj | #link http://pypi.python.org/pypi/proboscis/1.0 | 17:12 |
| jaypipes | heckj: there is no need for a test driver... | 17:12 |
| jaypipes | heckj: nose already supports skipping based on conditions just fine | 17:12 |
| jaypipes | heckj, AntoniHP: as an example, see https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_servers.py | 17:12 |
| heckj | it is an addition to Nose - an extension, not a test driver, that allows specifications of dependencies between tests | 17:12 |
| nati | I suppose we already discussed about Dependability. Class way and method way | 17:13 |
| nati | Is this different topic | 17:13 |
| nati | ? | 17:13 |
| dwalleck | I think the core issue is not whether we can have dependencies between tests, but should we | 17:13 |
| heckj | jaypipes: please take a look at it before you just dismiss it. I've been using it with some success to resolve some dependency issues between tests | 17:13 |
| jaypipes | heckj: I have looked at it. :) | 17:13 |
| *** johnpur has joined #openstack-meeting | 17:13 | |
| nati | heckj: This moudle looks cool. | 17:14 |
| heckj | dwalleck - is that the concern? If so, apologies for the random link. It wasn't clear from AntoniHP's email | 17:14 |
| dwalleck | I think the better question is "what is the problem we are trying to solve by having test dependencies" | 17:14 |
| jaypipes | so, basically, I'd like to know from AntoniHP what about nosetests does not allow for 1) to be taken care of.. | 17:15 |
| *** vandana has joined #openstack-meeting | 17:15 | |
| dwalleck | heckj: No, but the "should we" question is the one we need to answer here as a group | 17:15 |
| *** sleepsonthefloo has joined #openstack-meeting | 17:15 | |
| jaypipes | right | 17:15 |
| AntoniHP | it does, within nosetests the tests could be easily depndable | 17:15 |
| dwalleck | I think we have some philosophical differences on test design, so the goal is to find a solution that will either directly address everyones concerns, or allow people to use Tempest in different ways to ease those concerns | 17:16 |
| nati | Dependency could be occur by nature when we wanna resuse testcode or exisiting resource. | 17:16 |
| AntoniHP | it is possible to be implemented in different ways, within nose, without nose, with extra driver etc | 17:17 |
| nati | So I think this is a matter of code style. "Class vs Method" | 17:17 |
| dwalleck | So if we want to reuse existing resources, wouldn't it be easier to have an external library/process handling that? | 17:17 |
| AntoniHP | but as nati says this is about style, what nose test case should NOT be equal to test case | 17:18 |
| dwalleck | It seems like that would be a more robust solution, and addresses the concern of execution time and reuse of resources | 17:18 |
| nati | hmm. I suppose we are doing same discussion. I suppose both of Class style and Method style has merit and demerit. | 17:19 |
| jaypipes | AntoniHP: I guess I'm failing to see how the 001_, 002_ test method examples in your email would be of benefit over something like this: http://pastebin.com/2pdV34Ph | 17:19 |
| nati | Then I think our next action is discuss with actual code example. | 17:19 |
| nati | then vote it? | 17:20 |
| dwalleck | But if reuse of resources isn't the core problem, help me understand where the desire for test dependencies comes from | 17:20 |
| AntoniHP | if there is some code between asserts and first assert fails then code is not executed | 17:20 |
| dwalleck | nati: Actually, I think the easier idea would be for someone to submit a patch to Tempest | 17:20 |
| dwalleck | That way it can follow the traditional code acceptance path, and make it easily visible | 17:20 |
| jaypipes | AntoniHP: but why would you have separate test methods for things that are dependent on each other to be done in an order? | 17:20 |
| nati | dwalleck: Yes. I suppose reuse is core problem and right way to solve it is use libs. | 17:20 |
| jaypipes | AntoniHP: you can't parallelize them, and so you only add fragility to the test case because now dependencies between test methods must be tracked. | 17:21 |
| jaypipes | AntoniHP: the only advantage that your approach gives is more verbose output on successful test methods, AFAICT | 17:22 |
| dwalleck | nati: I'm working on that solution as part of my next sprint. There's varying levels of complexity to how it could be implemented, but it will be done in some form or fashion | 17:22 |
| AntoniHP | jaypipes: they can be paralellized as classes | 17:22 |
| jaypipes | AntoniHP: and I'm confused why anyone would care about successful methods -- I only care about stuff that errors or fails? | 17:22 |
| dwalleck | jaypipes: ++ | 17:22 |
| Ravikumar_hp | jaypipes +1 | 17:23 |
| jaypipes | AntoniHP: but in the case where you put a series of dependent actions in a single test method, the methods of a class can be run in parallel even with a shared resource... | 17:23 |
| Ravikumar_hp | i care sucess or failed . | 17:23 |
| AntoniHP | jaypipes: it provides context to results | 17:23 |
| jaypipes | AntoniHP: how so? | 17:23 |
| jaypipes | AntoniHP: could you provide example? | 17:23 |
| Ravikumar_hp | i want to fail dependent tests also instead of skiipinf | 17:23 |
| vandana | AntoniHP: most of the asserts are dependent on the previous assert to have passed so would it be useful to run the second assertion if the first one failed? | 17:23 |
| Ravikumar_hp | skiiping | 17:23 |
| nati | I think by using class way. The test log looks more easy to read without adding logging code. | 17:24 |
| nati | #sorry typo | 17:24 |
| nati | The merit of Class way is the log looks more easy to read without adding logging code. | 17:24 |
| AntoniHP | jaypipes: sometimes yes, and sometimes no - this also provides easy entry point for automated handling of errors | 17:24 |
| dwalleck | AntonioHP: So what if, regardless of test design practice, you could see the results of all assertions in the results. Is that the goal you're trying to reach? | 17:24 |
| AntoniHP | vandana: yes, in my example a failed response to API call could still create new object | 17:25 |
| Ravikumar_hp | for reporting purpose - Success , Failed ... 1) I will fail volume-attachment tests if create volume is failed , | 17:25 |
| AntoniHP | dwalleck: yes, that is why I'm totally not insitent on using this way - I proposed different solutions | 17:25 |
| jaypipes | AntoniHP: sorry, perhaps this is just lost in translation :) could you provide some example output that shows the benefit for automated handling of errors? | 17:26 |
| AntoniHP | create oject call -> verify response from call -> verify that object exists | 17:26 |
| jaypipes | Ravikumar_hp: our point is that if you need to "skip" a dependent set of actions based on an early bailout or failure, the dependent set of actions should be in the same test case method... | 17:27 |
| jaypipes | AntoniHP: but if those calls were in the same test method, the assert message would indicate which step failed... | 17:28 |
| vandana | but won't there be a lot of overhead in figuring out these dependent assertions | 17:28 |
| *** nati_ has joined #openstack-meeting | 17:28 | |
| AntoniHP | so result .F. would point to problems with API, FSS network connectivity, ..F to nova scheduler not working | 17:29 |
| *** nati has quit IRC | 17:29 | |
| AntoniHP | and then .FF would be different to .F. | 17:29 |
| jaypipes | AntoniHP: but so would a single F, with the error output message indicating the step that failed... | 17:29 |
| dwalleck | jaypipes: Right, like you did with images based on what's in the system. It makes sense for the test suite to be aware of it's surroundings and resources | 17:29 |
| dwalleck | So right now I get failures like this.... | 17:30 |
| dwalleck | ====================================================================== | 17:30 |
| dwalleck | ERROR: The server should be power cycled | 17:30 |
| dwalleck | ---------------------------------------------------------------------- | 17:30 |
| dwalleck | Traceback (most recent call last): | 17:30 |
| dwalleck | File "/var/lib/jenkins/jobs/zodiac chicago smoke/workspace/zodiac/zodiac/tests/servers/test_server_actions.py", line 33, in setUp | 17:30 |
| dwalleck | self.server = ServerGenerator.create_active_server() | 17:30 |
| dwalleck | File "/var/lib/jenkins/jobs/zodiac chicago smoke/workspace/zodiac/zodiac/tests/__init__.py", line 27, in create_active_server | 17:30 |
| dwalleck | client.wait_for_server_status(created_server.id, 'ACTIVE') | 17:30 |
| dwalleck | File "/var/lib/jenkins/jobs/zodiac chicago smoke/workspace/zodiac/zodiac/services/nova/json/servers_client.py", line 193, in wait_for_server_status | 17:30 |
| dwalleck | raise exceptions.BuildErrorException('Build failed. Server with uuid %s entered ERROR status.' % server_id) | 17:30 |
| dwalleck | BuildErrorException: u"u'Build failed. Server with uuid e0845137-61d7-48b8-9db8-128db00cd7b5 entered ERROR status.' | 17:31 |
| AntoniHP | we aim to automate, so if such logic is not in test, we would need to parse output messages then | 17:31 |
| dwalleck | Ack | 17:31 |
| dwalleck | https://gist.github.com/3da4cc395268f5ca36cb | 17:31 |
| dwalleck | Try that instead, bit easier to read :) | 17:31 |
| jaypipes | AntoniHP: automate what exactly? the reading of test results to put on some report? Then we can just use xunit output, no? | 17:31 |
| AntoniHP | jaypipes: exactly ! | 17:31 |
| AntoniHP | jaypipes: by having separate entries in units, we do not need to be very smart about parsing error messages | 17:32 |
| jaypipes | AntoniHP: but all that would mean is a simple --with-xunit | 17:32 |
| AntoniHP | jaypipes: no, because you need to parse a message in result | 17:32 |
| AntoniHP | to see which case has happend | 17:32 |
| dwalleck | I can't really post a link to my Jenkins reports, but that's pretty much what I have now with the --with-xunit | 17:33 |
| AntoniHP | otherwise you have a code that pinpoints failure, and captured output could be used for technical data | 17:33 |
| jaypipes | AntoniHP: If your automation depends on the sequence of E, F, ., and S in the test output, then something is more fundamentally wrong than the order of the test method execution IMHO | 17:33 |
| nati_ | jaypipes: Create server sometimes fails sometime oK. And some test fails not because of this. | 17:34 |
| jaypipes | AntoniHP: for instance, what happens when you insert a new 00X method and F.. becomes F.F.? How does your automated reporting handle that? | 17:34 |
| jaypipes | AntoniHP: you would have to make a corresponding change to your automation report, no? | 17:35 |
| AntoniHP | that is question to donaldngo_hp | 17:35 |
| jaypipes | nati_: that's a totally separate issue :) | 17:35 |
| AntoniHP | but still this allows for a) continuing execution of following steps | 17:36 |
| jaypipes | AntoniHP: I guess my hesitation is to change to a test class/method style in order to just support a certain type of output to the test run. | 17:36 |
| donaldngo_hp | cant we achieve what antoni wants (which is each test class is a testing scenario with dependent steps) and what tempest provides which is code resusabitlity through service classes | 17:36 |
| AntoniHP | jaypipes: I agree with that statement | 17:36 |
| dwalleck | AntonioHP: My case is that if an assertion fails, I probably don't want to make any more assertions, and the rest will likely fail and dirty my results | 17:37 |
| jaypipes | donaldngo_hp: but what we are saying is that test *methods* should contain all dependent series of actions, not the class. That way, there is no need to have dependency tracking. | 17:37 |
| AntoniHP | I think fundamentally this problems goes from using unit testing framework for working on other types of tests | 17:37 |
| jaypipes | AntoniHP: virtually every functional/integration testing framework derives from unit test frameworks. | 17:37 |
| vandana | dwalleck: +1 | 17:37 |
| Ravikumar_hp | jaypipes: yes . current test methods already contains dependent actions. | 17:38 |
| AntoniHP | yes, but I have a feeling that we are still quite bound by thinking of those tests as single unit tests | 17:38 |
| jaypipes | AntoniHP: they aren't. :) test *methods* are single functional tests. | 17:38 |
| donaldngo_hp | jaypipes: there will be still dependencies in one way or another. in antonis approach i think its a logical grouping of steps to run. using methods you still have to keep track that you need to do a before b before c ect | 17:39 |
| jaypipes | AntoniHP: with the test class housing shared resources the test methods use. | 17:39 |
| AntoniHP | jaypipes: my proposal is test *classes* are single functional tests with few test *methods* or generators (implementation detail) | 17:39 |
| *** carlp has quit IRC | 17:39 | |
| dwalleck | I don't think switching frameworks would solve that problem. Whether unit, integration, or functional, each test has a goal. It makes assertions towards that result and then ends | 17:39 |
| jaypipes | donaldngo_hp: no, that's wrong. if the test method is a series of dependent actions, assertions in a or b will mean c will not be executed... | 17:40 |
| jaypipes | donaldngo_hp: sorry, shouldn't say that's "wrong"... :) just my opinion... | 17:40 |
| dwalleck | I think what we're talking about only applies to some tests as well. I'm not sure I could see that style in use for negative tests, say one verifying that if I use an invalid image, a server won't build | 17:40 |
| dwalleck | Would it be fair to say that we care most about these results for the more core/smoke/positive tests that we have? | 17:41 |
| dwalleck | err, more=most | 17:41 |
| jaypipes | AntoniHP: and in your approach, if test method 002_xxx failed, then test method 003_xxx should be skipped, right? | 17:41 |
| *** carlp has joined #openstack-meeting | 17:41 | |
| AntoniHP | no | 17:41 |
| AntoniHP | if 1) fails then 2) and 3) skips, if 1) succeeds then 2) and 3) execute | 17:42 |
| AntoniHP | because we can get malformed API response, yet be able to actually boot VM | 17:42 |
| dwalleck | but then if 3 depends on 2 and it not skipped, 3 will fail, which would be a false positive | 17:42 |
| jaypipes | AntoniHP: right, and we are arguing that those three steps are assertions that should be in a single test method called test_basic_vm_launch(), otherwise you need to add stuff to test case framework to handle dependencies between test methods | 17:43 |
| AntoniHP | dwalleck: yes, there are different scenarious possible, sometimes test would be just like assertions and sometimes not | 17:43 |
| dwalleck | I think we need code here... | 17:43 |
| *** gyee has joined #openstack-meeting | 17:43 | |
| AntoniHP | jaypipes: that is false statement, generators allow to execute logical flows wihtout any nosetest additions | 17:44 |
| dwalleck | I can assert that I can create tests that give the same results, but are not dependent | 17:44 |
| jaypipes | AntoniHP: and putting the assertions in a single test method allows to execute logical flows without generators ;) | 17:44 |
| dwalleck | Which then break the dependency chain, allow for class level parallel execution, and for isolated test execution | 17:44 |
| AntoniHP | jaypipes: no, because first raised assetion stops the test method | 17:45 |
| jaypipes | AntoniHP: yes, that's what you want! | 17:45 |
| AntoniHP | jaypipes: not in case of integration test, as I mentioned before malformed response from REST call does not indicate final result of initial call | 17:45 |
| dwalleck | AntoniHP: So how about this...why not re-write some of the core servers tests in the style you propose | 17:45 |
| jaypipes | AntoniHP: that's a totally diffrerent test than "launch this VM in a normal operation" though :( | 17:46 |
| dwalleck | That way we're talking about concrete things instead of concepts | 17:46 |
| jaypipes | dwalleck: ++ | 17:46 |
| AntoniHP | ok, I will do this | 17:46 |
| dwalleck | I think it would be easier to be able to put this all on the table and compare things with real world examples | 17:46 |
| jaypipes | agreed | 17:47 |
| dwalleck | And that way we can see the results, compare the output, and see what is different and/or lacking | 17:47 |
| jaypipes | alrighty, let's let AntoniHP put some example code up to a pastebin/gist... | 17:48 |
| dwalleck | Until then, I don't think further discussion will help much | 17:48 |
| AntoniHP | ok | 17:48 |
| *** hggdh has quit IRC | 17:49 | |
| dwalleck | Good, I think that will help quite a bit | 17:52 |
| donaldngo_hp | how about we set up some time where we can see the code on someones desktop? i think we would all reach the end goal a lot faster then our current approach of code pasting | 17:52 |
| *** oubiwann1 has joined #openstack-meeting | 17:52 | |
| nati_ | donaldngo_hp++ | 17:53 |
| *** carlp has quit IRC | 17:53 | |
| dwalleck | donaldngo_hp: I think that's a good idea. I'd still like to have a chance to see and run it before as well | 17:53 |
| donaldngo_hp | we can discuss real time | 17:53 |
| *** derekh has quit IRC | 17:53 | |
| jaypipes | donaldngo_hp: I think I'd actually prefer pastes and the public mailing list for discussion... | 17:54 |
| dwalleck | And it may help if I also share what the tempest results I'm using now look like. I think there's quite a bit that comes out of the --with-x-unit results that are fairly helpful | 17:54 |
| *** oubiwann has quit IRC | 17:55 | |
| donaldngo_hp | dwallect: would love to see what your report looks like | 17:56 |
| dwalleck | awesome, I'll find a way to get that viewable | 17:56 |
| jaypipes | donaldngo_hp: are you using xUnit output for the feed into your reports? | 17:56 |
| dwalleck | And then we can see better what we have, and what is missing | 17:56 |
| donaldngo_hp | jaypies: yea we are using xunit to product xml and then aggregate into junit style report | 17:57 |
| donaldngo_hp | *produce | 17:57 |
| *** AlanClark has joined #openstack-meeting | 17:57 | |
| jaypipes | donaldngo_hp: k | 17:57 |
| *** carlp has joined #openstack-meeting | 17:57 | |
| dwalleck | donaldngo_hp: Ahh, then you're probably seeing pretty much what I am | 17:58 |
| donaldngo_hp | i can send our report out to the group as well | 17:58 |
| jaypipes | donaldngo_hp: yes, please do! :) | 17:58 |
| jaypipes | let's use the main mailing list, though, with a [QA] subject prefix... the team mailing list is proving unreliable at best :( | 17:58 |
| dwalleck | Though I'd like to add more...for example, my devs love that I say that a server failed to build, but without more info (the server id, IP, etc), it's not much help. I'm trying a few things to make that better | 17:59 |
| * dwalleck ideally would like to pull error logs directly from the core system, but not today | 17:59 | |
| *** yehudasa has quit IRC | 17:59 | |
| *** carlp has quit IRC | 17:59 | |
| jaypipes | dwalleck: that's what the exceptions proposed branch starts to address :) | 17:59 |
| dwalleck | jaypipes: yup! | 18:00 |
| jaypipes | dwalleck: and I've been creating a script that does a relevant log file read when running tempest against devstack... | 18:00 |
| *** yehudasa has joined #openstack-meeting | 18:00 | |
| *** yehudasa has quit IRC | 18:01 | |
| dwalleck | jaypipes: I was thinking along the lines of that. It's good start, but I'm afraid of how verbose it could be | 18:01 |
| jaypipes | dwalleck: well, the script I have grabs relevant logs, cuts them to the time period of the test run, and then tars them up :) | 18:02 |
| *** yehudasa has joined #openstack-meeting | 18:02 | |
| *** rnirmal has joined #openstack-meeting | 18:02 | |
| jaypipes | dwalleck: figured it would be useful for attaching the tarball to bug reports, etc | 18:02 |
| dwalleck | nice! | 18:03 |
| dwalleck | That sounds very useful | 18:03 |
| jaypipes | dwalleck: yeah, just can't decide whether the script belongs in devstack/ or tempest! | 18:03 |
| dwalleck | good question | 18:03 |
| *** jdurgin has joined #openstack-meeting | 18:04 | |
| AntoniHP | when you decide can you post link to it on the list? | 18:04 |
| dwalleck | sounds like an alternate plugin for tempest for those using devstack | 18:04 |
| *** mdomsch has quit IRC | 18:04 | |
| jaypipes | dwalleck: speaking of that... one other thing we all should decide on once we come to consensus on the style stuff is when to have tempest start gating trunk :) currently, only some exercises in devstack are gating trunk IIRC... | 18:04 |
| *** nati_ has quit IRC | 18:04 | |
| jaypipes | AntoniHP: absolutely! | 18:04 |
| dwalleck | jaypipes: I was thinking of the same thing. When I saw the gating trunk email, I was excited until I realized it wasn't on Tempest :) | 18:05 |
| jaypipes | dwalleck: right.. I added the tempest/tools/conf_from_devstack script recently to allow someone to generate a tempest config file from a devstack installation... very useful after running stack.sh ;) | 18:05 |
| jaypipes | dwalleck: since stack.sh wipes everything and installs new base images, which are needded in the tempest conf :) | 18:06 |
| * jaypipes needs to blog about that... | 18:06 | |
| *** carlp has joined #openstack-meeting | 18:06 | |
| dwalleck | Hmm...I would say once we can confidently say we have a solid set of smoke tests that we consider to be reliable. That seems like a reasonable goal | 18:06 |
| jaypipes | dwalleck: ++ | 18:07 |
| jaypipes | dwalleck: we're getting there... | 18:08 |
| dwalleck | I think we're close. The one thing I'm wrestling with is that it's bit hard to visualize coverage based on the bug list in Launchpad | 18:08 |
| jaypipes | dwalleck: agreed. though the tags help a bit.. | 18:09 |
| *** novas0x2a|laptop has joined #openstack-meeting | 18:09 | |
| dwalleck | jaypipes: They do. I'm still going to keep bouncing that idea around in my head | 18:10 |
| *** jdurgin has quit IRC | 18:10 | |
| dwalleck | Well good folks I need to bow out, off to the next meeting | 18:10 |
| *** yehudasa has quit IRC | 18:10 | |
| *** yehudasa has joined #openstack-meeting | 18:11 | |
| dwalleck | Or not :) Jay is stepping away for a sec | 18:11 |
| dwalleck | nati_: Are you still here? | 18:12 |
| jaypipes | AntoniHP: how about we just comment on the code on the mailing list? ok with you? | 18:13 |
| *** jdurgin has joined #openstack-meeting | 18:13 | |
| AntoniHP | I think using list would be more productive, as it is less interactive and code needs time to be read | 18:13 |
| jaypipes | AntoniHP: not a problem. | 18:13 |
| jaypipes | ok good discussion so far, we will continue on the ML. | 18:13 |
| jaypipes | #endmeeting | 18:13 |
| *** openstack changes topic to "Openstack Meetings: http://wiki.openstack.org/Meetings | Minutes: http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/" | 18:13 | |
| openstack | Meeting ended Wed Jan 11 18:13:56 2012 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:13 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-01-11-17.05.html | 18:13 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-01-11-17.05.txt | 18:14 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-01-11-17.05.log.html | 18:14 |
| dwalleck | sounds good! | 18:14 |
| *** dwalleck has quit IRC | 18:14 | |
| *** Ravikumar_hp has quit IRC | 18:14 | |
| AntoniHP | ok, then c u on mailing list | 18:15 |
| *** Ravikumar_hp has joined #openstack-meeting | 18:15 | |
| *** jdurgin has quit IRC | 18:16 | |
| *** yehudasa has quit IRC | 18:17 | |
| *** carlp has quit IRC | 18:17 | |
| *** bengrue has joined #openstack-meeting | 18:18 | |
| *** rickl has quit IRC | 18:18 | |
| *** yehudasa has joined #openstack-meeting | 18:18 | |
| *** yehudasa has quit IRC | 18:19 | |
| *** carlp has joined #openstack-meeting | 18:20 | |
| *** yehudasa has joined #openstack-meeting | 18:20 | |
| *** yehudasa has quit IRC | 18:21 | |
| *** carlp has quit IRC | 18:21 | |
| *** jdurgin has joined #openstack-meeting | 18:22 | |
| *** Ravikumar_hp has quit IRC | 18:22 | |
| *** yehudasa has joined #openstack-meeting | 18:22 | |
| *** jdurgin has quit IRC | 18:22 | |
| *** darraghb has quit IRC | 18:23 | |
| *** yehudasa has quit IRC | 18:23 | |
| *** yehudasa has joined #openstack-meeting | 18:24 | |
| *** vandana has quit IRC | 18:24 | |
| *** yehudasa has quit IRC | 18:24 | |
| *** dendrobates is now known as dendro-afk | 18:25 | |
| *** vandana has joined #openstack-meeting | 18:25 | |
| *** carlp has joined #openstack-meeting | 18:26 | |
| *** yehudasa has joined #openstack-meeting | 18:26 | |
| *** yehudasa has quit IRC | 18:27 | |
| *** yehudasa has joined #openstack-meeting | 18:27 | |
| *** jdurgin has joined #openstack-meeting | 18:27 | |
| *** dendro-afk is now known as dendrobates | 18:28 | |
| *** jdurgin has quit IRC | 18:28 | |
| *** carlp has quit IRC | 18:28 | |
| *** yehudasa has quit IRC | 18:28 | |
| *** yehudasa has joined #openstack-meeting | 18:29 | |
| *** sean_roberts has joined #openstack-meeting | 18:29 | |
| *** yehudasa has quit IRC | 18:30 | |
| *** yehudasa has joined #openstack-meeting | 18:31 | |
| *** jdurgin has joined #openstack-meeting | 18:32 | |
| *** yehudasa has quit IRC | 18:32 | |
| *** carlp has joined #openstack-meeting | 18:32 | |
| *** jdurgin has quit IRC | 18:33 | |
| *** carlp has quit IRC | 18:33 | |
| *** yehudasa has joined #openstack-meeting | 18:33 | |
| *** yehudasa has quit IRC | 18:34 | |
| *** yehudasa has joined #openstack-meeting | 18:35 | |
| *** carlp has joined #openstack-meeting | 18:35 | |
| *** yehudasa_ has joined #openstack-meeting | 18:37 | |
| *** yehudasa has quit IRC | 18:37 | |
| *** carlp has quit IRC | 18:37 | |
| *** donaldngo_hp has quit IRC | 18:38 | |
| *** yehudasa_ has quit IRC | 18:38 | |
| *** yehudasa_ has joined #openstack-meeting | 18:38 | |
| *** jdurgin has joined #openstack-meeting | 18:38 | |
| *** jdurgin has quit IRC | 18:39 | |
| *** yehudasa_ has quit IRC | 18:39 | |
| *** carlp has joined #openstack-meeting | 18:39 | |
| *** yehudasa_ has joined #openstack-meeting | 18:40 | |
| *** carlp has quit IRC | 18:40 | |
| *** adjohn has joined #openstack-meeting | 18:40 | |
| *** yehudasa_ has quit IRC | 18:41 | |
| *** yehudasa_ has joined #openstack-meeting | 18:41 | |
| *** jdurgin has joined #openstack-meeting | 18:42 | |
| *** yehudasa_ has quit IRC | 18:42 | |
| *** yehudasa_ has joined #openstack-meeting | 18:44 | |
| *** jdurgin has quit IRC | 18:44 | |
| *** yehudasa__ has joined #openstack-meeting | 18:45 | |
| *** yehudasa_ has quit IRC | 18:45 | |
| *** carlp has joined #openstack-meeting | 18:46 | |
| *** yehudasa__ has quit IRC | 18:46 | |
| *** jdurgin has joined #openstack-meeting | 18:48 | |
| *** carlp has quit IRC | 18:48 | |
| *** carlp has joined #openstack-meeting | 18:49 | |
| *** jdurgin has quit IRC | 18:49 | |
| *** carlp has quit IRC | 18:52 | |
| *** heckj has quit IRC | 18:53 | |
| *** jdurgin has joined #openstack-meeting | 18:55 | |
| *** donaldngo_hp has joined #openstack-meeting | 18:57 | |
| *** jdurgin has quit IRC | 18:57 | |
| *** jdurgin has joined #openstack-meeting | 18:58 | |
| *** jdurgin has quit IRC | 18:59 | |
| *** deshantm has quit IRC | 19:01 | |
| *** rnirmal has quit IRC | 19:01 | |
| *** bcwaldon has left #openstack-meeting | 19:02 | |
| *** jdurgin has joined #openstack-meeting | 19:05 | |
| *** jdurgin has quit IRC | 19:07 | |
| *** jdurgin has joined #openstack-meeting | 19:09 | |
| *** jdurgin has quit IRC | 19:10 | |
| *** jdurgin has joined #openstack-meeting | 19:15 | |
| *** jdurgin has quit IRC | 19:15 | |
| *** dolphm has joined #openstack-meeting | 19:17 | |
| *** jdurgin has joined #openstack-meeting | 19:21 | |
| *** jdurgin has quit IRC | 19:24 | |
| *** jdurgin has joined #openstack-meeting | 19:26 | |
| *** mahmoh has left #openstack-meeting | 19:35 | |
| *** rnirmal has joined #openstack-meeting | 19:53 | |
| *** renuka has quit IRC | 19:56 | |
| *** AlanClark has quit IRC | 20:03 | |
| *** danwent has left #openstack-meeting | 20:32 | |
| *** yehudasa has joined #openstack-meeting | 20:47 | |
| *** dwalleck has joined #openstack-meeting | 20:59 | |
| *** dwalleck has joined #openstack-meeting | 21:05 | |
| *** bengrue has quit IRC | 21:09 | |
| *** AlanClark has joined #openstack-meeting | 21:16 | |
| *** bhall has joined #openstack-meeting | 21:18 | |
| *** bhall has joined #openstack-meeting | 21:18 | |
| *** reed has joined #openstack-meeting | 21:20 | |
| *** dwalleck has quit IRC | 21:29 | |
| *** dprince has quit IRC | 21:32 | |
| *** heckj has joined #openstack-meeting | 21:34 | |
| *** rnirmal has quit IRC | 21:57 | |
| *** _adjohn has joined #openstack-meeting | 21:58 | |
| *** adjohn has quit IRC | 21:58 | |
| *** _adjohn is now known as adjohn | 21:58 | |
| *** dwalleck has joined #openstack-meeting | 22:09 | |
| *** adjohn has quit IRC | 22:20 | |
| *** adjohn has joined #openstack-meeting | 22:21 | |
| *** Kiall_ has quit IRC | 22:26 | |
| *** dolphm has quit IRC | 22:27 | |
| *** dwalleck has quit IRC | 22:32 | |
| *** markvoelker has quit IRC | 22:44 | |
| *** adjohn has quit IRC | 23:01 | |
| *** reed has quit IRC | 23:01 | |
| *** sleepsonthefloo has quit IRC | 23:01 | |
| *** sleepsonthefloo has joined #openstack-meeting | 23:05 | |
| *** reed has joined #openstack-meeting | 23:05 | |
| *** adjohn has joined #openstack-meeting | 23:06 | |
| *** mattray has quit IRC | 23:43 | |
| *** AlanClark has quit IRC | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!