22:01:05 #startmeeting qa 22:01:06 o/ 22:01:06 Meeting started Thu Mar 5 22:01:05 2015 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:11 The meeting name has been set to 'qa' 22:01:16 Hi, who's here today? 22:01:32 hi 22:01:39 hi 22:01:43 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_March_5th_2015_.282200_UTC.29 22:01:43 o/ 22:01:48 ^^^ Today's agenda 22:02:08 hi 22:02:45 o/ 22:03:00 sdague, jogo: courtesy ping 22:03:04 ok let's get started 22:03:14 o/ 22:03:18 #topic Tempest test removal proposal reviews 22:03:40 o/ 22:03:46 so this meeting is the first meeting with tempest test removals proposed per: 22:03:49 #link https://wiki.openstack.org/wiki/QA/Tempest-test-removal 22:04:00 there are only 2 and I expect them to be quick 22:04:02 o/ 22:04:17 the 2 removals proposed are: 22:04:20 #link https://etherpad.openstack.org/p/tempest-test-removals 22:04:50 the first one on the list is from stevebaker: 22:05:01 tempest.scenario.orchestration 22:05:09 \o 22:05:20 which is removing the heat scenario tests, which have been skipped since the begining of the cycle 22:05:35 seems uncontroversial to remove 22:05:37 (so step 2 doesn't really apply) 22:05:42 stevebaker: yep 22:05:54 does anyone have an objection to removing this? 22:06:22 nope, lets do it! 22:06:22 mtreinish: nope 22:06:26 no 22:06:27 why have they been skipped? 22:06:32 (and no objection) 22:06:57 jogo: transient failures at the time due to slow nodes. we're not seeing the same issue in heat functional tests 22:07:00 ok I'll remove my -2 and we can merge it 22:07:18 jogo: it says bug 1374175 22:07:19 bug 1374175 in heat "test_server_cfn_init failed in gate-tempest-dsvm-neutron-heat-slow: AssertionError: Timed out waiting for 10.1.0.4 to become reachable" [Undecided,Incomplete] https://launchpad.net/bugs/1374175 - Assigned to Steven Hardy (shardy) 22:07:21 can we quickly talk abou removal of heat cli test? 22:07:32 stevebaker: ahh makes sense 22:07:46 stevebaker: can we wait until after the other removal on the etherpad? 22:08:49 sure 22:09:07 the other proposed removal from the etherpad is from sdague: 22:09:23 tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor and tempest.api.compute.admin.test_flavors_negative.FlavorCreateNegativeTestJSON 22:09:34 yeh, it was updated this morning to delete the whole test file 22:09:57 as I now have the ability to test the user/admin split in the nova patch series as well 22:10:08 oh, ok I didn't see that but that doesn't make much of a difference 22:10:18 the Depends-On link was updated to the correct point in the nova patch series 22:10:27 yeh, should be the same 22:10:30 sdague: that link doesn't look correct 22:10:34 it doesn't go anywhere 22:10:51 apparently, truncated 22:10:53 let me fix 22:11:04 sure it's just missing the "I" in front 22:11:22 yeh 22:11:24 it was 22:11:32 ok, updated 22:11:45 but aside from the typo based on the numbers these tests seem like something else that's not really controversial 22:12:11 especially given none of these tests have failed once since Nov. in the gate 22:12:25 (according to the numbers in the etherpad) 22:12:38 does anyone have an objection to removing the flavor negative tests? 22:13:05 no, +1 from me 22:13:06 nope 22:13:09 no 22:13:19 no. +1 22:13:41 ok, then I'll remove my -2 and let it move forward 22:14:01 stevebaker: ok, what did you want to ask about the heat cli tests? 22:14:44 also, the heat test removal needs a rebase, because of idempotent_id land 22:14:46 rather than creating a new python-heatclient functional job I'd like to move the test to the heat functional job, and make python-heatclient gate on that 22:15:11 heat already depends on python-heatclient 22:15:52 that was all, I just wanted to mention in case there are objections 22:16:03 that seems confusing to me, but I think how you build those jobs is pretty much the heat team's call 22:16:15 stevebaker: well how you gate is up to you guys. BUt that seems weird to me, unless you are running the heats tests on every heatclient patch 22:16:34 I think the important thing is to get the cli calls out of tempest, because they don't really fit there 22:16:35 but it's your call 22:16:36 we would, they would co-gate 22:16:47 stevebaker: ok 22:17:31 ok then does anyone else have something to discuss on tempest test removals? Otherwise let's move on 22:18:25 #topic Devstack 22:18:48 dtroyer: so anything new and exciting from devstack this week? 22:19:08 I don't have much on my list as I've been focused on catching up with other things this week 22:19:20 I've just proposed to remove lib/dib https://review.openstack.org/#/c/161919/ 22:19:54 stevebaker: hmm I think I remember reading a spec from adam_g doing something similar at some point 22:20:00 I'll take a look after the meeting 22:20:07 dtroyer: ok, sure 22:20:17 it was a dead-end experiment which was never used 22:20:25 i actually had a spec to formalize our use of it, but at this point i think its okay to remove 22:20:35 adam_g: heh, ok 22:20:43 ironic is actually moving away from DIB built ramdisks so... 22:20:47 adam_g: it probably should be an external plugin if you want to use dib there 22:20:50 pip install diskimage-builder seems to do the job 22:20:58 sdague, yea 22:21:09 stevebaker: looks great, thanks! 22:21:23 yay for deleting stuff from devstack :) 22:21:36 indeed! 22:21:38 speaking of devstack plugins, ive been playing with moving the ironic bits out of devstack and into a plugin.. do we have a documented process for deprecating and removing the devstack code? 22:22:10 adam_g: not really, it's been more of just do it :) 22:22:41 okay. i guess we can keep it around with deprecation warnings everytime its sourced for a while 22:23:11 I think that doing it for something that has grenade jobs on it is going to be hard until grenade is modularized for it 22:23:29 sdague: yeah we've hit that already on a few things 22:23:47 but, my brain power has all been on nova this week, so I haven't thought that through yet 22:24:05 adam_g: we do have DEPRECATED_TEXT that is displayed at the end of stack.sh, if you want to add something to that it should be easily seen. 22:24:23 dtroyer, nice, thanks 22:24:54 the only thing I have on devstack this week is jogo and I have been on the quest to get our gate runtime under control. 22:25:16 also, for anyone interested in devstack plugins: https://review.openstack.org/#/c/161535/ deals with installation of distro packages required by plugins. didn't see anything in the spec about that 22:25:41 mostly around pre-caching wheels in nodepool, because the wheel compilation takes a non-negligible amount of time 22:26:00 mtreinish: ++ 22:26:31 non-negligible ~= 5 minutes 22:26:34 we're consistently seeing devstack take ~20+ min at this point, which is no good 22:26:44 yeh, numpy is 3 minutes iirc 22:27:19 ok, does anyone have anything else to discuss on devstack? 22:27:31 nope, I need to drop as well, see folks tomorrow. 22:27:47 o/ 22:28:23 #topic Grenade 22:28:44 I'm not expecting anything for this, things have been pretty quiet on the grenade front 22:28:52 jogo, dtroyer: anything to add? :) 22:29:33 nope 22:29:45 ok does anyone else have anything on grenade? 22:30:40 #topic Tempest Configuration Doc (mtreinish) 22:30:51 ok this is one of the leftover from last week 22:31:10 I wanted to bring up that we've started a tempest configuration guide in the docs 22:31:18 #link http://docs.openstack.org/developer/tempest/configuration.html 22:31:36 right now it's pretty sparse and just contains an overview on how to do the auth/creds setup 22:32:07 but I think considering the number of questions I've been fielding from people struggling with this the past few weeks it would be good if we could expand this 22:32:34 I was hoping people would volunteer to help tackle a bit and add on to it 22:33:06 mtreinish: do you have a list of options causing trouble based on what you were asked? 22:34:04 dkranz: most of the confusion I've seen was around enabled features and available services (which seemed strange to me because those should be the easy ones) 22:34:26 oh and the ec2 configuration throws everyone for a loop 22:34:53 mtreinish: heh 22:34:56 mtreinish, dkranz: as part of the ssh auth bp we have some work on configuration, so we'll update that bit (timeouts, ssh credentials and so) 22:35:54 andreaf: cool 22:36:17 ok does anyone else have something on this topic? 22:36:31 andreaf: +1, ppl get confused for ssh options we have currently. i saw bug for that. 22:36:48 mtreinish: how about bringing it to ML(openstack or openstack-dev)? I don't know how many people are using Tempest, though. 22:37:02 masayukig: +1 that's a good idea 22:37:24 #action mtreinish to bring up the new configuration guide on the ML 22:37:59 masayukig: I think there are many, many people using tempest 22:38:00 andreaf: would you be opposed to documenting what you have to do now for ssh? 22:38:16 dkranz: great! :) 22:39:13 mtreinish: I can give it a go 22:39:20 andreaf: cool, thanks 22:39:39 #action andreaf to work on adding ssh section to the config doc 22:39:48 ok let's move on 22:39:57 #topic new tag for interop tests (hogepodge) 22:40:02 hogepodge: around? 22:40:11 o/ 22:40:33 hogepodge: the floor is yours 22:40:41 I'd like to propose a new tag for tests, working on a blueprint for it right now. 22:41:23 I feel that there's a need for strict api testing that the current set of api tests doesn't meet. 22:42:03 It should be possible to just point tempest at a cloud with only an endpoint and credentials, and maybe a minimum of pre-existing resources (network id and image id, for example) and check for api functionality. 22:42:35 hogepodge: Isn't that what tempest does now? Coverage is a different issue. 22:42:36 My proposal is a new test tag, [interop] which has the requirements of testing api endpoints with only minimum configuration required. 22:43:02 dkranz lots of api tests want to have admin access for a number of different reasons 22:43:11 hogepodge: As I said earlier in IRC, that is going to cover most non-admin tests 22:43:23 hogepodge: yes, but we are fixing that 22:43:39 hogepodge, dkranz: sounds like "smoke" revisited 22:43:40 hogepodge: for the most part if you don't provide admin it should just skip those. (there are bugs there though, we're working through them) 22:44:06 andreaf: I think it is a little different. smoke is supposed to be a "minimal" set of testsw 22:44:19 andreaf: and we need to revisit that as well 22:44:27 if there's overlap, or other spaces that's handled by this, then that's great. 22:44:45 dkranz: well that's one of the goals for the code sprint to clean up the usage of the smoke tag. 22:44:48 The defcore committee is seeing otherwise, though. 22:44:51 hogepodge: no, I think a new tag is needed but it should be "not interop" 22:44:51 But I agree this might be a superset on smoke 22:45:01 err, actually probably just an overlapping set 22:45:11 because smoke will include scenario and this probably won't 22:45:29 hogepodge: there are a few tests that are non-admin but depend on some configuration that is hard for a user to know. 22:45:36 hogepodge: that is how I would interpret this 22:46:07 dkranz: that's part of it. An endpoint, identity, and modest resources should really be sufficient. 22:46:17 hogepodge: the interop tests are those that do not depend on values in tempest.conf that can be different for different deployments 22:46:22 I don't want to chase an idea that doesn't have community support, though. 22:46:37 hogepodge: I don't think it necessarily doesn't have support 22:47:02 hogepodge: well I think write up a spec to flush out the idea. Then we can work through it when there is a solid idea out there 22:47:10 hogepodge: IMO, only a small number of tests need to be tagged, and then if you don't provide admin credentials, only what you want will run 22:47:29 mtreinish: +1 22:48:01 hogepodge: dkranz : mtreinish i am confused about interops requirement, is it tests with minimum conf or a set of functionality to test irrespective of how much conf needed 22:48:12 dkranz: One thing I'd like to see is a pool of tests that can be used to really define interoperability. The defcore and committee and board need guidance on what's important to test and how to measure compatibility between deployments. 22:48:40 hogepodge: isn't that what defcore is supposed to define? 22:49:09 mtreinish: Right now the committee looks at api tests, ranks them against a criteria, then determines the list. 22:49:14 mtreinish: yes but I think the presumption is it can be extracted from existing tempest tests 22:49:15 gmann: I think that's why we need a spec to clearly outline the requirements here 22:49:51 mtreinish: we'd like more involvement from the PTLs and community members on what's important. The assumption is that if an api test exists it expresses importance. 22:50:09 the tests should just be the verification mechanism. I don't think they should be used to define it 22:50:30 but anyway I don't want to get too far off track 22:50:37 does anyone have anything else on this? 22:50:39 mtreinish: it's a chicken and egg problem. 22:50:52 mtreinish: I think we should assume all non-admin tests cover things that are supposed to be interoperable and identity exceptions 22:51:16 mtreinish: I really don't understand why there is controversy here, if there is any 22:51:35 hogepodge: not really, because we can add a test at any point. 22:51:39 but let's move on 22:51:45 dkranz: there are tests which use admin credentials for preparation, but they do not test an admin api, and they might belong to interop 22:51:52 because we're at <10mins 22:52:29 #topic Bugs 22:52:45 #link https://etherpad.openstack.org/p/Tempest-bug-report 22:53:05 gmann: anything new on bugs this week? 22:53:26 masayukig: you had the rotation this week right? 22:53:30 mtreinish: nothing much. masayukig had triage this week 22:54:23 mtreinish, gmann: yes. I've done. 22:54:58 ok anything else on bugs? Otherwise let's move one? 22:55:02 s/one/on 22:55:13 and people volunteered for triage rotation. we have triage rotation fill till Apr 6 22:55:22 that's all from my side 22:55:31 yeah I noticed that, I was surprised :) 22:55:39 :) 22:55:48 #topic Critical Reviews 22:56:06 ok does anyone have a review they'd like to get additional eyes on? 22:56:18 https://review.openstack.org/#/q/topic:+bp/multi-keystone-api-version-tests+status:+open,n,z 22:57:12 #link https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/test-accounts-continued,n,z 22:57:30 mtreinish: you were faster :) 22:57:46 #link https://review.openstack.org/#/q/topic:+bp/resource-cleanup+status:open,n,z 22:57:54 #link https://review.openstack.org/#/q/topic:+bp/multi-keystone-api-version-tests+status:+open,n,z 22:58:30 I actually had a question on the multi-auth bp, but perhaps after the meeting as we're short on time 22:58:48 andreaf: I had look yesterday on keystone one. ll review today 22:58:57 andreaf: sure we can pick that up in -qa after the meeting 22:59:00 gmann: thanks 22:59:06 ok are there any other reviews to bring up? 22:59:26 #link https://review.openstack.org/#/c/158017/ 22:59:45 that's just a doc patch for tempest-lib, but I think it'll be good to make sure I understood the restclient correctly :) 23:00:03 ok if there aren't any other reviews we'll end here 23:00:09 thanks everyone 23:00:26 #endmeeting