09:00:49 <gmann> #startmeeting qa
09:00:50 <openstack> Meeting started Thu Mar  9 09:00:49 2017 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:00:53 <openstack> The meeting name has been set to 'qa'
09:01:04 <gmann> hi, who all here today ?
09:01:06 <samP> hi o/
09:01:13 <gmann> samP: hi.
09:01:14 <samP> This me samapth from NTT
09:01:22 <blancos> o/
09:01:22 <zhufl> hello
09:01:28 <gmann> samP: nice to see you
09:01:40 <samP> gmann: hi..
09:02:00 <chandankumar> gmann: toscalix samP zhufl \o/
09:02:15 <gmann> hi everyone
09:02:31 <gmann> andreaf: masayukig jordanP ?
09:02:43 <tosky> hi
09:02:44 <gmann> masayukig is not around today
09:02:57 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_March_9th_2017_.280900_UTC.29
09:03:03 <gmann> ^^ today agenda
09:03:24 * masayukig is here but read-only-mode...
09:03:52 <andreaf> o/
09:03:57 <gmann> we have Pike priority here - #link https://etherpad.openstack.org/p/pike-qa-priorities
09:04:17 <gmann> and most of them with assignee also.
09:04:31 <gmann> #topic Previous Meeting Action review
09:04:44 <gmann> let's checkout the action items
09:04:56 <gmann> 1. andreaf email ML with our plan on scenario tests
09:05:03 <gmann> i think andreaf did that
09:05:18 <gmann> 2. andreaf setup a scenario only job for tempest
09:05:20 <andreaf> gmann yeah just looking for a link...
09:06:11 <gmann> #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/113113.html
09:06:19 <andreaf> thanks...
09:06:22 <andreaf> gmann: so for 2.
09:06:26 <andreaf> it's WIP
09:06:41 <gmann> 2. andreaf setup a scenario only job for tempest
09:06:50 <gmann> andreaf: thanks, link ?
09:06:55 <andreaf> gmann: I have a few patches in Tempest, d-g and project-config #link https://review.openstack.org/#/c/442565/
09:07:29 <andreaf> #link https://review.openstack.org/#/c/441980/ Tempest one is merged
09:07:52 <gmann> andreaf: local.conf things from job directly not stable yet ?
09:08:04 <gmann> i was thinking we can avoid d-g flags
09:08:16 <andreaf> gmann: and project-config one is my local branch, haven't posted it yet
09:08:37 <gmann> andreaf: ok, thanks for working on those.
09:08:38 <andreaf> gmann: yes local conf works for tempest settings and I used that for the project-config bit
09:09:00 <andreaf> gmann: but which tox env to use is not under local conf control
09:09:11 <gmann> ah yea
09:09:31 <gmann> ll check all series tomorrow
09:09:33 <andreaf> gmann: I don't like new flags either, but the alternative is to not use a dedicated tox env, which I though it was worse
09:09:39 <andreaf> gmann: thanks
09:09:43 <gmann> and currently we have 11 scenario tests running on normal jobs
09:10:20 <gmann> andreaf: yea tox env use is pretty nice
09:10:33 <gmann> 3. * andreaf check with luzC about grenade specs
09:10:38 <gmann> andreaf: again for you :)
09:11:10 <andreaf> gmann: yeah luzC mentioned she was going to withdraw those specs
09:11:19 <andreaf> and start new ones
09:11:55 <gmann> withdraw old one ?
09:12:31 <gmann> or its all new one what was discussed in PTG like rolling upgrade support etc
09:13:29 <andreaf> gmann: yeah so they are not grenade specs anymore
09:13:45 <andreaf> gmann: rolling upgrade and so will be done by other means
09:14:13 <gmann> in different project or way ?
09:15:26 <gmann> anyways let's get more clarity once we have spec up
09:15:51 <gmann> next AI- 4. oomichi to look at API versions for test jobs
09:15:55 <gmann> #link https://etherpad.openstack.org/p/tempest-api-versions-in-pike
09:16:30 <gmann> i think this will be very nice step if we can get rid of deprecated and unsupported version testing from tempest
09:16:49 <andreaf> yeah there are a few patches up already oomichi has been driving this forward
09:17:09 <gmann> it will make lot of load down on maintenance as well as on gate
09:17:18 <andreaf> yeah
09:17:31 <gmann> yea, oomichi will finish those fast i know :)
09:17:48 <gmann> cool
09:17:52 <andreaf> only one note on my side - config is a stable interface, and so are default values
09:17:52 <gmann> 5. everyone - review the ethercalc @
09:18:17 <andreaf> changing default values for a config is a backward incompatible chnage, so we need to give notice for that :)
09:18:28 <gmann> andreaf: default values also yes i think
09:18:41 <gmann> andreaf: i think i gave +1 or +2 on that
09:18:59 <gmann> but agree we should not change default value
09:19:24 <gmann> not sure deprecation process on default values change
09:19:45 <gmann> we can do just in help ?
09:19:58 <andreaf> gmann: yeah I guess so
09:20:40 <gmann> andreaf: ok.
09:21:02 <gmann> next AI:  ethercalc
09:21:23 <gmann> its all done and scenario tests are sorted out
09:21:33 <gmann> 6. sdague to look into cinder v1 admin tests
09:21:49 <gmann> andreaf: we can remove even admin tests for v1 right
09:23:19 <andreaf> heh I have to check I think sdague mentioned that it was actually ok as it is, but not 100% sure
09:23:47 <gmann> humm. i want to remove api version selection config options we have
09:24:04 <gmann> anyways let's check sdague on that
09:24:06 <gmann> #topic Gate Status
09:24:34 <gmann> so after cutting off scenario tests, gate kind of stable seems..
09:25:23 <gmann> but i am suspecting some ssh failure still even from API or scenario tests :)
09:25:37 <andreaf> gmann: the failures on the main tempest job are still too high: #link https://goo.gl/Ms7OVZ
09:26:18 <andreaf> we are still hitting libvirt crashes a lot #link  http://status.openstack.org/elastic-recheck/
09:26:18 <gmann> andreaf: oh, i think volume attach tests may be?
09:26:52 <andreaf> gmann: can you look into those?
09:27:13 <gmann> andreaf: sure, but i can do tomorrow only
09:27:43 <andreaf> I wanted to check if the libvirt crashes happen on API or scenario - I have the feeling it will be on API side, because API tests still generate good load on the system
09:28:01 <gmann> #action  gmann to check libvirt crashes issues on tempest job
09:28:10 <andreaf> on the other side if it's on scenario we might be able then to isolate the issue, get a reproducer and ask libvirt folks for a fix
09:28:12 <gmann> andreaf: yea i suspect attach detach tests
09:28:37 <gmann> nice
09:28:46 <andreaf> anyways, there is still work to be done on the gate, so that stays the highest prio for me
09:29:05 <gmann> +1
09:29:13 <gmann> gate stability is main goal
09:29:21 <andreaf> in case the issue is load related, one way forward could be to split API and scenario in two jobs and reduce concurrency on the API side
09:29:35 <gmann> openstack developement should go smoothly
09:30:03 <andreaf> another approach would be to isolate "heavy" API tests and either split them up or make them scenarios
09:30:17 <gmann> andreaf: we can list heavy API tests and then figure out those to be serial or parallel
09:30:21 <andreaf> so I will keep looking at both options
09:30:45 <andreaf> gmann: yeah I started looking at subunit-describe-calls to automate that
09:31:01 <gmann> i think splitting the heavy one either APi or scenario is nice
09:31:15 <gmann> andreaf: nice. thanks that will help
09:31:22 <zhufl> +1
09:31:41 <chandankumar> gmann: one query how to determine a test is heavy or light based on time or resources?
09:31:42 <zhufl> api tests should be simple enough
09:31:47 <gmann> let's keep eyes on gate things
09:32:01 <andreaf> chandankumar: heavy from resources pov
09:32:08 <gmann> chandankumar: yea time taken and what all kind of operation it does
09:32:16 <andreaf> chandankumar: so how many VMs. volumes, networks are created by a test
09:32:23 <gmann> yea
09:32:41 <chandankumar> andreaf: gmann got that, Thanks :-)
09:32:56 <zhufl> and some scenario testcases can be removed from Tempest maybe
09:32:58 <gmann> andreaf:  ll check some of them and prepare some list
09:33:10 <andreaf> gmann: ok thank you!
09:33:22 <gmann> zhufl: not sure about that. but we can go case by case
09:33:22 <zhufl> e.g, to suspend-resume-suspend-resume
09:33:42 <gmann> honestly i do not like to remove tests from tempest :)
09:34:01 <gmann> zhufl: we can figure that out where to run instead of deleting
09:34:03 <andreaf> zhufl: yeah some may not belong to tempest at all, but I guess that's a separate discussion for now
09:34:13 <gmann> but yea if those are covered on project side then yes
09:34:21 <gmann> let's move
09:34:24 <gmann> #topic Pike Goals
09:34:32 <gmann> 1. Python 3.5 https://governance.openstack.org/tc/goals/pike/python35.html
09:34:40 <gmann> i think this is completed
09:34:49 <andreaf> heh I was hoping to have some work done on these, but I have not had the time yet
09:35:04 <gmann> andreaf: on 3.5?
09:35:08 <andreaf> gmann: so the process is that I need to put up specs for each goal
09:35:25 <andreaf> gmann: and add the specs to the governance repo in the overall goal
09:35:28 <gmann> andreaf: and update patch on goverance
09:35:37 <gmann> ok
09:35:44 <chandankumar> gmann: i can check and confirm by runnign tempest on py3.5
09:35:47 <chandankumar> ?
09:35:53 <andreaf> gmann: and in each spec we need to say what needs to be done for python 3 and wsgi respectively
09:36:12 <andreaf> gmann: and this is not for tempest only but for every project under the QA umbreall
09:36:22 <gmann> andreaf: or just update status on that like  - #link https://review.openstack.org/#/c/440832/
09:36:45 <gmann> andreaf: yea, all projects
09:36:59 <gmann> blancos: how about patrole ?
09:37:12 <andreaf> so I guess there won't be much real work to be done but I just need to go through the list and make sure everything is covered
09:37:15 <gmann> do we have tests running on 3.5 job there
09:37:21 <andreaf> and what is not should go in the high prio list
09:37:25 <gmann> andreaf: yea
09:37:43 <blancos> gmann I'd have to doublecheck but I think we have a py35 gate
09:37:47 <andreaf> so I was hoping to have the list for today but I didn't manage yet, so it will be next week
09:38:11 <gmann> #action andreaf to add sepcs on Pike Goal, py3.5 and wsgi for all QA projects
09:38:20 <gmann> andreaf: ^^ :)
09:38:47 <gmann> blancos: cool, please update andreaf after checking on that.
09:39:04 <blancos> andreaf gmann I just looked, we do :)
09:39:42 <gmann> blancos: THIS? #LINK http://logs.openstack.org/17/443417/1/check/gate-patrole-python35/adfe79b/
09:39:58 <gmann> blancos: i think that runs just unit tests on 3.5 not patrole on 3.5
09:40:15 <andreaf> so integration tests must pass when a service runs on py3.5 but it the goal does not say anything about running integration tests on py3.5
09:40:31 <gmann> andreaf: is it?
09:40:43 <andreaf> but I would like to have that as a soft goal as well, not high prio like the main goal perhaps but it would be nice
09:41:03 <blancos> gmann Oh I'm sorry you're right. We're working on CI gates right now, I'll check with my colleague who's working on that
09:41:17 <gmann> i thought tests module too should be running on 3.5
09:41:27 <gmann> ll check anyways
09:41:32 <andreaf> blancos, gmann: ok I'll prepare a spec / etherpad and we can all comment on it then
09:41:53 <gmann> andreaf: cool, thanks, i already added AI for you
09:41:55 <gmann> anyways lets speed up
09:41:59 <andreaf> gmann: yeah unit / functional tests
09:42:00 <andreaf> but we don't have functional tests
09:42:01 <gmann> #topic Specs Reviews
09:42:20 <gmann> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
09:42:28 <gmann> andreaf: i see
09:43:14 <gmann> i think samP created new one/restored HA one # link https://review.openstack.org/#/c/443504/
09:43:26 <samP> gmann: yes
09:43:28 <gmann> let's talk on that separately
09:43:35 <gmann> anything else on specs?
09:43:40 <samP> gmann: sure
09:43:52 <andreaf> ok thanks I need to review that one :)
09:44:10 <andreaf> samP: is it WIP or ready for review?
09:44:12 <gmann> #topic Tempest
09:44:18 <gmann> - Warnings for internal interfaces? (andreaf)
09:44:23 <samP> adiantum: WIP for now,
09:44:35 <andreaf> samP: ok thanks
09:44:41 <samP> andreaf: sorry, WIP for now, I will update my todo items
09:44:45 <gmann> andreaf: you mean warning on each internal interface but we do not know how many in use
09:45:05 <andreaf> gmann: yeah I wanted to chat about if / how we advertise tempest internal interfaces
09:45:30 <andreaf> so something like raising a warning if an internal module is imported by a project that is not tempest
09:45:48 <gmann> andreaf: so doc on stable interface as list should be enough? and everything else is internal
09:46:04 <andreaf> I'm not sure how to implement that in way that is not too invasive and verbose, and I'm not sure if it's a good idea
09:46:05 <gmann> i know we do that currently and people still use internal variable too :)
09:46:32 <tosky> is there a way to identify them with some code analysis? It would be easier to identify the top-requested functions before starting to see warnings all over
09:46:43 <gmann> andreaf: ok, that will be good but on import warning it will be tempest too
09:46:53 <andreaf> gmann: well docs are ok I guess, but people does not necessarily read docs - which is normal
09:47:10 <tosky> gmann: can we defer the warning after identifing the required missing interfaces?
09:47:22 <gmann> tosky:  we can grep on codesearch but how many,
09:47:48 <andreaf> gmann: yeah I have some scripts to find out imports
09:47:51 <gmann> tosky: yea thats main goal, to have more and more required interface as stable
09:48:13 <gmann> andreaf: nice. but list might be huge right
09:48:23 <tosky> yes, but starting with warnings before having alternatives may be... counterproductive
09:48:33 <andreaf> gmann: #link https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh
09:48:47 <gmann> tosky: +1 i am on that side always :) but let's see how many we can provide
09:49:02 <gmann> tosky: but thing is it takes time to give complete interface as stable
09:49:21 <gmann> andreaf: cool, thanks
09:49:36 <andreaf> tosky: well apart from credential providers and perhaps test.py I think the alternative is to write the test in the plugins...
09:49:41 * gmann 12 min left
09:49:53 <gmann> next - #link https://review.openstack.org/#/q/project:openstack/tempest+status:open
09:50:08 <andreaf> tosky: well remote client perhaps as well, but I don't think we have much more high value things to move to lib
09:50:09 <gmann> ^^ are open review on tempest keep doing the good work
09:50:23 <gmann> Bug Triage:
09:50:37 <gmann> #link https://etherpad.openstack.org/p/tempest-weekly-bug-report
09:50:43 <prateek> https://review.openstack.org/#/c/443051/, can someone please review this too ?
09:50:46 <gmann> seems like nobody did last week
09:51:01 <tosky> andreaf: "get me the proper network used for the test, either created with the dynamic project or preconfigured"
09:51:03 <gmann> but we have report from luzC on previous week
09:51:10 <andreaf> gmann: lucZ sent an email about bug triage
09:51:17 <tosky> andreaf: that's required for any serious scenario tests
09:51:38 <gmann> andreaf: yea, i added in report #link https://etherpad.openstack.org/p/tempest-weekly-bug-report
09:51:43 <gmann> we have 7 new bugs
09:52:08 <gmann> jwhite is next week. ll ping him if i can reach to him
09:52:20 <gmann> anything on tempest side ?
09:52:48 <tosky> a potential issue with external plugin configuration, but we can handle it on -qa later
09:52:53 <gmann> if nothing on DevStack  grenade, o-h we can skip as time is less
09:53:00 <tosky> (well, not potential)
09:53:20 <gmann> tosky: yea, do we have bug on that, saw conversation on QA channel but not fully
09:53:34 <chandankumar> gmann: do we really need this while building doc https://github.com/openstack/tempest/blob/master/doc/source/conf.py#L18 ?
09:53:53 <chandankumar> to generate a list of plugins
09:54:08 <chandankumar> or i move it to a seperate tox section?
09:54:36 <gmann> chandankumar: need to check that is for listing all plugin on sample right
09:54:39 <gmann> #topic Patrole
09:54:47 <chandankumar> gmann: ok
09:54:51 <gmann> blancos: your turn, anything you want to bring in
09:55:13 <gmann> blancos: not sure about admin things on tests which i need to check the framework first
09:55:22 <blancos> I had 3 questions related to Patrole, but I can save them for -qa since we're short on time
09:55:43 <gmann> blancos: go ahead we can see if out of time
09:56:32 <blancos> Okay. 1. I understand that at the PTG it was decided base classes would become stable interfaces. I was wondering if there was any talk about doing that for waiters as well
09:56:56 <gmann> blancos: waiter too. i will check that
09:57:08 <blancos> 2. A couple were asking at a meeting yesterday if we had an IRC room. We don't
09:57:19 <blancos> Would we use QA's or get our own?
09:57:21 <gmann> i reemember we had patch for waiter but need to check where that went
09:57:57 <gmann> blancos: we cover in QA meeting but if you feel you need more time then its all ncie to have sub meeting too
09:58:00 <gmann> we do in nova
09:58:04 <gmann> andreaf: ^^ what u say
09:58:23 <gmann> blancos: keep your 3rd question on QA :) sorry
09:58:26 <gmann> #topic Destructive Testing
09:58:40 <samP> I hv just proposed a new spec (443504) in favor of Timur’s spec.
09:58:44 <gmann> so destructive testing is being started by samP
09:58:51 <gmann> samP: big thanks on that
09:58:54 <andreaf> gmann: well in terms of IRC I would rather keep one room
09:58:57 <gmann> samP: cool
09:58:59 <samP> gmann: np
09:59:04 <samP> I will address all the comments to Timur’s spec in my new spec.
09:59:15 <andreaf> gmann: in terms of meeting, if there is a need for a submeeting (enough to discuss there) I'm fine with that
09:59:22 <gmann> samP: in barcelona we mainly wanted user story on spec side
09:59:39 <gmann> samP: not sure you have in spec, if not can you put some of them
09:59:52 <samP> gmann: I will add them
09:59:54 <gmann> samP: it will be easy to visualize
09:59:57 <gmann> samP: thanks
10:00:12 <gmann> andreaf: ok
10:00:37 <samP> gmann: I will update this spec and clear my todos soon
10:00:38 <gmann> let's move to QA
10:00:40 <andreaf> gmann: but I would keep a patrole section in the main meeting for a quick update
10:00:45 <gmann> thanks all for joining
10:00:49 <gmann> #endmeeting