09:00:23 <gmann> #startmeeting qa
09:00:24 <openstack> Meeting started Thu Mar 23 09:00:23 2017 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:00:28 <openstack> The meeting name has been set to 'qa'
09:00:39 <gmann> who all here today?
09:00:45 <masayukig> \o
09:00:50 <ltosky[m]> o/
09:00:57 <chandankumar> \o
09:01:09 <martinkopec> o/
09:01:28 <gmann> hello everyone
09:01:30 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_March_23rd_2017_.280900_UTC.29
09:01:35 <gmann> ^^ today agenda
09:01:38 <felipemonteiro_> o/
09:01:58 <zhufl> hello
09:01:58 <gmann> #topic Previous Meeting Action review
09:02:14 <gmann> there was action item for jordan for ML on mem allcoation
09:02:22 <gmann> #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/114235.html
09:02:48 <gmann> seems like there are many patches to shrink the mem failure peak
09:03:14 <gmann> but most of the failure from API tests mainly attach/detach volume
09:03:41 <gmann> ll discuss that in gate topic
09:03:50 <gmann> #topic The Forum, Boston
09:04:21 <gmann> so there are 2 forum/sessions in Boston
09:04:39 <gmann> 1. Onboarding sessiosn #link https://etherpad.openstack.org/p/BOS-QA-onboarding
09:05:15 <gmann> this will be 90 min slot with combined with QA/infra/stable/release team
09:06:00 <gmann> and requirement team
09:06:20 <gmann> we will have around 15-20 min slot per each team
09:06:47 <gmann> this is more to talk and show project scope and visibility for new contributors in summit
09:06:51 <gmann> #link https://etherpad.openstack.org/p/BOS-QA-onboarding
09:06:59 <gmann> ideas for that is here ^^
09:07:08 <masayukig> gmann: ok, but it looks like the number of ideas are too much for the time..
09:07:28 <chandankumar> gmann: are these sessions going to be recorded?
09:07:37 <gmann> yea, i also thought we should have separate 90 min but i think andreaf is ok with 15-20 min :)
09:07:58 <gmann> chandankumar: do not know. it will be classroom type sessions
09:08:24 <gmann> actually there were room shortage and had to merge with those 5 team.
09:08:33 <prateek> gmann, would those be practical hands on sessions or more on the theory side ?
09:08:37 <gmann> let's see how many new contributor we can get from there
09:08:53 <gmann> prateek: theory due to time limits
09:09:01 <prateek> gmann, ok..
09:09:06 <masayukig> gmann: yeah..
09:09:20 <gmann> but there will be open area which we can use for team and new contributor and for hands on if they want
09:09:25 <ltosky[m]> So a big list of RTFM, given the time? :)
09:09:41 <prateek> gmann, sounds good
09:09:54 <gmann> yea we have to manage with that.
09:10:08 <gmann> 2. Brainstorming Forum
09:10:10 <gmann> #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/114399.html
09:11:12 <gmann> these will be more cross projects, dev-ops like and each team needs to submit their proposal and selection team  will schedule those by april 10
09:11:27 <gmann> deadline to submit sessions is april 2nd
09:11:40 <gmann> there is etherpad andreaf created to gether the ideas #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/114399.html
09:11:54 <gmann> please updates you ideas on that.
09:12:27 <masayukig> gmann: the link of the etherpad?
09:12:29 <gmann> as 2nd april is just after next meeting i am putting Ai for andreaf
09:12:41 <gmann> sorry
09:12:42 <gmann> #link https://etherpad.openstack.org/p/BOS-QA-brainstorming
09:13:01 <masayukig> gmann: thx :)
09:13:19 <gmann> #action andreaf to propose the Forum sessions before 2nd april on #link http://forumtopics.openstack.org/
09:13:46 <gmann> current proposed sessions can be seen here #link http://forumtopics.openstack.org/
09:13:55 * andreaf o/
09:14:15 <gmann> andreaf: just put AI for you on Forum sessions submission
09:14:41 <andreaf> gmann: yeah sure
09:14:45 <gmann> andreaf: any idea how long those sessions will be ?
09:15:07 <andreaf> gmann: uhm good question I have to check - I think it said in some ML
09:15:30 <gmann> ok
09:15:56 <andreaf> gmann: it's an opportunity to talk with openstack users / operators - so I like the idea of a session about downstream use of tempest plugins and other qa tools
09:16:39 <andreaf> gmann: there's still a bit of time to propose things in the etherpad, if you do please put your name as well :)
09:16:40 <gmann> yea we can get valuable (good/bad) feedaback from them
09:17:03 <gmann> andreaf: yea
09:17:04 <andreaf> gmann: yeah thanks for proposing that idea on the etherpad
09:17:17 <gmann> thanks :)
09:17:24 <gmann> andreaf: and for Onboarding sessions, 15-20 min enough ?
09:17:27 <chandankumar> andreaf: gmann are we keeping brainstome topic on etherpad somewhere?
09:17:42 <chandankumar> i mean forum topics
09:17:42 <gmann> chandankumar: #link  https://etherpad.openstack.org/p/BOS-QA-brainstorming
09:18:14 <gmann> andreaf: or we catch interested contributor in open area and explain them in more details ?
09:18:19 <chandankumar> gmann: sorry i mean forum topics
09:18:31 <andreaf> gmann: yeah well I initially thought we could use 90 min - but since it's a class type of setting I think it will be more more presentation + q&a style
09:18:36 * masayukig is worried about that some operator might say "Tempest is totally crap!" :-p
09:18:44 <gmann> chandankumar: those are Fourm topic only
09:19:06 <chandankumar> gmann: ack, i will add some more thoughts there
09:19:15 <gmann> masayukig: heh, that will be nice and opportunity to convince or improve ourself
09:19:34 <masayukig> gmann: heh, yeah
09:19:40 <andreaf> gmann: yeah my  idea is to do a quick presentation and hopefully attract contributors
09:19:40 <gmann> andreaf: ok, but now we cannot get more time right
09:19:59 <gmann> masayukig: we will miss Jordan in those cases :)
09:20:00 <andreaf> gmann: yeah 15min should be enough for a presentation
09:20:16 <andreaf> LoL
09:20:20 <gmann> andreaf: cool, some fancy slides is expected :)
09:20:39 <gmann> ok so please put more and more ideas there
09:20:48 <gmann> #topic Gate Stability - status update
09:20:57 <andreaf> masayukig, gmann: we did a lot of effort over the years to make tempest better for downstream usage - so any constructive input about where to improve things is welcome
09:21:15 <gmann> yea
09:21:21 <gmann> gate status -> #link https://goo.gl/ptPgEw
09:21:21 <masayukig> andreaf: yup
09:21:34 <jordanP> (hi, I am late)
09:21:44 <gmann> jordanP: hi
09:21:58 <gmann> so its little bit high failure from tomorrow
09:22:11 <gmann> main failure on attach detach API tests seems
09:22:32 <andreaf> jordanP: right on time! do you having anything to share about the gate and memory footprint?
09:22:40 <gmann> should we short list the heavy API tests like scenario tests ?
09:23:14 <jordanP> humm there's the "reduce the number of apache workers" and "enable ksm in guest" devstack patches
09:23:23 <jordanP> don't know what the status of those are
09:23:44 <jordanP> I've not be following the gate stability lately, is it as bad as usual ?
09:23:59 <gmann> one is merged - #link https://review.openstack.org/#/c/446741/
09:24:00 <andreaf> jordanP: well it's decent but not as good as it should
09:24:10 <gmann> yea
09:24:20 <jordanP> #link reduce number of apache workers for keystone: https://review.openstack.org/#/c/445910/
09:24:34 <gmann> ksm one is up -> #link https://review.openstack.org/#/c/447119/
09:25:11 <jordanP> the patch that disable nova-cert in gate jobs has been merged
09:25:20 <andreaf> another thing I saw in the conversation was that apparently the py35 job is doing much  better in term of memory footprint - it does not run swift which is part of it, but perhaps py35 is doing a better job in terms of memory mgmt
09:25:24 <gmann> oh we still had that?
09:25:33 <jordanP> https://review.openstack.org/#/c/446986/
09:25:54 <gmann> jordanP: thanks
09:26:16 <jordanP> that saves only 50Mo of RAM thought, but it's still good
09:26:34 <andreaf> I would like to propose a session in the forum about how to control / monitor over time the footprint of different services in the gate
09:27:05 <jordanP> +1
09:27:10 <gmann> andreaf:  +1
09:27:10 <masayukig> ++
09:27:16 <chandankumar> that would be awesome ++
09:27:25 <andreaf> ideas around that are welcome, if you have please add them in the etherpad :)
09:28:25 <andreaf> atm we have a lot of data but I feel perhaps not the right one so if's not obvious what'a using what and how things are changing over time
09:28:55 <andreaf> if we had historical data we could see if a service is regressing significantly in terms of memory footprint
09:29:07 <jordanP> lot of data is good, making sense of them is harder
09:29:11 <gmann> andreaf: putting those on graphite ? to monitor periodically
09:29:12 <andreaf> the issue with few data points is that there is so much variation in the test nodes that it's hard to tell
09:29:36 <andreaf> gmann: yeah graphite of somewhere we can see that data over time
09:29:50 <jordanP> well for eachand every runnning service we can ask ouselves "do we really need this" ?
09:29:57 <ltosky[m]> but with enough data it should be possible to group by node type
09:30:52 <andreaf> ltosky[m]: yeah if we have data over time we can work with it
09:31:18 <jordanP> I suggested in a ML email to kill a couple of unecessary swift services
09:31:33 <jordanP> the ones related to replication and reconcialiation
09:31:37 <andreaf> jordanP: thanks
09:31:53 <jordanP> I have less and less time to work on openstack so I won't do it myself
09:32:03 <jordanP> but I think we can save at least 100 Mo there
09:32:25 <masayukig> jordanP: :( and :)
09:32:46 <andreaf> I'd like to think we can keep better control on the status of our test nodes on a regular basis, because now we are doing a lot of effort in addressing this from many sides
09:33:08 <gmann> how about enabling services from job definition ? may be hard to configure each job
09:33:10 <andreaf> but one we are back in good shape we should have means to stay there
09:33:20 <jordanP> sure sure, would love to have this plotted in Graphite and all, but the house is on fire :)
09:34:14 <andreaf> jordanP: yeah that's why I say this could be a session for the forum for the mid term
09:34:29 <jordanP> yep, sounds goo
09:35:16 <gmann> or shrink the default enabled services ...
09:35:35 <masayukig> so, 25 mins..
09:35:39 <gmann> anyways let's move and keep monitor failure
09:35:42 <gmann> #topic Specs Reviews
09:35:51 <gmann> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z
09:36:18 <gmann> spec state seems same so if no one has anything we will move next ?
09:36:42 <gmann> #topic Tempest
09:36:47 <gmann> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open
09:36:54 <gmann> ^^ open reviews
09:37:33 <gmann> one update was oomichi removed the cinder v1 tests
09:37:36 <chandankumar> gmann: https://review.openstack.org/370630
09:37:42 <chandankumar> review needs feedback
09:37:44 <gmann> and working on merging v2 and v3 one
09:38:08 <chandankumar> gmann: and tempest cleanup doc patch https://review.openstack.org/#/c/444041/
09:38:23 <gmann> chandankumar: ll check but tomorrow
09:38:28 <gmann> chandankumar: thanks
09:38:54 <gmann> Bug Triage:
09:38:59 <gmann> #link https://etherpad.openstack.org/p/pike-qa-bug-triage
09:39:21 <gmann> chandankumar: had rotation this week
09:39:24 <gmann> #link https://etherpad.openstack.org/p/tempest-weekly-bug-report
09:39:26 <gmann> chandankumar: go ahead
09:39:44 <chandankumar> gmann: some small higlights
09:39:46 <chandankumar> Bugs with no reply from assignee from last 6 months are unassigned with a comment. (Mostly from wishlist)
09:39:56 <chandankumar> Added a new tag'upstream-gate' to list only upstream tempest gate failure bugs
09:40:04 <gmann> upstream-gate ?
09:40:05 <chandankumar> New Bugs: 11 -> 8
09:40:18 <gmann> you mean failure on gate with that tag?
09:40:31 <chandankumar> gmann: failure ongate with that gate
09:40:43 <gmann> k
09:41:02 <chandankumar> https://wiki.openstack.org/wiki/Bug_Tags list bug tags for different project
09:41:12 <chandankumar> tempest is missing there so i add it there by EOD
09:41:13 <gmann> chandankumar: yea assignee things you can cleanup by leaving a comment there
09:41:28 <chandankumar> gmann: cleaned already
09:41:34 <chandankumar> as discussed in the mornign
09:41:40 <chandankumar> *morning
09:41:41 <gmann> thanks
09:41:50 <gmann> chandankumar: did  see tempest tag on that wiki
09:42:18 <gmann> next rotation is prateek
09:42:26 <gmann> #topic Patrole
09:42:32 <prateek> gmann, ack
09:42:35 <gmann> #link https://review.openstack.org/#/q/project:openstack/patrole
09:42:39 <gmann> prateek: thanks
09:42:49 <chandankumar> gmann: one more thing *  https://bugs.launchpad.net/tempest/+bug/1672817
09:42:49 <openstack> Launchpad bug 1672817 in tempest "Mapping client is needed for v3/OS-FEDERATION/mappings API" [Undecided,New] - Assigned to Nicolas Helgeson (nhelgeson)
09:42:51 <chandankumar> * https://bugs.launchpad.net/tempest/+bug/1504641
09:42:52 <openstack> Launchpad bug 1504641 in OpenStack Compute (nova) "Listing volumes respects osapi_max_limit but does not provide a link to the next element" [Wishlist,New]
09:42:56 <gmann> felipemonteiro_:  ^^
09:43:05 <chandankumar> needs be moved to wishlist
09:43:21 <gmann> chandankumar: ok, leave comment also ll check later
09:43:24 <felipemonteiro_> we've achieved much higher gate stability, with both gates now passing, but we're still watching them carefully
09:43:26 <chandankumar> that's it from myside
09:43:39 <jordanP> good work chandankumar, thanks
09:43:54 <gmann> chandankumar: thanks. nice effort
09:44:05 <chandankumar> jordanP: gmann you are welcome :-)
09:44:11 <felipemonteiro_> idea is to keep monitoring them, fix errant bugs, and make admin gate voting (within our project), after we are confident it is stable
09:44:24 <gmann> felipemonteiro_: any plan to run patrole tests on project side ?
09:44:47 <felipemonteiro_> gmann: what do you mean by "project side"?
09:45:03 <gmann> felipemonteiro_: on project gate job like nova
09:45:04 <jordanP> I am not huge +1 gmann on this. We would need to weight the pros and cons
09:45:22 <gmann> felipemonteiro_: so that any mis changes on policy side can be detected bu patrole at same time
09:45:31 <jordanP> like this may mean a lot of more infra VMs
09:45:42 <gmann> jordanP: sure but policy testing is really critical
09:45:59 <felipemonteiro_> gmann: ultimately yes, but only once high stability is achieved. for nova, for example, i think i've seen 1 failure related to attach volume
09:46:14 <gmann> and i am 100% sure many projects miss those currently
09:46:19 <felipemonteiro_> but nova tests are otherwise among the most stable
09:46:35 <andreaf> gmann, jordanP, felipemonteiro_: I think this is something that needs to be discussed with the projects
09:46:44 <jordanP> +1
09:46:45 <gmann> felipemonteiro_: ok let's judge that later once it is stable
09:46:46 <felipemonteiro_> andreaf: agreed
09:46:55 <andreaf> if we get one or two onboard and showcase benefits we can propose for more
09:47:13 <gmann> andreaf: yea i discussed in nova api meeting yesterday and can be decide later once patrole is stable
09:47:32 <chandankumar> andreaf: gmann can we get a nightly job which run one time with all the tempest plugins with all scenario tests to detect plugin tempest issues?
09:47:36 <chandankumar> jordanP: ^^
09:47:36 <gmann> felipemonteiro_: i updated comment on discovery policy testing. that is kind of deprecated policy from nova side so not worth to tests
09:48:06 <gmann> chandankumar: and can that finish since morning :)
09:48:06 <andreaf> chandankumar: we are talking about patrole now?
09:48:23 <chandankumar> gmann: andreaf sorry i missed it
09:48:29 <gmann> felipemonteiro_: thanks for updates.
09:48:30 <felipemonteiro_> gmann: thanks! that was my assumption, so i hadn't invested more energy in it
09:48:37 <gmann> let's move as time is short
09:48:46 <gmann> #topic DevStack
09:48:50 <gmann> anything on devstack side?
09:49:02 <andreaf> gmann: well one sec please
09:49:19 <gmann> andreaf: sure
09:50:14 <andreaf> regarding patrole I just wanted to say that if we wait too much it will never be stable enough since project may break it
09:50:27 <andreaf> so yes we might need to wait a little bit I would start and try to get non voting jobs up for some projects
09:50:44 <andreaf> regarding chandankumar question about plugins
09:50:57 <gmann> andreaf: yea for n-v +1. but putting non stable tests it might trigger false alarm on them
09:51:18 <felipemonteiro_> andreaf: we've started discussing it with keystone, we can go to nova and other projects. but we just got "high stability" a few days ago
09:51:32 <gmann> but yea i am ok for nova to start if tests results are pretty stable
09:51:33 <andreaf> felipemonteiro_: ok thanks
09:51:52 <gmann> because we are changing the policy things there and should not break existing way
09:52:03 <gmann> felipemonteiro_: cool, thanks
09:52:08 <andreaf> gmann: sorry there was a question from chandankumar re plugins as well
09:52:31 <andreaf> chandankumar: running all plugins means running all services as well, I don't think that's feasible
09:52:32 <gmann> andreaf: yea but with all plugins i am afraid it will run till morning
09:52:51 <gmann> and night job means? its day for somewhere :)
09:53:18 <andreaf> gmann, chandankumar: and we have an issue without any  plugin to keep the memory footprint in place
09:53:29 <gmann> yea
09:53:49 * gmann 7 min left
09:54:07 <chandankumar> andreaf: but i am taking about a one time a day or a week scheduled job
09:54:14 <andreaf> gmann: sorry back to you
09:54:20 <andreaf> chandankumar: I don't think that will work at all
09:54:20 <gmann> anything on devstack from anyone?
09:54:31 <gmann> chandankumar: let's discuss more on qa on this. sorry
09:54:33 <andreaf> chandankumar: let's chat in QA after the meeting
09:54:37 <gmann> yea
09:54:53 <chandankumar> andreaf: gmann sure i will ping after an hour, switching to other call
09:54:55 <gmann> let's move to next then
09:54:58 <gmann> #topic Upgrade Testing
09:55:08 <gmann> Grenade
09:55:14 <gmann> Rolling upgrade
09:55:21 <gmann> andreaf: you know if any spec up for those?
09:55:49 <andreaf> gmann: no I haven't heard anything back from the osic folks
09:55:54 <gmann> sdague: was looking for those i think
09:56:12 <gmann> luzC: ^^
09:56:37 <gmann> ok let's jump to o-h
09:56:41 <gmann> #topic OpenStack-Health, Stackviz
09:56:49 <gmann> masayukig: ^^ any updates you want to bring in
09:57:19 <masayukig> yeah, I pushed a patch to change the default range at home page of o-h
09:57:39 <masayukig> the first page loading should be quicker than before
09:57:40 <gmann> masayukig: nice and what will be dafault now
09:57:52 <masayukig> and I pushed another patch for the rss
09:58:24 <gmann> nice
09:58:24 <masayukig> It could be useful to know the failure in the gate
09:58:35 <gmann> masayukig: link?
09:58:41 <masayukig> one sec
09:58:54 <gmann> masayukig: no prob you can give later
09:59:05 <masayukig> https://review.openstack.org/#/c/444722/
09:59:06 <masayukig> heh
09:59:09 <masayukig> #link https://review.openstack.org/#/c/444722/
09:59:10 <gmann> masayukig: thanks
09:59:22 <gmann> anything else?
09:59:24 <masayukig> that all
09:59:27 <gmann> #topic Destructive Testing
09:59:35 <gmann> masayukig: thanks for all nice updates
09:59:35 * masayukig one min..
10:00:15 <gmann> on destructive testing samP updated offline that he is working with different different stack holder to get the user story
10:00:22 <gmann> and will update the spec
10:00:44 <gmann> i think time up, sorry i have to close
10:00:53 <gmann> thanks all, let's jump to QA
10:00:55 <gmann> #endmeeting QA