17:00:31 <jlvillal> #startmeeting ironic_qa
17:00:32 <openstack> Meeting started Wed Jan 13 17:00:31 2016 UTC and is due to finish in 60 minutes.  The chair is jlvillal. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:36 <openstack> The meeting name has been set to 'ironic_qa'
17:00:48 <jlvillal> Hello all
17:00:57 <jlvillal> As always the agenda is here: https://wiki.openstack.org/wiki/Meetings/Ironic-QA
17:00:58 <kcalman_> o/
17:01:12 <jlvillal> #announcements
17:01:27 <jlvillal> I don't have anything to announce.
17:01:37 <aarefiev> hi
17:01:37 <jlvillal> Looks like this might be a short meeting if only two of use here
17:01:42 <jlvillal> three!
17:01:48 <rpioso> o/
17:01:51 <rajinir> o/
17:01:56 <jlvillal> Cool
17:02:01 <krtaylor> o/
17:02:05 <jlvillal> Does anyone have any announcements?
17:02:17 <jlvillal> Moving on in
17:02:18 <jlvillal> 5
17:02:20 <jlvillal> 4
17:02:22 <jlvillal> 3
17:02:24 <jlvillal> 2
17:02:27 <jlvillal> 1
17:02:35 <jlvillal> #topic Grenade Testing
17:02:37 <[3]cdearborn> o/
17:02:54 <jlvillal> I am still working on Grenade testing.
17:03:17 <jlvillal> This really means working on setting up an environment that copies openstack/devstack-gate and works on my system.
17:03:37 <jlvillal> Unfortunately my system is behind a proxy server and causes me what seems like endless issues.
17:04:20 <jlvillal> I have gotten the Grenade job to get to the creating the IPA image stage, which dies because of proxy issues. Hope to fix that today.
17:05:07 <jlvillal> I have also been seeing where it fails in the gate
17:05:23 <jlvillal> I have been using this patch to trigger "check experimental" which runs the grenade job: https://review.openstack.org/#/c/255834/
17:06:30 <jlvillal> That's all I have to report right now.
17:06:37 <jlvillal> Moving on unless questions.
17:06:41 <jlvillal> 4
17:06:42 <jlvillal> 3
17:06:43 <jlvillal> 2
17:06:45 <jlvillal> 1
17:07:00 <jlvillal> #topic Functional testing
17:07:19 <jlvillal> #info Email thread about functional testing in Ironic on openstack-dev
17:07:51 <jlvillal> So Serge started an email thread about functional testing
17:08:15 <jlvillal> I forget his IRC nickname :(
17:08:41 * jroll is here
17:08:41 <jlvillal> Seems like people want to be able to do 'tox -efunc' functional testing
17:08:53 <jlvillal> And would like to have support in Tempest also
17:09:27 <jroll> the tempest thing is somewhat separate, in my mind
17:09:39 <jlvillal> #info Functional testing should support 'tox -efunc'
17:09:44 <jroll> it's more of "should we keep writing tempest tests, even thuogh we have functional testing"
17:09:49 <jroll> "yes, we should"
17:09:56 <jlvillal> Thanks.
17:10:09 <jroll> (why? tempest is valuable for testing a deployment)
17:10:17 <jlvillal> Yep!
17:11:05 <jlvillal> #info Continue to have or add functional testing in Tempest. For one thing it is valuable for testing a deployment. Along with other reasons.
17:11:23 <jlvillal> Anyone have anything to add?
17:11:43 <jlvillal> Okay, moving on in 5
17:11:44 <jlvillal> 4
17:11:45 <jlvillal> 3
17:11:47 <jlvillal> 2
17:11:54 * krtaylor need to understand the differences better
17:12:08 <jlvillal> Between what and what?   krtaylor
17:12:18 <jlvillal> tox and tempest?
17:12:20 <krtaylor> functional and tempest
17:12:44 <krtaylor> or I should say stand-alone and tempest?
17:12:53 <jlvillal> So we would like to add a 'tox -efunc' to do functional testing.
17:13:02 <jlvillal> That makes it easy for developers to run functional testing.
17:13:09 <krtaylor> y, understood
17:13:25 <jlvillal> Tempest is great and all but needs to have an OpenStack environment up and running. Which is rather large overhead for a developer.
17:13:46 <jlvillal> Things they test will likely have an overlap. Maybe they both test the same API.
17:13:53 <jlvillal> That is my understanding
17:13:58 <jroll> so in my mind: tempest is a thing for testing a real deployment of ironic (probably with nova, maybe without if some tests are skipped), whereas functional testing is a thing that can run in a minute that can do similar validation, but less "full"
17:14:00 <jlvillal> Not sure if that makes it any clearer krtaylor
17:14:13 <jroll> in functional testing, backends can be mocked (for example neutron)
17:14:25 <jroll> and we might use in-memory oslo.msg or something
17:14:44 <jlvillal> Thanks jroll
17:14:46 <krtaylor> jroll, that helps
17:14:56 <jlvillal> Okay now moving on in 4
17:14:57 <krtaylor> so it has to do with developer speed
17:14:58 <jlvillal> 3
17:15:03 <jroll> yeah basically
17:15:04 <jlvillal> Yes :)
17:15:14 <jlvillal> 2
17:15:20 <jlvillal> 1
17:15:24 <jlvillal> Okay moving on :)
17:15:32 <jlvillal> #topic 3rd Party CI (krtaylor)
17:15:41 <jlvillal> The floor is yours krtaylor
17:15:47 <krtaylor> no progress at all
17:15:59 <krtaylor> swamped with downstream work
17:16:17 <jlvillal> Okay, thanks. And I know the feeling....unfortunately....
17:16:25 <kcalman_> I had a question regarding setting up 3rd party CI
17:16:33 <jlvillal> Go ahead!
17:16:35 <krtaylor> hoping to dig out a bit this week
17:16:59 <kcalman_> will there only be one release cycle/branch  at a time under test?
17:17:14 <kcalman_> or both the current and the next?
17:17:22 <rajinir> updated the status for dell idrac -> https://etherpad.openstack.org/p/IronicCI  ,  please review
17:17:27 <jlvillal> Do you mean like only testing tip of master?
17:17:56 <krtaylor> you can chose whether to test stable branches or tip, are you asking for requirements?
17:18:02 <kcalman_> yes, or master and current released version (e.g., liberty)
17:18:40 <jlvillal> I would have thought focus would be on master. But I will defer to krtaylor and jroll on that.
17:18:47 <krtaylor> I don't recall that we addressed that in the spec
17:19:03 <jroll> I guess right now we'd focus on master, next cycle master+mitaka, next cycle master+mitaka+N
17:19:21 <kcalman_> OK
17:19:22 <krtaylor> well, since we didn't require testing until now, it makes sense :)
17:19:28 <jroll> or rather, third party CI should test cycles/branches from when they start testing forward
17:19:36 <krtaylor> ++
17:19:50 * jroll updates the spec
17:20:02 <kcalman_> but going forward, will changes be expected to be backported to current release, on only in the new development/next release?
17:20:26 <kcalman_> ("current" => released)
17:20:36 <jroll> I'm not sure what you mean kcalman_ - some changes are backported, some are not. depends on the change
17:20:49 <jroll> I don't expect you to test branches that existed before your CI existed
17:20:52 <jroll> (but you can if you'd like)
17:21:06 <kcalman_> like feature vs bug fix?
17:21:27 <jroll> right, it's bug fixes only and depends on severity
17:21:33 <kcalman_> OK
17:21:44 <jroll> more info here: http://docs.openstack.org/project-team-guide/stable-branches.html
17:21:52 <kcalman_> thanks
17:21:58 <jlvillal> #info jroll will update spec on what branches (for example master, stable/liberty, etc..) should be tested by 3rd Party CI
17:22:13 <krtaylor> thanks jroll
17:22:18 <jlvillal> +1
17:22:18 <rajinir> Posting again, Updated the status for dell idrac -> https://etherpad.openstack.org/p/IronicCI  ,  please review
17:22:47 <kcalman_> there was another question with the scope of CI testing expected for Mitaka
17:22:49 <jlvillal> #info Status updated for Dell idrac -> https://etherpad.openstack.org/p/IronicCI  , please review
17:23:05 <rajinir> Question: For M feature freeze, the etherpad says receive and post comments on ci-sanbox. Is that it?
17:23:13 <kcalman_> should that be the build plumbing only, or is there a no-op Ironic test that should be executed?
17:23:23 <jroll> rajinir: yes
17:23:48 <jroll> kcalman_: we only require that you are able to receive and post comments from CI sandbox
17:23:54 <jroll> by the end of mitaka cycle
17:24:01 <jroll> (or rather M-3, iirc)
17:24:02 <kcalman_> OK great
17:24:10 <rajinir> cool, thanks
17:24:18 <jlvillal> Anything else?
17:24:54 <rajinir> We have  ton of questions for N release, we will probably reach out to the infra team and ironic QA
17:25:17 <jlvillal> #info 3rd Party CI is only required to receive and post comments from CI sandbox by M-3 time frame, to be compliant with Ironic policy.
17:25:31 <jlvillal> rajinir, understood.
17:25:38 <jlvillal> Okay, if nothing else will move on
17:25:41 <jroll> #link https://review.openstack.org/267049
17:25:43 <jroll> for spec update
17:26:08 <jlvillal> thanks
17:26:14 <jlvillal> Moving on in 4
17:26:16 <jlvillal> 3
17:26:19 <jlvillal> 2
17:26:25 <jlvillal> 1
17:26:38 <jlvillal> #topic Open discussion / General QA topics
17:26:59 <jroll> I have a thing
17:27:09 <jroll> I've been working on the move to a tempest plugin
17:27:23 <jroll> it's nearly done on the ironic side, could use some reviews
17:27:49 <jroll> https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:master+topic:tempest_plugin
17:27:56 <jroll> (needs an update too, apparently)
17:28:02 <jlvillal> Thanks
17:28:14 <jroll> the downside is that tempest is branchless, so we can't remove the tests from tempest until liberty is EOL :(
17:28:33 <jroll> I'll be looking into the best way to work that this week
17:28:40 <jlvillal> #info Tempest plugin patches need reviews: https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:master+topic:tempest_plugin
17:29:11 <jlvillal> That is a long time. I assume that is a couple years for Liberty EOL???
17:29:58 <jlvillal> This probably does not fall into this meeting. But there seem to be a lot of gate failures. I have one patch that fails again and again and it not touching any executing code.
17:30:09 <jlvillal> I assume other people are noticing similar things
17:30:44 <jroll> it's usually a couple cycles
17:31:13 <jroll> also, yes, oh my.
17:31:15 <jroll> http://tinyurl.com/j5yc4yr
17:31:46 <jlvillal> Last patch I see merged in was on Friday. Not counting patch into devstack/lib/ironic
17:31:48 <jroll> still timeout stuff, I assume. we need to keep pushing through that
17:32:27 <jlvillal> I would love a pointer on how to create those graphite graphs. I tried graphite.openstack.org and couldn't figure it out :(
17:32:44 <jroll> https://github.com/sdague/failopotamus
17:32:46 <jroll> :D
17:32:52 <jroll> I'm using python failgraph.py -d 400 gate-tempest-dsvm-ironic-pxe_ipa
17:32:53 <jlvillal> Awesome! Thanks :)
17:33:24 <jlvillal> Put that into his notes.
17:33:33 <jlvillal> Okay, if nothing else we can adjourn.
17:33:53 <jlvillal> Adjouring in 5
17:34:07 <jlvillal> Or Adjourning in 5 (spell correctly this time)
17:34:08 <jlvillal> 4
17:34:11 <jlvillal> 3
17:34:13 <jlvillal> 2
17:34:15 <jlvillal> 1
17:34:18 <jlvillal> Thanks everyone!
17:34:21 <jlvillal> Ciao
17:34:25 <jlvillal> #endmeeting