17:02:12 <jlvillal> #startmeeting ironic_qa
17:02:13 <openstack> Meeting started Wed Oct 12 17:02:12 2016 UTC and is due to finish in 60 minutes.  The chair is jlvillal. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:16 <openstack> The meeting name has been set to 'ironic_qa'
17:02:21 <mjturek> o/
17:02:21 <sambetts> o/
17:02:24 <watanabe_isao> o/
17:02:30 <jlvillal> Sorry I'm late. Got distracted fixing a bug :(
17:02:32 <[1]cdearborn> o/
17:03:12 <jlvillal> As always the agenda is at:
17:03:16 * jlvillal fights with Synergy
17:03:17 <jroll> \o
17:03:30 <jlvillal> https://wiki.openstack.org/wiki/Meetings/Ironic-QA
17:03:38 <jlvillal> #topic Announcements
17:04:04 <jlvillal> #info Xenial fix for ironic-lib was done. Thanks vsaienk0
17:04:34 <jroll> ++
17:04:43 <jlvillal> #info New version of ironic-lib broke ironic unit tests. https://bugs.launchpad.net/ironic/+bug/1632762  Almost fixed
17:04:44 <openstack> Launchpad bug 1632762 in Ironic "Unit tests fail with ironic-lib 2.2.0" [Critical,New]
17:04:59 <clarkb> is that related to slow iscsi io?
17:05:11 <jroll> clarkb: no, bad mocks in unit tests
17:05:13 <clarkb> ah
17:05:16 <jlvillal> clarkb: No. That was because we got sort of wonky unit tests.
17:05:21 <jroll> clarkb: the iscsi thing turned out to be MTU problems
17:05:23 <jroll> and is fixed
17:05:35 <jlvillal> clarkb: What jroll said on MTU
17:05:40 <clarkb> thanks
17:06:01 <jlvillal> clarkb: FYI: https://review.openstack.org/385433
17:06:12 <jlvillal> #info patch proposed to enable Xenial for ironic-lib https://review.openstack.org/385433
17:06:21 <jlvillal> Anything else?
17:06:37 <jlvillal> Okay. Moving on then.
17:06:43 <jlvillal> #topic Multi-node Grenade
17:07:34 <jlvillal> #info Multiple patches are in the queue to get reviewed. Last I checked the multinode test was green with the entire patch series.
17:08:14 <jlvillal> I don't have much to add there. I am working on setting up a local test environment for multi-node so I can test the patches and then be ready for adding Grenade to it.
17:08:19 <jlvillal> Any questions/comments?
17:09:06 <jlvillal> Okay moving on
17:09:14 <jlvillal> #topic 3rd Party CI from krtaylor
17:09:24 <jlvillal> krtaylor: The floor is yours.
17:09:47 <jlvillal> sambetts: Can you start us off as I don't see krtaylor
17:10:03 <jlvillal> sambetts: jroll: On the agenda is some questions from watanabe_isao
17:10:16 * jroll looks at agenda
17:10:35 <sambetts> So, there were several breakages last week caused by the SUBNETPOOL_PREFIX issue in devstack
17:10:52 <sambetts> the Cisco CI still hasn't recovered fully
17:10:53 <watanabe_isao> mostly I think we should make it clear and my advice, I have write it in the agenda.
17:11:09 <jroll> watanabe_isao: I agree, I've been working on a draft for some of this
17:11:16 <jroll> I'm sorry I've been slow about it
17:11:35 <sambetts> they have merged a fix into devstack to be able to optionally disable subnetpools now so that should help
17:11:42 <jlvillal> sambetts: I know clarkb has been pushing on getting a fix for that.
17:12:04 <sambetts> ++
17:12:09 <clarkb> I think the neutron team has a better understanding of the issue after the most recent discussion so hopefully we can get to a place that makes them happy
17:12:19 <sambetts> awesome
17:12:49 <watanabe_isao> jroll, what I am most worry about is we don't have a tool to check the CI reply time.
17:13:30 <jroll> watanabe_isao: yeah, it's sad. someone could build one, or extend ci-watch. :)
17:13:49 <jroll> I don't expect us to be super strict about runtime
17:14:09 <sambetts> with regards to the result turnover for a third party CI, I don't think we yet enforce a return time on the ironic ones, but we have discussed what is a resonable time frame in the past
17:14:11 <jroll> but we need to have some sort of rules, waiting more than 8 hours isn't useful
17:15:44 <watanabe_isao> jroll, and about the pass/fail rates, it is not mentioned in the space, but if all drivers update their CI account in Driverlog, it would be easier to check for that in stakalitics.
17:16:05 <jroll> watanabe_isao: cool
17:16:57 <watanabe_isao> irmc has done that, it is pretty easy.
17:17:23 <jroll> yeah, krtaylor had updated some of them, I think
17:19:12 <jroll> anything else on this topic?
17:19:15 <jlvillal> Anything else?
17:19:22 <watanabe_isao> jroll, sambetts, about the reasonable time,
17:19:33 <watanabe_isao> it changed day by day.
17:20:06 <jroll> what changed?
17:20:08 <watanabe_isao> do we have a max patch set per day target to clear for the 8/4 hours time?
17:20:19 <sambetts> From the spec: Tests will be expected to complete and report back to gerrit within 8 hours of the patch submission until the end of Newton release and within 4 hours by the end of Ocata development cycle.
17:21:00 <jroll> watanabe_isao: what do you mean by max patch set per day?
17:21:03 <jlvillal> watanabe_isao: I would assume it should be able to keep up with the patches that are submitted. Not sure how many is normal per day.
17:21:07 <jroll> watanabe_isao: it's every patch that touches code
17:21:23 <jroll> there's no "max", just "all"
17:21:36 <sambetts> http://graphite.openstack.org/render/?width=586&height=308&_salt=1447868663.13&lineMode=connected&from=00:00_20141115&until=23:59_20151118&target=summarize%28stats_counts.zuul.pipeline.check.openstack.ironic.total_changes,%20%271d%27%29
17:21:46 <jroll> sambetts: ++
17:22:02 <watanabe_isao> sambetts, thanks, I am asking this.
17:22:30 <sambetts> the time scales on that graph need updating but thats what we based the spec on
17:23:12 <sambetts> so as you can see it needs to deal with spikes of 35+ patches and still return within 8hours for all of them
17:23:22 <jroll> http://graphite.openstack.org/render/?width=586&height=308&_salt=1447868663.13&lineMode=connected&from=00:00_20151115&until=23:59_20161118&target=summarize(stats_counts.zuul.pipeline.check.openstack.ironic.total_changes,%20%271d%27)
17:23:24 <jroll> D:
17:23:25 <sambetts> thats 8hrs each btw
17:23:43 <jroll> that has... dramatically increased
17:23:43 <sambetts> jroll: woah...
17:24:13 <jroll> hm
17:24:18 * jroll needs to put some thought into this
17:24:35 <jroll> that seems... high
17:24:37 <watanabe_isao> jroll, sambetts, so the document patched included?
17:24:45 <jlvillal> Of course if someone uploads the same patch five times in a 5 minute period. It should kill the running job each time a new revision appears.
17:24:48 <jroll> watanabe_isao: yes, that graph is all patches
17:27:11 <watanabe_isao> jroll, thanks, however 600 per day is a big number...
17:27:46 <sambetts> I'm not sure that it right, there are definatly not 600 patch messages in the ironic irc channel
17:27:55 <jroll> yeah, something is off there
17:28:03 <jroll> watanabe_isao: like I said, I need to put some thought into this
17:28:21 <watanabe_isao> jroll, sure.
17:28:25 <jroll> it does seem we've been posting about 100 patchsets per day, though
17:29:00 <jroll> though if you imagine one hour per CI job, that's only 4 per hour, so 4 simultaneous runs
17:29:14 <jroll> I don't think that's unreasonable
17:30:16 <watanabe_isao> jroll, if one driver has 3 sub drivers, it makes it from 4 to 12 nodes.
17:30:37 <jroll> watanabe_isao: right, I don't think that's an unreasonable amount of hardware
17:30:55 <jroll> anyway, I will think about this a bit and try to write something up
17:31:32 <jroll> watanabe_isao: oh, and if the rule is 4 hours, divide it by 4
17:31:38 <jroll> so 1-3 nodes
17:31:47 * sambetts is going to have to drop, sorry guys, see you later
17:31:51 <jroll> if runtimes can be kept to one hour
17:31:53 <jroll> later sambetts :)
17:31:54 <jlvillal> sambetts: Thanks!
17:32:06 <watanabe_isao> sambetts, bye.
17:34:00 <jroll> watanabe_isao: does that help? make sense?
17:34:07 <jroll> we need to tweak the rules, I suspect
17:34:12 <watanabe_isao> jroll, I don't quite get the divide it by 4, I was thinking you mean that 4 patchset per hour comes at max, when it is 100 patchset per day?
17:34:54 <jroll> watanabe_isao: yeah, thinking more, ignore that
17:36:20 <watanabe_isao> jroll, and if a CI test can make sure itself be done in 1 hour, then the 8 or 4 conditions are just the same.
17:36:55 <jroll> watanabe_isao: yeah, I was wrong, I need more analysis to actually tell what this means (because patches aren't evenly distributed across time)
17:38:24 <jroll> watanabe_isao: but otherwise, does this make sense
17:38:26 <jroll> ?
17:38:37 <watanabe_isao> jroll, yes we are considering the "100 patchsets" are been post int he American time half day(12 hours) which means it may not be 4 per hour but 8 per hours.
17:39:00 <jroll> right
17:39:45 <jlvillal> Europe time zone pastes often too.  Seems like more of a 12-16 hour period.
17:39:51 <jlvillal> s/pastes/posts/
17:40:20 <jroll> so beyond "jroll needs to analyze, think, and write about this" is there anything else on this topic?
17:41:07 <watanabe_isao> jroll, I'm ok, and waiting for your answer about this. :)
17:41:18 <jroll> ok, thanks for bringing it up :)
17:41:42 <jroll> jlvillal: let's move on :)
17:41:48 <jlvillal> jroll: Thanks
17:41:51 <jlvillal> #topic Opens
17:41:55 <jlvillal> Anyone have anything?
17:42:04 <jroll> see dmitry's ML post?
17:42:12 <jroll> people should read that thing :)
17:42:13 <jlvillal> #info ironic gate is now working again. Unit tests are fixed.
17:42:41 * jlvillal looks and says Ah
17:43:21 <jlvillal> #info dtantsur has proposed consolidating and extending Ironic CI jobs
17:44:22 <jlvillal> http://lists.openstack.org/pipermail/openstack-dev/2016-October/105558.html
17:44:53 <jlvillal> #info Jordon Pittier is trying to remove Ironic jobs from Tempest
17:45:12 <jlvillal> http://lists.openstack.org/pipermail/openstack-dev/2016-October/105598.html
17:45:15 <jroll> hrm
17:45:19 <jroll> thiagop was working on that
17:45:26 <jroll> whatever happened to him?
17:45:33 * jlvillal doesn't know
17:46:04 <jroll> that email went to my spam O_o
17:46:11 <jlvillal> :(
17:46:42 <jlvillal> jroll: For me Gmail says: This message was not sent to Spam because of a filter you created. Edit Filters
17:46:59 * jlvillal had made filter that said never send openstack-dev email to spam.
17:47:18 <jroll> oh, neat
17:47:34 <jlvillal> Anything else to discuss?
17:47:42 * jroll suddenly realizes he isn't in -qa D:
17:47:45 <jroll> I have nothing
17:47:53 <jlvillal> heh
17:47:58 <jlvillal> Okay. Thanks everyone!
17:48:05 <jlvillal> #endmeeting