17:02:06 <jlvillal> #startmeeting ironic_qa
17:02:07 <openstack> Meeting started Wed Sep 28 17:02:06 2016 UTC and is due to finish in 60 minutes.  The chair is jlvillal. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:11 <openstack> The meeting name has been set to 'ironic_qa'
17:02:18 <jlvillal> Sorry I'm late. I was fighting with the kettle
17:02:33 <jlvillal> Trying to get some hot water before the meeting. Failed :(
17:02:40 <cdearborn> o/
17:02:41 <rpioso> o/
17:02:52 <jroll> \o
17:02:53 <sambetts> o/
17:02:59 <jlvillal> #topic Announcements
17:03:36 <jlvillal> I don't have any announcements
17:03:39 <jlvillal> Anyone else
17:03:41 <jlvillal> ?
17:03:47 <jroll> grenade is down
17:03:49 <jroll> seems relevant :)
17:03:51 <jlvillal> As always the agenda is: https://wiki.openstack.org/wiki/Meetings/Ironic-QA
17:04:02 <jlvillal> Doop!
17:04:22 <jlvillal> #info Grenade job is broken. Unable to contact host issue
17:04:33 <sambetts> Devstack subnet pools seems to have broken several third party CIs
17:04:47 <sambetts> including mine..
17:04:51 <jlvillal> sambetts: I was asking Vasyl if he though it could be an issue.
17:05:06 <jlvillal> s/though/thought/
17:05:33 <jlvillal> sambetts: Do you think it could be related?
17:05:44 <jlvillal> And how did you fix your CI?
17:06:00 <jlvillal> #topic Grenade gate job broken
17:06:12 <sambetts> I havn't yet, its still broken, I'm waiting for https://review.openstack.org/#/c/378063/
17:06:36 <sambetts> if that gets accepted, otherwise I'm going to have to rework my local.conf it seems
17:06:53 <jlvillal> #info There was a patch regarding subnetpools that 'might' be related to the breakage: https://review.openstack.org/#/c/356026/
17:07:26 <jroll> might be worth someone putting up an ironic patch to disable that and see what happens
17:07:34 <rajinir> o/
17:07:35 <jlvillal> #info vsaienk0 is looking at it, thinks it might be neutron being too slow: https://review.openstack.org/#/c/377653/
17:07:51 <jlvillal> I'll give it a shot after the meeting.
17:08:16 <jlvillal> Anything else regarding broken Grenade job?
17:08:40 <jlvillal> #topic Multi-node Grenade
17:09:01 <jlvillal> #info Work continuing on getting patches merged. List is at: https://etherpad.openstack.org/p/ironic-newton-grenade-whiteboard
17:09:23 <jlvillal> #info Some patches have been merged. But still a lot left.
17:09:37 <jlvillal> That's all I have.
17:10:09 <jlvillal> #topic 3rd Party CI (krtaylor)
17:10:26 <jlvillal> I don't see krtaylor in the meeting.
17:10:31 <jlvillal> sambetts: Do you have anything?
17:10:37 <rajinir> Is there guidelines to recheck tests that are failing ?
17:10:51 <jlvillal> rajinir: Can I get more details?
17:11:00 <jlvillal> rajinir: Is this in you are writing a 3rd Party CI?
17:11:03 <rajinir> recheck patches that failed in CI.
17:11:17 <rajinir> Yes, the CI builds are broken because of subnet pool issue
17:11:28 <sambetts> I have pinned my version of devstack in my CI before the subnet pools merged in the hopes that  I can get it working while we work out the problems with the new version
17:11:35 <jlvillal> rajinir: Okay, that is unrelated to 3rd Party CI, I think.
17:11:37 <krtaylor> o/ hey everybody, sorry I'm late
17:12:02 <jlvillal> rajinir: But basically don't do 'recheck' until the issue is determined. We are not sure what is the cause of the broken grenade gate job yet.
17:12:04 <rajinir> jlvillal: The thirdparty Ci builds are broken for us
17:12:35 <sambetts> rajinir: you should have a separate recheck command for your third party CI
17:12:36 <jlvillal> rajinir: Each 3rd Party CI should expose a command to only recheck their job.
17:13:03 <rajinir> We do, but going and rechecking every single build that failed for the past two days can be painful
17:13:04 <jlvillal> rajinir: And that command should be stated by the 3rd Party CI in the message it posts.
17:13:10 <rajinir> Probably can write a script
17:13:16 <jlvillal> sambetts: krtaylor Know the command format.
17:13:29 <jlvillal> ??
17:13:33 <krtaylor> well, technically, it is just recheck, it should trigger all systems
17:13:44 <jlvillal> krtaylor: Right we don't want to do that. That is wasteful.
17:13:48 <rajinir> Have fixed it now with the upstream pending review. Builds are working now
17:14:03 <krtaylor> yes you do, you don't want to get in a situation where one passes another doesnt
17:14:03 <sambetts> I think rajinir is asking do they need to recheck it for all the patches their CI failed on
17:14:11 <krtaylor> and you just recheck until it passes
17:14:24 <jlvillal> krtaylor: What sambetts said.
17:14:29 <krtaylor> however, that is not done generally
17:14:57 <krtaylor> sambetts, it can be either the test system that rechecks or the developer
17:14:58 <sambetts> as in is it the Third party CI maintainers responsibilty to recheck all the patches
17:15:17 <sambetts> or the patch owners
17:15:34 <jlvillal> rajinir: Yes you could probably right a script to figure out every patch your CI voted -1 and execute a recheck of your CI.
17:15:34 <jroll> I'd say patch owner / reviewer
17:15:43 <sambetts> ++
17:15:49 <krtaylor> there are those that believe each side is responsible, I believe it is up to the project to decide
17:15:50 <rajinir> +1
17:16:11 <jlvillal> Anyway, I will let krtaylor run this part of the meeting :)
17:16:25 <krtaylor> hehheh, it seems to be running fine :)
17:16:35 <sambetts> I would suggest that when a third party CI has issues, an email is sent to the ML to state that people are to expect the failures so they don't spam rechecks
17:17:00 <krtaylor> yes, a dev can also check http://ci-watch.tintri.com/project?project=ironic
17:17:02 <sambetts> and then when the issues is resolved a second one is sent out to make people aware the issue is fixed and should recheck their own patches
17:17:17 <krtaylor> it is good practice
17:17:56 <sambetts> I didn't know about that, that site is awesome!
17:18:01 <krtaylor> the goal is to get the CI dashboard hosted as an official infra service, then there are things that can be automated, such as email
17:18:01 <sambetts> #link http://ci-watch.tintri.com/project?project=ironic
17:19:20 <krtaylor> the effort stalled
17:19:31 <krtaylor> there is an infra spec somewhere...
17:19:42 * krtaylor shuffles around in his bookmarks...
17:20:27 <rajinir> Link is useful. Thanks
17:20:32 <krtaylor> we had proposed at one time for CI systems to update their status on the ThirdPartySystems wiki
17:20:56 <krtaylor> but it is hacky at best to show that on the index page
17:21:16 <krtaylor> and no programmatic way to do it
17:21:22 <krtaylor> but I digress
17:21:30 <krtaylor> anything else on CI?
17:21:57 <sambetts> Nothing from me
17:22:05 <rajinir> — me too
17:22:31 <krtaylor> k, I'll turn it back over to jlvillal
17:22:47 <jlvillal> Thanks krtaylor
17:22:51 <jlvillal> #topic Open Discussion
17:22:57 <jlvillal> Anyone have anything?
17:23:26 <jlvillal> <openstackgerrit> John L. Villalovos proposed openstack/ironic: Testing a revert to see if fixes Grenade  https://review.openstack.org/378928
17:23:36 <jlvillal> I'll see if that makes any difference
17:24:08 <jlvillal> If there is nothing else...
17:24:11 <krtaylor> I do, re: molten iron - is there enough interest in test systems using it to propose it as a sub-project?
17:24:32 <krtaylor> rajinir, you all are using it too right?
17:25:06 <rajinir> krtaylor: In progress, yes, would be nice to add that as sub-project
17:25:27 <krtaylor> there are a couple of patches to add tests
17:25:55 <krtaylor> reviews would be nice, that will get it closer to a cookie cutter project proposal
17:27:04 <jlvillal> Thanks krtaylor.
17:27:06 <jlvillal> Anything else?
17:27:09 <krtaylor> #link https://review.openstack.org/#/c/371729/
17:27:26 <rajinir> Will review, sure, thks
17:27:35 <krtaylor> #link https://review.openstack.org/#/c/370218/
17:27:42 <krtaylor> I think thats them
17:27:51 <krtaylor> hamzy on my team is working on that
17:28:17 <krtaylor> well, if there is enough interest, I'll put it on the list to discuss at summit
17:28:34 <sambetts> ++
17:28:58 <krtaylor> thats it for me jlvillal
17:29:14 <rajinir> ++
17:29:24 <jlvillal> Thanks everyone.
17:29:31 <jlvillal> I think we are done.
17:29:37 <sambetts> thanks jlvillal
17:29:39 <krtaylor> thanks!
17:29:42 <jlvillal> #endmeeting