17:02:06 <jlvillal> #startmeeting ironic_qa 17:02:07 <openstack> Meeting started Wed Sep 28 17:02:06 2016 UTC and is due to finish in 60 minutes. The chair is jlvillal. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:02:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:02:11 <openstack> The meeting name has been set to 'ironic_qa' 17:02:18 <jlvillal> Sorry I'm late. I was fighting with the kettle 17:02:33 <jlvillal> Trying to get some hot water before the meeting. Failed :( 17:02:40 <cdearborn> o/ 17:02:41 <rpioso> o/ 17:02:52 <jroll> \o 17:02:53 <sambetts> o/ 17:02:59 <jlvillal> #topic Announcements 17:03:36 <jlvillal> I don't have any announcements 17:03:39 <jlvillal> Anyone else 17:03:41 <jlvillal> ? 17:03:47 <jroll> grenade is down 17:03:49 <jroll> seems relevant :) 17:03:51 <jlvillal> As always the agenda is: https://wiki.openstack.org/wiki/Meetings/Ironic-QA 17:04:02 <jlvillal> Doop! 17:04:22 <jlvillal> #info Grenade job is broken. Unable to contact host issue 17:04:33 <sambetts> Devstack subnet pools seems to have broken several third party CIs 17:04:47 <sambetts> including mine.. 17:04:51 <jlvillal> sambetts: I was asking Vasyl if he though it could be an issue. 17:05:06 <jlvillal> s/though/thought/ 17:05:33 <jlvillal> sambetts: Do you think it could be related? 17:05:44 <jlvillal> And how did you fix your CI? 17:06:00 <jlvillal> #topic Grenade gate job broken 17:06:12 <sambetts> I havn't yet, its still broken, I'm waiting for https://review.openstack.org/#/c/378063/ 17:06:36 <sambetts> if that gets accepted, otherwise I'm going to have to rework my local.conf it seems 17:06:53 <jlvillal> #info There was a patch regarding subnetpools that 'might' be related to the breakage: https://review.openstack.org/#/c/356026/ 17:07:26 <jroll> might be worth someone putting up an ironic patch to disable that and see what happens 17:07:34 <rajinir> o/ 17:07:35 <jlvillal> #info vsaienk0 is looking at it, thinks it might be neutron being too slow: https://review.openstack.org/#/c/377653/ 17:07:51 <jlvillal> I'll give it a shot after the meeting. 17:08:16 <jlvillal> Anything else regarding broken Grenade job? 17:08:40 <jlvillal> #topic Multi-node Grenade 17:09:01 <jlvillal> #info Work continuing on getting patches merged. List is at: https://etherpad.openstack.org/p/ironic-newton-grenade-whiteboard 17:09:23 <jlvillal> #info Some patches have been merged. But still a lot left. 17:09:37 <jlvillal> That's all I have. 17:10:09 <jlvillal> #topic 3rd Party CI (krtaylor) 17:10:26 <jlvillal> I don't see krtaylor in the meeting. 17:10:31 <jlvillal> sambetts: Do you have anything? 17:10:37 <rajinir> Is there guidelines to recheck tests that are failing ? 17:10:51 <jlvillal> rajinir: Can I get more details? 17:11:00 <jlvillal> rajinir: Is this in you are writing a 3rd Party CI? 17:11:03 <rajinir> recheck patches that failed in CI. 17:11:17 <rajinir> Yes, the CI builds are broken because of subnet pool issue 17:11:28 <sambetts> I have pinned my version of devstack in my CI before the subnet pools merged in the hopes that I can get it working while we work out the problems with the new version 17:11:35 <jlvillal> rajinir: Okay, that is unrelated to 3rd Party CI, I think. 17:11:37 <krtaylor> o/ hey everybody, sorry I'm late 17:12:02 <jlvillal> rajinir: But basically don't do 'recheck' until the issue is determined. We are not sure what is the cause of the broken grenade gate job yet. 17:12:04 <rajinir> jlvillal: The thirdparty Ci builds are broken for us 17:12:35 <sambetts> rajinir: you should have a separate recheck command for your third party CI 17:12:36 <jlvillal> rajinir: Each 3rd Party CI should expose a command to only recheck their job. 17:13:03 <rajinir> We do, but going and rechecking every single build that failed for the past two days can be painful 17:13:04 <jlvillal> rajinir: And that command should be stated by the 3rd Party CI in the message it posts. 17:13:10 <rajinir> Probably can write a script 17:13:16 <jlvillal> sambetts: krtaylor Know the command format. 17:13:29 <jlvillal> ?? 17:13:33 <krtaylor> well, technically, it is just recheck, it should trigger all systems 17:13:44 <jlvillal> krtaylor: Right we don't want to do that. That is wasteful. 17:13:48 <rajinir> Have fixed it now with the upstream pending review. Builds are working now 17:14:03 <krtaylor> yes you do, you don't want to get in a situation where one passes another doesnt 17:14:03 <sambetts> I think rajinir is asking do they need to recheck it for all the patches their CI failed on 17:14:11 <krtaylor> and you just recheck until it passes 17:14:24 <jlvillal> krtaylor: What sambetts said. 17:14:29 <krtaylor> however, that is not done generally 17:14:57 <krtaylor> sambetts, it can be either the test system that rechecks or the developer 17:14:58 <sambetts> as in is it the Third party CI maintainers responsibilty to recheck all the patches 17:15:17 <sambetts> or the patch owners 17:15:34 <jlvillal> rajinir: Yes you could probably right a script to figure out every patch your CI voted -1 and execute a recheck of your CI. 17:15:34 <jroll> I'd say patch owner / reviewer 17:15:43 <sambetts> ++ 17:15:49 <krtaylor> there are those that believe each side is responsible, I believe it is up to the project to decide 17:15:50 <rajinir> +1 17:16:11 <jlvillal> Anyway, I will let krtaylor run this part of the meeting :) 17:16:25 <krtaylor> hehheh, it seems to be running fine :) 17:16:35 <sambetts> I would suggest that when a third party CI has issues, an email is sent to the ML to state that people are to expect the failures so they don't spam rechecks 17:17:00 <krtaylor> yes, a dev can also check http://ci-watch.tintri.com/project?project=ironic 17:17:02 <sambetts> and then when the issues is resolved a second one is sent out to make people aware the issue is fixed and should recheck their own patches 17:17:17 <krtaylor> it is good practice 17:17:56 <sambetts> I didn't know about that, that site is awesome! 17:18:01 <krtaylor> the goal is to get the CI dashboard hosted as an official infra service, then there are things that can be automated, such as email 17:18:01 <sambetts> #link http://ci-watch.tintri.com/project?project=ironic 17:19:20 <krtaylor> the effort stalled 17:19:31 <krtaylor> there is an infra spec somewhere... 17:19:42 * krtaylor shuffles around in his bookmarks... 17:20:27 <rajinir> Link is useful. Thanks 17:20:32 <krtaylor> we had proposed at one time for CI systems to update their status on the ThirdPartySystems wiki 17:20:56 <krtaylor> but it is hacky at best to show that on the index page 17:21:16 <krtaylor> and no programmatic way to do it 17:21:22 <krtaylor> but I digress 17:21:30 <krtaylor> anything else on CI? 17:21:57 <sambetts> Nothing from me 17:22:05 <rajinir> — me too 17:22:31 <krtaylor> k, I'll turn it back over to jlvillal 17:22:47 <jlvillal> Thanks krtaylor 17:22:51 <jlvillal> #topic Open Discussion 17:22:57 <jlvillal> Anyone have anything? 17:23:26 <jlvillal> <openstackgerrit> John L. Villalovos proposed openstack/ironic: Testing a revert to see if fixes Grenade https://review.openstack.org/378928 17:23:36 <jlvillal> I'll see if that makes any difference 17:24:08 <jlvillal> If there is nothing else... 17:24:11 <krtaylor> I do, re: molten iron - is there enough interest in test systems using it to propose it as a sub-project? 17:24:32 <krtaylor> rajinir, you all are using it too right? 17:25:06 <rajinir> krtaylor: In progress, yes, would be nice to add that as sub-project 17:25:27 <krtaylor> there are a couple of patches to add tests 17:25:55 <krtaylor> reviews would be nice, that will get it closer to a cookie cutter project proposal 17:27:04 <jlvillal> Thanks krtaylor. 17:27:06 <jlvillal> Anything else? 17:27:09 <krtaylor> #link https://review.openstack.org/#/c/371729/ 17:27:26 <rajinir> Will review, sure, thks 17:27:35 <krtaylor> #link https://review.openstack.org/#/c/370218/ 17:27:42 <krtaylor> I think thats them 17:27:51 <krtaylor> hamzy on my team is working on that 17:28:17 <krtaylor> well, if there is enough interest, I'll put it on the list to discuss at summit 17:28:34 <sambetts> ++ 17:28:58 <krtaylor> thats it for me jlvillal 17:29:14 <rajinir> ++ 17:29:24 <jlvillal> Thanks everyone. 17:29:31 <jlvillal> I think we are done. 17:29:37 <sambetts> thanks jlvillal 17:29:39 <krtaylor> thanks! 17:29:42 <jlvillal> #endmeeting