15:00:22 <anteaya> #startmeeting third-party
15:00:24 <openstack> Meeting started Mon Aug  1 15:00:22 2016 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:27 <openstack> The meeting name has been set to 'third_party'
15:00:39 <asselin> o/
15:00:44 <anteaya> hello asselin
15:00:45 <lennyb> hi
15:00:48 <anteaya> how are you today?
15:00:53 <anteaya> hey lennyb
15:00:55 <asselin> hi anteaya lennyb
15:02:05 <anteaya> does anyone have anything they would like to discuss today?
15:02:21 <lennyb> some tempest review for multi node scenario tests#link https://review.openstack.org/#/c/343294/  #link https://review.openstack.org/#/c/335447/
15:02:32 <rhedlind> hi
15:02:47 <anteaya> hey rhedlind
15:03:41 <anteaya> lennyb: have you asked anyone in the #openstack-qa channel for reviews on those patches?
15:03:54 <lennyb> anteaya: will do
15:04:42 <anteaya> thanks
15:04:52 <anteaya> since tempest is their repo
15:05:10 <anteaya> I think having them at least okay the direction at this stage is a good idea
15:05:37 <anteaya> any more on lennyb's tempest patches?
15:05:47 <lennyb> no
15:05:56 <anteaya> okay great, thanks lennyb
15:06:07 <anteaya> does anyone have anything else they would like to talk about today?
15:06:37 <rhedlind> if anyone has time to help me with a CI issue, I could use some help
15:06:51 <lennyb> rhedlind: what issue?
15:07:29 <rhedlind> Zuul and Jenkins are not syncing up. Zuul sees the trigger events coming in but it never kicks of any jobs in jenkins
15:07:46 <lennyb> what do you see in debug log?
15:08:10 <lennyb> did it work with sandbox-ci?
15:08:56 <rhedlind> trying to find an example of a cinder event
15:08:57 <lennyb> do you have existing /var/run/zuul/zuul.pid ?
15:09:10 <rhedlind> it used to be up and running openstack/cinder testing
15:11:18 <mmedvede> one good thing to do is to check state of gearman worker queue, i.e. 'echo status | nc <zuul_ip> 4730'. It should show all the functions jenkins has registered with gearman
15:11:31 <lennyb> in my CI in such cases the issue was usually bad layout.yaml syntax . Did you validated layout.yaml ?
15:12:07 <asselin> in your project-config you should have a tox.ini. tox -e zuul should be the command to validate
15:12:09 <lennyb> rhedlind: there are a lot of online yaml validators
15:12:09 <rhedlind> oh, I forgot about the validation feature. I will try that right now
15:12:48 <lennyb> rhedlind: also check if zuul.pid exists
15:13:15 <asselin> #link layout.yaml and jenkins job build configuration validation: http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/tox.ini#n39
15:13:50 <rhedlind> ok, I will take a look at this ideas. thanks!
15:15:32 <anteaya> any more on this topic?
15:15:40 * lennyb 1sec
15:15:59 <rhedlind> no, I don't want to take up any more of your time on it. Thanks for the ideas!
15:16:18 <lennyb> rhedlind: also check if zuul is not locked in loop. I've noticed that from time to time. a lot of exceptions in zuul.log
15:16:27 * lennyb done
15:16:49 <anteaya> okay thanks rhedlind for the topic
15:17:17 <anteaya> does anyone have anything else they would like to discuss?
15:17:23 <lennyb> yes
15:17:56 <anteaya> go ahead lennyb
15:17:58 <lennyb> from time to time I've noticed that in case that gearman cant see a job to run or no valid slaves are avaliable
15:18:18 <lennyb> it comments on gerrit with NOT_REGISTERED job
15:18:39 <anteaya> yes
15:18:43 <lennyb> this issue is very hard to monitor for me. All I found as a solution is grep the log
15:18:50 <anteaya> ah
15:18:54 <lennyb> any ideas how to prevent it?
15:19:11 <mmedvede> +1 on preventing it
15:19:17 <lennyb> or monitor it?
15:19:32 <anteaya> well infra used to get it when the jenkins output specified one node and the gearman was looking for another kind of node
15:19:56 <anteaya> we changed how nodes are specified and that has removed a lot of NOT_REGISTERED for us
15:20:10 <anteaya> and as for monitoring we have humans tell us when there is a problem
15:20:12 <anteaya> however
15:20:26 <anteaya> I can see you would not be able to rely on the same mechanism
15:20:37 <mmedvede> it is still a problem when you restart zuul, and jenkins did not have time to register all jobs yet
15:20:40 <lennyb> anteaya: the last time human said it to me, our CI was disabled due to this issue :)
15:20:49 <mmedvede> imo, zuul should not throw a fit every time it does not see a job :)
15:20:50 <anteaya> so as of right now, I personally don't know of any way to monitor or prevent it
15:21:12 <anteaya> lennyb: yes, right, then you need a better way other than humans
15:21:25 <lennyb> maybe we can disable commenting in such cases
15:21:34 <anteaya> okay well perhaps you can draft an email to the infra list lennyb
15:21:40 <mmedvede> lennyb: I was thinking patching zuul to prevent it
15:21:47 <lennyb> mmedvede +1
15:21:54 <anteaya> and we can discuss it and inlude the rest of the infra folks and other third party operators
15:22:16 <anteaya> mmedvede: let's raise some awareness first so your patch will get some attention
15:22:17 <lennyb> anteaya: ok, I will drop an email
15:22:32 <anteaya> since folks will be aware of your situation and hence the motivation for the patch
15:22:35 <mmedvede> I was thinking local patch, not going into community
15:22:38 <anteaya> does that sound reasonable?
15:22:47 <anteaya> mmedvede: okay well that is your choice
15:22:50 <mmedvede> but I'll see if it is possible
15:22:56 <anteaya> but upstream might be willing to consider it
15:22:58 <anteaya> thanks
15:23:07 <lennyb> mmedvede: we are community if it's good for you it will be good for me as well :)
15:23:16 <anteaya> :)
15:23:19 <mmedvede> right :)
15:23:39 <anteaya> do we have any more on this matter?
15:23:49 <mmedvede> lennyb: if I find a way, I'll push a patch. It was more of an annoyance for me with "NOT_REGISTERED"
15:24:03 <lennyb> mmedvede ok
15:24:44 * lennyb finished with this issue
15:24:55 <anteaya> thank you for bringing it up for discussion
15:25:11 <anteaya> does anyone have any other item they would like to discuss today?
15:26:27 <anteaya> does anyone have any objection to me closing the meeting?
15:27:13 <anteaya> thanks everyone for you kind attendance and participation today
15:27:20 <anteaya> enjoy the rest of your day/evening
15:27:27 <anteaya> see you all next week
15:27:29 <anteaya> thank you
15:27:31 <anteaya> #endmeeting