15:02:04 #startmeeting third-party 15:02:05 Meeting started Mon Mar 20 15:02:04 2017 UTC and is due to finish in 60 minutes. The chair is lennyb. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:02:08 The meeting name has been set to 'third_party' 15:02:13 Hello 15:03:32 o/ 15:03:41 mmedvede, hi, 15:03:59 do you run fedora 24 in your CI? 15:04:05 o/ 15:04:41 hello asselin 15:04:50 lennyb: no, we only run Ubuntu at the moment. 15:04:53 o/ 15:04:54 I am getting strange error http://paste.openstack.org/show/603440/ 15:04:58 hello pots 15:05:15 mmedvede what verion of ubuntu? 15:05:43 lennyb: our dsvm tests use Xenial 15:06:55 mmedvede, I see. 15:07:37 asselin, btw, your suggestion regarding zuul configuration for using one time node is not working for us, since we use multijob plugin. 15:08:02 post, asselin, mmedvede - anything you would like to discuss today? 15:08:04 what is the multijob plugin? Is that jenkins? 15:08:37 asselin, yes jenkins multijob plugin https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin 15:08:55 one job to run a list of other jhobs 15:09:27 it is also supported by JJB 15:10:40 * asselin_ having network issues 15:11:09 I also face something similiar to #link https://bugs.launchpad.net/zuul/+bug/1270029 . with zuul 2.5.1. I am trying to check the proposed workaround 15:11:09 Launchpad bug 1270029 in Zuul "zuul doesn't connect to gearman server" [Undecided,New] 15:11:32 lennyb: I see...I don't think zuul and multijob plugin will play nice together. 15:12:18 lennyb: I think I saw this as well 15:12:31 asselin_, is there a way to make jobs dependable in zuul? 15:13:01 like Build JOB -> Install JOB -> Test Job 15:13:29 lennyb: I don't think so...maybe zuul v3 has that. 15:13:51 lennyb: otherwise zuul does have post-build jobs ability, but not quite like you want 15:13:57 zuul v3 (maybe even 2.5) should have that 15:13:59 mptacekx, what are you doing to overcome it? mmedvede uses zuul restart one a week. 15:14:38 mmedvede, I will check 2.5, thanks 15:15:10 lennyb: but you should be able to structure the individual parts of the job in JJB, then combine them into one high level job...but you do lose the parallel part that I suppose you're looking for. 15:15:23 lennyb: I saw it just once in couple of months with zuul 2.5.1, had to restart zuul to wake him up. Not a real cure 15:16:53 asselin_, I use multijob plugin for this. 15:17:41 I can also run some jobs in parallel, there. 15:18:03 Is there anything else you would like to share/ask/discuss today? 15:18:09 lennyb: so remind me what the issue is again? Using multijob plugin with nodepool? 15:18:59 asselin_, I must add quite-period of 1min to a job, since it take s time for a nodepool to remove a slave from the jenkins. 15:19:41 and since all the job takes about 10min, 3 minutes of quite-period is 30% of the run time :) 15:20:13 but it's not that critical. so let's move to another things, if there are 15:20:15 lennyb: ok I see...yeah, might need to use that workaround since the tooling isn't designed to work as you want. 15:20:32 lennyb: i tried the quiet-period change but it had no effect--is there any way to fix this in zuul? 15:20:34 lennyb: ok. btw in zuulv3 it will switch to ansible and you'll have better support for the use case 15:21:09 pots, did you try asselin_ 's suggestion from the last week> 15:21:32 i don't recall what it was? 15:22:38 * lennyb looking 15:22:58 #link http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2017-03-13.log.html 15:24:24 this here? http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2017-03-13.log.html#t2017-03-13T15:04:54 15:24:24 oops, i did not see that. 15:24:50 anything else on this issue? 15:25:42 any other issues? 15:25:58 i've already got the parameter-function: single_use_node entry 15:26:16 pots, what is not working in your setup? 15:26:48 pots: are you using it in zuul's layout.yaml? 15:27:25 i'm using openstackci-puppet on top of devstack, with two jobs defined (one for fc, one iscsi). whenever a job completes, it looks like zuul starts the next job using the old nodepool vm, which fails. then nodepool comes around and kills the old vm a minute later. 15:27:44 so basically every other job fails 15:27:53 pots: can you share your zuul layout.yaml file? 15:28:16 pots, do you have jenkins? 15:28:20 does it look like what I shared before? http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2017-03-13.log.html#t2017-03-13T15:05:19 15:28:32 - name: ^dsvm-tempest.*$ parameter-function: single_use_node 15:29:33 yes, it looks like that. 15:29:41 sorry, can't seem to cut & paste today 15:29:49 btw I need to leave in 2 mins. pots if you still have issues we can discuss in openstack-infra. Others there might be able to help. 15:30:04 discuss later* 15:30:04 thanks, asselin_ 15:30:10 ok, i will see you there. thanks! 15:30:28 * asselin_ leaves now 15:30:37 pots, do you use jenkins? 15:30:59 yes, i'm following the recipe for openstackci-puppet. same issue with jenkins 2.x and 1.651 and 1.656 15:32:00 pots, do you see 'quite period' in jenkins job? 15:32:32 yes 15:32:35 newer jenkins filters out injected vars by default. Zuul coordinates single use VM action with jenkins via a job parameter iirc. You may need to whitelist that var 15:33:21 pots, so quite period should work anyway, maybe increase the delay. 15:33:22 i got the impression that quiet-period only affected the normal jenkins repo triggers and would not necessarily impact gearman 15:33:49 but i will try a much larger value and see if it has any effect. 15:34:29 pots, in our case Jenkins waits before starting job, in the meantime nodepool have enough time to delete slave from the jenkins 15:36:42 i was wondering if there might be some way to modify zuul/gearman behavior. e.g. doesn't gearman choose a specific executor before giving the job to jenkins? 15:37:16 pots, try asking in infra channel 15:38:30 will do. one last question, i was wondering what the conventional wisdom was re: setting up a new cloud for the CI to run on, something closer to a devstack that can survive a reboot? 15:39:03 pots, we have CI on RDO based cloud 15:39:13 pots: lennyb ^ seem my earlier comment 15:39:49 clarkb thanks, i resolved that issue earlier. 15:39:50 pots, but we are considering to check Fuel as cloud provider for CI 15:39:59 pots: we have a full blown cloud that we maintain (not based on devstack) 15:40:16 pots: did you resolve it by whitelisting vars? and if so did you whitelist the var that tells gearman to not reuse a host? 15:40:37 its something like OFFLINE_NODE or something 15:41:03 clarkb: I believe the way most third-party CI's do is to disable the security feature of jenkins, so no need to whitelist variables. But pots' setup might be different 15:41:13 clarkb: I thought I whitelisted everything 15:42:31 clarkb: asselin_ showed me an additional parameter for jenkins that works. 15:44:50 i don't see any OFFLINE_XXX parameter in the jenkins jobs 15:46:56 pots, asselin_ proposed zuul based param(I dont think it will be reflected in Jenkins). I proposed quite-period that can be seen in jenkins. 15:47:19 params['OFFLINE_NODE_WHEN_COMPLETE'] = '1' is what single_user_node sets in the gearman communication from zuul to jenkins 15:47:47 so that should appear in the job parameters in the jenkins UI? 15:49:18 it may, jenkins is weird about what params it shows iirc 15:51:11 it seems like a race condition, i think when i looked at the logs you could see the wrong thing happening in the same second. 15:51:36 my systems are very slow, so that just makes everything more fun. 15:52:26 lennyb so would you recommend chucking xenial/devstack for RHEL7/packstack for an all-in-one to run the CI on? 15:52:36 the actual implementation of offline node when complete should toggle the offline bit when job is running then jenkins puts it into effect when the job is completed (so there could be a race there somewher) 15:53:58 any way to debug the gearman stuff? 15:54:02 pots, I am not familiar with xenial,packstack. we use rdo #link https://www.rdoproject.org . It's RedHat oriented, but not really packstack 15:55:09 i'm just looking for an All-In-One solution that works out of the box 15:56:29 pots, rdo works out of the box. to be honest, we had to add networks manually, but it was quite straight forward. I had a lot of positive things regarding Fuel, but did not use it yet 15:56:50 i will look into that, thanks. 15:57:19 any other shares/questions/proposals ? before our time ends? 15:57:24 i tried openstack-ansible but it did not pass it's own tests 15:57:51 lennyb: not for me, thanks all 15:58:38 Great, mmedvede, clarkb, pots, mptacekx, asselin_ .. thanks. see you next week 15:58:41 pots: yes one thing you could do to help debug it is while a job is running use the nodepool hold command to hold that node (it won't be deleted) then check that the node is marked offline properly when the job completes 15:59:34 clarkb, can I 'release' node without deleting it after hold? 16:00:33 release it from what perspective? you can delete it in jenkins so jenkins will stop doing things to it 16:00:43 but if you delete it in nodepool it iwll get deleted from the cloud 16:01:47 clarkb, ignore my question. thanks. 16:01:54 #endmeeting