15:01:32 <lennyb> #startmeeting third-party
15:01:33 <openstack> Meeting started Mon Jan 23 15:01:32 2017 UTC and is due to finish in 60 minutes.  The chair is lennyb. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:36 <openstack> The meeting name has been set to 'third_party'
15:01:45 <lennyb> Hello
15:02:50 <mptacekx> Hi
15:03:57 <asselin_> hi
15:04:03 <mptacekx> may I propose one topic ? (3rd party CI - using hugepages on nodepool VM's)
15:04:20 <lennyb> hello everybody
15:04:34 <lennyb> mptaceks, go ahead
15:04:50 <lennyb> mptacekx, go ahead
15:05:22 <mmedvede> o/
15:05:29 <lennyb> mptacekx, please ask
15:06:01 <mptacekx> thanks, on our CI we're testing ovs-dpdk on VM's. So we have hugepages configured on them, alas after some time (4+ weeks) env looks to be slower and less stable
15:07:04 <mptacekx> the question is whether anyone from this round have some experience with hugepages on 3rd party CI's ?
15:07:08 <lennyb> mmedvede, asselin_ do you use hugepages in VMs? any ideas/proposals how to clean/fix this?
15:07:29 <asselin_> no we don't
15:07:46 <mmedvede> we do not as well
15:07:56 <mptacekx> we also see gradual increase of "inactively" used RAM on hypervisors, expecting that something is not getting cleaned
15:08:18 <mptacekx> it disappears after reboot, which is naturally not the best choice how to fix that :-)
15:08:30 <asselin_> seems like an issue in the underlying cloud.
15:08:38 <asselin_> have you searched for exisitng bug reports?
15:09:08 <mptacekx> I guess it's too specific, the question is whether we should configure hugepages also on hypervisor level
15:10:17 <mptacekx> btw usually we don't do any maintenance in CI's (inc. reboots) ?
15:10:51 <mptacekx> currently running zuul v2.1 -> would like to move to v2.5.1 soon (eventually 3.x)
15:11:22 <asselin_> I don't think it's related to zuul
15:11:35 <asselin_> do you use nodepool?
15:11:49 <mptacekx> jup
15:12:28 <lennyb> mptaceks, we used hugepages in vm in another project and if I recall correctly we configured hugepages in hypervisor as well. but we never run this setup for that long
15:12:35 <asselin_> nodepool does create and delete a lot of VMs. Perhaps there's a bug in the underlying cloud that is "leaking" resources after a certain number of create/delete cycles
15:12:55 <mptacekx> I also don't think it's related to zuul, I put it here just to my general question about maintenance in other CI's and reboot ?
15:13:49 <mmedvede> re maintenance, we restart zuul every week as a preventative measure
15:13:53 <mptacekx> probably we should configure them and also we would have better overview about how they are occupied/freed then
15:14:19 <mmedvede> mptacekx: what version of OpenStack are you running?
15:14:37 <mptacekx> no reboot on hypervisors ? "just service restart) are hypervisors running all the time ...
15:14:42 <lennyb> mmedvede, why? if there a reason/instability why not open a bug?
15:14:42 <mptacekx> mitaka
15:14:51 <mmedvede> there used to be a few memory leaks in the past
15:15:11 <mmedvede> ok, I am not aware on any memory leaks in mitaka
15:16:09 <mmedvede> lennyb: zuul appears to be leaking ram for me. And I am not running the latest
15:16:58 <lennyb> mmedvede, i see, we are using 2.5, but from time to time it has some strange behavior. maybe I will add restart as well :)
15:17:27 <mmedvede> lennyb: I brought it up at some point, and there were patches to fix the leak, which helped. But it is still getting to very hight RAM utilization if you do not periodically restart it
15:17:34 <lennyb> any other ideas/questions regarding hugepages issue?
15:18:15 <lennyb> next issue
15:18:16 <mptacekx> on v 2.1 we also had that damned issue with keeping list of all projects in layout.yaml, otherwise zuul might stalls (in some discussions I found it should be fixed in v 2.5.x), so I hope it won't be neccessary in v 2.5.x scope
15:18:36 <mptacekx> thanks for your time regarding hugepages
15:18:56 <lennyb> mptacekx, thanks for bringing this up
15:19:05 <lennyb> I've added a patch to overcome nodepool-base element installation with all openstack repo https://review.openstack.org/#/c/424000/1
15:19:47 <lennyb> I am working on nodepool and asselin_ work on comon ci solution is very helpful
15:20:35 <lennyb> I wander if jenkins configuration and installation of additional plugins can be done through the puppets
15:21:46 <asselin_> lennyb, additional plugin install yes. jenkins config yes, but it's a bit involved
15:22:26 <lennyb> asselin_, thanks, I will check when I will have some time
15:23:03 <lennyb> I also notices that if there are some critical errors/expcetions in zuul/nodepool it's not printed to the logs and you need to run it manually to see the problem
15:24:10 <asselin_> lennyb, yes, I've seen that too...especially for issue that happen during service startup
15:24:54 <lennyb> asselin_, yeap, no .pid file is a good symptom that something wrong
15:25:50 <lennyb> mmedvede, thanks for helping with nodepool
15:26:40 <mmedvede> lennyb: glad to help
15:26:44 <lennyb> asselin_, mmedvede, mptacekx any other issues/announcements/topics to share?
15:28:26 <asselin_> not from me
15:29:20 <mmedvede> I have nothing
15:30:05 <lennyb> asselin_, mmedvede, mptacekx thanks for coming. I am closing the meeting
15:30:38 <lennyb> #endmeeting