15:01:55 <edmondsw> #startmeeting PowerVM Driver Meeting
15:01:56 <openstack> Meeting started Tue Nov  6 15:01:55 2018 UTC and is due to finish in 60 minutes.  The chair is edmondsw. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:00 <openstack> The meeting name has been set to 'powervm_driver_meeting'
15:02:06 <edmondsw> #link agenda https://etherpad.openstack.org/p/powervm_driver_meeting_agenda
15:02:44 <edmondsw> efried mdrabe ping
15:05:26 <edmondsw> may be a short meeting if nobody's around :)
15:06:04 <efried> o/ sorry
15:06:17 <edmondsw> np
15:06:24 <edmondsw> #topic In-Tree Driver
15:06:31 <edmondsw> we merged a couple things this week:
15:06:38 <edmondsw> 1) https://review.openstack.org/#/c/610174/
15:06:45 <edmondsw> 2) https://review.openstack.org/#/c/468560/
15:07:16 <edmondsw> I don't recall anything else to discuss IT... anyone else?
15:08:14 <edmondsw> alright, moving on
15:08:22 <edmondsw> #topic Out-of-Tree Driver
15:08:43 <edmondsw> we merged the corresponding tempfile change OOT similar to IT: https://review.openstack.org/#/c/613342/
15:09:55 <edmondsw> and https://review.openstack.org/#/c/614781/
15:10:28 <edmondsw> oh, and there was a corresponding IT change for that, which I should have mentioned above : https://review.openstack.org/#/c/614643/
15:11:41 <edmondsw> the only other thing I have for the OOT driver is that there's been a customer issue with NovaLink GPFS volumes since we switched to the LOOP backstore
15:12:04 <edmondsw> from what I hear, seems to be a kernel issue, but the NovaLink guys are working on a workaround
15:12:49 <edmondsw> if they don't get that resolved, we may have to revert the change that we put in a while back to switch to loop: b86a9480c3c39f357b862c6a57ca6a2eb092558c
15:13:22 <edmondsw> anything else for OOT discussion?
15:14:19 <edmondsw> so quiet today
15:14:21 <edmondsw> #topic Device Passthrough
15:14:23 <edmondsw> efried ^
15:14:46 <efried> Nothing to report. I still owe updates to the specs and code.
15:15:06 <edmondsw> ack
15:15:14 <edmondsw> #topic PowerVM CI
15:15:53 <edmondsw> we found the CI not voting again late last week and mmedvede helped fix it
15:16:14 <edmondsw> I believe it was a nodepool issue, same as last time. Not sure what's causing that
15:16:34 <edmondsw> mujahid doesn't seem to be around to give us his thoughts
15:17:32 <edmondsw> http://ci-watch.tintri.com/ doesn't seem to be running anymore, so mmedvede setup http://ciwatch.mmedvede.net
15:18:12 <edmondsw> we appear to be voting there, though I'm seeing more red than I'd like
15:18:38 <edmondsw> I'll ask mujahid to take a look at that
15:18:47 <edmondsw> #topic Open Discussion
15:19:06 <edmondsw> efried and I will be at the summit in Berlin next week, so no meeting
15:19:40 <edmondsw> floor is open for any other topics
15:20:12 <efried> FYI I've been working on perf improvements in the resource tracker. It's nothing powervm specific, but it's what's consuming my time the past couple of work days.
15:20:25 <edmondsw> ack
15:21:11 <mmedvede> with regard to nodepool problem I helped to fix - it may happen again I am afraid. nodepool is stuck with too many open files error. I did not change ulimits
15:21:28 <edmondsw> mmedvede any idea what causes that?
15:21:46 <edmondsw> I'm with you, expecting this to keep happening unless we figure out what's causing it
15:22:27 <edmondsw> or put in some mitigation or something
15:22:27 <mmedvede> edmondsw: no, other than it is somehow not closing files
15:22:36 <efried> edmondsw: We fixed a file descriptor leak recently...
15:22:47 <efried> oh, it was in tempfile usage. And it never made it to prod.
15:22:54 <mmedvede> simple mitigation would be periodic restart of nodepool every day
15:23:05 <edmondsw> mmedvede would we lose things with the restart?
15:23:47 <mmedvede> nodepool keeps its state in database. Still slight possibility something would be lost, but it is usually safe to restart
15:24:00 <edmondsw> good to know
15:24:22 <edmondsw> maybe we can setup a cron job for the middle of the night or something
15:24:31 <edmondsw> I'll suggest that to mujahid
15:24:54 <edmondsw> I remembered one other thing I should mention... and mmedvede you'll also be interested in this
15:25:01 <edmondsw> a while back we hit an issue with greenlet 0.4.14 where they'd made a change that didn't work on Ubuntu 16.04
15:25:29 <edmondsw> they've since fixed that in greenlet with a different change in 0.4.15, which I believe I already noted previously
15:25:48 <edmondsw> but I was also working with Ubuntu to fix the gcc bug that was the real issue
15:26:18 <edmondsw> and I have now successfully tested a new gcc5 package on xenial-proposed with greenlet 0.4.14
15:26:59 <edmondsw> so once they promote that from proposed (which they've been waiting to do until I'd confirmed the fix), even 0.4.14 should be ok
15:27:15 <edmondsw> eom
15:27:38 <mmedvede> this is good news, thank you
15:28:41 <edmondsw> yep... shouldn't have to worry about other things besides greenlet breaking on Ubuntu 16.04 due to that gcc bug
15:29:00 <edmondsw> alright, I think that's it for today
15:29:04 <edmondsw> #endmeeting