15:01:55 #startmeeting PowerVM Driver Meeting 15:01:56 Meeting started Tue Nov 6 15:01:55 2018 UTC and is due to finish in 60 minutes. The chair is edmondsw. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:58 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:02:00 The meeting name has been set to 'powervm_driver_meeting' 15:02:06 #link agenda https://etherpad.openstack.org/p/powervm_driver_meeting_agenda 15:02:44 efried mdrabe ping 15:05:26 may be a short meeting if nobody's around :) 15:06:04 o/ sorry 15:06:17 np 15:06:24 #topic In-Tree Driver 15:06:31 we merged a couple things this week: 15:06:38 1) https://review.openstack.org/#/c/610174/ 15:06:45 2) https://review.openstack.org/#/c/468560/ 15:07:16 I don't recall anything else to discuss IT... anyone else? 15:08:14 alright, moving on 15:08:22 #topic Out-of-Tree Driver 15:08:43 we merged the corresponding tempfile change OOT similar to IT: https://review.openstack.org/#/c/613342/ 15:09:55 and https://review.openstack.org/#/c/614781/ 15:10:28 oh, and there was a corresponding IT change for that, which I should have mentioned above : https://review.openstack.org/#/c/614643/ 15:11:41 the only other thing I have for the OOT driver is that there's been a customer issue with NovaLink GPFS volumes since we switched to the LOOP backstore 15:12:04 from what I hear, seems to be a kernel issue, but the NovaLink guys are working on a workaround 15:12:49 if they don't get that resolved, we may have to revert the change that we put in a while back to switch to loop: b86a9480c3c39f357b862c6a57ca6a2eb092558c 15:13:22 anything else for OOT discussion? 15:14:19 so quiet today 15:14:21 #topic Device Passthrough 15:14:23 efried ^ 15:14:46 Nothing to report. I still owe updates to the specs and code. 15:15:06 ack 15:15:14 #topic PowerVM CI 15:15:53 we found the CI not voting again late last week and mmedvede helped fix it 15:16:14 I believe it was a nodepool issue, same as last time. Not sure what's causing that 15:16:34 mujahid doesn't seem to be around to give us his thoughts 15:17:32 http://ci-watch.tintri.com/ doesn't seem to be running anymore, so mmedvede setup http://ciwatch.mmedvede.net 15:18:12 we appear to be voting there, though I'm seeing more red than I'd like 15:18:38 I'll ask mujahid to take a look at that 15:18:47 #topic Open Discussion 15:19:06 efried and I will be at the summit in Berlin next week, so no meeting 15:19:40 floor is open for any other topics 15:20:12 FYI I've been working on perf improvements in the resource tracker. It's nothing powervm specific, but it's what's consuming my time the past couple of work days. 15:20:25 ack 15:21:11 with regard to nodepool problem I helped to fix - it may happen again I am afraid. nodepool is stuck with too many open files error. I did not change ulimits 15:21:28 mmedvede any idea what causes that? 15:21:46 I'm with you, expecting this to keep happening unless we figure out what's causing it 15:22:27 or put in some mitigation or something 15:22:27 edmondsw: no, other than it is somehow not closing files 15:22:36 edmondsw: We fixed a file descriptor leak recently... 15:22:47 oh, it was in tempfile usage. And it never made it to prod. 15:22:54 simple mitigation would be periodic restart of nodepool every day 15:23:05 mmedvede would we lose things with the restart? 15:23:47 nodepool keeps its state in database. Still slight possibility something would be lost, but it is usually safe to restart 15:24:00 good to know 15:24:22 maybe we can setup a cron job for the middle of the night or something 15:24:31 I'll suggest that to mujahid 15:24:54 I remembered one other thing I should mention... and mmedvede you'll also be interested in this 15:25:01 a while back we hit an issue with greenlet 0.4.14 where they'd made a change that didn't work on Ubuntu 16.04 15:25:29 they've since fixed that in greenlet with a different change in 0.4.15, which I believe I already noted previously 15:25:48 but I was also working with Ubuntu to fix the gcc bug that was the real issue 15:26:18 and I have now successfully tested a new gcc5 package on xenial-proposed with greenlet 0.4.14 15:26:59 so once they promote that from proposed (which they've been waiting to do until I'd confirmed the fix), even 0.4.14 should be ok 15:27:15 eom 15:27:38 this is good news, thank you 15:28:41 yep... shouldn't have to worry about other things besides greenlet breaking on Ubuntu 16.04 due to that gcc bug 15:29:00 alright, I think that's it for today 15:29:04 #endmeeting