Tuesday, 2018-02-20

*** AlexeyAbashkin has joined #openstack-powervm00:24
*** AlexeyAbashkin has quit IRC00:28
*** svenkat has joined #openstack-powervm00:49
*** svenkat has quit IRC01:12
*** svenkat has joined #openstack-powervm01:27
*** esberglu has quit IRC02:04
*** esberglu has joined #openstack-powervm02:04
*** esberglu has quit IRC02:09
*** esberglu has joined #openstack-powervm02:31
*** esberglu has quit IRC02:39
*** esberglu has joined #openstack-powervm02:40
*** esberglu has quit IRC02:44
*** AlexeyAbashkin has joined #openstack-powervm03:24
*** AlexeyAbashkin has quit IRC03:29
*** apearson has joined #openstack-powervm03:49
*** esberglu has joined #openstack-powervm03:51
*** esberglu has quit IRC03:56
*** svenkat has quit IRC04:12
*** esberglu has joined #openstack-powervm04:48
*** esberglu has quit IRC04:52
*** k0da has joined #openstack-powervm05:04
*** apearson has quit IRC05:35
*** mdrabe has quit IRC06:11
*** mdrabe has joined #openstack-powervm06:13
*** AlexeyAbashkin has joined #openstack-powervm07:53
*** k0da has quit IRC09:11
*** k0da has joined #openstack-powervm09:24
*** k0da has quit IRC10:51
*** svenkat has joined #openstack-powervm13:25
*** apearson has joined #openstack-powervm13:34
*** esberglu has joined #openstack-powervm13:48
esberglu#startmeeting powervm_driver_meeting14:01
openstackMeeting started Tue Feb 20 14:01:25 2018 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.14:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
*** openstack changes topic to " (Meeting topic: powervm_driver_meeting)"14:01
openstackThe meeting name has been set to 'powervm_driver_meeting'14:01
esbergluedmondsw: efried: You guys around? Sorry didn't realize the recurring meeting notices were done14:03
efriedoh, yeah, I forgot because no calendar entry.14:03
efriedBut I'm here.14:03
esberglu#topic In-tree Driver14:04
*** openstack changes topic to "In-tree Driver (Meeting topic: powervm_driver_meeting)"14:04
esbergluThe bp looked good to me, I think edmondsw is gonna add the interface hotplug stuff still?14:04
esbergluOnly other open IT item for me is snapshot14:06
esbergluHoping we can get vSCSI, snapshot and localdisk in pretty early in the cycle14:06
esbergluBecause the migration stuff is gonna require a lot more work, both coding and testing14:07
esbergluAnd getting CI up to speed14:07
esberglu#topic Out-of-tree Driver14:09
*** openstack changes topic to "Out-of-tree Driver (Meeting topic: powervm_driver_meeting)"14:09
efriededmondsw and I learned things about MagicMock and one of our test cases yesterday.14:09
efriedabout the former: it doesn't blow up when you try to iterate on it (Mock does).  And about the latter: we weren't testing what we thought we were testing.  Which happens a lot.14:10
esbergluefried: You mean we have a bunch of tests not doing what we thought?14:12
efriedProbably.  But at least this one for sure.14:12
efriedInterested in the recap?14:13
esbergluefried: Yeah14:13
efriedThis was for driver capabilities.  We've been setting has_imagecache dynamically, because whether the driver supports it depends on which disk driver we're using.14:14
efriedLocaldisk supports it, the others don't.14:14
efriedSo we had a test that said assertFalse(driver.capabilities['has_imagecache'])14:14
efriedwhich seems reasonable enough14:15
efriedThe dynamic setting used to be code in the driver that did something like:14:15
efriedi.e. "take the stuff in the disk_drv.capabilities dict and overwrite/add whatever's in self.capabilities"14:16
efriededmondsw wasn't happy with the "add" part, because it meant we were pulling capability keys into our compute driver that aren't part of the ComputeDriver interface.14:16
efriedSo to get rid of that, he changed it to be explicit about just pulling in has_imagecache:14:17
efriedself.capabilities['has_imagecache'] = disk_drv.capabilities['has_imagecache']14:17
efriedWhereupon the above assertion started to fail.14:17
efriedAnd when he changed it to assertTrue, it passed.14:17
efriedWhich should not happen.14:17
efriedIf we were testing this properly, both of those modes of assignment should have been equivalent in terms of the has_imagecache key.  So it should have been the same before & after.  Which in fact should have been a signal that the code change was valid.14:19
esbergluBut the code change is valid, so we realized it was a problem with our test14:20
efriedWell, in broad terms, yeah.  But we couldn't suss what the problem could possibly be.14:20
efriedThe first thing I suspected was that one or both of those .capabilities was a Mock.  But that didn't make sense, because you get an exception if you try to [key access] or .update a Mock.14:21
efriedSo either code path should have raised an exception.14:21
efriedBut it turns out that the disk_drv.capabilities dict was a *Magic*Mock14:22
efriedAnd MagicMock, in contrast with Mock, has default impls for magic methods.  Like __getitem__ (returns a mock) and __iter__ (yields as if an empty list).14:23
efriedSo when we were doing .update(), we were updating the driver's capabilities - where has_imagecache is defaulted to False - from an empty list of disk driver capabilities.  Which left it False.14:24
efriedBut when we did the assignment, we were assigning a mock to the value.  And a mock evaluates to True.14:24
esbergluAh okay I see14:25
esbergluI'm not seeing where in the code the capabilities are a MagicMock vs Mock14:26
efriedYeah, that was tricky to track down.  The driver is a real driver, which we were doing on purpose so we could get "real" values in the capabilities dict14:28
efriedthe setUp for this test class is installing a fixture that actually mocks the disk driver constructor14:29
efriedSo we're getting a real compute driver, but when it goes to create the disk driver, that mock patch takes over and we get a MagicMock.14:29
esbergluOkay I follow14:30
efriedWhat's next?14:32
esbergluefried: How do we fix it?14:32
efriedDon't let the disk driver get mocked.14:32
efriedI suggested moving the test case up to the first class in the suite, which doesn't do any of the fixture stuff.14:33
efriedAnd also adding test cases where we use different disk drivers to make sure it's True only for LocalStorage.14:33
efriedAnd also adding assertions to ensure that the extraneous keys aren't being created.14:34
esbergluSounds good14:34
esberglu#topic Device Passthrough14:35
*** openstack changes topic to "Device Passthrough (Meeting topic: powervm_driver_meeting)"14:35
efriedIOW, what I'm sure edmondsw thought was going to be an ace across the net is rapidly metastasizing14:35
esbergluYep sounds that way14:35
efriedNo news from me on dev passthrough.  We're frantically writing & reviewing specs ahead of the PTG.14:35
efriedI abandoned that resource class affinity spec.14:36
efriedAnd put up this new one (unrelated) last night: https://review.openstack.org/#/c/546009/14:36
esberglu#topic PowerVM CI14:38
*** openstack changes topic to "PowerVM CI (Meeting topic: powervm_driver_meeting)"14:38
efriedI believe I'm still waiting on prompts from edmondsw et al for next steps on dev passthrough.14:38
efriedsorry, I'm a step behind you all day.14:38
esbergluSo the big thing for CI for queens is getting multinode jobs so we can do migration testing14:39
esbergluI haven't looked into that at all really in the past, so I need to do some research there14:39
efriedYeah.  Big fun times.14:39
efriedI can tell you this: I will be of absolutely no help.14:39
efriedI have managed to go my entire career without ever migrating a partition14:40
esbergluHuh, would have bet against that :)14:40
esbergluThe other thing for CI I wanted to bring up is that stale vopt issue we were seeing14:41
esbergluSince fixing up the post stack cleaner, that has gone WAY down. However it did hit a handful of times in the last week14:41
esbergluSo we either need to re-explore that vopt naming change or try to figure out where the vopts are coming from now14:42
efriedDid you manually clean up (or check) any stale vopts that were lying around from before we fixed the cleaner?14:42
esbergluefried: Yes14:43
efriedSo we're somehow still ending up with stale ones.14:43
*** k0da has joined #openstack-powervm14:44
esbergluYep, looks that way14:44
efriedI wonder if it could be timing issues with multiple tests (possibly even from multiple patches) running at once.14:44
efriedLike, the offending vopt isn't actually stale; it's just in the process of being created/mapped or unmapped/deleted when the other test hits this code path.14:45
efried...and it's still the naming issue.14:45
esbergluefried: If that were the case I don't think the post stack cleaner update would have cut down the occurrences so much14:45
efriedNo, I'm saying both issues that we talked about were problems.  The post-stack cleaner took care of most of them, but not all.14:46
efriedUnless... is it possible the cleaner fix didn't get deployed to whatever server hit the problem?14:46
esbergluefried: Nope, the latest of that gets pulled every run14:46
efriedHow often is this happening?  Did we just see this once in a little blip and never again?14:47
esberglu7 times since I left on wednesday14:49
esbergluWhere it was a handful a day before the cleaner fix14:49
esbergluEither way, needs further debugging. I will look into it and let you know if I get stuck14:51
efriedSpread out across multiple nodes?14:51
esbergluefried: Yep14:51
efriedI was thinking we could reinstate the naming fix and see if it clears up entirely.14:51
esbergluefried: And if it doesn't, that points to a possible timing issue14:52
esbergluI'm on board with that14:52
efriedight.  will look for that review.  I need to un-abandon the renaming patch, eh?14:53
* efried looks...14:53
efriedDone: https://review.openstack.org/#/c/540453/14:53
esbergluefried: Cool I'll put up the change to patch it in right after this14:54
esbergluefried: That's all for me today, anything else?14:55
efrieddon't think so.14:55
*** openstack changes topic to "This channel is for PowerVM-related development and discussion. For general OpenStack support, please use #openstack."14:55
openstackMeeting ended Tue Feb 20 14:55:49 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:55
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2018/powervm_driver_meeting.2018-02-20-14.01.html14:55
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2018/powervm_driver_meeting.2018-02-20-14.01.txt14:55
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2018/powervm_driver_meeting.2018-02-20-14.01.log.html14:55
efriedThanks esberglu14:56
esbergluI'll add the meeting invites again. It would be nice if you got notified when your recurring meetings end14:56
efriedMm.  Patent.14:57
esbergluefried: 634915:18
efriedesberglu: +215:19
*** tjakobs has joined #openstack-powervm15:23
*** apearson has quit IRC16:01
*** apearson has joined #openstack-powervm16:06
*** AlexeyAbashkin has quit IRC16:26
*** k0da has quit IRC17:32
*** AlexeyAbashkin has joined #openstack-powervm17:57
*** AlexeyAbashkin has quit IRC18:02
*** openstackgerrit has quit IRC18:03
*** AlexeyAbashkin has joined #openstack-powervm18:13
*** efried has quit IRC18:15
*** efried has joined #openstack-powervm18:15
*** AlexeyAbashkin has quit IRC18:17
*** k0da has joined #openstack-powervm19:10
*** k0da has quit IRC19:40
*** AlexeyAbashkin has joined #openstack-powervm20:23
*** AlexeyAbashkin has quit IRC20:27
*** k0da has joined #openstack-powervm20:59
*** k0da has quit IRC21:04
*** k0da has joined #openstack-powervm21:16
*** AlexeyAbashkin has joined #openstack-powervm21:23
*** AlexeyAbashkin has quit IRC21:27
*** svenkat has quit IRC21:28
*** apearson has quit IRC22:06
*** apearson has joined #openstack-powervm22:06
*** apearson has quit IRC22:06
*** tjakobs has quit IRC22:38
*** tjakobs has joined #openstack-powervm22:41
edmondswefried we definitely found a rabbit hole with this test issue.22:44
edmondswI spent some more time on it last night and early this morning, and the driver fixture is causing issues even in that Init class that doesn't have any fixtures.22:44
edmondswI wrote new tests that pass when run individually, but fail (because of the fixture) if I run the whole test suite22:45
edmondswso something isn't cleaning up properly? I don't know much about how fixtures work, so I'll have to dig into that22:45
edmondswafter I get back on Thursday22:45
edmondswI'll go ahead and throw up what I've done in case you want to see it, but I know it's not working22:46
*** openstackgerrit has joined #openstack-powervm22:49
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: stop setting extra capabilities  https://review.openstack.org/54598422:49
efriedsuper weird.  That is definitely not supposed to happen.23:02
*** openstackgerrit has quit IRC23:04
*** tjakobs has quit IRC23:07
*** AlexeyAbashkin has joined #openstack-powervm23:23
*** svenkat has joined #openstack-powervm23:25
*** AlexeyAbashkin has quit IRC23:27
*** svenkat has quit IRC23:44

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!