Friday, 2017-06-02

*** AndyWojo has joined #openstack-powervm00:48
*** svenkat has quit IRC01:07
*** thorst has joined #openstack-powervm01:11
*** arunmani_ has joined #openstack-powervm01:58
*** thorst has quit IRC02:02
*** thorst has joined #openstack-powervm02:29
*** thorst has quit IRC02:30
*** thorst has joined #openstack-powervm02:41
*** thorst has quit IRC02:41
*** thorst has joined #openstack-powervm02:51
*** thorst has quit IRC02:51
*** edmondsw has joined #openstack-powervm03:00
*** arunmani_ has quit IRC03:01
*** edmondsw has quit IRC03:04
*** thorst has joined #openstack-powervm03:22
*** thorst has quit IRC03:40
*** arunmani_ has joined #openstack-powervm03:50
*** chhavi has joined #openstack-powervm04:32
*** thorst has joined #openstack-powervm04:37
*** thorst has quit IRC04:42
*** arunman has joined #openstack-powervm05:32
*** thorst has joined #openstack-powervm05:38
*** thorst has quit IRC05:43
*** tjakobs has joined #openstack-powervm05:56
*** tjakobs has quit IRC06:02
*** edmondsw has joined #openstack-powervm06:36
*** edmondsw has quit IRC06:41
*** chas has joined #openstack-powervm07:26
*** thorst has joined #openstack-powervm07:39
*** thorst has quit IRC07:44
*** thorst has joined #openstack-powervm08:40
*** thorst has quit IRC08:59
*** chas_ has joined #openstack-powervm09:15
*** k0da has joined #openstack-powervm09:17
*** chas has quit IRC09:19
*** thorst has joined #openstack-powervm09:56
*** thorst has quit IRC10:01
*** thorst has joined #openstack-powervm10:57
*** thorst has quit IRC11:02
*** edmondsw has joined #openstack-powervm11:19
*** thorst has joined #openstack-powervm11:41
*** thorst has quit IRC11:45
*** thorst has joined #openstack-powervm12:01
*** arunman has quit IRC12:05
*** svenkat has joined #openstack-powervm12:07
*** chas_ has quit IRC12:25
*** chas has joined #openstack-powervm12:25
*** chas has quit IRC12:31
*** jwcroppe has quit IRC12:36
openstackgerritMerged openstack/networking-powervm master: Fix docstrings for neutron-lib constants
*** kylek3h has joined #openstack-powervm12:52
*** jpasqualetto has joined #openstack-powervm12:53
*** jwcroppe has joined #openstack-powervm12:54
*** mdrabe has joined #openstack-powervm13:10
*** chas has joined #openstack-powervm13:12
*** chas has quit IRC13:16
*** chas has joined #openstack-powervm13:18
*** chas has quit IRC13:18
*** chas has joined #openstack-powervm13:18
efriedthorst What actual good do LifecycleEvent notifications do us?13:21
efriedFrom what I can tell, if the state of an instance is changed out of band, nova does things that are at best pointless, but often invasive and just wrong.13:23
esbergluefried: thorst: edmondsw: Okay CI is looking much better since the new novalink packages and the updated drmgr13:23
efriedesberglu Cool deal.13:24
esbergluBack to "normal" failure rates. Next we just need to figure out those timeout tests13:24
efriedthorst If I shut down the LPAR out of band, nova runs power-off (which turns out to be a no-op, but generates traffic).  But if I boot a shut-down LPAR out of band, nova shuts it back down - which seems kind of awful.13:28
*** tjakobs has joined #openstack-powervm13:29
thorstefried: nope13:33
thorstthat's the contract with nova13:33
thorstwe stick with it even if we dislike it13:33
thorstif we dislike it, we try to change it in core nova13:33
thorstbut the reason for that is really how nova handles libvirt13:33
efriedthorst Well, there's a conf option to disable the events entirely.13:33
thorstthey destroy and rebuild a VM whenever you power it on.13:33
efriedAnd... we could also just not send them.13:33
thorstI'd rather send them and let the user decide to turn them off.13:34
efriedthorst In the case of pvc, does the user control their nova.conf, or does pvc take responsibility for that?13:38
thorstpvc is turnkey...user doesn't go into config files13:38
efriedthorst Okay, so pvc could decide to set that conf option.  Problem solved.13:38
thorstany deployer could, yep13:39
thorstbut that doesn't mean we shouldn't have that in our driver...13:39
edmondswesberglu do we set OS_TEST_TIMEOUT and if so where and to what?13:57
openstackgerritOpenStack Proposal Bot proposed openstack/networking-powervm master: Updated from global requirements
*** arunman has joined #openstack-powervm14:03
esbergluedmondsw: Not that I know of. I think that would just be something we throw into the local.conf files14:08
esbergluActually maybe not. Why do you ask?14:09
edmondswesberglu that controls how long tempest will wait for a test to complete before it raises a timeout14:10
edmondswesberglu maybe we should try bumping that14:10
esbergluedmondsw: Yeah sure14:10
edmondswlong term I would like to figure out why things are taking so long, of course, and see what we can do to address that14:10
edmondswwhat storage driver are we using for something like test_admin_delete_servers_of_others OOT?14:11
openstackgerritEric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for  Lifecycle events
edmondswesberglu we don't control the tests, so we can't add things like that, but we do control what storage driver we use, right?14:14
edmondswefried your thoughts on this?14:16
*** smatzek has joined #openstack-powervm14:16
*** smatzek_ has joined #openstack-powervm14:17
*** smatzek__ has joined #openstack-powervm14:18
efriededmondsw That would be a stopgap, of course.  My fear is that it'll sweep the problem under the rug and we'll never go oil that wheel cause we've muffled the squeaking.14:18
edmondswefried yeah, totally agree14:19
efriedIt will also cause me to mix metaphors horribly.14:19
edmondswefried what about the storage driver question?14:19
efriededmondsw I'm not sure what the question was.14:21
*** smatzek has quit IRC14:21
edmondswwhat kind of storage are we using OOT for things like test_admin_delete_servers_of_others?14:21
edmondswis there a faster storage option that we could configure to use instead?14:22
*** smatzek_ has quit IRC14:22
edmondswefried one of the reasons I was thinking of trying OS_TEST_TIMEOUT bump was to see if it would help at all... i.e., determine whether the tests would still fail or if they're just being too slow for the current timeout14:23
*** kriskend has joined #openstack-powervm14:23
edmondswor do we already have a sense of that, whether it's a speed vs. hung/blocked issue?14:24
efriededmondsw A fair point.  As for which storage driver, pretty sure we only have SSP capability in the CI right now.  But that's the fastest one anyway, for pretty much any operation you'd care to name.14:26
edmondswI thought that might be the case, but had to ask :)14:26
esbergluefried: edmondsw: Just catching back up. Yeah we just have SSP. As far as speed vs. blocked I don't think we've determined anything14:27
esbergluUnless efried unconvered something I'm not aware of14:27
efriedI have not uncovered anything.14:28
edmondswesberglu so let's try bumping OS_TEST_TIMEOUT and see if that gets rid of, or even impacts, the number of timeouts we're seeing14:28
edmondswjust as a temporary measure14:29
edmondswefried agreed?14:29
esbergluedmondsw: Sure14:29
efriedesberglu Can you do that in a sandboxy way, versus committing a change in the repo?14:29
efriedJust want to make sure we don't forget about it.14:30
esbergluefried: Yeah I should be able to14:36
*** smatzek__ has quit IRC14:37
esbergluThat line means I can just set OS_TEST_TIMEOUT in /etc/environment and it will then get passed in?14:46
efriedesberglu Perhaps?14:47
efriedPretty sure the testr command starts off by outputting the env vars it's using, so you should be able to tell pretty quick.14:47
esbergluefried: Yeah it does. I doubled the timeout from 500s to 1000s for now14:47
edmondswesberglu the default was 500 not 1200?14:49
esbergluedmondsw: Yeah I saw that as well. But if you look at the test it fails in 8 min 20 sec (500s)14:51
edmondswesberglu maybe from this:
edmondswesberglu or this:
esbergluedmondsw: Yeah I was trying to track down where it was getting set but gave up when I realized it was 500. Overriding in /etc/environment should work14:52
thorstedmondsw: I still owe you a box right?14:53
edmondswthorst yep14:53
efriedAnd then I owe you a setup tutorial14:53
thorstk.  Will PM you the details.  It'll need a fresh install.14:53
efriedthorst Send him the install wiki page too.  I can never find the damn thing.14:55
thorstefried: yes sir mr. efried sir14:55
efriedthorst (edmondsw) please take a look at and let me know if this is reasonable so I can proceed with UT fixup.15:02
thorst-1 doesn't pass pep815:02
efriedthorst Yeah, I know, I have made no attempt to do anything other than functional testing (which looks good).15:02
thorstefried: I can't look at anything unless its all green15:03
efriedthorst Just looking for buy-in on the high points of the approach.15:03
thorst(I'm kidding)15:03
edmondswpulling it up15:03
efriedthorst edmondsw In live testing, the result of the change is that we don't emit any lifecycle events at all during spawn/destroy (which is good); but we still see a couple of get_instance()s per VM (which I think is better than before, but it would be nice to figure out a way to reduce that).15:08
efriedThe next step to improve things would be to maintain a cache of LPAR states, which would allow us to skip out before get_instance if it hasn't changed.15:09
efriedThat's getting real close to the Pareto point, though...15:09
efriedCause I would need to figure out a way to expire entries from the cache.15:11
efriedCouldn't find a simple "expiring cache" impl in a standard python lib.15:12
efriedCould expire entries by handling DELETED events from powervm; and/or by polling the whole LPAR list periodically.15:13
efriedAgain, not sure whether the gain is worth all that extra logic.15:13
thorstI have beef with your cache15:18
thorstnoted in the review.15:19
efriedthorst MY cache?  MY cache??  It's YOUR cache!15:19
thorstbeef is had15:21
*** k0da has quit IRC15:43
efriedthorst Responded to your comments.  Verbosely.  Pending further discussion, I stand by things like the cache and the task_state stuff.  Let me know what you think.16:00
*** jwcroppe has quit IRC16:01
*** jwcroppe has joined #openstack-powervm16:02
thorstefried: I think the part I was missing was that this is a short lived cache.16:03
edmondswefried didn't get through it all yet, but gave you some comments to start looking at16:03
thorstI assumed it was built when the driver started.16:03
efriedthorst Yeah, that would be... bad.16:03
efriedesberglu Is this where you were seeing the permission problem?
*** chas has quit IRC16:09
*** chas has joined #openstack-powervm16:10
efriededmondsw Thanks, responded.16:14
*** chas has quit IRC16:14
*** mdrabe has quit IRC16:30
*** mdrabe has joined #openstack-powervm16:44
*** jwcroppe has quit IRC16:58
*** jwcroppe has joined #openstack-powervm17:02
*** chhavi has quit IRC17:04
esbergluefried: Yeah it is17:19
*** esberglu has quit IRC17:24
*** esberglu has joined #openstack-powervm17:24
*** esberglu has quit IRC17:29
*** esberglu has joined #openstack-powervm17:37
*** dwayne has quit IRC17:43
*** jwcroppe has quit IRC17:58
*** arunman has quit IRC18:01
*** dwayne has joined #openstack-powervm18:04
*** jwcroppe has joined #openstack-powervm18:11
*** chas has joined #openstack-powervm18:11
*** chas has quit IRC18:15
esbergluefried: edmondsw: thorst: Still hitting the timeouts with the value doubled18:32
efriedCool, so most likely points to a real bug.18:33
esbergluWhat's the longest reasonable time that those tests should take? The timeout is now between 16 and 17 minutes18:34
edmondswesberglu what efried said. I would think 16-17 min is plenty18:38
openstackgerritEric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events
openstackgerritEric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events
*** k0da has joined #openstack-powervm18:54
openstackgerritEric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events
*** kylek3h has quit IRC19:11
*** jwcroppe has quit IRC19:12
*** chas has joined #openstack-powervm20:12
*** chas has quit IRC20:16
openstackgerritEric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events
*** dwayne has quit IRC20:44
*** kriskend has quit IRC20:52
*** jpasqualetto has quit IRC21:10
*** thorst has quit IRC21:11
*** svenkat has quit IRC21:11
*** thorst has joined #openstack-powervm21:15
*** thorst has quit IRC21:19
*** k0da has quit IRC21:30
*** thorst has joined #openstack-powervm21:39
*** mdrabe has quit IRC21:47
*** thorst has quit IRC21:58
*** edmondsw has quit IRC22:03
*** svenkat has joined #openstack-powervm22:04
*** tjakobs has quit IRC22:11
*** chas has joined #openstack-powervm22:12
*** chas has quit IRC22:17
*** toan has joined #openstack-powervm22:33
*** toanster has quit IRC22:40
*** jwcroppe has joined #openstack-powervm22:47
*** thorst has joined #openstack-powervm22:49
*** thorst has quit IRC22:49
*** thorst has joined #openstack-powervm22:55
*** esberglu has quit IRC22:59
*** esberglu has joined #openstack-powervm23:00
*** esberglu has quit IRC23:04
*** edmondsw has joined #openstack-powervm23:11
*** thorst has quit IRC23:14
*** edmondsw has quit IRC23:16

Generated by 2.15.3 by Marius Gedminas - find it at!