Tuesday, 2017-01-03

*** adreznec has joined #openstack-powervm00:01
*** apearson has quit IRC00:04
*** k0da has quit IRC00:08
*** chas__ has quit IRC00:08
*** chas has joined #openstack-powervm00:08
*** chas has quit IRC00:13
*** chas has joined #openstack-powervm03:10
*** chas has quit IRC03:15
*** adreznec has quit IRC03:34
*** adreznec has joined #openstack-powervm03:34
*** chas has joined #openstack-powervm05:11
*** chas has quit IRC05:16
*** kotra03 has joined #openstack-powervm05:22
*** chas has joined #openstack-powervm07:12
*** chas has quit IRC07:17
*** chas has joined #openstack-powervm07:25
*** k0da has joined #openstack-powervm08:21
*** k0da has quit IRC09:18
*** jpasqualetto has joined #openstack-powervm10:44
*** jpasqualetto has quit IRC11:34
*** smatzek has joined #openstack-powervm11:41
*** jpasqualetto has joined #openstack-powervm11:46
*** efried has quit IRC12:19
*** efried has joined #openstack-powervm12:20
*** tblakes has joined #openstack-powervm13:00
*** kotra03 has quit IRC13:06
*** svenkat has joined #openstack-powervm13:28
*** thorst has joined #openstack-powervm13:30
efriedthorst, adreznec, esberglu: agenda item for today's meeting: We need to figure out how to have separate skip lists for CI run against in-tree and out-of-tree.13:38
*** mdrabe has joined #openstack-powervm13:44
thorstefried: https://wiki.openstack.org/wiki/Meetings/Nova13:50
thorstper our earlier discussion13:50
thorstand agree on topic13:50
thorstefried: you also need to chat with qing wu about how we want the driver work to continue.  Now that you're back I think the two of you work as a team on it.13:51
*** jwcroppe has quit IRC13:53
*** jwcroppe has joined #openstack-powervm13:53
*** wangqwsh has joined #openstack-powervm13:54
*** apearson has joined #openstack-powervm13:55
*** jwcroppe_ has joined #openstack-powervm13:57
*** tblakes has quit IRC13:58
*** jwcroppe has quit IRC13:58
efriedthorst, ack.14:02
efriedAre adreznec and/or esberglu working today?  To run the meeting?14:02
thorstI don't see either online14:04
thorstwe could have an informal meeting14:04
efriedWe need esberglu for my topic.14:05
thorstbut we have wangqwsh here, so we could have a nova driver discussion14:05
adreznecSorry, running late today14:06
adreznecI think Eric is out though14:06
adreznecUntil next week or something14:06
thorstdo we want to talk through the driver stuff?14:08
thorstnamespace change...how the WIP change sets are going...etc...14:08
adreznecMay as well14:08
adreznec#startmeeting powervm_driver_meeting14:08
openstackMeeting started Tue Jan  3 14:08:45 2017 UTC and is due to finish in 60 minutes.  The chair is adreznec. Information about MeetBot at http://wiki.debian.org/MeetBot.14:08
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:08
openstackThe meeting name has been set to 'powervm_driver_meeting'14:08
adreznecSuper formal now14:09
thorstso namespace change?14:09
adreznec#topic In-tree driver status14:09
adreznecSo where did we leave off there? I know the three of us each had discussions before the holidays14:10
efried#link https://review.openstack.org/41373614:10
adreznecAnd efried sent out a note on things14:10
thorstI think we all agreed we need a namespace change14:10
thorstneed being flexible...  It's the least evil14:10
adreznecYeah, no good way around it14:11
thorstefried's got that link...we have nova_powervm, but in the e-mail he suggested powervm_ext14:11
thorstI kinda like powervm_ext better...14:11
adreznecyeah, I know you and I had discussed something similar14:12
adreznecI'm in agreement as well - makes things clear to users that this is the external driver14:12
thorstwhat did the hyper-v guys do though/14:12
adreznecthey just called it compute-hyperv14:12
thorstI know you had looked that up earlier.14:12
adreznecSame as the project name14:12
thorstahh, I think powervm-ext is better?14:12
thorstat least for our needs...14:12
*** tblakes has joined #openstack-powervm14:13
efriedI'm fine with that.  Any other opinions?14:13
efried#action efried to change namespace from nova_powervm to powervm_ext14:13
adreznectheir full oot namespace is nova.virt.compute_hyperv14:13
adreznecI don't think there's any real standard though14:14
adreznecSo lets go with powervm_ext14:14
thorstrip it14:14
efried#action wangqwsh to confirm this fixes whatever issues he was seeing.14:14
efriedBut before we merge, we need to resolve the other stuff I brought up.  Lemme reread...14:15
adreznecYeah, was just bringing up that email14:15
thorstyour bullet-point 3...do we need networking-powervm changes as well to support this14:15
thorstbasically, one solution would be the networking-powervm agent listens to powervm or powervm-ext...14:16
thorstthough, I'm not sure that is really the case...14:16
adreznecI guess that depends14:16
thorstI think both agent types know how to deal with pvm_sea, much like KVM, hyper-v and PowerVM all know how to deal with a type of 'ovs'14:16
adreznecAre we supporting SEA with the in-tree driver today?14:16
thorstno, but eventually we will...14:16
efriedWhat kind of network does the current in-tree driver support?14:17
adreznecSo I guess the question is do we just change it to powervm_ext until we do14:17
efriedWe're not going to pass squat in CI without a network.14:17
adreznecor do we allow both for now14:17
adreznecI think LBr/OvS for simplicity?14:17
*** tlian has joined #openstack-powervm14:17
efriedIs the code for those guys in place in the first change set?14:18
*** kylek3h has joined #openstack-powervm14:18
efriedCause there once again we're inflating that change set, which we're really wanting to avoid.14:18
thorstso networking isn't going to come in yet.  That was going to be WIP614:19
thorstI didn't get to that one yet14:19
thorstin our BP we said that we'd use OVS, but that has CI changes that we need to work through14:19
thorstand yeah...we're not going to get far on Tempest with these initial change sets.  That's chicken/egg...known problem14:19
efriedMeaning we're going to accept failing CI initially?  Is the community going to go for that?  I've been getting the impression they wouldn't.14:20
thorstI think it depends.14:21
efriedOr back to the discussion on paring down CI for in-tree-only14:21
thorsthaving the passing powervm_ext CI will help14:21
thorstno, we have to have two CI's.14:21
thorstone in tree, one out of tree14:21
thorstand its really 'one CI', just running two different jobs.14:21
thorstdifferent tests for the in-tree (while we get support in - though I suspect change set 1 won't pass much of anything)14:22
efriedWell, wanted to punt that to the other #topic, but briefly: we're going to have a volume problem if we literally run two CIs on every change set.14:22
thorstefried: I'm not sure about that...we're sitting OK volume wise atm14:22
thorstlike to the point we could run much more.14:22
thorstwe solved a big bottleneck mid Dec.14:22
efriedOkay, good deal.14:23
efriedSo back to VIRT_DRIVER.14:23
thorstand if we are bad on capacity, we'll figure it out14:23
efriedI still contend we need to understand more about what that thing means before we start mucking with it.14:24
adreznecefried: we'll also theoretically be getting more systems... details are still being worked there14:24
adreznecIn devstack, VIRT_DRIVER is what enables the plugins for a nova driver14:24
adreznecSo for example, if you set VIRT_DRIVER= libvirt, it'll run https://github.com/openstack-dev/devstack/blob/d0df7c88f2c4d8e929c635beca55e6efc69be2f5/lib/nova_plugins/hypervisor-libvirt14:25
thorstadreznec: that brings up a good point...do we need a devstack change for powervm as it goes in tree (maybe that's a discussion for later, once we're further along)14:26
adreznecThere are a couple of other misc places it pops up as well, checks for setting specific variables in things like OVS config if a specific VIRT_DRIVER is specified14:26
adreznecWe need lib/nova_plugins/hypervisor-powervm14:26
efriedThat's what wangqwsh was going to work on, per email thread, yes?14:26
adreznecI thought so14:27
wangqwshyes, hypervisor-powervm is used to fix the VIRT_DRIVER type in devstack14:28
efriedPerhaps the question is whether it's actually appropriate for networking-powervm to be using VIRT_DRIVER to decide which network plugins it loads.14:29
efriedVersus, perhaps, a conf setting.14:30
thorstefried: I contend it doesn't...where are you seeing that it does?14:30
*** jpasqualetto has quit IRC14:30
efried#link https://github.com/openstack/networking-powervm/blob/master/devstack/settings#L614:31
thorstahhh, OK.14:31
thorstnetworking-powervm's *devstack* (which I guess I was missing) assumes that.14:31
adreznecI mean we could definitely find something else to key off14:32
thorstok...I was off in kansas then.14:32
thorstwell, I think that's a good default.14:32
adreznecJust seemed like a fair thing to do at the time as there was no case to ever load the driver without having VIRT_DRIVER=powervm14:32
thorstright...and at the time it was the only one that worked.14:32
thorstnow we have OVS14:32
adreznecI think we have three options at this point14:32
*** edmondsw has joined #openstack-powervm14:32
thorstbut I think if using powervm_ext, I think it is appropriate to default to enabling the sea-agt and sriov-agt14:33
thorstI'd assert it's always OK (if PowerVM anything) to start the sriov-agt14:33
efriedSo the code is smart enough not to blow up if VIRT_DRIVER=foo and there's no hypervisor-foo - else our oot driver would have blown up.14:33
adreznec1) Make it an explicit conf setting. 2) Change it to just powervm_ext and add powervm back later. 3) Support both14:33
efriedI vote for #1.14:34
adreznecAnd yeah efried, devstack will ignore things if hypervisor-foo doesn't exist14:34
efriedOOT isn't using it for anything else at the moment.14:34
*** smatzek has quit IRC14:34
efriedToday we have [ml2] with mechanism_drivers={comma-separated list, e.g. pvm_sea,pvm_sriov}14:37
efriedin ml2_conf.ini14:37
efriedNow, I still don't really get what's a plugin and what's ML2 and what's a driver and what's a ... whatever.14:37
*** edmondsw_ has joined #openstack-powervm14:38
thorstefried: yeah, its weird...14:38
efriedSo if it's not appropriate for us just to key off of the above, we could make a similar conf option that tells us what to load up.14:38
*** edmondsw_ has quit IRC14:38
thorstI think I'm OK with option 1...but we should call that out in nova_powervm's change set that you now need to explicitly set the neutron plugin14:38
adreznecI'd be okay with either, we just need to be explicit on the decision14:38
thorstactually, I think that devstack defaults to ovs now14:38
thorstso we should probably default to that14:38
*** tlian2 has joined #openstack-powervm14:38
thorstbut if they explicitly set sea, we should disable OVS...14:38
efriedIs this a devstack thing, or a runtime thing?14:39
thorst(we've gone down a hell of a rabbit hole)14:39
thorstdevstack thing14:39
efriedOkay, so local.conf, not nova/neutron.conf14:39
thorstnova will figure out what to do based off the vif binding14:39
efriedAh, and blow up if there's no appropriate driver/plugin/widget/thingy.14:40
thorstkey is...only one L2-ish plugin can run at once (with the exception of macvtap and sr-iov, which can run in parallel)14:40
thorstif running sea, can't run ovs.  And vice versa14:40
efriedOkay, so we need #action somebody to propose this to networking-powervm.14:41
*** tlian has quit IRC14:41
adreznecI can throw something together14:41
thorstlol - as I was typing he saved me14:41
efried#action adreznec to propose networking-powervm change to move from VIRT_DRIVER to local.conf option to decide which networking thingies to install.14:42
efriedWhat's next?14:44
efriedWe may be able to get some mileage out of the dual-CI discussion without esberglu here.14:44
efriedAnd load him up with corresponding #actions for when he gets back ;-)14:44
thorstheh, lets just take a moment to congratulate him (without him here) on how well the CI has been running14:45
* efried bows head for introspective moment of silence.14:46
*** jpasqualetto has joined #openstack-powervm14:47
thorstso, what's the discussion CI wise.14:47
thorstsingle CI - run two jobs for each nova patch.  One in tree, one out of tree.  Different config files.  Configs changing rapidly for in tree as function gets added.14:47
adreznecDifferent configs and different test lists14:48
*** jwcroppe_ has quit IRC14:48
thorstwe want to use OVS...which will be a deeper discussion14:48
thorstbut one we've been having (lightly) for the OSA discussion14:48
thorstso this will help our OSA CI as well14:48
efriedAnybody remember where we are wrt converting test names to idempotent IDs?14:49
thorsthe has a patch set up for that14:49
thorstnot sure if merged.14:49
thorstits on morpheus...let me look up14:50
efriedgot it14:50
efriedah, good, he's got the test names in comments next to the IDs.14:50
efriedThat's what I wanted to make sure of.14:50
efriedElse maintainability nightmare.14:50
thorstefried: agree.14:51
efriedI gave that change set +214:51
thorstits been a month  :-)14:52
efriedWe'll let esberglu merge it, cause I want him to make sure it really works (i.e. getting parsed properly)14:52
efriedadded comment to that effect.14:53
efriedWhat else?14:53
efried(is there a way to queue this dual-CI agenda item up for next time when esberglu is here?)14:54
thorstefried: make an action for him to lead that discussion?14:55
efried#action esberglu to drive discussion: two CI runs per change set: in- and out-of-tree.  In-tree much smaller, with plan to grow in lockstep with in-tree driver merges.14:57
efried(where "smaller" really means "larger skip list")14:57
efriedWe done for now?14:58
thorstyeah, lets call it14:58
thorstglad to have you back efried  :-D14:59
efriedGlad to be here.14:59
efriedadreznec, gavel?14:59
efried(he's waiting for precisely top of the hour)14:59
efried(or getting coffee)15:00
*** smatzek has joined #openstack-powervm15:01
openstackMeeting ended Tue Jan  3 15:02:00 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:02
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-01-03-14.08.html15:02
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-01-03-14.08.txt15:02
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-01-03-14.08.log.html15:02
adreznecSorry, Steve stopped by my office15:02
efriedadreznec, thorst - can you help me out here?  Is this still accurate?  https://review.openstack.org/#/c/413736/2/doc/source/devref/usage.rst@7315:04
efriedWhat's the significance of --namespace nova_powervm in that command?15:05
efriedDoes that correspond to the [powervm] section in the conf?  I.e. should that have been --namespace powervm anyway?15:06
thorstI think it was to get us to load properly...but not entirely sure...15:06
efriedactually no, it appears as though the --namespace is the root package name.  The [powervm] comes from the opt code.15:08
efriedSo that line stays the same (in particular, does _not_ change to powervm_ext)15:08
thorstconf should all be the same between the two15:09
thorstwith powervm_ext being a superset of powervm in tree15:09
thorstfor now15:09
efriedCause we're changing nova_powervm/virt/powervm/nova_powervm to nova_powervm/virt/powervm_ext, not powervm_ext/virt/powervm/powervm_ext15:10
thorstcorrect - nova_powervm/virt/powervm_ext15:10
adreznecyeah, both the nova_powervm/virt and nova/virt paths should end up as powervm_ext15:11
openstackgerritEric Fried proposed openstack/nova-powervm: Change namespace to nova.virt.powervm_ext  https://review.openstack.org/41373615:15
*** kotra03 has joined #openstack-powervm15:15
*** jwcroppe has joined #openstack-powervm15:15
efriedadreznec, oh, I was going to leave nova_powervm/virt/powervm alone.15:15
*** tjakobs has joined #openstack-powervm15:16
adreznecefried: I suppose that works too15:16
efriedCause a) it's already a different namespace, and b) it's hidden by nova/virt/powervm_ext15:16
adreznecJust saw thorst mention making it nova_powervm/virt/powervm_ext there15:16
efriedSorry, that was my bad.15:17
efriedThat would make the change set a LOT bigger.  If we can get away with it this way, let's do that.15:17
adreznecYep, agreed15:17
efriedadreznec, thorst - so having been congratulatory about the CI running smoothly, it seems like we've been failing a lot of runs (most?  all?)15:22
thorstnot all15:22
thorstgo check the jenkins15:22
thorstmany are passing ( at least they were a few days back )15:22
efriedWell, I know I've been deleting tens of emails a day about failures.15:23
efriedAbove only has two failed tests.  So we not gonna worry about it til esberglu gets back?15:23
thorstsame...but we expect failures15:23
thorstchange sets will fail.15:23
thorstwe need to compare to the KVM results15:23
thorstwhich esberglu was doing before break15:24
adreznecWe're failing more than KVM right now15:24
adreznecSomething must be wonky15:24
efriedokay, so if there's a whole column of Xes, it's likely the change set that's effed; but if there's mostly greens and we're red, it's probably us.15:26
efriedSo - important to investigate ASAP, or wait for esberglu?15:26
*** wangqwsh has quit IRC15:33
thorsthmm...lets see if its an obvious problem?15:58
thorstefried: ^^15:58
thorstlike a SSP filled up or something15:58
efriedthorst, in the above example, they're both timeouts.16:00
thorstso one question...did new tests get added?16:01
thorstthat we don't support...one is a migration test16:01
thorstare the two failures consistent I guess...its always those tests or its random16:02
efriedthorst, at a glance, appears regular but not consistent.16:03
thorstnext question.  Is it always on specific hosts16:05
efriedthorst, how do I tell?16:05
thorstlog parsing I think16:06
thorstlooking for the key16:06
efriedconsole log appears to have an IP.16:06
thorstwhich it shouldn't....16:07
thorstand I think that is the client address...not the NL address16:07
*** jpasqualetto has quit IRC16:07
efriedin the console log?16:07
efriedat the very top?  9. address?16:07
efried(which, by the way, is not the same for any of the failures I'm looking at)16:08
thorstyeah, that's a client address16:08
thorstso it'll never be the same16:08
thorstI know there's a way to do it16:09
thorstI just forget how16:09
efriedUsed to be a neoXX name in the log, but not anymore.16:09
thorstyeah....probably because we eased up logging16:10
efriedI thought we got rid of this stupid "No value for $TERM" thing.16:11
thorstmanaged system UUID16:12
thorstwe'll have to work with that for now...16:12
efriedthorst, so do we need to keep digging into this now, or is it okay to wait for esberglu?16:14
efriedthorst, is our CI voting on nova-powervm changes?16:19
efriedgating, that is16:20
efriedSee https://review.openstack.org/#/c/413736/ which is marked -116:20
*** jpasqualetto has joined #openstack-powervm16:20
adreznecefried: theoretically it is16:20
adreznecAt least according to esberglu16:20
efriedSeems to be working ;-)16:20
*** jpasqualetto has quit IRC16:21
thorstefried: lets keep digging16:23
thorstit is odd...16:23
thorstthat some are passing.16:23
efriedthorst, okeydokey.16:24
thorstefried: simple fix to get us going may be to just disable those tests for now...16:28
thorstlooks like it is three tests...16:28
thorstif we don't have sorted before EOD, we should probably just push that through quick.16:28
efriedLooking at the first one (test_list_migrations_in_flavor_resize_situation) - creation succeeds; resize "succeeds" (or at least completes), and it's the confirmResize that's timing out.16:29
thorstI wonder if its a change I made...16:29
thorstefried: https://review.openstack.org/#/c/391164/16:30
thorstcould that play into it/16:30
efriedbecause not updating status?  That seems like a decent candidate.16:31
efriedwould explain intermittent-ness of failures.16:32
efriedthorst, don't we eventually emit the event anyway?16:32
thorstsomething overrides it16:32
thorstpower-off -> power-on means we wouldn't issue the power-off16:33
thorstbecause it is a reflective state.16:33
thorstI'm not sure if that plays into resize...16:33
efriedthorst, doesn't look like it.  We just POST the new specs to the LPAR.16:35
efriedwtf is the resize method named in the driver?16:38
efriedOnly Resize task call I see in driver is in finish_migration16:39
thorstbecause resize is a migration16:39
thorstto the same host16:39
thorstits a super weird thing.16:40
efriedokay, then I guess we are powering it off to do the resize.16:40
thorstyeah, I think we are16:43
thorstooo, could be a power off timeout?16:43
thorst(though that should be quick...its just sitting in SMS I think)16:44
efriedthorst, do you have the compute log open?16:44
thorstwhich one...I've been perusing many16:44
efriedI'm looking at this one:
efriedinstance is 2af9198d-b098-4919-ab38-7aa4c751d26916:46
thorsthmm...all within 10 minutes basically16:48
thorstand it timed out in the test...16:48
efried2017-01-03 07:37:19.467 issue power-on.16:49
efried2017-01-03 07:37:20.755 that completes.16:49
efried2017-01-03 07:37:23.443 Resized/migrated instance is powered off. Setting vm_state to 'stopped'16:49
efried2017-01-03 07:37:25.514 is where it gets interesting.16:49
thorstyeah, looks like it goes spastic then.16:50
efriedlooks like the DB still says it's stopped, so instead of updating the DB, it actually stops the VM!16:50
efriedThat seems wrong.16:50
thorstefried: no, that's the purpose of sync_power_state I thought16:50
thorstto bring it in line with the db.16:50
thorstbut maybe we shouldn't sync the power state on VMs with active tasks?16:50
thorstthough...its the compute manager that issues that.16:51
efriedSo we need to back up and figure out where the misstep occurs.16:52
efried2017-01-03 07:37:10.197 we start powering off.16:56
efried2017-01-03 07:37:10.205 lifecycle event 'Started'16:56
efriedbut 2017-01-03 07:37:10.429 skips syncing the power state because it's migrating.16:56
thorstwhy did we start it again is the question.16:56
efriedSo yeah, could be because we're delaying the event because we're SHUTTING_DOWN16:57
efriedWe didn't.16:57
thorstbut this timestamp: 2017-01-03 07:37:25.75116:57
thorstindicates we had an event that told us we were starting16:57
thorstwhich I thought just flew through16:58
*** kotra03 has quit IRC16:59
efriedIs that the same event that's been bouncing around since we first skipped it at 2017-01-03 07:37:10.429 ?16:59
efriedDoes it do that?  Keep events queued while operations are ongoing?16:59
efriedlooks like maybe not.  The events are tagged with a ms-since-epoch timestamp, which is different between those two events.17:00
efriedWhy does this happen: 2017-01-03 07:37:23.443 ??17:04
efriedthree seconds earlier, power-on completed.17:05
efriedthorst, what does resize actually do?  Why do we do this rename business?  Are there actually two instances with the same UUID at some point?17:08
*** jpasqualetto has joined #openstack-powervm17:09
*** chas has quit IRC17:09
thorstefried: kylek3h is actually the one who wrote it...but my understanding is that it changes the name because at some point there are two LPARs (the old size and the new size) on the host (if same host)17:09
*** chas has joined #openstack-powervm17:09
efriedSo we could be seeing power on/off for those separate instances?17:10
efriedHow does that work on our hypervisor if they have the same UUID?17:10
thorstgood questions...looking.  kylek3h you around to help answer that quick/17:14
*** chas has quit IRC17:14
kylek3hWhat's up?17:14
thorstkylek3h: with a resize on the same host...do we ever have two partitions matching?17:16
thorsttwo partitions representing the same VM?17:16
thorstI'm dug a bit more in now...doesn't look like we actually have two VMs on the same host.  We just rename the instance to help us identify it later.17:16
kylek3hright...should just be a rename....it's been a while since I looked at all that.17:17
thorstyeah, our CI is just hitting issues with it and we're trying to decompile the intention  :-)17:17
kylek3hcould have two with the same uuid on different systems but that they shouldn't be running at the same time.17:17
thorstyeah, that seems reasonable...17:18
thorstbut on the same host, we only have one VM.17:18
kylek3hI thought I had a big comment above some of that code documenting the semantics...17:18
thorstyeah, we're rushing.  :-)17:20
thorstbut its pretty obvious now.17:20
efriedSo here's what I don't understand:17:21
efried2017-01-03 07:37:20.755 Task pwr_vm completed in 1 seconds17:21
efriedThen three seconds later:17:21
efried2017-01-03 07:37:23.443 Resized/migrated instance is powered off. Setting vm_state to 'stopped'.17:21
efriedI think the problem is that the power-on finished, but for whatever reason nova didn't get the event and set the DB state to active.17:21
efriedSo when we get to :23.443, it still has cached (somewhere?? not in the DB??) that it's off, so it sets it to stopped in the DB.17:21
efriedMaybe that event is what's hitting at17:23
efried07:37:25.515 Emitting event <LifecycleEvent: 1483450645.52, 2af9198d-b098-4919-ab38-7aa4c751d269 => Started>17:23
efried...which is too late.  Though I still don't get why that event triggers sync_power_state, and/or why sync_power_state overrides the event and/or real state of the partition.17:23
thorstYeah, I have the code here...sec, let me find it on github17:25
thorstI wonder if we just need to update the 'instance.power_state' as part of the power on/off jobs...17:26
efriedthorst, that code is conditioned on the VM power state being SHUTDOWN, though - which it shouldn't be, should it?17:28
thorstI don't see any other driver having to do that...17:28
efriedWhere does instance.power_state normally get set?17:31
thorstin the compute manager.17:32
thorstit calls _get_power_state in the manager.  Which then calls to 'get_info' in the driver.  Which then calls down to 'InstanceInfo' in vm.py17:33
thorstso power_state should always be the real state of the VM (at least if its set via _get_power_state)17:34
*** thorst is now known as thorst_afk17:35
*** thorst_afk is now known as thorst17:51
thorstefried: well, turns out esberglu isn't back until the 16th17:51
thorstefried: I have a patch I'm going to propose up...see if it fixes that issue...18:00
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix  https://review.openstack.org/41631518:05
thorstefried: ^^18:05
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix  https://review.openstack.org/41631518:06
efriedthorst, lgtm.  If CI passes, though, we don't necessarily know the problem is fixed, do we?18:14
thorstefried: yeah, who knows if that's a legit fix or not.18:24
thorstjust seemed like it could be the cause18:24
*** k0da has joined #openstack-powervm18:37
*** chas has joined #openstack-powervm19:10
efriedthorst, fyi, the other failure in that run looks to be the same issue.19:14
*** chas has quit IRC19:15
efried...and your proposal passed CI, for whatever that's worth.19:15
adreznecefried: thorst should we run rechecks on that one a few times?19:16
efriedadreznec, good idea.19:16
adreznecFor a bit more confidence?19:16
*** efried has quit IRC19:35
*** adi_____ has quit IRC19:35
*** efried has joined #openstack-powervm19:36
*** adi_____ has joined #openstack-powervm19:41
thorstagree on that19:43
thorstthx for kicking it off19:43
efriedthorst, where did we land on the whole upload thing?19:47
efriedDid y'all manage to sort that out without my interference?19:48
efried...err, "help"?19:48
thorstwell, we have upload working19:48
thorstremote upload through the function is still in that hack'ed patch set19:48
thorstwhich I am still a proponent of19:48
thorstbut we've got the glance case sorted out...no complaints that I'm aware of19:48
thorstlet me ask kris19:48
thorstaccording to Kris - was working well, no one has tried it this new year.19:57
*** kriskend has joined #openstack-powervm19:57
adreznecthorst: Yeah, we haven't heard anything more on it after delivering the full package20:01
adreznecHolidays and all20:01
*** openstackgerrit has quit IRC20:03
adreznecFYI thorst efried - Not that it directly impacts us right now, but we may want to keep an eye on the OSA SRIOV patch series (e.g. https://review.openstack.org/#/c/415903/) to see how things develop from a compatibility perspective20:14
thorstefried: 3 tests failed in this pass.20:39
thorstits uploading logs now...20:39
*** jpasqualetto has joined #openstack-powervm20:40
thorstsame failures as before...20:40
efriedthorst, what's docker?20:54
efried(adreznec ^^)20:57
efriednever mind, I think I get it.20:59
efriedI don't know if this is good exposure, but they used nova-powervm as an example/template for their OOT driver: https://review.openstack.org/#/c/408148/521:00
efriedI guess imitation is the sincerest form of flattery, and all that.21:00
thorst:-)  neat21:01
thorstefried: so I'm looking at the difference between a good run and a bad run.21:05
thorstsame patch set21:05
*** chas has joined #openstack-powervm21:11
*** chas has quit IRC21:15
thorstdefinitely the event thing...21:18
thorstwell...wait...I got my success and failure ones reversed.21:19
thorsto, no I didn't...21:19
thorstI think I see the fix...21:20
*** smatzek has quit IRC21:20
*** openstackgerrit has joined #openstack-powervm21:28
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix  https://review.openstack.org/41631521:28
*** chas has joined #openstack-powervm21:51
*** chas has quit IRC21:56
*** jwcroppe has quit IRC21:58
*** smatzek has joined #openstack-powervm22:01
*** svenkat has quit IRC22:02
*** kriskend has quit IRC22:05
*** thorst has quit IRC22:07
*** thorst has joined #openstack-powervm22:08
*** chas has joined #openstack-powervm22:12
*** thorst has quit IRC22:13
*** chas has quit IRC22:16
*** edmondsw has quit IRC22:21
*** edmondsw has joined #openstack-powervm22:21
*** tjakobs has quit IRC22:24
*** edmondsw has quit IRC22:25
*** thorst has joined #openstack-powervm22:32
*** thorst has quit IRC22:37
*** tblakes has quit IRC22:40
*** smatzek has quit IRC22:44
*** thorst has joined #openstack-powervm22:53
*** thorst has quit IRC22:57
*** mdrabe has quit IRC23:01
*** openstack has joined #openstack-powervm23:15
*** k0da has quit IRC23:27
*** apearson has quit IRC23:29

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!