Wednesday, 2017-02-01

*** dwayne has joined #openstack-powervm00:01
*** k0da has quit IRC00:03
*** kylek3h has joined #openstack-powervm00:36
*** esberglu has quit IRC01:21
*** esberglu has joined #openstack-powervm01:52
*** esberglu has quit IRC01:56
*** esberglu has joined #openstack-powervm03:05
*** esberglu has quit IRC03:25
*** esberglu has joined #openstack-powervm03:25
*** efried has quit IRC03:28
*** esberglu has quit IRC03:30
*** efried has joined #openstack-powervm03:41
*** tlian has quit IRC03:59
*** thorst_ has quit IRC04:00
*** thorst_ has joined #openstack-powervm05:00
*** tjakobs has joined #openstack-powervm05:03
*** thorst_ has quit IRC05:04
*** tjakobs has quit IRC05:09
*** thorst_ has joined #openstack-powervm05:31
*** thorst_ has quit IRC05:35
*** thorst_ has joined #openstack-powervm07:31
*** thorst_ has quit IRC07:36
*** esberglu has joined #openstack-powervm07:50
*** esberglu has quit IRC07:55
*** k0da has joined #openstack-powervm08:18
*** thorst_ has joined #openstack-powervm09:32
*** thorst_ has quit IRC09:38
*** esberglu has joined #openstack-powervm09:39
*** esberglu has quit IRC09:44
*** efried has quit IRC10:06
*** edmondsw has joined #openstack-powervm10:30
*** thorst_ has joined #openstack-powervm10:34
*** edmondsw has quit IRC10:35
*** esberglu has joined #openstack-powervm10:39
*** thorst_ has quit IRC10:39
*** esberglu has quit IRC10:44
*** esberglu has joined #openstack-powervm12:24
*** esberglu has quit IRC12:28
*** dwayne has quit IRC12:45
*** thorst_ has joined #openstack-powervm12:45
*** edmondsw has joined #openstack-powervm12:46
*** edmondsw has quit IRC12:46
*** edmondsw has joined #openstack-powervm12:46
*** tblakes has joined #openstack-powervm13:09
*** efried has joined #openstack-powervm13:19
*** newbie|3 has joined #openstack-powervm13:32
*** dwayne has joined #openstack-powervm13:47
*** apearson has joined #openstack-powervm14:03
*** tlian has joined #openstack-powervm14:07
*** esberglu has joined #openstack-powervm14:13
*** mdrabe has joined #openstack-powervm14:13
*** esberglu has quit IRC14:13
*** esberglu has joined #openstack-powervm14:14
*** esberglu_ has joined #openstack-powervm14:18
*** esberglu has quit IRC14:21
thorst_esberglu_: lots of failures in CI.  How we holding up?14:25
*** apearson has quit IRC14:28
*** apearson has joined #openstack-powervm14:30
*** esberglu has joined #openstack-powervm14:35
*** esberglu_ has quit IRC14:39
*** kjw3 has joined #openstack-powervm14:43
esbergluthorst_: Here are the failures since I turned it back on yesterday. (not including bad patches / our in tree runs)14:56
esbergluI'm thinking we may still have an issue with the power off stuff14:56
thorst_efried: do you have any time to chase those?15:01
esbergluA lot of the rebuild tests may be explained by that plus there are some that are still explicitly failing to power off15:01
efriedWe're talking about the in-tree CI failing on power-off?15:02
efriedOkay, yeah, I can take a look.15:02
esbergluNo the out of tree15:02
thorst_thx dude15:03
esbergluI think some of the others may be bad patches or not supported.15:04
esberglup.s. I put the results from the in-tree runs at the bottom of that paste15:05
esbergluNot all of them are from the latest patch, but just to get an idea where it's at15:06
esbergluSeeing about 5-7 failures a run so far there15:07
esbergluI need to dig in and see which are unsupported and if there are real failures15:08
efriedSo hold on15:08
efriedwhat is it I'm looking at?15:08
efriedSpecifically just those two power-off failures?15:08
efriedthorst_ esberglu ^^15:09
thorst_yeah, lets see if there are power off failures...and then figure out why...15:09
thorst_other random failures, consider noise for the time being15:09
thorst_or a separate thing to chase15:09
efriedokay.  Two power-off failures on just two runs in two separate test cases (though both rebuild) seems like random noise.15:10
efriedBut I'll look.15:10
esbergluThose two yeah. But I think some of the rebuilds might be hitting the same thing15:10
efriedSo I'll reiterate what I said yesterday: I expect the in-tree to hit those power-off failures, because pypowervm doesn't have the latest fix.15:11
thorst_eww...15:12 do we get rid of that noise?15:12
thorst_patch it in as part of the we did with the local2remote15:12
thorst_until we can bump to
efriedWell, we could disable the new timeout code in-tree.15:12
efriedOr that, but that seems a tad dishonest.15:12
thorst_I don't think dishonest.15:13
thorst_I think it will be pushed through by the time we're actually core reviewed15:13
thorst_I just don't want to be seeing failures for a month15:13
efriedShrug, okay.  So esberglu - wherever you're pulling in 2382, pull in 4761 as well.15:14
efriedUgh, the CI env doesn't just conveniently patch those in from git, does it?  It has, like, a copy of the diff in a text file checked in separately?15:15
esbergluBut that doesn't explain the power off issues for oot15:16
efriedNono, separate issue.15:16
esbergluOkay just making sure we were on the same page15:16
efriedbtw, esberglu, not sure how you're doing your data collection, but I'm seeing two power-off failures in
esbergluThat list definitely isn't perfect. Just gives a rough idea of what we are seeing15:18
esbergluI want to make some sort of tool to automate this, I just haven't looked into it yet15:18
*** apearson has quit IRC15:21
esbergluMaybe jenkins has a plugin already15:21
*** kriskend has joined #openstack-powervm15:31
*** apearson has joined #openstack-powervm15:32
*** tjakobs has joined #openstack-powervm15:32
efriedesberglu thorst_ - So from what I'm seeing, this is trying the power-off unforced with a 60s timeout, and then when that fails, it's *not* retrying.15:39
efriedThe code path that I believe leads to this is when the LPAR doesn't have RMC.15:40
thorst_well then my pypowervm 'fix' wasn't good15:40
efriedYeah, your pypowervm fix was fine - we just weren't hitting it.15:40
thorst_I didn't think about RMC.15:40
efriedI'm questioning whether we should declare VMPowerOffFailure if we timeout that path (no RMC).15:40
efriedOr just ignore it.15:41
efriedWe don't add the immediate flag when RMC is dead - just change from osshutdown to shutdown.15:41
efriedSooo... I wonder if we should force-retry when that guy fails.15:42
efriedAdd another noodle to that spaghetti code.15:42
efriedThis is really getting to the point where I'm gonna refactor the sonofab!tch15:42
efriedIt's out of control.15:42
thorst_efried: I'd support that.  But the net is, if RMC is down, an osshutdown doesn't work, then a shutdown doesn't the damn power out.15:43
thorst_I think it was built the way it was because someone wanted to keep LoC down.15:43
efriedsupport which?  The force-retry in that case, the refactor, or both?15:43
efriedYeah, at this point I believe refactoring will actually reduce the LOC.15:44
efriedThinking recursive.15:44
efriedinstead of loop15:44
efriedAnyway, I'll do it in two steps.15:44
efriedFirst change set I'll add the force flag and retry in that scenario.15:44
efriedThen I'll do a refactor next.15:44
efriedOff I go.15:44
*** newbie|3 has quit IRC15:54
thorst_cool beans15:55
*** jpasqualetto has joined #openstack-powervm15:56
efriedthorst_ 476716:04
efriedesberglu: What does it take to pick up ^^?  Does it need to merge, do things need to be rebuilt/redeployed, ...?16:04
esbergluManagement node redeploy (~2 hours)16:06
thorst_and getting it into the VMs16:06
thorst_so you have to flush ready nodes?16:06
efriedSho would be nice if we had a way to try these pypowervm patches in the CI before they merge.16:07
esbergluI could kick off an image build and flush the nodes manually, but it ends up taking just as long16:07
thorst_staging env...16:08 this case...16:08
esbergluI could do a manual run on the staging env16:09
thorst_I'd say go prod16:10
thorst_for throughput16:10
esbergluFine by me16:11
*** mdrabe has quit IRC16:23
*** mdrabe has joined #openstack-powervm16:29
*** k0da has quit IRC16:35
esbergluefried: thorst_: Sorry I wasn't clear before. 4767 still needs to be merged to test on the production environment16:48
* efried grumbles16:48
esbergluThen patch in both 4761 and 4767 for in-tree16:50
*** kjw3 has quit IRC17:00
efriedesberglu Yeah.  As well as local2remote.17:01
esbergluAnd abort fifo17:01
*** kriskend has quit IRC17:28
*** tjakobs has quit IRC17:28
*** burgerk has quit IRC17:33
*** kylek3h is now known as kylek3h_away17:34
*** esberglu has quit IRC17:55
*** esberglu has joined #openstack-powervm17:56
*** esberglu_ has joined #openstack-powervm17:57
*** esberglu has quit IRC18:00
*** esberglu_ has quit IRC18:01
*** esberglu has joined #openstack-powervm18:09
*** esberglu has quit IRC18:09
*** esberglu has joined #openstack-powervm18:09
*** apearson has quit IRC18:29
*** apearson has joined #openstack-powervm18:32
*** apearson has quit IRC18:44
*** apearson has joined #openstack-powervm18:46
*** apearson has quit IRC18:50
*** apearson has joined #openstack-powervm19:12
esbergluReminder that tomorrow is the 1st release candidate for *-powervm projects. This means that a stable/ocata branch will be created.19:25
*** efried has quit IRC19:25
esbergluAny further changes for the ocata release will have to be made on the master branch and backported to the stable/ocata19:25
*** apearson has quit IRC19:28
*** efried has joined #openstack-powervm19:48
*** jpasqualetto has quit IRC19:48
*** k0da has joined #openstack-powervm19:55
*** apearson has joined #openstack-powervm20:02
*** jpasqualetto has joined #openstack-powervm20:05
*** jpasqualetto has quit IRC20:35
*** edmondsw has quit IRC21:44
*** edmondsw has joined #openstack-powervm21:53
*** thorst_ has quit IRC22:09
*** thorst_ has joined #openstack-powervm22:16
*** thorst_ has quit IRC22:20
*** thorst_ has joined #openstack-powervm22:40
*** edmondsw has quit IRC22:41
*** thorst_ has quit IRC22:44
*** tblakes has quit IRC22:46
*** tlian has quit IRC22:51
*** tjakobs has joined #openstack-powervm22:51
*** tlian has joined #openstack-powervm22:58
*** esberglu has quit IRC23:01
*** dwayne has quit IRC23:02
*** tlian2 has joined #openstack-powervm23:10
*** mdrabe has quit IRC23:10
*** tlian has quit IRC23:13
*** tlian2 has quit IRC23:15
*** k0da has quit IRC23:17
*** tlian has joined #openstack-powervm23:24
*** apearson has quit IRC23:36
*** tjakobs has quit IRC23:51
*** dwayne has joined #openstack-powervm23:57

Generated by 2.14.0 by Marius Gedminas - find it at!