Thursday, 2016-11-17

*** tjakobs has quit IRC00:02
*** chas has quit IRC00:09
*** chas has joined #openstack-powervm00:09
*** chas has quit IRC00:14
*** jwcroppe has joined #openstack-powervm00:21
*** jwcroppe has quit IRC00:49
*** jwcroppe has joined #openstack-powervm00:49
*** esberglu_ has quit IRC01:11
*** esberglu has joined #openstack-powervm01:11
*** esberglu has quit IRC01:16
*** svenkat has joined #openstack-powervm01:33
*** thorst_ has joined #openstack-powervm01:46
*** esberglu has joined #openstack-powervm01:56
*** esberglu has quit IRC02:30
*** esberglu has joined #openstack-powervm02:30
*** esberglu has quit IRC02:34
openstackgerritSridhar Venkat proposed openstack/networking-powervm: ProvisionRequest does not distinguish event source
*** seroyer has joined #openstack-powervm03:01
*** tlian has quit IRC03:18
*** thorst_ has quit IRC03:24
*** thorst_ has joined #openstack-powervm03:26
*** seroyer has quit IRC03:31
*** thorst_ has quit IRC03:32
*** svenkat has quit IRC03:35
*** jwcroppe has quit IRC03:54
*** jwcroppe has joined #openstack-powervm03:55
*** thorst_ has joined #openstack-powervm04:30
*** thorst_ has quit IRC04:38
*** kotra03 has joined #openstack-powervm05:16
*** thorst_ has joined #openstack-powervm05:36
*** thorst_ has quit IRC05:43
*** chas has joined #openstack-powervm05:54
*** chas has quit IRC06:01
*** chas has joined #openstack-powervm06:06
*** chas has quit IRC06:11
*** esberglu has joined #openstack-powervm06:25
*** esberglu has quit IRC06:29
*** esberglu has joined #openstack-powervm06:30
*** esberglu has quit IRC06:35
*** chas has joined #openstack-powervm06:38
*** thorst_ has joined #openstack-powervm06:42
*** chas has quit IRC06:43
*** thorst_ has quit IRC06:48
*** esberglu has joined #openstack-powervm07:20
*** esberglu has quit IRC07:24
*** thorst_ has joined #openstack-powervm07:45
*** thorst_ has quit IRC07:53
*** esberglu has joined #openstack-powervm08:19
*** esberglu has quit IRC08:24
*** thorst_ has joined #openstack-powervm08:51
*** thorst_ has quit IRC08:58
*** esberglu has joined #openstack-powervm09:14
openstackgerritArun Mani proposed openstack/nova-powervm: Object upload using swift occasionally fails with 'TypeError' exception.
*** esberglu has quit IRC09:18
*** YuYangWang has quit IRC09:33
*** openstackgerrit has quit IRC09:48
*** openstackgerrit has joined #openstack-powervm09:49
*** thorst_ has joined #openstack-powervm09:56
*** thorst_ has quit IRC10:02
*** esberglu has joined #openstack-powervm10:09
*** esberglu has quit IRC10:13
*** k0da has joined #openstack-powervm10:40
*** esberglu has joined #openstack-powervm10:59
*** thorst_ has joined #openstack-powervm11:01
*** esberglu has quit IRC11:03
*** thorst_ has quit IRC11:08
*** esberglu has joined #openstack-powervm11:59
*** esberglu has quit IRC12:04
*** thorst_ has joined #openstack-powervm12:06
*** thorst_ has quit IRC12:12
*** seroyer has joined #openstack-powervm12:23
*** edmondsw has joined #openstack-powervm12:48
*** thorst_ has joined #openstack-powervm12:51
*** esberglu has joined #openstack-powervm12:54
*** esberglu has quit IRC12:58
*** seroyer has quit IRC13:01
*** seroyer has joined #openstack-powervm13:03
*** svenkat has joined #openstack-powervm13:08
*** seroyer has quit IRC13:14
*** esberglu has joined #openstack-powervm13:27
*** wangqwsh has joined #openstack-powervm13:30
adreznecWhat's a CI? Never heard of it13:31
esberglu#startmeeting ci_scrum13:31
openstackMeeting started Thu Nov 17 13:31:23 2016 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:31
openstackThe meeting name has been set to 'ci_scrum'13:31
esberglu#topic status13:31
adreznecIn the future we should do powervm ci scrum, then all the logs go under the same directory out on eavesdrop13:32
adreznecHaven't had enough coffee to raise hand13:32
esbergluI looked at the dir online and it just says ci_scrum?13:33
adreznecYeah, in the past I was doing powervm ci meeting13:34
adreznecSo they were going to
adreznecIf you keep giving it the same name each time they all go to the same directory13:34
adreznecNot a big deal13:34
esbergluOh I had the wrong bookmark13:35
esbergluWe must have done it there once or something13:35
thorst_Status - stop everything until CI is running?13:35
esbergluI did end up getting a run to go through last night13:35
esbergluBut still seeing the same behavior13:36
thorst_elaborate?  The pypowervm issue certainly isn't fixed13:36
esbergluYeah. But I put USE_CONSTRAINTS=false13:36
thorst_that worked?13:36
esbergluDidn't we think that would work around?13:36
esbergluIt did not work13:37
thorst_we thought it might.13:37
thorst_is everyone aware what the pypowervm issue is?13:37
thorst_adreznec?  I'm not sure if you were yet13:37
adreznecSo we're still seeing constraints issues?13:37
adreznecWhy here and not in the upstream gate?13:37
thorst_devstack installs from the requirements file.  Which then blows away our local install (with the local2remote patch)13:37
thorst_before we dive into that (and I'd like to do a deep dive on that)13:39
thorst_are there any other items we need to chase esberglu?13:39
thorst_otherwise the meeting will get away on us with that rabbit hole13:39
esbergluI still need to do work on the post stack vm cleaner script13:39
esbergluAnd neo 6,7,8 stop working last night. Probably just need to restart vio_daemon13:40
thorst_esberglu: was that just the lu clean up?13:41
adreznecJust... stopped working?13:41
adreznecwithout the CI running/doing anything?13:41
thorst_adreznec: apearson is adding in a 'heartbeat' of sorts for that.13:41
* adreznec sighs13:41
thorst_so manual restarts for now, update of novalink in future should help13:41
esbergluthorst_: It looked like the LU thing. It happened after I was clearing out the nodes so I could deploy with USE_CONSTRAINTS13:41
esbergluBut it wouldn't let me delete the LUs or the VMs13:42
esbergluI didn't spend too much time looking at it, was more interested in getting a run through the CI13:44
thorst_esberglu: alright.13:45
thorst_#action thorst to follow up with apearson on vio_daemon thing13:45
thorst_#action esberglu to work on the clean up script for lu.  Possibly restart vio_daemon depending on how far along that is in NL13:45
thorst_any other items?13:46
*** seroyer has joined #openstack-powervm13:47
*** tlian has joined #openstack-powervm13:48
esbergluNot from me13:48
esbergluYou guys ready to talk about the pypowervm versioning then?13:49
thorst_esberglu: do you have a link to a run that we saw this issue/13:49
adreznecSo if I understand correctly it's happening because pypowervm is now a global-req, which because our stuff now as a real requirement.txt requirement on pypowervm is causing pypowervm to be installed down the develop branch path I put in, erasing our pre-installed version of pypowervm13:50
thorst_adreznec: well, it doesn't erase it13:50
thorst_but yeah, effectively.13:50
esbergluCrap not what I meant to paste13:51
adreznecDoes it not install over the top of it?13:52
thorst_adreznec: Search for this timestamp: 2016-11-16 20:27:46.70313:52
efriedIt would be relatively easy to fix this in our CI - our scripts download the requirements project first, then hack the pypowervm lines out of global-requirements.txt and upper-constraints.txt, then proceed with the rest of devstack - but fixing it for our devstack in general is going to be tougher.13:52
efriedHere's one suggestion, not particularly pretty:13:52
efriedWe start by re-syncing into
efriedThen moving forward, we write any new pypowervm function into nova-powervm itself (or whichever out-of-tree driver needs it), while continuing to develop against
efriedThen periodically, we release a new version of pypowervm to pip (,, ... just kidding) and update the global requirements.13:52
efriedOnce that merges, we update nova-powervm accordingly, removing that functionality and repointing its usage from the temporary nova-powervm code to the released pypowervm code.13:52
thorst_the challenge with that is that we don't want frequent updates to the global-requirements.txt.13:53
efriedRight, so when I say "periodically" above, we're talking once or twice per release.13:53
efried(openstack release, that is)13:54
thorst_then we lose the flexibility that we have now of getting fixes across multiple components (in a short period of time) for the out of tree driver13:54
adreznecYeah, that's kind of a pain13:54
efriedIt's pretty rare that we need something from pypowervm in multiple out-of-tree drivers, neh?13:54
thorst_I think that's 100% right for fact I'd like the in-tree to use basically once that goes out, even after pypowervm goes to
thorst_at least support it.13:55
*** tblakes has joined #openstack-powervm13:57
thorst_I like the idea of having the out of tree potentially overwrite the req.13:59
efriedCan we accept that it won't work for devstack in general?  Only for CI?14:00
thorst_we need a devstack'd solution14:00
thorst_I'd accept that we don't have the right answer for now...and maybe the work around is that we have something in the nova-powervm (or c/n) devstack scripts tweak the upper-constraints14:00
thorst_and that we need to dwell on this a bit.14:01
efriedWould need to confirm this, but I believe the devstack scripts run really early, as soon as the project is collected.14:01
efriedThe problem there is that we don't know for a given devstack run whether nova-powervm is collected before or after the requirements project.14:02
efriedIt's not deterministic.14:02
efriedWhich is kind of the point of global requirements - to at least make the levels of everything deterministic regardless of the order they happen to come in.14:03
thorst_so it only installs for projects that require it.  Right now that's nova-powervm, networking-powervm, and ceilometer-powervm (and at some point nova)14:03
thorst_so it would be called before that14:03
thorst_but potentially, quickly changed.14:03
thorst_because nova would be the first thing called.14:04
thorst_so it's deterministic right now...14:04
thorst_I think the 'hack' (and it is a hack) is two stages.14:04
efriedWe can't get away with having nova's devstack scripts muck with the requirements.14:04
thorst_1) our devstacks tweak the upper-limits14:05
thorst_2) once in nova, we know that nova will go before nova-powervm.  So there nova-powervm would have to uninstall what nova put in, then let the other install go through.14:05
thorst_the other install being what we developed in 114:05
thorst_core nova won't muck with it.  That's a no go.14:06
thorst_nor should it.14:06
thorst_that'd be...awful.14:06
efriedthorst_, I think we can implement both pieces at the same time.14:08
efriedBasically *-powervm uninstalls any pypowervm that happens to be there and installs the right version.14:08
thorst_efried: possibly, but if we separate out and get one before the other...then I'd like to do what's quickest14:08
thorst_(unless its like a couple hour difference)14:09
adreznecUninstalling 'any pypowervm that happens to be there' basically assumes it was pip installed14:09
adreznecUnless we're going to get in the game of doing apt/yum removes, etc14:10
adreznecWhich sounds gross14:10
thorst_adreznec: well, we'd just pip uninstall14:11
thorst_not apt-get uninstall14:11
adreznecI think part of this comes back to our pypowervm versioning being broken14:12
adreznecRight, because if we had proper versioning this wouldn't be a real issue14:12
adreznecWe could bump upper-constraints to something reasonable14:13
adreznecand not worry about it ripping this version out from under us14:13
efriedadreznec, wouldn't we still have the moving-target problem?14:14
efriedAssuming we can't update upper-constraints twice a week whenever we put something new in pypowervm14:14
thorst_and wouldn't it be well know that version 2.0.x doesn't exist, so how can that realistically be an upper-constraint14:14
adreznecSure, but so does everyone else in openstack... it makes things a bit less flexible, I agree14:15
*** mdrabe has joined #openstack-powervm14:15
efriedRight, I have to assume you can't put a version number that doesn't exist into requirements.  Or that'd be a snap.14:15
thorst_unless you can?14:15
adreznecEven if you could, I'd imagine the reqs team would -1 that patchset if they caught it14:16
adreznecOtherwise it kind of defeats the entire purpose of upper-constraints, no?14:16
efriedFor sure.14:16
efriedThat's what we're trying to do.14:16
adreznecLook, our upper-constraint is version 9000!14:16
efriedDefeat the entire purpose of upper-constraints.14:16
efriedI'm still finding it tough to believe that nobody else runs into this on a regular basis.14:17
adreznecOur goals are very different from their goals here since this is basically a development branch of an out-of-tree driver14:17
adreznecYeah, I looked at the compute-hyperv driver14:18
thorst_back to the 'out-of-tree driver should isolate its weirdness in itself'14:18
adreznecBut they just bump os-win14:18
adreznecand requirements14:18
efriedHow frequently?14:18
adreznecNot that frequently14:19
efriedSo thorst_, give it to me again why the lockstep approach wouldn't work?  (Accumulating function in nova-powervm and then, a couple times a release, "moving" it over to pypowervm.)14:19
thorst_efried: well that was my request earlier.  Can we get more frequent releases of pypowervm without driving the whole train along with it.14:20
thorst_which perhaps we can.14:21
esbergluI know this won't work for all cases. But for now in the CI couldn't we just update the pypowervm requirement everywhere we need to before devstack even starts14:21
efriedesberglu, yes, but we need a solution that works for devstack in general.14:21
thorst_esberglu: no, cause its too high a version, so upper constraints installs over it14:21
thorst_esberglu: sorry, I misread.14:21
esbergluefried: Yeah I just meant for now. So we can get CI up14:21
thorst_esberglu: yeah, lets I guess do it for now.14:22
esberglu#action esberglu: Change pypowervm requirements before CI runs14:23
thorst_alright...lets do that for CI, cool off and figure out how the heck we do this moving forward.14:25
esbergluSounds good. Anything else before I end the meeting? I know we are way long14:25
efriedIt looks like, if we wanted to do this in the *-powervm projects' devstack scripts, it would install over the top of
efried...which gets pulled down *with* each *-powervm in a way we have no control over.14:26
thorst_yep.  Lets close up meeting here14:27
thorst_we can still discuss14:27
openstackMeeting ended Thu Nov 17 14:27:28 2016 UTC.  Information about MeetBot at . (v 0.1.4)14:27
openstackMinutes (text):
*** tblakes has quit IRC14:40
*** wangqwsh has quit IRC14:47
openstackgerritArun Mani proposed openstack/nova-powervm: Object upload using swift occasionally fails with 'TypeError' exception.
*** kotra03 has quit IRC14:55
*** arunman has joined #openstack-powervm14:57
*** apearson has joined #openstack-powervm14:59
*** seroyer has quit IRC15:06
*** tblakes has joined #openstack-powervm15:09
arunmanthorst_: I have a review out for the swift issue. Please take a look when you have time15:17
arunmanthis is just a hack in nova-powervm, till we fix it the right way from swift client side15:17
thorst_arunman: why do we do it only on the store?  Couldn't this happen on the get as well/15:17
arunmanthis is only hit with _upload_object_call on the swift module15:20
*** seroyer has joined #openstack-powervm15:20
arunmanit's actually _upload_object_job15:20
*** tjakobs has joined #openstack-powervm15:23
thorst_arunman: ahh, cool.15:36
thorst_that makes the workaround better.15:36
thorst_arunman: I did a -115:39
thorst_see comments as to why and we can chat it out15:39
arunmanthorst_: sure15:40
arunmanOn changing the retry attempt to 3, this is just to give it one more try in case if there is ClientException and TypeError. The ClientException will most probably go off in the first retry attempt.15:43
arunmanOn why we NOT fail deploy here... this is only in case if a TypeError is hit. After few attempts I've seen that the periodic task updates swift with the required data.15:45
arunmanBut yes... if RR is attempted before swift gets updated, RR will fail or the VM on destination host will not have recent data15:46
thorst_arunman: and we can't let remote rebuild fail.  We've seen hyper issues where the volume got deleted in that case15:49
thorst_so its better to fail the deploy there.15:50
thorst_does the TypeError not go away with a retry?15:50
arunmanmost cases yes. It takes a while to go away15:51
arunmanI'll look at handling deploy case15:52
thorst_arunman: can you define 'takes a while to go away'16:00
thorst_does it block deploys for several minutes?16:00
thorst_if so, maybe we increase the retry count much higher (and add a small delay)16:00
arunmanit's like... we keep doing a call to swift for upload. Once TypeError is hit, you keep seeing this for sometime. But, this goes off on its own after a while and the subsequent uploads will work16:06
*** arunman has quit IRC16:06
*** arunman has joined #openstack-powervm16:07
arunmanI'm not able to predict the behavior... even if we increase the retry count to a higher value, can't be sure it will work.16:08
thorst_but we lower the probability16:09
thorst_I guess I'm wondering if we need a new session, or a new swift client, or something else16:09
thorst_or, it just disappears.16:09
thorst_so I would propose we retry more often...and then if it does fail...well it fails.16:09
thorst_that way we don't get into data integrity issues.16:10
arunmanI'm of the opinion that, it just disappears16:11
arunmanyeah.. we were failing  it anyway. My fix was just to handle it gracefully and not to fail any operation and we expect the periodic task to update it.16:14
arunmanBut if we retry more often and then fail it... I'm still OK with that16:14
thorst_yeah, that's my preference16:16
thorst_I don't think I'll allow anything through that allows us to deploy without NVRAM (when you set the driver to *require* NVRAM be persisted to Swift)16:17
arunmanThe same applies for volume attach and detach. How would you suggest to handle those?16:18
thorst_anything that changes the NVRAM, if we can't persist to NVRAM, we need to block that action16:18
thorst_that's my view.16:18
thorst_the contract is that the transaction completed wholly and fully...and the task flow should unwind it when it fails.16:19
arunmanGot that16:19
*** arunman has quit IRC16:25
*** mdrabe has quit IRC16:39
*** mdrabe has joined #openstack-powervm16:44
*** thorst_ has quit IRC17:59
*** thorst_ has joined #openstack-powervm18:00
svenkatthorst_ , efried: want to discuss here. with respect to default value of None for vif_type parameter in ProvisionRequest init.18:03
svenkatvif_type parameter is valid and consumed only in for_event path and not in for_wrapper path. heal_and_optimize is not impacted.18:04
svenkatfor_event path checks vif_type in ProvisionRequest and matches it with agent’s vif_type.18:05
efriedsvenkat, Agree for now we should leave it defaulting to None like you've got it.  If we discover that there's a deterministic way to correlate wrapper type to vif type, we may incorporate that later - but certainly not in this change set.  thorst_, you agree?18:05
openstackgerritEric Berglund proposed openstack/nova-powervm: DNM: CI Check2
*** jwcroppe has quit IRC18:12
*** thorst_ has quit IRC18:26
*** thorst_ has joined #openstack-powervm18:26
*** thorst_ has quit IRC18:58
*** thorst_ has joined #openstack-powervm19:05
thorst_sorry - was AFK.  The only deterministic way is via vSwitch and vNIC config19:09
thorst_if they're attached to a specific vSwitch we kind of know what they are19:10
thorst_efried: can we do another review of 4533 and 4504?19:16
thorst_looks like we got jenkins going19:16
efriedthorst_, okay.  Where do you want to start?19:16
thorst_I think we've both should both probably re-review19:17
efriedthorst, okay, this lgtm.  Shall we both +2?19:18
efriedthorst_ ^^  I +2ed.  You do same, then merge.19:19
thorst_onto 4533...surprised it didn't have a merge conflict19:19
thorst_efried: 4533 will have another pass.19:20
efriedthorst_, ferwhat?19:21
thorst_the vOpt changes are unrelated19:21
thorst_so I want to undo that19:21
efriedokay.  I was just inspecting  those.  Seem harmless, but yeah, unrelated.19:22
efriedthorst_, hold on a tick.19:22
efriedI don't think the vio_file removal thing is relevant, really.19:22
efriedYou're basically trying to remove it again the same way we tried to remove it originally.19:22
thorst_it just should always be None19:23
efriedDo you have some reason to believe that'll work where it didn't before because the vdisk is gone?19:23
thorst_because now we're deleting it19:23
thorst_the intention from kyle earlier was you needed the file to know how to remove it yourself.19:23
thorst_and I don't really agree...we delete it now19:23
efriedWe always deleted it.19:23
efriedOr tried, anyway.19:23
efriedWe returned it if it didn't get deleted for some reason.19:23
efriedWhich I contend we should still do.19:23
efriedThat piece of code is tangential to your change.19:24
efriedUnless you assert that the deletion may fail while the vdisk exists, but succeed after we remove it.19:24
thorst_yep, I think that could happen19:25
thorst_at least, I think that's what kriskend saw19:26
efriedthorst_ So even then, I want to return the vio_file if _that_ deletion also fails.19:27
thorst_o bleh19:27
thorst_I *hate* that variable19:28
thorst_no one in their right mind will use it19:28
efriedThen it's harmless.19:28
thorst_but I do see your point19:28
thorst_consistency with a bad past there.19:28
thorst_sometimes I hate your correctness19:28
efriedSometimes I hate being correct.19:29
efriedUsually not, though.19:29
*** tblakes has quit IRC19:34
thorst_efried: pushed another up19:38
efriedthorst_, beautiful, +219:42
efriedCleaner, no?19:42
*** tblakes_ has joined #openstack-powervm19:45
*** openstackgerrit has quit IRC19:48
*** tblakes_ is now known as tblakes19:48
*** openstackgerrit has joined #openstack-powervm19:49
*** k0da has quit IRC19:51
*** kylek3h has quit IRC19:57
thorst_efried: yep19:59
thorst_and kriskend is happy now19:59
*** tjakobs has quit IRC20:00
*** tjakobs_ has joined #openstack-powervm20:00
*** tblakes has quit IRC20:33
*** kriskend has joined #openstack-powervm20:48
kriskendthorst_ If pypowervm was dead when I did a bunch of VM deletes from openstack, after fixing pypowervm, shouldn't the reaper clean up my instances?20:50
thorst_assuming they got far enough to create instances...yes, eventually20:50
kriskendhow often does it run?20:50
thorst_I thought on boot20:51
kriskendso I had instances, I killed pypowervm by uninstalling it and not fixing the sym link20:51
kriskendthen I deleted the instances20:51
kriskendwell restarted nova20:51
kriskendthen deleted20:51
kriskendof course nova didn't work20:52
kriskendthen I fixed pypowervm and restarted nova20:52
thorst_ahhh, nova probably convinced itself it deleted them already20:52
kriskendI thought my instances would get cleaned up at that point... but no20:52
kriskendthey are gone in openstack but still on my compute20:52
thorst_so you already reaped20:53
kriskendsigh, so manual cleanup it is20:53
*** kylek3h has joined #openstack-powervm21:02
*** k0da has joined #openstack-powervm21:17
openstackgerritEric Fried proposed openstack/nova-powervm: devstack INSTALL_PYPOWERVM after nova-powervm
kriskendthorst_ so trying out your close the pipe patch, I deployed 10 vms21:45
kriskend4 fails with checksum errors21:45
kriskendand they all rolled back except for the lv21:45
kriskendcuz I don't have that patch21:45
kriskendwhoops I lied , 6 failed..21:46
kriskendglance seems really unhappy on this system21:46
openstackgerritEric Fried proposed openstack/networking-powervm: devstack INSTALL_PYPOWERVM after networking-pwervm
openstackgerritEric Fried proposed openstack/ceilometer-powervm: devstack INSTALL_PYPOWERVM after ceilometer-pvm
kriskendif glance can really fail this much, should we consider doing retries?21:56
*** svenkat has quit IRC22:00
*** edmondsw has quit IRC22:09
*** apearson has quit IRC22:17
thorst_kriskend: possibly.22:17
*** thorst_ has quit IRC22:18
*** thorst_ has joined #openstack-powervm22:19
*** thorst_ has quit IRC22:24
*** thorst_ has joined #openstack-powervm23:02
*** mdrabe has quit IRC23:03
*** seroyer has quit IRC23:05
*** esberglu has quit IRC23:18
*** esberglu has joined #openstack-powervm23:19
*** kriskend has quit IRC23:19
*** esberglu has quit IRC23:23
*** tjakobs_ has quit IRC23:28

Generated by 2.14.0 by Marius Gedminas - find it at!