Thursday, 2017-01-26

*** burgerk_ has quit IRC00:18
*** esberglu has joined #openstack-powervm00:24
*** esberglu has quit IRC00:29
*** edmondsw has joined #openstack-powervm01:12
*** edmondsw has quit IRC01:17
*** thorst_ has joined #openstack-powervm02:01
*** thorst_ has quit IRC02:06
*** thorst_ has joined #openstack-powervm02:50
*** thorst_ has quit IRC02:50
*** thorst_ has joined #openstack-powervm03:17
*** thorst_ has quit IRC03:17
*** apearson has joined #openstack-powervm03:20
*** esberglu has joined #openstack-powervm03:51
*** esberglu has quit IRC03:55
*** Jay1 has joined #openstack-powervm03:57
*** smatzek has quit IRC04:29
*** Jay1 has quit IRC05:08
*** thorst_ has joined #openstack-powervm05:18
*** thorst_ has quit IRC05:23
*** esberglu has joined #openstack-powervm05:40
*** esberglu has quit IRC05:44
*** apearson has quit IRC06:19
*** edmondsw has joined #openstack-powervm07:12
*** edmondsw has quit IRC07:16
*** thorst_ has joined #openstack-powervm07:19
*** AlexeyAbashkin has joined #openstack-powervm07:22
*** thorst_ has quit IRC07:23
*** esberglu has joined #openstack-powervm07:29
*** esberglu has quit IRC07:33
*** thorst_ has joined #openstack-powervm07:34
*** thorst_ has quit IRC07:39
*** AlexeyAbashkin has quit IRC09:10
*** k0da has joined #openstack-powervm09:13
*** thorst_ has joined #openstack-powervm09:35
*** thorst_ has quit IRC09:40
*** thorst_ has joined #openstack-powervm09:51
*** thorst_ has quit IRC09:56
*** esberglu has joined #openstack-powervm10:12
*** esberglu has quit IRC10:17
*** edmondsw has joined #openstack-powervm10:48
*** edmondsw has quit IRC10:52
*** k0da has quit IRC11:13
*** thorst_ has joined #openstack-powervm11:52
*** thorst_ has quit IRC11:57
*** smatzek has joined #openstack-powervm12:20
*** XiaYan has joined #openstack-powervm12:34
*** esberglu has joined #openstack-powervm12:41
*** esberglu has quit IRC12:46
*** thorst_ has joined #openstack-powervm12:47
*** edmondsw has joined #openstack-powervm13:14
*** esberglu has joined #openstack-powervm13:26
esberglu#startmeeting powervm_ci_meeting13:31
openstackMeeting started Thu Jan 26 13:31:10 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:31
openstackThe meeting name has been set to 'powervm_ci_meeting'13:31
esberglu#topic status13:32
esbergluI spent some time yesterday looking through what the OOT CI driver. Other than that I've been working on getting a passing tempest spawn on the IT driver13:34
thorst_how close are we with IT?13:35
esberglu#topic In Tree CI13:37
esbergluI'm at the point where it is requesting a server on the compute api. According to the docs, you can set networks='none'13:37
esbergluand it will create a server without networks13:37
esbergluBut it isn't accepting it as a valid request13:37
esbergluIt might be a version problem13:38
thorst_do you have a pastebin that we can work off of?13:38
thorst_I'd like to see that after meeting13:38
thorst_but that's good to hear that we are that far along  :-)13:38
esbergluAnd hopefully once we get through the rest api it will spawn successfully13:39
thorst_well, it should13:40
thorst_I've done it myself...unless it changed recently13:40
thorst_but that's good news13:40
thorst_good work13:41
esbergluOther than that, just creating the white list for IT13:41
esberglu#action esberglu: Create In Tree Whitelist13:42
esberglu#action: esberglu: Finish the tempest no network spawn test13:42
esbergluI think that's pretty much it for in tree unless you had anything13:42
thorst_not for in tree13:43
esberglu#Topic out of tree driver13:44
esbergluI haven't looked into the failures this morning, but there are a decent amount coming through13:45
thorst_I think I know why13:45
thorst_I think a lot of them are power_off failures.13:45
thorst_good news about that power off change I put in13:45
thorst_it's respecting the timeout values now13:45
thorst_bad news, a lot of images are using a default timeout of 60 seconds13:46
thorst_and I didn't say 'force off after 60 seconds'13:46
thorst_which I apparently should have.13:46
thorst_well, 'force off after timeout'13:46
thorst_I'd like to catch efried today and push that through.13:46
esbergluCool, glad we have a plan on that13:47
thorst_before we'd just wait up to 20 minutes...and it'd usually power off.13:47
thorst_now, that timing window is small and we have pain  :-(13:47
esbergluThat clears up some of the errors.13:49
esbergluBut we are still seing an occaisional "no valid host found"13:49
esbergluOr seeing servers fail to build13:49
esbergluI will go through the failures again today and see what we are hitting again13:50
esbergluThat all I had on OOT13:51
esbergluAny other topics you want to do?13:51
esbergluWant to talk drivers at all?13:51
*** efried has joined #openstack-powervm13:52
efriedDid I miss the CI meeting?13:54
esbergluIts still going if you had topics13:55
efriedLink me to the minutes?13:55
efriedoh, okay.13:55
efriedHave you tried 4754 yet?13:55
esbergluNo I went down the wrong path for a while yesterday trying to get that spawn test working13:56
esbergluAlmost working now13:56
esbergluI then was gonna test that13:56
esbergluTest 4754 that is13:57
efriedcool beans.  Do you need it to be merged before you can try it, or can you pick it up in-flight?13:57
esbergluNo merge needed. I will just do a manual run on the staging env13:57
efriedk.  Let me know if you need any help setting it up.13:57
thorst_sorry - I had someone stop by.  I didn't have anything else13:58
thorst_efried: can you read the eavesdrop logs from about 10-20 minutes ago13:58
thorst_about the power off stuff13:58
efriedThe os_ci_tempest.conf for in-tree use the default base regex (all-inclusive)13:58
*** openstackgerrit has joined #openstack-powervm13:59
efriedthorst_: it stops at 13:4413:59
thorst_alright, I'll catch you up after the nova IRC meeting13:59
efriedSounds good.13:59
thorst_(it'll also be there by then I bet)14:00
efriedSorry, guys - for some reason Thursday mornings always seem to start slow for me.14:00
esbergluNo biggie14:00
openstackMeeting ended Thu Jan 26 14:00:50 2017 UTC.  Information about MeetBot at . (v 0.1.4)14:00
openstackMinutes (text):
efriedWhat's Michael Still's IRC handle?14:01
efriedthorst_: ^14:01
efriedlooks like 'mikal'14:03
*** esberglu has quit IRC14:05
*** XiaYan has quit IRC14:06
*** esberglu has joined #openstack-powervm14:06
efriedoh, I did have a CI topic.  Remind me after the nova meeting.  It goes something like this: Can we eliminate some or all of the tests that don't hit our code?  That would make our CI run (a lot) faster.  Do those tests actually tell us anything?  (Maybe they would theoretically uncover errors running standard in-tree code on Power hardware?)14:06
*** esberglu has quit IRC14:07
*** mdrabe has joined #openstack-powervm14:07
*** esberglu has joined #openstack-powervm14:07
thorst_efried: this is basically the fix I think we need
thorst_for the 'power off' discussion.  Hopefully the commit message explains...but the CI is failing because some power off with a timeout of 60 seconds.  Then we periodically fail there...because the VM may not care.  So I think we need to switch to a force immediate after the timeout has been hit14:08
thorst_it looks like that is what libvirt does as I'm comfortable doing it in our code  :-)14:10
*** esberglu has quit IRC14:14
*** esberglu has joined #openstack-powervm14:16
*** k0da has joined #openstack-powervm14:19
efriedthorst_: The 60s timeout comes down from nova?14:19
efried"libvirt does it" is slightly compelling, but without that, I would be pretty resistant to this idea.14:20
efriedShutting down forcefully can result in data loss.14:20
efriedWe don't care in tempest, but in real life...14:20
efriedWe're basically coding this up so there'll never be a power off failure.14:21
efriedbut by masking the fact that the graceful power off actually did fail.14:21
thorst_and that actually appears to be the design14:21
efriedwhich doesn't seem... ethical.14:21
thorst_I guess if timeout is -1...I could do a forever thing14:21
thorst_that seems more appropriate actually14:21
thorst_let me read the conf prop a little more closely14:22
*** tblakes has joined #openstack-powervm14:22
efriedI wouldn't mind having a little data on how long it typically does take to power off gracefully.14:23
thorst_well, the problem is it matters where you are in the boot cycle14:23
thorst_and then different OSes do things differently too14:23
efriedIf it's, like, 70s, then maybe we respect the timeout with a fudge factor.14:23
thorst_so for our CI, if we're in like a bootp process (cause we have a non-bootable image), it'll never come up14:23
thorst_if its in SMS still (because slow boot), I think its fine14:23
thorst_IBMi I believe is slow14:23
thorst_AIX can be slow, because workloads can say 'hold on'....14:24
thorst_I think...not sure on that?14:24
efriedBut we don't have the ability to know whether we're in OF or SMS - PHYP reports all of that as "on", right?14:24
efriedI guess there's the status LED.14:24
efriedBut I don't relish the thought of coding to the zillion possible values of that guy.14:25
thorst_efried: well, also there's the what do we think we should do, versus what is the OpenStack contract...14:25
thorst_regardless of what we think14:25
efriedDoes power-off accept a timeout from the user, ever?14:25
efriedI've never noticed such an option in horizon, but from the CLI?14:25
thorst_its set in the image.14:25
thorst_or defaults to this value.14:26
thorst_the images in tempest just use the 60s14:26
efriedOkay, the way I read that doc, it does seem to imply that you're supposed to pull the plug if it's not down by the timeout.14:27
thorst_(I actually thought that was how I had it originally, but forgot that new Force enum)14:27
efriedPutting the onus on the user to set a reasonable timeout in their image metadata seems like a pretty heavy burden.14:28
efriedBut if that's the OpenStack contract... I guess I'm okay with it.14:28
thorst_yeah, I agree.14:28
thorst_I think for our typical workloads, we'd generally suggest that be higher.14:28
thorst_want to give those db's time to gracefully shut down14:29
efriedIs that something we should elucidate in our devref?14:29
efriedOr just count on the user to know their OpenStack esoterica?14:29
thorst_I think its really up to the distro...14:29
thorst_but I'm kinda iffy on it.14:30
thorst_I don't think I want to put it in...lets wait for someone to correct us on that...14:30
thorst_(if we're in fact wrong)14:30
efriedThe doc implies -1 isn't a valid value.14:30
efriedIn fact, even zero doesn't appear to be valid.14:31
thorst_so I *can't* do an if -1 go for a while14:31
efriedSo actually, the condition you have for setting the force flag will never work.14:32
efriedUnless there's some other way that value gets set within nova, that ignores/overrides the config opt.14:32
*** smatzek has quit IRC14:33
thorst_well, if they tell me to force it, it doesn't matter the timeout14:34
thorst_I just rip it I thought14:34
efriedIf *who* tells you to force it?14:35
efriedThe power_off API doesn't have a separate force option.14:35
efriedI guess that must be the answer to my above question.14:35
efriedIf you force power off from the API, it sends you a zero, regardless of the conf option.14:35
efriedSo - I think I'm good with your change.14:36
efriedLet me scrutinize it for spelling mistakes ;-)14:36
thorst_and definitely apostrophese14:36
efriedNope, looks good to me.  Let's wait for the PowerVM CI before merging, tho.14:37
thorst_efried: agree14:38
thorst_give it your +2 though, so I can just W+1 it when that comes in14:38
thorst_don't worry, I am ethical enough not to push it through until I get the CI +1'd14:38
thorst_also, I responded to your comments back in the fileio patch14:39
thorst_I did not take any action in the change set (after rewriting that one section, only to find out it doesn't matter)14:39
*** esberglu has quit IRC14:41
efriedthorst_: saw that, agree with your reasoning.  Gave +1, will promote once csky and gfm do the same (since they had comments earlier).14:43
thorst_I thought they did on previous patch and this was really just a rebase14:43
efriedI don't see +1s from them.14:43
thorst_o bleh, that was the iscsi one14:44
thorst_let me go pester them14:44
efriedMy gerrit views are out of control.14:53
efriedthorst_: can you push this one?
efriedand this one
efriedI'll follow up with more cleanup change sets if needed.14:54
openstackgerritMerged openstack/networking-powervm: Move deprecated pci_passthrough_whitelist
*** tjakobs has joined #openstack-powervm15:04
*** smatzek has joined #openstack-powervm15:09
*** esberglu has joined #openstack-powervm15:09
*** mdrabe has quit IRC15:33
*** burgerk has joined #openstack-powervm15:35
*** mdrabe has joined #openstack-powervm15:43
*** openstackgerrit has quit IRC15:48
thorst_power off for master branch should be going through now16:02
thorst_hopefully that leads to more stable runs this afternoon16:02
efriedthorst_: gave +2 to File I/O.16:03
efriedthorst_: Got time for ?16:03
thorst_yeah, I also want to peak at that OVS one from qingwu16:03
efriedesberglu: Did you determine for sure that and its brethren are not needed?16:04
*** openstackgerrit has joined #openstack-powervm16:06
openstackgerritMerged openstack/nova-powervm: Force immediate on failure
esbergluI will keep an eye on CI with that force immediate merged16:13
esbergluefried: It isn't needed for our CI. But if someone else was trying to use a custom pypowervm they would need a workaround.16:13
esbergluBut those patches didn't work when I tested them all concurrently.16:13
efriedI'll abandon for now, I guess, unless adreznec says otherwise.16:14
efriedthorst_: is ever likely to be resurrected?  I can't remember where we stand on the whole glance thing.16:15
adreznecefried: Nope, I don't think we need them if they're not needed for CI16:16
efriedk, abandoned, thx16:16
adreznecThe only people who'd be likely to need custom pypowervm is us16:16
efriedthorst_: How is configdrive.required_by(instance) normally set?  I'm testing out the cfg_drv change set and that condition doesn't seem to be firing.  I can force it with the conf option, but is that how we usually do it?16:20
thorst_efried: that's honestly how I've always done it16:20
thorst_I think you can also set image metadata to flag it16:21
thorst_but I just do the conf16:21
efriedokay.  I've never noticed that.16:21
thorst_I *think* that's what most deploys do too...16:21
openstackgerritMerged openstack/nova-powervm: Mock data cleanup 1
efriedthorst_: cfg_drv isn't working because the code path "requires" the mgmt_cna.16:31
efriedFirst of all, I'm assuming we can't do mgmt_cna yet because no network?16:31
efriedSecond, assuming #1 is true, can we just rip out the mgmt_cna from that code path, or do we actually need it in order to build the cfg drv?16:31
thorst_mgmt_cna is separate from networking16:36
thorst_it is an additonal network16:36
thorst_but cfg drive shouldn't REQUIRE the mgmt cna16:36
efriedOkay, so for now, rip it out?16:37
thorst_there is in fact a config option to turn off mgmt cna altogether...16:37
thorst_rip it out from what?16:37
thorst_which patch16:37
thorst_yeah, that should be something we add in a separate change set16:39
thorst_right now qingwu has the mgmt cna (and some vnic stuff....) in the OVS patch16:39
thorst_maybe we need the mgmt CNA to be a patch on top of the OVS one16:39
thorst_because it extends into multiple things16:39
openstackgerritMatt Rabe proposed openstack/nova-powervm: Add vopt removal params to the power on job in spawn
openstackgerritEric Fried proposed openstack/networking-powervm: Use neutron-lib portbindings api-def
efriedthorst_: adreznec: ^^ I didn't hear from boden, so I went ahead and made the changes.  I reckon we're in the driver's seat on this one - merge whenever we're ready.16:46
adreznecHmm ok16:46
thorst_sounds good16:47
*** apearson has joined #openstack-powervm16:47
esbergluthorst_: Pretty much hitting the same failures today as yesterday. I will update again once some runs go through with the force immediate patch16:53
esbergluDid we decide we were okay with disabling the test with the security group in use error?16:53
openstackgerritMatt Rabe proposed openstack/nova-powervm: Add vopt removal params to the power on job in spawn
thorst_I'm OK with it.  efried may want to weigh in...but I'd prefer to do it17:06
efriedLike I said yesterday, I would like to know more about why this fails.  If it's our concurrency level or it fails for everyone like this, community bug.  Otherwise, we need to look at why it fails with our code and whether _we_ have a bug.17:08
thorst_so maybe some research quick on what tests were running when that failed17:11
thorst_looking at the n-cpu log17:11
efriedFor now I guess I'm okay putting it in the part of the skip list that says, "We're skipping this because it fails, but we need to investigate why"17:11
efriedBut we need to be careful that bit doesn't grow too large or get stale.17:11
thorst_yeah, figure it out in staging17:12
thorst_rather than hold production hostage17:12
efriedthorst_: community code uses shared procs always?17:18
efriedand uncapped17:18
thorst_yeah, openstack code always runs that way.  It's a shared system, so we do that.  I think there was an extra specs thing to override though, but that would be the controversial bit17:24
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Experimental: File I/O Volume Driver Implementation
thorst_adreznec: efried: if good with that commit update, can we then push through?17:40
efriedthorst_: +217:41
efriedthorst_: adreznec: - added detail on those two proc tunables.  See whatcha think.17:47
mdrabeesberglu: Is the CI vote gating on all nova-powervm changes?17:49
thorst_mdrabe: yes17:50
thorst_all of them.17:50
thorst_won't get in unless it passes  :-)17:50
mdrabek cool17:50
thorst_it's amazing  :-)17:50
mdrabeIt is, also means I don't have to put "DNM yet" in the vopt removal change17:51
efriedmdrabe: Do you have the pypowervm change set up yet?  I don't see it.17:59
mdrabeclbush already merged it18:00
efriedthanks, I see it now.18:01
thorst_efried: I think it has NL updates too18:01
thorst_so the CI systems would need to be updated18:01
efriedthorst_: - updated to remove mgmt_cna stuff, tested, works.18:24
efriedAlso rebased 'em all.18:25
efriedthorst_ adreznec esberglu: Follow up from earlier18:25
efriedCan we eliminate some or all of the tests that don't hit our code?  That would make our CI run (a lot) faster.  Do those tests actually tell us anything?  (Maybe they would theoretically uncover errors running standard in-tree code on Power hardware?)18:25
thorst_efried: adreznec and I are in a meeting  :-(18:26
efriedFine, be that way.18:26
*** tblakes has quit IRC18:30
*** tblakes has joined #openstack-powervm18:52
thorst_all of the latest jobs have failed the CI19:01
thorst_ruh roh19:01
openstackgerritMatt Rabe proposed openstack/nova-powervm: Add vopt removal params to the power on job in spawn
openstackgerritTaylor Jakobson proposed openstack/nova-powervm: Add image cache to nova-powervm
*** tblakes has quit IRC19:10
esbergluthorst_: Just got back from lunch and saw the CI. Looks like they aren't even stacking. Looking into logs and checking devstack commits19:15
thorst_OK - if we need to turn something off...we should until we get it sorted19:16
thorst_or at least turning off the voting19:16
thorst_not voting...but basically red showing up from our CI on nova patches19:16
*** tblakes has joined #openstack-powervm19:16
esbergluthorst_: Most of those failures were expected19:40
esberglu7 failures were the in tree driver patches19:40
esberglu2 WIP patches that are failing everything19:40
esberglu3 failing haven't rebased with the force immediate19:40
esbergluAnd the 4 have failed with tests that were in that list yesterday19:41
openstackgerritMerged openstack/nova-powervm: Experimental: File I/O Volume Driver Implementation
thorst_esberglu: phew!20:31
thorst_good analysis20:31
*** apearson has quit IRC20:35
efriedesberglu: is on your radar?  What do we need to do to get CI passing here?20:43
efried( adreznec ^^ )20:43
adreznecefried: esberglu I think we'd need a CI local.conf change20:56
adreznecFor the dsvm AIOs20:56
esbergluYeah to the local.conf.aio20:56
adreznecAdding enable_service pvm-q-sea-agt to it20:56
adreznecI think we could add that and it wouldn't impact anything20:56
adreznecThen recheck, and then merge things20:56
adreznecSince enabling the service twice should end up with the second being a no-op20:57
esbergluYeah. It will take a redeploy of the CI though. Or we can put it in and wait for the next image template to pick it up20:58
adreznecesberglu: Have a preference on the approach?20:59
esbergluI vote for the latter and just wait to merge the change until we know that those line are in the local.conf on the image template21:00
*** apearson has joined #openstack-powervm21:02
esbergluthorst_: adreznec: efried: I am gonna tag the *-powervm projects for the ocata 3 milestone today21:06
efriedThis means we'll have a stable/ocata branch and have to do all the backporty stuff?21:07
thorst_let me merge one thing quick21:07
thorst_o wait, nm...its in21:07
thorst_rip it!21:07
adreznecWhich thing21:07
adreznecfile i/o?21:08
adreznecyeah I just merged it21:08
adreznecFigured you wouldn't care21:08
thorst_I was wondering how that merged21:08
esbergluefried: Nope, that's next thursday (release candidate 1). Ocata 3 is supposed to signify the end of new feature development21:10
esbergluAnd then we tag the first release candidate next thursday and create the stable/ocata branch21:11
esbergluAnd start doing backports if needed21:11
esbergluAnything else people want to get in before the tag?21:13
*** edmondsw_ has joined #openstack-powervm21:18
*** edmondsw_ has quit IRC21:18
openstackgerritMerged openstack/nova-powervm: Add image cache to nova-powervm
*** smatzek has quit IRC21:31
*** kriskend has joined #openstack-powervm21:39
*** tblakes has quit IRC21:51
*** tblakes has joined #openstack-powervm21:56
efriedesberglu adreznec thorst_ - Was about to rebase 4757 on top of 4754; but maybe y'all want to do it the other way around?22:00
efriedAre we delaying 4754 for some reason?22:00
efriedIt is backward compatible for OOT, so we oughtta be able to merge it and keep on keepin on.22:00
esbergluYeah. You tested it with no whitelist to confirm that OOT would work?22:01
esbergluCool. +1 from me, I'll let the others look at it one more time22:03
*** smatzek has joined #openstack-powervm22:04
adreznecefried: No reason I can think of, I'll review that next22:04
*** smatzek has quit IRC22:06
*** smatzek has joined #openstack-powervm22:06
efriedesberglu - I did the rebase and the skip list mod on 475722:06
esbergluAwesome thanks22:07
efriedyahyoubetcha.  Only problem is, now we have to get someone else to review it ;-)22:07
*** thorst_ has quit IRC22:15
*** apearson has quit IRC22:16
*** edmondsw has quit IRC22:29
*** burgerk has quit IRC22:36
*** thorst_ has joined #openstack-powervm22:47
*** thorst_ has quit IRC22:51
*** mdrabe has quit IRC22:57
*** kriskend has quit IRC22:59
*** kriskend has joined #openstack-powervm22:59
*** dwayne has quit IRC23:04
*** tlian has quit IRC23:13
*** smatzek has quit IRC23:16
*** tlian has joined #openstack-powervm23:21
*** tblakes has quit IRC23:21
*** esberglu has quit IRC23:23
*** esberglu has joined #openstack-powervm23:25
*** esberglu has quit IRC23:29
*** kriskend has quit IRC23:30
*** tlian has quit IRC23:32

Generated by 2.14.0 by Marius Gedminas - find it at!