Thursday, 2017-02-09

*** edmondsw has joined #openstack-powervm00:03
*** edmondsw has quit IRC00:08
*** openstackgerrit has joined #openstack-powervm01:49
openstackgerritWang Qing wu proposed openstack/nova-powervm master: Refine oslo config help description and code format
*** thorst_ has joined #openstack-powervm01:54
*** thorst_ has quit IRC01:58
*** openstackstatus has quit IRC02:58
*** openstackstatus has joined #openstack-powervm03:01
*** ChanServ sets mode: +v openstackstatus03:01
*** tlian has quit IRC03:04
*** thorst_ has joined #openstack-powervm03:08
*** thorst_ has quit IRC03:08
*** thorst_ has joined #openstack-powervm03:38
*** thorst_ has quit IRC03:38
*** edmondsw has joined #openstack-powervm03:39
*** edmondsw has quit IRC03:44
*** thorst_ has joined #openstack-powervm04:38
*** thorst_ has quit IRC04:44
*** thorst_ has joined #openstack-powervm06:40
*** thorst_ has quit IRC06:45
*** burgerk has joined #openstack-powervm08:26
*** thorst_ has joined #openstack-powervm08:40
*** thorst_ has quit IRC08:46
*** burgerk has quit IRC08:53
*** openstackgerrit has quit IRC09:03
*** k0da has joined #openstack-powervm09:34
*** thorst_ has joined #openstack-powervm10:41
*** thorst_ has quit IRC10:46
*** smatzek has joined #openstack-powervm12:21
*** seroyer has joined #openstack-powervm12:35
*** thorst_ has joined #openstack-powervm12:41
*** thorst_ has quit IRC12:42
*** thorst_ has joined #openstack-powervm12:43
*** jpasqualetto has joined #openstack-powervm13:03
*** openstackgerrit has joined #openstack-powervm13:09
openstackgerritMerged openstack/nova-powervm master: Fix VirtualInterfacePlugException message info
*** svenkat has joined #openstack-powervm13:14
*** xia has joined #openstack-powervm13:15
*** edmondsw has joined #openstack-powervm13:16
*** esberglu has joined #openstack-powervm13:22
esberglu#startmeeting powervm_ci_meeting13:31
openstackMeeting started Thu Feb  9 13:31:38 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:31
openstackThe meeting name has been set to 'powervm_ci_meeting'13:31
*** wangqwsh has joined #openstack-powervm13:32
efried#topic Power-off issues?13:32
efriedSo the good news (esberglu please confirm) is that PS11 of the refactor ran through the CI with no appreciable difference in failure rate.13:34
efriedWhich means it's at least working to the point of backward compatibility, at least in the code paths that are hit by the CI.13:35
efriedWith the caveat that that doesn't include some things, including IBMi.13:35
efriedThe bad news (well, neutral news, really) is that we're still seeing failures.13:35
thorst_still seeing power off failures or general failures?13:35
efriedWhich I really don't want to address in the current patch set, which I'd like to reserve for pure refactor.13:35
efriedpower off failures, at least.13:36
efriedI'm still working on the whole "enumerating all the job options" thing.  Should have a patch set up for a look by lunchtime, I believe.13:36
thorst_OK - just to verify...with the patch, we still see power off failures?13:37
thorst_or just 'failures'13:37
esbergluStill power off failures13:37
efriedBefore we merge that sucker, I would like to run some live tests on IBMi.  nvcastet has volunteered to help me out in some fashion there.  I think by sliding me a disk with an IBMi image on it.13:37
thorst_so efried, was the refactor not supposed to help with power off failures?13:38
*** tblakes has joined #openstack-powervm13:38
thorst_just make it more managable?13:38
efriedJust make it easier to debug and fix said failures.13:38
thorst_confusion cleared.13:38
efriedSo I think once I get the PowerOpts patch up, I'll first investigate those failures and try to put up a separate change set (on top of the refactor) that addresses them.13:39
efriedWith esberglu's handy-dandy live patching gizmo, we ought to be able to run that through the CI fairly easily, yes?13:39
efried#action efried to finish proposing PowerOpts13:39
efried#action efried to investigate power-off failures and propose a fix on top.13:39
efried#action efried to live test on IBMi (and standard, for that matter).13:40
efriedAnything else on the power-off issue for now?13:40
efriedesberglu Other topics?13:41
esberglu#topic CI redeploy13:41
esbergluJust wanted to say that the redeploy finished last night13:41
thorst_jobs going through yet?13:42
esbergluSo now we are running across the board13:42
esbergluI haven't looked at any results yet though13:42
thorst_that's good to know for the CI host server....CPU utilization on that sucker is like 10%13:42
thorst_after we moved everything to the SAN13:42
esbergluThat's all I had for that, just wanted to update13:44
esberglu#topic In Tree CI13:44
esbergluI think we need to talk about how we want to handle moving the in-tree runs from silent to check when we are ready13:45
esbergluBecause if we start posting results, it will fail everything until PS1 is through13:45
esbergluwhich is a lot of red coming from our CI13:45
efriedCan it be as simple as checking for the presence of, say, our
efriedOr do we not know that until too late in the process?13:46
*** xia has quit IRC13:47
efriedI guess we could inspect the commit tree and bail out if we don't see that first change set's commit hash / Change-Id.13:47
thorst_efried: yep13:47
thorst_that's what we should do13:47
thorst_if the commit message (probably?) has the word powervm in it, we publish.13:48
*** xia has joined #openstack-powervm13:48
thorst_and for the oot driver, if the commit message has a set of files from the nova project and contains the word powervm, we should just not run (because we'll fail)13:48
thorst_(due to duplicate options)13:48
efriedOr if the file list in the change set contains '/powervm/'?13:48
efriedWait, why do we need to do something special for OOT?13:49
thorst_The OOT driver will always fail on an IT driver change set.13:49
efriedOh, you mean we don't run the *in-tree* CI on *out-of-tree* patch sets.13:49
thorst_because the OOT driver has duplicate options13:49
efriedGotcha.  So it should be as simple as whether the change set is in the nova-powervm project, neh?13:49
thorst_so if a patch set comes in that is in tree for PowerVM, we should avoid running the OOT driver change13:50
thorst_otherwise we post a +1 and a -1 in the same patch13:50
efriedSorry, yeah, I had it backwards.13:50
thorst_once it merges, we can remove the opts from the oot.13:50
thorst_and be happy again13:50
efriedSo esberglu Do you know how to make all of that happen?13:50
efriedI can help out with the git commands if you need.13:51
efried#action esberglu to set up mutually-exclusive running/publishing of CI results for in- and out-of-tree.13:51
efried#action efried to assist as needed.13:51
efried(that's not going to show up right in the minutes)13:52
esbergluCool. That's all I had for in-tree13:53
esbergluAny other topics?13:53
thorst_I'm assuming that once we get in-tree going, we flip back to ansible CI?13:53
thorst_I know that the openstack-ansible team is still waiting there.13:53
adreznecFYI I discussed that a bit with Jesse last week13:54
thorst_ok - yeah, that was my next question13:54
thorst_do they understand we still are targeting that?13:54
thorst_(seems like they do)13:54
adreznecGave him a bit of status on where we were with CI (the whole in-tree driver, etc)13:54
adreznecYeah, theydo13:54
adreznec*they do13:54
thorst_k.  Assume you'll connect up more at PTG?13:54
adreznecThat was the plan13:54
thorst_that was the only other thing I had13:55
adreznecJust curious - wangqwsh esberglu how much work do you think is left there?13:55
thorst_I know the whole OVS thing needs to be solved...13:56
wangqwshopenstack can be installed via osa, but not run tempest to test it13:56
wangqwshso need to compose some codes for tempest for powervm osa ci13:58
efriedIs that what Nilesh is supposed to be doing?13:59
thorst_Nilesh is supposed to do some tempest tests with it, yeah13:59
thorst_we know that other env's have gotten that running13:59
adreznecRight, you can definitely run tempest against OSA with PowerVM. For the most part it really shouldn't be all that different than running it against a devstack AIO14:00
adreznecSince it's just calling into the APIs14:00
esbergluCool. Sounds like we are starting to get that back on the radar, but we aren't too far away14:04
esbergluAnything else?14:04
wangqwshyes, when we can continue to do for powervm osa ci? after the in-tree ready, right?14:05
*** mdrabe has joined #openstack-powervm14:08
esbergluIf anyone has free cycles they can go for it. I reserved systems for the infrastructure14:08
wangqwsha question related to convert instance's uuid to powervm uuid.14:08
esbergluOtherwise yes, after in-tree14:08
efriedwangqwsh Is that a CI-related question, or should it wait til after the meeting?14:08
wangqwshnot ci question14:09
openstackMeeting ended Thu Feb  9 14:09:46 2017 UTC.  Information about MeetBot at . (v 0.1.4)14:09
openstackMinutes (text):
efriedwangqwsh Go.  Then I've got another topic after that.14:10
wangqwshint(uuid[0], 16) & 714:10
wangqwshi found this code in convert_uuid_to_pvm method14:10
wangqwshwhy do we do this for powervm uuid.14:10
thorst_the powervm UUID format does not match the openstack UUID format14:11
thorst_it is one character difference14:11
thorst_yeah, true.14:11
thorst_net is, there are different UUID formats, and OpenStack's is slightly different from PowerVM's.14:11
*** burgerk has joined #openstack-powervm14:12
wangqwshis that meanings the powervm uuid uses the number in the front of uuid?14:12
thorst_all of the differences between the UUIDs are found in the convert_uuid_to_pvm method14:13
thorst_you can reliably go from an OpenStack UUID -> PowerVM UUID, but not in reverse.14:13
efriedRight, the first bit of the PowerVM UUID has to be zero, so we mask it off.  But going back to OpenStack, there's no way to know if we need to turn that bit back on.14:14
wangqwshthx :)14:14
efriedthorst_ adreznec From the Nova meeting agenda: "Move and/or repropose specs for Pike in the nova-specs repo."  Do we need to take action here?  I see "Series goal: Accepted for pike" on our blueprint page (  Do we need to copy/move to specs/pike?14:14
efriedWe need to get our bp reapproved for pike before they'll lift our -2.  So if we need a new spec before we can ask for that approval...14:15
thorst_I think that we update our review to move to the pike branch14:15
thorst_and that should handle it?14:15
efriedmaster is pike14:16
efriedit's already there.14:16
efriedI asked the question in the nova meeting, and they said we have to get the bp reapproved.14:16
efriedSo I'm asking whether we need to move the spec to specs/pike and get that merged before we can request bp reapproval.14:16
efriedThinking prolly yes.14:16
efriedVolunteers to do that (and give it a re-read to make sure it's still sane)?14:17
*** burgerk has quit IRC14:17
efriedIf not, I can queue that up after all the stuff.14:17
efriedNot a huge hurry, I guess.14:17
* efried is off to get food14:18
thorst_whoops, sorry.14:19
thorst_yeah, I'd say you drive that post power stuff...or mixed in with power stuff while we're waiting for retries.14:19
*** apearson has joined #openstack-powervm14:20
*** smatzek has quit IRC14:20
thorst_adreznec: how big of a disk should I make for Nilesh's new OSA host?14:27
thorst_100 gig or 200?14:27
adreznec100 gig should be fine14:28
thorst_k...I just seem to remember we needed more once upon a time14:28
adreznecUnless we want lots of AIX images14:28
adreznecThen we need more...14:28
adreznecBecause they're so hug14:28
thorst_200 it is.14:29
thorst_8 gb mem good?14:30
thorst_or moere14:30
*** xia has quit IRC14:30
*** xia has joined #openstack-powervm14:31
adreznecThat should be ok for a small environment14:32
thorst_I can always add more later14:32
thorst_disk was more important to get right initially14:32
*** xia has quit IRC14:36
thorst_adreznec: so I'm thinking I still need to get him 3 unique vlans with super short names (ex. br-st?).  Is that still right/14:41
*** nbante has joined #openstack-powervm14:44
*** esberglu has quit IRC14:48
adreznecthorst_: Sorry, yeah, we'll still need VLANs. They don't necessarily have to be unique (if you're short for some reason?) but we'd need unique ranges in each of the 314:48
thorst_well, I'll own the whole network...14:48
adreznecWell then14:49
thorst_so I can make them unique.  The question is how big do they need to be.  I seem to remember I had to use low VLAN numbers (ex. 3, 4, 5) for Ashana because the device names got too long otherwise14:49
thorst_is that still a thing?14:49
adreznecI don't think that the VLANs affect the container networks14:50
adreznecSince the VLAN devices aren't exposed to the containers14:50 you're saying no its not a thing and I have every right to be mad if it becomes a thing14:50
thorst_got it.14:50
adreznecOnly br-mgmt, br-st, etc14:50
* adreznec hides under a rock14:50
thorst_nbante: can you PM me your server that you're using?  I've forgotten which it is, but I'll need to get VLANs strung to it14:56
nbantenovalink IP neo/neo12314:57
nbanteFSP IP :
thorst_nbante: PM means private message in future14:58
thorst_we don't put IPs in here  :-)14:58
nbanteyes..I came to know ..sorry :)14:58
thorst_I'll look at getting it wired in a few hours...need to run to a meeting...14:59
nbantewill take care next times..sure14:59
* adreznec imagines thorst_ out pulling VLANs on the datacenter floor15:00
* thorst_ adreznec knows that I only want to remove wires, not add them (or add VLANs)...yet adreznec still makes me15:00
*** esberglu has joined #openstack-powervm15:03
*** esberglu_ has joined #openstack-powervm15:04
*** esberglu has quit IRC15:07
*** smatzek has joined #openstack-powervm15:10
*** kriskend has joined #openstack-powervm15:11
*** tblakes has quit IRC15:20
*** apearson has quit IRC15:28
*** wangqwsh has quit IRC15:34
*** burgerk has joined #openstack-powervm15:39
*** burgerk_ has joined #openstack-powervm15:39
*** burgerk has quit IRC15:43
*** jpasqualetto has quit IRC15:46
*** efried has quit IRC15:50
*** k0da has quit IRC15:56
esberglu_thorst_: efried: :q15:56
esberglu_I'm not sure if I can just avoid running a test based on if its an in-tree patch15:57
esberglu_I think the run will have to start before I can detect it15:57
thorst_could we add a script that checks the patch we add?15:58
esberglu_And once the run starts, I could check for it, but I'm not sure if I can avoid posting a result15:58
thorst_and bails out or something if its bad?15:58
*** jpasqualetto has joined #openstack-powervm15:59
thorst_well, we definitely need to think through that more...15:59
thorst_adreznec: any thoughts/15:59
esberglu_We could just report the OOT as a success on the IT patches15:59
esberglu_And post some sort of message saying it isn't a true OOT run15:59
thorst_that seems...dubious.16:00
thorst_but maybe sufficient...16:00
thorst_efried would be a good voice to have in that discussion but he's no in IRC atm16:00
adreznecYeah I was worried about this actually working16:02
esberglu_I'm still looking to see if there's a way that I can abort the run and not post any results16:03
*** efried has joined #openstack-powervm16:09
*** tblakes_ has joined #openstack-powervm16:13
*** apearson has joined #openstack-powervm16:14
*** mdrabe has quit IRC16:15
*** tblakes_ is now known as tblakes16:17
*** mdrabe has joined #openstack-powervm16:24
*** esberglu_ has quit IRC16:52
*** esberglu has joined #openstack-powervm16:53
*** esberglu has quit IRC16:57
*** apearson has quit IRC16:59
*** seroyer has quit IRC17:01
*** edmondsw has quit IRC17:03
*** edmondsw_ has joined #openstack-powervm17:03
*** seroyer has joined #openstack-powervm17:15
*** esberglu has joined #openstack-powervm17:16
*** k0da has joined #openstack-powervm17:55
*** nbante has quit IRC17:59
*** tlian has joined #openstack-powervm18:25
*** jpasqualetto has quit IRC18:26
*** tblakes has quit IRC18:37
*** jpasqualetto has joined #openstack-powervm18:39
*** tblakes has joined #openstack-powervm18:43
*** apearson has joined #openstack-powervm18:51
*** esberglu has quit IRC18:53
*** esberglu has joined #openstack-powervm18:53
*** esberglu has quit IRC18:58
*** esberglu has joined #openstack-powervm19:11
*** jpasqualetto_ has joined #openstack-powervm19:26
*** jpasqualetto has quit IRC19:26
*** jpasqualetto has joined #openstack-powervm19:32
esbergluadreznec: efried: thorst_: I think our best bet may be to use a few jenkins plugins.19:33
esberglu Then we can have it make the check and set the build to ABORT19:33
thorst_call the plugins from the job?19:33
thorst_like a script in the job19:33
*** jpasqualetto_ has quit IRC19:36
esbergluWe would just have to add some steps to our job template19:45
esbergluTo call the plugins19:46
esbergluI've got a plan, I'm gonna try it out on staging19:49
thorst_efried: I need you to take a peak at the direction of 483619:50
efriedthorst_ ack19:51
efriedaaaaghhh!  Slot maps!!!19:52
*** tblakes has quit IRC19:52
thorst_its a slot map19:53
thorst_just wait for the corresponding nova-powervm change19:54
thorst_it is accordingly awful as well.19:54
efriedthorst_ Why do we need to do this, if we don't need to do similar for the other UDID-havin things?19:57
efriedSorry, I mean I get storing the path as an extra_spec - I'm talking about the whole lookup-by-extra-spec thing.19:58
efriedI guess I'll see that answer in nova-powervm?19:58
thorst_I can't pre-fetch the UDID of a file19:58
thorst_so I can't look it up via UDID19:58
thorst_because I don't know the UDID19:58
efriedBut why should you need to?19:59
thorst_so that a rebuild op puts the file in the same vscsi slot19:59
thorst_and the order doesn't get screwed up19:59
efriedSo UDIDs of the other types are preserved across migrations (of whatever type) but not of FileIO?20:00
*** jpasqualetto has quit IRC20:00
mdrabethorst_ Is there anything else you _would_ know about the file that could be put in the slot_map extra_spec to identify it?20:01
adreznecI thought UDIDs for fileio WERE going to be preserved across migration as long as the path stayed the same...20:01
efriedMeh, the path should be sufficient, and the requirement would remain the same.20:01
thorst_efried: they may be the same across migrations20:01
thorst_but the problem is, I can't figure out WHAT that UDID even is20:01
thorst_Cinder doesn't give it to me like it would from FC20:01
thorst_afk for a a phone call20:02
mdrabeDoes the file path change?20:04
mdrabeIf it doesn't I think that would be used as the stg_key in the slot_map20:04
thorst_so to be clear...the REST API determines a UDID based off the path20:05
thorst_so the UDID and path shouldn't change20:05
thorst_the problem is Cinder *only* gives me the path20:05
thorst_so when I'm on the can I look it up if I only have the path20:05
thorst_or....I can ask REST what algorithm they use to generate a UDID from the path...and embed that in pypowervm...but then if they ever change that, we're screwed20:06
mdrabeI'm saying20:06
*** tblakes has joined #openstack-powervm20:06
mdrabeWhy do you even need the UDID?20:06
mdrabeIf you know the path at both ends, and that path is unique and guaranteed to stay the same, then you can store the slots keyed off that and lookup on the target just fine20:07
efriedYeah, I see mdrabe's point.  Use the file path as the key, and leave extra_specs blank.20:07
efriedThe only downside I see is that it's not consistent with the other storage types that also inherit from VDisk-ish.20:07
efriedSo does that mean we have to special-case other code to account for that?20:07
thorst_that's why I didn't do that20:09
thorst_because it does in fact have a UDID20:09
thorst_and that UDID gets pulled in via the vscsi mapping register20:09
thorst_see register_vscsi_mapping20:09
thorst_so I'd have to kinda screw up the 'api' (if we can call it that) to fake it out20:09
thorst_and I was trying not to screw up the API even more20:09
efriedmdrabe Thoughts?20:11
mdrabephone call sec20:11
efriedthorst_ Why can we figure out from cinder the UDID for a PV/LU/VDisk, but not for a FileIO?20:12
efriedCause if you're gonna say we look it up from REST, I'm gonna say we should do that for FileIO as well.  You know, to keep the "API" consistent.20:13
*** jpasqualetto has joined #openstack-powervm20:14
thorst_efried: PVs - comes from cinder20:20
thorst_LU's - I don't think I can.20:20
-openstackstatus- NOTICE: Restarting gerrit due to performance problems20:20
thorst_and we don't yet20:20
thorst_vSCSI PV's - I also run the lua_discovery thing that gives it to me20:20
*** jpasqualetto_ has joined #openstack-powervm20:20
*** jpasqualetto_ has quit IRC20:21
thorst_I suppose I could ask REST for an API that - for a given path - gives me the UDID20:21
thorst_that's kinda like a lua discovery....20:21
thorst_and then if they change the algorithm, its all one spot.  No version issues...20:21
thorst_hmm....I like that.20:21
*** jpasqualetto has quit IRC20:24
efriedthorst_ +120:26
thorst_fits better...20:27
efriedmdrabe gets a twinkie20:27
efriedthorst_ New rev up for 4776 (power refactor) including abstractions for all the additional params.20:27
thorst_bah, he's not the one on weight watchers20:27
efriedNo, mdrabe is lean and mean.20:28
mdrabecept it's girl scout cookie season20:28
mdrabeBut yea, I like the REST API doin that. Though I still don't see why the UDID is needed20:29
efriedI'll grant that the other way is "quicker" (can save a REST call), but it's hackier.20:30
*** apearson has quit IRC20:32
efriedthorst_ Sorry, NOW a new rev of 4776 is up.20:33
mdrabeBut efried is storing the file path as the key really hackier?20:36
mdrabeIf you're trying to identify files in an FS, whatsamatter with using the full path20:37
efriedmdrabe That part would be for consistency in the slot manager code.20:50
efriedWhich isn't the be-all-end-all.20:50
efriedBut I think overall it makes the code prettier to do it the other way.20:51
mdrabeI'm slowly convincing myself keying off the path is the way to go20:53
mdrabeThe reason we used UDID for the other storage types was because that's all we had, I think20:54
*** apearson has joined #openstack-powervm21:03
thorst_mdrabe: well, it is the unique identifier as well21:07
mdrabeThe file path might not work actually21:08
mdrabeIf you've got multiple clusters21:09
mdrabeThough I supposed the extra_spec could be used for that21:09
*** tlian has quit IRC21:11
*** seroyer has quit IRC21:23
*** kriskend has quit IRC21:29
*** smatzek has quit IRC21:33
*** seroyer has joined #openstack-powervm21:47
*** thorst_ has quit IRC21:57
*** thorst_ has joined #openstack-powervm21:58
*** svenkat has quit IRC21:59
*** thorst_ has quit IRC22:02
*** smatzek has joined #openstack-powervm22:07
*** smatzek has quit IRC22:16
*** thorst_ has joined #openstack-powervm22:23
*** burgerk__ has joined #openstack-powervm22:23
*** burgerk_ has quit IRC22:26
*** thorst_ has quit IRC22:28
*** burgerk__ has quit IRC22:31
*** tblakes has quit IRC22:48
*** edmondsw_ has quit IRC22:56
*** esberglu has quit IRC23:06
*** mdrabe has quit IRC23:06
*** esberglu has joined #openstack-powervm23:20
*** esberglu has quit IRC23:25
*** apearson has quit IRC23:32
*** seroyer has quit IRC23:34
*** k0da has quit IRC23:46
*** edmondsw has joined #openstack-powervm23:55
*** edmondsw has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!