14:06:23 <adreznec> #startmeeting PowerVM Driver Meeting
14:06:23 <openstack> Meeting started Tue Nov 15 14:06:23 2016 UTC and is due to finish in 60 minutes.  The chair is adreznec. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:06:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:06:27 <openstack> The meeting name has been set to 'powervm_driver_meeting'
14:06:34 <adreznec> Helps if I use the right command
14:06:52 <adreznec> #topic Current status
14:07:19 <adreznec> All right, so let's start with the upstream driver related topic
14:07:44 <adreznec> Now that the blueprint is approved/spec merged, efried1 has taken the WIP off all of this change sets
14:08:10 <thorst> wooo
14:08:29 <adreznec> Our global requirements patch for pypowervm has a +1 from Jenkins, but needs responses to the requirements team questions
14:08:30 <thorst> and our very sensible version number is going splendidly
14:08:36 <adreznec> Yes
14:08:44 <adreznec> We'll call it semver++
14:08:50 <adreznec> One + for each additional digit
14:09:00 <efried1> For the record, I tried to produce the upper-constraints patch using the process outlined in the README.  Never got it to work.
14:09:03 * adreznec grumbles off in the background
14:09:09 <efried1> Hence typo
14:09:31 <efried1> Didn't figure it was rocket science.  And it's not.  I'm just a doofus.
14:09:49 <efried1> I'll take care of that questionnaire this morning.
14:10:00 <adreznec> Was an easy fix anyway, and it means our very excellent versioning scheme can stay intact
14:10:01 <efried1> #action efried to update requirements change set per comments.
14:10:19 <efried1> #action efried1 to update requirements change set per comments.
14:10:30 <efried1> (I guess I'm efried1 since the last time my laptop crashed)
14:10:38 <thorst> I can't wait for efried2
14:10:48 <efried1> Me neither.
14:10:49 <efried1> What's next.
14:10:59 <adreznec> Do we need action on any of the other patches yet?
14:11:00 <thorst> CI stability?
14:11:12 <efried1> oh, hold on,
14:11:21 <efried1> let's talk a little bit about Tony's comment.
14:11:38 <efried1> Do we respond to it in the change set (and attempt to explain the IBM legal reasons we don't use semver)?
14:12:12 <adreznec> I'm... not sure it matters either way?
14:12:18 <efried1> Or maybe something flippant like, "yeah, we don't like it either, but there's a good (if boring) reason for it"
14:12:21 <efried1> or just ignore it.
14:12:29 <adreznec> I mean it's definitely awful
14:12:43 <adreznec> But I'm not sure it's actually a big deal to anyone at the end of the day
14:12:55 <adreznec> Besides me
14:13:00 <efried1> I'm just talking about for the sake of responding to Tony's comment.
14:13:51 <efried1> <silence>
14:13:54 <adreznec> I don't think it really matters as long as we can answer all the other questions
14:13:59 <efried1> Roger that.
14:14:05 <adreznec> From the questionnaire
14:14:06 <thorst> yeah, I'm not sure its a big deal.  I'm OK saying 'we don't really like it either'...but I'm not sure we need to make it a big deal
14:14:10 <adreznec> Right
14:14:11 <thorst> I think its a bigger deal to us than them
14:14:35 <efried1> The other thing on current patch sets: do we want to preserve the nova-powervm versioning history on the live migrate data object, or start it at 1.0 in nova proper?
14:14:42 <adreznec> We can just say Watson designated the version
14:14:44 <adreznec> And leave it at that
14:14:47 <adreznec> :P
14:14:48 <efried1> (see comment: https://review.openstack.org/#/c/391284/3/nova/objects/migrate_data.py@301
14:14:49 <efried1> )
14:15:35 <thorst> efried1: I can respond there if you'd like
14:15:46 <thorst> I'm sure they don't know the history of us being out of tree
14:15:47 <thorst> but that's why
14:16:05 <efried1> So we do want to preserve our out-of-tree version history.
14:16:41 <thorst> it simplifies the transition
14:16:53 <efried1> Right, for people currently using the out-of-tree driver.
14:16:54 <thorst> for operators
14:16:55 <adreznec> To an extent, yeah
14:17:10 <adreznec> I guess it depends on if we ever want to support anyone migrating from the out-of-tree driver to the in-tree driver
14:17:26 <efried1> Heck, maybe we should even bump it up to 1.2
14:17:32 <efried1> or 1.1.0.0.9
14:17:45 <adreznec> +2
14:18:38 <adreznec> I guess we can respond with a comment saying we'd like to preserve it for ops
14:18:52 <adreznec> And if they push back we can talk about the implications of that
14:19:28 <adreznec> Seem reasonable thorst efried1?
14:19:46 <thorst> yep
14:19:55 <efried1> So
14:19:55 <efried1> #action thorst to respond to https://review.openstack.org/#/c/391284/3/nova/objects/migrate_data.py@301
14:19:55 <efried1> ?
14:20:03 <adreznec> So it would seem
14:20:22 <adreznec> All right
14:20:25 <efried1> Okay.  The last thing would be socializing the driver skeleton change set.
14:20:37 <efried1> I'm still not clear how that's supposed to happen normally.
14:21:04 <efried1> Guess we're not in a super hurry to get that merged until we have CI stable.  Next #topic?
14:21:39 <adreznec> So I think we need to talk with the Nova cores about whether we should be on that subteam priority reviews list now
14:21:57 <thorst> sorry - afk for a bit.
14:22:07 <efried1> adreznec, I don't know what that is.
14:22:14 <adreznec> and list our 'top 5' reviews there like the other driver teams are
14:22:20 <adreznec> Let me dig that up quick
14:23:05 <adreznec> Like this: https://etherpad.openstack.org/p/ocata-nova-priorities-tracking
14:24:08 <efried1> Oh, for sure.
14:24:10 <adreznec> Basically whether or not we should have a PowerVM Subteam section on there where we start listing these reviews
14:24:29 <adreznec> We should bring this up in the next Nova meeting then
14:24:39 <efried1> Sounds like a good plan.
14:25:01 <adreznec> #action adreznec thorst and efried1 to bring up PowerVM subteam/reviews in the next Nova meeting
14:25:12 <adreznec> Okay
14:25:26 <adreznec> #topic Priority reviews
14:25:44 <adreznec> Do we have anything open that needs code review priority?
14:26:01 <efried1> There's nothing on my radar in the community.
14:26:10 <adreznec> FYI for the team - until https://review.openstack.org/#/c/396934/ merges, multi-architecture deploys are broken
14:26:20 <adreznec> But that already has +2/+1s this morning
14:26:25 <adreznec> So just awareness
14:26:41 <adreznec> All right
14:26:50 <adreznec> Sounds like nothing high priority
14:27:05 <adreznec> #topic Priority bugs/defects
14:27:24 <adreznec> Same thing here - any key issues people want to direct attention to?
14:27:48 <efried1> Other than the CI, nothing I'm aware of.  thorst may have something.
14:28:24 <efried1> Can we move on to CI?  That could be a substantial discussion.
14:28:27 <adreznec> Yep
14:28:31 <esberglu> Ok
14:28:33 <adreznec> #topic CI discussion
14:28:43 <adreznec> esberglu: efried1, it's all yours
14:28:55 <efried1> esberglu, what's the latest?
14:29:09 <esberglu> So amartey thinks that upgrading novalink *may* fix the LUs hanging around after the runs
14:29:24 <esberglu> #action esberglu update novalink in CI env
14:29:31 <esberglu> As far as the actual runs
14:29:44 <adreznec> Upgrading it to what?
14:29:54 <adreznec> Latest 1.0.0.4? Latest dev?
14:30:25 <wangqwsh> however, i found 183 can not retrieve vm info
14:30:41 <esberglu> Sorry I got disconnected. Latest 1.0.0.4
14:30:52 <adreznec> Ok
14:31:16 <adreznec> Well I guess we can give it a shot
14:31:16 <esberglu> I put the latest versions of the patches from efried on yesterday afternoon
14:31:29 <esberglu> But still not working
14:31:43 <esberglu> I let one run through last night
14:31:47 <esberglu> here are the logs
14:31:48 <esberglu> http://184.172.12.213/15/328315/19/check/nova-powervm-pvm-dsvm-tempest-full/9564270/
14:32:00 <esberglu> Still not finding any valid hosts for many of the tests
14:32:10 <esberglu> I haven't dug into the logs yet from that run
14:33:50 <efried1> #action efried1 esberglu to analyze http://184.172.12.213/15/328315/19/check/nova-powervm-pvm-dsvm-tempest-full/9564270/
14:34:10 <efried1> We may be at a point where we should put all hands on deck until the CI is stabilized.
14:34:56 <efried1> Throwing more people at it doesn't help the runs themselves finish faster.  But having all eyes on the results right away...
14:35:45 <adreznec> Yeah, I know people are getting pulled in a number of directions right now
14:35:52 <adreznec> Maybe even just a one day sprint
14:36:05 <adreznec> Focused on CI improvements could help
14:36:45 <adreznec> Unfortunately the holiday season is starting to creep up on us
14:36:54 <adreznec> So finding availability could be tricky
14:37:38 <esberglu> Yeah I'm out starting friday through all of thanksgiving week. No internet Friday to Wednesday
14:38:33 <adreznec> All right. I guess we'll have to table that discussion for now and discuss it outside the meeting
14:38:53 <adreznec> So what else do we have on CI?
14:39:11 <efried1> Seems service nova-compute on host powervm-ci-powervm-devstacked-25844 is down. Last heartbeat was 2016-11-15 01:54:39. Elapsed time is 126.14957
14:39:19 <efried1> Looks like the compute service died at some point in there.
14:39:49 <thorst> btw - I 100% agree that we need an all hands on deck for CI
14:39:51 <efried1> Compute log has no interrupt or anything - it just stops.
14:39:52 <adreznec> Well that would certainly explain issues finding a host
14:40:02 <thorst> does it just stop because we hit 5 hours?
14:40:05 <thorst> and zuul just shut it down
14:40:13 <esberglu> No the run went through in like 40 minutes
14:40:21 <esberglu> It went super fast because it couldn't find any hosts
14:40:36 <thorst> o
14:40:37 <thorst> huh
14:40:38 <efried1> 2016-11-14 19:54:43.560 is the last time stamp in the compute log.
14:41:40 <esberglu> The 5 hour runs happen when it times out waiting for hosts to become active (which we are past at this point I believe)
14:42:26 <esberglu> So should I let another run through? Maybe the host dying was just a one-off thing?
14:42:58 <thorst> esberglu: the 5 hour thing happens when we wait for the marker LU
14:43:04 <thorst> which we *may* be past, hopefully
14:43:12 <thorst> just needed an uber clean of the env
14:43:26 <esberglu> Yeah and efrieds latest patch
14:43:43 <thorst> right.
14:43:50 <thorst> we also need the novalinks updated
14:43:55 <thorst> did you see that from apearson?
14:44:00 <esberglu> Yep
14:44:05 <esberglu> already did an action for it
14:44:08 <thorst> so are those the immediate next steps?
14:44:17 <adreznec> Yep
14:44:24 <adreznec> Already had the novalink upgrade as an action
14:44:50 <thorst> then all hands on deck for the next run to debug
14:45:15 <adreznec> #action esberglu to try another CI run with latest patch and report results
14:45:37 <adreznec> Ok
14:45:46 <adreznec> Anything else on the CI front?
14:45:58 <esberglu> I think wangqwsh was hitting some OSA CI stuff?
14:46:04 <wangqwsh> yes,
14:46:32 <wangqwsh> the memory is extended to 12g, galera can work.
14:46:44 <wangqwsh> but it's failed to boot a vm.
14:46:57 <wangqwsh> i am trying to understand drew's mail
14:48:04 <wangqwsh> do not know how to set the network on neo14...
14:48:30 <adreznec> Ah, this was the OVS network issues email
14:49:42 <wangqwsh> yes.
14:50:08 <thorst> I kind of want to wait on the OVS bits until we have CI stabilized
14:50:15 <thorst> we need our core CI working first...
14:50:17 <adreznec> thorst probably has a more detailed description than this, but basically where we're at now (IIRC) is that we need to create a PHYP vswitch for each tempest VM running OSA, then put the OVS trunk on that
14:50:18 <adreznec> Right
14:50:29 <thorst> adreznec is right.
14:51:04 <adreznec> thorst: yeah, I agree with you
14:51:15 <wangqwsh> ok
14:51:28 <adreznec> we should probably put the OSA CI work on temporary hold while we all debug the main CI issues
14:52:18 <adreznec> thorst: I will note that on the patch Kyle put up to fix the OSA multi-arch stuff that got broken, Jesse did comment about investigating a multi-arch CI test
14:52:33 <adreznec> Nothing we need to do now
14:52:40 <adreznec> But something for the back burner
14:53:28 <adreznec> So, CI stability is going to take priority
14:53:48 <adreznec> Do we have anything else to discuss here? Or are we just on deck for the next set of results?
14:54:37 <adreznec> Ok, well if there's nothing else I'm going to call it
14:54:43 <adreznec> Thanks for joining everyone
14:54:58 <adreznec> #endmeeting