14:08:38 <esberglu> #startmeeting powervm_driver_meeting
14:08:39 <openstack> Meeting started Tue Nov 29 14:08:38 2016 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:08:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:08:42 <openstack> The meeting name has been set to 'powervm_driver_meeting'
14:08:43 <adreznec> Yep
14:08:45 <adreznec> That was it
14:08:45 <adreznec> :P
14:09:01 <thorst_> o/
14:09:07 <thorst_> efried is off today I think
14:09:33 <esberglu> #topic Status
14:10:32 <esberglu> The latest CI run went through last night with only 6 failures
14:10:49 <thorst_> I'm reviewing those.  Some seem like legit nova failures
14:11:18 <esberglu> Run in question if anyone needs it
14:11:20 <esberglu> http://184.172.12.213/34/403934/1/check/nova-powervm-pvm-dsvm-tempest-full/57319c1/
14:11:37 <esberglu> It looked like most of them were the same failure
14:11:52 <esberglu> I haven't looked into the logs yet though
14:12:02 <adreznec> Ah ok
14:12:08 <adreznec> Was going to ask if any were new tests
14:12:21 <esberglu> Yeah one of them is a new one that I need to disable
14:12:26 <adreznec> Were the runs that failed all against master?
14:12:59 <esberglu> Yeah
14:13:00 <thorst_> yeah, this is a nova failure (at least one of them)
14:13:07 <thorst_> I'll go investigate that.
14:13:11 <thorst_> propose something up...
14:13:22 <thorst_> o wait...no, this is on us
14:13:25 <esberglu> #action esberglu: Disable unsupported test
14:13:43 <thorst_> #action thorst: fix confirm_migration in nova_powervm
14:14:15 <adreznec> test_delete_server seems like one that should pass...
14:14:19 <adreznec> Unless the name is bad
14:14:48 <thorst_> yeah, they're failing because of a bug
14:14:52 <thorst_> they changed a driver signature
14:14:56 <adreznec> Ah
14:14:59 <thorst_> we didn't notice (no CI  :-)
14:15:09 <adreznec> Yeah
14:15:20 <adreznec> Sorry, was still unzipping the n-cpu log
14:15:22 <thorst_> nothing to worry about
14:15:26 <thorst_> I got it
14:15:30 <adreznec> Ok
14:15:42 <thorst_> hooray for CI though
14:16:42 <thorst_> so, next steps are to get that fixed...see if we can't get a few solid runs through...
14:16:45 <thorst_> then party?
14:16:53 <adreznec> You're sure this delete one is related? The 404 back from REST?
14:17:02 <esberglu> Yeah I think that's a different issue
14:17:18 <thorst_> maybe not...but the tempest-DeleteServersTestJSON hit this
14:17:25 <thorst_> TypeError: confirm_migration() takes exactly 4 arguments (5 given)
14:17:36 <thorst_> and I see that 4 times in the logs
14:17:43 <thorst_> each on different instances.
14:18:00 <esberglu> And the 5th failure is the one I need to disable
14:18:03 <thorst_> so its at least one of the things.
14:18:14 <adreznec> Right
14:18:17 <esberglu> So all we have left is the first test_delete_server failure
14:18:19 <adreznec> Ok, lets start there then I guess
14:18:27 <esberglu> Which is something about a security group in use
14:19:57 <thorst_> I wonder if that is a side effect of these other servers still existing though
14:20:02 <thorst_> cause we failed to delete them
14:20:12 <thorst_> and its just one of those concurrency things
14:20:52 <openstackgerrit> Drew Thorstensen (thorst) proposed openstack/nova-powervm: Update to new confirm_migration signature  https://review.openstack.org/404234
14:21:27 <esberglu> Could be. Btw did you guys see that change for the arch import statement?
14:22:03 <esberglu> Well at least thorst_ did
14:22:12 <adreznec> I saw the review come in, didn't have a chance to look at it yet
14:22:17 <thorst_> yeah, I'm ready to W+1 that
14:22:25 <thorst_> adreznec: should take 2 seconds and would unwedge
14:22:53 <esberglu> https://review.openstack.org/#/c/403934/
14:23:21 <esberglu> And that's the patch this CI run is from
14:23:27 <adreznec> lgtm
14:25:11 <thorst_> so what's next?
14:25:27 <thorst_> I propose that we update the requests library in the tempest VMs, the novalinks, and the undercloud
14:25:54 <thorst_> because efried told me we saw some of those requests bugs in a few runs a while back...so I'd like to squash that completely.
14:26:12 <adreznec> thorst_: do we have enough confidence that's solving real issues?
14:26:26 <adreznec> If so, we should probably get our dependency updated
14:26:27 <thorst_> adreznec: I think it is several issues
14:26:36 <adreznec> On requests >= 2.12.1
14:26:38 <thorst_> we have a fix from efried that we need to finish
14:26:42 <thorst_> that's solid
14:26:43 <adreznec> I was going to test that out today
14:26:52 <adreznec> Make sure it doesn't break anything
14:27:02 <adreznec> If it doesn't I'll put up a nova-powervm req change for it
14:27:09 <thorst_> the requests >= 2.12.1 has worked much better for kriskend's three runs of 10 deploys
14:27:28 <thorst_> adreznec: do we want it in nova-powervm or just install it and instead pursue with glanceclient?
14:27:52 <adreznec> We could try
14:27:52 <thorst_> this is one of those situations where its not us...
14:28:04 <adreznec> That would mean a g-r bump probably
14:28:13 <adreznec> So it could take while to filter through
14:28:14 <thorst_> or a g-r exclude
14:28:19 <thorst_> right
14:28:29 <thorst_> but I think that's the right approach
14:28:32 <adreznec> Do we think this is an issue specifically with 2.11.1?
14:28:39 <thorst_> unclear
14:28:47 <thorst_> that's definitely what people have when we hit the issue
14:28:49 <adreznec> Ok... probably need testing there then
14:28:59 <adreznec> To confirm if we need an exclude or a minimum bump
14:29:24 <thorst_> yeah.  I'd say for now we just add it to the CI env and workaround from there
14:29:27 <openstackgerrit> Merged openstack/nova-powervm: Change arch to fields.Architecture  https://review.openstack.org/403934
14:29:33 <adreznec> Sure
14:29:38 <thorst_> actually
14:29:44 <thorst_> upper-constraints will kill us there
14:29:51 <thorst_> they have it tagged to 2.11.1
14:30:08 <thorst_> so even if we put it in the CI, devstack will detect that and then update it
14:30:14 <thorst_> so I think we need to have a g-r debate.
14:30:37 <thorst_> #action thorst to open bug to get upper-constraints updated for requests package
14:31:13 <adreznec> Fun fun
14:31:19 <thorst_> moving on   :-D
14:31:20 <esberglu> I could remove the upper in the same way that we are removing the pypowervm constraints temporarily
14:31:35 <thorst_> esberglu: lets wait it out and see if its bad
14:31:39 <esberglu> Ok
14:31:48 <thorst_> I'd rather go through formal process
14:32:34 <thorst_> any other items to discuss?  I'm assuming OSA progress has been limited due to Thanksgiving and then focus on stablizing core CI?
14:32:52 <esberglu> I haven't done anything with it since before break
14:32:53 <adreznec> Yeah, I don't think there's been any movement on CI there
14:33:06 <thorst_> wangqwsh: do you have any updates on it?
14:33:20 <adreznec> and the only upstream patch I'm aware of is the one I have open for fixing the interfaces template
14:33:40 <wangqwsh> no, still block on crt_trunk_with_free_vlan
14:34:40 <thorst_> the block there is?
14:34:48 <thorst_> (sorry - my memory is hazy from holiday)
14:35:33 <wangqwsh> i am trying to boot a vm by manual, however the nova compute said KeyError: '06CC64F5-608A-4D4D-8C95-65B80CDFA96D'
14:36:04 <wangqwsh> i found this error was from pypowervm/tasks/cna.py - crt_trunk_with_free_vlan
14:36:35 <thorst_> can you send a pastebin with the stack?
14:36:46 <thorst_> not urgent, but I should probably dig into that
14:37:11 <wangqwsh> sure.
14:37:34 <thorst_> I'm also thinking that wangqwsh should learn from esberglu how to redeploy the CI
14:37:49 <thorst_> so that esberglu can take a break every once and a while (ex. go on holiday)
14:38:06 <wangqwsh> yes,
14:38:15 <esberglu> Especially since I am gonna be out for 2+ weeks after christmas
14:38:38 <wangqwsh> pleasure :)
14:38:45 <thorst_> esberglu: Can you take an action to set up a jump box (that has the patches that aren't yet merged) and train wangqwsh on how to do the deploys?
14:39:11 <esberglu> #action esberglu: Teach wangqwsh how to deploy CI
14:39:16 <thorst_> wangqwsh: you'll get thrown in on this quickly, cause mainline CI is our highest priority (though we have a light at end of tunnel)
14:39:40 <wangqwsh> ok
14:41:44 <thorst_> cool
14:42:08 <esberglu> thorst_: Once we get your change in, what projects should I open up the CI for? The *-powervm projects don't have much action right now, but the core projects will have a ton
14:43:10 <esberglu> Just kick off a few runs and make sure they are alright and then start opening up to the core projects after?
14:43:12 <thorst_> well, we want core once we've stablized
14:43:19 <thorst_> three phases
14:43:25 <thorst_> 1) Stabilize in powervm projects
14:43:39 <thorst_> 2) Open up to other projects for higher velocity -> check results
14:43:49 <thorst_> 3) Publish logs (but do not vote) for nova project
14:44:01 <thorst_> and for now, other projects should probably just be nova
14:44:55 <adreznec> Yeah
14:45:08 <adreznec> I vote *-powervm first, then nova, then neutron/ceilometer
14:45:26 <adreznec> then OSA once we get the base CI stable
14:46:00 <thorst_> I'd really like 3 being done by Jan 1
14:46:02 <thorst_> :-)
14:46:05 <thorst_> if not sooner
14:46:20 <esberglu> I think we can get it done before then
14:46:36 <thorst_> optimism
14:46:39 <thorst_> :-)
14:47:09 <esberglu> Anything else?
14:47:49 <thorst_> not from me
14:47:54 <thorst_> lots of good progress though!
14:48:21 <adreznec> Yeah, that's all I've got
14:48:26 <adreznec> Good to see things back running again :)
14:48:35 <esberglu> Man I forgot to change the topic the whole meeting. Oh well
14:48:39 <esberglu> #endmeeting