Thursday, 2017-03-30

*** thorst has quit IRC00:01
*** jpasqualetto has joined #openstack-powervm00:22
*** thorst has joined #openstack-powervm00:36
*** thorst has quit IRC00:53
*** edmondsw has quit IRC01:29
*** mdrabe has joined #openstack-powervm01:48
*** thorst has joined #openstack-powervm01:51
*** mdrabe has quit IRC01:57
*** thorst has quit IRC02:04
*** thorst has joined #openstack-powervm02:31
*** thorst has quit IRC02:38
*** tjakobs has joined #openstack-powervm03:12
*** tjakobs has quit IRC03:18
*** thorst has joined #openstack-powervm03:36
*** thorst has quit IRC03:55
*** jayasankar_ has joined #openstack-powervm04:56
*** nbante has joined #openstack-powervm05:08
*** jayasankar_ has quit IRC05:14
*** jayasankar_ has joined #openstack-powervm05:27
*** shyama has joined #openstack-powervm05:39
*** thorst has joined #openstack-powervm05:52
*** thorst has quit IRC05:57
*** thorst has joined #openstack-powervm06:53
*** thorst has quit IRC06:57
*** shyama has quit IRC07:20
*** k0da has joined #openstack-powervm07:20
*** shyama has joined #openstack-powervm07:21
*** k0da has quit IRC07:50
*** thorst has joined #openstack-powervm07:54
*** nbante has quit IRC08:07
*** thorst has quit IRC08:12
*** k0da has joined #openstack-powervm09:05
*** thorst has joined #openstack-powervm09:09
*** thorst has quit IRC09:13
*** esberglu has joined #openstack-powervm09:58
*** esberglu has quit IRC10:02
*** thorst has joined #openstack-powervm10:10
*** jayasankar_ has quit IRC10:12
*** thorst has quit IRC10:14
*** edmondsw has joined #openstack-powervm10:21
*** jayasankar_ has joined #openstack-powervm10:24
*** edmondsw has quit IRC10:26
*** smatzek has joined #openstack-powervm10:42
*** thorst has joined #openstack-powervm11:11
*** thorst has quit IRC11:15
*** shyama has quit IRC11:18
*** shyama has joined #openstack-powervm11:19
*** jwcroppe has joined #openstack-powervm11:27
*** thorst has joined #openstack-powervm11:34
*** a1fisher has joined #openstack-powervm12:11
*** edmondsw has joined #openstack-powervm12:22
*** jpasqualetto has quit IRC12:23
*** jpasqualetto has joined #openstack-powervm12:23
*** efried has joined #openstack-powervm12:27
*** apearson has joined #openstack-powervm12:38
*** jpasqualetto has quit IRC12:45
*** kylek3h_ has quit IRC12:49
*** jwcroppe has quit IRC12:53
*** dwayne__ has quit IRC12:55
*** k0da has quit IRC12:56
*** jpasqualetto has joined #openstack-powervm12:58
*** k0da has joined #openstack-powervm13:11
*** mdrabe has joined #openstack-powervm13:15
efriedadreznec Were you going to propose requirements update bots for networking- and ceilometer-powervm?13:23
efriedDo you have a way of doing same for pypowervm?  I think that guy is out of sync at this point.13:23
adreznecefried: Ah yeah, forgot about that. I'll get to that today here13:24
adreznecpypowervm is just manual at this point13:24
adreznecno scripting around that13:24
efriedthorst I don't see a good way to get around coordinated upload without changing pypowervm.13:27
efriedWe would have to spoof the local_api trait.  I kinda don't want to do that.13:27
*** smatzek has quit IRC13:28
*** kylek3h has joined #openstack-powervm13:29
*** esberglu has joined #openstack-powervm13:31
efriedthorst I'm also not sure there's a good way to support FUNC without threads.13:32
efriedI'm going to work on a biggish pypowervm change that gets rid of coordinated and deprecates FUNC.13:33
efried'cept coordinated is in public APIs too.  So I have to deprecate that guy too.13:36
thorstefried:  func may need threads but n ot the pipe13:43
thorstso it halves the threads13:43
thorstbut i think we could still make it work  :-)13:44
*** dwayne__ has joined #openstack-powervm13:51
*** jayasankar_ has quit IRC13:55
*** jpasqualetto has quit IRC13:58
*** burgerk has joined #openstack-powervm14:03
*** burgerk has quit IRC14:08
esbergluefried: thorst: adreznec: All of the OOT CI runs are failing as of last night14:10
thorstesberglu: yeah, I thought that volume was quite high14:11
thorstis it immediately obvious why its failing?14:11
esbergluis timing out trying to create vif14:11
thorstthe VIF is attached to a VM, right?14:11
*** jpasqualetto has joined #openstack-powervm14:11 is14:13
thorstsuper odd, because most of the other tests create vifs...14:14
thorstso what's special about this one14:14
esbergluYeah and nothing has changed in the test14:14
esbergluBut you can see all the other ones going through fine14:14
thorsthmm....I can even see it sending the device up from neutron14:18
thorst"Sending device up to Neutron for fa:16:3e:62:cb:5d"14:18
thorstare any other CI's experiencing issues?14:19
esberglubrb, coffee break14:23
thorstesberglu: its always that test?14:24
thorstbasically what I'm seeing is that neutron sends the event saying 'provisioned vif' and its just not showing up in the nova logs as received14:24
thorstyeah, we never got the 'external_instance_event' passed back to nova, which is why it failed14:29
thorstnot sure why we didn't get that back14:29
thorstefried: you familiar with this at all?  I don't think so, it's events, but not the novalink ones...14:30
efriedI looked at it a leeeetle bit when we were doing the SR-IOV stuff.  svenkat may be more familiar.14:30
efriedAre we using SEA in the CI?14:31
thorstefried: yeah, this is SEA14:32
esbergluthorst: Yep always that test14:32
efriedAnd is our SEA plugin/agent responsible for forwarding events?14:32
thorstefried: kinda?14:32
thorstsea agent sends the device up...that goes to the neutron server...neutron server sends event for device up over to nova api...nova api sends that external instance event to nova-compute (I think)14:32
efriedAnd you're saying we see it leave neutron, but not get to nova?14:34
efriedSo it'd be out of our hands by that point anyway?14:34
thorstI see it leave the sea_agent14:35
thorstand not get to nova14:35
thorstso I guess I need to find out what logging takes place in the neutron server.14:35
thorstefried: just for your knowledge...the thing that sends it to nova is neutron/notifiers/nova.py14:37
*** dwayne__ has quit IRC14:41
efriedthorst Is it bad form for one driver to import libs from another driver's namespace?14:41
efriedLet me be more specific: IterableToFileAdapter actually came from xenapi, and it's still there.14:41
efriedSo can I import nova.virt.xenapi.image.utils and use it from there?14:42
efriedI understand why we wouldn't want to do that OOT, because we wouldn't know if it changed underneath us.14:42
efriedBut in-tree....14:42
*** tjakobs has joined #openstack-powervm14:42
thorstefried: Absolutely bad form14:49
efriedBetter to duplicate it?14:49
thorstthat one, yes14:50
thorstwe had it in earlier iterations14:50
efriedIf there was a nova/image/, I would suggest moving it there.14:50
thorstjust go back to the Mitaka release, and we have it there14:50
efriedBut there isn't.14:50
efriedYes, in the OOT driver.14:50
efriedI'm talking in-tree.14:50
efriedNow we'll have duplicated code in-tree.14:50 is supposed to mirror OOT (and then when merged OOT can cut some stuff out)14:50
efriedmmph, I've definitely not been using a philosophy as strong as "mirror".14:51
*** mdrabe has quit IRC14:51
efriedBut okay.14:51
thorstefried: well, you're bringing changes back to oot...14:52
thorstwhich is essentially keeping them symmetrical14:52
efriedI'm not retrofitting 100% of the UT improvements.14:52
thorstback to the CI issue...I see in the neutron server this is happening14:53
thorst2017-03-29 23:16:12.793 [00;32mDEBUG neutron.notifiers.nova [[01;36mreq-61ad865a-d06d-43f5-bda3-ce1d90c20308 [00;36mservice neutron[00;32m] [01;35m[00;32mIgnoring state change previous_port_status: DOWN current_port_status: DOWN port_id 4626c642-d1d6-4bfa-9c3a-7ca952f60853[00m [00;33mfrom (pid=6399) record_port_status_changed /opt/stack/neutron/neutron/notifiers/[00m14:53
thorst2017-03-29 23:16:15.271 [00;32mDEBUG neutron.notifiers.nova [[01;36mreq-2362ce84-51c4-4298-9f59-0954b33733fc [00;36mNone None[00;32m] [01;35m[00;32mIgnoring state change previous_port_status: DOWN current_port_status: BUILD port_id 4626c642-d1d6-4bfa-9c3a-7ca952f60853[00m [00;33mfrom (pid=6401) record_port_status_changed /opt/stack/neutron/neutron/notifiers/[00m14:53
thorst2017-03-29 23:16:15.277 [00;32mDEBUG neutron.notifiers.nova [[01;36mreq-7d356fc3-dd81-4d53-9d31-adad673ce9b2 [00;36mNone None[00;32m] [01;35m[00;32mIgnoring state change previous_port_status: DOWN current_port_status: BUILD port_id 4626c642-d1d6-4bfa-9c3a-7ca952f60853[00m [00;33mfrom (pid=6401) record_port_status_changed /opt/stack/neutron/neutron/notifiers/[00m14:53
thorstit never gets a current_port_status of ACTIVE...14:53
thorstlike all the other ports.14:53
thorstI even see a message from neutron: "Provisioning for port 4626c642-d1d6-4bfa-9c3a-7ca952f60853 completed by entity L2."14:59
esbergluWhich is very odd seeing the previous message of15:01
esberglu"Transition to ACTIVE for port object 4626c642-d1d6-4bfa-9c3a-7ca952f60853 will not be triggered until provisioned by entity L2"15:01
*** mdrabe has joined #openstack-powervm15:01
*** newbie|2 has joined #openstack-powervm15:03
*** jwcroppe has joined #openstack-powervm15:07
thorstesberglu: try doing this for me...set the neutron agent configuration option 'heal_and_optimize_interval' to some super high value...liek 5000015:15
esbergluOk. Starting a manual run now15:17
thorstI think what is happening is that the heal and optimize is pulling down the VIF and then setting it back to a build state15:20
thorstwhich really shouldn't matter...but I think it may be interfering...15:22
esbergluWe had issues with that before on the undercloud15:23
*** dwayne__ has joined #openstack-powervm15:26
*** k0da has quit IRC15:32
thorstif that's the fix...what we can do is just update networking-powervm to only run the heal if we don't have provision requests pending15:41
thorstI thought we had put that in earlier...15:41
esbergluthorst: Thought that we set heal_and_opt or thought we updated networking-powervm?15:58
thorstesberglu: I don't follow?16:05
thorstI thought we set it somewhere...but it looks like it is running in this case for some reason16:06
*** shyama has quit IRC16:42
*** shyama has joined #openstack-powervm16:43
-openstackstatus- NOTICE: The Gerrit service on is being restarted briefly to relieve performance issues, and should return to service again momentarily.17:04
esbergluthorst: No luck setting heal_and_optimize_interval17:23
thorstesberglu: are there new logs there?17:24
*** newbie|2 has quit IRC17:28
esbergluthorst: I can send you the login info for the instance, otherwise I can upload them somewhere17:29
thorstesberglu: can we get them uploaded onto our log server?17:38
esbergluthorst ^17:50
esbergluprobably should have zipped them but oh well17:50
*** k0da has joined #openstack-powervm17:51
*** adi______ has joined #openstack-powervm17:54
*** jayasankar_ has joined #openstack-powervm17:56
*** adi_____ has quit IRC17:59
*** adi______ is now known as adi_____17:59
*** jayasankar_ has quit IRC18:03
shyamaefried: I am looking at UT for . The current test for this class mocks the whole feed task and the sub task methods do not get executed18:06
shyamahow do i get them to be run as part of the UT18:06
*** k0da has quit IRC18:28
*** gianpero has joined #openstack-powervm19:18
gianperoHi, anybody knows if there is any plan to integrate VMware vRealize with IBM PowerVM NL?19:20
thorstgianpero: I know that PowerVC (which uses PowerVM and NovaLink) have VMware vRealize integration19:20
thorstso I think its there today...  If VMware vRealize supports a generic OpenStack then I'd assume it could support PowerVM NL's OpenStack drivers.19:21
thorstbut again, PowerVC uses those OpenStack drivers too.19:21
gianperothorst: But it isn't a direct between vRA and NL?19:21
thorstgianpero: not that I know of.19:22
thorstI think its all through OpenStack19:22
gianperoso I have to install the nova module on vRA, on the NL LPAR19:23
thorstgianpero: I don't think so...I thought it was just install an OpenStack (ex. PowerVC) which manages PowerVM19:24
thorstthen connect vRA to that OpenStack19:24
thorstbut I'm not a vRA expert.19:24
gianperoam asking because, am not sure which one is the routemap on orchestration from IBM19:25
gianperoI had read about vRA, ICO, and OpenStack19:26
gianperonot sure where to point our efforts19:26
thorstgianpero: I can give you a name of someone from PowerVC who could maybe help out with that?  If you want that name, just PM me and I'll send you his e-mail19:28
gianperothorst: tks, I'll PM you.19:29
*** efried has quit IRC19:43
*** efried has joined #openstack-powervm19:55
*** shyama has quit IRC20:33
*** esberglu has quit IRC20:44
*** k0da has joined #openstack-powervm20:49
*** esberglu has joined #openstack-powervm21:02
*** edmondsw has quit IRC21:30
*** edmondsw has joined #openstack-powervm21:31
*** edmondsw has quit IRC21:35
*** k0da has quit IRC21:46
efriedthorst So one thing: Are we worried about an old pvm-novalink using a new pypowervm and thereby suffering from the perf problems of API upload?21:51
efriedapearson How old would the pvm-novalink have to be?21:51
thorstefried: I'm not too worried about it21:53
thorstbecause we can point to a new NL level to get good perf21:53
*** mdrabe has quit IRC21:58
*** esberglu has quit IRC22:00
*** jpasqualetto has quit IRC22:02
*** thorst has quit IRC22:03
*** efried has quit IRC22:03
*** dwayne__ has quit IRC22:06
apearsonefried - i'm not really sure when this was 'fixed'.  We don't know if it was a combination of the web servlet technology we had, or whether it was a problem in java7.  We moved off of 7 after the first release...but we've done an upgrade to our servlet stuff in pretty much every release, so depending on where the fix came...22:09
apearsonThat said, no I'm not worried...novalink is designed to be kept up to date.  If they're seeing performance issues, updating to the latest fix release will address it.22:09
*** gianpero has quit IRC22:29
*** apearson has quit IRC22:35
*** tjakobs has quit IRC22:44
*** gianpero has joined #openstack-powervm22:50
*** jwcroppe has quit IRC22:57
*** dwayne__ has joined #openstack-powervm23:00
*** thorst has joined #openstack-powervm23:04
*** thorst has quit IRC23:08
*** thorst has joined #openstack-powervm23:14
*** thorst has quit IRC23:17
*** edmondsw has joined #openstack-powervm23:31
*** edmondsw has quit IRC23:35
*** tjakobs has joined #openstack-powervm23:40
*** openstack has joined #openstack-powervm23:44
*** tjakobs has quit IRC23:45
*** thorst has joined #openstack-powervm23:47
*** thorst has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!