Thursday, 2017-08-17

*** thorst has joined #openstack-powervm00:01
*** thorst has quit IRC00:02
*** edmondsw has joined #openstack-powervm00:37
*** edmondsw has quit IRC00:42
*** thorst has joined #openstack-powervm01:03
*** thorst has quit IRC01:08
*** esberglu has joined #openstack-powervm01:20
*** thorst has joined #openstack-powervm01:38
*** thorst has quit IRC01:39
*** thorst has joined #openstack-powervm01:42
*** esberglu has quit IRC01:52
*** svenkat has quit IRC02:03
*** thorst has quit IRC02:20
*** edmondsw has joined #openstack-powervm02:26
*** edmondsw has quit IRC02:30
*** thorst has joined #openstack-powervm03:20
*** thorst has quit IRC03:25
*** jwcroppe has joined #openstack-powervm03:51
*** edmondsw has joined #openstack-powervm04:14
*** jwcroppe has quit IRC04:17
*** edmondsw has quit IRC04:18
*** thorst has joined #openstack-powervm04:22
*** thorst has quit IRC04:26
*** openstack has quit IRC04:42
*** openstack has joined #openstack-powervm04:45
*** openstackstatus has joined #openstack-powervm04:45
*** ChanServ sets mode: +v openstackstatus04:46
*** kylek3h has joined #openstack-powervm04:46
*** zerick has joined #openstack-powervm04:49
*** adreznec has joined #openstack-powervm04:49
*** thorst has joined #openstack-powervm05:22
*** thorst has quit IRC05:27
*** YuYangWang has joined #openstack-powervm05:53
*** YuYangWang has quit IRC05:55
*** YuYangWang has joined #openstack-powervm05:56
*** edmondsw has joined #openstack-powervm06:02
*** edmondsw has quit IRC06:06
*** jwcroppe has joined #openstack-powervm06:18
*** jwcroppe has quit IRC06:23
*** thorst has joined #openstack-powervm06:23
*** thorst has quit IRC06:28
*** YuYangWang has quit IRC06:56
*** YuYangWang has joined #openstack-powervm06:56
*** YuYangWang has quit IRC07:14
*** YuYangWang has joined #openstack-powervm07:14
*** thorst has joined #openstack-powervm07:24
*** thorst has quit IRC07:29
*** edmondsw has joined #openstack-powervm07:50
*** edmondsw has quit IRC07:54
*** thorst has joined #openstack-powervm08:25
*** thorst has quit IRC08:30
*** thorst has joined #openstack-powervm09:26
*** thorst has quit IRC09:30
*** edmondsw has joined #openstack-powervm09:38
*** edmondsw has quit IRC09:42
*** YuYangWang has quit IRC10:23
*** thorst has joined #openstack-powervm10:27
*** thorst has quit IRC10:31
*** svenkat has joined #openstack-powervm11:11
*** svenkat has quit IRC11:13
*** mawi has joined #openstack-powervm11:21
*** edmondsw has joined #openstack-powervm11:26
*** thorst has joined #openstack-powervm11:27
*** edmondsw has quit IRC11:30
*** thorst has quit IRC11:32
*** thorst has joined #openstack-powervm11:45
*** svenkat has joined #openstack-powervm11:47
*** svenkat_ has joined #openstack-powervm11:51
*** svenkat has quit IRC11:52
*** svenkat_ is now known as svenkat11:52
mawihello. I can't instantiate vm on power box (with pvm_sea networking) due to the following error: ERROR nova.compute.manager [instance: 24dc2b4d-d861-4def-9ef9-f1573850c4a5] PortBindingFailed: Binding failed for port 2761eb50-d979-4e03-9902-c1879d82bd3d12:12
mawiI would appreciate any suggestions how to  further investigate the root cause of it.12:12
thorstmawi - were we chatting earlier (sorry - mapping nic's from other means to IRC)12:13
thorstwondering if you were the one we wanted to get the nova.conf / neutron.conf's from our CI12:14
mawiyes :)12:14
thorstmind waiting ~1 hour for esberglu to pop online?12:14
thorststill early where he's at12:15
mawinot at all, I will wait. Thanks!12:15
*** AndyWojo has joined #openstack-powervm12:20
*** edmondsw has joined #openstack-powervm12:25
*** esberglu has joined #openstack-powervm13:07
mawiHello esberglu. I am having problem with instantiating vm on power due to some networking configuration problems13:08
mawiI am getting ERROR nova.compute.manager [instance: 24dc2b4d-d861-4def-9ef9-f1573850c4a5] PortBindingFailed: Binding failed for port 2761eb50-d979-4e03-9902-c1879d82bd3d13:08
mawiwould you suggest what  to start with in order to find the root cause?13:09
esberglumawi: Could you post the full trace on please?13:14
mawiwill do13:14
mawiPaste #61868113:19
thorstesberglu: I was hoping you could point mawi to our nova.conf and neutron.conf's from our CI runs13:19
thorstso that he can compare to his own13:19
mawithe stack trace is from nova-compute.log on NL system13:19
mawiI am not sure if in my configuration the networking-powervm agent is used at all (couldn't see that in log)13:20
thorstI suspect that even though the stack is in nova-compute, the error is on the neutron side13:20
thorstnova is just complaining the neutron didn't bind it properly13:20
thorstusually that means that the neutron server conf doesn't know about the pvm sea type13:20
mawifrom the neutron-server.log I see following:13:22
mawi2017-08-17 10:32:47.984 2788 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['linuxbridge', 'pvm_sea']13:23
mawi2017-08-17 10:32:47.988 2788 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['linuxbridge', 'pvm_sea']13:23
mawi2017-08-17 10:32:48.316 2788 INFO neutron.plugins.ml2.managers [req-edd87be2-3f2b-49ab-98a3-69e94fc40cd8 - - - - -] Initializing mechanism driver 'pvm_sea'13:23
thorsthmm...that is encouraging.13:23
thorstthat means you must have installed the networking-powervm as well13:23
thorstthe next thing I do is I set the neutron server log to debug13:24
thorstthen try a VM deploy.13:24
thorstin the nova-compute log you'll see when we start to wait for the 'vif binding'13:24
thorstthen go the the neutron server log and see what it's saying about a vif binding around that time13:24
thorstusually it has some esoteric error...13:24
thorstmawi: really dumb question.  You're using a VLAN network right?13:25
thorstnot vxlan, flat or gre?13:25
thorst(networking-powervm only supports vlan)13:26
mawiexcuse me this stupid question but are you talkin about following setting in the /etc/neutron/plugins/ml2/ml2_conf.ini file:13:29
mawitype_drivers = flat,vlan13:29
thorstwell, networking-powervm only supports vlan13:29
thorstso that may be one bit13:29
esberglu^ neutron and nova.conf. A lot of the nova.conf is probably irrelevant but I posted it in full13:30
mawithank you esberglu, would you mind sending me also your ml2_conf.ini file?13:30
esberglumawi ^13:32
mawitanks a lot13:33
mawiAm I assuming properly, that on the compute node (NL) PowerVM Shared Ethernet agent will be started13:33
thorstyou need to start it there13:33
mawiautomatically and I don't have to start it13:33
thorstyou start the networking-powervm and nova-compute there.13:34
thorstand optionally ceilometer-powervm13:34
mawinetworking-powervm as a service ?13:34
thorstjust like OVS neutron agent13:34
thorstthey're independent / codependent services  :-)13:34
mawias I downloaded the code from github, I don't have such a service. What is the poper way to start it?13:35
thorstsame way you start the nova compute?13:36
mawibut I installed nova-compute as ubuntu package13:36
thorstsurely not the powervm bit?13:36
thorstI don't know of anyone who's done that yet  :-)13:37
mawiwell hold on13:37
mawiwhat I did on NL was:13:37
mawiapt install software-properties-common add-apt-repository cloud-archive:ocata apt install nova-compute then I cloned git repo nova-powervm and linked directories from ocata branch to python dist-packages on NL13:40
mawiI alos cloned repo for networking-powervm and linked ocata branch to my python installation. What would be the proper way to install it?13:41
mawiI wasn't able to find ubuntu repositories with networking-powervm or nova-powervm13:42
thorstmawi: so the "proper" way is to use a distribution.  Ubuntu hasn't brought in all the powervm packages into their repos for their distro.  Though I suppose you could get the OVS one working (assuming its a new enough version)13:42
thorstwe are integrated into the OSA distro though13:42
thorstI think that's just for OVS though, I don't think that has SEA yet13:43
thorstadreznec: can you confirm?13:43
adreznecthorst: Yeah, OSA is OVS/LBr only, no SEA13:43
adreznecmawi: thorst when you say you linked the ocata branch for networking-powervm to your python installation, what do you mean exactly?13:44
adreznecDid you pip install it? Something else?13:44
mawifrom NL I git cloned the repository do local file system, changed branch to ocata, so I had these files from ocata branch in local filesystem. Finally I did13:46
mawisudo ln -s /home/padmin/nova-powervm/nova/virt/powervm_ext /usr/lib/python2.7/dist-packages/nova/virt13:46
mawisudo ln -s /home/padmin/nova-powervm/nova_powervm /usr/lib/python2.7/dist-packages13:47
mawiso the python could see the files among othe packages13:47
mawiwhat do you mean by OSA and OVS (open vswitch?)?13:48
adreznecOSA = OpenStack-Ansible, a deployment tool for openstack -
adreznecOVS is Open vSwitch, another network provider supported by PowerVM if you're using NovaLink with SDN13:48
adreznecRather than using SEA13:49
adreznecAre you actually able to start the Nova compute service with those links in place?13:50
mawiok. I am aware that there is SDN option too. However it looks lika I am having problems with the basic and simpler SEA option now..13:50
mawiyes, I have nova-compute up and running and it load the nova-powervm driver13:51
adreznecHmm interesting, ok. Wasn't sure if it'd work just linking over the nova/virt directory like that since nova usually has other files in there besides the virt drivers, but cool.13:52
adreznecSo for networking there are a couple differences from how you did the compute stuff. In that case you're just re-using the service file from the nova-compute package you installed but telling it to load nova-powervm instead13:52
adreznecBut for networking-powervm there doesn't seem to be a corresponding "generic" package you can install then point and tell it to use networking-powervm instead13:53
mawiok. so I am still missing part how would I properly install networking-powervm on ubuntu NL lpar13:54
adreznecYou should be able to get things up and running if you were to pip install networking-powervm, then configure your neutron options as per
adreznecthorst: It's been a while since I've done this, but I believe pip installing networking-powervm lays down a startable agent then13:56
adreznecnetworking-powervm-sharedethernet-agent or similar13:57
mawiwell. I have read the doc under the link you sent many times :). This configuration should be done on the neutron controller system right? No configuration is needed on  te NL (compute node) side right?13:57
mawiI also previously did: sudo pip install nova-powervm== ceilometer-powervm== networking-powervm==
mawiand it installed correctly but I couldn't get nova-compute running.13:59
adreznecmawi: Some of that configuration has to be done on the compute/NL side as it could be unique for each node. Which mechanism agent (sea or sriov) or bridge-mappings, for example,13:59
mawiok. (the doc didn't state this clearly). Do I need to install additional neutron packages?14:00
adreznecthorst or esberglu could probably help you with that configuration better than I could at this point14:00
*** apearson has joined #openstack-powervm14:00
adreznecI think pip installing networking-powervm should pull in the requisite dependencies on the compute node.14:01
adreznecYour control node would obviously be different depending on how you installed that (ubuntu I'm assuming)14:01
mawione of the issues with pip I had was that without specifying version to be installed explicitly it didn't install the latest one and even after using the latest ones, it led to breaking some package dependecies on ubuntu14:02
mawiI can try that again if you think this is the way to go14:03
mawiyes. the control node is x86 ubuntu 16.04 with ubuntu openstack packages14:04
adreznecmawi: Sorry, had to change locations for a meeting quick. Trying pip would probably be the fastest option. The other option I can think of offhand would be to look at what the neutron-openvswitch-agent package for ubuntu openstack is doing and either replicate that but using the networking-powervm code, or install that and change the configuration to load networking-powervm instead14:06
esbergluefried: edmondsw:
adreznecSince there's no official package for networking-powervm14:07
esbergluThe time.sleep call is timing out?14:07
adreznecFor the Ubuntu distro14:07
adreznecSorry, I have to handle this meeting now, but I'll check back in after. Hopefully if you have questions in the mean time thorst or esberglu can help out14:08
mawithanks for you support14:08
edmondswesberglu I assume that means that tempest timed out and it just so happened to be during the sleep14:11
esbergluedmondsw: If that were the case the timeout would be raised from here though?14:13
edmondswesberglu yep... I assume it did, and the trace is just misleading14:14
edmondswtime.sleep can't raise a tempest-specific TimeoutException :)14:14
edmondswesberglu wait... that's not the tempest TimeoutException, it's a fixtures one...14:17
esbergluedmondsw: Weird right?14:18
esbergluI've been looking at this failure wrong for a while because of that14:18
*** mawi has quit IRC14:19
edmondswI think that should get you going14:23
esbergluAh so there is a separate test timeout14:24
edmondswesberglu what do we set for OS_TEST_TIMEOUT ?14:25
edmondswlooks like we could just disable that by setting it to 014:26
edmondswnot that we probably want to...14:26
esbergluedmondsw: I don't think we set it, so w/e the default is14:26
edmondswthat explains it then, right?14:27
edmondswso I think we need to either bump that or play with TIMEOUT_SCALING_FACTOR for the specific tests where we're seeing timeouts14:27
esbergluedmondsw: I'm thinking we just double the OS_TEST_TIMEOUT for now. We can modify later as needed. Cool?14:44
edmondswI think that's probably ok, but we'll need to keep an eye on things. These timeout settings are getting high...14:46
edmondswefried agreed?14:46
esbergluedmondsw: Upping the timeout shouldn't affect anything except these long running tests though14:48
esbergluWe can drop the build_timeout since it is currently longer than the OS_TEST_TIMEOUT14:49
edmondswuntil/unless something goes wrong, yes14:49
edmondswgo ahead14:49
efriedcatching up...14:51
efriedSo we've identified why stuff was timing out quicker than we thought it shoulda?14:52
esbergluefried: Yeah14:53
efriedGood.  And yes, I'm on board with doubling that timeout and seeing if that helps.14:54
esbergluIs the best way to just set it in /etc/environment? I still don't fully understand what can see env. vars if using export. If you export, do all child processes have access?14:55
edmondswesberglu I would have asked you that question :)14:57
edmondswdoes it log what the vars are? If so, you could try it and see14:57
esbergluI *think* that we could export and it would pass to child processes. I know that /etc/environment will work14:58
esbergluedmondsw: Not that I know of14:59
esbergluWe use /etc/environment for other env. vars. might as well keep it consistent14:59
esbergluHere's the next question15:01
esbergluIs a spawn, rebuild, and termination taking over 500s reasonabe?15:01
*** tjakobs has joined #openstack-powervm15:01
efriedesberglu Sorry, missed the discussion.  Yes, export gets it to child processes.  That's what export is for.  Do that instead of /etc/environment.  But only if you are sure we can't just put it in tempest.conf, which seems like a much more reasonable place for it.15:32
esbergluThere isn't a tempest.conf option for it15:42
edmondswtempest seems to be a weird hodgepodge of using conf and using env vars15:46
esbergluefried: If setting OS_TEST_TIMEOUT in the os_ci_tempest.conf files do I still need export? Or does sourcing make that unnecessary?15:48
efriedesberglu I believe you need to export it.15:49
efriedesberglu Which you can do from within the .conf btw.15:50
efriedSourcing is like the code was in the same file.15:50
*** k0da has joined #openstack-powervm16:20
efried  <== anyone feel like tackling this beast?16:38
openstackLaunchpad bug 1711410 in nova-powervm "get_available_resource/get_inventory should account for resources used by the NovaLink, VIOSes, and non-OS partitions" [Undecided,New]16:38
efriedesberglu Are there tempest tests for PCI passthrough, that we currently have disabled?16:45
*** gianpero has joined #openstack-powervm16:46
*** gianpero has quit IRC16:47
esbergluefried: Not seeing any17:17
esbergluedmondsw: Did you see efried's latest comment on the OS_TEST_TIMEOUT change? Thoughts?17:20
*** k0da has quit IRC17:40
edmondswesberglu replied17:57
*** esberglu has quit IRC18:13
*** esberglu has joined #openstack-powervm18:14
*** esberglu has quit IRC18:18
*** esberglu has joined #openstack-powervm18:24
*** k0da has joined #openstack-powervm18:50
*** efried has quit IRC19:18
*** efried has joined #openstack-powervm19:24
*** apearson has quit IRC19:31
*** apearson has joined #openstack-powervm19:36
*** tjakobs has quit IRC20:22
*** tonyb has joined #openstack-powervm21:05
*** svenkat has quit IRC21:43
*** esberglu has quit IRC21:47
*** esberglu has joined #openstack-powervm21:54
*** thorst has quit IRC22:00
*** k0da has quit IRC22:09
*** thorst has joined #openstack-powervm22:11
*** thorst has quit IRC22:23
*** thorst has joined #openstack-powervm22:28
*** apearson has quit IRC22:31
*** edmondsw has quit IRC22:36
*** thorst has quit IRC22:51
*** openstackgerrit has joined #openstack-powervm23:00
openstackgerritEric Fried proposed openstack/nova-powervm master: Spec: PowerVM Support for PCI Passthrough
*** svenkat has joined #openstack-powervm23:41

Generated by 2.15.3 by Marius Gedminas - find it at!