Thursday, 2018-11-01

« Wednesday, 2018-10-31 Index Friday, 2018-11-02 »
*** markstur has quit IRC00:10
*** erlon has quit IRC00:30
*** erlon has joined #openstack-manila00:43
*** zul has quit IRC01:04
*** erlon has quit IRC01:58
*** dustins has joined #openstack-manila03:15
*** dustins has quit IRC03:25
*** pcaruana|elisa| has joined #openstack-manila07:40
*** pcaruana|elisa| has quit IRC07:59
*** pcaruana has joined #openstack-manila08:05
*** e0ne has joined #openstack-manila09:43
*** e0ne has quit IRC10:32
*** e0ne has joined #openstack-manila10:36
*** ganso has joined #openstack-manila10:49
openstackgerritRodrigo Barbieri proposed openstack/manila master: PLEASE DO NOT MERGE ME  https://review.openstack.org/61473711:03
*** pcaruana has quit IRC11:05
*** zul has joined #openstack-manila11:26
*** luizbag has joined #openstack-manila11:28
*** luizbag has quit IRC11:34
*** pcaruana has joined #openstack-manila11:52
*** erlon has joined #openstack-manila11:57
*** jmlowe has quit IRC12:47
*** zul has quit IRC12:52
*** e0ne has quit IRC12:55
*** zul has joined #openstack-manila12:56
*** eharney has joined #openstack-manila13:01
*** zul has quit IRC13:04
*** zul has joined #openstack-manila13:06
*** jmlowe has joined #openstack-manila13:07
*** belmoreira has joined #openstack-manila13:17
openstackgerritRodrigo Barbieri proposed openstack/manila-specs master: Add spec for Manage-Unmanage of Share Servers  https://review.openstack.org/60734213:26
*** dustins has joined #openstack-manila13:34
*** e0ne has joined #openstack-manila13:40
*** gouthamr has left #openstack-manila13:54
*** tpsilva has joined #openstack-manila13:55
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [CI][LVM] Run the LVM job on Bionic Beaver  https://review.openstack.org/60876114:07
*** e0ne_ has joined #openstack-manila14:09
*** e0ne has quit IRC14:10
*** gouthamr has joined #openstack-manila14:39
*** markstur has joined #openstack-manila14:55
tbarronmanila meeting has started on #openstack-meeting-alt15:03
*** jmlowe has quit IRC15:10
*** e0ne_ has quit IRC15:34
*** e0ne has joined #openstack-manila15:34
gouthamrwe have documentation suggesting the use of linuxbridge network driver for the manila generic share driver16:00
gouthamrbut we don't test it, it's likely broken16:01
tbarronganso sorry to cut you off16:01
gouthamrganso: ^ about not being able to reproduce at the gate16:01
tbarrongouthamr: it's not just our doc, the neutron doc w.r.t. openstack installation also instructs use of linuxbridge16:01
gouthamrit's quite easy for small deployments to setup linux bridge, so maybe we should run one of our jobs with it16:02
gouthamrand we might see the issue?16:02
gansotbarron: np16:02
tbarronganso: gouthamr and I have also been getting emails about this, probably from the same person16:02
gansoI helped 2 people that were deploying the generic driver with linux bridge16:02
gansoboth of them initially had connection issues16:02
gansobut one of them supposedly (I don't know how) sorted the connection issue out, and was able to connect manually to the service instance16:03
gansobut the driver was not able to connect, for some reason16:03
gansowe ran tcpdump and everything, the ssh handshake from the driver was seen, but the driver could not connect16:03
bswartzwas the ssh handshake successful?16:04
gansowe checked security group rules, etc16:04
bswartzOne of the bad cases I've seen is when the service VM can't talk to the metadata server, and cloud-init fails to get the SSH key16:04
gansobswartz: I saw ssh handshake packages, I am not sure if it was successful. So it wasn't a connection issue in that case16:04
bswartzIn that case, SSH cannot succeed, and you have to use password login16:04
gansobswartz: ohhhhhh that could be the case16:04
bswartzIIRC, the gate uses password login all the time because we were never able to track down the cloud-init problem16:05
gouthamrbswartz: the reporter used password as well16:05
gansoyes, but setting up the username and password in the driver still didn't allow it to suceed16:05
bswartzganso: are you sure you used the username and password that was pre-installed in the service image?16:05
bswartzWhich values did you use?16:05
gansothe person could do it manually, the share server was in error state but he could ssh into it. The driver couldn't, not even with username and password instead of ssh keys16:06
bswartzmanila/manila?16:06
gansobswartz: yes16:06
bswartzOkay maybe you've found yet another issue16:06
gansoso yesterday, the last time I spoke to him, I asked him to run a devstack in another env and we would try to compare every configuration that could be relatable16:07
bswartzWhen I hit series of problems last year I just banged my head into the wall until I found a less nettlesome issue to work on16:07
bswartzThe context I hit it in was when trying to use my alternative (lightweight) service image16:07
gansohmmmmmm16:09
gansoanyway, thanks for the tip, I will try to check for the metadata server connectivity16:09
tbarronganso: we shouldn't need metadata server working for password auth though, right?16:10
gansotbarron: yes, we shouldn't16:10
tbarronganso: or is there something else from cloud-init that is required?16:10
gansotbarron: I am not sure why the driver still can't ssh without the ssh key16:11
gansotbarron: I don't know what else is required. Another weird thing is that, as far as I know, the metadata server and cloud-init are both needed for the instance to obtain its IP16:11
tbarronis the driver using the same ssh client as when one runs the cmd manually from the vm?16:11
*** belmoreira has quit IRC16:11
gansobswartz: ^ can confirm that? was the problem you had the instance had its IP?16:11
gansotbarron: the driver uses paramiko16:12
tbarrons/from the vm/from the node where manila share runs to the vm/16:14
tbarronganso: so I wonder if a simple python program with paramiko and passwd auth would have the same problem connecting to the vm.16:14
gansotbarron: yea I wonder that too, that could be a good test. Write a simple python code to connect via paramiko and see what is required or why it fails16:15
bswartzIt's been a while since I looked into this16:15
bswartzBut when I dug deep down I saw SSH failures16:16
bswartzAnd the thing that makes it worse is that paramiko lies in its error messages16:16
bswartzSome of the error messages that come out of paramiko are completely false16:16
* tbarron notes that octavia uses service vms and wonders if they ssh to them via paramiko16:16
bswartzYou can verify this by stepping into paramiko with a debugger while the driver is trying to connect to the service VM16:17
gansobswartz: since I was just giving instructions via email I was relying on the other person to be able to debug16:18
bswartzYeah I'm just saying that it's incredibly hard to see what's going on because (1) it's all virtualized and seeing the internals requires special tooling and (2) the system is actively lying to you about why thing didn't work16:19
gansotbarron: one more thing I wanted to say was that, both of these folks that approached me with that issue attempted reinstalling everything with OVS instead of LB and still didn't get it to work16:19
bswartzIf there was an alternative to paramiko I would suggest using it16:20
bswartzOtherwise the only solution is to contribute bugfixes to paramiko16:20
gansobswartz: it is already difficult to deal with liars, but when the machine is lying it is even worse xD16:21
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver  https://review.openstack.org/61480216:24
gansogouthamr: so what were your breakthroughs?16:25
tbarronganso: I guess the mystery is why we don't have this issue with paramiko in gate.  We were thinking the diff was something about ovs vs linuxbridge but I guess not.16:25
gouthamrganso: see ^16:25
*** dustins has quit IRC16:25
gouthamri switched the IP address to br-ex - what we discussed, and what the quagga documentation seemed to suggest16:25
bswartzgouthamr: the commit message says bionic, but the code comments say xenial16:26
bswartzwhich is it?16:26
gouthamr:) bswartz: i'm testing centos and xenial with https://review.openstack.org/61480216:26
gansogouthamr: nice! for some reason it didn't work when I did it. Maybe it had to be done in the proper order16:26
gansogouthamr: I created a debug patch this morning but I didn't see any zebras in it16:26
*** e0ne has quit IRC16:26
gouthamrganso: perhaps, yes.. i did a fresh stack and freaked out :P16:27
gansothe 10-second sleep did not work16:27
gouthamr?16:27
gouthamri saw the patch16:27
bswartzgouthamr: why does the commit message say bionic then?16:27
gouthamrbswartz: https://review.openstack.org/608761 is the true bionic change - i.e, switch the job to run on bionic16:27
gansogouthamr: I was looking for a route added by zebra, but there wasn't any. There was supposed to be one16:27
bswartzSo is the problem with xenial or bionic?16:28
gouthamrganso: hmm, strange thought there was one - looking16:28
gouthamrbswartz: i haven't played with quagga on Xenial, ganso has.. gate's currently using centos - netapp-ci is using Xenial16:28
gansobswartz: the problem is with bionic16:28
gouthamrbswartz: i don't want to break folks so decided to split out the quagga changes to test xenial and centos in isolation16:29
gouthamrganso: how much lag have you seen for the zebra route to show up?16:32
bswartzgouthamr: I'm going to -1 your comment in that case16:41
gouthamrbswartz: sure, am confused, but will address it :)16:42
gansogouthamr: I am not sure, shouldn't be more than 10 seconds16:45
*** belmoreira has joined #openstack-manila17:02
gouthamrregarding reproducing the connectivity bug, we can eliminate one variable by testing with manila.network.linux.interface.BridgeInterfaceDriver as the interface_driver on the gate17:02
gouthamrwe run two jobs with DHSS=true/generic17:03
gouthamrin two separate repos - manila, manila-tempest-plugin17:03
gouthamrone of these can be running with LinuxBridge instead of OVS as the mech driver17:03
*** belmoreira has quit IRC17:19
*** belmoreira has joined #openstack-manila17:20
*** e0ne has joined #openstack-manila17:31
*** e0ne has quit IRC17:47
*** luizbag has joined #openstack-manila17:54
*** e0ne has joined #openstack-manila17:59
*** e0ne has quit IRC18:02
*** e0ne has joined #openstack-manila18:05
*** e0ne has quit IRC18:07
*** jmlowe has joined #openstack-manila18:22
gansogouthamr: sounds good18:27
gansogouthamr: I am confused about https://review.openstack.org/#/c/608761/ and https://review.openstack.org/#/c/61480218:41
gansogouthamr: the former is running on bionic and has quagga working18:42
gansogouthamr: why do we need the latter (it is running on centos)?18:42
* gouthamr cool, just got zuul's run on it18:42
gouthamrganso: yep, i was going to remove the plugin changes from  https://review.openstack.org/#/c/608761/18:42
gouthamrand make it dependent on https://review.openstack.org/#/c/61480218:43
gansogouthamr: oh ok18:43
gouthamrso that https://review.openstack.org/#/c/614802 can test centos18:43
gansogouthamr: great!18:43
gouthamrlooks like the ipv6 tests passed: http://logs.openstack.org/02/614802/1/check/manila-tempest-minimal-dsvm-lvm-centos-7/be8e085/job-output.txt.gz#_2018-11-01_17_53_05_14068718:43
gouthamrbut the job timed out18:43
gouthamrganso: don't see the zebra route here: http://logs.openstack.org/02/614802/1/check/manila-tempest-minimal-dsvm-lvm-centos-7/be8e085/job-output.txt.gz#_2018-11-01_17_33_03_31055818:44
gouthamrganso: so the logging isn't very useful at the moment, i'll just remove it18:45
gansogouthamr: yea me neither, IPv6 is passing so it is working18:45
gouthamrganso: yes, and those private networks are created in the test?18:45
gansogouthamr: yes18:46
gansogouthamr: we should still the 1 route for the private network created by the plugin. It might be that 10 seconds is not enough18:46
gansogouthamr: but I don't think it is worth it to send another patch and increase the sleep18:46
gouthamrganso: yes, i'm confused about that as well.. its working locally too18:47
gouthamryeah18:47
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver  https://review.openstack.org/61480218:48
*** belmoreira has quit IRC18:48
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver  https://review.openstack.org/61480218:49
*** pcaruana has quit IRC19:05
*** luizbag has quit IRC19:16
*** luizbag_ has joined #openstack-manila19:21
*** belmoreira has joined #openstack-manila19:32
*** dustins has joined #openstack-manila20:05
*** dustins has quit IRC20:06
*** dustins has joined #openstack-manila20:06
*** dustins has quit IRC20:07
*** dustins has joined #openstack-manila20:08
*** eharney has quit IRC20:08
*** e0ne has joined #openstack-manila20:11
*** belmoreira has quit IRC20:12
gansoxyang: ping20:18
xyanghi20:19
*** luizbag_ has quit IRC20:20
gansoxyang: Hey Xing =) Could you please take a look at those 3 specs? https://review.openstack.org/#/c/607342/, https://review.openstack.org/#/c/391805 and https://review.openstack.org/#/c/609537  ?20:20
xyangganso: sure20:20
gansoxyang: thank you! =D20:20
xyangnp20:21
*** e0ne has quit IRC20:28
*** belmoreira has joined #openstack-manila20:35
*** belmoreira has quit IRC20:35
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver  https://review.openstack.org/61480220:54
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [CI][LVM] Run the LVM job on Bionic Beaver  https://review.openstack.org/60876120:54
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver  https://review.openstack.org/61480220:55
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver  https://review.openstack.org/61480221:03
gouthamrlast time ^21:03
*** erlon has quit IRC21:05
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [CI] Switch Xenial tempest jobs to Bionic Beaver  https://review.openstack.org/60492921:13
*** dustins has quit IRC21:31
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [DevRef] Add code review guideline  https://review.openstack.org/60959821:43
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [CI][LVM] Run the LVM job on Bionic Beaver  https://review.openstack.org/60876121:49
*** threestrands has joined #openstack-manila22:17
*** tpsilva has quit IRC23:04
*** ganso has quit IRC23:17
« Wednesday, 2018-10-31 Index Friday, 2018-11-02 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!