| *** whoami-rajat has joined #openstack-manila | 01:18 | |
| *** baojg has joined #openstack-manila | 01:34 | |
| *** lseki has joined #openstack-manila | 02:58 | |
| *** enriquetaso has joined #openstack-manila | 03:54 | |
| *** enriquetaso has quit IRC | 03:59 | |
| *** enriquetaso has joined #openstack-manila | 04:26 | |
| *** lseki has quit IRC | 05:08 | |
| *** rwadolowski has joined #openstack-manila | 05:14 | |
| *** enriquetaso has quit IRC | 05:44 | |
| *** kopecmartin|off is now known as kopecmartin | 06:26 | |
| rwadolowski | tbarron, are you alive ? :) | 08:52 |
|---|---|---|
| tbarron | rwadolowski: barely, just had my first sip of morning coffee. 'sup? | 09:43 |
| rwadolowski | I am looking into: https://bugs.launchpad.net/manila/+bug/1806576 | 09:52 |
| openstack | Launchpad bug 1806576 in Manila "CEPHFS/NFS driver fails with error code 108" [Undecided,Triaged] - Assigned to Tom Barron (tpb) | 09:52 |
| rwadolowski | it's occured in my env | 09:52 |
| rwadolowski | so if you have some time, we can try to debug-fix it | 09:53 |
| tbarron | rwadolowski: are you running two ganesha servers against the same ceph cluster? | 09:55 |
| rwadolowski | yep, exactly | 09:55 |
| rwadolowski | i changed cephfs_volume_path_prefix | 09:55 |
| rwadolowski | still same | 09:55 |
| tbarron | do they use the same CepFS file system and metadata and data pools or separate filesystems? | 09:56 |
| rwadolowski | my setup is using manila-share in container (kolla) and nfs-ganesha on this same host, but outside container | 09:56 |
| rwadolowski | this same | 09:56 |
| tbarron | i saw in the backlog your discussion w/ gouthamr regarding the dbus socket | 09:58 |
| tbarron | do goth ganesha instances use the same socket path? | 09:59 |
| tbarron | both | 09:59 |
| rwadolowski | yes, so i decided to go outside docker with ganesha | 09:59 |
| rwadolowski | http://paste.openstack.org/show/751221/ this is config for backend. On each compute there is different ip and path | 10:00 |
| tbarron | outside docker, inside docker are both fine but I would worry about using the same socket path | 10:00 |
| * tbarron looks | 10:00 | |
| rwadolowski | I have each ganesha on different controler | 10:00 |
| tbarron | ah | 10:01 |
| tbarron | and you are using different cephfs_volume_pat_prefix for each backend I guess? | 10:02 |
| rwadolowski | exactly, hostname of each controller, example /{{controller_hostname}} | 10:03 |
| tbarron | looks like we have an incidental logging issue in that traceback in the bug: Share instance information in exception can not be written to db because it contains {} and it is not a dictionary | 10:05 |
| tbarron | do you have that error too? | 10:05 |
| rwadolowski | yes, i will paste my error, sec | 10:06 |
| tbarron | with the main issue being a stat failure? | 10:06 |
| rwadolowski | http://paste.openstack.org/show/751222/ | 10:07 |
| rwadolowski | yes | 10:08 |
| *** carloss has joined #openstack-manila | 10:09 | |
| tbarron | and this works with the first ganesha bacj end even when two are configured? | 10:09 |
| tbarron | back | 10:10 |
| * tbarron broke right hand so typing is awkward | 10:10 | |
| rwadolowski | yes, only one manila(ganesha) can create shares, the other one are throwing errors | 10:12 |
| tbarron | so as you see the error is from the ceph volume library which is part of ceph rather than manila itself (not even in the manila driver code) | 10:12 |
| tbarron | not trying to dump the prob elsewhere by saying that btw | 10:13 |
| tbarron | it'a a python libraru though | 10:13 |
| tbarron | if you would take a minute to put those pastebins as well as that for the working back end in the bug that will be great | 10:15 |
| tbarron | also manila, ganesha, and ceph versions you are using | 10:15 |
| rwadolowski | yeah, no problem | 10:15 |
| rwadolowski | i will try to debug it deeper, maybe ceph-devs will help me :) | 10:16 |
| tbarron | rwadolowski: https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L625 | 10:17 |
| tbarron | it's catching ObjectNotFound but not error code 108 mebbe ... | 10:20 |
| rwadolowski | probably this error we're facing https://github.com/ceph/ceph/blob/ae4f94c868e87e7a60d35b7832db7988fd60ab53/src/common/freebsd_errno.cc#L100 | 10:21 |
| openstackgerrit | melissaml proposed openstack/manila master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658325 | 10:21 |
| tbarron | rwadolowski: if you fix this you'll get your name on the ceph t-shirt for the next release :) | 10:22 |
| tbarron | batrick: ^^^^^^^ | 10:23 |
| tbarron | rwadolowski: he's on the west coast US so won't be looking now but may know right off | 10:23 |
| tbarron | also gregsfortytwo ^^^^ | 10:24 |
| tbarron | rwadolowski: you can also ask in freenode #ceph or on <ceph-users@lists.ceph.com> | 10:26 |
| tbarron | rwadolowski: btw does the working back end also have ceph_volume_path_prefix configured? or is it using the default? | 10:32 |
| tbarron | cephfs_volume_path_prefix | 10:32 |
| *** lpetrut has joined #openstack-manila | 10:35 | |
| rwadolowski | configured, each of backends have different path | 10:35 |
| tbarron | rwadolowski: k | 10:37 |
| tbarron | rwadolowski: and at driver init time there aren't errors showing up in manila share service log around this code (for the second back end?) - | 10:38 |
| tbarron | https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L204 | 10:39 |
| *** baojg has quit IRC | 10:41 | |
| * tbarron wonders if it's an auth/caps/client keys issue | 10:44 | |
| rwadolowski | i can generate seperate keypairs and try to do it | 10:44 |
| tbarron | rwadolowski: just thinking out loud ..., the ceph guys will know better | 10:45 |
| tbarron | 108 is pretty obscure | 10:46 |
| rwadolowski | no errors at startup | 10:55 |
| tbarron | rwadolowski: ack | 11:08 |
| rwadolowski | tbarron, got it | 11:17 |
| * tbarron is all ears | 11:18 | |
| rwadolowski | caps | 11:18 |
| tbarron | do you need a separate client key for each back end? | 11:19 |
| rwadolowski | different caps for each and it is working like a charm | 11:19 |
| rwadolowski | for each manila :) | 11:19 |
| tbarron | rwadolowski: yeah i guess that makes sense | 11:20 |
| tbarron | nice work! | 11:21 |
| rwadolowski | new question is if we can change something on the ceph side | 11:22 |
| rwadolowski | so probably tools like kolla should generate caps for each endpoint, but this sounds more like a workaround | 11:23 |
| tbarron | it may be a security feature, how does the cluster know both subtrees are being managed by the same admin? | 11:25 |
| tbarron | rwadolowski: but it's a reasonable question for the ceph folks | 11:25 |
| rwadolowski | good point | 11:25 |
| rwadolowski | i am thinking about this: https://github.com/openstack/manila/blob/2609950d1e8d0b57c01fead23101ee7b419dbfb7/manila/share/drivers/cephfs/driver.py#L208 | 11:29 |
| rwadolowski | what are cons of disabling eviction at all | 11:29 |
| tbarron | rwadolowski: that's a Q for the ceph devs but i'm thinking it's a cleanup stale client sessions thing | 11:35 |
| tbarron | http://docs.ceph.com/docs/mimic/cephfs/eviction/ | 11:35 |
| *** rwadolowski has quit IRC | 11:38 | |
| *** enriquetaso has joined #openstack-manila | 12:22 | |
| *** enriquetaso has quit IRC | 12:23 | |
| *** enriquetaso has joined #openstack-manila | 12:23 | |
| *** lseki has joined #openstack-manila | 12:57 | |
| *** rwadolowski has joined #openstack-manila | 13:08 | |
| *** ianychoi_ is now known as ianychoi | 14:00 | |
| *** lpetrut has quit IRC | 14:43 | |
| *** lpetrut has joined #openstack-manila | 14:47 | |
| *** lpetrut has quit IRC | 14:57 | |
| *** josecastroleon has quit IRC | 15:32 | |
| *** lpetrut has joined #openstack-manila | 16:06 | |
| *** s0ru has joined #openstack-manila | 16:12 | |
| openstackgerrit | Soledad Kuczala proposed openstack/manila master: Adding documentation for User Messages in Manila Documentation https://review.opendev.org/645010 | 16:23 |
| *** s0ru has quit IRC | 16:47 | |
| *** enriquetaso has quit IRC | 16:50 | |
| *** enriquetaso has joined #openstack-manila | 16:55 | |
| *** whoami-rajat has quit IRC | 16:58 | |
| *** kopecmartin is now known as kopecmartin|off | 17:04 | |
| *** lpetrut has quit IRC | 17:17 | |
| batrick | rwadolowski: are you using freebsd? | 17:45 |
| batrick | on Linux that's just ESHUTDOWN | 17:45 |
| batrick | if you look at src/include/rados.h that's also EBLACKLIST (ceph specific errno) | 17:46 |
| batrick | so it's possible your ganesha got blacklisted somehow | 17:46 |
| batrick | these errors are happening during setup or after using it for some time? | 17:46 |
| *** whoami-rajat has joined #openstack-manila | 18:09 | |
| *** dviroel has quit IRC | 21:52 | |
| *** carloss has quit IRC | 21:57 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-specs master: Move unimplemented specifications to separate folder https://review.opendev.org/657894 | 22:20 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-specs master: Move unimplemented specifications to separate folder https://review.opendev.org/657894 | 22:20 |
| *** whoami-rajat has quit IRC | 22:39 | |
| *** enriquetaso has quit IRC | 23:39 | |
| *** enriquetaso has joined #openstack-manila | 23:41 | |
| *** lseki has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!