Thursday, 2019-02-28

openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Fix version selector when for proxy-style URLs  https://review.openstack.org/63980500:26
*** dviroel has quit IRC00:54
*** takamatsu has quit IRC03:25
*** ganso has quit IRC03:38
*** enriquetaso has quit IRC04:21
*** takamatsu has joined #openstack-manila05:05
*** takamatsu has quit IRC05:10
*** kshitija has joined #openstack-manila06:21
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Fix version selector when for proxy-style URLs  https://review.openstack.org/63980506:32
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Fix version selector when for proxy-style URLs  https://review.openstack.org/63980506:35
*** kshitija has quit IRC06:42
*** kshitija has joined #openstack-manila06:44
*** kshitija has quit IRC07:20
*** kshitija has joined #openstack-manila07:24
*** kshitija has quit IRC07:36
*** kshitija has joined #openstack-manila07:38
*** kshitija has quit IRC08:05
*** kshitija has joined #openstack-manila08:21
*** pcaruana has joined #openstack-manila08:32
*** tosky has joined #openstack-manila08:39
*** e0ne has joined #openstack-manila09:32
*** luizbag has joined #openstack-manila10:38
*** cmoura has quit IRC10:47
*** ganso has joined #openstack-manila10:56
*** mauricio_ps has joined #openstack-manila10:56
*** dviroel_ has joined #openstack-manila10:56
*** erlon has joined #openstack-manila11:08
amitotbarron: ping11:14
tbarronamito: 'sup?11:16
amitotbarron: all good. I have a suggestion for the HTTP codes thing... would you like me to propose it in a separate patch or add to yours?11:17
tbarronamito: probably just propose it separately, that way if I like it I can +2 it and abandon mine11:19
amitotbarron: what I did, basically, is keep the http_codes.py file, but change it to import from httplib if we're in Python 2, or use HTTPStatus if we're in 3. Since pylint still doesn't eat it well, I added manila.api.http_codes to ignored-modules. I think ignoring that module is a small price to pay (it also means we only ignore it where it's imported, it still lints that module when you run pylint on 'all').11:19
tbarronamito: +211:20
amitotbarron: thanks :)11:21
tbarronamito: thank you!11:21
amitotbarron: sure, np!11:23
*** thaihust__ has joined #openstack-manila11:25
gansotbarron, gouthamr, bswartz, toabctl: Hi folks! Friendly reminder that Manage/Unmanage in DHSS=True patches are ready in gerrit waiting for your reviews: https://wiki.openstack.org/wiki/Manila/SteinCycle#Manage.2FUnmanage_Share_Servers11:27
gansotbarron, gouthamr, bswartz, toabctl: All comments and concerns have been addressed so far, all tests are being run and passing for the Container and NetApp driver11:28
tbarronganso: yes, and thank you for your work on this, including the recent extra work :)11:28
thaihust__hi, i've deployed OpenStack Manila 3 nodes. I got a problem below11:28
gansotbarron: =)11:28
*** e0ne has quit IRC11:30
thaihust__when i create a share, manila-share on 1st node get request and create a share ok. I try stopping manila-share on this node, then delete the share created11:30
thaihust__manila stucked at status deleting11:30
thaihust__despite the fact that 2 remaining manila-share nodes still alive11:32
thaihust__i wanna know how to configure HA for manila-share components11:33
*** e0ne has joined #openstack-manila11:36
tbarronthaihust__: sec11:38
tbarronthaihust__: are you deploying using an openstack distribution with its packages?11:50
tbarronthaihust__: anyways, there's a 'host' field in the DEFAULT section of manila.conf11:52
tbarronif you don't see it you're using its default value11:53
tbarronthaihust__: https://paste.fedoraproject.org/paste/9xttP57v19NYfmYOIH~RdQ11:54
tbarronthaihust__: you'll want the value of this to be the same on all three nodes for manila-share11:55
thaihust__no, i build all component in docker11:55
tbarronthaihust__: so you want it in manila.conf for the containers running manila share11:56
tbarronthaihust__: manila share service11:56
thaihust__i've already set host value be the same on all node as you recommeded11:56
thaihust__but i got a new error11:56
tbarronthaihust__: leave them to the hostname for the other services11:56
tbarronthaihust__: you'll have to mess with the database for the shares created before they were set the same11:57
thaihust__i set up exactly as you recommended, when i type: manila service-list, i got 3 scheduler service and only 1 manila-share name: cloud-fs@cephfsnative11:59
thaihust__with cloud-fs is the host name i set for manila-share11:59
thaihust__but as i told you above11:59
tbarronthaihust__: ok maybe my theory about why you're stuck deleting is wrong :)12:00
tbarronthaihust__: i'd turn on debug and look at the manila share-logs on the surviving nodes as well as the scheduler logs to see if12:02
tbarronthaihust__: manila share service on one of the surviving nodes is even attempting the delete operation12:02
tbarronthaihust__: and to see if the surviving share services are doing their periodic reports to the scheduler nodes12:03
thaihust__i got a new problem: when i create the 1st share, everything ok, i watch manila-share on 1st controller handle this request. But when i create new shares, requests were load balanced to others controllers, manila-share stuck at creating new share12:03
tbarronthaihust__: if you have multiple nodes they'll load balance among the nodes for api and scheduler services but you only want to run one manila-share service at any time, are you doing that somehow?12:05
tbarronthaihust__: in our distro downstream we use pacemaker for that12:05
tbarronmanila-share service (like cinder-volume) does not have active-active support yet12:06
thaihust__could i use keepalived instead of pacemaker12:07
tbarronthaihust__: probably12:08
tbarronthaihust__: i use what our downstream distro uses already for cinder, etc.  -- what our HA folks think is most reliable, etc.12:08
tbarronthaihust__: but I'm not an HA expert12:09
thaihust__i'll try keepalived, thanks for your info12:11
tbarronthaihust__: good luck; in any case you'll want to run in debug and inspect share-service logs on the nodes that are up to see who is catching the message from the scheduler to do the operation and what problem is happening at that point12:12
openstackgerritArne Wiebalck proposed openstack/manila master: Fix error message when updating quota values  https://review.openstack.org/64001912:25
toabctltbarron, what is the best way (in your opinion) to use ceph with manila but use cifs and/or nfs on the clients?12:44
toabctlthe generic driver?12:44
tbarrontoabctl: well *in my opinion* it is to use cephfs back end with nfs ganesha and samba for cifs12:45
tbarrontoabctl: but I haven't built the samba part12:45
tbarrontoabctl: only on back of envelope12:45
toabctl:)12:45
tbarrontoabctl: i don't know of anyone who has deployed generic driver in production for real12:46
toabctltbarron, so nfs-ganesha would be deployed manually and mounts a manila backend?12:46
toabctlyeah. that's why I'm asking. I don't think the generic driver is very useful in real world12:47
tbarrontoabctl: by which I mean without introducing single point of failure on the data path, having limited number of cinder attaches for volumes, etc. etc.12:47
toabctl or do we have a manila driver that handles nfs-ganesah?12:47
tbarronnetworking issues too12:47
tbarrontoabctl: we have a manila nfs-ganesh back end12:47
tbarrontoabctl: sec12:47
* toabctl lost overview about the backend drivers...12:47
tbarronhttps://docs.openstack.org/manila/latest/admin/cephfs_driver.html#for-cephfs-nfs-shares12:49
tbarrontoabctl: ^^12:49
tbarrontoabctl: for devstack, this may be helful: https://github.com/tombarron/vagrant-libvirt-devstack12:51
tbarrontoabctl: buried in the ansible templates there you'll see devstack local.conf for deploying devstack that will deploy ganesha for you as well as an integrated cephfs back end12:52
tbarrontoabctl: and here's how we do it downstream with tripleo: https://www.slideshare.net/Inktank_Ceph/ceph-day-berlin-practical-cephfs-and-nfs-using-openstack-manila12:53
tbarrontoabctl: i'd love to see non-tripleo deployments also supporting cephfs with nfs back end12:54
tbarrontoabctl: because i consider it the best end-to-end 100% open source way to deploy manila12:54
tbarrontoabctl: glusterfs could also qualify there if the folks working with it would involve themselves again with the upstream manila community12:55
tbarronwith PTL hat on I don't favor cephfs over other open source solutions, I'd like multiple open source back ends that12:56
tbarronpeople are willing to support12:56
tbarronand I want viable, supported proprietary back ends as well12:57
*** mauricio_ps has quit IRC12:58
*** erlon has quit IRC12:58
*** luizbag has quit IRC12:58
*** enriquetaso has joined #openstack-manila13:13
*** erlon has joined #openstack-manila13:15
*** mauricio_ps has joined #openstack-manila13:16
*** luizbag has joined #openstack-manila13:16
openstackgerritAmit Oren proposed openstack/manila master: [pylint] fix errors with http_client status codes  https://review.openstack.org/64002513:20
*** s0ru has joined #openstack-manila13:22
*** lseki has joined #openstack-manila13:26
*** thgcorrea has joined #openstack-manila13:36
toabctltbarron, thx for the info!13:45
tbarrontoabctl: yw, as you can see I have a keen interest in that topic13:50
tbarrontoabctl: would love to see healthy cooperation and competition in that area13:51
openstackgerritAmit Oren proposed openstack/manila master: [pylint] fix errors with http_client status codes  https://review.openstack.org/64002513:53
vkmctoabctl, amito can you check this one? https://review.openstack.org/#/c/638320/ :)13:57
amitovkmc: no problem, I'll review it this evening as I'm on some downstream work at the moment :)13:58
vkmcamito, sure, thanks!13:58
*** s0ru has quit IRC13:59
*** jmlowe has quit IRC14:01
*** mmethot has joined #openstack-manila14:06
*** kopecmartin has joined #openstack-manila14:08
*** a-pugachev has joined #openstack-manila14:14
*** a-pugachev has quit IRC14:17
*** ianychoi has joined #openstack-manila14:20
*** jgrosso has joined #openstack-manila14:28
*** jmlowe has joined #openstack-manila14:29
*** carlos_silva has joined #openstack-manila15:00
*** e0ne has quit IRC15:16
*** e0ne has joined #openstack-manila15:19
bswartzI just noticed @slow_me_down for the first time15:22
openstackgerritAmit Oren proposed openstack/manila master: [pylint] fix errors with http_client status codes  https://review.openstack.org/64002515:27
amitobswartz: what's that?15:32
bswartzamito: I'm wondering what it is15:33
bswartzIt's in dummy.py15:34
gouthamrbswartz amito: https://review.openstack.org/#/c/400331/15:35
gouthamrjust sets predetermined artificial delays in execution of driver methods in the dummy driver to simulate storage backend delays15:36
bswartzgouthamr: +115:36
amitobswartz: seems like it just adds some delay before the function starts15:37
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Fix version selector when for proxy-style URLs  https://review.openstack.org/63980515:50
*** a-pugachev has joined #openstack-manila15:59
*** a-pugachev has quit IRC15:59
gansotbarron, gouthamr: regarding https://review.openstack.org/#/c/638681/16:00
gansowe've been investigating the proper fix across manila16:00
gansowe are able to fix in the database in 2 ways16:01
* tbarron listens16:01
ganso1 is very simple: rename size property in models.py to instance_size and create a property size that returns either self.size or self.snapshot.size16:01
gansobut in the database, the share_snapshot_instance database will continue to store size as null16:02
gansojust to add some context, in the database, regular snapshots create instances with null size, and snaphots created via group snapshots create instances with the correct size16:02
gansoso, approach 1 is simple, no impact, but doesn't fix the real issue.16:03
gansoapproach 2 is to fix for every new share snapshot instance created, so DB will always create the snapshot instance with the proper size regardless of how it is created16:03
gansobut we would need a database migration to fix it for all the existing DB entries16:04
gansothus copying the snapshot size to the share snapshot instance size where it is null16:04
gansoand approach 3 is even simpler, just go with the patch already proposed 638681 xD16:04
gansoin my mind, approach 2 is the ideal one, but also the riskier16:05
gouthamrganso: i like option 1, is there any problem in having a property in the snapshot instance model that overrides an existing attribute?16:05
gansogouthamr: yes, in order to not override it, we rename the existing attribute to instance_size("size",Column(int))16:05
tbarronganso: just so I understand, why is doing a DB migration *riskier*?16:06
gansogouthamr: and every time we do a create or update to the share_snapshot_instance model, we make sure the field is properly named "instance_size" instead of "size", as it is today16:06
tbarronganso: i can understand that it's more work ...16:06
gouthamrganso: oh, hmm, yeah. I think that's clean16:06
gansotbarron: it is slightly risky in the sense that we are not 100% sure we won't find something different than what we expect in production clouds when running the db migration16:07
gouthamrganso: option 3 will be a -2 from me, just saying :)16:08
gansotbarron: we don't expect anything to be different, but we never know. Here's an example:16:08
gouthamrganso: glad you guys found the root of the problem, but it's sketchy if we can't agree upon a generalized solution across all drivers - most of them won't have tests that would have caught this16:09
gansotbarron: if there's an old production cloud that has snapshot since the old Liberty days when there wasn't share snapshot instances or group snapshots yet, they could have something different there. The bug happens on new snapshot instances created as of today16:09
gansogouthamr: proposal 1 and 2 don't affect any driver at all16:10
tbarronganso: k, I understand what you are saying then16:10
gansogouthamr: it is fixed in the DB transparently16:10
tbarronthe other thing is that if want to backport this fix to stable it can't involve db schema changes16:10
gansotbarron: oh right, good point, didn't think about that16:10
gansotbarron: so if there is a db migration it is not backportable16:11
tbarronwell i'm not sure #1 is either16:11
tbarrongouthamr: ^^?16:11
gansotbarron: hmm why wouldn't it?16:11
gansotbarron: it looks like to me we will need the 2 fixes separated, one that we can backport, and the db migration that will fix the database for the current and next releases16:12
gouthamrdon't see a problem with backporting a change to the model representation16:12
tbarronganso: i guess changing properties in db isn't really a schema change but ...16:12
tbarronjust thinking out loud and making sure16:13
gansotbarron: it is just the model representation, the database column name isn't changed16:13
tbarronyes16:13
gouthamrif the fix isn't there, the bug is as it is today, there's really no user impact if drivers used to check the size like the netapp driver does16:13
gouthamri.e, the real size would be set in the backend always, anyway16:13
tbarronbut the "virtual database" as consumed by the upper layers has changed16:14
tbarronso not sure we'd really want to backport to stable/*16:14
gansotbarron: not really, because of the property16:14
gansotbarron: if the upper layer uses the snapshot_instance_create and update db methods, as it should be, then the upper layer will not notice a change16:14
*** enriquetaso has quit IRC16:14
tbarronganso: what I'm saying is that we do queries using the properties, etc. not just the real columns16:14
gansotbarron: if by "we" you mean other parts of manila or other drivers etc, we will submit the fix upstream and see what breaks16:15
tbarronit may be ok, just want to make sure you guys think this through :)16:15
tbarronsounds like you are converging on #116:16
*** e0ne has quit IRC16:16
tbarronwe really don't need to backport for python 3 support anyways I think16:16
gansotbarron: based on the fact that we can't backport db migrations, #1 will have to be done I guess... #2 is a very good improvement to be done as well for the current and future releases16:16
tbarronwe need py3 for Stein16:17
gansohmmm16:18
tbarronso consider the possiblity of #2 for Stein and forwards if you think it's the better fix16:18
gansoso, well, yea we need to decide between #1 or #216:18
gansotbarron: yea it is better, but riskier :P16:18
tbarronsounds like gouthamr prefers #1 anyways?16:18
gansotbarron: you know production clouds better than me, so if you think #2 is safe16:18
gansotbarron: then I believe you16:18
* tbarron watches ganso try to wiggle off the trout hook16:19
gansolol16:19
gansogouthamr: what do you think ?16:20
*** e0ne has joined #openstack-manila16:23
openstackgerritTom Barron proposed openstack/manila master: Move grenade job to bionic and run with python 3  https://review.openstack.org/63072716:23
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [pylint] Fix/ignore pylint errors in test modules  https://review.openstack.org/63806716:36
*** jmlowe has quit IRC16:43
*** e0ne has quit IRC17:05
openstackgerritMerged openstack/puppet-manila stable/queens: Support cephfs_volume_mode parameter  https://review.openstack.org/63964517:14
*** luizbag has quit IRC17:16
*** luizbag has joined #openstack-manila17:16
*** kopecmartin is now known as kopecmartin|off17:17
*** jgrosso has quit IRC17:39
*** yajushiSri has joined #openstack-manila17:53
*** luizbag has quit IRC17:56
*** jmlowe has joined #openstack-manila18:03
*** kshitija has quit IRC18:37
*** e0ne has joined #openstack-manila18:51
*** mauricio_ps has quit IRC18:54
*** luizbag has joined #openstack-manila18:57
*** luizbag_ has joined #openstack-manila19:02
*** e0ne has quit IRC19:03
*** luizbag has quit IRC19:04
*** thaihust__ has quit IRC19:08
*** mauricio_ps has joined #openstack-manila19:14
*** enriquetaso has joined #openstack-manila19:14
*** pcaruana has quit IRC19:14
*** whoami-rajat has quit IRC19:20
gansogouthamr, tbarron: is there any way we can try to reach agreement on the above discussion? This is a high priority for us and I have a team mate that is waiting for this agreement so we can finish the implementation and send it upstream19:25
gouthamrganso: thought we agreed...  we won't backport a database migration, so if the fix can be done without one, great..19:27
gansogouthamr: ok, we will proceed with this fix, and if we ever want to improve the bugfix fixing it in the database, we make a separate patch19:28
gansogouthamr: thanks19:28
gouthamrganso: yw, thanks for the bugfix!19:29
*** senrique_ has joined #openstack-manila19:50
*** enriquetaso has quit IRC19:52
*** senrique_ has quit IRC19:55
*** senrique_ has joined #openstack-manila19:55
*** thgcorrea has quit IRC20:02
openstackgerritRodrigo Barbieri proposed openstack/manila master: Add manage/unmanage of shares in DHSS=True  https://review.openstack.org/63583120:06
* gouthamr grrr20:06
gouthamrganso: imma complete the API review on PS19, and do the others on PS20, seeing as you've only addressed one of my comments :)20:07
gansogouthamr: omg how many are there20:08
gouthamrganso: you wait20:08
gouthamr:P20:08
*** mauricio_ps has quit IRC20:09
*** luizbag_ has quit IRC20:13
openstackgerritMerged openstack/manila master: Fix error message when updating quota values  https://review.openstack.org/64001920:15
*** e0ne has joined #openstack-manila20:16
*** senrique_ is now known as enriquetaso20:20
*** lseki has left #openstack-manila20:23
openstackgerritLucio Seki proposed openstack/manila-tempest-plugin master: Add functional tests for manage/unmanage of shares in DHSS=True  https://review.openstack.org/63823120:23
openstackgerritTom Barron proposed openstack/manila master: Only allow IP access type for CephFS NFS  https://review.openstack.org/64018520:38
openstackgerritLucio Seki proposed openstack/manila-tempest-plugin master: Add functional tests for manage/unmanage of shares in DHSS=True  https://review.openstack.org/63823120:41
*** erlon has quit IRC21:01
gouthamrganso: while you wait can you review these test changes? it'll be good to have them merged: https://review.openstack.org/628921 https://review.openstack.org/63570321:07
openstackgerritLucio Seki proposed openstack/manila-tempest-plugin master: Add functional tests for manage/unmanage of shares in DHSS=True  https://review.openstack.org/63823121:11
gansogouthamr: I'm on the bus home and I will work on addressing the comments as soon as I get home so I can push a patch to get the CI to run tonight. I'll review those changes tomorrow morning21:13
gouthamrganso: oh, no rush.. didn't realize you're four hours ahead! :) goodnight!21:14
gansogouthamr: 5 hours actually, if you're in seattle at this moment21:17
gouthamrganso: nice.. I'll be wrong again in 10 days :P21:18
*** e0ne has quit IRC21:19
*** e0ne has joined #openstack-manila21:42
openstackgerritLucio Seki proposed openstack/manila-tempest-plugin master: Add functional tests for manage/unmanage of shares in DHSS=True  https://review.openstack.org/63823121:43
*** e0ne has quit IRC21:44
*** e0ne has joined #openstack-manila21:45
*** zul has quit IRC21:53
*** e0ne has quit IRC22:01
openstackgerritRodrigo Barbieri proposed openstack/manila master: Add manage/unmanage of shares in DHSS=True  https://review.openstack.org/63583122:54
*** carlos_silva has quit IRC22:59
openstackgerritRodrigo Barbieri proposed openstack/manila-tempest-plugin master: Add functional tests for manage/unmanage of shares in DHSS=True  https://review.openstack.org/63823123:04
gansogouthamr: ok I addressed the comments (some I replied with some questions), and I have triggered the CIs. I will have dinner now. Do you have more comments inbound today?23:05
gouthamrganso: 'course i do, i was going to post on your patch that you don't address them tonight..23:06
gouthamrwe're going to see some midair collisions if new patchsets come in23:06
gansogouthamr: what do you mean? I dont mind with post the comments related to previous patchsets, since they are on different files23:09
gouthamrganso: sure, np if you don't mind.. i meant that i'm working through those files in batches; and like you saw, we're still in the API23:10
gansogouthamr: hmm maybe you could re-review just after you finished your first pass through all the code, and dont mind the changes based on your comments that I push up23:14
gouthamrganso: sure..23:14
*** enriquetaso has quit IRC23:22
*** dviroel_ has quit IRC23:41

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!