| *** dr_gogeta86 has quit IRC | 00:04 | |
| *** dr_gogeta86 has joined #openstack-swift | 00:07 | |
| *** gyee has quit IRC | 00:11 | |
| itlinux_ | DHE: it asked me to verify again.. | 00:13 |
|---|---|---|
| itlinux_ | not cool! since I was just on | 00:13 |
| mattoliverau | itlinux_: Swift (via keystone auth) only really has 2 types of users, or rather 3. Users who match as an operator, match as a reseller_admin and those who don't match. The first has access to the projects account, the second have access to every account and the last need to be given access via ACLs by the operators to access certain containers. | 00:32 |
| mattoliverau | itlinux_: the 'operator_roles' in the keystone middleware, simply says what keystone roles will map users to the first | 00:33 |
| mattoliverau | the 'reseller_admin_role' says which keystone role maps to the second | 00:33 |
| mattoliverau | so in your example, 'operator_roles = admin, swiftoperator' just means anyone in the keystone roles admin or swiftoperator will be given swift_operator privilige. | 00:34 |
| mattoliverau | So from a swift POV there is no difference. | 00:34 |
| itlinux_ | so operator is user who can consume, create containers | 00:35 |
| mattoliverau | yes, for their projects account. | 00:36 |
| itlinux_ | does the map identify that the user is admin and gets admin permissions? | 00:36 |
| mattoliverau | your getting confused with whats in the list. which is just a map to keystone roles | 00:36 |
| itlinux_ | or they still considered as operators | 00:36 |
| mattoliverau | anyone in a keystone role what appears in the 'operator_roles' list will be a swift operator | 00:37 |
| mattoliverau | anyone in the keystone role 'reseller_admin_role' will be an reseller admin | 00:37 |
| mattoliverau | anyone in the keystone role that appears in 'reseller_admin_role' will be an reseller admin | 00:38 |
| itlinux_ | ahh ok so I now have one called swiftuser, and I create it in keystone which does nothing within openstack but user can consume since I added that user to the operators which means they can create and delete ttheir own conttainers, objects.. | 00:38 |
| itlinux_ | how do I check if the admin is mapped to reseller_admin? | 00:39 |
| itlinux_ | since the proxy-server.conf does not show any admin just shows operator_roles.. | 00:40 |
| mattoliverau | yeah, they can create containers and objects in the account they're tenant is. So everyone in the project that has the swiftuser group (in keystone) will be able to create/delete containers and objects in the AUTH_<project_id> account | 00:40 |
| mattoliverau | what does the 'reseller_admin_role' line say in the config? | 00:40 |
| itlinux_ | yes that's what I want.. and it's working but I am trying to check what is mapped to admin | 00:41 |
| itlinux_ | I do not have that line | 00:41 |
| mattoliverau | what do you mean the keystone group admin? | 00:41 |
| itlinux_ | in the proxy-server.conf | 00:41 |
| mattoliverau | where does it appear in the keystoneauth mapping in proxy-server.conf | 00:42 |
| mattoliverau | if the 'reseller_admin_role' is missing, then it will default to the 'ResellerAdmin' group in keystone (if that exists). | 00:42 |
| itlinux_ | one sec.. will share | 00:42 |
| itlinux_ | I do have ResellerAdmin | 00:43 |
| mattoliverau | so admin in your case, _wont_ map to the reseller_admin_role | 00:43 |
| mattoliverau | if your talking about an admin group | 00:43 |
| mattoliverau | if admin is a user.. then just add them to the ResellerAdmin group in keystone | 00:43 |
| itlinux_ | http://paste.openstack.org/show/727106/ | 00:45 |
| mattoliverau | if you want the admin group to be reseller_admins, then you need to specify: `reseller_admin_role = admin` in the keystoneauth section in proxy-server.conf. (and remove it from the operator-roles) | 00:45 |
| itlinux_ | ok | 00:45 |
| mattoliverau | so, those options maps the swift priviliges to keystone groups. Add keystone users to the groups that map the to piviliges you want them to have in keystone to give them the required rights in swift | 00:46 |
| itlinux_ | what groups do you see.. am I missing something.. | 00:47 |
| mattoliverau | itlinux_: ok, so in your example: operator_roles = admin, swiftoperator, ResellerAdmin, swiftuser | 00:48 |
| itlinux_ | yes | 00:48 |
| itlinux_ | I have swiftuser which I added to a group now.. | 00:48 |
| itlinux_ | so that group which has access to the project does have permissions to use swift | 00:48 |
| itlinux_ | and I set quota to the account | 00:49 |
| itlinux_ | so I think that's the correct way to make it happen. | 00:49 |
| mattoliverau | your saying any keystone user in one of the keystone groups (admin, swiftoperator, ResellerAdmin, swiftuser) is a swift operator in the project account | 00:50 |
| itlinux_ | ok that do not have any other permissions other than using swift | 00:50 |
| mattoliverau | Because you are missing 'reseller_admin_role', anyone in the ResellerAdmin group in keystone is a reseller_admin | 00:50 |
| itlinux_ | what's the best way to check who is part of the ResellerAdmin? | 00:51 |
| itlinux_ | I guess I can add the line reseller_admin = admin | 00:51 |
| mattoliverau | ask keystone, any user in that group is a reseller admin | 00:51 |
| itlinux_ | ok let me check one sec.. | 00:52 |
| mattoliverau | you should put ResellerAdmin in the operators role when it's also reseller_admins role. as that might confuse operators. Because anyone currently in that group is _more_ then an operator | 00:52 |
| mattoliverau | *shouldn't | 00:53 |
| itlinux_ | ok so I should remove that user from the Operators.. | 00:53 |
| mattoliverau | remember they are roles _not_ users. | 00:54 |
| mattoliverau | you should remove the ResellerAdmin role from that list. | 00:54 |
| itlinux_ | I see I do not remember to have assigned an ResellerAdmin to anyone.. | 00:54 |
| itlinux_ | or to any groups | 00:54 |
| itlinux_ | so that's ok | 00:54 |
| mattoliverau | cool | 00:54 |
| mattoliverau | but then you have no reselleradmins | 00:54 |
| itlinux_ | in fact I do not think I have any reseller admin | 00:55 |
| mattoliverau | a reseller admin also have access to swift, but to anyones account | 00:55 |
| itlinux_ | I am checking admin | 00:55 |
| itlinux_ | http://paste.openstack.org/show/727107/ | 00:56 |
| itlinux_ | looks like admin does have reseller admin | 00:57 |
| mattoliverau | yeah looks like it. so your admin user is a memeber of the swiftoperator, admin and ResellerAdmin roles. So it matches the operator_roles and the reseller_admin_role options in your config (ResellerAdmin matches reseller_admin_role because the option is missing so defults) | 00:59 |
| mattoliverau | which is fine. | 00:59 |
| itlinux_ | ok | 01:00 |
| itlinux_ | cool... | 01:00 |
| mattoliverau | Because ResellerAdmin matches the reseller_admin_role option you don't need it in the operator_roles option because they would already have access | 01:00 |
| itlinux_ | ok super.. so I have it the way it should be now.. and created a new role called swiftclient and added to the list so anyone on that list is good to use swift | 01:01 |
| itlinux_ | what I need now as you stated above I need a doc which describes the role type reselleradmin, swiftoperator etc.. | 01:02 |
| mattoliverau | yup. for the project's swift account they're in that role in. So if you have 1 user with in the swiftclient role in 2 different projects, they can access both project accounts as operators. If your a user but not in that group, then you can potentually create objects etc but only if an operator adds an ACL for you. So there is also giving people access that way. | 01:04 |
| itlinux_ | yes I assigned swiftclient to the group.. | 01:06 |
| mattoliverau | The documentation is kinda spread out. So not really in one place. I'll try and find time to write something more clear and in one place and push up when I get the chance. | 01:06 |
| itlinux_ | I really do not want to deal with single users.. | 01:06 |
| mattoliverau | :) | 01:06 |
| itlinux_ | thanks even it's 2-3 places if you have a link that's ok I will create a new intternal doc | 01:07 |
| mattoliverau | itlinux_: https://docs.openstack.org/swift/latest/overview_auth.html#access-control-using-keystoneauth | 01:09 |
| itlinux_ | thanks | 01:09 |
| mattoliverau | and a little: https://docs.openstack.org/swift/latest/middleware.html#keystoneauth | 01:13 |
| *** links has joined #openstack-swift | 01:15 | |
| itlinux_ | thanks mattoliverau: | 01:16 |
| *** itlinux_ has quit IRC | 01:26 | |
| *** psachin has joined #openstack-swift | 02:21 | |
| *** links has quit IRC | 02:37 | |
| *** vinsh has quit IRC | 04:41 | |
| viks_ | mattoliverau: Nice explanation.. Thanks | 05:31 |
| mattoliverau | viks_: ta | 05:36 |
| openstackgerrit | Merged openstack/swift master: imported some docs from the old user-guide https://review.openstack.org/588093 | 06:01 |
| *** ccamacho has joined #openstack-swift | 06:16 | |
| *** hoonetorg has quit IRC | 06:42 | |
| *** hoonetorg has joined #openstack-swift | 06:54 | |
| openstackgerrit | HCLTech-SSW proposed openstack/swift master: Add ability to undelete an account. https://review.openstack.org/507808 | 06:55 |
| *** rcernin has quit IRC | 07:03 | |
| *** Guest62477 has quit IRC | 07:36 | |
| *** mvk_ has quit IRC | 07:46 | |
| *** eandersson has quit IRC | 07:46 | |
| openstackgerrit | HCLTech-SSW proposed openstack/python-swiftclient master: Add ability to exclude file from upload. https://review.openstack.org/568547 | 08:06 |
| *** pcaruana has joined #openstack-swift | 09:00 | |
| *** ejat has joined #openstack-swift | 09:22 | |
| *** cbartz has joined #openstack-swift | 09:44 | |
| *** hoonetorg has quit IRC | 09:54 | |
| *** hoonetorg has joined #openstack-swift | 10:08 | |
| *** mikecmpbll has joined #openstack-swift | 10:12 | |
| *** cbartz has quit IRC | 10:56 | |
| *** viks_ has quit IRC | 11:04 | |
| *** mikecmpbll has quit IRC | 11:15 | |
| *** mikecmpbll has joined #openstack-swift | 11:21 | |
| *** mikecmpbll has quit IRC | 11:37 | |
| *** mvenesio has quit IRC | 12:25 | |
| *** dewanee has joined #openstack-swift | 12:53 | |
| dewanee | hu all | 12:53 |
| dewanee | *hi | 12:53 |
| *** mikecmpbll has joined #openstack-swift | 13:30 | |
| *** ccamacho has quit IRC | 14:07 | |
| *** ccamacho has joined #openstack-swift | 14:07 | |
| *** ccamacho has quit IRC | 14:08 | |
| *** ccamacho has joined #openstack-swift | 14:08 | |
| thurloat | if you use EC with low configured parity fragments, thats the best storage efficiency with swift, right? | 14:45 |
| thurloat | obviously, not excellent durability | 14:45 |
| DHE | N+M (Data+parity) fragments will survive M failed hard drives, and consumes (N+M)/N times the space of the file itself... | 14:50 |
| DHE | so, yes... but you can play with it by raising N as well | 14:50 |
| DHE | it's a bit extreme, but 20+5 would survive any 5 dead drives and only consume +25% the storage capacity | 14:51 |
| tdasilva | just found this presentation concerning EC, looks pretty good, contains a bit of info on storage efficiency: https://www.snia.org/sites/default/files/JasonResch_Erasure_Codes_SDC_2013.pdf | 14:51 |
| DHE | 10+5 is a more sensible policy I think | 14:52 |
| thurloat | but the 10+5/20+5 is a ratio, right? or does it fly for the entire cluster? like if 5 disks die in 5k disks, we start having issues? | 14:53 |
| thurloat | but rather if it happens that the 5 that die are the parity disks for that piece of data | 14:54 |
| DHE | there's a notation difference here. I'm writing 10+5 (data + parity), but these guys write it as 10-of-15 (needed -of- total) | 14:54 |
| thurloat | understood | 14:55 |
| DHE | if you do have 5 disks fail and they happen to be all parity disks, that doesn't change much. at most it means that retrieving the object will not require much CPU work since no parity calculations are actually involved | 14:55 |
| thurloat | ah this pdf answers all the questions, thanks tdasilva | 14:57 |
| tdasilva | thurloat: yw :) | 14:57 |
| tdasilva | basically, there's more to it, it's not a simple answer :) | 14:58 |
| DHE | which is why a lot of the swift literature describes it as for "cold storage". It works best with lots of drives and there's a significant CPU cost. | 14:59 |
| thurloat | yea i haven't looked at anything EC related yet really. | 14:59 |
| thurloat | significant at the proxy server or object server level/ | 14:59 |
| thurloat | or both? | 14:59 |
| DHE | I did an experiment with a 3+2 EC scheme on swift and a Ryzen7 1700 (3 GHz) CPU doing the parity work. I estimate pegging a CPU core will give me around 400 megabits/second of IO | 14:59 |
| tdasilva | proxy | 14:59 |
| DHE | mostly the proxy, but recovering from failed hardware will require CPU on the object servers | 15:00 |
| DHE | for the reconstruction | 15:00 |
| tdasilva | ah yes! reconstructor | 15:00 |
| thurloat | beefy proxy servers aren't an issue to assemble, hoping to keep the cpu requirements of the object servers low though | 15:00 |
| thurloat | and if you set up a 20+5 configuration, and have a 200kb file, it's going to get chopped into 20 data pieces the same as a 2GB large object fragment, right? | 15:03 |
| thurloat | or whatever size you roll | 15:04 |
| *** ccamacho has quit IRC | 15:05 | |
| DHE | I don't know the specific of chunk sizes... I mean, that's 10 kilobytes per node/drive... | 15:06 |
| thurloat | throw a CDN infront of it, nonethewiser | 15:09 |
| thurloat | small files stay cached :P | 15:09 |
| DHE | delayed reconstruction as a means of reducing network traffic for reconstructs... interesting... | 15:09 |
| DHE | small files seeing a lot of use shouldn't use EC policies. | 15:10 |
| thurloat | yea this specific project is short-term archive | 15:10 |
| thurloat | write once, read twice | 15:11 |
| thurloat | just under half a pb | 15:11 |
| *** psachin has quit IRC | 15:21 | |
| thurloat | so your 3+2 system (or 3 of 5) pegged a CPU core serving 400mbit worth of traffic, and since CPU cost scales with width, a 10+5/10 of 15 would require 3x more CPU to drive? | 15:25 |
| thurloat | ish | 15:26 |
| *** eandersson has joined #openstack-swift | 16:10 | |
| *** spotz has quit IRC | 16:18 | |
| *** gyee has joined #openstack-swift | 16:35 | |
| notmyname | good morning | 16:58 |
| thurloat | o/ | 16:58 |
| notmyname | hmm EC conversation | 16:58 |
| notmyname | to be safe, I'd prefer to see EC used when the resulting fragment size is >=10MB | 16:59 |
| notmyname | *especially* when you're using a lot of fragments | 17:00 |
| notmyname | DHE: I love the idea of ryzen CPUs with swift. I'm still hoping for a more low-power, high-thread/core one, though | 17:02 |
| DHE | notmyname: this was just my little home lab. one of my "storage nodes" was a 100megabit laptop with its internal hard drive and a USB drive as a "second disk". my main PC as the proxy has a ryzen7... | 17:03 |
| notmyname | heh | 17:03 |
| thurloat | anyone tried running ARM object servers? | 17:15 |
| thurloat | speaking of low power | 17:15 |
| notmyname | thurloat: I've got swift running at home on https://www.hardkernel.com/main/products/prdt_info.php?g_code=G150229074080 | 17:18 |
| notmyname | so yeah, it can work | 17:18 |
| thurloat | hehe | 17:18 |
| notmyname | not exactly what I'd choose for a data center. the only trick is getting some dependencies compiled/installed | 17:18 |
| thurloat | any crazy bottlenecks? | 17:18 |
| notmyname | yeah. don't write logs to a microsd card ;-) | 17:19 |
| thurloat | bahaha | 17:19 |
| thurloat | rip sd card | 17:19 |
| notmyname | it's not something I've stressed tested per se. It's just for home/family use | 17:19 |
| notmyname | but it's fine for that | 17:19 |
| thurloat | i'm going to be experimenting with a bunch of rock64s just to see what happens | 17:19 |
| notmyname | I'd recommend using a 64-bit cpu | 17:19 |
| notmyname | isntead of 32 | 17:20 |
| thurloat | the RK3328 is 64 bit iirc | 17:20 |
| tdasilva | notmyname: can you expand on your home setup there? how many nodes are you running? why did you choose the HC1, did you consider the MC1. any pointers? what are you running for OS? | 17:22 |
| thurloat | hot topic! | 17:23 |
| tdasilva | heh...i've been eyeing the odroid for a while ;) | 17:23 |
| notmyname | tdasilva: I have an MC1 (so 4 nodes) and 4 HC1s with 2.5" spinning drives. one of the MC1 nodes is the proxy. the 4 HC1s are ACO | 17:23 |
| tdasilva | cool | 17:24 |
| tdasilva | did you do anything special for power? | 17:25 |
| tdasilva | I saw some people suggesting using something like: https://www.amazon.com/gp/product/B077349XDW?pf_rd_p=d1f45e03-8b73-4c9a-9beb-4819111bef9a&pf_rd_r=5SW20Q0FQHN8XHGA7SJD | 17:26 |
| notmyname | http://d.not.mn/home_swift1.jpg | 17:27 |
| notmyname | http://d.not.mn/home_swift2.jpg | 17:27 |
| thurloat | tdasilva: you can use an ATX psu and put barrel connectors onto the pci-e wires | 17:27 |
| tdasilva | nice! | 17:27 |
| notmyname | yeah, so the disadvantage is that I'm running everything off of one wall plug | 17:28 |
| notmyname | but that makes for neater cable management ;-) | 17:28 |
| thurloat | have you tested the power draw? | 17:29 |
| thurloat | with a wall meter/killawatt? | 17:29 |
| notmyname | I have not. but it's 2 5v 20A power supplies. so it can't be that much | 17:30 |
| notmyname | this is the power supply I have https://www.amazon.com/gp/product/B06XK2DDW4/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 | 17:38 |
| tdasilva | notmyname: can you merge this? https://review.openstack.org/#/c/587108/ | 18:01 |
| patchbot | patch 587108 - swift (stable/pike) - Native Zuul v3 tox jobs | 18:01 |
| notmyname | yeah, let me go through the stable/* patches | 18:02 |
| notmyname | thanks for reminding me | 18:02 |
| *** itlinux has joined #openstack-swift | 18:04 | |
| notmyname | ok, all the backport proposals with a core +1 have been marked to land. that leaves https://review.openstack.org/#/c/585506/ and https://review.openstack.org/#/c/585355/ as still open | 18:13 |
| patchbot | patch 585506 - swift (stable/queens) - py36: Fix test_get_logger_sysloghandler_plumbing | 18:13 |
| patchbot | patch 585355 - swift (stable/pike) - Fix SLO delete for accounts with non-ASCII names. | 18:13 |
| notmyname | which seem to have corresponding patches already approved on other stable branches | 18:14 |
| notmyname | ok, those are landed too | 18:14 |
| notmyname | timburke will be happy(-er) when he gets back online tomorrow ;-) | 18:15 |
| tdasilva | notmyname: thanks | 18:30 |
| *** itlinux has quit IRC | 21:54 | |
| *** rcernin has joined #openstack-swift | 22:32 | |
| mattoliverau | morning | 23:06 |
| DHE | notmyname: come to think of it, ryzen7 has 8 cores, boost speeds above 3 GHz, and does ECC memory (motherboard permitting)... that's not bad. and wow, the mobile version has a quad-core HT model with 2.2 GHz base clock at 15 watts... | 23:20 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
