Tuesday, 2018-06-05

*** lifeless_ has joined #openstack-keystone00:01
*** oikiki has quit IRC00:01
*** lifeless has quit IRC00:02
*** bigdogstl has joined #openstack-keystone00:06
*** bigdogstl has quit IRC00:11
*** masuberu has joined #openstack-keystone00:15
*** bigdogstl has joined #openstack-keystone00:17
*** masber has quit IRC00:19
*** bigdogstl has quit IRC00:24
*** bigdogstl has joined #openstack-keystone00:26
*** bigdogstl has quit IRC00:33
*** bigdogstl has joined #openstack-keystone00:34
*** rcernin_ has joined #openstack-keystone00:37
*** rcernin has quit IRC00:37
*** bigdogstl has quit IRC00:39
*** bigdogstl has joined #openstack-keystone00:41
*** bigdogstl has quit IRC00:46
*** bigdogstl has joined #openstack-keystone00:58
gagehugokmalloc looking01:01
*** Dinesh_Bhor has joined #openstack-keystone01:02
*** bigdogstl has quit IRC01:03
*** harlowja has quit IRC01:07
*** bigdogstl has joined #openstack-keystone01:12
*** bigdogstl has quit IRC01:20
wxylbragstad: https://www.lbragstad.com/blog/openstack-summit-vancouver-recap  reading now. ;)01:21
wxyops, misoperation. ignore it.01:22
openstackgerritwangxiyuan proposed openstack/python-keystoneclient master: WIP: functionality for registered limits  https://review.openstack.org/57200601:25
*** r-daneel has joined #openstack-keystone01:43
*** namnh has joined #openstack-keystone01:53
*** Dinesh__Bhor has joined #openstack-keystone02:00
*** namnh_ has joined #openstack-keystone02:02
*** namnh has quit IRC02:02
openstackgerritMerged openstack/keystone master: Correct test_v3_oauth1.test_bad_authorizing_roles_id  https://review.openstack.org/57191202:02
*** namnh_ has quit IRC02:02
*** namnh has joined #openstack-keystone02:03
*** Dinesh_Bhor has quit IRC02:03
*** Rhvs has quit IRC02:06
openstackgerritMerged openstack/keystone master: Decouple bootstrap from cli module  https://review.openstack.org/55890302:07
*** Rhvs has joined #openstack-keystone02:13
*** itlinux has joined #openstack-keystone02:21
*** liuzz has joined #openstack-keystone02:23
openstackgerritHarry Rybacki proposed openstack/keystone master: WIP: Ensure default roles created during bootstrap  https://review.openstack.org/57224302:26
*** gongysh has joined #openstack-keystone02:27
*** lifeless_ has quit IRC02:38
*** lifeless has joined #openstack-keystone02:39
*** dave-mccowan has quit IRC02:54
*** alex_xu has quit IRC03:00
*** alex_xu has joined #openstack-keystone03:01
openstackgerritMerged openstack/keystone master: Correct test_v3_oauth1.test_change_user_password_also_deletes_tokens  https://review.openstack.org/57191303:01
openstackgerritMerged openstack/keystone master: Correct test_v3_oauth1.test_deleting_project_also_invalidates_tokens  https://review.openstack.org/57191403:06
openstackgerritChason Chan proposed openstack/keystone master: Docs: Remove the TokenAuth middleware  https://review.openstack.org/57224803:13
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert Keystone to use Flask  https://review.openstack.org/56837703:14
*** hoonetorg has quit IRC03:23
*** sonuk has joined #openstack-keystone03:26
*** hoonetorg has joined #openstack-keystone03:40
*** bigdogstl has joined #openstack-keystone03:55
*** bigdogstl has quit IRC04:00
*** germs has quit IRC04:01
*** germs has joined #openstack-keystone04:02
*** germs has quit IRC04:02
*** germs has joined #openstack-keystone04:02
*** germs has quit IRC04:06
*** links has joined #openstack-keystone04:22
*** gongysh has quit IRC04:31
*** sapd has quit IRC04:45
*** sapd has joined #openstack-keystone04:45
*** Dinesh__Bhor has quit IRC04:59
*** AlexeyAbashkin has joined #openstack-keystone05:00
*** gyee has quit IRC05:00
*** mvk has joined #openstack-keystone05:05
*** sonuk_ has joined #openstack-keystone05:08
*** sonuk has quit IRC05:11
*** sonuk has joined #openstack-keystone05:12
*** sonuk_ has quit IRC05:15
*** AlexeyAbashkin has quit IRC05:17
*** lifeless has quit IRC05:40
*** AlexeyAbashkin has joined #openstack-keystone05:41
*** lifeless has joined #openstack-keystone05:41
*** gongysh has joined #openstack-keystone05:43
*** Dinesh_Bhor has joined #openstack-keystone05:43
*** AlexeyAbashkin has quit IRC05:44
*** AlexeyAbashkin has joined #openstack-keystone05:44
*** Dinesh_Bhor has quit IRC05:51
*** Dinesh_Bhor has joined #openstack-keystone05:54
*** AlexeyAbashkin has quit IRC05:59
*** germs has joined #openstack-keystone06:03
*** germs has quit IRC06:03
*** germs has joined #openstack-keystone06:03
*** pooja-jadhav is now known as pooja_jadhav06:04
*** martinus__ has joined #openstack-keystone06:05
*** germs has quit IRC06:08
*** links has quit IRC06:08
*** ispp has joined #openstack-keystone06:12
*** jaosorior has joined #openstack-keystone06:14
*** isssp has quit IRC06:14
*** links has joined #openstack-keystone06:26
openstackgerritwangxiyuan proposed openstack/keystone master: Fix db model inconsistency for FederatedUser  https://review.openstack.org/56624206:32
openstackgerritwangxiyuan proposed openstack/keystone master: Enable Foreign keys for sql backend unit test  https://review.openstack.org/55802906:32
openstackgerritwangxiyuan proposed openstack/keystone master: Enable foreign keys for unit test  https://review.openstack.org/55819306:32
*** pcaruana has joined #openstack-keystone06:54
*** pcaruana is now known as pcaruana|worksho07:03
*** sonuk_ has joined #openstack-keystone07:03
*** sonuk has quit IRC07:07
openstackgerritAdrian Turjak proposed openstack/keystone master: Rename token_utils back to fernet_utils  https://review.openstack.org/56620807:34
*** lifeless has quit IRC07:40
*** lifeless has joined #openstack-keystone07:42
openstackgerritAdrian Turjak proposed openstack/keystone master: [WIP] Implement auth receipts spec  https://review.openstack.org/57228607:43
adriantlbragstad, cmurphy, mordred: ^ VERY early code for auth receipts. The logic in theory is right, but it entirely untested and has no unit tests.07:44
adriantWas interesting learning how the provider logic actually worked07:44
adriantthe actual code for making the auth receipt logic work in the auth process is easy... all the work is in that blasted provider layer.07:45
adriantI'll refine it a bit tomorrow, and start actually trying to get it to work, as well as start adding tests.07:46
*** pcaruana|worksho is now known as pcaruana07:50
*** akovi has joined #openstack-keystone07:50
cmurphyadriant: yay \o/07:50
adriantmy goal is to have the code in a fully ready to review state next week, before feature proposal freeze, so I've got time to refine before feature freeze :P07:51
adriantand hopefully start working on figuring out how to make it play nice in KeystoneAuth.07:51
akoviHi Keystone team! Is it a possibility that this patch https://review.openstack.org/#/c/568877/ is causing Mistral devstack jobs to break? http://logs.openstack.org/85/527085/29/check/mistral-devstack/56e57ce/07:51
cmurphyadriant: you are so on top of things07:52
adriantcmurphy: hah, I wish!07:52
akoviIt seems like the config changes have not been released? http://logs.openstack.org/85/527085/29/check/mistral-devstack/56e57ce/controller/logs/apache/mistral_api_log.txt.gz#_2018-06-04_19_23_37_22348707:53
cmurphyakovi: hmm that sure looks related07:55
akovicmurphy: thanks, could you please ask someone to look at it?07:58
openstackgerritAdrian Turjak proposed openstack/keystone master: [WIP] Implement auth receipts spec  https://review.openstack.org/57228607:59
cmurphyakovi: better idea, could you file a bug?08:02
*** germs has joined #openstack-keystone08:04
*** germs has quit IRC08:04
*** germs has joined #openstack-keystone08:04
*** AlexeyAbashkin has joined #openstack-keystone08:05
*** blake has joined #openstack-keystone08:07
*** germs has quit IRC08:08
akovicmurphy: thanks, opened https://bugs.launchpad.net/mistral/+bug/177514008:11
openstackLaunchpad bug 1775140 in Mistral "Keystoneauth does not consistently add the collect-timing parameter" [Undecided,New]08:11
cmurphyakovi: mistral seems to be doing something weird with keystoneauth, it's manually adding the timeout option here http://git.openstack.org/cgit/openstack/mistral/tree/mistral/utils/openstack/keystone.py#n3108:15
cmurphyakovi: it shouldn't be doing that, it should be using keystoneauth's oslo.config register methods like in here https://docs.openstack.org/keystoneauth/latest/migrating.html#step-by-step-migration-example08:15
cmurphythat must be why the new option isn't registered08:16
akovioh, great, I'll try to fix it08:17
akovione question: what should be fixed here? :) I have no clue why the timeout option is registered here in this way08:22
cmurphyakovi: er i guess that doc doesn't show the example, but here's an example from keystonemiddleware http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/keystonemiddleware/auth_token/_opts.py#n22008:22
cmurphyit needs to call register_auth_conf_options with the CONF object and the keystone_authtoken group08:23
akoviOk, I did something. We'll see how far it goes https://review.openstack.org/572300 @cmurphy thanks for your help, very much appreciated08:33
cmurphyakovi: no problem08:35
*** namnh has quit IRC08:37
*** gongysh has quit IRC09:14
*** lifeless has quit IRC09:25
*** issp has joined #openstack-keystone09:28
*** lifeless has joined #openstack-keystone09:29
*** bigdogstl has joined #openstack-keystone09:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/keystone master: Ensure default roles created during bootstrap  https://review.openstack.org/57224309:37
*** bigdogstl has quit IRC09:40
jaosoriorhrybacki: Did some of the missing work there ^^09:41
*** Dinesh_Bhor has quit IRC09:42
*** eandersson has quit IRC09:50
*** blake has quit IRC10:03
*** germs has joined #openstack-keystone10:04
*** germs has quit IRC10:09
*** rcernin_ has quit IRC10:42
*** links has quit IRC11:09
*** akovi has left #openstack-keystone11:10
*** liuzz has quit IRC11:12
*** lifeless has quit IRC11:12
*** lifeless has joined #openstack-keystone11:13
*** ianw has quit IRC11:13
*** pooja_jadhav has quit IRC11:16
*** bhagyashri_s has quit IRC11:16
*** bhagyashri_s has joined #openstack-keystone11:17
*** pooja_jadhav has joined #openstack-keystone11:17
*** edmondsw has joined #openstack-keystone11:19
*** links has joined #openstack-keystone11:25
*** sonuk has joined #openstack-keystone11:30
*** sonuk_ has quit IRC11:30
*** ianw has joined #openstack-keystone11:32
*** AlexeyAbashkin has quit IRC11:36
*** AlexeyAbashkin has joined #openstack-keystone11:46
*** raildo has joined #openstack-keystone12:02
*** germs has joined #openstack-keystone12:05
*** sonuk has quit IRC12:08
*** germs has quit IRC12:10
*** eandersson has joined #openstack-keystone12:19
*** neha_alhat has joined #openstack-keystone12:30
neha_alhatcmurphy mordred: Hi, Want to know that change added in patch[1] is released from keystoneauth library or not? How can I found that?12:33
neha_alhatcmurphy mordred: [1]: https://review.openstack.org/#/c/568878/12:33
cmurphyneha_alhat: it has been released, which i found out by using `git tag --contains 80323289`12:37
neha_alhatcmurphy: Ok, thanks.. will check12:38
neha_alhatcmurphy: Can you please give me high overview of releases of third party lib and how that are used by core projects like cinder,nova?12:48
cmurphyneha_alhat: when we want to release a library we submit a change to http://git.openstack.org/cgit/openstack/releases and the release team reviews and approves it, and then the automation behind the scenes pushes a git tag and pushes a tarball to pypi, then we make an update to http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt to change the upper bound for the library12:52
cmurphyand then the other projects automatically start using it12:52
*** dave-mccowan has joined #openstack-keystone12:54
neha_alhatcmurphy: ok. thank you for your inputs12:55
cmurphyno problem12:56
*** mchlumsky has joined #openstack-keystone12:59
*** r-daneel has quit IRC13:12
lbragstadadriant: awesome - it's good to see that impl up for review13:32
lbragstadlast release we (specifically cmurphy) had a helluva time fighting the gate despite the implementation for application credentials being ready to go :)13:33
*** ayoung has quit IRC13:34
*** ayoung has joined #openstack-keystone13:35
lbragstadkmalloc: i'll trade you flaskification reviews for a look at the hierarchical limits spec13:35
kmalloclbragstad: sold13:36
kmalloclbragstad: but... you need to answer my question re loading middleware/external apps13:36
kmalloclbragstad: before reviewing13:37
kmallocjust "do i need to do those things or is a release-note sufficient for now, and/or can the relnote be a followup"13:37
lbragstadif we just do a release note - then people's custom middleware won't work once we land the flask patches, right?13:38
kmallocright, if anyone has custom middleware they're loading via paste-ini13:39
kmallocmost folks layer things via apache modules or via proxy.13:40
kmallocwsgi middleware is... weird.13:40
lbragstadthe other option would be to support that through configuration/13:42
kmallocthat was the way i';d implement it13:47
kmallocthrough keystone config.13:47
kmalloci'll need to add a conditional for the oslo_debug middleware, but i could make it more generic13:47
kmallocloading in API endpoints/extensions is something i'm not thrilled to do, but i *could* implement that as well.13:47
kmalloci'm inclined to say "no, we don't do that"13:47
lbragstadcmurphy: knikolla thoughts* ^13:48
kmalloc"use a separate wsgi app for that"13:48
kmallocso 2 questions: 1) Loadable Middleware (before our pipeline)13:48
cmurphyi'm a little too far removed to have an opinion right now13:48
kmallocand 2) Loadable wsgi routes to some other app not-keystone (something embeded in our APi)13:48
kmallocayoung: ^ you have customer like interactions13:48
kmallocayoung: thoughts?13:49
kmallocayoung: context, flaskification patches drop paste.deploy, so without added code you can't wedge new middleware into it (as a deployer) and/or cannot load an extension directly in as a routable path13:50
kmallocpaste let people totally muck with what was run before, during, and after keystone13:50
cmurphyit's not just custom middleware though, is it? things like healthcheck or osprofiler or s3 stuff is all configurable in paste13:50
ayoungWe do Tripleo.  Break that and all hell escapes13:50
kmallocayoung: what are you loading in?13:50
kmallocbecause paste is a trainwreck13:51
ayoungpaste is just another file.  Leave it there but empty so the install process does not break and Tripleo will be fine13:51
kmallocand should have died a long time ago (it's a dead/unmaintained project as well)13:51
lbragstadcmurphy: those things are converted over to flask apps since we include them by default13:51
ayoungI suspect that someone out there is doing custom middleware, but RH would not support it13:51
kmallocno, no, not talking about paste being empty, i mean we don't use paste.ini at all13:51
cmurphylbragstad: okay then what if people want to turn them off?13:51
kmallocayoung: i don't care about empty unused files.13:51
ayoungSo, some alternative way to load it instead is probably OK13:51
lbragstadit's still configurable (at least osprofiler is)13:52
cmurphyah ok13:52
kmalloccmurphy: most of the middleware is not really turn-off-able without other headaches. healthcheck, osloprofiler lines up the list13:52
kmallocand osprofiler has it's own config options13:52
kmallocs3 middleware is application side, keystone has the s3 api baked in.13:53
ayoungkmalloc, so I can give the entirely selfish and non-community supporting advice of KILL IT! KILL IT DEAD!13:53
kmallocayoung: so, is triple-o adding or changing middleware in keystone?13:53
ayoungbut only cuz I know it won't affect me13:53
kmallocayoung: i am in the same boat, trying to kill it. but trying to be reasonable13:53
ayoungnah, Tripleo might have to adjust to the lack of the file, but would be otherwise unaffected13:53
kmallocok. cool13:54
kmalloccmurphy: i'm planning on adding a wsgi conditional for debug middleware, but if we need something more configurable, i'll add more configuration-y things13:54
ayoungA replacement paste.ini file saying "this file is no longer used and will be removed in a future release.  If you were modifying this file, you were in a state of sin and you are stuck."13:55
ayoungOrt something a littel nicer maybe13:55
kmallocayoung: comment here: https://review.openstack.org/#/c/571979/ to keep the file but not use it (with comments in it)13:55
kmallocayoung: please. so i don't forget13:55
kmallocayoung: and why it's important (just a, so we are nice to triple-o is fine)13:56
cmurphyokay fwiw i just checked and neither of suse's cloud products allow for custom paste configs for keystone13:56
kmalloccmurphy: almost nobody changes the paste-ini, we found this with the middleware deprecation(s)13:56
lbragstadcmurphy: ++13:57
kmallocand if they did, they did changes we no longer support.13:57
kmalloccmurphy: thanks for checking :)13:57
cmurphywe do seem to muck with it for trove for some reason but that's not really relevant here :)13:57
kmallocright, i'm concerned with keystone specifically :)13:57
*** r-daneel has joined #openstack-keystone13:59
ayoungkmalloc, done14:01
cmurphylbragstad: I finally got around to looking at the limits spec, I'm happy with it if kmalloc or ayoung want to push it through but i noted some concerns14:02
knikollareminder about the edge computing meeting14:02
kmallocknikolla: thanks14:02
kmallocknikolla: what was the info on that meeting?14:02
ayoung"strict two-level hierarchical"  still?14:02
kmallocayoung: yes, that is what was agreed upon initially14:03
cmurphyayoung: ja14:03
kmallocbut we can add in the "store upwards" comments if they aren't there14:03
ayoungI thought we realized that was neither necessary nor sufficient14:03
ayoungit is the size of the tree, not the depth, that matters14:03
kmallocright and the store upwards work is relevant in either case14:03
*** germs has joined #openstack-keystone14:06
*** germs has quit IRC14:11
ayoungso, I think that 2 level is broken.  Either people are not doing hierarchy at all, or they are making a decent use out of it, and we can't assume that they are 2 level14:14
ayoungthat means we have no quota for the normal case14:14
ayoungand, if this is where the effort goes, it is going to cause a lot of churn14:15
ayoungand...I can't see where it addresses the fact that endpoints don't communicate.14:16
ayoungIt is one user hard coding a solution that meets their needs at the expense of the normal case.  I can't see why we would not reject that.14:17
*** vrv_ has joined #openstack-keystone14:18
*** spilla has joined #openstack-keystone14:22
ayounglbragstad, how wedded are you to the idea of 2 level?14:22
kmallocayoung: we already are planning more enforcement models.14:26
kmallocthis was the agreed upon 1st hierarchical one14:26
kmallocbut, i'm open to expanding to more depth, just the issue is the "fluid" quota part14:26
ayoungkmalloc  so the only thing that is fundamentally broken (and this will be for all) is the per-endpoint part14:26
kmallocwhere quota might shift from one branch of the tree to another non-intuatively14:26
kmallocayoung: right, and I'm going to take a stab at that14:27
*** itlinux has quit IRC14:27
ayoungbut...I think 2 level is a mistake in communication14:27
kmallocbut we still need the enforcement model(s)14:27
ayoungpeople are going to Hear "keystone only supports two levels"14:27
ayoungand I can see doing damage control for that for years14:27
kmallocthat is a fair reasoning to push to a different model14:28
ayoungI'd rather make a push to get the proper tree stuff working, and then 2 level can be an option after14:28
kmalloccmurphy ^ thoughts? the communication bit is important.14:28
ayoungbut we need to solve per enpoint no matter what14:28
ayoungnext cup of coffee is going to be Death Wish.  Thank you kmalloc14:29
cmurphyi wasn't able to be at the quota session in yvr but in the ops feedback session and in dublin the feedback we got was that 2-level solved the 80% use case14:29
* ayoung needs to dig out grinder14:29
kmallocayoung: ^_^14:29
ayoungcmurphy, that leaves 20% in the lurch.14:29
kmalloccmurphy: thats fair, i would like to figure a way to avoid the comms issue though14:29
cmurphyayoung: which we'll address after we get *something* working14:29
cmurphyif we try to hit the 100% we'll never get anything done14:30
kmallocayoung: i'm ok with an 80% solution if we can clearly communicate future/forward14:30
kmallocand not need damage control14:30
ayoungthe question is, if the multi level is the same amount of complexity, does 2 level get us anything?14:30
kmallocsoftware wont be perfect.14:30
*** dklyle has joined #openstack-keystone14:30
ayoungand...could we do 2 level in the context of full?14:30
kmallocayoung: i think the 2-level buys us less non-intuative quota affecting the tree in weird ways14:30
kmallocthat was the key issue14:30
ayoungthere is that, too.14:30
kmalloca->b and a->c->d, where d may not have quota because B consumed it all14:30
kmallocthat was the biggest concern, more than the technical "this is expensive to calculate"14:31
ayoungyeah, I hit the same issue in my analysis14:31
ayoungOh, wait14:31
ayoungno, A NEEDS to be able to set Quota on B and D14:31
* lbragstad is in a meeting14:31
lbragstadbut i'll read scroll back later14:32
ayoungA can't overcommit on quota14:32
kmallocright, it's overcommit14:32
ayoungso...I just reread what you wrote and youi are correct14:32
kmalloclbragstad: the edgecompute one?14:32
ayoungthe quotas are additive per level, so if A messes with his people, they get starved14:32
cmurphy-_- why are community meetings not in irc14:33
kmalloclbragstad: i want to sync w/ you after, i couldn't get zoom working (need a vm) because i don't trust .deb packages from websites14:33
kmalloclbragstad: i'll plan to have a VM setup for the next one as needed.14:33
kmalloccmurphy: because that is what they chose =/ i don't like it14:34
kmallocbut people want video calls14:34
cmurphythe reason we settled on 2-level was we kept running into logic issues trying to work out more than three levels, sdague documented in https://review.openstack.org/44120314:34
cmurphywe want to start getting the integration with oslo.limits working and so we need something to work with, after we can do that we can have a second stage with user feedback for what actual use cases there are14:35
kmallocright, i'm not expecting a perfect solution14:36
kmalloci want to make sure we have a good solution for that 80%, but not shooting ourselves in the foot comm wise14:36
*** spilla has quit IRC14:36
lbragstadcmurphy: i'm not sure :( to be fair... i'm not sure I've seen any of these folks in irc14:36
kmalloclbragstad: yeah edgecompute is much the same as some of the other groups that are not as irc-centric. a lot of the business world (and this group is business world) tends to video-call.14:37
lbragstadildikov: is recording things, too14:37
* kmalloc nods.14:38
kmallocis there a good person to bring up that zoom.us is not super friendly to some folks and that there wasn't a dial-in phone# i could use.14:38
cmurphykmalloc: we're documenting a limits model endpoint to hit, once this is implemented there will be two different models available, I don't really buy that anyone would assume we're planning to stop there14:38
kmalloci really wish i could have been there.14:38
kmalloccmurphy: right. lets just make sure we're very clear on where we aren't stopping :)14:39
knikollathere's the other edge group which does irc meetings IIRC14:39
cmurphykmalloc: thumbsup14:39
kmalloccmurphy: i agree with ayoung it's important to make sure we aren't in damage control mode14:39
knikollathe fog edge massively distributed clouds group14:39
kmallocbut having a workable solution for integration purposes is equally important14:39
*** spilla has joined #openstack-keystone14:40
*** eandersson_ has joined #openstack-keystone14:45
*** eandersson has quit IRC14:45
ayoung ERROR, Child limits exceed parent limits.14:46
ayoungThat is the heart of it.  Which means that you always check both the child limits and the parent limits.  Period14:47
ayoungthat is from Garbutt's doc14:47
ildikovlbragstad: kmalloc: recording of the meeting will be posted on the Edge wiki soon15:03
ildikovlbragstad: kmalloc: I will try to encourage people in the future to try to take some notes on the IRC channel for easier recap on important info15:04
lbragstadildikov: kmalloc cmurphy knikolla my rough notes https://gist.github.com/lbragstad/913dcb4e79495beadbb64f855f6a619c15:04
cmurphyildikov: some official-ish minutes would be nice, it's hard to digest an hour-long recording after the fact :)15:05
*** pcaruana has quit IRC15:05
ildikovcmurphy: I know, I will try to write up a summary some time this week15:06
ildikovcmurphy: I wanted to ask people at the beginning of the meeting to take notes, but forgot15:06
ildikovcmurphy: and despite of being a woman with good multitasking skills, I can't type and talk at the same time... :/15:07
cmurphyildikov: that's okay, i can barely do either of those things one at a time ;)15:07
ildikovcmurphy: yeah, I can relate to that too :)15:08
*** wxy| has joined #openstack-keystone15:15
openstackgerritwangqi proposed openstack/oslo.limit master: fix broken link  https://review.openstack.org/57244215:18
*** itlinux has joined #openstack-keystone15:19
openstackgerritmelissaml proposed openstack/keystone-specs master: Replace Chinese quotes to English quotes  https://review.openstack.org/54477315:19
openstackgerritPavlo Shchelokovskyy proposed openstack/keystone master: Filter by entity_type in get_domain_mapping_list  https://review.openstack.org/57244615:25
lbragstadayoung: responding to your question about 2 levels versus 315:30
lbragstadthe decision to go to 2 levels was primarily driven by support use cases15:30
lbragstadand usability15:31
lbragstadif someone has limit use cases that don't fit the strict two level, then they can use flat and manually adjust the knobs how ever they want15:32
lbragstaduntil we work out an enforcement model that works for their use cases15:32
lbragstadafaik, that was the primary driver behind the concept of enforcement models15:32
lbragstadkmalloc: as far as the flask bits go15:34
lbragstadkmalloc: if we keep things configurable with what we have by default today, then i'm fine15:34
*** lifeless_ has joined #openstack-keystone15:35
lbragstadkmalloc: that said, it might be worth a note to the mailing list raising this issue, just describing what we want to do and what that means for people relying on it - if there are any15:35
lbragstads/mailing list/operator and development mailing list/15:35
*** lifeless has quit IRC15:36
lbragstadwe're not going to be, or probably aren't the first project faced with this decision15:36
lbragstadwith the amount of projects on paste and looking to move from it15:36
kmalloclbragstad: we wont have the same level of configurability in flask without paste15:37
kmalloclbragstad: fundamentally15:37
kmalloclbragstad: so but...15:37
kmalloclbragstad: so you want me to enable custom middleware via options15:38
lbragstadright - what i meant is things like the ability to turn osprofiler on and off15:38
kmallocoh ok15:38
openstackgerritHarry Rybacki proposed openstack/keystone master: Ensure default roles created during bootstrap  https://review.openstack.org/57224315:38
kmallocthen that is a non-issue, i wont be changing that15:38
lbragstadlike - what we have by default in paste today15:38
kmalloci'll add a toggle (followup) for debug15:38
kmallocsince we can do that today15:38
kmalloci'll create a new [wsgi] group for future things like that15:38
lbragstadthat'd be good15:38
kmallocif we need to enable custom middleware, we can add that functionality to [wsgi] group15:39
kmalloci'll get release note, and toggle done15:39
kmallocthen mail to the ML for /v2.015:39
kmallocand then we can land flaskification15:39
kmallocand i'll start work on the blueprint stuff and code shuffle15:39
kmalloclet me go review limits15:40
kmalloclbragstad: what review # is the limits spec?15:41
kmallocfound it15:41
kmallocayoung: that spec cannot explicitly deal with per-endpoint15:42
kmallocayoung: it is not in the scope of the spec itself, per-endpoint is a followup15:42
kmallocwe don't have the mechanism to share the data yet15:42
ayounggotta be addressed or it is going to be assigned quote * endpoint15:42
kmallocthat is currently, acceptible and how it works with the current quota system15:43
ayoungmake it explicit15:43
kmallocplease move your -2 to a -1 and comment i'll not kick it through if that level of explicit is needed15:43
kmallocheck i'll respin to add that15:43
kmallocbut please don't hold up if that is the -2, that is a -1 worthy comment (and should be addressed before landing)15:44
kmallocif the -2 is because of insufficiency, we need to get you, lbragstad, and cmurphy on the same page.15:44
kmallocso we can move it forward. i am not clear on the -2 reason.15:44
lbragstadayoung: i need more detail about the -2 - otherwise you're leaving me fishing through your blog post hoping wxy and i can figure out what the issue is15:45
*** mchlumsky has quit IRC15:45
kmalloci am 100% ok making it explicit that this does not address per-endpoint quota and that will be built on top of what we're doing now15:45
kmallocftr. and i think we should call that out15:46
ayounglbragstad, per endpoint is essential.  And I would like a decent argument that 2 level is more important than solving multi level other than "we spent more time on it"15:46
ayoungI am not convinced that 2 level is a good thing.15:46
lbragstadayoung: i already answered that question, it was related to support15:46
ayoungI am convinced that multi-level is needed, and will solve 2 level15:46
lbragstadif you have a tree of more than 2 levels, and a cousin project uses the last 2 cores of the entire tree15:47
*** mchlumsky has joined #openstack-keystone15:47
ayounglbragstad, what is going to happen is we are going to get 2 level, and then we are going to turn our attention elsewhere15:48
kmalloclbragstad: it's the fluid quota model with oversub.15:48
lbragstadwhat am i supposed to put in the support ticket if i'm a user on the other side of the tree?"15:48
kmallocthat is the issue there15:48
ayoungthe 2 level as definied here is , I suspect, still vulnerale to gaming15:48
kmallocit's oversub vs strict.15:48
*** wxy| has quit IRC15:48
ayoungmulti level with decent error messages:15:48
lbragstadi isn't susceptible to gaming when you calculate usage of the tree15:48
lbragstadit isn't*15:48
*** wxy| has joined #openstack-keystone15:49
kmallocstrict [no oversub], while game-able [in some cases, see adam's blog and i can explain further], doesn't suffer from the weird error state15:49
ayoung"The action would exceed your quota assigned by 'Parent'."  Versus "the action would exceed quote for 'Parent' assigned by 'System'.15:50
openstackgerritwangqi proposed openstack/oslo.limit master: fix link  https://review.openstack.org/57244215:50
ayoungso..get the endpoint thing at a minimum.  But with edge, I really see that as a deal breaker15:50
*** jmlowe has quit IRC15:50
ayoungwe need to be able to assign quota per endpoint.15:50
kmallocayoung: ok, so if we solve the per-endpoint bit [forward looking/explicit statement on what is implemented there] can the -2 be removed15:50
ayoungdoes not have to be for all quota15:50
kmalloci can add in the per-endpoint stuff15:51
ayoungI don't love it, but sure15:51
ayoungI'd rather you did it right, as it is going to cause more work for everyone, but not a hill for me to stand on15:51
kmallochonestly i don't understand the need for oversub on children15:52
kmallocbut my view is largely oversub is bad(tm) in quotas.15:53
lbragstadi'm still missing context as to why everything is terribly broken15:53
kmalloclbragstad: it isn't.15:53
lbragstadit certainly sounds like it is15:53
ayoungkmalloc, is oversub a "two layer" thing ?15:53
kmallocoversub is the reason for two-layer limit15:53
ayoungwhat is it called in there?15:54
kmallocit is about UX.15:54
kmalloc[end user ux]15:54
kmallocit is the argument that a->b and a->c->d causes issues if b consumes all the quota15:54
kmallocand d doesn't get any when it should have available15:54
kmallocbecause D's quota + b + C > a15:54
kmalloccommunicating that with N depth is very very very hard to do clearly15:55
kmallocwithout oversub on children, non-issue15:55
kmallocN depth is 100% a-ok15:55
kmallocand not hard to communicate clearly15:55
kmalloc"you are over quota"15:55
kmalloclbragstad: i'll add details about per-endpoint sharing of quota15:55
lbragstadi'd like to understand it before you push a new version15:56
kmallocshould be straightforward to do, which should reolve the bulk of the -215:56
ayoungEndpoints are going to break this15:56
ayoungsame issues with gaming15:56
kmallocbasically, "this initially doesn't share quota across endpoints, so each nova is granted the entire volume of quota"15:56
ayoungI want to split my quota over two endpoint, I want to delet quote from one endpoint, etc15:56
ayoungwe had, like 5000 meetings on edge at the summit15:56
kmalloc"we will build on top of this, use of etcd to store the shared quota state"15:57
ayoungis that really a viable approach?15:57
ayounganyway, I'll downgrade to -115:57
kmallocayoung: ^ is that sufficient (better words to come)?15:57
*** bigdogstl has joined #openstack-keystone15:57
kmallocbefore i start generating all the verbiage and updating the spec.15:57
kmallocand yes, i will commit to writing the bulk of that etcd bit.15:58
ayoungkmalloc, I really don't know what is the right solution.  The division of services we have in OpenStack may make it an impossible problem to solve, and I don't have the cycles to spend on it now15:58
kmalloci am asking from a unblock the spec perspective.15:58
ayoungmaking changes in keystone without communicating them to the services seems "broken by design"15:59
ayoungI unblocked15:59
kmallocbut you're still not really answering my question15:59
kmallocif that had been in the spec, would you have -2'd based upon per-endpoint data sharing issues15:59
ayoungkmalloc, I think that per endpoint might be too hard to solve, so the comment you have there is fine.16:00
ayoungAnd that was the deal breaker that lead to the -216:00
*** gyee has joined #openstack-keystone16:00
ayoungI don't think 2 level really is the right approach, but I'm not implementing, so I can live with it16:00
*** bigdogstl has quit IRC16:02
*** links has quit IRC16:03
*** germs has joined #openstack-keystone16:07
*** germs has quit IRC16:07
*** germs has joined #openstack-keystone16:07
*** germs has quit IRC16:09
*** germs has joined #openstack-keystone16:09
*** germs has quit IRC16:09
*** germs has joined #openstack-keystone16:09
*** germs has quit IRC16:09
*** germs has joined #openstack-keystone16:10
*** harlowja has joined #openstack-keystone16:28
lbragstadrm_work: ping16:33
*** issp has quit IRC16:44
*** harlowja has quit IRC16:45
*** bigdogstl has joined #openstack-keystone16:53
knikollakmalloc: the x1?16:58
knikollayoga or carbon16:58
*** jaosorior has quit IRC16:58
lbragstadrm_work: fyi - http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-06-05-16.00.log.html#l-16416:58
kmallocknikolla: i can have an x1c6 [1080p], P50, T480 [1080p], or macbook pro 15/13 or macbook air16:59
knikollakmalloc: no hidpi screens?16:59
kmallocknikolla: i think i'm going to go x1c6, but honeslty it wont replace my current x1c4 because 1080p17:00
kmallocknikolla: not offered as the "standard" configs17:00
kmallocall 1080p17:00
kmallocmakes me sad.17:00
kmallocmy x1c4 is a personal machine.17:00
kmallocand hidpi is a hard requirement for me to use anything else.17:00
knikolladamn :/ non hidpi is a blocker for me.17:00
knikollai asked for an xps 13 with a 4k screen. but answer was let's talk again end of summer.17:01
kmalloci'll buy a laptop for myself and use it over a corp one without hidpi17:02
*** masuberu has quit IRC17:02
kmallocthough, $2k+ isn't in my budget right now.17:02
knikolla++, that was what i was gonna do until i realized i couldn't afford it.17:02
kmallocso, x1c4 for the time being [even though it's on it's last legs]17:02
*** masuberu has joined #openstack-keystone17:02
kmallocspeaker busted, WWAN sim no longer reading, screen issues, battery issues, and warranty is up17:03
knikollathat sucks.17:03
kmallocand you saw the laptop, it wasn't in bad shape/abused (at the summit)17:03
knikollayeah, i'm suprised. thinkpads usually last forever.17:03
knikollamacbook pro (late 2013) is still working flawlessly17:04
knikollaand ubuntu works amazingly well17:04
lbragstad#startmeeting keystone-office-hours17:04
openstackMeeting started Tue Jun  5 17:04:49 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.17:04
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:04
openstackThe meeting name has been set to 'keystone_office_hours'17:04
*** wxy| has quit IRC17:11
rm_worklbragstad: ah yeah, i thought we basically came to a conclusion on that in-channel at the time, and that we'd review the available developer volunteers and approaches in Denver at the PTG, but that the general thought was "YES WE WANT THIS" it just wasn't prioritized17:25
rm_workso i didn't think we needed to discuss it in the meeting :P17:25
rm_workbut, reading through that now17:25
*** lifeless has joined #openstack-keystone17:26
rm_worki THINK people had the right idea17:28
*** AlexeyAbashkin has quit IRC17:28
*** lifeless_ has quit IRC17:28
rm_workwe really shouldn't be *removing records*. and yeah, obviously a deleted project would not be significantly different than a disabled one -- honestly delete just being an alias to disable is practically acceptable from my PoV, it's just a terminology issue17:29
rm_workwe can't retrain end users to "please use disable instead of delete" because that just isn't feasible17:29
rm_workimagine trying to get every random user of your cloud (who you mostly have no power over, they're just people who sign up / pay you) to use a different method that doesn't mesh with every other project in existence17:30
rm_work"When we don't want a resource anymore, we delete it" <-- is what everyone thinks, for everything17:30
rm_worki wasn't ever saying we need an undelete (though I'm sure it could essentially be done manually without too much trouble, just to get the project itself back in an urgent case)17:31
openstackgerritMorgan Fainberg proposed openstack/keystone-specs master: Hierarchical Unified Limits  https://review.openstack.org/54080317:31
lbragstadkmalloc: i was just about to start reading this17:32
kmalloclbragstad: ^ addressing ayoung's concerns with the explicit endpoint data sharing and some reasoning for the strict two-level, calling out we allow oversubscription17:32
rm_worki think ayoung and I agree in principle and he's on the same page as I am generally17:32
kmallocrm_work: the way i would accept this change (soft delete): change how delete works to always be soft delete, still do the "delete" process but maintain records17:33
rm_workthat's basically all i want17:33
kmallocrm_work: i am a hard -2 for a "new and special delete"17:33
kmallocall deletes should become soft (for projects and other must maintain records)17:33
rm_workdelete should just ... set the disabled flag, hide the record from returning in lists17:33
kmallocno, do not used "disabled"17:33
kmallocdelete should still remove grants17:34
rm_workah, k, yeah, that's prolly fine17:34
rm_worki just mean "it should also do all the same things disable does"17:34
rm_workIE, can't create tokens17:34
kmallocdisable is a lot less invasive17:34
ayoungwhat is the word for the trivial case of something17:34
rm_workand whatever else, because a deleted project is a superset of disabled17:34
kmallocdisable is toggle, soft delete should never [really] expected to be toggled back except <extreme> cases17:35
ayounglike...when a complicated solution has a base case that could be covered by a simpler solution?17:35
kmallocyou will also need a new API to purge records [sanely] or a keystone-manage command17:35
kmallocbased upon <before datetime>17:35
rm_workyeah i imagined something like what ... someone brought up, one sec17:35
lbragstadstepping away to grab lunch quick and i'm going to go through the unified limit stuff17:35
*** isssp has joined #openstack-keystone17:35
rm_workah yeah, you17:36
ayoungkmalloc, I'm with you on the "don't just use disabled" etc.17:36
rm_work16:43:25 <kmalloc> and track deleted time with a "cleanup" option so "remove records for projects with deleted and deleted before <datetime>17:36
kmallocayoung: ++ disabled has it's use.17:36
rm_workactually i just said that because i thought it'd be MORE palatable17:36
ayoungrm_work I hear ya17:36
kmalloc"you didn't pay, we're turning your spigot off"17:36
rm_worki don't really care how it works, as long as the project DB record stays in the DB, and doesn't show up to users but does show up to admins17:36
kmallocdelete is "you still haven't paid and we're closing your account"17:37
kmallocrm_work: you'll probably want a new api for deleted record introspection17:37
ayoungand we should be able to disable, but still get tokens to delete resources for a disables project...maybe with app creds or something.  But that is a different feature17:37
rm_workand further deletes and such should probably 404? but that's getting deep into implementation details17:37
kmallocayoung: yep, different features17:37
kmallocrm_work: if it's already deleted it works the same as already deleted today ;)17:37
rm_workyeah, for most other projects you can do a ?deleted=True and it'll allow showing/listing deleted objects17:38
kmallocrm_work: sure, just make sure it's independent of disabled=True or whatever we use for that17:38
rm_workyes, for all intents and purposes to a regular user it should seem like nothing changed from the way it works now17:38
rm_workbut for admins, we should be able to go in and still see the project if we want17:38
kmallocso the (implementation detail warning) change is a going to be a new column and some constraint changes for project_name17:39
kmallocas delete frees project name17:39
kmallocand some changes to how delete propagates (delete from -> update)17:39
*** ispp has quit IRC17:40
kmallocif you make a unique constraint and deleted can be null, be aware that null is never considered to collide in mysql17:40
kmallocso (PName, Null) doesn't collide with (Pname, Null)17:40
rm_workheh, noted17:40
rm_workyeah i honestly am not 100% sure whether I WILL be able to work on this or not, by the time Denver rolls around. will see.17:40
rm_workI would like to. I don't at first glance feel like this should be too involved17:41
rm_workbut i am sure it will rabbit-hole quickly ;P17:41
ayoungso what is the word.  The 2 level quota is the (blank) of the multi level approach17:41
ayounglogical extrame?17:41
ayoungrm_work, not quite the word I am looking for...that means the essense, and I am looking for the "trivial case" or something17:44
ayoungand maybe that is the term I want?17:44
ayounganyway, I see no benefit of 2 level.  It requires the same overhead as the multilevel17:44
ayoungjust forces you to a flat, wide tree, but it will be just as expensive to calculate and query17:45
ayoungFor any project, we need to know its parent17:45
ayounginside of nova17:45
ayoungthe multi level will then chain from parent to grand parent and so on17:46
ayoungthe two level will not, but the same amount of info needs to be synced between keysteon and e.g. nova17:46
ayoungwe can put restrictions on people moving quota around such that you can never allocate more quota to your child projects than you yourself have, and then there is no "you parent quote was exceeded"17:47
ayoungthat is just a gotcha that we need to make sure we cover, not fundamental to the two level17:47
ayoungso, yeah, 2 level dumb17:47
kmallocexcept even in the case of that, you run into17:47
kmallocnon-strict enforcement17:47
kmallocor does it.17:48
kmalloc*me drinks more coffee*17:48
kmallocright so no oversub quota17:48
ayoungno over sub quota17:49
kmallocayoung: oversub is the fundamental piece17:49
kmallocthat dictates need for two-level17:49
ayoungkmalloc, ah, I reemmber17:50
ayoungyeah, the oversub is a side effect of not squaring things with Keystone17:50
ayoungit is kinda fundamental, but I think 2 level suffers from it as well17:50
kmallocbut oversub is a method that people do use17:50
ayoungsay A->B  A->C each has 4 units of a parent 8 unit quota17:50
kmalloc"i give you 100 cores, and each child 100 cores even though i only have 100"17:51
kmallocthe quota system can tell youre in aggregate over17:51
ayoungB maexs out17:51
kmallocbut people like being able to oversub and then just buy up when children aggregate hit limit17:51
ayoungmaxes out at 4, C has 017:51
ayoungA then removes from B, creates D17:51
kmallocoversub is a headache, but common practice17:51
ayoungA->D gets 417:51
ayoungD allocates all17:52
kmallocno, the not squaring with keystone is solved with the store upwards concept17:52
ayoungC still has 4 quota, but it puts A over the limit17:52
kmallocso we can be sure we can (cheaply?) calculate the total usage17:52
kmallocoversub is a choice made on representation to child projects17:52
ayoungstore upwards means what?  That A cannot remove quota from B?17:53
ayoungCuz that is really where the oversub comes from17:53
kmallocno, consumption of quota is stored up to the parent17:54
kmallocin aggregate17:54
ayoungso A cannot assign quota to B?17:55
ayoungno hierarchy? That seems to contradict the name of the spec17:55
kmallocso when i make a claim on quota in C (a->b->c) it stores a record upwards that consumption for C is used and A gets a record of aggregate of b+c17:55
kmallocit's about storing claim aggregate usage upwards17:55
kmallocin the hierarchy17:55
kmallocit means worst case to check usage we go to the top of the tree17:56
kmallocthat doesn't have to square with keystone ever17:56
ayoungbut the cost is based on the number of nodes in the tree, not the depth17:56
kmallocbecause if A tries to game the system i am fine with saying "over quota"17:56
kmalloci am less ok with something D does affecting a different branch of the tree17:56
kmallocunder a17:56
kmalloca->b->c and a->d17:56
ayoungyeah, I don't think that is possible17:57
kmallocwith oversub it is17:57
ayoungits omnly possible if A plays games, not D17:57
kmallocoversub means C+B+D might have more "allowed [not consumed]" quota than A17:57
kmallocso D could use all the quota for B and C17:57
kmallocwith no games being played by A17:57
ayoungLet me mull it over to see if there are peer-to-peer games, but I think that only A can do *that* to its Pledges17:57
kmallocthat is bad UX17:57
ayoungoversub is only possible if A plays games17:58
ayoungby not removing actual resources when it removes quota17:58
kmallocin the model proposed oversub is explicitly allowed without games17:58
kmallocA may allocate it's entire quota to A, B, C, and D17:58
kmallocat the same time17:58
ayoungthat is just "don't divide the quota on sub projects" and is suppored by multi level as well17:58
kmallocbut the consumption check prevents use once A's quota is hit17:58
ayoungyeah, that is covered in multilevel17:59
kmallocso, 2 fundamental issues:17:59
ayoungyou are saying the fact that A can have deep trees just makes it more surprising?17:59
kmalloc1) Oversub makes for icky ux beyond ~2 layers17:59
ayoungGot it17:59
ayoungwhat if we don't allow oversub beyond to levels?17:59
kmallocbecause D has no insight to it's peers really17:59
kmallocso it's very surprising when D runs out of quota17:59
ayoungthat was the idea of quota pools18:00
ayoungthe pool id is the id of the project that owns the quota18:00
ayoungwhich is you or someone in your tree18:00
kmalloci think the solution to store aggregate consumption to the parent(s) is easier than needing to defrag quota pools.18:00
ayoungkmalloc, perhaps18:00
kmallocit's net the same thing.18:00
ayounggot ameeting18:00
kmallocchat later18:01
*** jmlowe has joined #openstack-keystone18:01
lbragstadok - does quota mean limit or usage?18:03
lbragstadwe've been referring to limit and usage as two distinct, but related, things18:04
lbragstadand i'm not exactly sure what people mean by quota now18:04
*** bigdogstl has quit IRC18:04
*** bigdogstl has joined #openstack-keystone18:09
kmalloclbragstad: i always specify Limit as what is stored in keystone (allowance)18:09
kmalloclbragstad: and quota claim/consumption as what is used (not stored in keystone)18:09
kmallocjust for clarity, quota is BOTH things, so it needs a specifier18:10
kmallocyou can't say quota without being explicit what side you're looking at18:10
ayoungkmalloc, OK,  store up can and should be done even in the multi-level approach18:10
kmallocwe may need to update those docs to reflect it18:10
ayoungthere mechanism is this:18:11
kmallocayoung: totally i added a comment to the spec to highlight it18:11
lbragstadkmalloc: how so? those docs look fine to me18:11
ayoungthere is no distinction between allocating quota for a new resource or splitting for a sub quota18:11
kmalloclbragstad: if we are unclear18:11
lbragstadthey describe keystone as being the place for limits and usage being calculated by services18:11
kmalloclbragstad: right i haven't dug deep, i said "may" assuming we crossed the definitions somewhere18:11
kmallocit wasn't a "it's wrong" comment, sorry, split attention between the two docs (new spec, old current documentation)18:12
ayoungso A->B->C  add a resource to C  we add the resource to C, reduce 1 fro C available, bubble up, reduce 1 from B available, bubble up reduce 1 from A.18:12
lbragstadso - is a "quota system" still referring to the coordination between limits and keystone and usages at the service?18:12
kmallocayoung: basically.18:12
kmalloclbragstad: both.18:12
lbragstadlimits in keystone*18:13
kmalloclbragstad: you really can't have one without the other (well you can, but it's silly)18:13
kmallocyou need both limits (allowance) and usage18:13
lbragstadright - and when people reference quota, that's what they are talking about18:13
lbragstadok - that helps18:13
kmallocif you're tracking usage without limits for billing purposes, it's just "usage"18:14
kmallocquota adds in a allowance/cap18:14
ayounglbragstad, so you still want to pursue 2 level?  And if so, only due to pressure to get spec approved?18:14
lbragstadayoung: i'm still about 10 steps behind understanding the reason why you don't like it18:14
lbragstadi'm trying to catch up18:15
ayounglbragstad, it think the 2 level restriction is not really buying anything, and is as expensive as the multi-level18:15
ayoungwhy half-ass something when you can whole-ass it?18:15
* ayoung apologizes18:16
lbragstadayoung: ok - so 1.) you don't see the value in it 2.) you consider it still expensive, yeah?18:16
ayounglbragstad, and 3) we will be doing damage control once we expand to the multi level18:17
lbragstadwhat makes you think it doesn't buy us anything?18:17
kmalloclbragstad: if we have to look up every child, it is expensive beyond a very small number of children18:17
lbragstadok - let's just start with one18:17
ayoungright, and that is not based on depth of tree, but on number of children18:17
lbragstadfor the sake of me being slow18:17
ayoungIf we have a wide flat tree with 1000 nodes the work is the same as a deep tree with 1000 nodes18:18
kmallocso, i wont comment to the two/not-two level. i am not opposed/for it [with exception of oversub/non-oversub ux issues]18:18
ayoungAnd I won't hold things up, just want it to be a deliberate decision18:19
lbragstadby work are you referring usage calculation?18:19
kmalloccollection and calculate usage when making a new claim18:19
kmallocthe "are we over quota" check18:19
ayoungand same for storage requirements18:19
lbragstadfwiw - the reasonsing being two-level wasn't because calculating usage was harder for either18:19
lbragstadif you have 1000 projects, usage calculation will be the same regardless of what the tree looks like18:20
lbragstadso - sure, i'll agree there18:20
lbragstadthe main reason for the two-level bit was from dublin18:20
kmallocand that issue, i have a solution for [see comment on spec] the wide tree issue18:20
lbragstadin filling out support tickets18:20
lbragstadif someone can't create an instance because someone in a third cousin project used up the last of the limit allocation, what is that person supposed to put in a support ticket?18:21
lbragstadthat is actually useful for operators?18:22
lbragstadwith two level project hierarchies, everyone knows the parent18:22
kmallocis oversubscription of children actually useful? i don't have an answer18:22
*** fiddletwix has joined #openstack-keystone18:22
kmalloci'm not being rhetorical here, is it useful, if it is, then the UX issue you describe is real18:23
lbragstadif i'm an operator18:23
kmallocif it isn't useful, then we can not allow oversub and dodge the issue.18:23
lbragstadand I manage a deployment with a depth of 30 levels18:23
lbragstadand someone at the 29th level reports a ticket like that18:24
lbragstadi'm going to have a *real* long day tracking down where in the tree in need to make adjustments18:24
kmallocright, but this only comes up if the aggregate limit (allowance) for all children is large than the parent18:24
kmalloc(width/depth included)18:24
kmallocso is oversubscription useful?18:24
lbragstadaccording to the requirements johnthetubaguy proposed from CERN, it sounds like it is18:25
kmallocok, that is fine then, i didn't have a good answer18:25
lbragstadlet me see if i can findit18:25
kmallocthat to me tells us that in the case we allow oversub we need to be either limiting depth or adding smarts like "quota=100, oversub_children_allowance=[true|false]" so the operator can divine where the top of that "omg something borked" is18:26
kmallocand i don't really like oversub as a dynamic property of the individual limit18:26
lbragstad"Sub project over commit is OK (i.e. promising your sub projects more is OK, sum of the commitment to subprojects>project is OK but should be given an error if it actually happens)"18:26
kmallocok, overcommit is a requirement, that answers my question18:27
kmallocovercommit is something we need to address in the quota limit model.18:27
lbragstadi imagine it being useful for deployments looking for resources to flow between sub-projects18:28
kmallocand that makes deep trees tricky18:28
lbragstadand not having to be tightly coupled to tinkering with limits18:28
kmalloci agree18:28
lbragstadwhen things are in flux18:28
kmallocbut i wanted to be sure we had that clearly delineated18:28
kmallocbecause if it wasn't, i was going to ask more critically why we are allowing overcommit18:29
lbragstadthat was the reason wxy and johnthetubaguy included it in their proposals, afaict18:29
* kmalloc nods.18:29
kmallocsoooooo then.18:29
*** jmlowe has quit IRC18:30
kmalloci see two ways out of this.. well 318:30
kmalloc1) as is, two level limit18:30
kmalloc2) overcommit=true (on the limit definition)18:30
kmalloc3) config value for max limit and hang the ux issues [please don't pick this one]18:30
kmalloc4) clever solution to error reporting to end users18:31
kmalloc(see, off-by one errors, common)18:31
kmallocand honestly, i am fine with any of those options.18:31
kmallocexcept 318:31
kmalloc3 is a bad option18:31
kmalloci'm inclined to say option 1 is the most direct.18:32
*** fiddletwix has quit IRC18:32
lbragstadok - so what is #118:32
lbragstadpretty much what we have detailed in the specification?18:33
*** jmlowe has joined #openstack-keystone18:34
kmallocas the spec sits18:34
kmallocnot much changes.18:34
*** fiddletwix has joined #openstack-keystone18:34
lbragstadok - what about #218:35
lbragstaddoes the not have the two-level requirement?18:35
kmallocnothing specific18:38
kmallocjust allows for operators to know where in the tree to start looking if 29 deep you say you're out of quota18:39
kmallocmaybe overcommit only starts ate level 2818:39
kmalloci am not a huge fan of that btw.18:39
kmallocjust figured i'd float options as I saw them18:39
kmalloci think the 2-level bit is about as good as we're going to get for now.18:39
*** bigdogstl has quit IRC18:40
lbragstadand #3 restricts creating new projects that exceed two levels deep?18:40
kmallocvia config18:41
kmallocwhich is just bad design18:41
kmallocbut... we do that kind of stuff elsewhere18:41
kmalloc"oh i want 20 deep, cool i set the config"18:41
lbragstadwell - is that in a way similar to option #1?18:41
kmallocbut i'd rather have it hard-set18:41
kmalloc3 means API behavior is different based on config18:41
*** edmondsw has quit IRC18:41
kmallocwhich i REALLY hate18:41
lbragstad#1 does that in a way by opting into the model18:42
kmalloci'm more inclined to give it a pass if it's not a 3 way config (am i using quota enforcement, what level deep, and keystone side)18:42
kmalloc#1 is "I opt into enforcement" cool, that changes api behavior anyway18:42
kmallocbecause enforcement is centrally managed18:42
kmalloc3 is "I opt into enforcement and enforcement may behave differently depending on config"18:43
*** EmilienM is now known as EmilienM|PTO18:43
*** mvk has quit IRC18:43
kmalloci prefer the more specific "X enforcement means X behavior"18:43
kmallocthan "X enforcement could be X, Y, Z, Q, R, G" behavior18:43
kmallocpick one, good luck knowing18:44
lbragstadso would we modify the project hierarchy depth to be ignored in option #1?18:44
kmallocwe could.18:44
kmallocor we create a new enforcement model that does explicitly that18:44
lbragstadwe do similar things with the security_compliance configuration section18:44
kmallocvs "two-level-overcommit-enabled"18:44
kmallocsecurity compliance is a bit more special because of how PCI-DSS works18:45
lbragstadwell - in that we explicity say "18:45
lbragstad"this only matters if you are using the sql driver"18:45
kmallocquota has no implications with outside certification on if you can process credit cards...if for some reason your cloud is in scope18:45
kmallocquota is very much internal to openstack18:46
kmallocvs potentially very externally impacting18:46
lbragstadwe could have something similar for the tree depth configuration option saying "this option is ignored if you're using the two-level strict enforcement model"18:46
kmallocand in fact, i'd deprecate the option in keystone to change the depth18:47
kmallocset it to some reasonable max18:47
kmallocand let the quota enforcement model dictate the amount18:47
kmallocif it changes from the "Reasonable max"18:47
kmalloclbragstad: i'd increase the max_depth to 10, and deprecate the option18:49
kmallocand if someone has extra deep tree(s), we let them stay just no new extra deep ones18:50
kmallocthen the quota model dictates the max depth, no multiple options to reconcile.18:50
* lbragstad thinks18:50
kmallocalso keep in mind what happens if someone enables this enforcement model and already has a tree 5 projects deep18:51
kmallocdoes it error and say "NO WAY"18:51
lbragstadcmurphy: brought that up in review18:52
lbragstadthe specification review*18:52
lbragstadwe do something like that with the fernet token provider18:52
lbragstadif you're using fernet tokens and we can't find the key repository we fail18:53
lbragstadon start up18:53
kmallocthats fine.18:53
kmallocjust as long as we have the expected behavior documented18:53
kmallocanyway, i've covered my view18:54
lbragstaddo we keep that same view if a deployment has a tree 4 projects deep?18:54
kmallocit should be consistent18:54
lbragstadand they attempt to use the strict-two-level enforcement model?18:54
kmallocif we are saying this enforcement only works with two-level, it only works with two level18:55
lbragstadfail on start up, have them switch back to flat, fix the project structure, and try again?18:55
kmallocerror with clear indication on all the failures18:55
kmallocnot just the first one18:55
kmallocproject heirarchy is too deep at locations X, y, z,18:55
kmalloc[i might make it a doctor command to check before swaping to an enforcement model]18:56
kmallocso you don't have to play "are we going to fail to start games"18:56
kmallocbut basically, "run this command, it will tell you if the quota enforcement model can work"18:56
kmallocif not, you need to fix the places it wont work18:57
*** links has joined #openstack-keystone18:59
*** spilla has quit IRC19:02
*** edmondsw has joined #openstack-keystone19:07
*** vrv_ has quit IRC19:08
lbragstadi'm going to re-parse the spec again19:11
*** edmondsw_ has joined #openstack-keystone19:12
*** edmondsw has quit IRC19:15
openstackgerritLance Bragstad proposed openstack/keystone-specs master: Update blueprint link in default roles specification  https://review.openstack.org/57252819:17
lbragstadhrybacki: ^19:18
*** AlexeyAbashkin has joined #openstack-keystone19:19
*** Alexey_Abashkin has joined #openstack-keystone19:22
*** AlexeyAbashkin has quit IRC19:23
*** Alexey_Abashkin is now known as AlexeyAbashkin19:23
*** bigdogstl has joined #openstack-keystone19:25
*** itlinux has quit IRC19:26
*** bigdogstl has quit IRC19:30
lbragstadkmalloc: in your first comment here - https://review.openstack.org/#/c/540803/12/specs/keystone/rocky/strict-two-level-enforcement-model.rst19:36
lbragstadwhere are we writing the usage information?19:36
*** AlexeyAbashkin has quit IRC19:38
*** links has quit IRC19:46
*** mvenesio has joined #openstack-keystone19:47
kmallocthat is data stored in the service layer [oslo.limit]19:48
kmallocif you look at the convo between me and melwitt it's a tough nut to crack19:50
lbragstadi just left a response to that19:50
lbragstadand my head hurts19:51
*** timothyb89_ is now known as timothyb8919:51
melwittwell, fwiw I think if oslo.limit calls the per project count functions in parallel, maybe we don't really have a problem19:52
melwitter, or we should be more okay without doing the usage caching thing than I was originally thinking19:53
lbragstadyeah - that's a good point19:56
lbragstadanother thing19:56
lbragstadCERN has pretty much been asking for this for a while19:56
lbragstadand they've done a good job of stretching the legs on other big initiatives19:56
lbragstad(cells v2.0?)19:56
lbragstadsay we try this, we will likely get good feedback from them we can use to improve the system and refine it19:57
kmallocmelwitt: oh hi!19:57
kmalloci really think we're going to see a wide tree issue more commonly than expected, wide enough that even parallel lookup is going to be painful19:57
kmallocbut, that said, i put that as a comment so we weren't holding anything up besides having a discussion19:58
*** bigdogstl has joined #openstack-keystone19:58
kmalloci dind't want to encode that behavior as part of the spec without some level of agreememnt on the approach.19:58
kmallocalso we can revisit if we hit performance issues, i seriously hope i'm wrong.19:59
melwittyeah, I definitely think there should be a plugin or choice that does not do the usage caching as a first cut19:59
kmallocfor what james was proposing [i keep forgetting his IRC nick], because multi-level was a thing we def. need a report up to avoid "game the system" issues.20:00
lbragstadmelwitt: ftr - you're talking about making requests to calculate usage in parallel, right?20:01
kmallocbut that is out of scope of this spec.20:01
lbragstadnot requests to kesytone?20:01
kmallocright in nova in this case20:01
kmallocusage lookup not allowance lookup20:01
melwittlbragstad: yes, so oslo.limit would spin up some green threads to count usage for project A, B, C in parallel20:01
lbragstadjust making sure i was reading things right20:02
lbragstadfactoring that in, i'm not sure where i would guess things to tip over without seeing the system working or pushing it20:02
openstackgerritMerged openstack/keystone-specs master: Update blueprint link in default roles specification  https://review.openstack.org/57252820:03
kmallocthe real tip over is just how many children there are, even in parallel.20:03
*** bigdogstl has quit IRC20:03
kmallocand concurrency to nova making new vms.20:03
kmallocor whatever service :)20:04
* kmalloc goes back to flask stuffs i think we beat this spec into submission20:04
melwittthere was a thread awhile back where someone did some testing in cinder with the "counting quotas" re-architecting we did in nova and the tip over was having a lot of resources to count in one project (in the absence of caching). example, 30k volumes in one project and things started slowing down to do the "enforce" call20:04
kmallocand i'm happy to upgrade to a +2 as it sits as long as other folks weigh in.20:04
melwittI do wonder if hierarchy is available, people would be more likely to create smaller projects, that would help things a lot20:05
kmallocmelwitt: i think people will.20:05
lbragstadto a certain extent though20:05
lbragstadsince we're still limiting to two levels20:05
melwittthis was the thread, though really hard to read in plain text because there are graphics http://lists.openstack.org/pipermail/openstack-dev/2018-March/128096.html20:06
melwittthe resource count is a database query sum(column) over the rows for a project and as you get a lot more records, that slows down20:07
melwittwell, in the cases where it has to be a sum() (like vcpus, ram). the count() are faster if that is possible (one resource per row)20:08
*** r-daneel has quit IRC20:08
lbragstadso - there is the resource per project tip-over point and the total projects to calculate usage for tip-over point20:09
*** edmondsw_ is now known as edmondsw20:10
lbragstadnova could slow down calculating that there are 10000 instances in a single project, or that there are 100 instead in 100 projects, yeah?20:10
melwittinstances would be fast because that's a count() in an index but if it were vcpus, that would be a sum() and the former (10k) would be slower than the latter (100 x 100) I think20:11
lbragstadoh - sure, good point20:12
lbragstadis this enough to classify what we have in the spec as a unsupportable design?20:16
lbragstador do we keep things marked as experimental just until we get a better idea at where things fall over?20:17
lbragstadand then iterate from there20:18
melwittI'm writing up a response but I was wondering if we can't design this in a way such that the initial model is the simplest and could be potentially chosen via config option if we add a new model that writes upwards, for example20:21
melwittin the future20:21
lbragstadi think that is reasonable20:22
lbragstadand it still leaves the "flat" enforcement we have today20:22
lbragstadwhich let's system operators enforce whatever hierarchical model they want if they're will to modify the limits manually20:23
lbragstadwhich doesn't help james as much :(20:23
lbragstadsince he's looking for the ability for domain/project administrator to do a certain level of that on their own20:23
*** bigdogstl has joined #openstack-keystone20:24
melwittif I understood correctly, they wanted to be able to delegate limit setting via hierarchy and that each project delegated to would have to manage their own limits manually20:24
*** pcichy has joined #openstack-keystone20:25
lbragstadi'm not sure we'll be able to do that with flat enforcement20:25
lbragstadsince the hierarchy isn't part of the limit validation process in keystone20:25
melwittI see20:25
lbragstadif you have A and it has two children B and C as a single tree20:26
lbragstadthen D is a separate tree with one child, D20:26
lbragstadif you give project admin on D the ability update limits, they'd be able to modify limits on A's tree20:26
melwittI see, yeah, so flat means flat on both sides, the limit setting and the enforcement20:27
lbragstadright - at least right now?20:27
lbragstadlike, flat RBAC enforcement and flat limit validation20:27
melwittwhich makes sense. but they were hoping for a hybrid20:27
lbragstadthat might still be useful though20:28
lbragstadif you have hierarchical RBAC enforcement without the hierarchical limit validation, does that work?20:28
melwittyeah, I think it'd be useful, just a matter of whether it's too many knobs or other UX issue. to be able to choose hierarchical limit validation + flat RBAC enforcement20:28
lbragstadyeah - flat being only system administrator can modify limits20:29
melwittI guess I didn't see why not but that could well be my keystone limit ignorance. I had been thinking of a toggle in oslo.limit, either it walks the hierarchy or it doesn't20:29
kmallocalso, keep in mind that hierarchical limit and flat enforcement needs write-upwards or the system can be gamed20:29
kmallocor we have to do the entire depth search/validate anyway20:30
kmallocadd quota to child, spin up instanced, remove quota, add removed quota to other child, spin up instances, rinse, repeat20:30
melwittI didn't think it would need to write upwards. if we're checking quota for project E, only call the count function for project E and then compare it to project E's limit and the parent limit but don't go and count all the siblings to compare with the parent limit. maybe I'm missing something20:31
kmallocright but what if you take quota away from E and give it to F20:31
kmallocand then spin up instances on F without turning down e20:31
kmallocthen do it for D and Q20:31
lbragstaddoing this from the perspective of an administrator of D20:32
kmallocyou end up with as many instances on as many projects as you want.20:32
*** bigdogstl has quit IRC20:32
lbragstadso long as it's under the limit the system administrator gave your tree at D20:32
kmallocas many instances = max quota for the ultimate parent20:32
melwittyeah, that is what you're trusting your delegated party with. if they do that, it would be over quota until resources are freed by users20:32
kmalloci make it a habit to not write CVEs by design ;)20:33
melwittwell, in this scenario these are trusted sub-admins but I see your point20:33
kmallocbut that feels like something that will be a CVE pretty quickly, because someone will not expect it... even if it's documented20:33
melwittin their case, they just want to alleviate the load of putting all the limit setting work on one admin20:34
lbragstadif it wasn't a trusted sub admin and a malicious customer20:34
kmalloci still don't trust a sub-admin in the grand scheme of things [have to play the "have security hat, will critique"] - it may even be a malicious subadmin, you want to keep quota for a given tree under X20:34
kmallocthe admin has users saying "OMG NEED MORE VMS" and they, with good intentions, give the quota knowing they can game the system20:35
kmallocsomewhat malicious compliance to the limits delegated to them20:35
melwittI know penick would be okay with that if he could get the flat enforcement but I definitely see your point that this would open up a lot of problems for other use cases, so maybe it's a nonstarter20:35
kmalloci view this as outside of the scope of the spec proposed though20:35
melwittyeah, it's outside the scope for sure20:36
lbragstadpenick would use the flat enforcement?20:36
kmallocand i am happy to hammer that issue into the dead horse that it is when we start working on it20:36
*** itlinux has joined #openstack-keystone20:36
kmallocpenick said flat enforcement, strict commit [no overcommit], heirarchy20:36
kmallochierarchical limits*20:36
melwittthat's what he said. but maybe with parallel queries the hierarchical enforcement would be fine. would have to check with him20:37
kmalloc++ we should check with him on that front.20:37
lbragstadyeah - if he's will to manage requests :(20:37
melwittkmalloc: what do you think about the idea of the first model not caching usage designed in a way to allow swapping in a different model (via config option) in the future? is that something that would be possible?20:38
kmallocmelwitt: i'm inclined to just make that a different model (maybe make it a model family, that can be swapped between)20:38
*** raildo has quit IRC20:38
lbragstadso long as the project structure adheres to the requirements of the new model20:38
lbragstadthat seems reasonable20:39
melwittI agree with you that we're going to hit performance issues but as lbragstad mentioned, things get a lot more complicated that way (claim releases services are responsible for) and out-of-sync possibilities20:39
melwittas far as trying to provide something that doesn't have a high barrier for entry20:40
kmallocmodel_family(hierarchical_enforcement, hierarchical_limits) => [parallel_check_model, cached_check_model, cached_check_model_with_oob_timed_updates], model_family(flat_enforcement, flat_limits) => [flat_enforcer_model], model_family(flat_enforcement, heirarchical_limits) => [...]20:40
kmallocand anything in a model_family is assumed to be compatible (with maybe a seed-the-cache command)20:40
kmallocso to start we end up with flat_enforcement and heirarchical/hierarchical [ugh i can spell/type]20:40
kmallocand we expand from there20:41
kmallocflat is what is merged today, hierarchical/hierarchical without caching is the current spec20:41
kmallocand we expand from there20:41
*** jmlowe has quit IRC20:41
melwittsounds cool20:41
lbragstadif james ends up using flat or two-level-strict20:42
kmallocand i can then add some etcd, endpoint-sync data storage bits for this as well20:42
lbragstadi'd be interested to hear his feedback, because i wouldn't be surprised if we could us it for a new model20:42
kmallocso multi-endpoint could have a single limit set that is enforced.20:42
kmallocjust develop for each family of limit/enforcer combinations20:42
*** r-daneel has joined #openstack-keystone20:45
*** pcichy has quit IRC20:47
*** martinus__ has quit IRC20:48
*** lifeless has quit IRC20:51
*** lifeless has joined #openstack-keystone20:51
*** bigdogstl has joined #openstack-keystone20:53
*** bigdogstl has quit IRC20:58
*** dave-mccowan has quit IRC21:16
lbragstadlast unified limit question for the day21:23
lbragstadwe currently support the ability to create multiple limits in a single POST request21:24
lbragstaddo we want to expose that through the CLI some how, or would that be weird?21:24
*** lifeless has quit IRC21:25
lbragstador do we just leave CLI support as a single limit per request for the sake of not cluttering the command line or reading multiple limits from a file21:26
melwittlike for multiple resources? or multiple projects? or both? I could see someone maybe wanting to do the former, but not sure tbh21:26
lbragstadwe currently have this https://developer.openstack.org/api-ref/identity/v3/index.html#create-registered-limits21:26
lbragstadand https://developer.openstack.org/api-ref/identity/v3/index.html#create-limits21:27
lbragstadso you have create limits in batches21:27
melwittokay, so the request format can be repeated, I see21:27
*** spilla has joined #openstack-keystone21:27
melwittor it's a list rather21:27
lbragstadyeah - like "here are all the limits i want to create, go"21:28
melwittthe only data point I have is the nova quotas let you do multiple resources for one project in one go https://developer.openstack.org/api-ref/compute/#update-quotas21:28
*** bigdogstl has joined #openstack-keystone21:28
melwittnot sure how many people do that, but it's there and it makes sense to want to set limits for several resources for a project21:28
lbragstadare users able to create multiple quotas in a single go from the CLI?21:29
melwittI believe so. let me double check21:29
melwittosc does it https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/quota.html#quota-set21:30
melwitt(per project)21:30
lbragstadoh - wow21:31
lbragstadso it is possible string together settings per project for multiple projects in a single request21:32
*** edmondsw has quit IRC21:34
melwittand this is the old novaclient ref https://docs.openstack.org/ocata/cli-reference/nova.html#nova-quota-update21:34
*** edmondsw has joined #openstack-keystone21:34
*** bigdogstl has quit IRC21:34
lbragstadso you could do openstack quota set $SETTINGS $PROJECT_1 $SETTINGS $PROJECT_221:35
melwittI doubt it, I think it's one project at a time only21:35
lbragstadoh - got it21:35
lbragstadi was thinking you could do it for multiple projects at one21:36
melwittjust multiple resources21:36
melwittI don't think multiple projects makes much sense but that's just MHO21:36
lbragstadfrom a CLI perspective you mean?21:36
lbragstadimo - if an operator wanted to do that i could see them having a large .json file with all their requests ready to go and just curling it to keystone directly21:38
*** edmondsw has quit IRC21:38
*** r-daneel_ has joined #openstack-keystone21:39
*** r-daneel has quit IRC21:39
*** r-daneel_ is now known as r-daneel21:39
melwittyeah from CLI perspective21:41
lbragstadmakes sense21:42
melwittmayve I'm wrong too :)21:43
lbragstadi'm not aware of any CLI commands that let you do that21:43
lbragstador support batch creation21:43
* melwitt nods21:44
*** bigdogstl has joined #openstack-keystone21:45
lbragstadthanks for the help today melwitt21:45
openstackMeeting ended Tue Jun  5 21:46:32 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:46
openstackMinutes:        http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-06-05-17.04.html21:46
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-06-05-17.04.txt21:46
openstackLog:            http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-06-05-17.04.log.html21:46
*** lifeless has joined #openstack-keystone21:47
*** itlinux has quit IRC21:53
*** itlinux has joined #openstack-keystone22:03
*** itlinux has quit IRC22:13
*** r-daneel has quit IRC22:21
*** mvenesio has quit IRC22:23
*** ayoung has quit IRC22:25
*** rcernin has joined #openstack-keystone22:25
*** ayoung has joined #openstack-keystone22:36
*** boris_42_ has joined #openstack-keystone22:47
*** bigdogstl has quit IRC22:53
*** bigdogstl has joined #openstack-keystone23:00
*** edmondsw has joined #openstack-keystone23:03
*** bigdogstl has quit IRC23:05
*** edmondsw has quit IRC23:07
*** bigdogstl has joined #openstack-keystone23:07
*** threestrands has joined #openstack-keystone23:08
*** harlowja has joined #openstack-keystone23:10
*** bigdogstl has quit IRC23:12
*** bigdogstl has joined #openstack-keystone23:22
*** bigdogstl has quit IRC23:31
*** annp has quit IRC23:38
*** annp has joined #openstack-keystone23:38
*** bigdogstl has joined #openstack-keystone23:44
*** bigdogstl has quit IRC23:49
*** lifeless has quit IRC23:54
*** lifeless has joined #openstack-keystone23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!