| *** rcernin has joined #openstack-meeting-alt | 00:11 | |
| *** rcernin_ has quit IRC | 00:12 | |
| *** rcernin has quit IRC | 00:13 | |
| *** rcernin has joined #openstack-meeting-alt | 00:14 | |
| *** tetsuro has joined #openstack-meeting-alt | 00:45 | |
| *** chhagarw has joined #openstack-meeting-alt | 00:47 | |
| *** gouthamr has joined #openstack-meeting-alt | 00:47 | |
| *** chhagarw has quit IRC | 00:51 | |
| *** tetsuro has quit IRC | 01:06 | |
| *** tetsuro_ has joined #openstack-meeting-alt | 01:06 | |
| *** markvoelker has joined #openstack-meeting-alt | 01:25 | |
| *** hongbin has joined #openstack-meeting-alt | 01:27 | |
| *** cloudrancher has quit IRC | 01:33 | |
| *** cloudrancher has joined #openstack-meeting-alt | 01:34 | |
| *** hongbin has quit IRC | 01:36 | |
| *** hongbin has joined #openstack-meeting-alt | 01:37 | |
| *** yamahata has quit IRC | 01:40 | |
| *** hongbin_ has joined #openstack-meeting-alt | 01:41 | |
| *** cloudrancher has quit IRC | 01:41 | |
| *** cloudrancher has joined #openstack-meeting-alt | 01:42 | |
| *** hongbin has quit IRC | 01:43 | |
| *** bhavikdbavishi has joined #openstack-meeting-alt | 01:43 | |
| *** bhavikdbavishi has quit IRC | 01:50 | |
| *** hongbin has joined #openstack-meeting-alt | 02:05 | |
| *** hongbin_ has quit IRC | 02:07 | |
| *** lei-zh has joined #openstack-meeting-alt | 02:22 | |
| *** bhavikdbavishi has joined #openstack-meeting-alt | 02:23 | |
| *** lei-zh has quit IRC | 02:30 | |
| *** lei-zh1 has joined #openstack-meeting-alt | 02:30 | |
| *** bhavikdbavishi has quit IRC | 02:40 | |
| *** dave-mccowan has quit IRC | 02:57 | |
| *** munimeha1 has quit IRC | 03:01 | |
| *** bhavikdbavishi has joined #openstack-meeting-alt | 03:35 | |
| *** lei-zh1 has quit IRC | 03:39 | |
| *** yamahata has joined #openstack-meeting-alt | 03:48 | |
| *** hongbin has quit IRC | 03:52 | |
| *** chhagarw has joined #openstack-meeting-alt | 03:58 | |
| *** yamamoto has quit IRC | 04:17 | |
| *** yamamoto has joined #openstack-meeting-alt | 04:17 | |
| *** yamahata has quit IRC | 04:19 | |
| *** yamahata has joined #openstack-meeting-alt | 04:19 | |
| *** lei-zh1 has joined #openstack-meeting-alt | 04:54 | |
| *** janki has joined #openstack-meeting-alt | 04:59 | |
| *** ttsiouts has quit IRC | 05:42 | |
| *** ttsiouts has joined #openstack-meeting-alt | 05:43 | |
| *** ttsiouts has quit IRC | 05:47 | |
| *** tetsuro_ has quit IRC | 05:50 | |
| *** ccamacho has joined #openstack-meeting-alt | 06:31 | |
| *** slaweq has joined #openstack-meeting-alt | 06:43 | |
| *** bhavikdbavishi1 has joined #openstack-meeting-alt | 06:53 | |
| *** bhavikdbavishi has quit IRC | 06:55 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:55 | |
| *** rcernin has quit IRC | 07:06 | |
| *** rdopiera has joined #openstack-meeting-alt | 07:09 | |
| *** alexchadin has joined #openstack-meeting-alt | 07:10 | |
| *** lei-zh1 has quit IRC | 07:24 | |
| *** lei-zh1 has joined #openstack-meeting-alt | 07:24 | |
| *** ttsiouts has joined #openstack-meeting-alt | 08:17 | |
| *** e0ne has joined #openstack-meeting-alt | 08:23 | |
| *** derekh has joined #openstack-meeting-alt | 08:29 | |
| *** gouthamr has quit IRC | 08:32 | |
| *** priteau has joined #openstack-meeting-alt | 08:45 | |
| *** dtrainor has quit IRC | 08:59 | |
| priteau | #startmeeting blazar | 09:00 |
|---|---|---|
| openstack | Meeting started Tue Oct 23 09:00:16 2018 UTC and is due to finish in 60 minutes. The chair is priteau. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: blazar)" | 09:00 | |
| openstack | The meeting name has been set to 'blazar' | 09:00 |
| priteau | #topic Roll call | 09:00 |
| *** openstack changes topic to "Roll call (Meeting topic: blazar)" | 09:00 | |
| *** bertys has joined #openstack-meeting-alt | 09:01 | |
| bertys | o/ | 09:02 |
| priteau | Good morning bertys | 09:02 |
| bertys | hi all | 09:02 |
| priteau | bertys: It's just the two of us from now, should we wait a few minutes to see if masahito and tetsuro join? | 09:04 |
| bertys | sure, have you pinged them on blazar IRC? | 09:04 |
| priteau | They're not on IRC at all | 09:05 |
| priteau | Well, let's start, maybe they'll join later. I sent them an email. | 09:09 |
| bertys | ok, thanks | 09:09 |
| priteau | Agenda: | 09:09 |
| priteau | 1. Project Update @ Berlin Summit | 09:09 |
| priteau | 2. Project Update @ Berlin Summit | 09:09 |
| priteau | 3. OpenStack-wide Goals | 09:09 |
| priteau | 4. stein-1 milestone | 09:09 |
| priteau | 5. AOB | 09:09 |
| priteau | #topic Project Update @ Berlin Summit | 09:09 |
| *** openstack changes topic to "Project Update @ Berlin Summit (Meeting topic: blazar)" | 09:09 | |
| priteau | bertys: I see you added a slide on How to Contribute, thanks! | 09:10 |
| *** gouthamr has joined #openstack-meeting-alt | 09:11 | |
| priteau | We have 20 minutes and only 7 slides, we may be able to add one or two to explain how Blazar works in more details, do you think that could be useful? | 09:11 |
| bertys | priteau: sure, would you be willing to present that part? | 09:14 |
| *** tetsuro has joined #openstack-meeting-alt | 09:14 | |
| priteau | bertys: Sure | 09:14 |
| priteau | Hi tetsuro | 09:14 |
| tetsuro | sorry to be late | 09:15 |
| tetsuro | And masa has told me he can't attend today | 09:15 |
| priteau | OK, thanks for letting us know | 09:16 |
| priteau | We were just discussing the Project Update talk | 09:16 |
| priteau | I was saying that, given that we have 20 minutes and only 7 slides so far, we may be able to add more content to explain how Blazar works internally | 09:16 |
| priteau | I can take care of it but I am open to suggestions. | 09:16 |
| tetsuro | Sounds a good idea. | 09:17 |
| priteau | There is still some content to add in the existing slide from Masahito and I | 09:18 |
| priteau | Let's try to get it finished by next meeting. | 09:19 |
| tetsuro | I think we can focus on how we can use blazar rather than how it works internally since there are more users than developers | 09:19 |
| priteau | tetsuro: That's a good idea :) | 09:19 |
| tetsuro | Masa has told me to tell everyone that he's going to finish it this week | 09:19 |
| tetsuro | s/it/the slide/ | 09:20 |
| priteau | Good to hear | 09:20 |
| priteau | Next topic | 09:21 |
| priteau | #topic Forum @ Berlin Summit | 09:21 |
| *** openstack changes topic to "Forum @ Berlin Summit (Meeting topic: blazar)" | 09:21 | |
| priteau | I have created the Etherpad for the Blazar forum session and tetsuro created the one for placement | 09:21 |
| priteau | Both are available on the Forum wiki page | 09:21 |
| priteau | #link https://wiki.openstack.org/wiki/Forum/Berlin2018 | 09:21 |
| priteau | Thank you tetsuro | 09:22 |
| *** alex_xu has quit IRC | 09:23 | |
| tetsuro | Oh, no I haven't I think you are referring to the placement extraction etherpad. | 09:23 |
| tetsuro | But I will today. | 09:23 |
| priteau | Yes, I just realized now! | 09:24 |
| priteau | I misread the Etherpad title | 09:24 |
| priteau | I don't think we have much more to do for now. We will need to review any comment that are added to the Etherpad before the session. | 09:25 |
| priteau | #topic OpenStack-wide Goals | 09:26 |
| *** openstack changes topic to "OpenStack-wide Goals (Meeting topic: blazar)" | 09:26 | |
| priteau | We've received a patch to implement upgrade checkers | 09:26 |
| priteau | #link https://review.openstack.org/#/c/611811/ | 09:27 |
| priteau | I haven't had time to review it yet, but it's got a +1 from Matt | 09:27 |
| *** lei-zh1 has quit IRC | 09:28 | |
| priteau | Please review when you can | 09:28 |
| priteau | #topic stein-1 milestone | 09:29 |
| *** openstack changes topic to "stein-1 milestone (Meeting topic: blazar)" | 09:29 | |
| priteau | stein-1 milestone is this week. As explained in a previous meeting, there is no more tagging for this milestone. | 09:30 |
| priteau | #link https://launchpad.net/blazar/+milestone/stein-1 | 09:30 |
| priteau | We've fixed a few issues but a lot are in progress or not yet started | 09:30 |
| priteau | I propose to move all undone bugs and blueprint to the stein-2 milestone, any objection? | 09:31 |
| priteau | Any comment? | 09:33 |
| tetsuro | No, we should do more reviews. | 09:33 |
| tetsuro | towards stein-2 | 09:35 |
| priteau | Agreed. My time is limited at the moment due to focus on another project but I am hoping it will get better after the Berlin Summit | 09:35 |
| priteau | #topic AOB | 09:37 |
| *** openstack changes topic to "AOB (Meeting topic: blazar)" | 09:37 | |
| priteau | bertys: Regarding your comment on https://review.openstack.org/#/c/604938/ | 09:39 |
| priteau | What about we assert that the nova-compute service is enabled and up when we add it to aggregate, and we leave a better handling as a todo for when we test health monitoring? | 09:40 |
| priteau | At the moment, there is only one host and if it is down, that's an infra or devstack issue. | 09:43 |
| bertys | priteau: this may be ok for now and I was actually thinking already longer term | 09:45 |
| priteau | Sometimes it's more productive to merge early and properly fix later ;-) | 09:46 |
| priteau | Could you update your comment for masahito to explain what is sufficient for merging the patch? | 09:48 |
| bertys | right and as you know, I still have some plans to execute those scenario tests in OPNFV | 09:48 |
| bertys | ok, will do | 09:49 |
| priteau | Thank you | 09:50 |
| priteau | Any other issue to discuss? | 09:51 |
| tetsuro | I'm on business travel next week to south east Asia, so I can't attend the meeting, but by then I'll fill up the Summit etherpad https://etherpad.openstack.org/p/BER-python-bindings-for-the-placement-api, so please add your comments there next week. | 09:51 |
| priteau | Thank you tetsuro! | 09:52 |
| priteau | If nothing else, we can finish meeting early this week. | 09:54 |
| priteau | Thank you for joining! | 09:54 |
| priteau | #endmeeting | 09:54 |
| *** openstack changes topic to "Documentation (Meeting topic: trove)" | 09:55 | |
| openstack | Meeting ended Tue Oct 23 09:54:59 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:55 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/blazar/2018/blazar.2018-10-23-09.00.html | 09:55 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/blazar/2018/blazar.2018-10-23-09.00.txt | 09:55 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/blazar/2018/blazar.2018-10-23-09.00.log.html | 09:55 |
| *** rossella_s has quit IRC | 09:57 | |
| *** bertys has quit IRC | 09:57 | |
| *** ianychoi has quit IRC | 10:22 | |
| *** ianychoi has joined #openstack-meeting-alt | 10:25 | |
| *** apetrich has quit IRC | 10:38 | |
| *** pbourke has quit IRC | 10:47 | |
| *** dtrainor has joined #openstack-meeting-alt | 10:48 | |
| *** pbourke has joined #openstack-meeting-alt | 10:48 | |
| *** apetrich has joined #openstack-meeting-alt | 10:54 | |
| *** erlon has joined #openstack-meeting-alt | 10:58 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-meeting-alt | 11:04 | |
| *** ttsiouts has quit IRC | 11:04 | |
| *** yamamoto has quit IRC | 11:08 | |
| *** yikun has quit IRC | 11:09 | |
| *** tetsuro has quit IRC | 11:12 | |
| *** yamamoto has joined #openstack-meeting-alt | 11:21 | |
| *** panda is now known as panda|lunch | 11:27 | |
| *** janki has quit IRC | 11:41 | |
| *** ttsiouts has joined #openstack-meeting-alt | 11:43 | |
| *** markvoelker has quit IRC | 11:45 | |
| *** ttsiouts has quit IRC | 11:48 | |
| *** ttsiouts has joined #openstack-meeting-alt | 12:02 | |
| *** dave-mccowan has joined #openstack-meeting-alt | 12:07 | |
| *** janki has joined #openstack-meeting-alt | 12:14 | |
| *** cloudrancher has quit IRC | 12:19 | |
| *** cloudrancher has joined #openstack-meeting-alt | 12:19 | |
| *** bhavikdbavishi has quit IRC | 12:22 | |
| *** janki has quit IRC | 12:25 | |
| *** janki has joined #openstack-meeting-alt | 12:25 | |
| *** janki has quit IRC | 12:27 | |
| *** tobberydberg has quit IRC | 12:30 | |
| *** janki has joined #openstack-meeting-alt | 12:34 | |
| *** markvoelker has joined #openstack-meeting-alt | 12:36 | |
| *** jcoufal has joined #openstack-meeting-alt | 12:37 | |
| *** markvoelker has quit IRC | 12:37 | |
| *** jchhatbar has joined #openstack-meeting-alt | 12:38 | |
| *** janki has quit IRC | 12:40 | |
| *** jchhatbar is now known as janki | 12:50 | |
| *** dustins has joined #openstack-meeting-alt | 12:51 | |
| *** dustins is now known as dschoenb|worksho | 12:52 | |
| *** dschoenb|worksho is now known as dustins | 12:52 | |
| *** ChanServ changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:53 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** bnemec has joined #openstack-meeting-alt | 13:08 | |
| *** e0ne has quit IRC | 13:19 | |
| *** yamamoto has joined #openstack-meeting-alt | 13:35 | |
| *** cloudrancher has quit IRC | 13:44 | |
| *** cloudrancher has joined #openstack-meeting-alt | 13:45 | |
| *** liuyulong has joined #openstack-meeting-alt | 13:48 | |
| *** Leo_m has joined #openstack-meeting-alt | 13:58 | |
| *** hongbin has joined #openstack-meeting-alt | 14:00 | |
| *** panda|lunch is now known as panda | 14:04 | |
| *** janki has quit IRC | 14:21 | |
| *** e0ne has joined #openstack-meeting-alt | 14:29 | |
| *** alexchadin has quit IRC | 14:30 | |
| *** ccamacho has quit IRC | 14:57 | |
| *** gagehugo has joined #openstack-meeting-alt | 15:02 | |
| *** Leo_m_ has joined #openstack-meeting-alt | 15:02 | |
| *** Leo_m has quit IRC | 15:04 | |
| *** Leo_m_ has quit IRC | 15:09 | |
| *** Leo_m has joined #openstack-meeting-alt | 15:09 | |
| *** cloudrancher has quit IRC | 15:12 | |
| *** cloudrancher has joined #openstack-meeting-alt | 15:13 | |
| *** ccamacho has joined #openstack-meeting-alt | 15:17 | |
| *** ccamacho has quit IRC | 15:17 | |
| *** kopecmartin is now known as kopecmartin|off | 15:33 | |
| *** wxy| has joined #openstack-meeting-alt | 15:34 | |
| *** ianychoi_ has joined #openstack-meeting-alt | 15:36 | |
| *** e0ne has quit IRC | 15:39 | |
| *** ianychoi has quit IRC | 15:40 | |
| *** liuyulong is now known as liuyulong|away | 15:41 | |
| *** ayoung has joined #openstack-meeting-alt | 15:50 | |
| lbragstad | #startmeeting keystone | 16:00 |
| openstack | Meeting started Tue Oct 23 16:00:22 2018 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: keystone)" | 16:00 | |
| openstack | The meeting name has been set to 'keystone' | 16:00 |
| lbragstad | #link https://etherpad.openstack.org/p/keystone-weekly-meeting | 16:00 |
| lbragstad | o/ | 16:00 |
| cmurphy | o/ | 16:00 |
| hrybacki | o/ | 16:01 |
| gagehugo | o/ | 16:01 |
| ayoung | Oyez oyez | 16:02 |
| wxy| | o/ | 16:02 |
| lbragstad | #topic Release status | 16:03 |
| *** openstack changes topic to "Release status (Meeting topic: keystone)" | 16:03 | |
| lbragstad | #info next week is Stein-1 and specification proposal freeze | 16:03 |
| ayoung | I assume we have real things to discuss prior to my two agenda items | 16:03 |
| lbragstad | we should be smoothing out concerns with specs sooner rather than later at this point | 16:04 |
| kmalloc | o/ | 16:04 |
| lbragstad | if you have specific items wrt to specs and want higher bandwidth to discuss, please let someone know | 16:04 |
| lbragstad | or throw it on the meeting agenda | 16:04 |
| lbragstad | ayoung do you want to reorder the schedule | 16:05 |
| lbragstad | ? | 16:05 |
| lbragstad | or is that what you're suggesting? | 16:05 |
| ayoung | Nah | 16:06 |
| ayoung | I'm going last | 16:06 |
| lbragstad | ok | 16:06 |
| lbragstad | #topic Oath approach to federation | 16:06 |
| *** openstack changes topic to "Oath approach to federation (Meeting topic: keystone)" | 16:06 | |
| lbragstad | last week we talked about Oath open-sourcing their approach to federation | 16:07 |
| ayoung | so replace uuid3 with uuid5 and I like | 16:07 |
| ayoung | couple other things; we could make it so the deployer choses the namespace, and could keep that in sync across their deployments, to get "unique" Ids that are still distributed | 16:07 |
| lbragstad | tl;dr they consume Athenz tokens in place of SAML assertion and have their own auth plugin for doing their version of auto-provisioning | 16:07 |
| lbragstad | you can find the code here: | 16:08 |
| lbragstad | #link https://github.com/yahoo/openstack-collab/tree/master/keystone-federation-ocata | 16:08 |
| lbragstad | i started walking through it and comparing their implementation against what we have, just to better understand the differences | 16:08 |
| lbragstad | you can find that here, but i just started working on it | 16:08 |
| lbragstad | #link https://etherpad.openstack.org/p/keystone-shadow-mapping-athenz-delta | 16:08 |
| ayoung | Could Athenz be done as a middleware module? Something that looks like REMOTE_USER/REMOTE_GROUPS? Or does it provide more information than we currently accept from SAML etc | 16:09 |
| lbragstad | it doesn't really follow the saml spec at all - from what i can tell, it gets everything from the athenz token and the auth body | 16:09 |
| lbragstad | the auth plugin decodes the token and provisions users, projects, and roles based on the values | 16:10 |
| ayoung | Because Autoprovisioning is its own thing, and we should be willing to accept that as a standalone contribution anyway. | 16:10 |
| lbragstad | yeah - i guess it's important to note that Oath developed this for replicated usecases and not auto-provisioning specifically, but the implementation is very similar to what we developed as a solution for auto-provisioning | 16:11 |
| ayoung | also...Oauth needs predictable Ids. | 16:12 |
| ayoung | I have a WIP spec to support those. It is more than just Users, it looks like | 16:12 |
| lbragstad | i'm not sure they need those if they come from the identity provider | 16:12 |
| lbragstad | which is athenz | 16:12 |
| ayoung | I think the inter-tubes are congested | 16:12 |
| ayoung | https://review.openstack.org/#/c/612099/ | 16:13 |
| lbragstad | why does athenz need predictable user ids? | 16:13 |
| ayoung | lbragstad, becasue they need to be the same from location to location | 16:14 |
| ayoung | so admin can't be one ABC on region1 and 123 in region2 | 16:14 |
| lbragstad | https://github.com/yahoo/openstack-collab/blob/master/keystone-federation-ocata/plugin/keystone/auth/plugins/athenz.py#L123-L129 | 16:14 |
| ayoung | they state they use uuid3(NAMESPACE, name) | 16:14 |
| lbragstad | the user id is generated by athens | 16:14 |
| lbragstad | athenz* | 16:15 |
| lbragstad | and keystone just populates it in the database, from what i can tell | 16:15 |
| *** rdopiera has quit IRC | 16:15 | |
| lbragstad | so long as you're using athenz tokens to access keystone service providers, you should have the same user id at each site? | 16:16 |
| ayoung | that is my understanding, yes | 16:16 |
| *** ttsiouts has quit IRC | 16:16 | |
| lbragstad | so their implementation has already achieved predictable user ids | 16:16 |
| *** gyee has joined #openstack-meeting-alt | 16:16 | |
| lbragstad | right? | 16:16 |
| *** ttsiouts has joined #openstack-meeting-alt | 16:17 | |
| lbragstad | if anyone feels like parsing that code, feel free to add your comments, questions, or concerns to that etherpad | 16:19 |
| lbragstad | it might be helpful if/when we or penick go to draft a specification | 16:19 |
| lbragstad | worst case, it helps us understand their usecase a bit better | 16:19 |
| wxy| | Will take a look later. | 16:19 |
| lbragstad | thanks wxy| | 16:20 |
| lbragstad | any other questions on this? | 16:20 |
| * knikolla will read back. am stuck in meetings as we have the MOC workshop next week. sorry for being AWOL this time period. | 16:20 | |
| lbragstad | no worries - thanks knikolla | 16:21 |
| lbragstad | alright, moving on | 16:21 |
| lbragstad | #topic Another report of upgrade failures with user options | 16:21 |
| *** openstack changes topic to "Another report of upgrade failures with user options (Meeting topic: keystone)" | 16:21 | |
| *** ayoung has quit IRC | 16:21 | |
| *** ttsiouts has quit IRC | 16:21 | |
| lbragstad | #link https://bugs.launchpad.net/openstack-ansible/+bug/1793389 | 16:21 |
| openstack | Launchpad bug 1793389 in openstack-ansible "Upgrade to Ocata: Keystone Intermittent Missing 'options' Key" [Medium,Fix released] - Assigned to Alex Redinger (rexredinger) | 16:21 |
| lbragstad | we've had this one crop up a few times | 16:21 |
| lbragstad | specifically, the issue is due to caching during a live upgrade | 16:22 |
| lbragstad | from pre-Ocata to Ocata | 16:22 |
| *** ayoung has joined #openstack-meeting-alt | 16:22 | |
| lbragstad | it's still undetermined if this impacts FFU scenarios | 16:23 |
| lbragstad | (e.g. Newton -> Pike) | 16:23 |
| lbragstad | but it boils down to the cache returning a user reference during authentication on Ocata code that expects user['options'] to be present, but isn't because the user was cached prior to the upgrade | 16:23 |
| ayoung | Gah...disconnect. I'll try to catch up | 16:24 |
| lbragstad | deployment projects have a work around to flush memcached as a way to force a miss on authentication and refetch the user | 16:24 |
| lbragstad | cmurphy odyssey4me and i were discussing approaches for mitigating this in keystone directly | 16:25 |
| lbragstad | there is a WIP review in gerrit | 16:26 |
| lbragstad | #link https://review.openstack.org/#/c/612686/ | 16:26 |
| lbragstad | but curious if people have thoughts or concerns about this approach | 16:26 |
| lbragstad | or if there are other approaches we should consider | 16:26 |
| ayoung | wouldn't deploying a fix like this flush the cache anyway? | 16:27 |
| ayoung | How could they ever get in this state? | 16:27 |
| lbragstad | the memcached instance has a valid cache for a specific user | 16:28 |
| ayoung | Is this a side effect of 0 downtime upgrades? Keep the cache up, even as we change the data out from underneath? | 16:28 |
| lbragstad | yeah - that's the problem | 16:28 |
| lbragstad | the cache remains up | 16:28 |
| lbragstad | thus holding the cached data | 16:28 |
| ayoung | that is going to be a problem in other ways | 16:28 |
| ayoung | needs to be part of the upgrade. Flush cache when we do .... | 16:29 |
| ayoung | contract? | 16:29 |
| ayoung | we change the schema in the middle. THe cache will no longer reflect the schema after some point | 16:29 |
| lbragstad | that's what https://review.openstack.org/#/c/608066/ does | 16:30 |
| lbragstad | but not in process | 16:30 |
| cmurphy | that's the problem, the question is whether we can be a bit more surgical instead of flushing the whole cache | 16:30 |
| ayoung | I see that, but it is on a row by row basis | 16:30 |
| ayoung | yeah, that review looks like it is in the right direction | 16:31 |
| ayoung | so...can we tell memcache to flush all of a certain class of entry? As I recall from token revocations, that is not possible | 16:31 |
| lbragstad | also - alex's comment on https://review.openstack.org/#/c/612686/ proves this could affect FFU | 16:31 |
| ayoung | it only knows about key/value stores | 16:31 |
| lbragstad | ayoung are you asking about cache region support? | 16:32 |
| ayoung | lbragstad, maybe. | 16:32 |
| ayoung | does each region reflect a specific class of cached objects? | 16:33 |
| lbragstad | some parts of keystone rely on regions, yes | 16:33 |
| lbragstad | computed role assignment have their own region, for example | 16:33 |
| lbragstad | same with tokens | 16:33 |
| ayoung | are regions expensive? Is there a reason to avoid using them? | 16:33 |
| *** iyamahat has joined #openstack-meeting-alt | 16:33 | |
| lbragstad | i'm not sure - that might be a better question for kmalloc | 16:34 |
| kmalloc | no | 16:34 |
| lbragstad | #link https://review.openstack.org/#/c/612686/1/keystone/identity/core.py,unified is an attempt at creating a region specifically for users | 16:34 |
| ayoung | could we wrap user, groups, projects etc each with a region, and then, as part of the sql migrations, flush the region | 16:34 |
| kmalloc | not expensive, but we have cases where we cannot invalidate an explicit cache key | 16:34 |
| kmalloc | e.g. many entries via kwargs into a single method | 16:34 |
| kmalloc | so we need to invalidate the entire region | 16:34 |
| lbragstad | #link https://review.openstack.org/#/c/612686/1/keystone/auth/core.py,unified@389 drops the entire user region (every cached user) | 16:34 |
| kmalloc | it is better to narrow the invalidation to as small a subset as possible | 16:34 |
| kmalloc | no reason to invalidate *everything* if only computed role assignments needs to be invalidated | 16:35 |
| ayoung | kmalloc, if we change the scheme on, in this case, users, we need to invalidate all cached users. Is that too specific? | 16:35 |
| kmalloc | you can do so. | 16:35 |
| ayoung | each class of object gets its own region? | 16:35 |
| kmalloc | so far yes | 16:36 |
| kmalloc | well... | 16:36 |
| kmalloc | each manager | 16:36 |
| ayoung | ok...so, we could tie in with the migration code, too, to identify what reqions need to be invalidated | 16:36 |
| lbragstad | correct - if that region needs to be invalidated | 16:36 |
| kmalloc | and some managers have extra regions, eg. computed assignments | 16:36 |
| ayoung | OK, so users and groups would go together, for example? | 16:36 |
| kmalloc | right now, yes | 16:36 |
| *** yamahata has quit IRC | 16:36 | |
| lbragstad | but - they could be two separate regions if needed | 16:37 |
| kmalloc | ++ | 16:37 |
| lbragstad | depends on the invalidation strategy | 16:37 |
| kmalloc | it's highly modular | 16:37 |
| ayoung | Backend is probably granular enough | 16:37 |
| lbragstad | or what needs to invoke invalidation, how often, etc... | 16:37 |
| ayoung | identity, assignment, resource | 16:37 |
| kmalloc | you can also force a cache pop by changing the argument(s)/kwargs [once https://review.openstack.org/#/c/611120/ lands] in the method signature | 16:37 |
| kmalloc | since we cache memoized | 16:38 |
| ayoung | yech | 16:38 |
| ayoung | lets not count on that. | 16:38 |
| kmalloc | it is a way caching works. | 16:38 |
| ayoung | I'd hate to hate to change kwargs just to force a cache pop | 16:38 |
| ayoung | yeah, and it is ok, just not what we want to use for this requirement | 16:38 |
| kmalloc | it is a way a lot of things on the internet work, explicit change to the request forcing a cache cycle | 16:38 |
| kmalloc | in either case you can force a cache pop. though i would not want to do that in db_sync | 16:39 |
| kmalloc | it might make sense to do an explicit region (all region) cache expiration/invalidation on keystone start | 16:40 |
| kmalloc | or as a keystone-manage command | 16:40 |
| kmalloc | hooking in all the cache logic into db_sync seems ... rough | 16:40 |
| *** iyamahat has quit IRC | 16:41 | |
| lbragstad | in that case, a single keystone node could invalidate the memcached instances | 16:41 |
| ayoung | what if db_sync set the values that would then be used by the manage-command | 16:41 |
| lbragstad | but that behavior also depends on cache configuration | 16:41 |
| ayoung | like a scracth table with the set of regions to invalidate? | 16:41 |
| kmalloc | ayoung: there is no reason to do something like that | 16:41 |
| kmalloc | really, just invalidate the regions | 16:42 |
| kmalloc | they will re-warm quickly | 16:42 |
| kmalloc | upgrade steps should be expected to need a cache invalidation/rewarm | 16:42 |
| lbragstad | performance will degrade for a bit | 16:42 |
| lbragstad | also - cmurphy brought up a good point earlier that it would be nice to find a solution that wasn't super specific to just this case | 16:42 |
| kmalloc | which is fine for an upgrade process. we already say "turn everything off except X" | 16:42 |
| lbragstad | since this is likely going to happen in the future | 16:42 |
| kmalloc | so, i'd say keystone-manage that forces a region-wide invalidation | 16:43 |
| kmalloc | [all regions] | 16:43 |
| ayoung | I'll defer. I thought we were going more specific, to flush only regions we knew had changed, but, this is ok] | 16:43 |
| kmalloc | for the most part our TTLs are very narrow | 16:44 |
| kmalloc | i'll bet most cache is popped just by timeout (5m) during upgrade process | 16:45 |
| kmalloc | or a restart of memcache servers as part of the deal | 16:45 |
| kmalloc | this is just explicit another option is to add a namespace value that we change per release of keystone | 16:45 |
| kmalloc | that just forces rotation of the cache based upon code base. | 16:46 |
| ayoung | ok, so keystone-manage cache-invalidate [region | all ] ? | 16:46 |
| *** armstrong has joined #openstack-meeting-alt | 16:46 | |
| kmalloc | fwiw, a namespace is just "added" to the cache key (before sha calculation) | 16:46 |
| kmalloc | which then forces a new keystone to always use new cache namespace | 16:46 |
| kmalloc | no "don't forget to run this command" | 16:47 |
| kmalloc | (though an explicit cache invalidate command might be generally useful regardless) | 16:47 |
| ayoung | cool. We good here? | 16:47 |
| lbragstad | i think so - we're probably at a good point to continue the discussion in review | 16:48 |
| kmalloc | we could use https://github.com/openstack/keystone/blob/master/keystone/version.py#L15 anyway. yeah we should continue discussion in review | 16:48 |
| ayoung | Cool...ok | 16:48 |
| lbragstad | #topic open discussion | 16:48 |
| *** openstack changes topic to "open discussion (Meeting topic: keystone)" | 16:48 | |
| ayoung | stwo things | 16:48 |
| ayoung | 1. service roles | 16:48 |
| lbragstad | we have 12 minutes to talk about whatever we wanna talk about | 16:48 |
| kmalloc | Flask has 2 more reviews, all massive code removals! yay, we're done with the migration | 16:48 |
| ayoung | we need a way to convert people from admin-everywhere to service roles | 16:48 |
| * kmalloc has nothing else to talk about there, just cheering that we got there | 16:49 | |
| ayoung | so...short version: | 16:49 |
| * kmalloc hands the floor to ayoung... and since ayoung is now holding the entire floor, everyone falls ... into the emptyness/botomless area below the floor. | 16:49 | |
| ayoung | we role in rules that say admin(not everywhere) is servicer role or is_admin_project and leae the current mechanism in place | 16:49 |
| ayoung | so, once we enable a bogus admin proejct in keystone, none of the tokens will ever have is_admin_project set | 16:50 |
| ayoung | then we can remove those rules | 16:50 |
| ayoung | it will let a deployer decide when to switch on service roles as the only allowed way to perform those ops | 16:51 |
| lbragstad | why wouldn't we just use system-scope and use the upgrade checks to make sure people have the right role assignments according to their policy? | 16:51 |
| ayoung | lbragstad, so... | 16:51 |
| ayoung | that implied a big bang change | 16:52 |
| ayoung | those never go smoothly | 16:52 |
| ayoung | we want to be able to have people get used to using system roles, but not break their existing workflows | 16:52 |
| lbragstad | but upgrade checkers are a programmable way to help with those types of things? | 16:52 |
| ayoung | will it make sure that Horizon works? | 16:52 |
| ayoung | Will it make sure 3rd party apps work? | 16:52 |
| ayoung | we want to leave the existing policy in place until they are ready to throw the switch | 16:53 |
| ayoung | and give them a way to throw it back | 16:53 |
| ayoung | right now, people are misusing admin tokens | 16:53 |
| ayoung | I've seen some really crappy code along those lines | 16:53 |
| kmalloc | ayoung: that is the idea behind the deprecated policy bits, they just do an logical OR between new and old | 16:54 |
| ayoung | we want to tell people: switch to using "service scoped tokens" and make it their choice | 16:54 |
| ayoung | yeah, but.... | 16:54 |
| kmalloc | until we remove the declaration of the "this is the deprecated rule" | 16:54 |
| ayoung | I don't want to have to try and synchronize this across all of the projects in openstack | 16:54 |
| kmalloc | you are going to have to. | 16:54 |
| ayoung | so...we absoutelty use those | 16:54 |
| kmalloc | it's just how policy works | 16:55 |
| ayoung | re-read what I said | 16:55 |
| ayoung | it allows us to roll in those changes, but keep things working as-is until we throw the switch | 16:55 |
| kmalloc | you can't just wave a wand here. | 16:55 |
| ayoung | I worked long and hard on this wand | 16:55 |
| *** raildo has joined #openstack-meeting-alt | 16:55 | |
| kmalloc | it is going to be a "support (new or old) or supply custom policy" | 16:56 |
| ayoung | so, the idea is we get a common definition of service scoped admin-ness | 16:56 |
| kmalloc | the switch is thrown down the line. | 16:56 |
| ayoung | yes! | 16:56 |
| kmalloc | and it likely will be an upgrade | 16:56 |
| kmalloc | where the old declaration is removed | 16:56 |
| kmalloc | but it COULD be re-added with a custom policy override | 16:56 |
| kmalloc | this has to be done per-project in that project's tree | 16:56 |
| ayoung | what happens if that breaks a critical component? | 16:57 |
| ayoung | they are not going to do a downgrade | 16:57 |
| kmalloc | 3 things: supply a fixed custom policy | 16:57 |
| kmalloc | (quick remediation) | 16:57 |
| kmalloc | 2) do better UAT and/or halt upgrade | 16:57 |
| kmalloc | 3) roll back to previous | 16:57 |
| kmalloc | custom policy to the old policy string is immediate and fixes "critical path is broken" | 16:58 |
| ayoung | So...nothing I am saying is going to break that. But it ain;'t going to work that smoothly | 16:58 |
| ayoung | so... | 16:58 |
| ayoung | here is the middle piece: | 16:58 |
| ayoung | make it an organizational decision to enable and disable the service scoped roles as the ONLY way to enforce that policy | 16:58 |
| ayoung | and isolate that decision | 16:58 |
| lbragstad | final minute | 16:59 |
| kmalloc | this feels like a deployer/installer choice. | 16:59 |
| kmalloc | fwiw | 16:59 |
| ayoung | OK...one other thing | 16:59 |
| kmalloc | not something we can encode directly | 16:59 |
| kmalloc | (just because of how we sucked at building how policy works in the past) | 16:59 |
| ayoung | I propse that the custom policies we discussed last week go to oslo-context instead of olso-policy | 16:59 |
| kmalloc | -2 | 16:59 |
| kmalloc | put them external in a new lib if it doesn't go in oslo-policy | 16:59 |
| ayoung | oslo-context is the openstack specific code. oslo-policy is a generic rules engine. | 16:59 |
| *** derekh has quit IRC | 17:00 | |
| ayoung | there is a dependency between them for this anyway | 17:00 |
| kmalloc | context is the wrong place to put things that are policy rules. | 17:00 |
| ayoung | so is olso-policy, tho | 17:00 |
| kmalloc | oslo context is a holder object for context data. | 17:00 |
| ayoung | but we insist on it for enforcing policy | 17:00 |
| kmalloc | put them in oslo-policy and then extract to new thing | 17:00 |
| kmalloc | or put it in new thing and fight to land it | 17:00 |
| lbragstad | oslo.context is often overridden for service specific implementations, too | 17:00 |
| ayoung | I think it stays in new thing, then | 17:00 |
| kmalloc | do not assume oslo.context even is in use. | 17:00 |
| kmalloc | i told you i recommend olos-policy for one reason only | 17:01 |
| kmalloc | just for ease of landing it | 17:01 |
| kmalloc | then extract | 17:01 |
| lbragstad | ok - we're out of time folks | 17:01 |
| kmalloc | but, i am happy to support a new thing as well | 17:01 |
| ayoung | cool. I'll push for new thing | 17:01 |
| kmalloc | it will just be painful to get adopted (overall) | 17:01 |
| lbragstad | reminder that we have office hours and we can continue there | 17:01 |
| lbragstad | thanks all! | 17:01 |
| kmalloc | but i am fine with +2ing lots of stuff for that as it comes down the line | 17:01 |
| lbragstad | #endmeeting | 17:02 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 17:02 | |
| openstack | Meeting ended Tue Oct 23 17:02:10 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-10-23-16.00.html | 17:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-10-23-16.00.txt | 17:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-10-23-16.00.log.html | 17:02 |
| * kmalloc is trying to not be in the way, but wants to keep things like that isolated to the general places they belong | 17:02 | |
| kmalloc | ayoung: you're not in -keystone btw | 17:02 |
| *** wxy| has quit IRC | 17:05 | |
| *** iyamahat has joined #openstack-meeting-alt | 17:05 | |
| *** iyamahat has quit IRC | 17:05 | |
| *** iyamahat has joined #openstack-meeting-alt | 17:06 | |
| *** e0ne has joined #openstack-meeting-alt | 17:08 | |
| *** iyamahat_ has joined #openstack-meeting-alt | 17:15 | |
| *** lbragstad is now known as lbragstad_f00d | 17:17 | |
| *** iyamahat has quit IRC | 17:18 | |
| *** yamahata has joined #openstack-meeting-alt | 17:24 | |
| *** tpsilva has joined #openstack-meeting-alt | 17:40 | |
| *** e0ne has quit IRC | 17:45 | |
| *** lbragstad_f00d is now known as lbragstad | 17:45 | |
| *** cloudrancher has quit IRC | 18:10 | |
| *** cloudrancher has joined #openstack-meeting-alt | 18:10 | |
| *** apetrich has quit IRC | 18:16 | |
| *** apetrich has joined #openstack-meeting-alt | 18:17 | |
| *** e0ne has joined #openstack-meeting-alt | 18:31 | |
| *** e0ne has quit IRC | 18:31 | |
| *** irclogbot_3 has joined #openstack-meeting-alt | 18:35 | |
| *** panda has quit IRC | 18:45 | |
| *** panda has joined #openstack-meeting-alt | 18:45 | |
| *** chhagarw has quit IRC | 18:57 | |
| *** caboucha has joined #openstack-meeting-alt | 19:11 | |
| *** e0ne has joined #openstack-meeting-alt | 19:15 | |
| *** david-lyle has joined #openstack-meeting-alt | 19:27 | |
| *** dklyle has quit IRC | 19:28 | |
| *** armstrong has quit IRC | 19:46 | |
| *** lbragstad has quit IRC | 19:50 | |
| *** lbragstad has joined #openstack-meeting-alt | 19:53 | |
| *** david-lyle is now known as dklyle | 19:53 | |
| *** cloudrancher has quit IRC | 20:21 | |
| *** cloudrancher has joined #openstack-meeting-alt | 20:22 | |
| *** ttsiouts has joined #openstack-meeting-alt | 20:56 | |
| *** erlon has quit IRC | 21:03 | |
| *** e0ne has quit IRC | 21:04 | |
| *** priteau has quit IRC | 21:09 | |
| *** dustins has quit IRC | 21:19 | |
| *** dklyle has quit IRC | 21:19 | |
| *** dklyle has joined #openstack-meeting-alt | 21:20 | |
| *** raildo has quit IRC | 21:23 | |
| *** cloudrancher has quit IRC | 21:35 | |
| *** cloudrancher has joined #openstack-meeting-alt | 21:36 | |
| *** slaweq has quit IRC | 21:39 | |
| *** cloudrancher has quit IRC | 21:40 | |
| *** cloudrancher has joined #openstack-meeting-alt | 21:40 | |
| *** dustins has joined #openstack-meeting-alt | 21:41 | |
| *** slaweq has joined #openstack-meeting-alt | 22:05 | |
| *** ttsiouts has quit IRC | 22:06 | |
| *** ttsiouts has joined #openstack-meeting-alt | 22:06 | |
| *** ttsiouts has quit IRC | 22:11 | |
| *** dustins has quit IRC | 22:15 | |
| *** rcernin has joined #openstack-meeting-alt | 22:24 | |
| *** bnemec has quit IRC | 22:25 | |
| *** slaweq has quit IRC | 22:38 | |
| *** caboucha has quit IRC | 22:44 | |
| *** diablo_rojo has quit IRC | 22:57 | |
| *** diablo_rojo has joined #openstack-meeting-alt | 23:08 | |
| *** hongbin has quit IRC | 23:10 | |
| *** tpsilva has quit IRC | 23:11 | |
| *** slaweq has joined #openstack-meeting-alt | 23:11 | |
| *** slaweq has quit IRC | 23:45 | |
| *** gyee has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!