| *** chrisshattuck has joined #openstack-keystone | 00:08 | |
| masunkar | Hello, I am trying to setup small openstack deployment, got an issue with keystone install, I am trying on ubuntu 17.04 + octa release, when i am trying to list images it is throwing error 500 on terminal, in logs EmptyCatalog: The service catalog is empty. | 00:08 |
|---|---|---|
| *** chrisshattuck has quit IRC | 00:09 | |
| masunkar | any pointer what might be wrong | 00:09 |
| *** edmondsw has joined #openstack-keystone | 00:09 | |
| masunkar | I can list catalog using openstack catalog list with out issues | 00:11 |
| *** aahh has joined #openstack-keystone | 00:13 | |
| aahh | @lbragstad : is it possible to get the project_id or name inside the identity/backend | 00:13 |
| aahh | without making use of the resource api | 00:13 |
| *** aselius has quit IRC | 00:18 | |
| *** otleimat has quit IRC | 00:23 | |
| *** edmondsw has quit IRC | 00:24 | |
| *** thorst has joined #openstack-keystone | 00:27 | |
| *** edmondsw has joined #openstack-keystone | 00:31 | |
| *** agrebennikov has quit IRC | 00:38 | |
| *** efried has quit IRC | 00:47 | |
| *** efried has joined #openstack-keystone | 00:57 | |
| kmalloc | aahh: you need to use resource_api | 00:59 |
| kmalloc | aahh: you cant guarantee (in many cases) the backend fore resource_api is the same as identity_api. recently we changed some things to allow for FKs between identity and resource | 01:01 |
| kmalloc | aahh: but in *general* it is best practice to request from resource_api if you need data for the project | 01:01 |
| kmalloc | keep in mind keystone will cache the result in the session thread local - so if you request resource_api.get_project(XXXX) multiple times, it will only hit the backend once. | 01:02 |
| *** aahh has quit IRC | 01:03 | |
| *** panbalag has joined #openstack-keystone | 01:05 | |
| *** harlowja has quit IRC | 01:11 | |
| *** thorst has quit IRC | 01:16 | |
| *** itlinux has joined #openstack-keystone | 01:20 | |
| *** edmondsw has quit IRC | 01:21 | |
| *** edmondsw has joined #openstack-keystone | 01:22 | |
| *** Shunli has joined #openstack-keystone | 01:22 | |
| *** ricolin has joined #openstack-keystone | 01:24 | |
| *** zhurong has joined #openstack-keystone | 01:25 | |
| *** thorst has joined #openstack-keystone | 01:28 | |
| *** thorst has quit IRC | 01:46 | |
| masunkar | can somebody help with https://bugs.launchpad.net/keystone/+bug/1715770 | 01:50 |
| openstack | Launchpad bug 1715770 in OpenStack Identity (keystone) "openstack image list throwing 500 error " [Undecided,New] | 01:50 |
| *** thorst has joined #openstack-keystone | 02:07 | |
| *** thorst has quit IRC | 02:07 | |
| *** zhurong has quit IRC | 02:11 | |
| *** zhurong has joined #openstack-keystone | 02:14 | |
| *** panbalag has left #openstack-keystone | 02:15 | |
| *** jamesbenson has joined #openstack-keystone | 02:31 | |
| *** jamesbenson has quit IRC | 02:36 | |
| *** masunkar has quit IRC | 02:45 | |
| *** masunkar has joined #openstack-keystone | 02:45 | |
| *** catintheroof has joined #openstack-keystone | 02:48 | |
| *** catinthe_ has joined #openstack-keystone | 02:50 | |
| *** catintheroof has quit IRC | 02:51 | |
| lbragstad | efried: any red flags here? https://bugs.launchpad.net/keystone/+bug/1715770 | 02:59 |
| openstack | Launchpad bug 1715770 in OpenStack Identity (keystone) "openstack image list throwing 500 error " [Undecided,New] | 02:59 |
| *** nicolasbock has quit IRC | 03:11 | |
| *** markvoelker has quit IRC | 03:12 | |
| *** edmondsw has quit IRC | 03:28 | |
| *** zxy has quit IRC | 03:38 | |
| *** zhouyaguo has joined #openstack-keystone | 03:45 | |
| *** catinthe_ has quit IRC | 03:56 | |
| *** chrisshattuck has joined #openstack-keystone | 03:59 | |
| *** thorst has joined #openstack-keystone | 04:08 | |
| *** thorst has quit IRC | 04:13 | |
| *** links has joined #openstack-keystone | 04:19 | |
| *** zsli_ has joined #openstack-keystone | 04:24 | |
| *** Shunli has quit IRC | 04:27 | |
| *** zhurong has quit IRC | 04:29 | |
| *** edmondsw has joined #openstack-keystone | 04:46 | |
| *** edmondsw has quit IRC | 04:52 | |
| *** zhurong has joined #openstack-keystone | 05:02 | |
| *** thorst has joined #openstack-keystone | 05:09 | |
| *** chrisshattuck has quit IRC | 05:12 | |
| *** markvoelker has joined #openstack-keystone | 05:13 | |
| *** thorst has quit IRC | 05:14 | |
| *** markvoelker has quit IRC | 05:47 | |
| *** aojea has joined #openstack-keystone | 05:48 | |
| *** dims has quit IRC | 05:54 | |
| *** jamesbenson has joined #openstack-keystone | 06:08 | |
| *** thorst has joined #openstack-keystone | 06:10 | |
| *** jamesbenson has quit IRC | 06:12 | |
| *** thorst has quit IRC | 06:15 | |
| *** harlowja has joined #openstack-keystone | 06:19 | |
| *** cfriesen has quit IRC | 06:20 | |
| *** zsli__ has joined #openstack-keystone | 06:25 | |
| *** zsli_ has quit IRC | 06:28 | |
| *** rcernin has joined #openstack-keystone | 06:32 | |
| *** edmondsw has joined #openstack-keystone | 06:35 | |
| *** harlowja has quit IRC | 06:40 | |
| *** edmondsw has quit IRC | 06:40 | |
| *** markvoelker has joined #openstack-keystone | 06:44 | |
| *** itlinux has quit IRC | 06:51 | |
| *** dims has joined #openstack-keystone | 06:51 | |
| openstackgerrit | zhengliuyang proposed openstack/keystone master: Two different API achieve listing role assignments https://review.openstack.org/501975 | 06:57 |
| *** zsli__ has quit IRC | 07:01 | |
| *** Shunli has joined #openstack-keystone | 07:07 | |
| *** thorst has joined #openstack-keystone | 07:11 | |
| *** thorst has quit IRC | 07:16 | |
| *** markvoelker has quit IRC | 07:18 | |
| *** tesseract has joined #openstack-keystone | 07:31 | |
| *** chlong has joined #openstack-keystone | 07:36 | |
| *** chlong has quit IRC | 07:45 | |
| *** thorst has joined #openstack-keystone | 08:12 | |
| *** markvoelker has joined #openstack-keystone | 08:15 | |
| *** thorst has quit IRC | 08:17 | |
| *** ioggstream has joined #openstack-keystone | 08:20 | |
| *** edmondsw has joined #openstack-keystone | 08:23 | |
| *** chlong has joined #openstack-keystone | 08:25 | |
| *** edmondsw has quit IRC | 08:27 | |
| openstackgerrit | Thomas Duval proposed openstack/oslo.policy master: New version of the modification of the HTTPCheck. https://review.openstack.org/501992 | 08:28 |
| *** chlong has quit IRC | 08:35 | |
| *** masunkar has quit IRC | 08:41 | |
| *** markvoelker has quit IRC | 08:49 | |
| *** chlong has joined #openstack-keystone | 08:51 | |
| *** chlong_ has joined #openstack-keystone | 08:54 | |
| *** chlong has quit IRC | 08:58 | |
| *** chlong_ has quit IRC | 09:04 | |
| *** thorst has joined #openstack-keystone | 09:13 | |
| openstackgerrit | Thomas Duval proposed openstack/oslo.policy master: Modification to add additional information in the HTTPCheck request. https://review.openstack.org/498467 | 09:16 |
| *** thorst has quit IRC | 09:17 | |
| *** chlong has joined #openstack-keystone | 09:21 | |
| *** chlong has quit IRC | 09:24 | |
| *** chlong has joined #openstack-keystone | 09:25 | |
| *** Shunli has quit IRC | 09:27 | |
| *** jamesbenson has joined #openstack-keystone | 09:44 | |
| *** jamesbenson has quit IRC | 09:48 | |
| *** chlong has quit IRC | 09:55 | |
| *** zhurong has quit IRC | 10:04 | |
| *** dklyle has joined #openstack-keystone | 10:09 | |
| *** edmondsw has joined #openstack-keystone | 10:11 | |
| *** david-lyle has quit IRC | 10:12 | |
| *** szaher has quit IRC | 10:12 | |
| *** thorst has joined #openstack-keystone | 10:14 | |
| *** szaher has joined #openstack-keystone | 10:15 | |
| *** edmondsw has quit IRC | 10:15 | |
| *** thorst has quit IRC | 10:18 | |
| *** nicolasbock has joined #openstack-keystone | 10:24 | |
| *** nicolasbock has quit IRC | 10:29 | |
| *** nicolasbock has joined #openstack-keystone | 10:42 | |
| *** markvoelker has joined #openstack-keystone | 10:46 | |
| -openstackstatus- NOTICE: Our CI systems experience a hickup, no new jobs are started. Please stay tuned and wait untils this resolved. | 10:47 | |
| *** rcernin has quit IRC | 11:09 | |
| *** thorst has joined #openstack-keystone | 11:14 | |
| *** thorst has quit IRC | 11:19 | |
| *** markvoelker has quit IRC | 11:20 | |
| *** efried is now known as fried_rice | 11:21 | |
| fried_rice | lbragstad Looking... | 11:22 |
| *** tesseract has quit IRC | 11:23 | |
| *** tesseract has joined #openstack-keystone | 11:27 | |
| *** zhouyaguo has quit IRC | 11:27 | |
| *** mvk has quit IRC | 11:31 | |
| fried_rice | lbragstad Appears to be using ksa at 2.18.0 or earlier. (Which I'm sure should be fine - just a data point.) | 11:34 |
| *** rcernin has joined #openstack-keystone | 11:36 | |
| fried_rice | Oh, look, 2.18.0 is in the output. I feel silly for doing my clever sleuthing. | 11:37 |
| *** dklyle has quit IRC | 11:44 | |
| *** dklyle has joined #openstack-keystone | 11:44 | |
| fried_rice | lbragstad Sorry, I don't see anything obvious. | 11:46 |
| fried_rice | What kind of auth is in play here? | 11:47 |
| fried_rice | Was thinking this could be that token expiration deal - but there should be a reauth triggered in BaseIdentityPlugin.get_access just before that call. | 11:47 |
| fried_rice | Oh, duh, also, a short-lived osc process shouldn't be subject to that. | 11:49 |
| *** edmondsw has joined #openstack-keystone | 11:59 | |
| *** thorst has joined #openstack-keystone | 12:02 | |
| *** edmondsw has quit IRC | 12:04 | |
| *** raildo has joined #openstack-keystone | 12:05 | |
| *** edmondsw has joined #openstack-keystone | 12:07 | |
| *** tesseract has quit IRC | 12:14 | |
| *** tesseract has joined #openstack-keystone | 12:14 | |
| *** markvoelker has joined #openstack-keystone | 12:16 | |
| *** ioggstream has quit IRC | 12:28 | |
| *** markvoelker has quit IRC | 12:34 | |
| *** markvoelker has joined #openstack-keystone | 12:34 | |
| lbragstad | fried_rice: yeah - i'm not sure, it looks like keystone populates a service catalog though | 12:41 |
| cmurphy | maybe a silly question but is it really projecct_name in the problem glance config or is it a typo in the bug comment? | 12:46 |
| lbragstad | cmurphy: oh - that's a valid question | 12:49 |
| lbragstad | i'm not sure what keystoneauth does in that case | 12:49 |
| *** burnz has quit IRC | 12:52 | |
| *** burnz has joined #openstack-keystone | 12:53 | |
| *** panbalag has joined #openstack-keystone | 12:57 | |
| fried_rice | lbragstad cmurphy Well, oslo_config behaves as though that line is absent - so what happens if project_name is missing? | 13:04 |
| lbragstad | i would assume it to change scoping | 13:05 |
| lbragstad | which might make sense because unscoped tokens do not contain a service catalog | 13:05 |
| fried_rice | whee, good eye cmurphy! | 13:07 |
| *** Dinesh_Bhor has quit IRC | 13:16 | |
| *** catintheroof has joined #openstack-keystone | 13:26 | |
| *** links has quit IRC | 13:31 | |
| knikolla | o/ morning/afternoon/evening everyone | 13:38 |
| lbragstad | knikolla: o/ | 13:39 |
| knikolla | lbragstad: was wondering what are you using for your blog | 13:40 |
| lbragstad | knikolla: squarespace, | 13:41 |
| *** catinthe_ has joined #openstack-keystone | 13:41 | |
| lbragstad | knikolla: i deployed my own wordpress for a while | 13:41 |
| knikolla | ah, it felt too polished for a wordpress install | 13:41 |
| lbragstad | but migrated to squarespace last spring | 13:41 |
| lbragstad | i know mhayden and cloudnull run really nice wordpress deploys though | 13:41 |
| lbragstad | takes some tinkering and some custom templating, but it can be done | 13:41 |
| lbragstad | i've noticed some outages with squarespace however | 13:42 |
| lbragstad | (they had one this week) | 13:42 |
| lbragstad | for me it's not a real big deal, but I do admin for other folks that have businesses on squarespace, and that's were it becomes more of a problem | 13:42 |
| lbragstad | ghost.io is really nice, too | 13:43 |
| lbragstad | i've only ever run the trial for ghost, but i really liked the simplicity | 13:43 |
| *** catintheroof has quit IRC | 13:44 | |
| knikolla | they sure look nice but way outside of my budget for a dev blog. | 13:44 |
| knikolla | i've been using jekyll/gh pages so far | 13:44 |
| lbragstad | yeah - when i signed up for squarespace the prices were lower and i used a discount | 13:44 |
| knikolla | thinking of going to medium | 13:44 |
| lbragstad | pelican also seems interesting | 13:45 |
| lbragstad | it's all python based | 13:45 |
| -openstackstatus- NOTICE: nodepool issue related to bad images has been resolved, builds should be coming back online soon. Restarted gerrit due to reasons. Happy Friday. | 13:46 | |
| knikolla | can't believe Sydney will be my 4th summit already. | 13:52 |
| lbragstad | kmalloc: responded to https://review.openstack.org/#/c/501885/1 | 13:53 |
| *** jamesbenson has joined #openstack-keystone | 14:15 | |
| *** agrebennikov has joined #openstack-keystone | 14:18 | |
| *** catintheroof has joined #openstack-keystone | 14:22 | |
| *** dave-mccowan has joined #openstack-keystone | 14:23 | |
| *** catinthe_ has quit IRC | 14:25 | |
| *** ioggstream has joined #openstack-keystone | 14:26 | |
| *** chrisshattuck has joined #openstack-keystone | 14:27 | |
| *** gyee has joined #openstack-keystone | 14:31 | |
| *** gyee has quit IRC | 14:39 | |
| *** gyee has joined #openstack-keystone | 14:41 | |
| *** cfriesen has joined #openstack-keystone | 14:45 | |
| *** nicolasbock has quit IRC | 14:47 | |
| *** nicolasbock has joined #openstack-keystone | 14:47 | |
| kmalloc | lbragstad: cool | 14:56 |
| lbragstad | kmalloc: trying to convert to a blanket update statement | 14:58 |
| lbragstad | hitting a few snags | 14:59 |
| *** dklyle has quit IRC | 15:02 | |
| lbragstad | kmalloc: currently, type is a member of the primary key constraint of the assignment table, i assume assignment_type is going to follow that pattern | 15:02 |
| lbragstad | in order to create a primary key constraint, aren't we going to have to make a new assignment table and do a rename? | 15:03 |
| *** david-lyle has joined #openstack-keystone | 15:03 | |
| lbragstad | similar to what we did here - https://github.com/openstack/keystone/blob/master/keystone/common/sql/migrate_repo/versions/073_insert_assignment_inherited_pk.py ? | 15:03 |
| *** agrebennikov has quit IRC | 15:04 | |
| *** agrebennikov has joined #openstack-keystone | 15:04 | |
| *** masunkar has joined #openstack-keystone | 15:05 | |
| *** masunkar_ has joined #openstack-keystone | 15:06 | |
| *** ricolin has quit IRC | 15:07 | |
| *** masunkar has quit IRC | 15:10 | |
| lbragstad | actually - i might have an idea | 15:12 |
| *** links has joined #openstack-keystone | 15:20 | |
| *** itlinux has joined #openstack-keystone | 15:21 | |
| *** links has quit IRC | 15:26 | |
| gagehugo | o/ | 15:38 |
| gagehugo | lbragstad https://review.openstack.org/#/c/494018/ merged | 15:39 |
| *** rmascena has joined #openstack-keystone | 15:39 | |
| lbragstad | nice! | 15:39 |
| *** raildo has quit IRC | 15:41 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Add project tags api-ref documentation and reno https://review.openstack.org/472396 | 15:44 |
| masunkar_ | lbragstad https://bugs.launchpad.net/keystone/+bug/1715770 is updated with few more details | 16:07 |
| openstack | Launchpad bug 1715770 in OpenStack Identity (keystone) "openstack image list throwing 500 error " [Undecided,New] | 16:07 |
| *** tesseract has quit IRC | 16:08 | |
| *** ioggstream has quit IRC | 16:10 | |
| *** masunkar_ has quit IRC | 16:12 | |
| *** otleimat has joined #openstack-keystone | 16:16 | |
| *** masunkar has joined #openstack-keystone | 16:18 | |
| *** mvk has joined #openstack-keystone | 16:19 | |
| kmalloc | knikolla: i wont be in sydney summit. | 16:31 |
| kmalloc | missing two summits in a row. | 16:31 |
| kmalloc | it's a weird feeling | 16:31 |
| kmalloc | wait... 3 sumits in a row. | 16:31 |
| *** szaher has quit IRC | 16:31 | |
| kmalloc | i skipped barcelona, boston, and now sydney | 16:31 |
| knikolla | kmalloc: :( | 16:33 |
| *** szaher has joined #openstack-keystone | 16:37 | |
| *** rcernin has quit IRC | 16:40 | |
| lbragstad | database migrations make my head spin | 16:49 |
| *** harlowja has joined #openstack-keystone | 17:00 | |
| *** aahh has joined #openstack-keystone | 17:07 | |
| *** jmlowe has quit IRC | 17:09 | |
| *** jmlowe has joined #openstack-keystone | 17:10 | |
| *** szaher has quit IRC | 17:15 | |
| kmalloc | yup | 17:17 |
| kmalloc | lbragstad: update assignment set assignment_type = type; | 17:26 |
| kmalloc | lbragstad: should copy the value from assignment.type to assignment.assignment_typel | 17:26 |
| kmalloc | lbragstad: should copy the value from assignment.type to assignment.assignment_type * | 17:26 |
| lbragstad | hmm - i thought i tried that.. let me push what i have here | 17:27 |
| lbragstad | i changed a bunch of stuff | 17:27 |
| kmalloc | you'll need to run it as connection.execute(<sql>) | 17:28 |
| kmalloc | instead of connection.update | 17:29 |
| kmalloc | or whathaveout | 17:29 |
| lbragstad | right | 17:38 |
| lbragstad | finally got the migration passing - but i think this is going to run into edge cases with sql versus sqlite | 17:38 |
| lbragstad | refactoring the backend to understand the new type | 17:38 |
| *** aselius has joined #openstack-keystone | 17:43 | |
| *** panbalag has quit IRC | 17:56 | |
| lbragstad | grabbing lunch quick | 18:00 |
| *** mjax has joined #openstack-keystone | 18:08 | |
| mjax | lbragstad: The token issued from openstack token issue, and the token generated for the user in keystone after password authentication are different right? | 18:22 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Implement project tags logic into manager https://review.openstack.org/499727 | 18:34 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Implement backend logic for project tags https://review.openstack.org/499726 | 18:34 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Add JSON schema validation for project tags https://review.openstack.org/484483 | 18:34 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Add database migration for project tags https://review.openstack.org/484456 | 18:34 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Add policy for project tags https://review.openstack.org/486757 | 18:34 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Refactor removal of duplicate projects/domains https://review.openstack.org/491574 | 18:34 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Implement project tags API controller and router https://review.openstack.org/499728 | 18:34 |
| *** rmascena is now known as raildo | 18:34 | |
| lbragstad | mjax: nope - those tokens are generated using the same process | 18:46 |
| mjax | lbragstad: hmm, I have a bit of confusion. My default configs on devstack have auth_method = password, but every time I make a command (eg openstack image list) it will reprompt me for a password. Does that mean I just need to change the authentication method to token in order to be able to make multiple requests after providing my password once | 18:51 |
| gagehugo | mjax I think openstackclient gets a new token for each command | 18:54 |
| gagehugo | so it'll need your password each time | 18:54 |
| mjax | gagehugo: i see, are there any config settings that I can modify so that it gets a token once, and then keeps using it until it expires? | 18:56 |
| gagehugo | mjax I'm not sure to be honest, I don't use it that often | 18:56 |
| gagehugo | I just export my password as an env variable, but that might not be the most secure | 18:57 |
| mjax | gagehugo: alright. Main reason is because I've written a custom identity driver that makes API calls to our company server for authentication. Wanted to minimize the potential load that could have | 18:57 |
| *** Zara has joined #openstack-keystone | 18:58 | |
| gagehugo | ah | 18:59 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: WIP: Make assignment type not an Enum https://review.openstack.org/501885 | 19:02 |
| lbragstad | kmalloc: ^ that's going to fail 3 tests because of FKs | 19:02 |
| lbragstad | kmalloc: but i already things it's too complicated | 19:03 |
| kmalloc | ? | 19:03 |
| kmalloc | say that last line again? | 19:03 |
| lbragstad | think* | 19:03 |
| kmalloc | ah | 19:03 |
| lbragstad | my fingers started the weekend early | 19:04 |
| kmalloc | uhm. | 19:04 |
| kmalloc | why are you trying to do this via the ORM? | 19:04 |
| kmalloc | it's going to really be super complex | 19:04 |
| kmalloc | and painful | 19:04 |
| lbragstad | yeah - we did that here https://github.com/openstack/keystone/blob/master/keystone/common/sql/migrate_repo/versions/073_insert_assignment_inherited_pk.py | 19:05 |
| kmalloc | i wouldn't do that | 19:05 |
| lbragstad | it is painful and probably wrong | 19:05 |
| kmalloc | i would just use update | 19:05 |
| kmalloc | directly | 19:05 |
| kmalloc | erm | 19:05 |
| kmalloc | execute | 19:05 |
| lbragstad | so - don't create a new table | 19:05 |
| lbragstad | ? | 19:05 |
| kmalloc | right | 19:05 |
| kmalloc | add the column | 19:06 |
| kmalloc | (nullable) | 19:06 |
| kmalloc | .execute('update assignment set assignment_type = type') | 19:06 |
| kmalloc | update new column to non-null | 19:06 |
| kmalloc | it should be done specifically in SQL, not in ORM | 19:08 |
| kmalloc | (the update) | 19:08 |
| kmalloc | the contract phase is where assignment_type is made non-null afaict | 19:08 |
| kmalloc | unless data migrate is assumed to be run when everything has been upgraded | 19:09 |
| kmalloc | again. the whole rolling upgrade bit makes next to no sense to me. | 19:09 |
| kmalloc | it's very inconsistent from what i gather | 19:09 |
| aahh | hi , is there any way to fetch the project name on the identity backend | 19:09 |
| kmalloc | is migrate after everything is upgraded to new code? | 19:09 |
| kmalloc | is it run before code upgrade? | 19:10 |
| lbragstad | nope | 19:10 |
| kmalloc | aahh: use resource_api | 19:10 |
| lbragstad | expand is run with all but one node on N | 19:10 |
| kmalloc | ok | 19:10 |
| lbragstad | migrate is run with all but one node on N | 19:10 |
| kmalloc | so the data update has to be in contract | 19:10 |
| lbragstad | then you start upgrading everything from N to N + 1 | 19:10 |
| kmalloc | again | 19:10 |
| lbragstad | the non-null bit does | 19:11 |
| kmalloc | i think the rolling upgrade process is broken | 19:11 |
| kmalloc | no, you have to do the update *when* everything is on N+1 | 19:11 |
| lbragstad | if you do a rolling upgrade and don't issue a real-only lock | 19:11 |
| kmalloc | you can't have N anywhere, or it might write a non-updated entry into the table | 19:11 |
| lbragstad | on the assignment tabel | 19:11 |
| kmalloc | if you have a non-N+1 node *ever* once the migrate is issued, read lock or not | 19:12 |
| kmalloc | i think the rolling upgrade process is busted . | 19:12 |
| lbragstad | well - if you used an N node to do something to the assignment table you'd get an error | 19:12 |
| lbragstad | saying it's read only or whatever | 19:13 |
| lbragstad | but the data wouldn't be modified | 19:13 |
| kmalloc | then you have to drop the N node before unlocking | 19:13 |
| lbragstad | right | 19:13 |
| kmalloc | and the read lock would affect all nodes | 19:13 |
| lbragstad | that's the trick | 19:13 |
| kmalloc | i.... | 19:13 |
| kmalloc | *shrug* | 19:13 |
| kmalloc | i think the process is poorly designed | 19:13 |
| lbragstad | i'm not saying it's right... i'm just saying it's an alternative | 19:13 |
| kmalloc | the weay i would do it: 1) expand schema | 19:14 |
| kmalloc | 2) update to new code (code smart enough to look in both places/write to both places) | 19:14 |
| kmalloc | 3) data migrate | 19:14 |
| kmalloc | 4) contract (where possible... one cycle out because code is writing to multiple places) | 19:14 |
| kmalloc | anyway. | 19:14 |
| lbragstad | so does that mean we need to support new Enum types for global role assignments this release? | 19:15 |
| lbragstad | if we don't remove type in the migration? | 19:15 |
| kmalloc | hm. | 19:15 |
| lbragstad | and write to both places? | 19:15 |
| kmalloc | except old code cant handle the new types | 19:15 |
| kmalloc | so, i guess that doesn't work either | 19:16 |
| kmalloc | you'll break previous keystones if it gets an enum type it doesn't understand (or any assignment it doesn't get) | 19:17 |
| lbragstad | right | 19:17 |
| kmalloc | you're going to have to make it a new table then ... i guess. | 19:17 |
| lbragstad | or... | 19:17 |
| kmalloc | not just a new column. | 19:17 |
| kmalloc | because we can't have rows that are null for .type | 19:18 |
| kmalloc | it will break old keystones. | 19:18 |
| lbragstad | yeah | 19:19 |
| kmalloc | ok so we have to do: new assignment table | 19:19 |
| aahh | would this be the right way to do it | 19:19 |
| kmalloc | solution | 19:19 |
| aahh | http://paste.openstack.org/show/620751/ | 19:19 |
| kmalloc | lbragstad: make global roles a new assignment table | 19:19 |
| kmalloc | assignment_global | 19:19 |
| kmalloc | and *only* use that for global roles | 19:19 |
| kmalloc | so old code never sees it | 19:20 |
| kmalloc | new code looks at both places. | 19:20 |
| lbragstad | new code only looking in the new global_assignment table for global roles, right? | 19:20 |
| kmalloc | yes | 19:20 |
| lbragstad | it essentially just aggregates everything together | 19:20 |
| lbragstad | yeah - that would work | 19:20 |
| lbragstad | so we just keep assignment.type as an Enum? | 19:21 |
| lbragstad | (i was under the impression the idea behind getting rid of Enum was to avoid migrations when we wanted to support new types) | 19:21 |
| kmalloc | so we make a contract that does the enum type changeover | 19:21 |
| kmalloc | explicitly. | 19:21 |
| kmalloc | and we make the code still do the "enforcement" of types where needed | 19:22 |
| kmalloc | it doesn't matter if enum exists or not to Queens code | 19:22 |
| kmalloc | so we can explicitly deal with changing the type in a contract | 19:22 |
| lbragstad | but let's say we get down the road and want to add another assignment type for some reason | 19:22 |
| lbragstad | are we going to have to add another table or Enum type? | 19:23 |
| kmalloc | right, so we require the contract to happen | 19:23 |
| lbragstad | (require a migration) | 19:23 |
| kmalloc | the contract fixes the table. | 19:23 |
| kmalloc | and does an alter to make it non-enum | 19:23 |
| kmalloc | sec | 19:24 |
| kmalloc | https://www.irccloud.com/pastebin/5Dn7SMAj/ | 19:26 |
| kmalloc | lbragstad: ^ | 19:26 |
| lbragstad | huh | 19:27 |
| lbragstad | ok | 19:27 |
| lbragstad | that makes sense | 19:27 |
| lbragstad | does alter table create a new table and perform a copy? | 19:28 |
| lbragstad | behind the scenes? | 19:28 |
| kmalloc | hm... | 19:28 |
| kmalloc | possibly | 19:28 |
| kmalloc | i am not sure | 19:28 |
| kmalloc | i would put that in a contract phase | 19:28 |
| kmalloc | ftr | 19:29 |
| kmalloc | it is assumed a contract will be run before (if N+1 = Y), you do the Y -> Y+1 upgrade | 19:29 |
| kmalloc | right? | 19:29 |
| lbragstad | the contract is only run after *all* nodes are on N + 1 | 19:30 |
| lbragstad | we don't support running contact with a mixed pool | 19:30 |
| kmalloc | right | 19:30 |
| kmalloc | but before you run Y->Y+1 | 19:30 |
| kmalloc | another upgrade | 19:30 |
| kmalloc | contract will be run | 19:30 |
| kmalloc | ? | 19:30 |
| lbragstad | yes - if i understand the question | 19:30 |
| kmalloc | so, to move from N+1 to N+2, you have to run contract at N+1 | 19:30 |
| lbragstad | 30 expand, 30 migrate, 30 contract, 31 expand, 31 migrate, 31 contract, .... | 19:31 |
| lbragstad | actually - that's not right | 19:31 |
| kmalloc | contract will be run before N+1 -> N+2 happens | 19:32 |
| kmalloc | right? | 19:32 |
| kmalloc | that is all i care about | 19:32 |
| lbragstad | yeah | 19:32 |
| kmalloc | then | 19:32 |
| lbragstad | N is a release or a migration number? | 19:32 |
| kmalloc | in a contract alter | 19:32 |
| kmalloc | release | 19:32 |
| lbragstad | ok | 19:32 |
| lbragstad | yes - all migrations must be run for a release before upgrading to another release | 19:32 |
| kmalloc | good | 19:32 |
| kmalloc | then we can just migrate the enum alter in contract phase | 19:33 |
| lbragstad | (unless the skip-level upgrade session next week changes that) | 19:33 |
| kmalloc | i'd push so very hard against that | 19:33 |
| kmalloc | we're struggling to get plain upgrade to upgrade working | 19:33 |
| lbragstad | i'm not quite sure what skip-level mean | 19:33 |
| lbragstad | means* | 19:33 |
| kmalloc | N->N+2 | 19:33 |
| kmalloc | no N+1 state | 19:33 |
| lbragstad | right - i'm not sure how they plan to implement that though | 19:33 |
| kmalloc | not sure if it's downtime | 19:33 |
| kmalloc | right now, skip-level with downtime is easy | 19:34 |
| lbragstad | it could mean downtime and stopping at N + 1 to run the migrations (which technically would work for us) | 19:34 |
| kmalloc | yep. and since no collapses are happening | 19:34 |
| kmalloc | *shrug* it's pretty trivial to do that | 19:34 |
| kmalloc | and isn't anything crazy | 19:34 |
| lbragstad | ok - so | 19:35 |
| lbragstad | can we document this here - https://etherpad.openstack.org/p/keystone-non-enum-migration ? | 19:35 |
| kmalloc | yeah give me a few | 19:36 |
| kmalloc | need to do a reboot here and toss in some more ram (new ram sticks arrived) | 19:36 |
| lbragstad | ack | 19:37 |
| kmalloc | lbragstad: 128GB of ram ;) | 19:37 |
| lbragstad | kmalloc: you can send any of that my way if you want | 19:37 |
| kmalloc | hehe | 19:37 |
| *** raildo has quit IRC | 19:59 | |
| *** aahh has quit IRC | 20:10 | |
| *** lucasxu has joined #openstack-keystone | 20:14 | |
| *** otleimat has quit IRC | 20:18 | |
| lbragstad | kmalloc: not sure if i've completely documented your approaches https://etherpad.openstack.org/p/keystone-non-enum-migration | 20:26 |
| lbragstad | but i have 4.5 so far | 20:26 |
| kmalloc | i'll look in a sec | 20:26 |
| kmalloc | lbragstad: added to the global_assignment one | 20:28 |
| kmalloc | it hits both marks | 20:28 |
| kmalloc | lbragstad: expanded more actually | 20:32 |
| kmalloc | there ya go | 20:32 |
| kmalloc | Approach #2 is the way I would do it | 20:32 |
| kmalloc | fwiw | 20:32 |
| lbragstad | reading | 20:38 |
| lbragstad | kmalloc: approach #2? | 20:39 |
| lbragstad | or approach #3? | 20:39 |
| kmalloc | sorry #3 | 20:40 |
| kmalloc | the one I added to | 20:40 |
| lbragstad | so - the switch to enum only happens when Queens code is running | 20:40 |
| kmalloc | the switch from enum is queens-only code | 20:41 |
| kmalloc | in the schema | 20:41 |
| kmalloc | which is why it's done in contract | 20:41 |
| kmalloc | technically it could happen in migrate, but i'm playing it super safe here | 20:41 |
| kmalloc | the impact should be zero to convert ENUM -> varchar | 20:42 |
| kmalloc | except, unrecognized values could end up in the column, which would make the non-queens keystone freak the f*** out | 20:42 |
| lbragstad | so - for the duration of the queens release, global roles would be persisted in assignment_global and everything else would be in assignment like it is today | 20:43 |
| kmalloc | yeah | 20:43 |
| kmalloc | and that is just for allowing no-freakouts for pike keystone | 20:43 |
| kmalloc | that said, we could drop the enum in migrate (minor locking outage) | 20:43 |
| kmalloc | and as long as no one created roles that keystone doesn't understand we could put the values for global roles in the .type column | 20:44 |
| kmalloc | we probably want to make queens code resilient to unknown role types | 20:44 |
| kmalloc | as well | 20:44 |
| *** lucasxu has quit IRC | 20:44 | |
| kmalloc | lbragstad: i added to option #5 | 20:46 |
| kmalloc | to clarify what could be done w/o a separate table and just an alter | 20:47 |
| lbragstad | when would you issue the alter? | 20:51 |
| lbragstad | in the contract? | 20:51 |
| kmalloc | in migrate? | 20:51 |
| kmalloc | i am not sure when the best time to do that would be | 20:51 |
| lbragstad | wouldn't that require pike to understand VARCHAR types? | 20:51 |
| kmalloc | oh. yeah still contract. | 20:51 |
| kmalloc | ugh, SQLAlchemy might balk at getting a string back when an enum is expected | 20:51 |
| kmalloc | we might not be able to do #5 | 20:52 |
| lbragstad | yeah - it does | 20:52 |
| kmalloc | we'd have to ask zzzeek | 20:52 |
| lbragstad | knikolla: ran into that yesterday | 20:52 |
| kmalloc | ok so #5 is off the table | 20:52 |
| kmalloc | we might be able to make a magic column type that can handle either ENUM or VARCHAR | 20:52 |
| kmalloc | ... but that might be a lot of work | 20:52 |
| lbragstad | kmalloc: that's probably above my sql skill level | 20:53 |
| lbragstad | but if someone has a way to do it i'm all ears | 20:53 |
| kmalloc | i could try | 20:55 |
| kmalloc | it's a lot of work. | 20:55 |
| kmalloc | let me poke at something real quick | 20:55 |
| lbragstad | kmalloc: no need to spin cycles on it if you don't have the bandwidth | 20:55 |
| lbragstad | kmalloc: let's say we do number #3 | 20:59 |
| kmalloc | okie | 20:59 |
| lbragstad | we could just have the queens code use two different sql models pointing to two separate tables | 20:59 |
| lbragstad | like you said | 20:59 |
| *** otleimat has joined #openstack-keystone | 20:59 | |
| lbragstad | if the queens code is asked to store a global role assignment, it does so in the assignment_global table | 21:00 |
| lbragstad | and during an upgrade let's say someone goes to validate a global role assignment against a pike node | 21:00 |
| lbragstad | it would fail, because pike doesn't understand that | 21:01 |
| lbragstad | right? | 21:01 |
| kmalloc | right | 21:01 |
| lbragstad | is that a problem? | 21:01 |
| kmalloc | it wouldn't know what the role is | 21:01 |
| kmalloc | it wouldn't even get the global role | 21:01 |
| lbragstad | to be able to have successful global role action for a pool of keystone nodes and have the next request not understand it? | 21:01 |
| lbragstad | (from a data model it would be fine, but from a mixed API upgrade perspective, it might seem inconsistent or confusing) | 21:02 |
| kmalloc | this is why i think the rolling upgrade is ludicrous | 21:03 |
| kmalloc | the API may respond differently between versions | 21:03 |
| kmalloc | this is crazypants imo | 21:03 |
| * lbragstad sigh | 21:03 | |
| kmalloc | you always have this issue running multiple versions of keystone | 21:03 |
| lbragstad | right | 21:04 |
| kmalloc | it's not that the API contract has changed | 21:04 |
| lbragstad | what if we say screw it and make it a partial outage | 21:04 |
| kmalloc | i'm fine with that | 21:04 |
| lbragstad | assignments are read-only if you want to do a rolling upgrade | 21:04 |
| lbragstad | that's the best we can do | 21:04 |
| kmalloc | that is just an addendum to approach #3 | 21:04 |
| kmalloc | the rest still needs to be that way because ENUM vs VARCHAR | 21:05 |
| lbragstad | i'm going to walk through a read-only case | 21:05 |
| *** rcernin has joined #openstack-keystone | 21:06 | |
| kmalloc | okie | 21:10 |
| *** edmondsw has quit IRC | 21:21 | |
| *** thorst has quit IRC | 21:32 | |
| lbragstad | kmalloc: modified approach #4 and added #4a | 21:39 |
| lbragstad | kmalloc: thoughts on #4a? | 21:43 |
| *** mvk has quit IRC | 21:48 | |
| *** ioggstream has joined #openstack-keystone | 21:55 | |
| *** jamesbenson has quit IRC | 21:59 | |
| *** jamesbenson has joined #openstack-keystone | 22:04 | |
| *** mvk has joined #openstack-keystone | 22:04 | |
| *** catintheroof has quit IRC | 22:08 | |
| *** jamesbenson has quit IRC | 22:08 | |
| kmalloc | 4a looks ok | 22:19 |
| *** rcernin has quit IRC | 22:19 | |
| kmalloc | it's still an outage | 22:19 |
| kmalloc | and might be a hard down vs partial outage | 22:20 |
| kmalloc | new keystone can't read non-enum | 22:20 |
| kmalloc | oh | 22:20 |
| kmalloc | assignment vs assignments | 22:20 |
| kmalloc | yeah | 22:20 |
| kmalloc | that is fine. | 22:20 |
| lbragstad | yeah | 22:21 |
| lbragstad | it's tricky :) | 22:21 |
| kmalloc | still a bit wonky | 22:21 |
| kmalloc | but not bad | 22:21 |
| kmalloc | *shrug* | 22:21 |
| kmalloc | you're still going to get "different responses" from the global role enabled keystone if a global role is created | 22:21 |
| lbragstad | well- a global role won't be createable until after the migration | 22:22 |
| lbragstad | assignments have to be read only | 22:22 |
| kmalloc | ok | 22:23 |
| kmalloc | then | 22:23 |
| kmalloc | not terrible | 22:23 |
| kmalloc | it just means queens is not really rolling upgradable | 22:23 |
| kmalloc | it's partial outage | 22:23 |
| lbragstad | i can't seem to find a way to make it consistent and writable during a rolling upgrade | 22:23 |
| lbragstad | right | 22:23 |
| *** aojea has quit IRC | 22:23 | |
| lbragstad | if you have pike and queens running at the same time and way writeable backends... you're going to have to deal with possible inconsistent APIs for the duration of the migration | 22:24 |
| lbragstad | want writeable* | 22:24 |
| kmalloc | i really think the rolling update should be scratched. | 22:31 |
| kmalloc | tbh | 22:31 |
| kmalloc | i have no issue with minimal downtime goal, aka schema can be upgraded independently in most cases from the code | 22:32 |
| kmalloc | but we should stop trying to make a guarantee that you can actually run different versions of keystone | 22:32 |
| *** thorst has joined #openstack-keystone | 22:32 | |
| kmalloc | if we ever update an api (especially around auth) with featuresets like global roles, it's not an API contract break really | 22:33 |
| kmalloc | it just means the API is returning additional (and sane) content | 22:33 |
| kmalloc | in accordance with the contract | 22:33 |
| *** thorst has quit IRC | 22:37 | |
| *** dave-mccowan has quit IRC | 22:39 | |
| *** edmondsw has joined #openstack-keystone | 22:49 | |
| *** chrisshattuck has quit IRC | 22:51 | |
| *** ioggstream has quit IRC | 22:53 | |
| *** edmondsw has quit IRC | 22:54 | |
| *** thorst has joined #openstack-keystone | 23:14 | |
| *** thorst has quit IRC | 23:19 | |
| *** otleimat has quit IRC | 23:19 | |
| *** gyee has quit IRC | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!