| *** panbalag has joined #openstack-keystone | 00:10 | |
| *** dave-mccowan has quit IRC | 00:10 | |
| *** odyssey4me has quit IRC | 00:11 | |
| *** odyssey4me has joined #openstack-keystone | 00:11 | |
| *** mvk has quit IRC | 00:16 | |
| *** panbalag has quit IRC | 00:18 | |
| *** rcernin has quit IRC | 00:19 | |
| *** rcernin has joined #openstack-keystone | 00:19 | |
| *** eEbx has quit IRC | 00:21 | |
| *** eEbx has joined #openstack-keystone | 00:21 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 00:24 | |
| *** zhurong has joined #openstack-keystone | 00:27 | |
| *** mvk has joined #openstack-keystone | 00:29 | |
| *** NM has joined #openstack-keystone | 00:38 | |
| *** felipemonteiro has quit IRC | 00:39 | |
| *** NM has quit IRC | 00:43 | |
| *** harlowja has quit IRC | 00:53 | |
| *** voelzmo has joined #openstack-keystone | 01:09 | |
| *** annp has quit IRC | 01:10 | |
| *** annp has joined #openstack-keystone | 01:11 | |
| *** felipemonteiro has joined #openstack-keystone | 01:19 | |
| *** gyankum has joined #openstack-keystone | 01:27 | |
| *** voelzmo has quit IRC | 01:32 | |
| *** germs has joined #openstack-keystone | 01:52 | |
| *** germs has quit IRC | 01:52 | |
| *** germs has joined #openstack-keystone | 01:52 | |
| openstackgerrit | wanghui proposed openstack/keystone master: Move openstackdocstheme to extensions in api-ref https://review.openstack.org/556704 | 01:55 |
|---|---|---|
| *** germs has quit IRC | 01:57 | |
| *** zhurong has quit IRC | 01:58 | |
| *** voelzmo has joined #openstack-keystone | 02:05 | |
| *** dave-mccowan has joined #openstack-keystone | 02:15 | |
| *** namnh has joined #openstack-keystone | 02:16 | |
| *** felipemonteiro has quit IRC | 02:20 | |
| *** dikonoo has joined #openstack-keystone | 02:21 | |
| *** dikonoor has quit IRC | 02:21 | |
| *** gyee has quit IRC | 02:26 | |
| *** daidv has joined #openstack-keystone | 02:34 | |
| *** voelzmo has quit IRC | 02:38 | |
| *** voelzmo has joined #openstack-keystone | 03:01 | |
| *** zhurong has joined #openstack-keystone | 03:23 | |
| *** voelzmo has quit IRC | 03:34 | |
| *** dave-mccowan has quit IRC | 03:49 | |
| *** voelzmo has joined #openstack-keystone | 03:51 | |
| *** germs has joined #openstack-keystone | 03:53 | |
| *** germs has quit IRC | 03:53 | |
| *** germs has joined #openstack-keystone | 03:53 | |
| *** germs has quit IRC | 03:58 | |
| openstackgerrit | wangxiyuan proposed openstack/oslo.limit master: Init repo https://review.openstack.org/556744 | 04:02 |
| *** voelzmo has quit IRC | 04:16 | |
| *** annp has quit IRC | 04:26 | |
| *** markvoelker has quit IRC | 04:27 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 04:33 | |
| *** AlexeyAbashkin has quit IRC | 04:37 | |
| *** zhurong has quit IRC | 04:39 | |
| *** Dinesh_Bhor has quit IRC | 05:01 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 05:03 | |
| *** dikonoo has quit IRC | 05:11 | |
| *** dikonoo has joined #openstack-keystone | 05:12 | |
| *** Mujahid has joined #openstack-keystone | 05:20 | |
| *** Mujahid has quit IRC | 05:25 | |
| *** markvoelker has joined #openstack-keystone | 05:28 | |
| *** germs has joined #openstack-keystone | 05:54 | |
| *** germs has quit IRC | 05:54 | |
| *** germs has joined #openstack-keystone | 05:54 | |
| *** germs has quit IRC | 05:59 | |
| *** dangtrinhnt has joined #openstack-keystone | 06:11 | |
| *** annp has joined #openstack-keystone | 06:12 | |
| *** namnh_ has joined #openstack-keystone | 06:17 | |
| *** namnh has quit IRC | 06:20 | |
| *** namnh has joined #openstack-keystone | 06:21 | |
| *** namnh_ has quit IRC | 06:21 | |
| *** namnh_ has joined #openstack-keystone | 06:22 | |
| *** namnh has quit IRC | 06:26 | |
| *** aojea has joined #openstack-keystone | 06:31 | |
| *** martinus__ has joined #openstack-keystone | 06:40 | |
| *** aojea has quit IRC | 06:40 | |
| *** pcaruana has joined #openstack-keystone | 06:53 | |
| *** jaosorior has quit IRC | 07:02 | |
| *** tesseract has joined #openstack-keystone | 07:15 | |
| *** rcernin has quit IRC | 07:25 | |
| *** gongysh has joined #openstack-keystone | 07:43 | |
| *** dims_ has joined #openstack-keystone | 07:48 | |
| *** dims has quit IRC | 07:49 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 07:51 | |
| *** zhurong has joined #openstack-keystone | 07:53 | |
| *** jaosorior has joined #openstack-keystone | 07:53 | |
| *** germs has joined #openstack-keystone | 07:54 | |
| *** germs has quit IRC | 07:54 | |
| *** germs has joined #openstack-keystone | 07:54 | |
| *** germs has quit IRC | 07:59 | |
| *** mvk has quit IRC | 08:27 | |
| *** gyankum has quit IRC | 08:41 | |
| *** gyan_ has joined #openstack-keystone | 08:41 | |
| *** gyan__ has joined #openstack-keystone | 08:44 | |
| *** gyan_ has quit IRC | 08:47 | |
| *** mvk has joined #openstack-keystone | 08:55 | |
| *** Dinesh_Bhor has quit IRC | 08:56 | |
| *** zhurong has quit IRC | 09:00 | |
| *** ilush has joined #openstack-keystone | 09:20 | |
| *** mvk has quit IRC | 09:36 | |
| *** mvk has joined #openstack-keystone | 09:37 | |
| *** germs has joined #openstack-keystone | 09:55 | |
| *** germs has quit IRC | 10:00 | |
| *** gyan_ has joined #openstack-keystone | 10:03 | |
| *** gyan__ has quit IRC | 10:05 | |
| *** gyan__ has joined #openstack-keystone | 10:05 | |
| *** gyan_ has quit IRC | 10:08 | |
| *** gongysh has quit IRC | 10:11 | |
| *** namnh_ has quit IRC | 10:18 | |
| *** gongysh has joined #openstack-keystone | 10:28 | |
| *** zhurong has joined #openstack-keystone | 10:31 | |
| *** gyan__ has quit IRC | 10:34 | |
| *** gyan__ has joined #openstack-keystone | 10:36 | |
| *** AlexeyAbashkin has quit IRC | 11:05 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 11:07 | |
| *** marius1 has joined #openstack-keystone | 11:08 | |
| *** openstackgerrit has quit IRC | 11:33 | |
| *** gyan__ has quit IRC | 11:34 | |
| *** marius1 has quit IRC | 11:42 | |
| *** panbalag has joined #openstack-keystone | 11:50 | |
| *** ilush has quit IRC | 11:52 | |
| *** panbalag has quit IRC | 11:54 | |
| *** gongysh has quit IRC | 11:54 | |
| *** germs has joined #openstack-keystone | 11:56 | |
| *** germs has quit IRC | 11:56 | |
| *** germs has joined #openstack-keystone | 11:56 | |
| *** germs has quit IRC | 12:01 | |
| *** edmondsw has joined #openstack-keystone | 12:13 | |
| *** ilush has joined #openstack-keystone | 12:19 | |
| *** raildo has joined #openstack-keystone | 12:20 | |
| *** NM has joined #openstack-keystone | 12:27 | |
| *** zhurong has quit IRC | 12:27 | |
| *** markvoelker has quit IRC | 12:28 | |
| *** markvoelker has joined #openstack-keystone | 12:28 | |
| *** aojea has joined #openstack-keystone | 12:38 | |
| *** voelzmo has joined #openstack-keystone | 12:40 | |
| *** dave-mccowan has joined #openstack-keystone | 12:41 | |
| *** odyssey4me has quit IRC | 12:43 | |
| *** odyssey4me has joined #openstack-keystone | 12:43 | |
| *** voelzmo has quit IRC | 12:53 | |
| *** aojea has quit IRC | 12:54 | |
| *** tmcm has joined #openstack-keystone | 13:04 | |
| tmcm | hello. | 13:04 |
| tmcm | i'm encountering some problems with a keystone database migration | 13:04 |
| tmcm | 2018-03-27 08:57:12.882 22821 INFO migrate.versioning.api [-] 13 -> 14... | 13:04 |
| tmcm | 2018-03-27 08:57:12.949 22821 CRITICAL keystone [-] Unhandled error: DBMigrationError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`user`, CONSTRAINT `user_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `project` (`id`))') [SQL: u'UPDATE user SET domain_id=%(domain_id)s WHERE user.id = %(id_1)s'] [parameters: {u'id_1': u'74aad9087e11452babbe657276b4d006', 'domain_id': | 13:04 |
| tmcm | u'0b7ac3f80ecf4347a683c5ec8570bf6a'}] | 13:04 |
| tmcm | has anyone seen something like this before? | 13:05 |
| *** panbalag has joined #openstack-keystone | 13:17 | |
| *** jdennis has quit IRC | 13:19 | |
| *** jdennis has joined #openstack-keystone | 13:20 | |
| knikolla | lbragstad: is this the same as that other one? ^^ | 13:21 |
| *** jaosorior has quit IRC | 13:33 | |
| *** panbalag has quit IRC | 13:37 | |
| lbragstad | knikolla: looksl ike it | 13:42 |
| knikolla | lbragstad: makes me wonder if we can at all support online migrations | 13:43 |
| knikolla | that specific migration is going to lock the table for reads too | 13:43 |
| knikolla | or fail when a read is happening. | 13:43 |
| lbragstad | or propose a backport that allows the migration to be successful | 13:44 |
| knikolla | oh wait. this is in mysql. the other was in postgresql? | 13:44 |
| lbragstad | that might be a good thing to work on for office hours | 13:44 |
| lbragstad | yeah - one of the ones i looked at was using postgres | 13:45 |
| *** panbalag has joined #openstack-keystone | 13:45 | |
| *** mchlumsky has joined #openstack-keystone | 13:48 | |
| tmcm | lbragstad: should it be possible to upgrade the keystone db from mitaka to queens? | 13:49 |
| lbragstad | it should be... | 13:49 |
| lbragstad | all migrations should be in available in the queens source | 13:50 |
| cmurphy | my proposal to drop that foreign key is looking a whole lot better now i bet :P | 13:50 |
| lbragstad | yeah - it is | 13:50 |
| lbragstad | someone opened a bug for this last time we talked about it, right? | 13:51 |
| lbragstad | this is the one? https://bugs.launchpad.net/keystone/+bug/1755906 | 13:51 |
| openstack | Launchpad bug 1755906 in OpenStack Identity (keystone) "Occasional deadlock during db_sync --contract during Newton to Pike live upgrade" [High,Confirmed] | 13:52 |
| tmcm | i see the following: | 13:52 |
| tmcm | 2018-03-27 09:51:23.348 24059 INFO migrate.versioning.api [-] 108 -> 109... | 13:52 |
| tmcm | 2018-03-27 09:51:23.793 24059 INFO migrate.versioning.api [-] done | 13:52 |
| tmcm | 2018-03-27 09:51:23.914 24059 INFO migrate.versioning.api [-] 0 -> 1... | 13:52 |
| tmcm | 2018-03-27 09:51:23.945 24059 INFO migrate.versioning.api [-] done | 13:52 |
| tmcm | is that expected? | 13:52 |
| tmcm | i'm starting at 96 | 13:52 |
| lbragstad | tmcm: what you're seeing there is all the old/legacy migrations being run | 13:53 |
| tmcm | ok | 13:53 |
| lbragstad | for context - all migration scripts were kept in a single directory prior to rolling upgrade support | 13:53 |
| lbragstad | https://github.com/openstack/keystone/tree/master/keystone/common/sql/migrate_repo/versions | 13:53 |
| cmurphy | lbragstad: yeah that's the one | 13:54 |
| lbragstad | then when we started working on the rolling upgrade requirements, we essentially froze that migration repository in favor of the expand, migrate, contract, repositories (which are located two directories up) | 13:54 |
| lbragstad | https://github.com/openstack/keystone/tree/master/keystone/common/sql | 13:55 |
| lbragstad | tmcm: so - on a fresh install for example, db_sync will run all the "legacy" migrations, then it will run all the "expand" scripts, then "migrate", and finish with "contract" | 13:55 |
| tmcm | good, your explanation matches my understanding | 13:56 |
| tmcm | however, i'm crashing in migrate | 13:56 |
| tmcm | 2018-03-27 09:55:16.209 24143 INFO migrate.versioning.api [-] 13 -> 14... | 13:57 |
| tmcm | 2018-03-27 09:55:16.368 24143 CRITICAL keystone [-] Unhandled error: DBMigrationError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`user`, CONSTRAINT `user_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `project` (`id`))') [SQL: u'UPDATE user SET domain_id=%(domain_id)s WHERE user.id = %(id_1)s'] [parameters: {u'id_1': u'74aad9087e11452babbe657276b4d006', 'domain_id': | 13:57 |
| tmcm | u'0b7ac3f80ecf4347a683c5ec8570bf6a'}] | 13:57 |
| *** germs has joined #openstack-keystone | 13:57 | |
| tmcm | looking at that module now | 13:57 |
| lbragstad | tmcm: would you be able to put a full trace in http://paste.openstack.org/ ? | 13:57 |
| lbragstad | sans sensitive information, if any... | 13:58 |
| tmcm | i'm looking in the repo now. what is the difference between migrate_repo and data_migration_repo | 13:59 |
| *** jamielennox has quit IRC | 14:01 | |
| *** germs has quit IRC | 14:02 | |
| *** jamielennox has joined #openstack-keystone | 14:02 | |
| *** r-daneel has joined #openstack-keystone | 14:02 | |
| lbragstad | tmcm: migrate_repo is the "legacy" repository | 14:02 |
| tmcm | got it | 14:02 |
| lbragstad | tmcm: https://docs.openstack.org/keystone/latest/contributor/database-migrations.html does a better job explaining the purpose and use of each | 14:02 |
| tmcm | thank you | 14:03 |
| *** jaosorior has joined #openstack-keystone | 14:03 | |
| tmcm | http://paste.openstack.org/show/715408/ | 14:04 |
| tmcm | that is from just "keystone-manage db_sync" | 14:05 |
| lbragstad | hmm - that looks slightly different that what was reported in https://bugs.launchpad.net/keystone/+bug/1755906 | 14:07 |
| openstack | Launchpad bug 1755906 in OpenStack Identity (keystone) "Occasional deadlock during db_sync --contract during Newton to Pike live upgrade" [High,Confirmed] | 14:07 |
| lbragstad | https://github.com/openstack/keystone/blob/9c2e977b7e1764b20a9d2f5b0df44869445f50c5/keystone/common/sql/data_migration_repo/versions/014_migrate_add_domain_id_to_user_table.py | 14:08 |
| lbragstad | this was actually something that was written in ocata, i think | 14:09 |
| tmcm | yeah, i'm looking at that now | 14:09 |
| lbragstad | does domain 0b7ac3f80ecf4347a683c5ec8570bf6a not exist? | 14:10 |
| tmcm | actually, it looks pretty similar to me given that #1755906 references postgres and i've got mariadb | 14:11 |
| tmcm | umm | 14:11 |
| tmcm | hold on | 14:11 |
| lbragstad | the trace from 1755906 is showing it failing during contract due to deadlock | 14:11 |
| tmcm | oh | 14:12 |
| tmcm | hrm | 14:12 |
| tmcm | ugh, damn | 14:16 |
| lbragstad | it looks like it's tripping on https://github.com/openstack/keystone/blob/9c2e977b7e1764b20a9d2f5b0df44869445f50c5/keystone/common/sql/data_migration_repo/versions/014_migrate_add_domain_id_to_user_table.py#L43-L45 | 14:16 |
| tmcm | that domain does not exist | 14:16 |
| tmcm | i wonder htf that happened | 14:16 |
| *** ykarel has joined #openstack-keystone | 14:17 | |
| ykarel | Hi can someone look my comment on https://review.openstack.org/#/c/543060/5 and confirm | 14:17 |
| lbragstad | tmcm: is that a federated user? | 14:17 |
| lbragstad | looks like it might be by reading the migration | 14:18 |
| lbragstad | ykarel: oh - yes... that will get cleaned up | 14:18 |
| lbragstad | or it can | 14:18 |
| lbragstad | we have a couple configuration options to clean up, too | 14:18 |
| ykarel | lbragstad, is it tracked somewhere, bug or bp? | 14:19 |
| lbragstad | ykarel: a bug would work | 14:19 |
| lbragstad | or we could just remove it and associate it with the removed-as-of-rocky blueprint | 14:19 |
| lbragstad | the second option is more appropriate, imo | 14:19 |
| ykarel | lbragstad, Ok | 14:20 |
| ykarel | please track it somewhere so it isn't missed | 14:20 |
| lbragstad | ykarel: if it makes you feel better, feel free to open a bug | 14:21 |
| *** panbalag has left #openstack-keystone | 14:23 | |
| ykarel | lbragstad, Ok will open may be tomorrow | 14:24 |
| tmcm | | 8 | 74aad9087e11452babbe657276b4d006 | 0b7ac3f80ecf4347a683c5ec8570bf6a | heat_domain_admin | | 14:25 |
| tmcm | i should probably just nuke that. heat is not even truly activated in my mitaka production cloud | 14:27 |
| lbragstad | ykarel: done - https://bugs.launchpad.net/keystone/+bug/1759289 | 14:28 |
| openstack | Launchpad bug 1759289 in OpenStack Identity (keystone) "keystone-manage token_flush fails unexpectedly" [Undecided,New] | 14:28 |
| ykarel | lbragstad, ack | 14:28 |
| *** ykarel is now known as ykarel|away | 14:29 | |
| lbragstad | tmcm: interesting, that user must have existed prior to the domain unique constraint | 14:29 |
| tmcm | i believe so | 14:29 |
| lbragstad | tmcm: you could try and create that domain | 14:31 |
| lbragstad | or just create *a* domain, then update that user's domain to the freshly created one | 14:31 |
| lbragstad | (just incase anyone is using that account) | 14:31 |
| lbragstad | but it doesn't sound like that is the case | 14:31 |
| lbragstad | kmalloc: do we need to deprecate keystone-manage commands when the backing functionality has been removed (e.g. keystone-manage token_flush)? | 14:32 |
| tmcm | i'm certain no one is using that account | 14:33 |
| tmcm | deleting that user before the db_sync allows it to finish | 14:33 |
| tmcm | thanks! | 14:33 |
| lbragstad | tmcm: no problem | 14:33 |
| *** wxy| has joined #openstack-keystone | 14:33 | |
| *** mchlumsky has quit IRC | 14:34 | |
| tmcm | on to the next database migration problem :) (glance, iirc) | 14:34 |
| lbragstad | lol | 14:39 |
| *** spilla has joined #openstack-keystone | 14:39 | |
| *** mchlumsky has joined #openstack-keystone | 14:41 | |
| kmalloc | lbragstad: deprecate, yeah, just so we don't break anyone (make them do nothing but emit a warning) | 14:43 |
| *** ykarel|away has quit IRC | 14:46 | |
| *** felipemonteiro has joined #openstack-keystone | 14:57 | |
| *** felipemonteiro_ has joined #openstack-keystone | 15:04 | |
| *** voelzmo has joined #openstack-keystone | 15:06 | |
| *** pcaruana has quit IRC | 15:10 | |
| *** dikonoo has quit IRC | 15:11 | |
| *** jaosorior has quit IRC | 15:12 | |
| *** germs has joined #openstack-keystone | 15:13 | |
| *** germs has quit IRC | 15:13 | |
| *** germs has joined #openstack-keystone | 15:13 | |
| *** ykarel|away has joined #openstack-keystone | 15:15 | |
| *** germs has quit IRC | 15:18 | |
| *** ilush has quit IRC | 15:20 | |
| *** openstackgerrit has joined #openstack-keystone | 15:26 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Log warning when using token_flush https://review.openstack.org/556889 | 15:26 |
| *** gagehugo has quit IRC | 15:43 | |
| *** gagehugo has joined #openstack-keystone | 15:44 | |
| *** spilla has quit IRC | 15:48 | |
| *** spilla has joined #openstack-keystone | 15:48 | |
| *** spilla has quit IRC | 15:53 | |
| *** gyee has joined #openstack-keystone | 16:00 | |
| kmalloc | Lbradstad: I would, so we don't break tooling, but we can gut it to just a warning "hey don't do this" in one fell swoop | 16:00 |
| *** germs has joined #openstack-keystone | 16:01 | |
| *** germs has quit IRC | 16:01 | |
| *** germs has joined #openstack-keystone | 16:01 | |
| kmalloc | Irccloud on mobile browser got worse, no name completion now :( | 16:01 |
| lbragstad | kmalloc: i took a crack at it https://review.openstack.org/#/c/556889/ | 16:02 |
| kmalloc | Cool. | 16:02 |
| lbragstad | reminder that the keystone weekly meeting is happening in #openstack-meeting-alt | 16:02 |
| *** voelzmo has quit IRC | 16:08 | |
| *** germs_ has joined #openstack-keystone | 16:08 | |
| *** germs has quit IRC | 16:10 | |
| *** dave-mccowan has quit IRC | 16:10 | |
| *** mvk has quit IRC | 16:11 | |
| *** germs_ has quit IRC | 16:11 | |
| *** germs has joined #openstack-keystone | 16:12 | |
| *** germs has quit IRC | 16:12 | |
| *** germs has joined #openstack-keystone | 16:12 | |
| *** germs has quit IRC | 16:22 | |
| *** pcaruana has joined #openstack-keystone | 16:25 | |
| *** felipemonteiro has quit IRC | 16:28 | |
| *** germs has joined #openstack-keystone | 16:29 | |
| *** germs has quit IRC | 16:29 | |
| *** germs has joined #openstack-keystone | 16:29 | |
| *** germs has quit IRC | 16:32 | |
| openstackgerrit | Merged openstack/keystone master: Updated from global requirements https://review.openstack.org/556405 | 16:33 |
| *** pcichy has joined #openstack-keystone | 16:37 | |
| *** germs has joined #openstack-keystone | 16:43 | |
| *** germs has quit IRC | 16:43 | |
| *** germs has joined #openstack-keystone | 16:43 | |
| *** felipemonteiro_ has quit IRC | 16:45 | |
| *** david-lyle has joined #openstack-keystone | 16:46 | |
| *** jgr is now known as jgrassler | 16:53 | |
| *** dikonoo has joined #openstack-keystone | 16:53 | |
| *** germs_ has joined #openstack-keystone | 16:55 | |
| *** germs has quit IRC | 16:56 | |
| *** dave-mccowan has joined #openstack-keystone | 16:56 | |
| *** NM has quit IRC | 16:56 | |
| hrybacki | proposed OO topics: Grooming the list we just talked about and then doing spec reviews we didn't get to? | 17:00 |
| *** NM has joined #openstack-keystone | 17:00 | |
| *** NM has quit IRC | 17:01 | |
| lbragstad | #startmeeting keystone-office-hours | 17:01 |
| openstack | Meeting started Tue Mar 27 17:01:33 2018 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:01 |
| *** openstack changes topic to " (Meeting topic: keystone-office-hours)" | 17:01 | |
| *** ChanServ changes topic to "Queens release schedule: https://releases.openstack.org/queens/schedule.html | Meeting agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting | Bugs that need triaging: http://bit.ly/2iJuN1h | Trello: https://trello.com/b/wmyzbFq5/keystone-rocky-roadmap" | 17:01 | |
| openstack | The meeting name has been set to 'keystone_office_hours' | 17:01 |
| hrybacki | bridge is open btw: https://redhat.bluejeans.com/8559013623/ | 17:01 |
| * lbragstad grabs water quick | 17:02 | |
| lbragstad | i'll be on in about 3 minutes | 17:02 |
| * hrybacki visits a water closet | 17:02 | |
| *** NM has joined #openstack-keystone | 17:03 | |
| cmurphy | :/ guys i love your faces but i can't have a multi-hour video conference every week, that starts to feel a lot like work | 17:03 |
| gagehugo | when you have them everyday, why not have another one? | 17:05 |
| gagehugo | :( | 17:05 |
| hrybacki | video calls are the only way I can stay on topic these days -_- my current role is wrecking me | 17:05 |
| *** wxy| has quit IRC | 17:05 | |
| cmurphy | i'm starting an etherpad for the help wanted list https://etherpad.openstack.org/p/keystone-help-wanted-list | 17:08 |
| cmurphy | if you need my input on the call just ping me | 17:08 |
| hrybacki | adding that to the OO etherpad cmurphy | 17:10 |
| kmalloc | cmurphy: can i entice you with doggo cam again? (soon to be puppy cam too) | 17:10 |
| kmalloc | (2 more weeks and puppy arrives) | 17:11 |
| cmurphy | kmalloc: hmmmmmmmm maybe | 17:11 |
| kmalloc | cmurphy: ooh sec. | 17:11 |
| cmurphy | puppies aside a video call isn't easily reconsumeable even if it's recorded, so it's harder for say wxy to come back to it and figure out what happened | 17:12 |
| *** felipemonteiro_ has joined #openstack-keystone | 17:14 | |
| lbragstad | true | 17:17 |
| kmalloc | https://usercontent.irccloud-cdn.com/file/ss1edaf7/OMG%20PUPPY | 17:18 |
| kmalloc | For your puppy consumption needs. | 17:18 |
| hrybacki | The video call was so that someone can drive a specific conversation e.g. look at this review with me. Like, someone outside of the core team for example could ask for more immediate input and get feedback if they are having troubles with just comments on reviews | 17:19 |
| *** dave-mccowan has quit IRC | 17:19 | |
| cmurphy | they can do that on irc | 17:19 |
| cmurphy | and we can jump on a call if irc isn't cutting it for a given discussion | 17:20 |
| hrybacki | I guess I was shooting for some consistency | 17:20 |
| *** AlexeyAbashkin has quit IRC | 17:21 | |
| hrybacki | but it is what the team wants :) | 17:21 |
| gagehugo | yeah I like the video call for focused conversation (ie reviews,specs) | 17:21 |
| gagehugo | but otherwise irc can probably cover most of what we need | 17:22 |
| cmurphy | please go ahead with it, it's just evening here for me so i'm going to relax and stuff and am available if needed specifically | 17:22 |
| *** spilla has joined #openstack-keystone | 17:29 | |
| lbragstad | jgrassler: updated https://review.openstack.org/#/c/396331/20 to summarize the meeting | 17:31 |
| *** dave-mccowan has joined #openstack-keystone | 17:34 | |
| jgrassler | lbragstad: Thanks! | 17:34 |
| lbragstad | jgrassler: no problem | 17:34 |
| jgrassler | lbragstad: I'll update the spec tomorrow morning (gotta dash now) | 17:34 |
| lbragstad | jgrassler: yeah - no worries | 17:34 |
| *** NM has quit IRC | 17:39 | |
| *** tesseract has quit IRC | 17:40 | |
| *** NM has joined #openstack-keystone | 17:43 | |
| lbragstad | hrybacki: reviewed https://review.openstack.org/#/c/523973/ | 17:45 |
| hrybacki | lbragstad: ack, thank you! looking now | 17:45 |
| lbragstad | looks good, just a few suggestions, but I think we can probably take this to some of the other projects and get feedback | 17:45 |
| hrybacki | lbragstad: ack. Need folks to ask a few hard questions so we can flesh it out in a meaningful way rather than just speculating imo | 17:46 |
| *** dikonoo has quit IRC | 17:49 | |
| *** pcaruana has quit IRC | 17:51 | |
| *** spilla has quit IRC | 17:59 | |
| kmalloc | lbragstad: the mysql and pgsql errors are different | 18:00 |
| kmalloc | one is an issue, loosk like with the dataset | 18:00 |
| kmalloc | the other is a deadlock | 18:00 |
| lbragstad | huh | 18:01 |
| lbragstad | for mysql, are you specifically referencing the issue brought up this morning by tmcm? | 18:01 |
| kmalloc | yep | 18:01 |
| kmalloc | that looks to be an issue with the FK constraint cannot be made, there is bad data | 18:01 |
| kmalloc | like project.id reference in a column that doesn't exist in the project table | 18:02 |
| kmalloc | (example) | 18:02 |
| lbragstad | kmalloc: yeah - we found out that the domain being referenced didn't actually exist | 18:02 |
| kmalloc | yep | 18:02 |
| kmalloc | the hard part here is... we don't really test pgsql | 18:02 |
| kmalloc | this might be a pgsql issue | 18:02 |
| lbragstad | there was some discussion about postgres support, but i don't know where that ended up | 18:03 |
| kmalloc | well, let me check the gate, but... i think we aren't testing pgsql meaningfully | 18:03 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Log warning when using token_flush https://review.openstack.org/556889 | 18:03 |
| lbragstad | i thought i remember various TC members being involved there | 18:03 |
| lbragstad | iirc - it was a long discussion and i never kept up with it | 18:04 |
| gyee | lbragstad: https://bugs.launchpad.net/keystone/+bug/1758460 | 18:04 |
| openstack | Launchpad bug 1758460 in OpenStack Identity (keystone) "UUID (or any persistent) token providers unable to validate federation token" [Undecided,New] | 18:04 |
| gyee | tell me with a straight face, how much do we care about UUID provider at this point, even in stable/pike :-) | 18:04 |
| kmalloc | gyee: not | 18:05 |
| kmalloc | gyee: it was deleted in Rocky | 18:05 |
| gyee | heh | 18:05 |
| gyee | gotta ask | 18:05 |
| kmalloc | if it is a major bug, we can fix as long as P isn't EOL | 18:05 |
| kmalloc | and if the bug is in Q, we can address it | 18:05 |
| gyee | P is near EOL | 18:05 |
| lbragstad | gyee: i already pulled that plug | 18:05 |
| cmurphy | lbragstad: it got a resolution https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.html but it was not really a real decision | 18:06 |
| lbragstad | and started rewriting all the interfaces in that part of keystone | 18:06 |
| kmalloc | but frankly, i wouldn't care about the P bug unless it's critical, and if the bug is in Q too | 18:06 |
| kmalloc | since technically we support uuid in both P and Q | 18:06 |
| gyee | that bug's been there since P | 18:06 |
| gyee | only for UUID provider though | 18:06 |
| kmalloc | as a stable core, i'd merge a fix for Q and backport to P | 18:06 |
| kmalloc | but i wouldn't write the code myself. | 18:07 |
| kmalloc | if that helps you out | 18:07 |
| gyee | that's enough info for me to convey back to the decision makers :-) | 18:08 |
| gyee | thanks guys | 18:08 |
| kmalloc | gyee: and i say that as the only keystone-stable-core member :P | 18:08 |
| kmalloc | gyee: my recommendation to the decision makers is "fernet" | 18:08 |
| gyee | kmalloc, agree | 18:09 |
| kmalloc | gyee: if you are writing the fix, propose straight to Q but comment that it cannot be merged to master because uuid has been removed. | 18:09 |
| gyee | kmalloc, nah, I'll push for fernet, no point of touching UUID | 18:09 |
| kmalloc | and once it's good (inc. a test) port to P | 18:09 |
| kmalloc | i figured | 18:09 |
| kmalloc | but just in case you have to ;) | 18:10 |
| kmalloc | and then ping me directly so we can push it through (if you end up needing it) | 18:10 |
| cmurphy | gyee: with my internal hat on, our other product already does fernet so you could copy that implementation :) | 18:10 |
| gyee | cmurphy, most of the stuff are already there, just the rotation bit needs work | 18:11 |
| kmalloc | lbragstad: i'm also going to reference the TC resolution regarding PGSQL there. | 18:12 |
| kmalloc | lbragstad: it's going to take a bit more work to know what and why PG is having the issues. | 18:12 |
| kmalloc | lbragstad: i'm guessing it is an issue with load on the DB and a table lock. | 18:12 |
| kmalloc | lbragstad: possibly a delete operation | 18:12 |
| *** voelzmo has joined #openstack-keystone | 18:18 | |
| lbragstad | gyee: in case you're interested https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:token-provider-refactor | 18:19 |
| lbragstad | i'm not sure if you are maintaining an out-of-tree token provider anywhere | 18:19 |
| *** voelzmo_ has joined #openstack-keystone | 18:20 | |
| lbragstad | kmalloc: you mean reference it in the bug? | 18:20 |
| gyee | lbragstad, no out-of-tree, I got pulled in to reverify the federation stuff recently | 18:20 |
| lbragstad | gyee: oh | 18:20 |
| gyee | but I haven't touch that stuff in awhile | 18:20 |
| kmalloc | lbragstad: yeah | 18:20 |
| lbragstad | kmalloc: ok | 18:20 |
| *** felipemonteiro__ has joined #openstack-keystone | 18:21 | |
| kmalloc | lbragstad: just commented on the code | 18:21 |
| kmalloc | lbragstad: i think we might need zzzeek to help us here. | 18:21 |
| hrybacki | lbragstad: where was that community tag stuff doc'd again? | 18:22 |
| hrybacki | lbragstad: disregard | 18:22 |
| lbragstad | here? https://governance.openstack.org/tc/reference/tags/index.html | 18:22 |
| lbragstad | kmalloc: thanks for digging | 18:22 |
| hrybacki | aye | 18:22 |
| *** voelzmo has quit IRC | 18:22 | |
| hrybacki | lbragstad: update pushed | 18:24 |
| * hrybacki fetches lunch | 18:24 | |
| *** felipemonteiro_ has quit IRC | 18:25 | |
| lbragstad | sweet | 18:26 |
| kmalloc | lbragstad: i downgraded the bug to medium | 18:27 |
| kmalloc | it's PGonly and it's contract | 18:28 |
| lbragstad | kmalloc: ack - so we're waiting on feedback then? | 18:28 |
| kmalloc | zzzeek: if you could help out some, trying to chase down potential deadlocks in a contract phase https://bugs.launchpad.net/keystone/+bug/1755906 | 18:28 |
| openstack | Launchpad bug 1755906 in OpenStack Identity (keystone) "Occasional deadlock during db_sync --contract during Newton to Pike live upgrade" [Medium,Confirmed] | 18:28 |
| kmalloc | zzzeek: i just don't see it, and it's happening in PG but not MySQL AFAICT | 18:28 |
| *** AlexeyAbashkin has joined #openstack-keystone | 18:35 | |
| zzzeek | kmalloc: we're spuporting postgresql again? | 18:35 |
| kmalloc | zzzeek: no, just a best effort, feel free to say "not my problem/can't help" | 18:36 |
| zzzeek | kmalloc: postgresql is very locky at the DDL level | 18:36 |
| kmalloc | just said I would ask -- mostly to be sure we aren't doing something dumb that could bite us in MySQL as well | 18:36 |
| kmalloc | zzzeek: yeah PG is very locky for integrity reasons(tm) | 18:36 |
| zzzeek | kmalloc: so, what did you have in mind here? | 18:37 |
| kmalloc | if you could look at the migration and give a "yeah no clear issues that would impact, this is an edge case" | 18:38 |
| kmalloc | that is good for me | 18:38 |
| zzzeek | kmalloc: are the deadlocks against the normal app server running SELECT statements? | 18:38 |
| kmalloc | the deadlock is happening in a contract phase (new FK constraint) while selects are happening | 18:38 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Make tags filter match subset rather than exact https://review.openstack.org/553108 | 18:39 |
| gagehugo | lbragstad added a releasenote | 18:39 |
| *** AlexeyAbashkin has quit IRC | 18:39 | |
| kmalloc | this is the no-downtime(limited downtime) upgrade thing | 18:39 |
| kmalloc | zzzeek: this is the migrtation in question https://github.com/openstack/keystone/blob/master/keystone/common/sql/contract_repo/versions/014_contract_add_domain_id_to_user_table.py | 18:40 |
| zzzeek | kmalloc: this is adding a foreign key in the contract, huh | 18:41 |
| kmalloc | because we can't add the forign key in expand... lets just say the no-downtime thing has been headaches | 18:42 |
| kmalloc | and pivoting the whole table is also... very detrimental | 18:42 |
| zzzeek | kmalloc: i dont see a quick win on this thered' ahve to be some hey make sure the app server isn't running while the migration happens thing, e.g. more locks | 18:42 |
| kmalloc | zzzeek: and for all i know the triggers is causing issues | 18:43 |
| zzzeek | kmalloc: online schema migrations for PG seems like a non-starter given the status of PG | 18:43 |
| kmalloc | good to know | 18:43 |
| kmalloc | i'll respond with that and reference this convo | 18:43 |
| zzzeek | kmalloc: yo're doing the triggers w/ PG as well? | 18:43 |
| kmalloc | sigh | 18:43 |
| *** voelzmo_ has quit IRC | 18:43 | |
| kmalloc | against all my protests | 18:43 |
| zzzeek | kmalloc: oh I was arguing in *favor* of the triggers :) | 18:43 |
| kmalloc | i greatly dislike them. | 18:43 |
| kmalloc | they're so very hard to debug. | 18:44 |
| kmalloc | esp. when the app can do all the logic. | 18:44 |
| zzzeek | kmalloc: im not saying this bug cant be fixed but it would require thinking, looking, coding, and testing and i dont know we have resources for that for PG | 18:44 |
| kmalloc | yeah thats fine | 18:44 |
| kmalloc | i'll reference this as well confirming my statement, it's just not something we have resources for | 18:44 |
| zzzeek | yes | 18:44 |
| kmalloc | but we will happily accept external help | 18:44 |
| kmalloc | and if they have a fix, we'll evaluate it | 18:45 |
| kmalloc | and include it if we can. | 18:45 |
| kmalloc | lbragstad: ^, marked the bug as incomplete | 18:49 |
| lbragstad | kmalloc: zzzeek thanks | 18:49 |
| kmalloc | lbragstad: if they can supply help, we'll accept it | 18:49 |
| lbragstad | gagehugo: works for me locally | 18:49 |
| kmalloc | lbragstad: otherwise, we should update our documentation, live-schema changes only supported/tested under MySQL, PGSQL is recommended that live-schema changes not be performed (downtime-only) | 18:50 |
| kmalloc | lbragstad: acutally,w e sould update docs for that anyway | 18:50 |
| lbragstad | right | 18:50 |
| lbragstad | kmalloc: when should we remove keystone-manage token_flush? | 18:52 |
| lbragstad | Solar? | 18:52 |
| kmalloc | lbragstad: sure. | 18:53 |
| *** ykarel|away has quit IRC | 18:54 | |
| *** germs_ has quit IRC | 19:00 | |
| *** voelzmo has joined #openstack-keystone | 19:03 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Log warning when using token_flush https://review.openstack.org/556889 | 19:04 |
| *** thomasduval has joined #openstack-keystone | 19:09 | |
| *** thomasduval has left #openstack-keystone | 19:10 | |
| hrybacki | lbragstad: cmurphy perhaps we could conduct an audit of our API vs scope levels | 19:14 |
| *** NM has quit IRC | 19:15 | |
| *** pcichy has quit IRC | 19:15 | |
| hrybacki | for example, I assumed user related actions would be domain-scoped as opposed to system-scoped. BUT if user/group actions are system-scoped that would resolve our earlier issue lbragstad | 19:16 |
| *** NM has joined #openstack-keystone | 19:17 | |
| *** voelzmo has quit IRC | 19:19 | |
| *** voelzmo has joined #openstack-keystone | 19:19 | |
| *** voelzmo_ has joined #openstack-keystone | 19:19 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Removal of deprecated direct driver loading https://review.openstack.org/350815 | 19:20 |
| *** voelzmo_ has quit IRC | 19:20 | |
| *** voelzmo has quit IRC | 19:20 | |
| lbragstad | hrybacki: i think some of that would audit would have been taken care of when we implement scope_types | 19:23 |
| hrybacki | lbragstad: ack. I'm just thinking doing part of it now would make sure we have a sane, consistent messgae in the spec | 19:24 |
| hrybacki | e.g. are user operations domain or system level discussion | 19:24 |
| lbragstad | sure | 19:24 |
| lbragstad | we could put a disclaimer for that specific case in the spec | 19:26 |
| *** itlinux has joined #openstack-keystone | 19:26 | |
| *** ilush has joined #openstack-keystone | 19:29 | |
| *** harlowja has joined #openstack-keystone | 19:34 | |
| *** mvk has joined #openstack-keystone | 19:34 | |
| *** ilush has quit IRC | 19:35 | |
| *** NM has quit IRC | 19:40 | |
| *** germs has joined #openstack-keystone | 19:55 | |
| *** germs has quit IRC | 19:55 | |
| *** germs has joined #openstack-keystone | 19:55 | |
| *** NM has joined #openstack-keystone | 19:57 | |
| *** edmondsw has quit IRC | 19:58 | |
| *** edmondsw has joined #openstack-keystone | 19:59 | |
| *** edmondsw_ has joined #openstack-keystone | 20:01 | |
| *** germs has quit IRC | 20:02 | |
| *** germs has joined #openstack-keystone | 20:02 | |
| *** germs has quit IRC | 20:02 | |
| *** germs has joined #openstack-keystone | 20:02 | |
| *** edmondsw has quit IRC | 20:04 | |
| *** edmondsw_ has quit IRC | 20:04 | |
| *** edmondsw has joined #openstack-keystone | 20:04 | |
| *** edmondsw has quit IRC | 20:04 | |
| *** tmcm has quit IRC | 20:17 | |
| *** germs has quit IRC | 20:30 | |
| gagehugo | is Solar the official name? | 20:31 |
| cmurphy | not yet | 20:32 |
| lbragstad | oh - shoot, i should probably wip my review then | 20:34 |
| *** aojea has joined #openstack-keystone | 20:34 | |
| gagehugo | I liked stein | 20:37 |
| *** ilush has joined #openstack-keystone | 20:53 | |
| *** edmondsw has joined #openstack-keystone | 20:59 | |
| *** edmondsw_ has joined #openstack-keystone | 21:00 | |
| *** dave-mccowan has quit IRC | 21:01 | |
| *** ilush has quit IRC | 21:02 | |
| *** edmondsw has quit IRC | 21:04 | |
| *** edmondsw_ has quit IRC | 21:05 | |
| *** raildo has quit IRC | 21:10 | |
| *** itlinux has quit IRC | 21:15 | |
| lbragstad | knikolla: are you happy with response here https://review.openstack.org/#/c/556022/ ? | 21:18 |
| *** itlinux has joined #openstack-keystone | 21:20 | |
| adriant | lbragstad: want me to update the auth receipt spec and move that down? | 21:23 |
| adriant | or would you prefer doing it as another patch? | 21:23 |
| lbragstad | adriant: oh - we can do that in another patch set... we talked about that in today's meeting and i said i was going to propose a follow on | 21:24 |
| lbragstad | but i haven't gotten to it yet | 21:24 |
| adriant | cool :) | 21:24 |
| adriant | sorry I wasn't able to attend | 21:24 |
| cmurphy | i think kmalloc was also hoping for a final look from ayoung | 21:25 |
| *** r-daneel has quit IRC | 21:25 | |
| lbragstad | adriant: no worries - it's not the best time for APAC | 21:26 |
| adriant | As I start making progress on the implementation WIP I'll come by for office hours and ask silly questions! | 21:26 |
| *** felipemonteiro__ has quit IRC | 21:28 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone-specs master: Log queens specifications with previous releases https://review.openstack.org/557060 | 21:28 |
| *** NM has quit IRC | 21:32 | |
| *** edmondsw has joined #openstack-keystone | 21:35 | |
| openstackgerrit | Merged openstack/keystone master: Make tags filter match subset rather than exact https://review.openstack.org/553108 | 21:35 |
| *** edmondsw has quit IRC | 21:36 | |
| *** oikiki has joined #openstack-keystone | 21:42 | |
| *** martinus__ has quit IRC | 21:45 | |
| kmalloc | cmurphy: only because ayound was commenting on it | 21:45 |
| kmalloc | cmurphy: but i'm fine with it +A now | 21:46 |
| kmalloc | lbragstad, adriant: ^ | 21:46 |
| openstackgerrit | Merged openstack/keystone master: Fix integer -> method conversion for python3 https://review.openstack.org/555339 | 21:46 |
| *** felipemonteiro has joined #openstack-keystone | 21:47 | |
| knikolla | lbragstad: pushed it :) | 21:51 |
| openstackgerrit | Merged openstack/keystone-specs master: Add spec for MFA auth receipts https://review.openstack.org/553670 | 21:52 |
| adriant | woo! | 21:53 |
| adriant | now I have to actually implement it. Which shouldn't be so bad | 21:54 |
| *** anyone is now known as eschwartz | 21:58 | |
| *** itlinux has quit IRC | 22:05 | |
| *** mchlumsky has quit IRC | 22:12 | |
| *** lbragstad has quit IRC | 22:15 | |
| *** rcernin has joined #openstack-keystone | 22:16 | |
| *** aojea has quit IRC | 22:24 | |
| *** germs has joined #openstack-keystone | 22:31 | |
| *** germs has quit IRC | 22:31 | |
| *** germs has joined #openstack-keystone | 22:31 | |
| *** lbragstad has joined #openstack-keystone | 22:32 | |
| *** ChanServ sets mode: +o lbragstad | 22:32 | |
| *** lbragstad has quit IRC | 22:32 | |
| *** germs has quit IRC | 22:36 | |
| *** felipemonteiro has quit IRC | 22:38 | |
| *** tmcm has joined #openstack-keystone | 22:56 | |
| *** felipemonteiro has joined #openstack-keystone | 23:07 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 23:12 | |
| *** AlexeyAbashkin has quit IRC | 23:17 | |
| *** felipemonteiro has quit IRC | 23:25 | |
| *** harlowja has quit IRC | 23:32 | |
| *** DevX has quit IRC | 23:33 | |
| *** germs has joined #openstack-keystone | 23:35 | |
| *** germs has quit IRC | 23:35 | |
| *** germs has joined #openstack-keystone | 23:35 | |
| *** germs has quit IRC | 23:36 | |
| *** germs has joined #openstack-keystone | 23:36 | |
| *** germs has quit IRC | 23:36 | |
| *** germs has joined #openstack-keystone | 23:36 | |
| *** germs has quit IRC | 23:58 | |
| *** germs has joined #openstack-keystone | 23:59 | |
| *** germs has quit IRC | 23:59 | |
| *** germs has joined #openstack-keystone | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!