| *** xingchao has joined #puppet-openstack | 00:02 | |
| *** xingchao has quit IRC | 00:07 | |
| *** markvoelker has joined #puppet-openstack | 00:32 | |
| *** markvoelker has quit IRC | 00:36 | |
| ayoung | PUT https://[::1]:35357/v3/projects/1494113d1d984061be09bd9dab38beb8/users/ae93abf775e24d06b15738ba4b094a3f/roles/a8ddbba58b36472dbe2c0b701933c74d is doing a role assignment for a user on a project, and it looks like a duplicate | 00:45 |
|---|---|---|
| ayoung | Not a bug in Keystone, except maybe if that is how it reports the error | 00:45 |
| *** ducttape_ has joined #puppet-openstack | 00:46 | |
| ayoung | INSERT INTO role (id, name, extra) VALUES (%s, %s, %s)'] | 00:47 |
| ayoung | that looks wrong | 00:47 |
| ayoung | I love that we are goind IPv6 though | 00:48 |
| ayoung | It looks like it is tripping over the old code we had to set the default Role...hhmm | 00:49 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/controllers.py#n401 | 00:51 |
| ayoung | mwhahaha, EmilienM ^^ | 00:51 |
| *** ducttape_ has quit IRC | 00:51 | |
| ayoung | It looks like something is trying to create a "Member" role | 00:51 |
| ayoung | and, since it is done as a side effect...it looks wrong | 00:52 |
| mwhahaha | So I wonder if we have a user role create that's doing it before tempest | 00:54 |
| mwhahaha | Or should tempest even be doing that | 00:54 |
| ayoung | mwhahaha, what test is that anyway? | 00:55 |
| mwhahaha | The setup for nova v2 flavors | 00:55 |
| mwhahaha | Is what is failing | 00:55 |
| mwhahaha | Like when it creates the credentials | 00:55 |
| mwhahaha | Is what I tracked down | 00:55 |
| ayoung | mwhahaha, so, it looks like an error in Keystone, and I wonder if it is due to the domain specific role code, but we've only now started seeing it? | 00:56 |
| mwhahaha | So I'm wondering if we are creating a similar role when we run the scenario | 00:56 |
| mwhahaha | I don't think we should allow a null in that column if it's going to result in a 409 | 00:57 |
| ayoung | OK...lets styart from here | 00:57 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/routers.py#n114 | 00:58 |
| ayoung | its a put so | 00:58 |
| ayoung | grant_controller, create_grant | 00:58 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/controllers.py#n625 | 00:59 |
| mwhahaha | I'll get to my computer in a bit but like I said I wonder if we're inadvertently creating the same user/role that tempest tries to use | 00:59 |
| ayoung | mwhahaha, I think it is more subtle | 00:59 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/controllers.py#n631 | 00:59 |
| ayoung | self.assignment_api.create_grant( | 01:00 |
| mwhahaha | But it seems like a bad thing to allow a null in a constraint like that | 01:00 |
| mwhahaha | It's the domain that's getting dropped right? | 01:00 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/core.py#n339 | 01:00 |
| ayoung | not sure | 01:01 |
| ayoung | a FULL stack trace would help | 01:01 |
| ayoung | keystone.common.sql.core is pretty far down the chain | 01:01 |
| ayoung | [SQL: u'INSERT INTO role (id, name, extra) VALUES (%s, %s, %s)'] [parameters: ('a312190b7554410ebc815289f7390e8f', u'Member', '{}')] | 01:01 |
| ayoung | looks like it is trying to create the GLocal Member role | 01:02 |
| ayoung | and it should not be doing that | 01:02 |
| ayoung | so first it checks that the role exists self.role_api.get_role(role_id) | 01:02 |
| ayoung | if domain_id: | 01:02 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/core.py#n1163 | 01:03 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/role_backends/sql.py#n70 | 01:03 |
| *** xingchao has joined #puppet-openstack | 01:03 | |
| ayoung | OK that is straight ahead | 01:03 |
| ayoung | just a fetch, no side effects | 01:04 |
| ayoung | skip the call to get_domain. does a d get_project self.resource_api.get_project(project_id) | 01:05 |
| ayoung | self.driver.create_grant(role_id, user_id, group_id, domain_id, | 01:05 |
| ayoung | project_id, inherited_to_projects) | 01:05 |
| mwhahaha | so it looks like perhaps it's old code? | 01:05 |
| mwhahaha | `domain_id` varchar(64) NOT NULL DEFAULT '<<null>>' is the last column in role | 01:05 |
| mwhahaha | so if it's not trying to add it, it'll get the <<null>> | 01:05 |
| mwhahaha | so is it like a v2 create against a v3 table or something | 01:06 |
| * mwhahaha is just guessing | 01:06 | |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/backends/sql.py#n51 | 01:06 |
| ayoung | inserts into the table | 01:06 |
| ayoung | mwhahaha, red herring | 01:06 |
| ayoung | domain is a new addition to the role table | 01:06 |
| ayoung | before that, all roles were global | 01:06 |
| ayoung | so it should be null 99.99% of the time | 01:07 |
| mwhahaha | delicious delicious herrings | 01:07 |
| ayoung | until people start doing domain specific roles...not there yet | 01:07 |
| mwhahaha | let me go track down the tempest create i had it earlier today | 01:07 |
| ayoung | the second call does not seem to line up with the first | 01:07 |
| ayoung | mwhahaha, is 22699 and 22700 the thread/process ID? | 01:08 |
| *** xingchao has quit IRC | 01:08 | |
| mwhahaha | yea i think so | 01:08 |
| ayoung | 2016-06-25 14:17:14.731 22700 INFO keystone.common.wsgi [req-6654dff5-e5e3-4bc1-ab8e-97066f64ba0f e50189d010664d2bb5e2d1ca9471605e a12c0da88dc542c8abdf55f77998b6f1 - default default] GET https://[::1]:35357/v3/roles | 01:08 |
| ayoung | that is the API being executed, not the line above then | 01:09 |
| ayoung | OK.lets take a look at that | 01:09 |
| ayoung | mwhahaha, feels like a race condition | 01:09 |
| mwhahaha | probably | 01:09 |
| mwhahaha | cause it is sporatic | 01:09 |
| mwhahaha | i noticed it on friday | 01:09 |
| mwhahaha | and recheck will pass sometimes | 01:09 |
| ayoung | V3 roles... | 01:10 |
| ayoung | Standard crud call, | 01:10 |
| mwhahaha | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/test.py#n351 is the function it's erroring in | 01:10 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/controllers.py#n125 | 01:11 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/core.py#n1175 | 01:11 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/role_backends/sql.py#n38 | 01:12 |
| ayoung | very straight ahead | 01:12 |
| ayoung | mwhahaha, so we had logic in the v2 api add_user_to_project that created the member role if it did not exist | 01:15 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/assignment/core.py#n194 | 01:16 |
| ayoung | it feels like that logic, but that should be v2 only and this is a v3 lsit_roles call, should not trigger that | 01:16 |
| ayoung | mwhahaha, the fact that it is the "Member" role leads me to think something got it out of the Config file, like this code does | 01:17 |
| mwhahaha | http://logs.openstack.org/05/332805/3/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/6718b50/console.html#_2016-06-25_14_28_57_994203 | 01:19 |
| mwhahaha | comes from tempest | 01:19 |
| ayoung | self.creds_client.create_user_role('Member') is wrong as a fish in beer | 01:19 |
| ayoung | well, maybe if you were battering and cooking it | 01:19 |
| ayoung | but certainly not swiming in my beer | 01:19 |
| ayoung | mwhahaha, But I think some of the logging has been dropped. | 01:20 |
| mwhahaha | wonder how to add more | 01:20 |
| ayoung | mwhahaha, where is tempest/common/dynamic_creds.py _create_creds | 01:21 |
| ayoung | http://git.openstack.org/cgit/openstack/tempest/tree/tempest//common/dynamic_creds.py#n112 | 01:21 |
| ayoung | mwhahaha, http://git.openstack.org/cgit/openstack/tempest/tree/tempest//common/dynamic_creds.py#n162 throw a try block around that | 01:21 |
| ayoung | the role is almost always created | 01:22 |
| mwhahaha | so bug in tempest? | 01:22 |
| ayoung | mwhahaha, looks like it | 01:22 |
| ayoung | mwhahaha, my guess is this | 01:22 |
| ayoung | two threads running at the same time. both do get_roles, see the role is missing, one inserts, the other inserts and errors | 01:23 |
| ayoung | although it does not seem to be doing a "get_roles" call first | 01:23 |
| mwhahaha | that's old code | 01:23 |
| mwhahaha | from 2015 | 01:23 |
| ayoung | Maybe the keystoneclient code is less forgiving? | 01:24 |
| mwhahaha | https://review.openstack.org/#/c/201816/ | 01:24 |
| mwhahaha | yea old code :D | 01:24 |
| mwhahaha | also year old | 01:24 |
| ayoung | mwhahaha, so look at Keystone client. My guess is it used to let a user create a role that already exists, and now it doesn't | 01:25 |
| ayoung | might even be Keystone server that used to allow that | 01:25 |
| ayoung | might have been in error...the constraint might have been put on for domain specific roles and is now catching the duplicate calls, where as before you were getting away with \putting crap in the database | 01:26 |
| ayoung | I blame you, personally./ | 01:26 |
| ayoung | the error is unfriendly, though | 01:26 |
| mwhahaha | :D | 01:27 |
| mwhahaha | just wonder how this started to be a random problem like this week | 01:27 |
| mwhahaha | i'll just blame mtreinish, i think that's what EmilienM usually does | 01:28 |
| * mwhahaha ducks | 01:28 | |
| EmilienM | it's not safe to ping me a saturday night | 01:29 |
| EmilienM | my sobriety is really unstable | 01:30 |
| mwhahaha | victory | 01:30 |
| mwhahaha | :D | 01:31 |
| EmilienM | https://dashboards.rdoproject.org/rdo-dev | 01:31 |
| mwhahaha | anyway tempest doesn't use keystone client | 01:31 |
| EmilienM | this dashboard is super cool | 01:31 |
| EmilienM | it tells us how far puppet CI is from openstack trunk | 01:31 |
| EmilienM | (1 day now) | 01:31 |
| EmilienM | mwhahaha: is it happenning, like all the time? | 01:31 |
| EmilienM | did something merged in tempest ? | 01:32 |
| mwhahaha | i noticed it on friday | 01:32 |
| mwhahaha | ayoung, http://git.openstack.org/cgit/openstack/tempest/tree/tempest/services/identity/v3/json/roles_client.py#n29 | 01:32 |
| ayoung | must be the server then | 01:32 |
| ayoung | I think you were lucky befopre | 01:32 |
| *** markvoelker has joined #puppet-openstack | 01:32 | |
| ayoung | UI think you were creating slews of Member roles and that is just scary | 01:32 |
| EmilienM | ayoung: "you" | 01:33 |
| mwhahaha | https://review.openstack.org/#/c/332805/ we rechecked a bunch of times | 01:33 |
| mwhahaha | and each time the tests switch on which one is failing | 01:33 |
| EmilienM | ayoung: we're deploying keystone & tempest from trunk | 01:33 |
| ayoung | EmilienM, You in the case meaning tempest | 01:33 |
| mwhahaha | sometimes scenario 1 sometimes 3 | 01:33 |
| EmilienM | so devstack might be affected no? | 01:33 |
| ayoung | should not | 01:33 |
| EmilienM | WHY? | 01:33 |
| ayoung | devstack creates the role exactly once | 01:33 |
| EmilienM | oops sorry | 01:33 |
| EmilienM | I told you I have beers in hand | 01:33 |
| EmilienM | ayoung: don't we ? | 01:33 |
| *** ntpttr- has joined #puppet-openstack | 01:33 | |
| ayoung | I don't see an evidence of code that allowed a role create to succeed if the role already existed | 01:33 |
| ayoung | that leads me to think that there was no uniqueness constraint on role name. I could go and check, but I am lazy | 01:34 |
| ayoung | EmilienM, watch out for the fish | 01:34 |
| EmilienM | I have to google it man | 01:35 |
| EmilienM | anyway | 01:35 |
| mwhahaha | anyway we can continue this on monday :) | 01:36 |
| mwhahaha | sorry to disturb teh drinking | 01:36 |
| EmilienM | ayoung: if you have time on Monday, we can investigate what our CI does that devstack is not doing | 01:36 |
| EmilienM | and figure if there is a bug somewhere | 01:36 |
| EmilienM | now I'll ban you from IRC to make you're enjoying the week end | 01:36 |
| EmilienM | damn, I don't know how to ban well, please leave | 01:36 |
| *** markvoelker has quit IRC | 01:37 | |
| EmilienM | it also reminds me we pin tempest for ubuntu nodes | 01:37 |
| ayoung | EmilienM, I'm avoiding putting kids to bed | 01:37 |
| ayoung | this is easier, believe me | 01:37 |
| ayoung | Putting kids to bed is NP Complete | 01:37 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge https://review.openstack.org/333506 | 01:38 |
| ayoung | EmilienM, file a bug against Tempest | 01:38 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge https://review.openstack.org/333507 | 01:38 |
| ayoung | its is their mistake | 01:38 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge https://review.openstack.org/333508 | 01:38 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge https://review.openstack.org/333509 | 01:38 |
| *** xingchao has joined #puppet-openstack | 01:38 | |
| EmilienM | right now I don't even know what to file | 01:38 |
| ayoung | EmilienM, http://git.openstack.org/cgit/openstack/tempest/tree/tempest//common/dynamic_creds.py#n162 throw a try block around that | 01:38 |
| ayoung | and now to go play dad | 01:38 |
| EmilienM | we'll do it on Monday with Alex | 01:39 |
| EmilienM | enjoy weekend folks /me afk | 01:39 |
| mwhahaha | https://bugs.launchpad.net/tempest/+bug/1596250 | 01:41 |
| openstack | Launchpad bug 1596250 in tempest "setup_credentials failure in FlavorsV2TestJSON" [Undecided,New] | 01:41 |
| ayoung | EmilienM, https://review.openstack.org/334209 | 01:42 |
| mwhahaha | going to throw a block around that ;0 | 01:42 |
| ayoung | mwhahaha, add to that review, please? | 01:42 |
| mwhahaha | will do | 01:42 |
| ayoung | and probablty should be a more specific exception | 01:42 |
| mwhahaha | k updated | 01:45 |
| *** ilbot3 has quit IRC | 01:46 | |
| mwhahaha | we can deal with it on monday, thanks ayoung | 01:46 |
| *** ducttape_ has joined #puppet-openstack | 01:47 | |
| *** ilbot3 has joined #puppet-openstack | 01:47 | |
| *** ducttape_ has quit IRC | 01:52 | |
| *** xingchao has quit IRC | 02:15 | |
| *** xingchao has joined #puppet-openstack | 02:16 | |
| *** xingchao has quit IRC | 02:21 | |
| *** xingchao has joined #puppet-openstack | 02:24 | |
| *** ducttape_ has joined #puppet-openstack | 03:49 | |
| *** ducttape_ has quit IRC | 03:54 | |
| *** xingchao has quit IRC | 03:55 | |
| *** moravec has quit IRC | 04:52 | |
| *** moravec has joined #puppet-openstack | 04:53 | |
| *** xingchao has joined #puppet-openstack | 05:21 | |
| *** xingchao has quit IRC | 05:50 | |
| *** xingchao has joined #puppet-openstack | 05:50 | |
| *** xingchao has quit IRC | 07:07 | |
| *** ilbot3 has quit IRC | 07:35 | |
| *** ilbot3 has joined #puppet-openstack | 07:36 | |
| *** gregfaust has quit IRC | 07:37 | |
| *** jmikolajczak has joined #puppet-openstack | 07:40 | |
| *** ducttape_ has joined #puppet-openstack | 07:52 | |
| *** jmikolajczak has quit IRC | 07:52 | |
| *** ducttape_ has quit IRC | 07:56 | |
| *** jmikolajczak has joined #puppet-openstack | 08:28 | |
| *** markvoelker has joined #puppet-openstack | 08:37 | |
| *** xingchao has joined #puppet-openstack | 08:38 | |
| *** yolanda has quit IRC | 08:39 | |
| *** markvoelker has quit IRC | 08:42 | |
| *** jmikolajczak has quit IRC | 08:49 | |
| *** ducttape_ has joined #puppet-openstack | 08:53 | |
| *** ducttape_ has quit IRC | 08:57 | |
| *** xingchao has quit IRC | 08:59 | |
| *** markvoelker has joined #puppet-openstack | 09:38 | |
| *** markvoelker has quit IRC | 09:43 | |
| *** ducttape_ has joined #puppet-openstack | 09:54 | |
| *** xingchao has joined #puppet-openstack | 09:57 | |
| *** ducttape_ has quit IRC | 09:58 | |
| *** xingchao has quit IRC | 10:32 | |
| *** markvoelker has joined #puppet-openstack | 10:39 | |
| *** markvoelker has quit IRC | 10:44 | |
| *** crinkle_ has quit IRC | 10:44 | |
| *** crinkle_ has joined #puppet-openstack | 10:44 | |
| *** aderyugin has quit IRC | 10:45 | |
| *** alvladeru has joined #puppet-openstack | 10:46 | |
| *** alvladeru is now known as aderyugin | 10:46 | |
| *** ducttape_ has joined #puppet-openstack | 10:55 | |
| *** ducttape_ has quit IRC | 10:59 | |
| *** xingchao has joined #puppet-openstack | 10:59 | |
| *** xingchao has quit IRC | 11:18 | |
| *** markvoelker has joined #puppet-openstack | 11:40 | |
| *** markvoelker has quit IRC | 11:44 | |
| *** ducttape_ has joined #puppet-openstack | 11:56 | |
| *** ducttape_ has quit IRC | 12:01 | |
| *** markvoelker has joined #puppet-openstack | 12:41 | |
| *** ducttape_ has joined #puppet-openstack | 12:43 | |
| *** markvoelker has quit IRC | 12:45 | |
| *** johnmilton has joined #puppet-openstack | 13:30 | |
| *** ducttape_ has quit IRC | 13:31 | |
| *** xingchao has joined #puppet-openstack | 13:39 | |
| *** markvoelker has joined #puppet-openstack | 13:41 | |
| *** xingchao has quit IRC | 13:44 | |
| *** markvoelker has quit IRC | 13:46 | |
| *** ducttape_ has joined #puppet-openstack | 14:24 | |
| *** xingchao has joined #puppet-openstack | 14:40 | |
| *** ducttape_ has quit IRC | 14:40 | |
| *** markvoelker has joined #puppet-openstack | 14:42 | |
| *** xingchao has quit IRC | 14:45 | |
| *** markvoelker has quit IRC | 14:47 | |
| *** ducttape_ has joined #puppet-openstack | 15:05 | |
| *** ducttape_ has quit IRC | 15:16 | |
| *** ducttape_ has joined #puppet-openstack | 15:31 | |
| *** _stowa has joined #puppet-openstack | 15:52 | |
| *** _stowa has quit IRC | 16:00 | |
| *** ducttape_ has quit IRC | 16:18 | |
| *** v1k0d3n has joined #puppet-openstack | 16:18 | |
| *** v1k0d3n has quit IRC | 16:24 | |
| *** yolanda has joined #puppet-openstack | 16:39 | |
| *** markvoelker has joined #puppet-openstack | 16:44 | |
| *** markvoelker has quit IRC | 16:48 | |
| *** ducttape_ has joined #puppet-openstack | 17:18 | |
| *** ducttape_ has quit IRC | 17:23 | |
| *** xingchao has joined #puppet-openstack | 17:42 | |
| *** markvoelker has joined #puppet-openstack | 17:44 | |
| *** xingchao has quit IRC | 17:47 | |
| *** markvoelker has quit IRC | 17:51 | |
| *** v1k0d3n has joined #puppet-openstack | 18:00 | |
| *** v1k0d3n has quit IRC | 18:05 | |
| *** imcsk8 has quit IRC | 18:13 | |
| *** v1k0d3n has joined #puppet-openstack | 18:14 | |
| *** imcsk8 has joined #puppet-openstack | 18:15 | |
| *** ducttape_ has joined #puppet-openstack | 18:19 | |
| *** ducttape_ has quit IRC | 18:24 | |
| *** ducttape_ has joined #puppet-openstack | 18:25 | |
| *** vinsh has joined #puppet-openstack | 18:31 | |
| *** xingchao has joined #puppet-openstack | 18:43 | |
| *** ducttape_ has quit IRC | 18:44 | |
| *** markvoelker has joined #puppet-openstack | 18:47 | |
| *** xingchao has quit IRC | 18:48 | |
| *** ducttape_ has joined #puppet-openstack | 18:49 | |
| *** ducttape_ has quit IRC | 18:50 | |
| *** markvoelker has quit IRC | 18:51 | |
| *** foxhunt_away is now known as foxhunt72 | 19:09 | |
| *** rcernin has joined #puppet-openstack | 19:34 | |
| *** xingchao has joined #puppet-openstack | 19:44 | |
| *** xingchao has quit IRC | 19:49 | |
| *** xarses has quit IRC | 20:01 | |
| *** ducttape_ has joined #puppet-openstack | 20:06 | |
| *** foxhunt72 is now known as foxhunt_away | 20:36 | |
| *** xingchao has joined #puppet-openstack | 20:45 | |
| *** xingchao has quit IRC | 20:49 | |
| *** v1k0d3n has quit IRC | 21:01 | |
| *** rcernin has quit IRC | 21:22 | |
| *** dh64 has quit IRC | 21:26 | |
| *** ayoung has quit IRC | 21:29 | |
| *** xingchao has joined #puppet-openstack | 21:46 | |
| *** markvoelker has joined #puppet-openstack | 21:49 | |
| *** xingchao has quit IRC | 21:50 | |
| *** markvoelker has quit IRC | 21:54 | |
| *** ducttape_ has quit IRC | 22:12 | |
| *** ducttape_ has joined #puppet-openstack | 23:45 | |
| *** xingchao has joined #puppet-openstack | 23:47 | |
| *** markvoelker has joined #puppet-openstack | 23:50 | |
| *** xingchao has quit IRC | 23:51 | |
| *** markvoelker has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!