Monday, 2017-01-16

*** jistr has quit IRC00:03
*** jistr has joined #openstack-keystone00:07
*** dave-mcc_ has joined #openstack-keystone00:08
*** dave-mccowan has quit IRC00:10
*** dave-mcc_ has quit IRC00:13
*** adrian_otto has joined #openstack-keystone00:30
*** dave-mccowan has joined #openstack-keystone00:41
*** thorst has joined #openstack-keystone00:41
*** thorst has quit IRC00:45
*** thorst has joined #openstack-keystone00:46
*** hoangcx has joined #openstack-keystone00:50
*** dave-mccowan has quit IRC00:51
*** adrian_otto has quit IRC00:52
*** voelzmo has joined #openstack-keystone01:35
*** voelzmo has quit IRC01:40
*** stevemar has quit IRC02:21
*** stevemar has joined #openstack-keystone02:21
*** links has joined #openstack-keystone02:25
*** ChanServ sets mode: +o stevemar02:26
*** links has quit IRC02:38
*** links has joined #openstack-keystone02:46
*** thorst has joined #openstack-keystone02:47
*** liujiong has joined #openstack-keystone02:50
*** thorst has quit IRC02:52
*** dave-mccowan has joined #openstack-keystone03:02
*** dave-mccowan has quit IRC03:06
*** severion has joined #openstack-keystone03:10
*** thorst has joined #openstack-keystone03:14
*** thorst has quit IRC03:14
*** dave-mccowan has joined #openstack-keystone03:31
*** voelzmo has joined #openstack-keystone03:36
*** voelzmo has quit IRC03:41
*** dave-mccowan has quit IRC03:45
*** tqtran has joined #openstack-keystone03:49
*** tqtran has quit IRC03:50
*** phalmos has joined #openstack-keystone03:54
*** phalmos has quit IRC04:13
*** adrian_otto has joined #openstack-keystone04:23
*** adrian_otto has quit IRC04:24
*** adrian_otto has joined #openstack-keystone04:24
*** adrian_otto has quit IRC04:33
*** adrian_otto has joined #openstack-keystone04:44
*** jperry has joined #openstack-keystone04:53
*** adrian_otto has quit IRC05:11
*** thorst has joined #openstack-keystone05:15
*** severion has quit IRC05:18
*** v1k0d3n has quit IRC05:18
*** thorst has quit IRC05:20
*** v1k0d3n has joined #openstack-keystone05:35
*** v1k0d3n has quit IRC05:48
*** v1k0d3n has joined #openstack-keystone05:49
*** Dinesh_Bhor has joined #openstack-keystone06:05
*** adrian_otto has joined #openstack-keystone06:06
*** adrian_otto has quit IRC06:14
*** adrian_otto has joined #openstack-keystone06:16
*** adrian_otto has quit IRC06:16
*** adrian_otto has joined #openstack-keystone06:21
*** voelzmo has joined #openstack-keystone06:27
*** voelzmo has quit IRC06:32
*** adrian_otto has quit IRC06:33
*** richm has quit IRC06:43
*** mnaser has quit IRC06:53
*** afazekas has quit IRC06:53
*** mordred has quit IRC06:53
*** tqtran has joined #openstack-keystone06:54
*** afazekas has joined #openstack-keystone06:57
*** Jack_I has joined #openstack-keystone07:00
*** mordred has joined #openstack-keystone07:02
*** afazekas has quit IRC07:02
*** g2[ATL] is now known as g207:03
*** mnaser has joined #openstack-keystone07:13
*** afazekas has joined #openstack-keystone07:13
*** tqtran has quit IRC07:15
*** thorst has joined #openstack-keystone07:16
*** edtubill has joined #openstack-keystone07:18
*** edtubill has quit IRC07:20
*** voelzmo has joined #openstack-keystone07:21
*** thorst has quit IRC07:21
*** tesseract has joined #openstack-keystone07:21
*** hogepodge_ has joined #openstack-keystone07:24
*** pcaruana has joined #openstack-keystone07:34
*** hogepodge_ has quit IRC07:37
*** liujiong has quit IRC07:52
*** liujiong has joined #openstack-keystone07:52
*** adriant has quit IRC08:03
*** stingaci has joined #openstack-keystone08:07
*** trananhkma has joined #openstack-keystone08:30
*** obre has joined #openstack-keystone08:32
*** zzzeek has quit IRC09:00
*** zzzeek has joined #openstack-keystone09:00
*** thorst has joined #openstack-keystone09:17
*** chrisplo_ has quit IRC09:18
*** chrisplo_ has joined #openstack-keystone09:20
*** thorst has quit IRC09:22
*** rha has joined #openstack-keystone09:33
*** rha has quit IRC09:33
*** rha has joined #openstack-keystone09:33
*** asettle has joined #openstack-keystone09:46
*** asettle is now known as Guest5850609:46
*** mvk has quit IRC09:47
*** Guest58506 has quit IRC09:53
*** liujiong has quit IRC10:04
*** jperry has quit IRC10:11
*** hoangcx has quit IRC10:14
*** mvk has joined #openstack-keystone10:20
*** voelzmo has quit IRC10:43
*** asettle has joined #openstack-keystone10:53
*** asettle has quit IRC10:54
*** asettle has joined #openstack-keystone10:59
*** asettle__ has joined #openstack-keystone11:01
*** asettle has quit IRC11:01
*** asettle__ has quit IRC11:10
*** asettle has joined #openstack-keystone11:11
*** asettle is now known as Guest7429711:11
*** richm has joined #openstack-keystone11:13
*** erlon_pto has joined #openstack-keystone11:15
*** Guest74297 has quit IRC11:17
*** thorst has joined #openstack-keystone11:18
*** thorst has quit IRC11:23
*** voelzmo has joined #openstack-keystone11:41
*** asettle_ has joined #openstack-keystone11:50
*** asettle_ is now known as asettle11:52
*** voelzmo has quit IRC11:55
*** voelzmo has joined #openstack-keystone11:56
*** dikonoor has joined #openstack-keystone12:13
*** thiagolib has joined #openstack-keystone12:19
*** dikonoor has quit IRC12:34
*** dikonoor has joined #openstack-keystone12:34
*** nicolasbock has joined #openstack-keystone12:34
*** jperry has joined #openstack-keystone12:41
*** thorst has joined #openstack-keystone12:45
*** raildo has joined #openstack-keystone12:58
*** jperry has quit IRC13:10
*** erlon_pto is now known as erlon13:33
bretonstevemar: :(13:44
stevemarbreton: meh13:45
stevemarbreton: at least its ~10 projects13:46
stevemarbreton: not 13013:46
*** dave-mccowan has joined #openstack-keystone14:14
*** edtubill has joined #openstack-keystone14:22
*** jperry has joined #openstack-keystone14:23
*** links has quit IRC14:30
*** v1k0d3n has quit IRC14:43
*** davechen has quit IRC14:44
*** davechen has joined #openstack-keystone14:45
*** dikonoor has quit IRC14:51
*** mriedem has joined #openstack-keystone15:04
mriedemhas anyone else been seeing these 500 errors in keystone?
stevemareh Lost connection to MySQL server15:05
stevemarmriedem: is it happening with other projects of just keystone?15:06
stevemardstanek: how did the bug smash go?15:09
mriedemlooks like it's just keystone15:09
stevemardstanek: we were dealing with a fire in OSC so i diverted my attention there, but will be back on keystone for the rest of the dev cycle15:09
stevemarmriedem: hmm15:09
*** chris_hultin|AWA is now known as chris_hultin15:11
dstanekstevemar: i think it went well. i was in meetings in the morning so i really couldn't participate15:11
mriedemnot sure if it matters but it's mostly on rax-ord nodes15:11
openstackgerritLance Bragstad proposed openstack/keystone-specs: Update shadow mapping spec
dstanekmriedem: that's really strange15:12
lbragstadsamueldmq ^ modified the spec15:13
*** edmondsw has joined #openstack-keystone15:20
*** jaosorior has joined #openstack-keystone15:25
openstackgerritMerged openstack/keystoneauth: Replace yaml.load() with yaml.safe_load()
*** edmondsw has quit IRC15:30
lbragstadmriedem that logstash query doesn't seem to be rendering for me15:38
lbragstadmriedem how often is this popping up?15:38
mriedem153 in 7 days in check and gate15:39
mriedem97% failure15:39
*** phalmos has joined #openstack-keystone15:39
mriedem79% on rax-ord odes15:39
lbragstadah - so it's not completely exclusive to rax-ord nodes15:39
mriedemwell 106 hits in voting jobs15:39
lbragstadwas the first occurrence 7 days ago?15:40
mriedemlooks like it started happening around 1/1015:41
mriedemwe only store up to 10 days of logs15:41
mriedemso that's under the 10 days15:41
mriedemso could be some performance impacting change made around 1/1015:41
mriedemthis is the bug i created btw
openstackLaunchpad bug 1656850 in OpenStack Identity (keystone) "DBConnectionError while validating tokens in CI runs" [Undecided,New]15:41
mriedemi did see something about listing revocation events in the stacktrace, and was 1/1015:43
mriedembut ^ is just a policy change to make things more restrictive15:43
lbragstadtoken validation should bypass that since I don't think we're using policy in that sense15:43
mriedem "Allow a service user to fetch a token that has expired."15:44
mriedemis that talking about the same change?15:44
mriedemlbragstad: what is the reseller feature?
*** stingaci has quit IRC15:48
lbragstadmriedem reselling is the ability to have HM in a sense that doesn't allow projects above you to inspect projects below you15:48
*** thiagolib has quit IRC15:48
lbragstadi.e. protecting your customers if you're reselling15:48
lbragstada service15:48
*** stingaci has joined #openstack-keystone15:48
mriedemdoes enabling that introduce more flows to the check_token operation?15:49
lbragstad(i think rodrigods worked on that stuff)15:49
lbragstadmriedem that's a good question - I am not sure15:49
*** dave-mccowan has quit IRC15:50
openstackgerritBoris Bobrov proposed openstack/keystone: Handling of 'region' parameter as None
*** mvk has quit IRC15:50
lbragstadit's a long shot - but a new version of pymysql hasn't been released in the last week15:51
lbragstadby looking at the trace - it doesn't even look like the revocation check is even getting to comparing the token against the values returned from the revocation API15:53
*** phalmos has quit IRC15:54
lbragstadlooks like the revocation API blows up trying to retrieve revocation events15:54
lbragstadfwiw - this would have been the change that allowed that for keystone server
*** thiagolib has joined #openstack-keystone15:56
stevemarlbragstad: hmm, we just got the keystonemiddleware bits merged16:00
stevemarhmm, about a month ago16:01
lbragstadyeah - i'm seeing dec 15v16:01
stevemarmriedem: did you guys land something that uses the allow_expired flag?16:02
mriedemstevemar: not that i'm aware of16:04
openstackgerritMerged openstack/keystone-specs: Update shadow mapping spec
mriedemstevemar: we're working through the nova changes to pass a service token with the user token, but none of that is enabled in the gate yet16:04
dstanekstill working on the gate mystery?16:05
*** dave-mccowan has joined #openstack-keystone16:05
lbragstadthis is the last point where keystone passes control to pymysql -
lbragstadbut that stuff was refactored like 3 months ago and we haven't had an issue with it16:05
lbragstadthe only other thing that would made me a little suspicious would be
dstanekwhat's a good example of the failure?16:05
lbragstadbut that is only modifying the token expiration comparison and that is all done before the call to list all revocation events happnes16:06
lbragstad^ that's was I'm working off of16:07
*** dave-mcc_ has joined #openstack-keystone16:08
dstaneki wish the mysql log was captured16:09
dstanekseems like maybe it becomes unresponsive16:09
*** dave-mccowan has quit IRC16:10
lbragstaddstanek right - it look infra related16:11
lbragstadand and I can't imagine we would have merged anything that can do that across keystone/keystonemiddleware/keystoneauth in the last week16:12
lbragstad(but I could be wrong)16:12
mriedemlbragstad: we haven't bumped pymysql in upper-constraints since sept
mriedemso i doubt it's a new release16:13
*** voelzmo has quit IRC16:13
mriedemat one point the mysql logs were captured...16:14
dstanekzzzeek: do you have any thoughts on ?16:15
lbragstadmriedem yeah - i double checked that to make sure there wasn't a new release of pymysql that broke us, but they haven't released anything since last year (and we've already accounted for it in uc)16:15
zzzeekdstanek: that error would indicate the network was cut off to MySQL or MySQL was stopped16:15
*** voelzmo has joined #openstack-keystone16:15
zzzeekdstanek: "connection refused" is pretty unambiguous16:16
lbragstad(that sounds pretty infra related)16:16
*** jose-phillips has joined #openstack-keystone16:16
dstanekzzzeek: there's nothing that we could have done application side to trigger that right?16:16
mriedemdstanek: oom
*** jose-phillips has quit IRC16:16
zzzeekdstanek: the "lost connection" thing can be application side but the next error 20 ms later makes it clear the network to mysql is not avalible16:17
dstanekmriedem: nice16:17
dstanekzzzeek: thanks. i just wanted to double check before i started pointing fingers, but mriedem may have just found the cause16:17
mriedemJan 16 04:12:53 ubuntu-xenial-rax-ord-6682904 kernel: Killed process 16686 (mysqld) total-vm:4707732kB, anon-rss:465436kB, file-rss:13016kB16:17
zzzeekdstanek: the errors show it trying to connect and failing many times just within 4:12:53 this is clear16:18
bretonyey, so kernel did it16:20
*** phalmos has joined #openstack-keystone16:22
dstanekbreton: but it's possible that keystone was the cause if we were blowing up the memory somehow16:23
*** adrian_otto has joined #openstack-keystone16:31
*** kiran-r has joined #openstack-keystone16:32
lbragstadJan 16 04:12:53 ubuntu-xenial-rax-ord-6682904 kernel: Free swap  = 7998540kB16:37
lbragstadJan 16 04:12:53 ubuntu-xenial-rax-ord-6682904 kernel: Total swap = 7999020kB16:37
lbragstadthis looks interesting16:46
lbragstadlvmetad invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=016:46
*** phalmos has quit IRC16:48
lbragstadlooks like that is specific to LVM16:51
lbragstadcc dstanek mriedem ^16:51
*** voelzmo has quit IRC16:52
*** jose-phillips has joined #openstack-keystone16:52
dstaneklbragstad: we need to figure out what process grew in memory between run16:53
*** jose-phi_ has joined #openstack-keystone16:53
lbragstaddstanek wouldn't that be lvmetad?16:54
*** jose-phillips has quit IRC16:54
dstaneklbragstad: maybe... not sure what the message means.16:56
dstaneklbragstad: i guess it could be some sort of disk caching error? i think lvmetad plays a role in that16:57
*** tqtran has joined #openstack-keystone16:57
*** kiran-r has quit IRC17:04
lbragstaddstanek from what I've read, it looks like it reports what causes the kernel to invoke the oom-killer17:05
*** dave-mcc_ has quit IRC17:06
*** phalmos has joined #openstack-keystone17:08
*** diazjf has joined #openstack-keystone17:09
lbragstaddstanek you don't suppose this is cinder related - do you?17:13
dstaneklbragstad: maybe...not really sure17:13
*** stingaci has quit IRC17:19
*** stingaci has joined #openstack-keystone17:23
*** jose-phi_ has quit IRC17:24
openstackgerritOpenStack Proposal Bot proposed openstack/python-keystoneclient: Updated from global requirements
*** stingaci has quit IRC17:28
*** phalmos has quit IRC17:29
*** stingaci has joined #openstack-keystone17:29
*** tesseract has quit IRC17:29
*** jose-phillips has joined #openstack-keystone17:30
*** stingaci has quit IRC17:30
*** stingaci has joined #openstack-keystone17:30
*** stingaci has quit IRC17:35
*** jose-phillips has quit IRC17:35
*** jose-phillips has joined #openstack-keystone17:36
*** jose-phi_ has joined #openstack-keystone17:40
*** phalmos has joined #openstack-keystone17:40
lbragstaddstanek hmm - it looks like systemd-udevd has invoked it a couple times, too17:41
*** jose-phillips has quit IRC17:43
*** adrian_otto1 has joined #openstack-keystone17:47
lbragstadwhatever seems to be invoking the oom-killer seems sporadic17:49
*** adrian_otto has quit IRC17:51
dstaneklbragstad: that would make sense - lvmetad listens for udev events17:52
*** spzala has joined #openstack-keystone17:59
*** clenimar has joined #openstack-keystone18:02
lbragstaddstanek looks like there are a bunch of things that initiate oom-killer though18:05
lbragstad(systemd-udevd, neutron-openvsw, ebtables, kthreadd)18:05
*** mvk has joined #openstack-keystone18:06
*** portdirect is now known as intlabs18:14
lbragstadgrabbing lunch18:15
*** intlabs is now known as portdirect18:17
morganstevemar: ALL the auth plugins (in keystone, the handlers) are supposed to set 'methods' right?18:17
morganstevemar: ... it seems like mapped, external, and a number of others ignore that part of the spec *sigh*18:17
morganin fact, oauth1 does too18:18
morganwow, like we didn't require this  at all =/18:18
morgangoing to just make authenticate deal with that.18:18
bknudsonthe method comes from the config file mapping18:32
bknudsonIIRC we had methods in the plugins originally and then decided we didn't need it anymore.18:32
morganbknudson: right, no this is from cases like rescope18:37
morganbknudson: we at least track the original request auth methods18:37
morganif that is not something we care about...18:37
morganwe can drop it18:37
morganright now only token bothers to set it18:38
morganthis is in the token itself18:38
morgani'm tracking it independently18:38
morganas well18:38
*** dave-mccowan has joined #openstack-keystone18:44
*** diazjf has quit IRC18:51
stevemarmorgan: at least no one has noticed until nowv *shrug*18:58
morganstevemar: we can probably drop that info from the tokens18:59
morgani am certain nothing uses it18:59
*** voelzmo has joined #openstack-keystone19:02
*** diazjf has joined #openstack-keystone19:03
stevemarmorgan: probably19:04
*** voelzmo has quit IRC19:06
stevemarmorgan: probably still needs to be supported in the request ;)19:12
morganthe request wont change19:14
morganjust the body of the token wouldn't have it19:14
*** voelzmo has joined #openstack-keystone19:18
*** voelzmo has quit IRC19:19
rodrigodswe need to add functional tests for the auth plugins19:19
rodrigodsi guess we can do that in the ksc functional tests19:19
openstackgerritLance Bragstad proposed openstack/keystone: Implement federated auto-provisioning
lbragstadknikolla around?19:21
lbragstadknikolla you had a comment on the shadow mapping review that asked if there was any background on why we wanted to associate a single domain to an identity provider -
openstackLaunchpad bug 1642687 in OpenStack Identity (keystone) "Missing domain for federated users" [Medium,In progress] - Assigned to Ron De Rose (ronald-de-rose)19:23
lbragstadrderose ended up working on that and implementing the fix ^19:23
*** avarner has joined #openstack-keystone19:29
*** phalmos has quit IRC19:33
*** avarner has quit IRC19:33
lbragstadstevemar ping19:41
*** phalmos has joined #openstack-keystone19:44
knikollalbragstad: my comment was about restricting the projects to be of the same domain. i agree on keeping the users in that domain.19:52
lbragstadknikolla ah - got it. I thought your comment was about the one-to-one mapping in general19:53
stevemarlbragstad: pong19:55
lbragstadstevemar i'm curious if you had a strong opinion on where federated development environment documentation should live/19:56
*** spilla has joined #openstack-keystone19:56
lbragstadin the federation section, or in the dev docs?19:56
lbragstad(I went with the dev docs)19:56
*** jerrygb has joined #openstack-keystone19:57
stevemarlbragstad: dev docs makes sense to me20:04
*** voelzmo has joined #openstack-keystone20:05
openstackgerritMerged openstack/python-keystoneclient: Updated from global requirements
*** chris_hultin is now known as chris_hultin|AWA20:11
*** jaugustine has joined #openstack-keystone20:12
*** spzala has quit IRC20:14
*** edmondsw has joined #openstack-keystone20:18
* morgan needs a non-enameled dutch oven.20:19
morgandolphm: which one do you have so i can be sure to avoid buying it and having the same issues with food being stuck in the lid.20:19
*** voelzmo has quit IRC20:22
* dolphm just went to go look in the kitchen...20:23
dolphmmorgan: Lodge L10DOL3 (7-quart)20:23
*** edmondsw has quit IRC20:23
morgandolphm: cool I shall not buy that one.20:25
morganfor now, the enameled one is enough20:25
morganbut i want a proper cast iron one w/o the enamel20:25
dolphmmorgan: besides the lid, it is pretty great. i was pondering it the other day and i think the last use i'd reserve for it is probably deep frying though, which i don't do very often20:26
morgandolphm: i'm in need of a new dutch oven so i can dedicate my current one for bread baking20:28
morgandolphm: and i love cooking with proper cast iron and carbon steel20:28
morganso, it's a win-win20:28
*** woodster_ has joined #openstack-keystone20:32
*** jose-phi_ has quit IRC20:33
*** dave-mccowan has quit IRC20:34
*** jose-phillips has joined #openstack-keystone20:36
*** adrian_otto1 has quit IRC20:40
*** spzala has joined #openstack-keystone20:49
openstackgerritMorgan Fainberg proposed openstack/keystone: Add user_mfa_rules table
openstackgerritMorgan Fainberg proposed openstack/keystone: Auth Method Handlers now return a response object always
*** chris_hultin|AWA is now known as chris_hultin20:58
*** jamielennox|away is now known as jamielennox20:59
*** chris_hultin is now known as chris_hultin|AWA21:07
*** jose-phillips has quit IRC21:12
*** Jack_I has quit IRC21:12
*** jose-phillips has joined #openstack-keystone21:13
openstackgerritMerged openstack/keystonemiddleware: Updated from global requirements
gagehugostevemar: exclude_patterns doesn't seem to acknowledge '../../keystone_tempest_plugin' at all21:29
*** dave-mccowan has joined #openstack-keystone21:32
*** adrian_otto has joined #openstack-keystone21:33
*** edmondsw has joined #openstack-keystone21:40
*** edmondsw has quit IRC21:44
stevemargagehugo: oh no?21:45
stevemargagehugo: what about just *keystone_tempest_plugin*21:46
openstackgerritJohn Dennis proposed openstack/keystoneauth: Use comma as separator in ECP Accept HTTP header
*** v1k0d3n has joined #openstack-keystone21:50
jamielennoxjdennis: i feel like we've made that fix before...21:52
*** dave-mccowan has quit IRC21:54
*** thorst has quit IRC21:55
*** spilla has quit IRC22:01
gagehugostevemar: nope :(22:08
*** spzala has quit IRC22:13
*** stingaci has joined #openstack-keystone22:20
*** thorst has joined #openstack-keystone22:24
*** stingaci has quit IRC22:24
*** thorst has quit IRC22:28
*** jerrygb has quit IRC22:34
*** chris_hultin|AWA is now known as chris_hultin22:36
*** jaosorior has quit IRC22:41
*** mriedem has quit IRC22:51
*** jperry has quit IRC23:05
*** medberry is now known as med_23:05
*** jaugustine has quit IRC23:05
*** v1k0d3n has quit IRC23:08
*** edtubill has quit IRC23:09
openstackgerritLance Bragstad proposed openstack/keystone: Implement federated auto-provisioning
*** spzala has joined #openstack-keystone23:14
*** spzala has quit IRC23:18
*** jerrygb has joined #openstack-keystone23:22
*** jose-phillips has quit IRC23:23
*** jerrygb has quit IRC23:26
*** jose-phillips has joined #openstack-keystone23:29
*** chris_hultin is now known as chris_hultin|AWA23:31
*** jerrygb has joined #openstack-keystone23:34
*** spzala has joined #openstack-keystone23:36
*** chris_hultin|AWA is now known as chris_hultin23:39
*** knikolla has quit IRC23:47
*** knikolla has joined #openstack-keystone23:47
*** knikolla has quit IRC23:47
*** knikolla has joined #openstack-keystone23:48
*** phalmos has quit IRC23:48
*** edmondsw has joined #openstack-keystone23:55
*** adrian_otto has quit IRC23:58
*** edmondsw has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!