Monday, 2017-12-18

*** hieulq has quit IRC00:46
*** hieulq has joined #openstack-barbican00:47
*** rm_work has quit IRC01:08
*** rm_work has joined #openstack-barbican01:09
*** daidv has joined #openstack-barbican01:27
*** namnh has joined #openstack-barbican01:27
*** daidv has quit IRC01:33
*** namnh has quit IRC01:36
*** dave-mcc_ has joined #openstack-barbican01:43
*** annp has joined #openstack-barbican01:45
-openstackstatus- NOTICE: The freenode network is currently the target of automated spam attacks, we have enabled temporary restrictions on targetted OpenStack channels which requires users to be logged on to NickServ. If you see spam in your channel, please report it in #openstack-infra. Thanks.01:46
*** dave-mcc_ has quit IRC02:02
*** ChanServ sets mode: +r 02:52
*** annp has quit IRC03:03
*** ssathaye has quit IRC05:55
*** namnh has joined #openstack-barbican06:37
*** namnh has quit IRC06:37
*** namnh has joined #openstack-barbican06:38
*** hieulq has quit IRC06:47
*** hieulq has joined #openstack-barbican06:48
*** alee has quit IRC07:33
openstackgerritNam Nguyen Hoai proposed openstack/barbican master: [WIP] Implement OVO [4]
*** namnh has quit IRC10:52
*** pbourke has quit IRC11:43
*** raildo has joined #openstack-barbican12:21
*** namnh has joined #openstack-barbican13:32
*** namnh has quit IRC14:00
*** namnh has joined #openstack-barbican14:01
*** namnh has quit IRC14:02
*** namnh has joined #openstack-barbican14:02
*** ChanServ sets mode: -r 14:04
*** alee has joined #openstack-barbican14:24
*** abishop has joined #openstack-barbican14:39
*** abishop has quit IRC14:44
*** cleong has joined #openstack-barbican14:49
*** testtest_ has joined #openstack-barbican15:26
*** corey_ has joined #openstack-barbican15:26
*** cleong has quit IRC15:27
*** testtest_ has quit IRC15:28
*** nkinder has quit IRC15:54
*** namnh has quit IRC16:36
*** catintheroof has joined #openstack-barbican16:38
*** david_8 has joined #openstack-barbican16:45
*** catintheroof has quit IRC16:45
*** david_7 has quit IRC16:49
*** catintheroof has joined #openstack-barbican17:23
*** catintheroof has quit IRC17:30
*** catintheroof has joined #openstack-barbican18:36
*** catintheroof has quit IRC18:36
*** catintheroof has joined #openstack-barbican18:36
*** catintheroof has quit IRC18:36
*** namnh has joined #openstack-barbican18:37
*** namnh has quit IRC18:42
*** catintheroof has joined #openstack-barbican18:56
*** corey_ is now known as cleong19:16
*** ssathaye has joined #openstack-barbican19:17
EmilienMalee: can you join #tripleo please?19:32
*** dave-mccowan has joined #openstack-barbican20:12
*** namnh has joined #openstack-barbican20:38
aleedave-mccowan, hey20:39
aleedave-mccowan, hey - got a barbican problem to troubleshoot --- have some time to help?20:41
aleedave-mccowan, cool20:42
aleedave-mccowan, so - in tripleo, I start up barbican-api and the keystone-listener and barbican-worker20:42
*** namnh has quit IRC20:42
aleedave-mccowan, when I started them all at the same time , I ended up in db deadlocks20:43
aleedave-mccowan, now I end up with the occassional deadlock20:43
aleedave-mccowan, trying to figure out whats going on20:46
aleedave-mccowan, the deadlock is happening when we populate the db with secret stores as per multile secret store setup20:46
aleedave-mccowan, is it possible that both the barbican-api and the barbican-worker go through this code?20:47
*** raildo has quit IRC20:48
dave-mccowando you think a process is not cleaning-up after a write?  or is a process holding one lock, while waiting for another resource?20:51
aleedave-mccowan, the latter20:51
aleedave-mccowan, I think two processes are trying to access the db concurrently and stepping over each other20:52
aleedave-mccowan, if both worker and barbican-api go thorugh the same code -- maybe they step on each other trying to make the same insert?20:53
dave-mccowanif there's only one resource, it wouldn't deadlock, would it?20:54
aleedave-mccowan, I'm just guessing , but it does seem like that code needs some kind of semaphore around it to make sure it only gets executed by a single thread20:55
aleeor process20:55
aleedave-mccowan, I'm trying to figure out how the worker starts up -- does it actually execute that code?21:00
*** ssathaye has quit IRC21:08
aleedave-mccowan, so that code is eventually triggered by setup_database_engine_and_factory() in repositories,py21:10
*** salmankhan has joined #openstack-barbican21:11
aleedave-mccowan, which is executed in, and server.py21:12
aleedave-mccowan, basically I think all of worker, notifier and api-server21:13
aleedave-mccowan, and concurrent access doing the same thing causes deadlock somehow ..21:14
aleedave-mccowan, how can we ensure only one goes through at a time?21:15
dave-mccowanalee the deadlock is always inserting into the secret_stores table?21:19
aleedave-mccowan, in the cases where I have seen it - yes21:20
*** ssathaye has joined #openstack-barbican21:20
*** catintheroof has quit IRC21:20
aleedave-mccowan, not saying it wont happen elsewhere --- but have not seen it elsewhere21:20
aleedave-mccowan, we have not seen this in devstack because more likely than not 1) we dont have listener and worker in devstack (2) we aren't testing multi-secret store config21:26
*** cleong has quit IRC21:26
aleedave-mccowan, we probably need to fix both of those things21:26
aleedave-mccowan, I do know that I did not start seeing these errors when I added the listener and worker21:27
*** salmankhan has quit IRC21:28
dave-mccowanthe multiple backend code is the freshest, so it makes sense new issues would be there.21:29
aleedave-mccowan, so how to fix -- some sort of file lock?21:30
aleedave-mccowan, or some kind of lock at the db layer ..21:30
aleedave-mccowan, do we implement locks anwyhere in the barbican code?21:31
dave-mccowani was just seaching db deadlocks.  the typical cause is doing things in a different order.  as long as each thread does the inserts in the same order, it shouldn't lock.21:31
dave-mccowanthe sql code should be managing the locks for us.21:31
aleedave-mccowan, but its the same insert being done by multiple processes ..21:32
*** openstackgerrit has quit IRC21:33
dave-mccowanthe table should be locked during the insert.  a problem could occur if inserts are done to both project table and secret store table, but in a different order for two threads.21:34
*** salmankhan has joined #openstack-barbican21:43
aleedave-mccowan, so whatya think?21:45
dave-mccowanalee i think a timing conflict between project table and secret store table.  since the project_id is a foreign key of secret_store, a process needs to lock both tables to insert into the secret store.  maybe one of the threads is still locking the project table, then goes to initialize the secret store, while another process starts with the secret store.21:48
dave-mccowani'd add more logging and then recreate.21:54
aleedave-mccowan, its not straightforwrad to recreate -- being a timeing thing and all22:03
*** salmankhan has quit IRC22:20
*** namnh has joined #openstack-barbican22:39
*** namnh has quit IRC22:43
*** catinthe_ has joined #openstack-barbican23:08
*** catinthe_ has quit IRC23:13
*** tonyb has quit IRC23:26
*** tonyb has joined #openstack-barbican23:36

Generated by 2.15.3 by Marius Gedminas - find it at!