Monday, 2015-09-07

*** Qiming_ has joined #senlin01:03
*** lawrancejing has joined #senlin01:10
*** LiuWei has joined #senlin01:16
*** Yanyanhu has joined #senlin01:32
Qiming_admin__, there?01:50
xuhaiweihi, Qiming, it's ok to ask a question now?01:55
*** elynn has joined #senlin01:57
xuhaiweiabout the set_status() method in actions/base01:57
*** lawrancejing has quit IRC01:57
xuhaiweiwhen want to set the status as READY, the db api will do action_abandon?01:58
xuhaiweiit seems not appropriate01:58
*** lawrancejing has joined #senlin01:58
Qiming_when we want to enable an action to be retried later, we set its status back to READY, so that the scheduler will schedule it for execution later01:59
Qiming_setting status to READY is not the goal, it is a preparation for 'abandon'02:00
openstackgerritYanyan Hu proposed stackforge/senlin: Reinitialize action properties after locking it
xuhaiweiwhen setting the action to READY, it means we want some high priority action to be ran first?02:02
Qiming_there is no priorities among actions at the moment02:03
Qiming_when an action's status is READY and its lock released by the 'abandon' operation, we are expecting that the scheduler will schedule it for execution eventually02:04
Qiming_the action's current status, before 'abandon' should be RUNNING02:05
*** zhenguo_ has joined #senlin02:07
*** elynn_ has joined #senlin02:08
*** elynn_ has quit IRC02:08
xuhaiweithe dependency is added before setting the action's status, so there should be already priorities at that time02:10
*** zhenguo has quit IRC02:15
*** zhenguo_ is now known as zhenguo02:16
Qiming_xuhaiwei, those are not 'priorities'02:16
Qiming_those are 'dependencies', expressed using the 'depends_on' and 'depended_by' fields in the action table02:17
xuhaiweishoud the actions marked as depended_by be ran first?02:17
Qiming_we don't know02:18
Qiming_we leave that to the scheduler02:18
Qiming_the scheduler is expected to run any action that is in READY state02:19
xuhaiweiso why the 'dependency' field is needed02:19
Qiming_it is used to emulate fork-join02:19
openstackgerritYanyan Hu proposed stackforge/senlin: Reinitialize action properties after locking it
Qiming_action A forks actions B, C, D, for example02:20
Qiming_using dependency, A puts itself into WAITING status, and B, C, D into READY status02:20
Qiming_when B, C, D are completed, they will put A back to READY status02:21
Qiming_then A continues to run02:21
xuhaiweiso if A is dependened by B, A should be ran earlier than B?02:22
Qiming_A will be run, because A is put into READY and B is in WAITING status02:23
Qiming_that is not a priority thing02:23
Qiming_priority, if you want to implement it, refers to fact that A is always executed before B, if both A and B are in READY status02:24
xuhaiweiok, got it02:24
*** branw has joined #senlin02:30
xuhaiweiso when we set the action to 'READY' status, we want it to be ran, but in action_abandon() we mark the status_reason as 'The action was abandoned.', that is confusing02:30
Qiming_xuhaiwei, you can propose a better, more generic logic if that is confusing you, :)02:34
xuhaiweiI thought about it and felt READY and RES_RETRY should be treated differently, but not sure about it02:42
openstackgerritYanyan Hu proposed stackforge/senlin: Reinitialize action properties after locking it
Qiming_Yanyanhu, about the patch 22086602:55
Qiming_I'm thinking of a revision to the ActionProc to solve the problem once for all02:55
Qiming_currently, line 468 is not a mandatory step02:56
Qiming_We can delete it "Step 1" completely02:56
Yanyanhuyes, if we can build a 'context' for db access02:57
Yanyanhuthen we can load the action obj only after the action is locked successfully02:57
Qiming_The warning on line 478 can be changed to reflect the possible reaons: either the action doesn't exist, or it has been locked by another worker02:58
Qiming_we then materialize the action object by moving line 468 after line 47902:59
Qiming_shit, this is incorrect either03:00
Yanyanhubut for db access in line475, we still need a correct context03:00
Qiming_the key is to use the right context03:00
Yanyanhuso still the context problem...03:01
Yanyanhuneed a complete refactoring here03:01
Qiming_right, has to be done step by step03:02
Qiming_first of all, modify __init__() to avoid the creation of new context if context is provided03:03
YanyanhuI have made a try to redesign context usage in action object as we discuessed last week, but met some issues03:03
Yanyanhuso the basic idea is for action derived from existing action, we didn't create new context03:04
Yanyanhuwe create new context just for those actions created for RPC request03:04
Qiming_when __init__ is called, the testing can be simplified03:05
Qiming_just check if context is provided03:05
Yanyanhuyes, if not, we use the user infor stored in target to initialize the new context for db acess03:07
*** Qiming__ has joined #senlin03:07
*** Qiming_ has quit IRC03:07
Yanyanhuoh, I recalled those two questions I met when trying to make this change03:07
*** elynn has quit IRC03:08
YanyanhuThe first thing is we need a way to transmit 'is_admin' property. It is not contained in target object.03:09
*** elynn has joined #senlin03:09
Yanyanhuumm, this is the only issue. The second one about target obj can be solved using another way.03:11
Qiming__"is_admin" ?03:12
Qiming__why is it important?03:12
Yanyanhuthe context need it for initialization03:12
Yanyanhuand the policy check will use it then03:13
Qiming__it is a problem need to be fixed03:13
Yanyanhuseems for DB, we don't use this property03:14
Yanyanhuanother interim solution can be setting is_admin=true when new context is created in action initialization progress since we don't rely on policy to do any check in action layer03:27
Yanyanhuor we remove context from action layer completely03:35
Yanyanhudesign another class for DB access03:36
Yanyanhuor in first step, we define a class derives from RequestContext and override its methods like __init__03:37
Qiming__another class sounds good03:38
Yanyanhuand init those attributes just needed by DB access03:38
Qiming__don't mix it with the current 'RequestContext'03:38
Qiming__there is no03:38
Qiming__such thing once the request is turned into an internal action03:39
Yanyanhuok, then we know the 'context' in DB access progress is actually different from the one used in API/engine service layer03:39
*** Qiming__ has quit IRC03:44
openstackgerritxu-haiwei proposed stackforge/senlin: Do some source cleanup
*** Qiming__ has joined #senlin05:23
openstackgerritMerged stackforge/senlin: Do some source cleanup
*** branw has quit IRC05:50
*** Shijia has joined #senlin05:52
Yanyanhuhi, Qiming, just quick looked through related code of olso_db, I think we may need to customize enginefacade module in oslo_db/sqlalchemy/ locally to use our own Session class to replace the default one.06:27
Yanyanhuthe above two lines need to be revised06:29
Qiming__that is going too far06:30
Qiming__we can just provide a DBSession class, for example, wrapping the current session reference into this new class06:31
Yanyanhuthis way06:31
YanyanhuI thought you mean adding some new properties like project into the Session class06:31
Qiming__don't introduce more inheritance please06:32
Yanyanhuso I tried to find a way to override the definition of Session06:32
Qiming__it will make our code very dependent to the implementation details of other projects06:32
Yanyanhuthat will be too complicated and we need to maintain those code06:32
Yanyanhuwill be nightmare06:32
Yanyanhuok, will do the job by defining a DBSession class06:33
Qiming__the first job in the queue is to revise the Action constructor I think06:34
Yanyanhuyes, we don't need context in action any more06:34
Yanyanhua DBSession obj will replace it06:34
Qiming__then we replace the (revised) context parameter with a DBSession object06:34
Yanyanhuin first step, we don't change the param name defined in DB layer06:35
Yanyanhuthat will be a huge change06:35
Qiming__We have to make sure each step made is not worsening the situation at least06:35
Qiming__think about it06:36
Qiming__suppose you have a DBSession class at hand, how would you change the Action intialization code?06:36
Qiming__change the intialization logic, and at the same time, replace the context with an object of a new class?06:37
Yanyanhuguess so06:37
Qiming__that is too dangerous a thing to do06:38
Qiming__the Action initialization part is already very sensitive to any changes06:38
Yanyanhuhmm, that's true06:39
Qiming__and I'd strongly suggest you to do this by steps06:39
Yanyanhuwill give the definition and DB model support of DBSession class06:40
Yanyanhuand then touch the Action base module06:40
Yanyanhuoh, not DBSession class06:41
Yanyanhulet me see06:41
Yanyanhuwe don't need a DB model for DBSession class06:42
Yanyanhujust covert it to dict before storing06:42
Qiming__DB model for DBSession class?06:42
Qiming__a new table?06:42
Yanyanhufalse statement06:43
Yanyanhuplz just ignore it :)06:43
Qiming__admin__, online?07:03
openstackgerritYanyan Hu proposed stackforge/senlin: Reinitialize action properties after locking it
openstackgerritMerged stackforge/senlin: Reinitialize action properties after locking it
openstackgerritYanyan Hu proposed stackforge/senlin: Implement DBSession class
Yanyanhuhi, Qiming__, just proposed the patch to add DBSession class, please help to review. Thanks08:02
Qiming__any reasoning behind the late initialization of the DBSession._session?08:03
Yanyanhuno special reason, just feel maybe this can reduce the opening time of session connection08:04
Qiming__don't understand08:06
YanyanhuI mean if we initialize the _session property of DBSession only when it is first referred, we don't need to create DB connection until the DBSession.session is really used to open DB transaction08:08
Qiming__if we don't use DBSession.session, why are we creating DBSession anyway?08:09
Yanyanhuyes, we will use it. But not just after it is initialized. For example, in action.__init__, a DBSession object will be created. But the _session will not be used until we first time invoke a db_api interface with DBSession object.08:11
Yanyanhuso I think we just need to initialize the _session in that time but not in action.__init__08:12
Qiming__so .... the point is, we need to think through the whole life cycle of the DBSession object08:12
Yanyanhuactually I'm wondering maybe we can have a place to store those opened session and control them by ourself08:12
Yanyanhurather than just let them been destroyed when the DBSession object is destructed08:13
Yanyanhubut this may be dangerous...08:13
Qiming__surely if will be dangerous, given that concurrency is so difficult08:13
Yanyanhuso I guess late initialization of DBSession._session might be able to provide some help for reducing the presure of DB connection08:15
Qiming__I'm not sure of that08:15
Yanyanhubut no big effect actually08:15
Qiming__In senlin's execution model08:17
Qiming__how would that save us a lot resource?08:17
Qiming__That was the reason I suggested you to start with refactoring the Action initialization as the FIRST step08:18
Qiming__We are constructing a context when calling _from_db_record()08:19
Qiming__we are always creating a new context when __init__ is called08:19
Qiming__all those logic need to be revised08:19
Qiming__without revising that, replacing the context with DBSession or whatever won't be making things any better08:20
Yanyanhuyes, that's true. But without the new DBSession class, I think we can't finish the refactoring just use RequestContext I think since there are too much trash properties in this class08:22
Yanyanhuof course the key is to fix rebuilding of context/DBSession each time action is reloaded from DB08:23
YanyanhuI guess we may need to know more about the DB session to understand how it effect the progress of DB accessing since we can't store the session object in DB. Will make more investigations about this issue.08:35
*** lixinhui has joined #senlin08:49
Qiming__in senlin's execution model08:51
Qiming__an action is always retrieved from DB before execution (ref: scheduler and action.ActionProc)08:52
Qiming__thus any db session created in senlin.common.context.RequestContext won't get used during action execution08:53
Qiming__an action object is constructed before execution and it becomes garbage once it is finished08:53
Qiming__we have DBSession generated for action object's during this phase08:54
Qiming__then ... where is the need to persist DBSession object?08:54
Yanyanhuyes, this is the problem.08:57
*** Shijia has quit IRC08:57
Qiming__DBSession is only supposed to be used by an Action instance, I don't feel the need to persist it08:57
YanyanhuBut we need to store some properties like project, user into DB since they should be part of DBSession08:58
Qiming__why into DB?08:58
Qiming__the DBSession object is always in memory08:58
Yanyanhuyes, DBSession object is always in memory. But session property of action will be stored into DB which actually contains information of DBSession object09:00
Yanyanhuwhen action is reloaded from DB, its context/session property will be re-initialized based on this record09:00
Qiming__is that a problem?09:01
Yanyanhuit is09:01
Yanyanhuthe ideal way is reuse the context/session object that in memory when action is reloaed09:01
Qiming__when reloaded from DB, an action ALWAYS creates a new (in-memory) DBSession for execution09:01
Qiming__the best way to detect/handle concurrency problems is to have each action working with its own DB session09:03
Yanyanhubut I think we may not be able to resolve this issue if we don't manage the lifecycle of orm.session.Session by ourself09:03
Yanyanhuthat's our goal09:03
Qiming__you may want to save your effort on that09:03
Qiming__if you dive into the orm model, you will get lost for sure09:04
Qiming__the session design is more complicated than we can imagine09:04
YanyanhuYes, I found it's almost impossible to do that09:04
Yanyanhujust need can't figure out how can we keep the session alive09:05
Qiming__and there is no benefit, other than the imagined session savings09:05
Yanyanhuduring the store/reload progress of action09:05
Qiming__... I don't think keeping session alive is that important09:06
Yanyanhuyes, that's right09:06
Yanyanhuthis is why the context object rebuilding in _from_db_record is allowed I think09:07
Yanyanhuthe difference between new/old context objects is their session property09:07
Qiming__we create new context/DBSession intentionally09:08
Yanyanhuthe context recreate in Action.__init__ can be removed09:08
Yanyanhuyou mean?09:09
Qiming__we create new DBsession (or 'context') when we load an action from DB09:10
Qiming__we do this creation only once09:10
Qiming__that is the first step09:10
Yanyanhuthen we spread it to derived actions(like node create action)?09:11
Yanyanhuspread the DBSession object09:11
Qiming__then, if we have proofs to show that inherit 'context'/'dbsession' is something worth to do, we do it09:11
Qiming__if not, we don't do it09:12
Qiming__by the way09:12
Yanyanhuinherit? You mean use the same DBSession object created in the first time action object is reloaded from DB?09:13
Qiming__I have tried that path ... having "child" actions to "inherit" DBsessions from "parent" action09:13
Qiming__it is not passible09:13
Qiming__it is impossible09:13
Yanyanhuyes, that's the problem09:13
Qiming__don't understand 'inherit'?09:13
Yanyanhuit's impossible since we are in a multi-region env09:13
Qiming__more than that09:13
Qiming__action triggering is done via IPC09:14
Qiming__you cannot pass in-memory object in that way09:14
Yanyanhuso DB is the only way we can transmit the action information09:14
Qiming__DBSession is an in-memory object09:15
Qiming__session is an in-memory object09:15
Qiming__how would you handle DBMS restart?09:15
Yanyanhuyes, but how we transmit them between two individual process without manage the memory sharing by ourself?09:16
Qiming__how do you persist a db-session object, serialize it and deserialize it?09:16
Qiming__we don't transmit them!09:16
Qiming__we don't pass session from one action to another09:17
YanyanhuQiming__> how do you persist a db-session object, serialize it and deserialize it?  This is just what I want to say...09:17
Qiming__we don't 'spread' it, we don't allow inherit it09:17
Yanyanhuseems some misunderstanding here...09:17
Qiming__each action will have a db-session (in-memory) property during its execution09:18
Yanyanhuthat's right09:18
Qiming__then why the question: but how we transmit them between two individual process without manage the memory sharing by ourself?09:18
YanyanhuI thought you mean we don't store action/session into DB09:19
Yanyanhuand directly share the same action/session object between multiple engine. I think this is impossible09:19
Qiming__we don't store it, yes, we don't09:19
Qiming__we don't share it09:19
Qiming__I never said sharing it09:20
Qiming__Qiming__> I have tried that path ... having "child" actions to "inherit" DBsessions from "parent" action09:20
Qiming__Qiming__> it is impossible09:20
Qiming__it is not possible even between parent and child actions09:20
Qiming__how could you enable it in other scenarios?09:21
Yanyanhuok. So if we neither store action into DB or share it, how we notify other workers to execute new actions created in current worker thread?09:23
Qiming__show me the line of code you are talking about?09:24
Yanyanhu? you mean the code of notify?09:24
Yanyanhuok, let me check it09:25
Yanyanhuthis line I think09:28
Qiming__1 min, talking to Ji Zi Lian09:28
Qiming__okay, why we need a context there?09:30
Yanyanhuwe don't09:31
Qiming__it seems we are passing 'context' on line 10209:32
Yanyanhuthis is a historical problem I think09:32
Qiming__but the context you get on line 54 is a completely different thing09:32
Yanyanhuyes, the same as the one from api to engine09:32
Yanyanhuthey are two objects in two different threads09:34
Yanyanhuthere is a serialization and deserialization progress09:36
Yanyanhuwhen this param transmitted through rpc09:36
Qiming__why is it needed?09:40
Yanyanhuit is not needed. But when we started to write the code, context is still there09:41
Yanyanhuin every coner of senlin service09:41
Yanyanhubut it is not needed actually09:42
Qiming__so ... pls help revisit that code09:42
Yanyanhuactually after the code execution is outof engine/service, we don't need context any more09:42
Yanyanhuwill recheck it09:42
Yanyanhuwill propose a patch for this09:44
xuhaiweisorry to interpret09:44
xuhaiweiis an action executed only when it is in 'READY' status?09:45
xuhaiweiI found the set_status is not used to all the actions09:46
xuhaiweiin engine/service layer, we dont set status for actions09:46
Yanyanhuhi, xuhaiwei, I guess the action is in INIT status before the first time execution of an action09:48
Yanyanhulet me have a look09:48
Yanyanhuafter the action is created, its status is INIT. Then after it is locked, it will be marked as RUNNING09:49
xuhaiweiyes, in engine/service layer, the actions are INIT status, but in the cluster/node actions layer, they are marked READY before being executed09:49
xuhaiweithat's right Yanyanhu, I just wonder if we need to check the action's status before locking it09:50
Yanyanhuhmm, I guess we don't.09:52
Yanyanhuor it's too complicated to check all cases?09:52
xuhaiweiif we don't check the status, an action marked FAILED will also get the lock , though currently no actions are marked failed09:54
Yanyanhuyou mean the lock of cluster?09:54
Yanyanhuor the lock of a cluster_action09:54
Yanyanhuso what you mean by 'an action marked FAILED will also get the lock'09:55
Yanyanhuyou mean an action marked as FAILED will be locked by a worker?09:55
xuhaiweicurrently we only set actions' status to READY, but if an action is marked as FAILED by  set_status, it will also be executed by the scheduler09:57
xuhaiweiIf it is not, I think we dont need to set status to action09:58
Yanyanhuhi, xuhaiwei, you're right if the work will check the action table by themselves periodically09:58
Yanyanhuand that is one of our goal in future09:58
*** lawrancejing has quit IRC09:59
Yanyanhubut currently, action dispatching only happens after the action is first time created09:59
Yanyanhuso we don't need to worry an action with FAILED status will be rescheduled by other worker threads10:00
xuhaiweiyes, currently not, I just think we need to improve the action execution10:02
*** elynn has quit IRC10:03
Yanyanhutheoretically, worker should check the action status before acquiring its lock10:03
Yanyanhuit should only acquire lock of those actions whose status is READY(or INIT?)10:04
xuhaiweiI think READY is meaningful10:04
Qiming__the current scheduler is not doing scheduling at all10:06
Qiming__it only accepts a request and start an action immediately10:06
Qiming__the design/goal was to have it work like a scheduler10:06
Qiming__so it will do things like invoke this api:
xuhaiweiso the status logic is not working actually, right?10:07
Qiming__the pitfall is as what I have just outlined above10:08
Qiming__if you equalize that to "the status logic is not working actually, ", fine10:08
Yanyanhuhi, xuhaiwei, the status logic is mainly used for dependency checking now I think10:09
Qiming__one of the most complicated handling of action status is here:
Qiming__if you read this set_status() function:
Qiming__you will get a better understanding how we have been making heavy use of action.status10:11
Qiming__when we call db_api.action_mark_failed() there, we are doing a lot in the database layer, all manipulating action.status10:14
xuhaiweiyes, but after I read the set_status method, I got a little confused, why in engine/service layer, we dont set status to action before executing it10:14
Qiming__we should10:14
Qiming__the code still works because "<Qiming__> the current scheduler is not doing scheduling at all"10:15
xuhaiweigot it10:16
xuhaiweiI'd like to make a patch to make it a little nearer to our goal10:16
Yanyanhuxuhaiwei, that's great :)10:18
*** Qiming__ has quit IRC10:53
*** Yanyanhu has quit IRC10:56
openstackgerritxu-haiwei proposed stackforge/senlin: Check action status before lock it
*** lixinhui has quit IRC11:12
*** lawrancejing has joined #senlin11:33
*** lixinhui has joined #senlin12:09
*** Qiming_ has joined #senlin12:12
*** lixinhui has quit IRC12:14
Qiming_xuhaiwei, still on?12:19
*** Qiming_ has quit IRC12:33
*** Qiming__ has joined #senlin12:33
*** lawrance_ has joined #senlin12:35
*** lawrancejing has quit IRC12:37
openstackgerritMerged stackforge/senlin: Cluster action test case (1)
openstackgerritMerged stackforge/senlin: Cluster action test case (2)
*** zhenguo has quit IRC13:00
*** lixinhui has joined #senlin13:02
openstackgerritCindia-blue proposed stackforge/senlin: Add a placement-policy to enable vSphere DRS functions
openstackgerritMerged stackforge/senlin: Add test case for Node Action
openstackgerritQiming Teng proposed stackforge/senlin: Cluster action test case (3)
*** lixinhui has quit IRC14:46
*** Qiming__ has quit IRC15:28
*** lawrance_ has quit IRC15:28
*** Qiming__ has joined #senlin23:41

Generated by 2.14.0 by Marius Gedminas - find it at!