Friday, 2015-08-07

*** Qiming has quit IRC00:02
*** Qiming has joined #senlin01:11
*** Yanyanhu has joined #senlin01:15
QimingHi, Yanyanhu01:24
Qimingline 2101:25
Qimingis the $ sign necessary?01:25
Qimingit is a weird syntax to me01:25
Yanyanhuem, seems so, since without it, we can't get correct DEVSTACK_LOCAL_CONFIG01:26
Yanyanhualso found other projects used the similar way to add more than one line into DEVSTACK_LOCAL_CONFIG01:26
Qimingokay, just want to make usre it is not an ignorance01:26
openstackgerritMerged stackforge/senlin: Do not update other properties if exception happens
*** Tennyson has joined #senlin01:37
*** ChrisSen has joined #senlin01:38
Yanyanhuhi, Qiming, about the functional test, have made some investigations but still didn't find a good way to fake the interaction between Senlin and other services...01:45
Qimingokay, keep digging then, :)01:46
*** mathspanda has joined #senlin01:46
Qimingit's not that easy a task I know01:46
Yanyanhuhmm, I'm thinking maybe we just put those cases where Senlin need to talk with other services into integration test?01:46
*** Qiming has quit IRC01:48
*** Qiming has joined #senlin01:48
Qimingwell .... then there would be many workflows not tested in the 'functional' tests01:49
Yanyanhuthat's true01:49
Yanyanhuand about functional test in other services, I found nova/cinder are using fixture to build their functional tests, but using that way, they fake the entire nova/cinder services I think... and may need to reorg the design of our service to support this way01:50
Yanyanhuactually I think nova/cinder's functional test is actually an enhanced version of unit test :)01:51
Qimingthat sounds the right thing to do though01:51
Qimingmaybe we can use fixtures just for sdk?01:52
Yanyanhuem, but that's not the functional test we discussed I think, they don't run nova/cinder in real environment actually01:52
Qimingwe fake sdk's calls as if compute is there, orchestration is there01:52
Qimingis this possible?01:53
YanyanhuI'm afriad we can't since they need fake the entire nova/cinder services to achieve this goal...01:53
Yanyanhuthen they can control every methods called in services01:53
Qimingwhat do you mean by fake the nova service?01:53
Qiminggiven that your goal is to test the nova service?01:54
Yanyanhuwill go downstairs, back soon01:54
Yanyanhuhi, I'm back01:59
YanyanhuI'm not quite sure how to use this way, but it seems that they run a 'fake' nova/cinder service using fixture support02:00
Yanyanhuthen mock methods when necessary02:01
Yanyanhuactually, they mainly use this to support their unit test02:01
Qimingthat is where we need to dig ?02:01
Yanyanhuem, yes02:01
Qimingwe borrow but we don't just copy, right?02:01
Yanyanhubut I think maybe it's not a proper way for us02:02
Qimingjust don't know02:02
Yanyanhusince for nova/cinder, they only have little requirement to talk with other services02:02
Yanyanhuthis is different from us02:02
Qimingright, what we need is to make a fake sdk02:03
Yanyanhuso the scale of their functional test is small02:03
Qimingjust like you have made a "TestProfile"02:03
Yanyanhuyes, that's what we want02:03
Qimingwe borrow, but we don't just copy02:03
Yanyanhujust not sure we can do that without mock the entire services, since sdk is a lib actually02:03
Qiminghow many interfaces are we actually using?02:04
Yanyanhuif so, maybe fake the driver is a better way?02:04
Yanyanhuem, if we want to test all policies and profile types, I think most of the methods in driver may need to be faked02:05
Yanyanhuanyway, will keep investigating this to see whether we can just fake part of the service using this way02:05
Yanyanhuif so, that would be the best I think02:06
Qimingwe just fake/mock what is called02:06
Qiminge.g. server_interfaces? keypairs? images? ....02:06
Qimingthere are so many things we are not using today02:06
Yanyanhuif our driver is implemented as plugin, that's much better :)02:06
*** jqiu has joined #senlin02:07
*** ChrisSen has joined #senlin02:07
*** jqiu has left #senlin02:07
Qimingwhen we are using them someday, we fake them then02:07
Yanyanhujust not sure whether we can fake a method inside a service that running in the real environment02:08
Qiminggoogle it?02:09
Yanyanhulet me make more deep digging02:09
Qimingif there are evidences that is impractical02:09
Yanyanhucan't access it... sigh, so just used Bing02:09
Qiming.... I mean "google" ...02:09
Qimingit would be great if we can switch modules without restarting a service02:10
Yanyanhuyes, definitely02:11
Qimingif that is proved not possible, we switch between modules and restart the service02:11
Qimingfor some projects, they even provide a different config file, even a different service wrapper for this purpose02:11
Yanyanhuservice wrapper is?02:12
Qimingeven that has to be done, okay, we'd better make the changes less intrusive02:12
Qimingthe so called 'fake service'02:13
Qimingyou got to control which part you will run the real code, which part you will run the mocks02:13
openstackgerritQiming Teng proposed stackforge/senlin: Tune scaling_in policy unit tests
openstackgerritQiming Teng proposed stackforge/senlin: Tune scaling_in policy unit test
ChrisSenI tried to create a cluster with a specified capacity of 2 nodes. One node was created successfully, but the creation of the second one failed with an error. But the error does not seem to be reproduceable, since the creation of the cluster succeeded entirely when I tried it again:03:15
openstackgerritQiming Teng proposed stackforge/senlin: Fix version middleware and test case
QimingChrisSen, looks like a intermittent or sporadic event?03:18
ChrisSenI think so. Yesterday the test was successful, today the error occurred only one time.03:20
ChrisSenI can keep an eye on it and will tell you, if it happens again.03:23
*** ChrisSen has quit IRC04:37
*** jruano has joined #senlin04:39
*** jdandrea has quit IRC04:39
*** ChrisSen has joined #senlin04:53
*** jruano has quit IRC04:58
openstackgerritQiming Teng proposed stackforge/senlin: Improve trigger base test case coverage
openstackgerritQiming Teng proposed stackforge/senlin: Improve event unit test coverage
Yanyanhuhi, Qiming, the commit msg and title of this patch is incorrect
Qimingok, fixed05:31
openstackgerritMerged stackforge/senlin: Fix coverage test
openstackgerritQiming Teng proposed stackforge/senlin: Tune scaling_out policy unit test
Yanyanhuhi, Qiming, about the reload support in wsgi, the reload operation is triggered by a SIGHUPInterrupt?05:48
openstackgerritQiming Teng proposed stackforge/senlin: Fix profiles API and test cases
ChrisSenIn the following scenario I tried to resize a cluster, but the cluster_resize action waits for one node_create action even though all nodes were created successfully (infinite loop). When you try to delete one node of the cluster a deadlock is caused, because the cluster is already locked by the never ending cluster_resize action.06:09
QimingChrisSen, thanks for the analysis06:12
Qimingit seems that all the later errors were caused by the "unsuccessful" CLUSTER_RESIZE operation06:13
Qimingcan we nail down the problem with a shorter story?06:14
Qiminge.g. create an empty cluster, resize it to 2 nodes06:15
Qimingthen check if the CLUSTER_RESIZE operation was completed, i.e. with lock released06:15
ChrisSenI can test that.06:15
Yanyanhuhi, Qiming, made a test and found oslo_service has supported restart service and reload configure.06:16
Yanyanhujust need to run service with deamon mode and send SIGHUP to it :)06:16
Qimingsmells like a concurrency control problem06:16
Yanyanhue.g. kill -106:16
Qimingwanna dance for that06:16
Yanyanhuyep :)06:17
Qimingthat sounds beautiful, isn't it06:17
Yanyanhuso will try to rework driver using plugin mode06:17
Yanyanhuyes, much better than the current way06:17
Yanyanhuwill file a bp for this work first06:17
Qimingyes, we borrow from others but we always try work the right way, :)06:18
Yanyanhuyes :)06:18
*** wizardYVE has joined #senlin06:29
openstackgerritMerged stackforge/senlin: Revise server delete-wait logic
openstackgerritQiming Teng proposed stackforge/senlin: Fix nodes API and test cases
*** xuhaiwei has quit IRC07:05
*** xuhaiwei has joined #senlin07:12
ChrisSenQiming, the creation of an empty cluster and resizing it to 2 nodes was successful (with lock released). But after that I was not able to create any node any more (neither with node-create, nor with cluster-create , nor with cluster-resize), it always fails with "KeyError: 'error'".07:16
ChrisSenI can restart my VM to run further tests for creating and resizing clusters and will try to summarize the findings.07:17
Qimingthx, ChrisSen07:17
openstackgerritQiming Teng proposed stackforge/senlin: Fix errors in policies API and test cases
Yanyanhuhi, Qiming, found my test environment break. Seems some werid tmp directories wee added to load_list of subunit.run07:34
Yanyanhuerror like this:07:34
Yanyanhu${PYTHON} -m discover -s .  --load-list /tmp/tmpzAu3rk07:34
Yanyanhudid this happen in your env?07:34
Qimingcheck if your .testrconf is correct07:35
Qimingmaybe do a rm -fr .testrepository07:35
Yanyanhuok, let me have a try07:35
Yanyanhuit worked will yesterday07:35
Yanyanhuit works07:37
Yanyanhuwhy clean .testrepository can solve the problem?07:37
Yanyanhuthe number of unit test cases in my env is 1111 now :)07:39
openstackgerritYanyan Hu proposed stackforge/senlin: Add environment support for driver plugin
Qiminghi, yanyan07:42
Qiminglooking at your patch07:42
Qimingcommenting here instead of on gerrit07:42
Qimingcustom_drivers seems not necessary07:43
Qimingthe likelyhood of having users to contribute new cloud drivers is low07:43
Qimingso we may don't need that07:44
Qimingremove CUSTOM_DRIVERS section then07:44
Yanyanhuok, understand07:44
Qimingit is not about 'drivers' I think07:44
Qimingit is more about a 'cloud_backends'07:45
Qimingfor example, we have 'openstack' as one option under /drivers/openstack07:45
Qimingso ... maybe we can learn from sdk's design07:46
Yanyanhubut I think we can only manage each individual plugin?07:46
Qimingwe get a backend, and having each backend provide an attribute 'compute', an attribute 'network' ...07:47
Qimingjust like sdk.conn.compute.create_server07:47
*** ChrisSen has quit IRC07:47
Qimingin that case, we don't need many drivers for each service07:48
Yanyanhuok, make sense. but maybe we can't ensure the second level module can have the same method?07:48
Qimingby default, we raise not supported07:48
Qimingnot implemented ...07:49
Yanyanhuem, so this will be a real driver07:49
Yanyanhunot like the current one, which only make simple interface mapping07:50
Qimingso we treat all modules from under openstack/ as a single driver07:50
Qimingmaking them each look like a different driver would not buy us anything, just problems07:51
Qimingone driver has create_server, the other driver has create_alarm07:51
Qimingyou will need "driver type" .... a nightmare07:52
Yanyanhuso, for server creation, we always invoke cloud_backend_name.compute.create_server07:52
Yanyanhuwith the same parameters07:52
Qimingin an ideal world07:52
Yanyanhuem, looks nice07:52
Qimingwe can eliminate the senlin/drivers/openstack dir07:52
Qimingwe call sdk directly07:53
Qimingwe still maintain that dir for reasons of stability (exception catching), independence (we don't rely on its success) ...07:53
Yanyanhuactually, in my mind I thought the drivers/openstack will contains several folders including 'compute', 'network', 'identity'...07:55
Qimingwhen the 'cloud_backend' is set to 'openstack', we rely on openstacksdk, when it is set to something else, hopefully, there will be a sdk for use ...07:55
Yanyanhuand also the same for drivers/aws or drivers/softlayer07:55
Qimingwhen there are more than one compute, more than one network service for openstack, that will make sense07:56
Yanyanhuyou mean there is only one file under drivers/openstack/ path?07:56
Yanyanhulike drivers/openstack.py07:56
Yanyanhuor drivers/softlayer.py07:56
Qimingdrivers/openstack/ will be left there07:56
Qimingwhy are we combining the existing drivers into a single file?07:57
Yanyanhuoh, you mean we just create a new directory to place these new drivers07:57
Qimingjust the drivers/ directory would be fine, right?07:58
Qiming `--drivers08:02
Qiming      +--- generic intefaces08:02
Qiming      +--- and driver modules08:02
Qiming      +--- openstack08:02
Qiming      |       +-- nova_v208:02
Qiming      |       +-- keystone_v308:02
Qiming      |       `-- neutron_v208:02
Qiming      `--- softlayer08:02
Qiming              +-- compute_v108:02
Qiming              `-- metric_v508:02
*** ChrisSen has joined #senlin08:06
openstackgerritYanyan Hu proposed stackforge/senlin: Add environment support for driver plugin
openstackgerritxu-haiwei proposed stackforge/senlin: Revise senlin senlin exception handling from SDK side
openstackgerritxu-haiwei proposed stackforge/senlin: Revise senlin exception handling from SDK side
openstackgerritMerged stackforge/senlin: Add environment support for driver plugin
xuhaiweiQiming, about the patch I just submitted, just a try to do it, no comment about it?08:49
Qiminglooks tood08:50
xuhaiweiok, since it is passed, will do more jobs about it in the next week08:51
xuhaiweiI will leave now, have a good weekend guys08:52
xuhaiweisee you next week08:52
Qimingyou too08:52
*** xuhaiwei has quit IRC08:52
Yanyanhuthis guys left too soon, even didn't get chance to say byebye to him :)08:54
openstackgerritMerged stackforge/senlin: Revise senlin exception handling from SDK side
openstackgerritYanyan Hu proposed stackforge/senlin: Add cloud_backend_name option into Senlin config
Yanyanhuhi, Qiming, about this patch which tries to add the cloud_backend_name option
QimingYanyanhu, seems you are "putting the horses bind the carts"09:00
Qiming210257 is supposed to be te last patch, right?09:01
Yanyanhuoh, just start of the blueprint I think09:02
Yanyanhuwill also added generic driver implementation09:02
Qimingonce you have a generic driver implementation09:02
Qimingthis option value will be used09:03
Yanyanhuyou mean I shouldn't use the same branch to propose multiple patches?09:05
*** mathspanda has quit IRC09:12
Qimingmaybe I was wrong09:14
Qimingthis option should be there in the first place, then the driver interface would determin which backend to use09:14
Yanyanhuyes, that's right I think09:15
Yanyanhuand the option should define the default cloud_backend senlin will talk with when invoking driver methods09:16
Yanyanhubut if cloud_bakend_name is provided in e.g. profile, we will use the specified one to get the correct driver module09:16
Yanyanhuso we can talk to multiple clouds at the same time09:17
Qimingcloud_backend_name provided in profile?09:17
Yanyanhumaybe not this name09:17
Yanyanhusince I think a profile should be able to decide which cloud it will talk with09:17
Yanyanhuin that case, we need to load correct driver based on profile definition09:17
Qimingthat is called context09:18
Yanyanhuthe context09:18
Qimingwe don't know what other clouds would look like yet09:19
Yanyanhuyes, but we can know its name and if we have a driver plugin for that cloud, we should be able to get it based on the name?09:20
*** Tennyson has quit IRC09:20
Qimingbackend is about the cloud type09:21
Yanyanhumaybe not the name, could be a unique symbol we can identify it09:21
Qimingyou are talking about a cloud instance09:21
Qimingthat will be indicated via some auth_url kind of thing09:21
Yanyanhufor serveral cloud instances with the same type, we just use the same drivers09:22
Qiminglink to stackforge namespace patch?09:22
Qimingdriver is a implementation thing09:22
Qimingwe will figure out the interface for users when there is a need09:22
Qimingwhen we really have multiple implementations09:22
Qiminglink to stackforge namespace patch?09:23
Yanyanhustackforge namespace patch is?09:23
Qimingthe patch about deprecating stackforge namespace09:24
QimingI cannot find the linke09:24
YanyanhuI thought you said we should do something about 'link to stackforge namespace patch' when new driver implementation is added...09:25
ChrisSenQiming, I retestet the cluster create and resize operations and met the same problems as before: 1. Creating an empty cluster and resizing it to 2 nodes succeeds. 2. Creating a cluster with minimum, maximum and default capacity and resizing it causes an infinite loop - either the cluster_create or the cluster_resize action waits for the node_create action to be finished even though this operation already succeeded. 3. After a while it is not possible t09:34
ChrisSeno create any nodes any more because node_create causes a "KeyError: 'error'". I could observe this behavior several times, but it can not be forecasted 100%.09:34
*** wizardYVE has quit IRC09:37
Qimingokay, please help file a bug?09:38
Qimingneed to check it when cycles allow09:39
Qimingthe worst case would be some concurrency problem on locks09:39
*** Qiming has quit IRC09:45
*** Yanyanhu has quit IRC09:45
*** Yanyanhu has joined #senlin09:45
*** Yanyanhu has quit IRC09:50
*** ChrisSen has quit IRC09:56
*** openstackgerrit has quit IRC10:45
*** openstackgerrit has joined #senlin10:49
*** mathspanda has joined #senlin11:19
*** mathspanda has quit IRC12:33
*** mathspanda has joined #senlin12:36
*** jruano has joined #senlin12:41
*** lixinhui has quit IRC13:01
*** lixinhui has joined #senlin13:02
*** jhroyal has joined #senlin13:03
*** jdandrea has joined #senlin13:21
*** jhroyal1 has joined #senlin14:03
*** jhroyal has quit IRC14:04
*** lkarm has joined #senlin14:21
*** lkarm has quit IRC14:28
openstackgerritLinPeiyu proposed stackforge/senlin: B64encode user_data content before handling it to nova
*** mathspanda has quit IRC15:39
*** Qiming has joined #senlin15:48
*** mathspanda has joined #senlin15:56
*** mathspanda has quit IRC16:08
*** mathspanda has joined #senlin16:09
*** Qiming has quit IRC16:12
*** mathspan_ has joined #senlin16:12
*** mathspanda has quit IRC16:15
*** mathspan_ has quit IRC16:20
*** jdandrea has left #senlin18:10
*** jhroyal1 has quit IRC18:17
*** jhroyal has joined #senlin18:33
*** jhroyal has quit IRC20:15
*** jruano has quit IRC20:45
*** jruano has joined #senlin23:30

Generated by 2.14.0 by Marius Gedminas - find it at!