Thursday, 2015-12-17

*** Qiming has quit IRC00:08
*** pratikmallya has joined #senlin00:42
*** Qiming has joined #senlin00:56
*** zhenguo has joined #senlin01:38
*** Yanyanhu has joined #senlin01:45
*** Yanyanhu has quit IRC02:03
*** elynn_ has joined #senlin02:05
*** elynn_ has quit IRC02:10
*** elynn_ has joined #senlin02:10
*** lixinhui has joined #senlin02:25
elynn_Seems gerrit UI has changed...02:25
Qimingyes, a new version02:25
elynn_Need time to familiar with it...02:26
elynn_Can't find the comment button now...02:28
Qimingnot logged in?02:29
elynn_oh, it's been moved from bottom to top, and the name changed to 'reply'...02:30
openstackgerritlvdongbing proposed openstack/senlin: Fix some nits in profile base
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
*** Yanyanhu has joined #senlin03:21
Yanyanhuthe UI of review page changed for gerrit updating?03:22
Qimingseems so, girl03:23
Yanyanhucan't find those buttons I was familiar with :)03:23
openstackgerritMerged openstack/senlin: Updated from global requirements
openstackgerritMerged openstack/senlin: Remove senlin.conf.sample
openstackgerritMerged openstack/senlin: Fix some nits in profile base
Qimingbuttons are moved, screen is busy, a lot of pop-up windows now03:37
Yanyanhuyes. Honestly, I prefer the old UI. There are too much thing in a small field now.03:38
Yanyanhumake it difficult to find something. Maybe I will feel better after I'm more familiar with the new UI.03:38
Qiminghope so03:39
elynn_I think the better place for new one is that you can see this patch will affect what other patches.03:40
QimingOne change I'd like to see is the 'Incoming reviews' list should be customizable03:42
Qimingthere are too many patches I reviewed before and now are blocked03:43
Qimingeither there have been not new patch set, or they have been -1'd without response03:43
Yanyanhupatch with -1 but without response should be abandoned a while after the last update date?03:45
Yanyanhue.g. 2 month?03:45
QimingI still have this one in the incoming reviews:
Qimingall over a year now03:46
YanyanhuI guess heat team should abandon these patches on behalf of proposer03:47
QimingI'm doing that03:47
Yanyanhuthey can restore it later when the work is resumed03:47
Qimingsometimes I have to play a bad guy03:47
Yanyanhumaybe a mail about this issue in mailinglist?03:48
Yanyanhuto raise some awareness about this problem03:48
*** lixinhui has quit IRC03:51
*** lixinhui has joined #senlin03:53
*** lixinhui has quit IRC04:15
*** pratikmallya has quit IRC04:16
*** elynn_ has quit IRC04:29
*** elynn_ has joined #senlin05:13
*** lixinhui has joined #senlin05:18
*** elynn_ has quit IRC06:17
*** elynn_ has joined #senlin06:17
openstackgerritjunxu proposed openstack/senlin: Fix policy_check for cluster binding scale-in and scale-out policies at sametime. 'policy_check' will check the cooldown of all scale policies for any scale action. This may have some problem, for example as follow.
*** pratikmallya has joined #senlin06:32
openstackgerritjunxu proposed openstack/senlin: Fix "" file's mode to 644
openstackgerritYanyan Hu proposed openstack/senlin: Set cluster.next_index to correct value after node join/leave
Qiminghelp needed07:35
Qimingwhen debugging db sync problems, found another bug07:35
Qimingdon't have bandwidth to fix it07:35
Qiminganybody can help?07:35
elynn_I can have a look.07:36
Qimingreproduce is simple07:36
elynn_What's the problem.07:36
Qimingcreate a node of heat stack07:36
Qimingdelete that stack from heat07:36
Qimingthen you cannot delete the node from senlin07:36
elynn_Qiming: I encounter this problem yesterday07:36
Qimingso please file a bug and fix it07:36
elynn_Then update openstacksdk can solve it.07:36
Yanyanhubut the heat stack was deleted correctly?07:37
Qimingheat stack doesn't exist at all07:37
QimingI wasn't able to delete heat stack07:37
Yanyanhuoh, I see07:37
Qimingit is complaining that some project cannot be deleted07:37
elynn_ignore_missing will be take care in the newest version I guess.07:37
QimingI manually deleted the stacks from heat database, it is a heat bug07:38
elynn_Seems the problem we see is not the same?07:38
elynn_You sad it's a heat problem?07:38
elynn_You create a stack, and then delete it will fail?07:39
Qimingyes, cannot delete07:39
elynn_in latest codes?07:40
elynn_Let me have a try...07:40
Qimingyes, latest code07:40
Qimingheat side trace:
Qimingit is trying to delete some projects, could be the stack_user_project07:41
Qimingoh, stack_domain_project07:41
Qimingbut somehow, it is invoking keystoneclient to do this07:41
Qimingkeystoneclient is deprecated ... don't know if it worth a fixing or not07:42
Qiminganyway, that stack cannot be deleted07:42
elynn_it's an endpointnotfound error?...07:42
Qimingthen I manually delete that stack in heat db07:42
Qimingendpoint for identity service ...07:42
elynn_Should I update my keystone codes/07:43
elynn_or keystoneclient codes...07:43
Qimingdon't know07:43
Yanyanhuhi, Qiming , this error didn't happen in my local env07:43
Qimingit is a long invocation chain, anything can be broken07:43
Yanyanhusenlin's code is latest, but heat is not I think07:44
Qimingpython-keystoneclient 2.0.007:44
elynn_There isn't any change in heat codes related with keystone07:44
elynn_Let me update my heat codes and have a try.07:44
Qimingthen maybe keystoneclient is buggy07:45
Qimingmy keystone is very new: keystone (, /opt/stack/keystone)07:45
Qimingpip install stalls .... due to firewall07:46
elynn_my keystoneclient version is 1.8.107:46
Yanyanhumine is 2.0.007:46
elynn_my keystone version is 9.0.0dev29607:47
Yanyanhukeystone version is 8.0.007:47
Qimingokay, my keystone is too old07:47
Yanyanhuso you're using latest heat with old keystone :)07:48
YanyanhuI'm using old version heat and keystone, so error didn't happen I guess07:49
elynn_Can't reproduce your problem.. My local env is fine follow your reproduce steps...07:49
Qimingelynn_, which problem?07:50
elynn_I can delete heat stacks created by senlin07:50
Qimingthe senlin side trace is here:
elynn_on my local env.07:50
Qimingif you delete that stack from heat07:50
Qimingor you purge_delete it from heat07:51
Qimingthat stack won't exist in Heat anymore07:51
elynn_purge_delete is broken in heat for now....07:51
elynn_there's a patch to fix it.07:51
Qimingthen when senlin tries to do _check_action_complete07:51
Qimingthe first thing it calls is to do a stack_get()07:51
Qimingusing its recorded physical id07:52
Qimingelynn_, not interested in heat07:52
elynn_Better don't use purge_delete for now.07:52
elynn_I see the same error log in senlin yesterday07:52
elynn_that stack not found error.07:52
Qiminglet's focus for the moment on things we can do better from senlin side, alright?07:53
elynn_then I uninstall openstacksdk and install the latest version of openstacksdk07:53
Qiming20 minutes passed07:54
QimingI haven't explained the problem we need to solve07:54
Qimingsigh ...07:54
Qimingrefer to this trace:
Qimingoh, no, this one:
Qimingwhen senlin tries to determine if the delete action is complete07:55
Qimingit will do a stack_get07:55
Qimingwith the previous physical id it remembered (i.e. stack id)07:56
Qimingthis won't cause any problem in heat normally07:56
elynn_I see your point...07:56
Qimingbecause in heat's interface, when you are asking for a stack with a stack id07:56
elynn_The problem is here
Qimingheat gives you the complte record, even if the stack is already deleted07:56
Qimingit is doing show as if you have specified 'show_deleted=True'07:57
Qiminghowever, this is not reliable07:57
Qimingwe don't know if a stack has disappeared from heat database or not07:57
Qiminglet's suppose 'purge_delete' works today, you completely deleted that stack from heat db07:58
Qimingthe effect is the same07:58
Qimingwhen we do stack_get(), we get a ResourceNotFound Exception07:58
Qimingand when we do stack_get(), there is no way to specify ignore_missing07:59
Qimingwe have to capture and handle this exception07:59
Qimingfull trace is pasted here:
elynn_I open a bug and I will fix it.
openstackLaunchpad bug 1527112 in senlin "Can't delete node if stack doesn't exists in heat db" [Undecided,New]08:00
Yanyanhuor maybe we should use stack_find rather than stack_get to ignore notfound exception08:00
Yanyanhucurrently, stack_get will always raise exception is stack with given id does not exist08:01
elynn_That might be a good idea.08:01
Qimingexactly the problem I'm trying to explain08:01
Qimingthe same applies to nova server I think08:02
Qimingwe got to make our side as robust as possible08:02
elynn_seems we just need to fix the heat profile, don't need to fix nova server profile. It won't happen when using nova.08:10
*** Liuqing has joined #senlin08:10
Qimingline 353 add ignore_missing?08:11
YanyanhuQiming, ignore_missing is true by default in nova driver08:12
Yanyanhusenlin nova driver08:12
Qimingokay, server_delete has that default to True08:12
Qimingwell ...08:13
elynn_wait_for_server_delete will handle not found.08:13
Qimingthe heat stack impl is different08:13
Qimingthe exception was thrown in check_action_complete()08:13
Qimingplease help make them look similar08:14
elynn_OK, will add wait_for_stack_delete for heat profile.08:14
Yanyanhuhi, elynn_, I think you can try to add this interface into senlin heat driver08:15
elynn_Yanyanhu: I mean that ;)08:16
Yanyanhucurrently, there is no directly support for this interface in sdk08:16
Yanyanhumaybe in future we can directly rely on sdk to implement it08:16
elynn_Is openstacksdk support wait_for_delete for stack?08:17
*** lixinhui has quit IRC08:18
Yanyanhuelynn_, no I think08:21
elynn_Let me have a try~08:21
Yanyanhuif possible, maybe we should add it this support in sdk :)08:22
Yanyanhufor heat stack resource08:22
Qimingscroll to the bottom08:23
Qimingthere is wait_for_status and wait_for_delete08:23
Yanyanhuyep, but not sure it works08:23
Yanyanhuah, based on the implementation, it should08:24
Qimingkeystone v9 is so slow ....08:27
elynn_I think qiming is right, it works for me.08:27
Yanyanhuit can simplify the code of heat stack profile significantly08:28
elynn_I will add unittest later.08:28
openstackgerritEthan Lynn proposed openstack/senlin: Add wait_for_stack_delete to heat driver
elynn_You can have a try first.08:30
Yanyanhusure, thanks :)08:34
*** pratikmallya has quit IRC08:38
openstackgerritEthan Lynn proposed openstack/senlin: Add wait_for_stack_delete to heat driver
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
elynn_Don't know why that patch doesn't automatically link to launchpad.08:51
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
Qimingcould be because that bug is not reviewed?08:53
Yanyanhuwhich one?08:53
Qimingbug #152711208:54
openstackbug 1527112 in senlin "Can't delete node if stack doesn't exists in heat db" [High,Triaged] - Assigned to Ethan Lynn (ethanlynn)08:54
Yanyanhuthis usually didn't happen08:55
Yanyanhujust occasionally met it08:55
Qimingit happens to elynn several times08:55
*** lixinhui has joined #senlin08:56
Yanyanhuin most times, the link will be built successfully even when bug has not been reviewed I think08:56
*** xujun has joined #senlin08:56
Yanyanhumaybe elynn_ 's account issue LOL08:56
Yanyanhujust kidding :)08:56
Qimingis it that we have to assign a bug?08:56
elynn_Only if the patch is merged, the link will appear to launchpad08:56
Yanyanhuoh, that's possible08:56
Qimingthere must be some tricks08:56
Qiming" Only if the patch is merged, the link will appear to launchpad"08:57
Qimingthat would be bad08:57
*** xujun has joined #senlin08:57
YanyanhuI guess assignee is needed to build the link>08:57
Qimingwe have to make sure that bugs are explictly assigned08:57
Qimingor else, people may jump to the same thing simultaneously08:57
Yanyanhuyes, agree08:58
Qimingit is weird that our local functional tests are all okay, but still fails at gate!!!09:07
Qimingso... tortured09:07
Yanyanhuyes, I suffered from this issue in last two weeks...09:08
Yanyanhuafter that, I tested concurrency issue by creating and deleting multiple big clusters concurrently09:09
Yanyanhurather than running functional test09:09
Yanyanhusince error stopped happening in my local env when running functional test...09:09
YanyanhuI create/delete 5 clusters concurrently with each of them has 200 nodes09:10
Yanyanhuusing test cloud_backend09:10
QimingRan: 13 tests in 359.9469 sec.09:10
Qiming - Passed: 1309:10
Qiming - Skipped: 009:10
Qiming - Expected Fail: 009:10
Qiming - Unexpected Success: 009:10
Qiming - Failed: 009:10
Yanyanhuit just passed..09:11
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
elynn_Hi Yanyanhu, how to run functional tests in local env?09:16
elynn_just tox -e functional?09:16
Yanyanhuyes, before that, you have to run senlin-engine and senlin-api09:17
elynn_Do I need to upload any image?09:17
Yanyanhuno, but you need to set the following configure option09:17
Yanyanhucloud_backend = openstack_test09:17
Yanyanhuby default, it is set to 'openstack'09:17
elynn_in senlin.conf?09:18
elynn_What does it mean?09:18
elynn_What's this option for?09:18
Qimingelynn_, it is a fake layer09:25
Qiminga fake driver layer intercepting all senlin's interactions with other services09:26
Qimingthis is a pure functional test09:26
Qimingif we are using real drivers that allow senlin to talk real services, it would be integration test09:26
Yanyanhuhi, elynn_ , sorry just left for a while09:27
elynn_Wow, so we don't need other services when running functional tests?09:27
elynn_that's so cool!09:29
QimingYanyanhu is the hero09:30
elynn_InternalError: No Image found for cirros-0.3.2-x86_64-uec09:30
elynn_Seems I still need an older version cirros09:30
Qimingor you should change your profile09:30
Qiming0.3.4 is the default09:31
elynn_The gate still using cirros-0.3.2?09:31
Qimingnot sure, gate is using latest devstack I think09:33
*** pratikmallya has joined #senlin09:39
*** pratikmallya has quit IRC09:43
Yanyanhuhi, elynn_ , the image_get function is also faked in functional test09:45
Yanyanhuall needed fake driver are in senlin/test/functional/drivers directory09:46
Yanyanhuso the image name won't fail the test I think09:46
elynn_Yanyanhu: Then that's weird...09:46
Yanyanhulet me check it09:46
elynn_Oh, I know why09:47
elynn_I forgot to restart senlin-engine...09:47
*** openstackgerrit has quit IRC09:47
elynn_Bye openstackgerrit09:47
Yanyanhu... why it quit09:47
Yanyanhurestarting engine will load the fake driver09:48
*** openstackgerrit has joined #senlin09:48
Qiminglooks like a sport for him09:48
Yanyanhugood exercise :)09:49
openstackgerritLiuqing Jing proposed openstack/senlin-dashboard: Add policy detail page
*** Liuqing has quit IRC10:12
*** Liuqing has joined #senlin10:13
openstackgerritLiuqing Jing proposed openstack/senlin-dashboard: Use the default `url` tag instead
*** Qiming has quit IRC10:20
*** Liuqing has quit IRC10:25
*** Yanyanhu has quit IRC10:26
*** elynn_ has quit IRC10:31
*** Liuqing has joined #senlin10:50
*** Qiming has joined #senlin11:08
*** Liuqing has quit IRC11:14
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
*** pratikmallya has joined #senlin12:41
*** pratikmallya has quit IRC12:46
openstackgerritMerged openstack/senlin: Fix "" file's mode to 644
*** gongysh has quit IRC13:02
*** gongysh has joined #senlin13:02
*** elynn_ has joined #senlin13:14
openstackgerritMerged openstack/senlin-dashboard: Add policy detail page
openstackgerritMerged openstack/senlin-dashboard: Use the default `url` tag instead
*** elynn_ has quit IRC14:20
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem
*** pratikmallya has joined #senlin15:12
*** pratikmallya has quit IRC15:50
*** xujun has quit IRC16:08
*** xujun has joined #senlin16:08
*** Qiming has quit IRC16:14
*** pratikmallya has joined #senlin16:26
*** pratikma_ has joined #senlin16:27
*** pratikmallya has quit IRC16:30
-openstackstatus- NOTICE: Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating.16:43
*** bdrich has joined #senlin17:45
*** bdrich has left #senlin17:58
*** bdrich has joined #senlin17:59
*** bdrich has quit IRC18:09
*** bdrich has joined #senlin18:09
*** bdrich has quit IRC18:34
jdandreaShould there be an extra os.pardir here per the comment on line 25? /blob/master/bin/senlin-api#L2918:50
jdandreaWhoops -
*** pratikma_ is now known as pratikmallya20:35
*** bdrich has joined #senlin20:39
*** bdrich has quit IRC21:45
*** yuanying has joined #senlin23:12
*** Qiming has joined #senlin23:46
*** pratikmallya has quit IRC23:51

Generated by 2.14.0 by Marius Gedminas - find it at!