Tuesday, 2016-06-21

*** elynn has joined #senlin00:00
*** eldon_ has quit IRC00:19
*** elynn has quit IRC00:42
openstackgerritMerged openstack/senlin: Initialize docker driver  https://review.openstack.org/32269200:44
*** yanyanhu has joined #senlin01:53
*** elynn has joined #senlin01:55
*** elynn has quit IRC02:01
*** elynn has joined #senlin02:01
elynnmorning senlin02:12
*** zzxwill has joined #senlin02:16
*** yuanying has quit IRC02:51
*** eldon_ has joined #senlin02:53
*** yuanying has joined #senlin02:54
eldon_may I ask some question about rpc call?02:55
eldon_we have 3 servers installed senlin.02:55
eldon_each one has its own api+engine.02:55
yanyanhuhi, eldon_02:55
eldon_so we have 3 apis and 3 engiens.02:56
eldon_now, when we stop one of the three engine.02:56
eldon_it will have 3 apis and 2 engines.02:56
eldon_and when I use senlin node-list to show nodes.02:57
eldon_it will fail one of the three times.02:57
yanyanhuso you configure loadbalancer for senlin api?02:58
eldon_any one had idea about it?02:58
eldon_I have lb for api use ha-proxy02:58
eldon_rabbitmq for rpc02:58
yanyanhueldon_, you can configure more than one engine worker for each senlin service02:59
eldon_I don't knoe it's rabbit's problem or senlin's02:59
yanyanhubut for the case you described, we don't have good support now02:59
eldon_yes, I think that's the point02:59
eldon_but how?02:59
yanyanhueldon_, it's senlin's problem now03:00
yanyanhusince senlin-api are not aware of engine failure03:00
yanyanhuand thus will always try to send rpc calls to engine03:00
eldon_can senlin-api find other engine not in it's server automatically?03:00
yanyanhuno, that is infeasible03:01
eldon_or do you know how nova make it work?03:01
yanyanhunova can't handle that case as well as I know03:01
yanyanhuI mean nova-api in one deployment can't talk with nova-compute in another deployment03:02
yanyanhubut you can configure multiple engines at backend, and if api can be aware of engine failure, it can stop sending rpc request to failed engine03:02
yanyanhuyes, I guess so. Since the rpc msg is passed through queue03:03
yanyanhuand the queue server and topic are fixed in configuration03:03
eldon_so, I think you mean nova has its backend and nova-api has function to process engine failure?03:04
eldon_senlin-api doesn't have this function yet.03:04
yanyanhuyes, I guess so03:04
eldon_ok,  thank you:)03:05
yanyanhueldon_, I was not sure about it:)03:06
yanyanhuI guess nova api will stop creating VM in a  compute node after that compute node crashed03:06
yanyanhuwhat you said is a really important issue03:06
eldon_yes. nova knows which nova-cpt is down.03:06
yanyanhueldon_, so the same problem won't happen in nova :)03:07
yanyanhuunless no any nova-compute nova is available in all nova deployment03:07
eldon_I thought it was rabbitmq's job, now you remind me about nova-api's function.03:08
yanyanhufor senlin, it does happen since senlin-api doesn't know senlin-engine failure03:08
yanyanhueldon_, yes :)03:08
yanyanhunova supports that I thiknk03:08
eldon_So i should review some code about senlin-api and nova-api03:08
yanyanhuI think this is an important feature we want to support03:08
yanyanhuI believe there has been some basic support for this feature. Engine failure detection is now supported03:09
yanyanhujust senlin-api doesn't know the result03:09
yanyanhuwill try to figure out how to implement it03:09
yanyanhuafter that, senlin-api won't send rpc calls to those failed engine03:10
eldon_yes, that's interesting :)03:10
yanyanhuwill file a bp for it and think how to support this feature :)03:11
yanyanhuthanks, eldon_ :)03:13
yanyanhuoh, BTW, do you guys have any new progress for senlin rally test03:14
yanyanhuor any new requirement?03:14
eldon_we are using our rally tasks doing some tests.03:15
yanyanhumore plugins for senlin has been added03:15
yanyanhuand also test example03:15
eldon_We haven't use your tasks as we uses codes from 2015-12-1603:15
yanyanhuI see. but the I think the plugin works as well for that version :)03:16
yanyanhusince the interface of senlinclient didn't change I think03:16
yanyanhuyou can have a try and just ping me if you meet any problem :)03:16
eldon_ok, we will add them in our docker image~03:17
*** eldon_ has quit IRC03:19
*** eldon_ has joined #senlin03:37
eldon_I think we should add an interface.03:37
eldon_ like senlin service-list03:37
yanyanhueldon_, we have one :)03:46
yanyanhuoh, let me check03:47
yanyanhuI know we have DB support for it03:47
yanyanhulet me check whether we have user interface for it03:47
yanyanhuelynn, around?03:47
yanyanhuhi, do we have user interface for checking senlin-engine service status?03:48
elynnsenlin-manage service list03:48
yanyanhunice :)03:48
yanyanhuhi, eldon_ you can use that cmd to check engine service status03:48
yanyanhuif using latest senlin code03:49
eldon_that is dynamic?03:49
*** zzxwill has quit IRC03:49
yanyanhuI think so03:49
eldon_if we have this interface, it will not hard to solve that problem I said early.03:50
yanyanhueldon_, sure03:50
yanyanhutechnically, there is no critical obstacle03:50
yanyanhujust need to figure out what is the best way to support it03:50
*** eldon_ has quit IRC03:54
*** yuanying has quit IRC04:02
*** yuanying has joined #senlin04:03
*** elynn has quit IRC04:34
*** elynn has joined #senlin04:53
*** elynn has quit IRC04:58
*** elynn has joined #senlin04:58
openstackgerritYanyan Hu proposed openstack/senlin: Releasenote for tempest API test  https://review.openstack.org/33193905:05
*** zzxwill has joined #senlin05:07
openstackgerritxu-haiwei proposed openstack/senlin: Add Container create/delete driver  https://review.openstack.org/33194705:35
*** eldon_ has joined #senlin05:36
openstackgerritYanyan Hu proposed openstack/senlin: Clean action records after deleting cluster/node  https://review.openstack.org/32933805:40
openstackgerritMerged openstack/senlin-dashboard: modify help_text message  https://review.openstack.org/32956005:57
openstackgerritMerged openstack/senlin-dashboard: Delete duplicate period  https://review.openstack.org/32973805:57
*** eldon_ has quit IRC06:05
openstackgerritxu-haiwei proposed openstack/senlin: Add Container create/delete driver  https://review.openstack.org/33194706:21
*** eldon_ has joined #senlin06:25
*** eldon_ has quit IRC06:31
openstackgerritYanyan Hu proposed openstack/senlin: Fix oslo.config.opts in setup.cfg  https://review.openstack.org/33197807:09
openstackgerritOpenStack Proposal Bot proposed openstack/senlin-dashboard: Imported Translations from Zanata  https://review.openstack.org/33199107:31
yanyanhuhi, elynn , around?07:40
elynnhi yanyanhu ?07:41
yanyanhuhi, I found senlin-manage service list doesn't work in my local env07:41
yanyanhuI got the following error: NoSuchOptError: no such option in group DEFAULT: periodic_interval07:41
elynnis it broken?07:41
yanyanhucan you have a try in your env?07:41
yanyanhuguess so07:41
elynnlet me07:41
yanyanhulooks like CONF is not loaded correctly07:42
elynnlet me find out why it doesn't work07:42
yanyanhuI guess we may need to add "cfg.CONF.import_opt('periodic_interval', 'senlin.common.config')" before refering to CONF.periodic_interval07:43
yanyanhubut I don't know why...07:43
elynnLet's have try07:43
elynnYou are right!07:45
elynnNot sure why it's needed now...07:45
yanyanhuit worked before I think07:46
yanyanhuI guess something changed in oslo.config causes this problem07:47
yanyanhulet me try to downgrade oslo.config version to make a try07:48
openstackgerritEthan Lynn proposed openstack/senlin: Fix NoSuchOptError when using senlin-manage  https://review.openstack.org/33199807:50
yanyanhudidn't work as well after down grading to 3.8.0...07:50
yanyanhuhi, elynn , I think you can just that line before L57, since only format_service needs it07:51
yanyanhudb_sync doesn't need that option07:52
elynnMaybe some codes change in senlin make this potential error appear07:52
yanyanhuyes, that's possible as well07:52
elynnyanyanhu, changing :)07:52
yanyanhuvery weird...07:52
openstackgerritEthan Lynn proposed openstack/senlin: Fix NoSuchOptError when using senlin-manage service list  https://review.openstack.org/33199807:55
yanyanhuhi, elynn, you just updated commit msg in patchset2...08:01
openstackgerritEthan Lynn proposed openstack/senlin: Fix NoSuchOptError when using senlin-manage service list  https://review.openstack.org/33199808:05
yanyanhuelynn, hmm, I noticed either import_opt or register_opt must be invoked before options in CONF can be visible08:05
yanyanhuonce an option is registered or imported, all other options become available08:06
elynnthat's odd08:06
yanyanhuI guess the reason manage.py worked before is some embedded importing includes the invoking of register_opt or import_opt08:06
elynnI guess so08:06
yanyanhuor oslo.conf is designed like that and we didn't noticed that before :)08:07
elynnsome other project08:07
elynnthey don't directly import oslo.config08:07
elynnThey use a wrapper to register their configs08:07
yanyanhuyou mean?08:08
yanyanhuany example?08:08
*** eldon_ has joined #senlin08:08
elynnSo at other places, keystone directly use CONF in common/config.py.08:09
elynna bit difference from us.08:09
elynnBut we are the same as heat.08:10
yanyanhuso all other modules will refer to this 'CONF'?08:11
yanyanhurather than create a global CONF08:11
eldon_can we deploy senlin-api and senlin-engine in different hosts?08:11
yanyanhua new CONF based on global configuration08:11
yanyanhueldon_, we can I think08:11
elynnIf I remember correctly08:11
yanyanhueldon_, if the message queue is configured correctly08:12
yanyanhulet me check it08:12
eldon_how it works?08:12
eldon_how can i configure it ?08:12
elynnhaven't go through these codes recent release.08:12
yanyanhueldon_, will confirm it and give you reply :)08:12
yanyanhuelynn, me neither...08:12
yanyanhuhi, eldon_ , you just need to configure "rabbit_hosts" correctly in you env to ensure both senlin-api and senlin-engine can reach it08:14
eldon_i see~08:14
yanyanhuand also rabbit_userid and pass08:14
eldon_and also this means, one api refers to one engine or multi apis refers to one engine08:17
eldon_but can't one api refers to multi engines08:17
eldon_def __init__(self):08:17
eldon_ 29         self._client = messaging.get_rpc_client(08:17
eldon_ 30             topic=consts.ENGINE_TOPIC,08:17
eldon_ 31             server=cfg.CONF.host,08:17
eldon_ 32             version=self.BASE_RPC_API_VERSION)08:17
eldon_server is fixed.08:18
eldon_rpc_opts = [08:18
eldon_ 94     cfg.StrOpt('host',08:18
eldon_ 95                default=socket.gethostname(),08:18
eldon_ 96                help=_('Name of the engine node. This can be an opaque '08:18
eldon_ 97                       'identifier. It is not necessarily a hostname, FQDN, '08:18
eldon_ 98                       'or IP address.'))]08:18
yanyanhuhi, eldon_, multiple engines will listen to the same server.topic I think08:21
yanyanhuwhich is the place API submit rpc call request08:21
yanyanhuonly one engine can get the chance to receive that request and handle it08:22
eldon_yanyanhu,  which website can I paste code?08:23
yanyanhuhmm, if so, the conclusion about engine failure detection in senlin-api we made this morning might be incorrect08:23
yanyanhuyou can paste it here08:24
eldon_I compared senlin with heat08:27
eldon_the init is different08:27
eldon_we fixed host in senlin rpc client, but heat doesn't fix it.08:28
yanyanhuyou mean?08:28
eldon_I guess, if we fix the host, it will not call function from that host.08:30
eldon_the structure is host/topic/function.08:31
yanyanhuhi, eldon_ ,that option defines the host of rabbit_mq08:31
yanyanhubut not the host of senlin engine08:31
yanyanhuyou can find an option named exactly as "host" in senlin.conf08:32
eldon_and if host is not defined, it will call function from any hosts where topic is same08:32
yanyanhuthat option defines the 'host' of senlin engine08:32
eldon_yes ,i get it.08:32
yanyanhuactually that is just a symbol, it doesn't have to be a ip address08:32
yanyanhueldon_, yes08:32
yanyanhuyou're right08:32
yanyanhuactually naming it to "server" could make it easier to understand :)08:33
eldon_so here in rpc/client.py, it is the engine host08:34
yanyanhuyes, the engine host whose default value is node108:34
yanyanhuI think08:34
eldon_we fix it to localhost, if we didn't configure host in senlin.conf08:34
eldon_that is not appropriate, right?08:35
yanyanhuyou mean localhost?08:35
yanyanhulocalhost is ok08:35
*** eldon_ has quit IRC08:35
yanyanhuit doesn't matter08:35
*** eldon_ has joined #senlin08:36
yanyanhuhi, eldon_, localhost is ok08:36
yanyanhuyou can even name it to 'abc'08:36
eldon_but if we only deploy api in host1, and not deploy engine in host1, it will not work.08:37
eldon_so, I suggest not pass server to get_rpc_client function08:37
eldon_just like heat.08:38
yanyanhuhmm, are the rabbit_hosts option in api node and engine node the same?08:38
eldon_yes, same.08:38
yanyanhueldon_, if so, we can't differentiate multiple senlin engines if they belongs to different deployments08:39
yanyanhurpc server of multiple senlin engines08:39
yanyanhuall of them will have the same topic08:39
yanyanhuwithout unique server being specified08:39
eldon_ok, maybe I should try first to test something:)08:40
yanyanhulet me double check the code of oslo.messging08:40
yanyanhueldon_, yes, you can make a test to observe the behavior of engine and api :)08:41
eldon_ok, thanks a lot08:41
yanyanhuno problem08:42
yanyanhuBTW, eldon_ ,  you can try to enable multiple engine worker to see whether it can meet your requirement for handling engine failure when enabling API loadbalancing08:45
yanyanhuif an engine dies completely and doesn't receive any rpc calls, I think it won't cause rpc request hanging08:46
eldon_now, it returns time out error.08:50
yanyanhuso there are multiple senlin-apis being loadbalanced08:51
yanyanhuand each of them has multiple senlin-engine at backend?08:51
yanyanhuoh, you are doing the rabitmq test08:51
eldon_I guess it's caused by the api.py:get_rpc_client function I said early.08:51
yanyanhuhmm, let me have a try08:52
yanyanhuneed to setup senlin in the second host, need several minutes08:56
yanyanhuhi, eldon_ is there any error log reported during engine and api started up09:00
eldon_no, they can start up successfully.09:00
yanyanhuok, that's means rabbit connection is correct09:01
yanyanhuI installed senlin-api and engine in the same host, but configured rabbit_host to another host09:04
yanyanhuit works well09:04
yanyanhuthe installation is slow... for poor network09:04
*** elynn has quit IRC09:05
yanyanhuhi, eldon_ so I guess deploying senlin-api and senlin-engine into different hosts will work as well09:07
*** eldon_ has quit IRC09:07
*** openstackgerrit has quit IRC09:18
*** openstackgerrit has joined #senlin09:19
*** ChanServ sets mode: +v openstackgerrit09:19
*** yanyanhu has quit IRC10:30
*** eldon_ has joined #senlin12:02
*** zzxwill_ has joined #senlin12:22
*** zzxwill has quit IRC12:23
*** yanyanhu has joined #senlin12:53
yanyanhuhi, eldon_, around?12:57
yanyanhufor coming meeting, I added a topic about deploying senlin-api and senlin-engine in different hosts12:58
yanyanhuhope can make a quick discussion about it with you12:58
yanyanhuhi, guys, meeting will start in 2 minutes12:59
*** elynn has joined #senlin12:59
yanyanhuin meeting channel, Qiming can not join it for family reason, so I will hold this meeting on behalf12:59
yanyanhuhi, elynn, meeting will start in openstack-meeting channel13:00
elynnwill join13:00
elynnJust in time :)13:01
yanyanhuhi, lixinhui , around?13:08
*** xuhaiwei has quit IRC13:13
elynneldon_, we are holding senlin meeting at channel #openstack-meeting13:14
elynnwelcome to join13:14
elynnyou can share us some work you do recently.13:14
eldon_I am very glad to have this chance.13:15
yanyanhuhi, eldon_ , plz join openstack-meeting channel13:17
yanyanhuwe can make a discussion about deploying senlin-engine and api individually there13:17
*** lixinhui_ has joined #senlin13:21
*** eldon_ has left #senlin13:24
*** eldon_ has joined #senlin13:24
*** lixinhui_ has quit IRC13:41
openstackgerritMerged openstack/senlin: Fix NoSuchOptError when using senlin-manage service list  https://review.openstack.org/33199813:42
*** openstackgerrit has quit IRC13:48
*** openstackgerrit has joined #senlin13:49
*** ChanServ sets mode: +v openstackgerrit13:49
*** yanyanhu has quit IRC14:05
*** eldon_ has quit IRC14:08
*** zzxwill_ has quit IRC14:35
*** zzxwill_ has joined #senlin14:36
*** elynn has quit IRC14:37
-openstackstatus- NOTICE: Gerrit is being restarted now to apply an emergency security-related configuration change16:06
*** zzxwill_ has quit IRC16:14
openstackgerritOpenStack Proposal Bot proposed openstack/python-senlinclient: Updated from global requirements  https://review.openstack.org/33237918:05
openstackgerritOpenStack Proposal Bot proposed openstack/senlin: Updated from global requirements  https://review.openstack.org/33239018:06
openstackgerritOpenStack Proposal Bot proposed openstack/senlin-dashboard: Updated from global requirements  https://review.openstack.org/33239118:06
*** Drago has joined #senlin18:25
*** Drago has quit IRC18:26
*** Drago has joined #senlin18:26
*** jdandrea_ is now known as jdandrea18:27
*** Drago has quit IRC23:23
*** Drago has joined #senlin23:23
*** Drago has quit IRC23:28

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!