Monday, 2017-01-16

*** elynn has joined #senlin01:26
*** yanyanhu has joined #senlin01:34
openstackgerritXinhui Li proposed openstack/senlin: Add mistral driver into Senlin
*** chohoor has joined #senlin01:40
*** root has joined #senlin01:44
*** root is now known as Guest4661401:44
Guest46614nick XueFeng01:48
*** Guest46614 is now known as XueFeng01:48
yanyanhuhi, elynn, around?01:55
openstackgerritRUIJIE YUAN proposed openstack/senlin: improve tempest test for action list
ruijiehi yanyanhu, can you help me review this patch when you have time. To make sure this is what we want for tempest test. :)02:22
yanyanhuruijie, sure, will check it :)02:23
ruijiethanks yanyanhu :)02:23
yanyanhumy pleasure02:23
yanyanhuhi, ruijie, just took a look, it is exactly what we want :)02:25
yanyanhuthanks a lot for working on it02:25
ruijiegreat! will try to finish it before release-3 :)02:26
yanyanhuruijie, thanks! I think there could be a big effort to cover all API test scenarios, so please take your time :)02:27
elynnHi yanyanhu02:27
yanyanhuhi, elynn02:28
elynnSaw your emals02:28
yanyanhuyep, that's what I was about to ask :)02:28
elynnAs Qiming said, the referencing part would be difficult to handle.02:30
yanyanhuelynn, yes it is I think02:31
elynnIf the user_data is the only part needs to be handled, then it's easy. But for future support for NFV, we might do some simple orchestration thing in senlin02:32
yanyanhuyes. for user_data, it won't be difficult. Basically a string replacement I think. But the attr reference between different resources is much more complicated02:33
yanyanhuit also introduces implicit dependency02:33
yanyanhumay need a generic way to handle this kind of depedency02:34
yanyanhuand decide how to build the workflow02:34
yanyanhubut we can start from a simple one I feel02:35
elynnyes, if we introduce attribute reference, there might be some tricky code in do_create.02:35
yanyanhue.g. fixed dependency and creating sequence?02:35
yanyanhuelynn, right02:35
elynnyanyanhu, you mean start from a simple reference only for user_data?02:36
yanyanhuin our case, we always know that the port is depends on sec-group02:36
yanyanhuand server always depends on port02:36
elynnFor this particular case, I plan to do so, create security_groups first, then create ports, then create user_data.02:37
yanyanhuelynn, no, I mean we use fixed dependent relationship when handling different resources like port, sec-group, or even volume02:37
yanyanhuthat can be a simple solution02:37
yanyanhualthough we may need more complicated/generic solution later02:37
elynnyes, and place it at contrib/ folder first.02:37
elynnAnd also another problem,02:38
elynnWe might need cluster to pass its metadata 'inputs' to node when it creates new nodes.02:38
XueFenghttps://etherpad.openstack.qorg/p/scaling-to-existing-nodes, this link can't open02:39
yanyanhuXueFeng, oh, let me check02:41
yanyanhuXueFeng, should be
yanyanhuthere is a "q" letter in the url which was added accidentally :)02:42
yanyanhuelynn, yes. maybe we can always include the input field of cluster's metadata into node's metadata?02:43
elynnyanyanhu, I think so, but that will change the senlin's code.02:45
elynnWe can't put all the code in contrib/ folder.02:45
yanyanhuyou mean the profile's do_create logic?02:45
yanyanhuI see.02:46
yanyanhuif so, we just implement this logic in this new profile type?02:46
elynnI mean the pass through the cluster's metadata 'inputs' to node02:46
yanyanhulet me see whether it is feasible02:46
yanyanhuit is feasible. Just the code will be ugly if we don't change the code of master branch...02:47
yanyanhucurrently, profile.create_object will be invoked with node obj as input parameter02:48
yanyanhuand also the context02:48
elynnyanyanhu, so I can try to get the cluster's metadata02:49
yanyanhuoh, context is not used when do_create is invoked02:49
yanyanhuelynn, yes, you can get it. Just it looks like a hack :)02:49
yanyanhuunless we add cluster's metadata to node's metadata02:50
elynnI would have to say there will be much ugly codes in this profile..02:50
yanyanhuby default using a generic way02:50
yanyanhuelynn, yes, it will be...02:50
Qiminghow about we add a 'config' property to a cluster?02:50
yanyanhubut since that could be a common requirement to access cluster's metadata in node operation02:50
yanyanhumaybe we add this logic?02:50
elynnQiming, property?02:51
QimingI'm a little bit concerned that directly copying metadata could inflate the dict quickly02:51
Qiminga property02:51
Qimingit can be a cluster-wide shared repo02:51
yanyanhuQiming, so what information will be wrapped in this property?02:51
yanyanhuummm, I see02:52
QimingI was thinking about this in another thread02:52
openstackgerritXinhui Li proposed openstack/senlin: Add mistral driver into Senlin
elynnit's in a dict schema?02:52
XueFengHi, YanYan.Now it's ok02:52
yanyanhucurrently, we are trying to define all cluster-wide control information in policies02:52
Qimingfor example, users have a pool of volumes to be referenced when creating nodes02:53
Qimingelynn, yes02:53
yanyanhubut there are really some information not suitable for storing in policy02:53
Qiminga cluster-wide configuration for network name, security group etc02:53
yanyanhuQiming, maybe a volume/storage policy :P02:54
yanyanhujust a kidding02:54
QimingI didn't invent the idea02:54
Qimingthe idea was from magnum02:55
Qimingalso from genesis02:55
elynnThe pool of volumes is a cluster or a policy?02:55
Qimingin their design, a cluster may have quite some cluster-wide properties to be shared among nodes02:55
yanyanhuQiming, I think cluster config is a good idea.02:56
elynnThat's a great idea02:56
Qimingif you have a {'volume_pool': pool_id} in cluster.config, when creating a node, we can check if we have that value02:56
yanyanhuJust we may need to distinguish the usage of cluster config and policy for some special cases02:56
elynnThen we can pull information from this property.02:57
yanyanhue.g. we can store available host list in both placement policy or cluster config02:57
Qimingwe will create new volumes if needed, but if the cluster has a config, we use it02:57
Qimingthe thing I am not sure is how to manage such a wild config property02:57
Qimingevery key we add into that property becomes a standard02:58
Qimingwhich is not supposed to be renamed, deleted randomly02:58
yanyanhuQiming, you mean the config property is not allowed to update?02:59
Qimingvalue can be updated for sure02:59
Qimingbut we have to be very careful in designing the key names (and name spaces)03:00
yanyanhuQiming, or we let user to decide the key?03:00
yanyanhubeside those ones for built-in usage?03:00
Qimingyes, then there are two namespaces at least: system, user03:00
yanyanhue.g. those keys used by built-in policies profiles03:00
elynnWhat are the key names will be there?03:01
yanyanhuQiming, yes03:01
Qimingelynn, that is what we are discussing03:01
yanyanhuor senlin: username03:01
elynnWhat's the difference between system and user?03:01
yanyanhuand the namespace can be used to match03:01
yanyanhuwill join a meeting, come back later03:01
elynnfor user namespace, it's updated allowed?03:01
Qiminggood question03:02
Qimingin other words, I don't have an answer, :D03:02
yanyanhuI think for user namespace, users can create/update any keys/values03:02
yanyanhuaccording to their need03:03
yanyanhubut for system ones, only value can be updated03:03
Qimingwhere will be place the update logic for user defined-keys?03:03
yanyanhucluster update I guess?03:03
Qimingupdate to where?03:03
yanyanhuit will be a sync op03:03
Qimingjust upate the cluster.config ?03:04
yanyanhuto config property?03:04
yanyanhuQiming, yes, I guess so03:04
Qimingthen node cannot reference them?03:04
yanyanhueach time node operations will to use cluster metadata, it is required to query the latest one03:05
yanyanhufrom db03:05
Qimingcluster config == node metadata?03:05
Qiminglemme show you an example structure from genesis03:05
Qiming110 type AsgUserdata struct {03:05
Qiming111     RootPass   string03:05
Qiming112     SSHKey     string03:05
Qiming113     DisgoCfg   string03:05
Qiming114     PassExpire string03:05
Qiming115 }03:05
yanyanhuQiming, no, I mean each time when some information stored in cluster config is required to finish node operations03:06
openstackgerritRUIJIE YUAN proposed openstack/senlin: improve tempest api for action show
Qiminghow do we know that?03:06
yanyanhuif we have cluster config, we don't use node metadata03:06
Qimingit is user defined, the keys and values are all provided by users03:06
yanyanhuQiming, yes, so for user define ones, built-in policy/profile won't be aware of them03:07
elynnWe could let profile retrieve the cluster.config itself? When needed03:08
Qimingthe reference and usage of "system" config is more clear to me, but for "user" config, I'm not sure how we will handle them03:09
yanyanhuhmm, let me see. If build-in policies/profiles are not aware of user defined config, it could be inadequate...03:09
Qimingfor "system" config, we add a key which is named carefully, and we will check it in profile operations and do something accordingly03:09
elynnI still don't know why build-in policies/profiles are not aware of user defined config?03:09
yanyanhudo need more think here...03:09
Qimingbuilt in policies can access them for sure03:10
yanyanhuor maybe we should ask, what is the case user want to define customized config?03:10
Qimingjust you are getting a collection of key-value pairs, provided by users03:11
Qimingand ... you don't know how to handle them ..03:11
yanyanhuQiming, yes, just they don't know that they means maybe03:11
Qimingright, I cannot think of a user case for this either03:11
Qimingit looks to me more like a metadata03:11
yanyanhuQiming, customized policy/profile :)03:11
Qiminge.g. cluster.metadata.department = "finance"03:12
Qimingokay, that makes sense03:12
Qimingwe are adding kind of labels to clusters then03:13
Qimingunless user provide their own profile/policy types, senlin won't touch them03:13
yanyanhuif those information are only for tag, they are more like metadata rather then config I think03:13
yanyanhuagree with u03:13
Qimingsenlin only provides basic CRUD operations to them03:13
Qimingalong with cluster operations03:13
elynnThere are two kinds of metadata here, one for user query, and one for senile internal used I think03:14
yanyanhuelynn, so we either add a config property for cluster, or we add a config field in cluster metadata03:16
elynnyanyanhu, yes, add a config field in metadata will not need to change current senlin codes until we think it thoroughly.03:17
yanyanhuleave for while, back soon03:19
Qiminga special key is needed then03:19
Qimingwe are not supposed to copy the whole metadata down to node level, :)03:20
elynnQiming, if profile can access cluster's metadata, then we don't need to copy the metadata to node.03:20
Qimingyes, we can03:21
*** Jeffrey4l_ has quit IRC03:21
*** Jeffrey4l has joined #senlin03:21
Qimingfrom the node object you get the cluster, then the cluster metadata03:21
elynnabout the security_groups part, I think we might need a policy to handle it instead create a new one in profile.03:22
yanyanhui'm back03:24
Qimingcurrently we support name only03:24
Qimingand we are treating that as the identity of a security group to be used03:24
Qimingand ... it is a list ...03:25
yanyanhuQiming, yes, sec-group takes effect in cluster-scope03:25
yanyanhurather than node03:25
Qimingsome intelligence we can build is to check if the specified security group can be found03:25
*** chohoor has quit IRC03:26
yanyanhuso handling it in policy sounds more reasonable03:26
Qimingif not, whether there are rules provided so that we can create one03:26
yanyanhubut we also need to consider the case that different nodes want different sec rules03:26
yanyanhumaybe based on roles03:26
Qiminghow about we handle these threads one by one?03:27
QimingI cannot think clearly if roles are brought into this picture at this stage ...03:28
yanyanhusure :)03:28
Qimingethan and I were discussing the creation of security groups03:29
yanyanhuso, get back on sec-group03:29
Qimingif we are revising nova server profile, within or outside the current code base, how we are gonna do that03:29
elynnyes back to sg..03:29
yanyanhuif we create it in profile, that means the creation of first node will include the creation of sec-group?03:30
yanyanhuand that sec-group will be logged into cluster's metadata and reused by the following created nodes?03:30
elynnQiming, so you mean I can check the sg name in profile and see if it's exists and then check if rules are added, each time node creating?03:30
Qimingelynn, very rough idea03:31
Qimingbut it will bring in a lot of problems I think03:31
Qimingtoo naive03:31
elynnI'm considering using a policy to create sgs and store the ids in cluster metadata.03:31
Qimingdifferent nodes will try create the same SG03:32
elynnThen profile creation can reference it when creating new nodes.03:32
Qimingusing policy sounds more feasible and practical03:32
*** wllabs has joined #senlin03:32
Qimingthe problem is the process of creating cluster, creating policy, attach policy ... is boring for such a feature03:33
elynnQiming, yes, that's the side effect...03:33
elynnWe can improve the current processes, When creating policies along with cluster.03:35
QimingI'm thinking about such a structure03:37
Qiming  properties:03:37
Qiming    profile_id: <blah>03:37
Qiming    metadata:03:37
Qiming      department: finance03:37
Qiming      author: zhang_san03:37
Qiming      ...03:37
Qiming      __builtin__:                # this can be promoted to a config property03:37
Qiming        security_groups:03:37
Qiming          - name: allow_ping03:37
Qiming            auto_create: true03:38
Qiming            rules:03:38
Qiming              - direction: ingress03:38
Qiming                protocol: ICMP03:38
Qiming                ...03:38
Qiming          - name: allow_ssh03:38
Qiming            auto_create: false03:38
elynnWhen does profile handle these metadata? during cluster creation?03:39
Qimingyup, if we go down this road, it will become a cluster asset03:39
elynnauto_create here means?03:40
elynnjust a reference?03:40
Qimingthere are two security groups defined03:40
Qimingfor allow_ping, we will check if an existing SG is there named 'allow_ping'03:41
Qimingwe do nothing if so03:41
Qimingthen if 'allow_ping' is not found, and 'auto_create' is true, we check if rules are provided to create one03:41
Qimingfor 'allow_ssh', auto_create is false, if we cannot find such a SG, and given that 'auto_create' is false, we cannot do anything, just error03:42
elynnSounds good, but what if we have others to create, like lb, should we go this road?03:44
yanyanhuumm, currently, sec-group is not enforced to name unique in nova03:45
yanyanhuso there could be situation that two sec-groups belong to the same user have the same name03:45
Qimingsecurity groups are neutron objects?03:46
yanyanhuyes, it is provided by neutron actually03:46
yanyanhualso not name-unique enforced in neutron03:47
Qimingif there is name unique issue ...03:47
Qimingwe can change the 'name' in the snippet above to 'id'03:48
elynnHow do we reference 'id' if it's not created yet?03:48
yanyanhuumm, that is a problem.03:50
yanyanhuand since the node creation is concurrent, so avoid need to consider concurrency issue...03:51
yanyanhue.g. two nodes' creation trigger the creation of two sec-groups...03:52
yanyanhuor we stored the creation result, e.g. sec-group id to cluster metadata?03:53
Qimingright, good question03:53
Qimingcluster object is always created before node objects03:54
Qimingso we may need to check and create SGs before creating nodes and delete them if necessary03:54
yanyanhuyes, delete logic is also needed03:54
Qimingcreated SGs will be stored somewhere in the cluster properties03:55
Qimingthere are other complexities as well, there are things you can update, there are others you cannot update03:55
Qimingthat is why I was suggesting a config property03:55
Qimingand even a separate group of APIs for it03:55
Qimingin the long run03:55
Qimingleaving for lunch03:56
Qimingstarving ...03:56
yanyanhuok, lets keep discussing later03:56
elynnsee U afternoon03:56
*** elynn has quit IRC04:11
wllabsscaling policy has a propoerties number is 204:21
wllabscreate a receiver-create -P count=1004:21
wllabsif triggering reciver, num of node what the cluster scaling node04:22
wllabsif triggering reciver, How many nodes will be scaling04:30
wllabsQiming, I set scaling out num is 3, but senlin cluster-scale-out mycluster only add one node04:46
*** fabian4 has quit IRC05:07
*** fabian4 has joined #senlin05:08
*** fabian4 has quit IRC05:09
*** fabian4 has joined #senlin05:10
ruijieit depends the max_size of your cluster05:24
*** elynn has joined #senlin05:24
ruijiee.g current you have 2 nodes in your cluster. and the max_size of your cluster is 3. And if you set the parameter "strict" to False when creating receiver, it will only create ONE node even if you set the number greater than 105:28
ruijieAnd for the receiver(scale_out) and policy, if you set the number when you create the receiver, the cluster_scale_out will use the number but not the number in your policy.05:31
wllabsok, thank you05:36
ruijieur welcome:)05:43
*** chohoor has joined #senlin05:54
wllabsdoes senlin can direct handle ceilomete?05:54
wllabs  scale_out_alarm:05:54
wllabs    type: OS::Ceilometer::Alarm05:54
wllabs    properties:05:54
wllabs      description: trigger when bandwidth insufficient05:54
wllabs      meter_name:
wllabs      statistic: avg05:54
wllabs      period: 6005:54
wllabs      evaluation_periods: 105:54
wllabs      threshold: 2800005:54
wllabs      repeat_actions: True05:54
wllabs      alarm_actions:05:54
wllabs        - {get_attr: [receiver_scale_out, channel, alarm_url]}05:54
wllabs      comparison_operator: ge05:55
wllabs      query:05:55
wllabs        metadata.user_metadata.cluster_id: {get_resource: cluster}05:55
yanyanhuwllabs, we supported ceilometer resource(e.g. alarm) creating in senlin before. But that functionality was finally deprecated for we think the building of alarm is actually out of senlin's scope06:00
yanyanhusenlin will handle the bottom half of autoscaling, including scale up/down the cluster size06:00
yanyanhubut we rely on external monitoring system to trigger the scaling06:01
wllabsthank you06:01
yanyanhuhi, will leave for a meeting, back later06:01
wllabsTake me on06:01
*** shu-mutou has joined #senlin06:10
*** Drago has quit IRC06:51
openstackgerritRUIJIE YUAN proposed openstack/senlin: revise tempest api test for cluster policy
openstackgerritShu Muto proposed openstack/senlin-dashboard: Address receiver panel to recent Horizon framework
openstackgerritShu Muto proposed openstack/senlin-dashboard: Add create action into Receiver panel
openstackgerritCao Xuan Hoang proposed openstack/senlin: Replaced e.message with str(e)
*** zhurong has joined #senlin08:05
*** wllabs_ has joined #senlin08:48
*** wllabs has quit IRC08:49
*** wllabs_ is now known as wllabs08:50
*** shu-mutou is now known as shu-mutou-AWAY09:14
yanyanhuhi, guys, just a reminding, please leave your +1 on those todos in the following etherpad, lets get through all those items in tomorrow's meeting, thanks09:26
openstackgerritRUIJIE YUAN proposed openstack/senlin: revise tempest api test for cluster-1
openstackgerritRUIJIE YUAN proposed openstack/senlin: revise tempest api test for cluster policy
openstackgerritRUIJIE YUAN proposed openstack/senlin: improve tempest test for action list
*** zhurong has quit IRC09:37
openstackgerritRUIJIE YUAN proposed openstack/senlin: improve tempest api for action show
*** XueFeng has quit IRC09:59
*** zhurong has joined #senlin10:08
ruijiejust be told tomorrow is out company's annual meeting ..10:15
ruijiemay miss the IRC meeting tomorrow.10:15
*** yanyanhu has quit IRC10:34
openstackgerritXueFeng Liu proposed openstack/senlin: Fix _add_listener cannot get project
*** chohoor has quit IRC10:58
*** root____4 has joined #senlin11:14
root____4QiMing, about ironic. It's the same with QiMing, about ironic. It's the same with novanova11:29
*** root____4 is now known as XueFeng11:29
XueFengQiMing, about ironic. We can think it's the same with nova11:30
openstackgerritmiaohb proposed openstack/senlin: Add db purge in senlin manage
*** wllabs_ has joined #senlin12:00
*** wllabs has quit IRC12:01
*** wllabs_ is now known as wllabs12:01
*** blinky_ghost has joined #senlin12:08
blinky_ghostHi all, I'm been looking at senlin can anyone explain me if it supports auto-healing, something like I have a cluster with 2 nodes and if one the nodes is deleted or in Error state, it creates another automatically? thanks12:13
XueFenghi, blinky. Senlin can do this.12:20
XueFengYou can create a policy which you want to do in recovering, and attach policy to cluster.12:22
blinky_ghostXueFeng: thanks, something like this: health_policy_poll.yaml  from examples12:24
blinky_ghostXueFeng: I applied this from file health_policy_poll.yaml with has policy  NODE_STATUS_POLLING. For testing I deleted one of the nodes of the cluster but senlin didn't rebuild it. What I'm missing?12:41
*** zhurong has quit IRC13:03
XueFenghi, you mean you delete node in nove layer?13:06
XueFengThis condition is a bug, need to fix.13:07
XueFengThis line we need add extra conditional13:11
XueFengdo recover will be a complicated actions, we will continue to perfer the function. Also we will support workflow to control action flow.13:18
openstackgerritMerged openstack/senlin-dashboard: Address receiver panel to recent Horizon framework
openstackgerritMerged openstack/senlin: Replaced e.message with str(e)
openstackgerritMerged openstack/senlin: improve tempest api for action show
*** blinky_ghost has quit IRC14:32
*** blinky_ghost has joined #senlin14:36
blinky_ghostXueFeng: Yes i deleted it with nova. How can I test this? When the instance is in error?14:38
XueFenghi,blinky. May you can cerate many instances until your resource is insufficient.14:55
XueFengThen you use senlin node-delete to delete some 'ACTIVE' nodes.14:56
XueFengThen test do_recover again.14:56
*** wllabs has quit IRC14:59
*** wllabs has joined #senlin14:59
*** ruijie-phone has joined #senlin15:04
ruijie-phonethis is15:06
*** ruijie-phone has quit IRC15:07
blinky_ghostXueFeng: I'm getting mycluster [f8fd34e9] CLUSTER_CHECK - error: CLUSTER_CHECK [a371e0c3] failed after deleted the node. 017-01-16 15:36:22.079 26340 ERROR senlin.drivers.openstack.sdk ResourceNotFound: ResourceNotFound: No Server found for a76e7bf4-f9a1-40a8-8a7a-55a2ffc9cca8 newnode [4f0ade66] NODE_CHECK - error: Node status is not ACTIVE. Shouldn't it create a new one now?15:37
*** Drago has joined #senlin15:49
XueFengThis because you have used nova delete before. Now you can use senlin-delete to delete this node. About If this condition need to create a new VM, will discuss tomorrow.15:53
*** Drago has quit IRC15:55
*** Drago has joined #senlin16:01
blinky_ghostXueFeng: but if I delete the node using senlin it only checks for the active node, because it assumes the other was deleted and does nothing.16:39
openstackgerritOpenStack Proposal Bot proposed openstack/senlin: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/python-senlinclient: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/senlin: Updated from global requirements
*** blinky_ghost has quit IRC19:00
*** elynn has quit IRC19:02
*** elynn_ has joined #senlin19:02
*** Jeffrey4l has quit IRC21:34
*** Jeffrey4l has joined #senlin21:35
*** xuhaiweixuhaiwei has joined #senlin23:30
*** elynn_ has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!