Tuesday, 2017-04-11

*** catintheroof has quit IRC00:19
*** zhurong has joined #senlin00:55
*** shu-mutou-AWAY has quit IRC01:19
*** yanyanhu has joined #senlin01:26
openstackgerritXueFeng Liu proposed openstack/senlin master: Add prepare and cleanup to all needed test cases  https://review.openstack.org/45533101:53
*** dixiaoli has joined #senlin01:54
*** dixiaoli has quit IRC01:58
*** dixiaoli has joined #senlin02:02
openstackgerritXueFeng Liu proposed openstack/senlin master: Add prepare and cleanup for nova server local tempest  https://review.openstack.org/45532502:07
QimingXueFeng, there?02:08
*** elynn has joined #senlin02:08
*** catintheroof has joined #senlin02:16
*** shu-mutou has joined #senlin02:21
*** catintheroof has quit IRC02:34
*** catintheroof has joined #senlin02:35
root____5hi, QiMing02:39
*** catintheroof has quit IRC02:40
QimingI'm confused by your patches to the tempest/common/utils02:40
Qimingand the big patch that your are making to the tempest test cases02:41
Qimingthe changes about nova key pair and subnet are only relevant to integration tests, right?02:42
Qimingbecause in api and functional tests, we use openstack_test backend, not the real openstack backend02:43
Qimingthere is no need to create keypairs or network/subnet for most of the api/functional tests02:43
root____5hi, Qiming02:50
XueFengIn local run tempest02:50
XueFengWe need create theses for each dynamical user/project02:51
XueFengtempest run with dynamical by default02:51
XueFengYou can do a teset. cd  /opt/stack/tempest, then run `nosetests -v senlin`02:53
Qimingdid you set cloud_backend = openstack_test in your senlin.conf before running api tests?02:56
Qimingor functional tests?02:56
XueFengOh, not set.02:57
XueFengLet me do a test02:57
*** dixiaoli has quit IRC03:00
*** dixiaoli has joined #senlin03:01
root____5QiMing, it's ok03:10
Qimingthose new logics are useful, however, for integration tests where we use real openstack drivers03:11
Qimingso please consider make them integration tests specific03:11
Qiminginstead of polluting the api and functional tests03:11
XueFengok, got it.03:12
XueFengWill remove the change for api and functional tests03:12
openstackgerrityangyide proposed openstack/senlin master: Improve node check for health_policy_poll  https://review.openstack.org/45550303:52
*** dixiaoli has quit IRC04:17
*** dixiaoli has joined #senlin04:19
*** dixiaoli has quit IRC04:23
openstackgerrityangyide proposed openstack/senlin master: Improve node check for health_policy_poll  https://review.openstack.org/45550305:05
*** dixiaoli has joined #senlin05:26
ruijiehi Qiming, online?05:52
ruijieabout this bp, I got 2 ideas, not sure which one is better06:01
ruijie1. do health check before scaling. 2. do health check after scaling06:01
ruijiesince the process is _execute() -> pre_op() -> do_xyz() -> post_op()06:02
*** elynn has quit IRC06:03
ruijieif we want to do health check before scaling, we need to sync resource status in pre_op() in deletion policy if exit or in do_xyz() if no deletion policy exit06:03
ruijiethat might be heavy weight to sync resource status in deletion policy since it is single thread06:04
ruijiesecond option is do health check after scaling, which means we don't care the status of resource for the current action, we do it synchronized with number of node_check action.06:05
Qiminggot your points06:06
Qimingso ... the question is ... why do we want a health check when doing scaling?06:06
ruijiefor scale in action, we will chose bad nodes first ..06:07
ruijiethis is the only question06:07
QimingI'm taking a step back to revisit the reason why we are doing this06:07
Qimingwe want to make sure that the scaling operation is doing things right, correct?06:08
ruijieyes Qiming06:08
Qimingthen the choice is evident06:08
Qimingit is for sure that introducing an extra step before scaling would mean some overhead06:09
Qimingthat was the reason we wanted to make 'health_check' optional06:10
ruijieso .. we do it before we chose candidates06:10
Qimingthis extra checking will be meaningless if we append it to the end of a scaling operation06:11
ruijieyes Qiming, but I am not sure if we need this process for scale out too06:12
Qimingin an ideal case, senlin will know the EXACT statuses of all cluster nodes before doing scaling06:13
Qimingwhether we want to do health check before scaling out can be left for future decision06:14
Qimingnow, let's focus on getting node status checked before *scaling in*06:15
ruijieokay, Qiming, that makes sense :)06:15
Qimingthe question becomes how do we implement it06:15
Qimingas you have mentioned, there are two places where you will need to invoke the health check06:16
Qimingone in the do_scale_in() (or do_resize()) action, the other is in the deletion policy06:17
Qiminganywhere else?06:17
ruijieno Qiming06:17
Qimingany impact on other policies such as LB, cross-az placement, cross-region placement?06:18
ruijienp, we can use engine_node.do_check() to do this work06:18
Qimingso you mean we are not forking new child actions for this06:19
ruijiein do_scale_in() we can actually use node_action to do it, but in deletion policy that is not easy06:20
Qimingokay. We are not supposed to create actions in a policy06:20
ruijieyes Qiming06:21
Qimingthen we don't have a lot of choices06:21
ruijieand we may need to handle nodes which status is WARNING since we introduced it for nodes that failed to added to lb pool06:21
Qimingokay, that makes sense06:22
ruijiethanks Qiming, much clear now :)06:24
Qimingthanks for bringing these up06:24
ruijienp :)06:24
yuanbinruijie, the health policy action recreate, I don't use this action?06:26
openstackgerritOpenStack Proposal Bot proposed openstack/senlin-dashboard master: Imported Translations from Zanata  https://review.openstack.org/45553506:34
ruijieem, yuanbin, you mean for cluster_recover?06:41
yuanbinruijie, yes06:42
openstackgerritRUIJIE YUAN proposed openstack/senlin master: handle node which status is WARNING  https://review.openstack.org/45554206:42
ruijiethat's 2 scenarios06:44
ruijiefor the health policy, we can monitor the node by pull the status of node or listen to rabbit, then we do the action to recover it, by recreating or rebuilding .etc06:45
ruijiedo health check in scaling action is another scenario, we may want to know the exact status of node before we do scale in or scale out, because we are supposed to remove bad nodes first06:47
yuanbinruijie, yes, the rebuilding i have some problems, my instance use cinder volume to boot, nova rebuild don't support cinder volume with root device06:48
yuanbinruijie, remove bad nodes, create new nodes from profile, i known.06:49
ruijieoh okay :) I am not very familiar with nova ..06:50
yuanbinruijie, I have a question, when i use cluster-update the new profile use cinder boot, so the cluster-update execute image_update will faild06:51
yuanbinruijie, so, I use cluster-update image change, execute recreate action, but the recreate don't realize06:52
ruijieI think Qiming may know this better :)06:54
Qimingsounds to me that you are not simply updating the image of a server06:57
Qimingyou are actually updating its block device mapping06:57
QimingI'm not 100% sure nova supports such an operation06:57
Qimingeven if nova does, senlin has not yet supported it.06:58
yuanbinQiming, yes, if use ceph backend boot instance with cinder volume, rebuild will be update cinder volume06:58
Qimingplease read the do_update logic has ruijie pointed above and feel free to propose a fix06:59
QimingI don't have a cinder env or a ceph deployment to experiment with this06:59
QimingI'm more than happy to learn things in this space though07:00
yuanbinQiming, I known senlin update logic, only update_image a  little question07:02
Qimingdid you mean question or problem?07:04
Qimingthere are many different scenarios to consider in the do_update workflow, we tried to optimize it so that we won't reboot/rebuild the server twice if we can solve the problem in one reboot07:05
Qiminghowever, I cannot assure you that the root device changes have been considered07:06
Qimingthat is the reason I think we need your help on fixing it07:06
yuanbinQiming, senlin call rebuild , if my profile use block_device_mapping_v2 create boot volume, nova rebuild don't successful in my test boot volume rebuild07:13
yuanbinQiming, this is nova rebuild probrom07:14
Qimingany workaround we can think of?07:14
Qimingfor example, some other nova apis we can leverage?07:14
openstackgerritRUIJIE YUAN proposed openstack/senlin master: handle node which status is WARNING  https://review.openstack.org/45554207:14
Qimingyuanbin, https://developer.openstack.org/api-ref/compute/#servers-run-an-action-servers-action07:15
yuanbinQiming, I want change senlin call nova rebuild to recreate, delete old profile exist node, and use new profile create node.07:17
Qimingthat makes sense07:17
Qimingsince we can either do rebuild or recreate for this purpose07:17
Qiminghow are we gonna make the decision to avoid breaking existing users?07:18
yuanbinQiming, nova will be support detach root device, use cinder create new volume and attach to instance, implementation  root device update07:18
Qimingif ("block_device_mapping_v2" is used) then we do recreate; else we do rebuild?07:19
yuanbinQiming, yes07:20
Qimingor ... we can check the boot_index property for deciding this?07:20
Qimingif bdmv2 is not used for root device, recreate would be an over-kill, right?07:20
Qimingokay, this means we have a decent workaround07:21
Qimingwe don't blindly recreate, but we do recreate when unavoidable07:22
Qimingrecreate is harmful ... because the instance id will be changed, the IP address allocated will be changed ...07:22
Qimingbut ... if recreate is the only way out, we should do it07:23
yuanbinI known, if recreate the ip will be change, the other(securite group, password, ...) maybe configure same07:24
openstackgerritchenyb4 proposed openstack/senlin master: fix node do_check invalid code  https://review.openstack.org/45557508:09
openstackgerritRUIJIE YUAN proposed openstack/senlin master: handle node which status is WARNING  https://review.openstack.org/45554208:25
*** elynn has joined #senlin08:38
*** openstackgerrit has quit IRC09:03
*** zhurong has quit IRC10:10
*** zhurong has joined #senlin10:11
*** ruijie has quit IRC10:13
*** elynn has quit IRC10:13
*** yanyanhu has quit IRC10:35
*** dixiaoli has quit IRC11:21
*** dixiaoli has joined #senlin11:21
*** dixiaoli has quit IRC11:25
*** XueFeng has quit IRC12:04
*** root____5 has quit IRC12:04
*** shu-mutou is now known as shu-mutou-AWAY12:27
*** zhurong has quit IRC12:54
*** yanyanhu has joined #senlin12:56
*** XueFengLiu has joined #senlin13:00
*** yanyanhu has quit IRC13:58
*** testtest has joined #senlin13:59
*** testtest has quit IRC14:00
*** testtest has joined #senlin14:03
*** testtest has quit IRC14:06
*** XueFengLiu has quit IRC14:06
-openstackstatus- NOTICE: latest base images have mistakenly put python3 in some places expecting python2 causing widespread failure of docs patches - fixes are underway14:28
-openstackstatus- NOTICE: we have rolled back centos-7, fedora-25 and ubuntu-xenial images to the previous days release. Feel free to recheck your jobs now.14:48
*** catintheroof has joined #senlin21:19
*** catintheroof has quit IRC21:21
*** catintheroof has joined #senlin21:21
*** dixiaoli has joined #senlin22:43
*** catintheroof has quit IRC23:29
*** catintheroof has joined #senlin23:30
*** catintheroof has quit IRC23:30
*** dixiaoli has quit IRC23:36

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!