Friday, 2016-03-11

*** zzxwill has quit IRC00:00
*** sridhar_ram1 has joined #senlin00:01
*** sridhar_ram has quit IRC00:03
*** zzxwill has joined #senlin00:06
*** Qiming has quit IRC00:08
*** zzxwill has quit IRC00:20
*** openstackgerrit_ has quit IRC00:23
*** zzxwill has joined #senlin00:24
*** openstackgerrit_ has joined #senlin00:25
*** zzxwill has quit IRC00:41
*** zzxwill has joined #senlin00:42
*** openstackgerrit_ has quit IRC00:48
*** openstackgerrit_ has joined #senlin00:50
*** zzxwill has quit IRC00:52
*** sridhar_ram1 has quit IRC00:57
*** zzxwill has joined #senlin00:59
*** sridhar_ram has joined #senlin01:01
*** Qiming has joined #senlin01:07
*** zzxwill has quit IRC01:13
*** zzxwill has joined #senlin01:18
*** zzxwill has quit IRC01:33
*** zzxwill has joined #senlin01:34
*** sridhar_ram has quit IRC01:35
*** zzxwill has quit IRC01:49
*** zzxwill has joined #senlin01:49
*** Yanyanhu has joined #senlin01:58
*** zzxwill has quit IRC02:00
*** zzxwill has joined #senlin02:12
*** elynn has joined #senlin02:19
*** zzxwill has quit IRC02:24
*** elynn has quit IRC02:25
*** elynn has joined #senlin02:25
*** zzxwill has joined #senlin02:30
Qiminghi, guys02:39
Qimingduring a discussion with some early users, I was answering their questions regarding how authentication is done, how multi-region deployment is performed02:40
Qimingso I was revisiting the api middleware source to make sure I'm talking accurately02:41
Qiminghowever, I am not feeling comfortable when I saw the trust middleware talks directly to db02:41
Yanyanhuyou mean the part about keystone middleware?02:41
Qimingthat is the question I want to raise02:42
Qimingshould we eliminate direct DB interactions from the API layer02:42
Qimingthat will make the architecture much cleaner02:42
YanyanhuQiming, this is what we wanted. but didn't find a better way to address problem before02:42
QimingI want to introduce some "internal" RPC interfaces for trust retrieval and creation02:42
Qimingin the api pipeline, trust comes after the context middleware02:43
Qimingso it is possible to decouple it from db02:43
Yanyanhuhmm, this is feasible. But on potential problem is the API request handling progress could be blocked if engine is busy02:44
QimingI can look into that if there is no objections on this02:44
Qimingif the engine is busy, you cannot make any progress anyway02:44
Qimingdon't believe that is a valid concern02:44
YanyanhuI mean the pgress inside middleware02:44
Yanyanhuyes, but that's a little bit different02:44
Qiminghave to leave for a moment02:45
Yanyanhuanyway, not a serious problem02:45
Yanyanhuok, ttyl02:45
lixinhuihave you ever meeting this problem02:50
lixinhuioctavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying02:50
lixinhuiOnce you said some config need to pay attention02:51
Yanyanhuhi, lixinhui, octavia is another lb service in openstack?02:51
lixinhuiit is the provider of lb02:51
lixinhuiin haproxy driv er02:51
lixinhuiI wonder what it is02:52
Yanyanhuoh, I see02:52
Yanyanhuabout the configure, I mean the driver type and I can't recall whether there is some authentication related options02:52
Yanyanhuby the driver, it's actually the service provider02:53
lixinhuiDifficult here is hard to find real error02:56
lixinhuibehind the PENDING-to....02:56
YanyanhuI remember I used to check haproxy related process or something to ensure it works well02:58
Yanyanhubut can't recall the detail how to check it...02:58
lixinhuiso you do not use  octavia03:00
Qimingsounds like a networking problem?03:03
Yanyanhulixinhui, yes, I didn't use it before03:03
Qimingafter your ha_proxy machine is up03:03
Qimingyou can try connect to it manually03:03
Qimingsometimes, it takes time to have the controller realize there are new destination reachable03:04
Qimingsometimes, the security group thing jumps into the way when you try connecting to a VM03:04
*** zzxwill has quit IRC03:12
*** zzxwill has joined #senlin03:16
*** elynn_ has joined #senlin03:26
*** elynn has quit IRC03:30
*** gmann has left #senlin03:48
*** elynn_ has quit IRC04:01
*** shu-mutou-AFK is now known as shu-mutou04:02
*** zzxwill has quit IRC04:11
*** zzxwill has joined #senlin04:12
*** zzxwill has quit IRC04:33
*** zzxwill has joined #senlin04:51
*** elynn_ has joined #senlin04:56
*** elynn_ has quit IRC05:00
*** elynn_ has joined #senlin05:01
*** zzxwill has quit IRC05:14
*** zzxwill has joined #senlin05:18
*** sridhar_ram has joined #senlin05:23
*** zzxwill has quit IRC05:27
*** zzxwill has joined #senlin05:30
*** jdandrea has quit IRC05:33
*** heyongli has joined #senlin05:39
*** jdandrea has joined #senlin05:40
*** heyongli has quit IRC05:41
*** heyongli has joined #senlin05:41
*** zzxwill has quit IRC05:43
*** zzxwill has joined #senlin05:44
*** heyongli has quit IRC05:46
*** heyongli has joined #senlin05:46
*** jdandrea_ has joined #senlin05:49
*** jdandrea has quit IRC05:50
*** zzxwill has quit IRC05:54
*** zzxwill has joined #senlin05:55
*** heyongli has quit IRC05:56
*** heyongli has joined #senlin05:57
Yanyanhuhi, Qiming, I'm considering retore the obsoleted BP of access permission control in Senlin. Since this is not a work that will be done in this cycle, I will mark it as under discussion.06:02
*** heyongli has quit IRC06:06
*** heyongli has joined #senlin06:07
Yanyanhuwill read it carefully :)06:07
Qimingwhat we are proposing is actually called "DYNAMIC" RBAC in keystone terms06:07
Qimingspes from ayoung is here:
Yanyanhuthanks a lot for this clue06:10
*** heyongli has quit IRC06:17
*** heyongli has joined #senlin06:17
*** openstackgerrit_ has quit IRC06:17
*** openstackgerrit_ has joined #senlin06:18
*** zzxwill has quit IRC06:25
*** heyongli has quit IRC06:27
*** zzxwill has joined #senlin06:28
lixinhuiQiming, it should irrelevant to secuity group06:33
*** zzxwill has quit IRC06:33
lixinhuiand the loadbalancer instance hide behind and nova can not list it06:33
*** zzxwill has joined #senlin06:33
lixinhuithe port id seems right06:33
lixinhuibut the error means the ha_procy driver can not connect to the built instance06:34
lixinhuiI will search more to see what is the process06:34
*** shu-mutou is now known as shu-mutou-AFK06:46
*** sridhar_ram has quit IRC06:56
*** zzxwill has quit IRC06:58
*** zzxwill has joined #senlin07:07
*** elynn_ has quit IRC07:09
Qimingwhat is the built instance referring to?07:10
Qimingif you cannot list it, try the admin account?07:11
lixinhuiadmin is the same07:14
*** elynn has joined #senlin07:14
lixinhuithe built instance is the loadbaancer imstance in lbaas v207:14
Qimingthat instance is a VM07:16
Qimingmaybe it is in a special project07:16
Qiminglixinhui, this line:
Qimingit means the amphora instance may need a security group, but by default it is empty07:18
Qimingthat is something you may want to verify07:18
Qiminganother possibility, it takes too long to wait for the instance to become reachable07:19
lixinhuiit will us ethe default07:19
Qimingokay, you have configured your default security group?07:20
lixinhuithe port it used to ccreate the instance is right one07:20
Qimingthe new "default" security group is not blocking any port range, right?07:20
Qimingokay, that almost eliminates the possibility of secgroup07:21
lixinhuiall the samples I searched about lbaas v2 is using octavia07:22
lixinhuiso I am seaeching how to check correctness of ha_proxy process07:22
Qimingyou got to log into that instance07:22
lixinhuilet me try to find some clue to do it07:23
Qimingone way to try07:23
Qimingexport OS_USERNAME=admin07:23
Qimingthen 'nova list --all-tenants 1'07:23
QimingI didn't try it before07:24
lixinhuiwill try it07:24
Qimingbut I really believe the nova instance is created somewhere07:24
*** gongysh has joined #senlin07:26
Qiminggongysh, hi07:30
*** zzxwill has quit IRC07:31
gongyshQiming,  hi07:32
Qiminghi, we have encountered some LBaaS and octavia problem07:33
Qimingwhere I believe you are THE expert, :)07:33
*** zzxwill has joined #senlin07:33
gongyshQiming,  no, I have not play with it yet. beyond mime.07:33
Qimingokay, give me a name07:34
QimingI'll let you go07:34
*** zzxwill has quit IRC07:38
Qiminggongysh, ???07:38
* gongysh looking ...07:39
Qimingamong these names:07:39
QimingAdam Harwellflux.adam@gmail.com07:39
QimingBertrand Lallaubertrand.lallau@gmail.com07:39
QimingBrandon Loganbrandon.logan@rackspace.com07:39
QimingDoug Wiegleydougwig@parkside.io07:39
QimingGerman Eichbergergerman.eichberger@hp.com07:39
QimingMichael Johnsonjohnsomor@gmail.com07:39
QimingStephen Balukoffstephen@balukoff.com07:39
gongyshQiming, IRC: #openstack-lbaas07:41
Qimingokay, :)07:41
Qimingyou are of no value now07:41
* Qiming pulls the trigger ...07:41
gongyshQiming,  shot dead.07:43
*** zzxwill has joined #senlin07:45
Yanyanhuhi, Qiming, just quickly went through the spec from adam young07:52
Yanyanhuit's very helpful for controlling http resource access based on role check07:52
Qimingit looks like one07:53
Yanyanhubut it's not satisfied enough to meet our requirement07:53
Qimingbut as ayoung mentioned07:53
Yanyanhue.g. control the access to each single entity07:53
Qimingapi level or pre-api level access control is still limited07:53
Qimingit is not touching the resources inside database07:54
Yanyanhuit's understandable07:54
Qimingso, with that in minde07:54
Yanyanhuso I guess maybe we can have both of them07:54
Qimingwe can work on the permission thing07:54
Yanyanhuthis is what I'm thinking now07:54
Yanyanhuwill try to mix them07:54
Yanyanhuleverage what keystone can provide us as much as possible07:55
Qiminghowever, I'm not so sure if a per-resource access control is needed07:55
Yanyanhuand implement the function it doesn't provision in Senlin07:55
Yanyanhuwill leave for a while07:55
Qimingwe may need 'chown', 'chmod' calls07:55
Yanyanhuyea, will further think about it07:55
Yanyanhuthis is what we want07:56
Yanyanhuleave now07:56
Yanyanhugo back later07:56
*** zzxwill has quit IRC08:02
*** zzxwill has joined #senlin08:03
Qimingso ...08:05
Qimingthis is one of the quiet weeks where we are not supposed to add new features08:05
Qiminguntil newton development opens08:06
QimingI'd like to give testing a higher priority08:06
Qiminge.g. tempest, rally, functiona, stress ... etc08:06
Qimingnone of those will break the existing code08:06
Qiming(hopefully) :)08:06
YanyanhuQiming, yes, actually I'm now planning to add some functional test for failure cases which can be implemented inside existing framework08:16
Yanyanhue.g. creating cluster with invalid profile-id08:16
Yanyanhusomething like this08:16
Yanyanhufor more complicated cases, or API consistency test, may need new design08:17
Qimingwe can start shifting to tempest I think08:18
Yanyanhuyou mean shift functional test to tempest? Or using tempest for API test08:19
YanyanhuI recalled maintaining functional test inside each project individually is recommened?08:19
Qimingat least, api surface test should be done via tempest plugin08:20
Qimingyes, no conflict there08:20
Qimingthe code should live in senlin08:21
YanyanhuI see08:21
Qimingwe got some pretty good guidance the other day from a tempest expert08:21
Qimingwe can check how it works by looking into congress08:21
Yanyanhuso the difference is where we define the job, devstack gate or tempest?08:21
Qimingfor example08:21
Qimingit can be a tox env specified in tox.ini08:22
*** zzxwill has quit IRC08:25
*** zzxwill has joined #senlin08:29
*** zzxwill has quit IRC08:37
openstackgerritQiming Teng proposed openstack/senlin: Add engine service RPC api for credentials
*** zzxwill has joined #senlin08:40
openstackgerritQiming Teng proposed openstack/senlin: Add engine service RPC api for credentials
*** gongysh has quit IRC08:46
*** zzxwill has quit IRC08:47
*** zzxwill has joined #senlin08:50
*** gongysh has joined #senlin08:51
lixinhuithat is the loadbalancer instance08:56
lixinhuibut can not ssh it08:56
lixinhuibecause of lack of router08:56
Qimingokay, checking08:58
Qimingyou cannot even ssh to it via
Qimingit has a security group named lb-mgmt-sec-grp09:00
Qimingand the management network is
Qimingyou will need a octavia_ssh_key to ssh, after switching to project eb59c8ab580b40c586c5bda06f51c8f809:01
Qimingif the ssh is still rejected, you can treat it as a normal SSH problem09:02
openstackgerritQiming Teng proposed openstack/senlin: RPC support for credential operations
lixinhuino reponse too by 10.0.0409:05
Qimingcan you ping it?09:06
lixinhuiI can not09:07
lixinhuibut sure for management net09:07
lixinhuiand I can find the key under /etc/octavia/.ssh/octavia_ssh_key09:08
*** elynn_ has joined #senlin09:09
*** elynn has quit IRC09:10
Qiming"but sure for management net" ... what does this mean?09:11
Qimingyou can reach it from management network?09:13
lixinhuiI can ping through the management net09:13
Qimingthen you should log in via management net09:13
lixinhuibut ssh faild although I can find the key09:13
lixinhuissh: connect to host port 22: No route to host09:14
Yanyanhusecurity group problem?09:14
Qimingyou can ping it09:14
lixinhuihave done this09:14
lixinhuineutron security-group-rule-create f29c45ff-dfd3-44a3-a1a6-b7716eff9041 --protocol tcp --port-range-min 22 --port-range-max 2209:14
lixinhuiping workds09:15
lixinhuif29c45ff-dfd3-44a3-a1a6-b7716eff9041 is the management net09:15
Qimingssh is giving an inaccurate error message ...09:15
Yanyanhudid you apply this security group when booting up VM?09:15
Qimingwhat are the first few characters of the octavia_ssh_key file?09:17
lixinhuiPermission denied (publickey).09:19
*** zzxwill has quit IRC09:20
*** zzxwill has joined #senlin09:21
Qimingokay, that is a private key09:22
Qimingssh -i /etc/octavia/.ssh/octavia_ssh_key doesn't work?09:23
lixinhuien, same error09:23
Qimingit didn't ask for assword?09:24
Qimingokay, the image has disabled password authentication09:25
Qimingdo you know the OS installed in the image?09:25
lixinhuibu default it should be ubuntu09:27
Qimingssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@  ?09:32
QimingI think we are creating a lot of problems because the installation and configuration of octavia wasn't complete09:33
Qimingwhen I tried read the source code of rest_api_driver.py09:33
QimingI found that octavia is actually doing a REST call09:34
Qimingwith a server certificate09:34
Qimingwhich should be configured in haproxy_amphora section, named 'server_ca'09:34
Qimingthat is the only thing required for the REST request09:35
lixinhuiit is set by default value09:36
lixinhuiand to avoid the mismatch problem09:37
Qiminghow is the server_ca generated?09:37
Qimingany script doing that?09:37
lixinhuiI ensure generate the octavia.conf very time now09:37
Qimingbut generating octavia.conf doesn't mean the generation of a server_ca09:38
Qimingokay, I see09:39
*** elynn_ has quit IRC09:41
lixinhuiit is indeed generated with the conf together09:41
Qimingokay, so it is generating certificates09:42
*** Yanyanhu has quit IRC09:42
Qimingif I were you I will add a LOG.error('%s' % reqargs) at here:
Qimingor, at line 242, try catch the exception and print it out09:45
QimingI really hate this design, which is making the most fragile link very difficult to debug09:46
Qimingafter you have modified the code09:47
*** elynn_ has joined #senlin09:47
Qimingyou don't have to reinstall devstack, alright?09:47
Qimingjust kill the octavia service and restart it09:47
lixinhuiI see09:47
Qimingalso, you may want to try your luck in the #openstack-lbaas channel09:49
QimingI think such a problem must be among the top 5 in their FAQ list09:49
lixinhuithere have some one report this bug09:49
lixinhuibut no further action there09:50
*** elynn_ has quit IRC09:52
lixinhuiI leave some message at the channel hope someone could help09:55
lixinhuithe error content is09:55
lixinhui2016-03-11 17:47:52.794 3785 ERROR octavia.amphorae.drivers.haproxy.rest_api_driver [-] {'url': '', 'verify': '/etc/octavia/certs/ca_01.pem', 'json': {'subnet_cidr': u'', 'gateway': u'', 'mac_address': u'fa:16:3e:94:b0:2c'}, 'timeout': (10.0, 60.0), 'headers': {'User-Agent': 'Octavia HaProxy Rest Client/0.5 ('}}09:55
*** zzxwill has quit IRC09:58
Qimingthe above log shows you that the rest api is invoked from the 10 network, not the 192.168 network10:00
*** zzxwill has joined #senlin10:01
Qimingsorry, it is about getting data from
Qimingand the port is now 944310:01
Qimingis that port opened?10:02
* Qiming is feeling very very very hungry ... 10:03
*** elynn_ has joined #senlin10:11
*** Qiming has quit IRC10:11
lixinhui~$ netstat -nap|grep 944310:14
lixinhui(No info could be read for "-p": geteuid()=1000 but you should be root.)10:14
lixinhuitcp        0      0  *               LISTEN      -10:14
*** elynn_ has quit IRC10:20
*** gongysh has quit IRC10:31
*** zzxwill has quit IRC10:35
*** zzxwill has joined #senlin10:38
*** zzxwill has quit IRC10:40
*** zzxwill has joined #senlin10:50
*** zzxwill has quit IRC10:54
-openstackstatus- NOTICE: Gerrit is going to be restarted due to bad performance10:55
*** ChanServ changes topic to "Gerrit is going to be restarted due to bad performance"10:55
*** ChanServ changes topic to "IRCLog: | Bugs: | Review:,n,z"11:01
-openstackstatus- NOTICE: Gerrit has been restarted successfully11:01
*** Qiming has joined #senlin11:01
*** zhenguo_ has quit IRC12:10
*** zzxwill has joined #senlin12:37
*** zzxwill has quit IRC13:41
*** zzxwill has joined #senlin13:44
*** Qiming has quit IRC13:54
*** Qiming has joined #senlin13:54
*** lixinhui_ has joined #senlin14:16
*** lixinhui_ has quit IRC14:17
*** zzxwill has quit IRC14:55
*** zzxwill has joined #senlin14:57
*** Qiming has quit IRC14:58
*** Qiming has joined #senlin14:58
*** zzxwill has quit IRC14:59
*** zzxwill has joined #senlin15:57
*** Qiming has quit IRC16:11
*** zzxwill has quit IRC16:39
*** zzxwill has joined #senlin16:42
*** zzxwill has quit IRC16:46
*** zzxwill has joined #senlin16:57
*** zzxwill has quit IRC17:23
*** zzxwill has joined #senlin17:31
*** zzxwill has quit IRC17:45
*** zzxwill has joined #senlin17:58
*** zzxwill has quit IRC18:04
*** zzxwill has joined #senlin18:17
*** zzxwill_ has joined #senlin18:31
*** zzxwill has quit IRC18:31
*** zzxwill_ has quit IRC18:38
*** sridhar_ram has joined #senlin18:58
*** sridhar_ram1 has joined #senlin19:01
*** sridhar_ram has quit IRC19:03
*** sridhar_ram1 is now known as sridhar_ram19:49

Generated by 2.14.0 by Marius Gedminas - find it at!