13:01:08 <joehuang> #startmeeting tricircle
13:01:09 <openstack> Meeting started Wed Feb 17 13:01:08 2016 UTC and is due to finish in 60 minutes.  The chair is joehuang. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:01:11 <BeanZhang> Thanks
13:01:13 <openstack> The meeting name has been set to 'tricircle'
13:01:37 <joehuang> #topic rollcall
13:01:46 <joehuang> #info joehuang
13:01:54 <yipei> #info yipei
13:01:54 <BeanZhang> #info BeanZhang
13:02:04 <zhiyuan_> #info zhiyuan
13:02:06 <joehuang> hi Yipei, your connection is not stable
13:02:09 <Pengfei> #info Pengfei
13:02:11 <yipei> #info yipei
13:02:11 <yipei> yes
13:02:31 <joehuang> #topic progress review
13:02:32 <yipei> i have already tried many times
13:02:54 <joehuang> you can use VPN
13:03:53 <joehuang> Hi Yipei, have you report a bug to DevStack for the multi-region installation issue
13:04:11 <yipei> yes, i have
13:04:18 <joehuang> link?
13:04:42 <yipei> https://bugs.launchpad.net/devstack/+bug/1540802
13:04:42 <openstack> Launchpad bug 1540802 in devstack "Installing devstack with multi-region setting fails" [Undecided,New]
13:05:53 <joehuang> Bean and Pengfei, how about your environment setup, is there any challenge?
13:06:15 <Pengfei> Bean and I had try devstack already, and we both succeed install devstack and tricircle plugin. But when we try the Step 6 ,command "nova aggregate-list" cannot show any result.
13:06:32 <BeanZhang> After we installed, we try to create pod instances for Tricircle and bottom OpenStack using curl POST request, but “nova aggregate-list” shows nothing, didn’t show the results as README, Name=ag_Pod1, AZ=az1.
13:07:03 <BeanZhang> we use the latest tricirle code from github, having 130 commits
13:07:21 <BeanZhang> in README, it says Tricircle API service will automatically create a aggregate when user creates a bottom pod
13:07:44 <BeanZhang> we have not found the reason
13:08:08 <joehuang> yes, we have tested many times, it's unusual that it doesn't work
13:08:47 <zhiyuan_> Was the result showing in step 5 as expectd?
13:09:11 <BeanZhang> yes, in step 5, it shows more than README
13:09:14 <joehuang> have you tried to log in to the mysql, and check to see if all tables are there
13:10:00 <BeanZhang> we haven't check the mysql tables
13:10:07 <Pengfei> ubuntu@devstack-1:~$ token=1af602f8c4fd479dab52d8ca2de72903 ubuntu@devstack-1:~$ curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \ >     -H "X-Auth-Token: $token" -d '{"pod": {"pod_name":  "RegionOne"}}' {"pod": {"dc_name": "", "pod_az_name": "", "pod_id": "38d38c23-4dd8-4111-be91-313ad18455f4", "az_name": "", "pod_name": "RegionOne"}}ubuntu@devstack-1:~$  ubuntu@devstack-1:~$  ubuntu@devsta
13:11:43 <BeanZhang> How to know there should be how many and which tables?
13:12:50 <BeanZhang> pengfei, zhiyuan and joe have left from my client
13:13:33 <zhiyuan_> login again, the tool I used was suddenly offline
13:13:50 <joehuang> sorry, my connection was lost
13:14:19 <Pengfei> Me too
13:14:21 <Pengfei> TT
13:14:31 <BeanZhang> i use LimeChat as an IRC client and it seems very stable
13:14:34 <zhiyuan_> What about the result of curl -X GET http://127.0.0.1:19999/v1.0/pods? Were the two pods created successfully?
13:14:53 <BeanZhang> let me have a try
13:15:57 <joehuang> It's very strange that there are two region-one records
13:16:54 <joehuang> duplicated entry exception will be raised if insertion from the API
13:16:56 <zhiyuan_> Please show the list of endpoints again, I missed that
13:18:32 <joehuang> region-one pod was added once or twice?
13:18:59 <BeanZhang> I entered curl -X GET http://127.0.0.1:19999/v1.0/pods? -H "Content-Type: application/json" -H "X-Auth-Token: $token"
13:19:22 <yipei> sorry about the bad connection
13:19:23 <BeanZhang> but Authentication required, echo $token 9a3506f375884c399bde48e992cb6634
13:19:54 <joehuang> sure you need  -H "X-Auth-Token: $token"
13:20:27 <joehuang> no CLI provided yet to insert the token by default, you have to add the token head
13:20:42 <zhiyuan_> oh, please remove the "?"
13:20:54 <yipei> i still try to solve the bug correctly, and the progress is a little slow. i will speed up in the next week
13:20:58 <Pengfei> curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \ >     -H "X-Auth-Token: $token" -d '{"pod": {"pod_name":  "RegionOne"}}'
13:21:10 <Pengfei> {"pod": {"dc_name": "", "pod_az_name": "", "pod_id": "38d38c23-4dd8-4111-be91-313ad18455f4", "az_name": "", "pod_name": "RegionOne"}}
13:21:25 <Pengfei> curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \ >     -H "X-Auth-Token: $token" -d '{"pod": {"pod_name":  "Pod1", "az_name": "az1"}}'
13:21:34 <Pengfei> {"pod": {"dc_name": "", "pod_az_name": "", "pod_id": "64f60853-00b4-43c6-a36e-59afdce62f9d", "az_name": "az1", "pod_name": "Pod1"}}
13:21:35 <joehuang> to Yipei, it's ok. Please also send a mail in the M-L for the bug
13:21:45 <yipei> ok
13:22:18 <joehuang> To Pengfei, the last one you post is successful and correct
13:22:46 <Pengfei> nova aggregate-list
13:22:53 <Pengfei> +----+------+-------------------+ | Id | Name | Availability Zone | +----+------+-------------------+ +----+------+-------------------+
13:23:16 <Pengfei> nothing ...TT
13:23:34 <joehuang> you need top region to insert first
13:23:40 <BeanZhang> curl -X GET http://127.0.0.1:19999/v1.0/pods -H "X-Auth-Token: $token"
13:23:47 <BeanZhang> {"pods": [{"dc_name": "", "pod_az_name": "", "pod_id": "056b3cc9-c8a5-454f-be2b-32bd2bbf57f2", "az_name": "", "pod_name": "RegionOne"}, {"dc_name": "", "pod_az_name": "", "pod_id": "48cd53e0-036f-43d2-908c-daa9769fcd87", "az_name": "az1", "pod_name": "ag_Pod1"}, {"dc_name": "", "pod_az_name": "", "pod_id": "d977b991-44f4-47f6-9ad7-5ffb3623be5e", "az_name": "az1", "pod_name": "Pod1"}]}
13:24:49 <BeanZhang> the token was expired just now
13:25:17 <BeanZhang> are the results right?
13:25:21 <zhiyuan_> A strange pod "ag_Pod1"
13:25:32 <joehuang> could you show the openstack region information
13:25:39 <joehuang> openstack region list
13:26:05 <BeanZhang> Region    | Parent Region | Description
13:26:11 <BeanZhang> Pod1      | None          |
13:26:16 <joehuang> the ag_Pod1 is post intended ot created by API automaticly?
13:26:17 <BeanZhang> RegionOne | None          |
13:26:30 <BeanZhang> ag_Pod1 is created by me
13:26:37 <BeanZhang> post intended
13:26:52 <BeanZhang> i see the 1  | ag_Pod1  | az1
13:27:10 <BeanZhang> so create it
13:27:21 <joehuang> you mean get pod list
13:27:39 <zhiyuan_> Need to check the log of nova_api_gw, pod api will send request to it to create aggregates
13:28:55 <BeanZhang> where is the log of nova_api_gw, /opt/stack/logs?
13:28:57 <joehuang> Hi, Bean, now you have one record in aggregate-list
13:30:39 <BeanZhang> nova aggregate-list shows still nothing
13:31:38 <joehuang> no error returned when post a new pod request?
13:31:46 <BeanZhang> no error
13:31:48 <zhiyuan_> opt/stack/new/screen-logs/screen-xxx
13:32:33 <BeanZhang> i enter the first post request again, it echos Top region already exists
13:32:34 <joehuang> So I think this maybe the nova client is not accessing the tricircle
13:32:57 <BeanZhang> i enter the second post request again, it echos Record already exists
13:33:06 <joehuang> could you use nova --debug aggregate-list to see which region the client issues a command to
13:33:21 <joehuang> sure, duplicated error
13:34:24 <joehuang> maybe the nova-client is issueing the command to a bottom pod
13:36:19 <joehuang> At least two regions in your environment, and if the command is issued to Tricircle Nova-APIGW,
13:37:26 <joehuang> it should look like a request to port: 19998
13:37:39 <Pengfei> sorry about the bad connection, I haven't go to school
13:37:48 <BeanZhang> --debug, it echos a lot of found extension, REQ, RESP
13:38:15 <joehuang> Nova-APIGW listened on port 19998
13:38:21 <joehuang> yes
13:38:31 <zhiyuan_> the import part is the url nova client uses
13:38:49 <zhiyuan_> will showed when using --debug
13:39:35 <BeanZhang> yeah, REQ: curl -g -i -X GET http://127.0.0.1:8774/v2.1/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2b9a5c81209c44dffcce741de090f29552a0435b"
13:39:42 <BeanZhang> "GET /v2.1/ HTTP/1.1" 200 383
13:39:48 <BeanZhang> RESP: [200] Content-Length: 383 X-Compute-Request-Id: req-d3c7bcfc-a55a-4b84-8f5a-b798c0e782d6 Vary: X-OpenStack-Nova-API-Version Connection: keep-alive X-Openstack-Nova-Api-Version: 2.1 Date: Wed, 17 Feb 2016 13:33:33 GMT Content-Type: application/json
13:39:56 <BeanZhang> RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://127.0.0.1:8774/v2.1/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.22", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}
13:40:05 <zhiyuan_> so the url is not correct
13:40:31 <joehuang> 8774 is nova port , right?
13:40:33 <BeanZhang> which url?
13:41:01 <joehuang> http://127.0.0.1:8774/v2.1/ is nova endpoint
13:41:14 <zhiyuan_> but we have two nova api
13:41:29 <zhiyuan_> one nova api gateway, one bottom nova api
13:41:55 <zhiyuan_> try "openstack endpoint list" to see which port is for nova api
13:41:57 <joehuang> nova-apigw listened on the port 19998, althouth service type is nova
13:42:14 <BeanZhang> before the results above, there are other logs
13:42:32 <BeanZhang> REQ: curl -g -i -X GET http://127.0.0.1:8774/v2.1/ab54b5c3a73c4dff9762ed7bac5d55a2 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2b9a5c81209c44dffcce741de090f29552a0435b"
13:42:33 <BeanZhang> INFO (connectionpool:207) Starting new HTTP connection (1): 127.0.0.1
13:42:35 <joehuang> From the above information "min_version": "2.1", "version": "2.22", it must be nova
13:43:05 <BeanZhang> but 404 not found,"GET /v2.1/ab54b5c3a73c4dff9762ed7bac5d55a2 HTTP/1.1" 404 52
13:43:11 <BeanZhang> RESP: [404] Date: Wed, 17 Feb 2016 13:33:33 GMT Connection: keep-alive Content-Type: text/plain; charset=UTF-8 Content-Length: 52 X-Compute-Request-Id: req-45145d94-c409-4230-a7c5-5783116d1ccf
13:43:15 <zhiyuan_> try this: nova --os-region-name RegionOne aggregate-list
13:43:23 <joehuang> so Bean and Pengfei, please check endpoint in KeyStone.
13:43:34 <BeanZhang> 1  | ag_Pod1    | az1
13:43:40 <BeanZhang> 3  | ag_ag_Pod1 | az1
13:43:43 <joehuang> yes ZHiyuan's work
13:44:00 <Pengfei> bingo!
13:44:14 <Pengfei> ubuntu@devstack-1:/home/stack/devstack$ nova --os-region-name RegionOne aggregate-list Your request was processed by a Nova API which does not support microversions (X-OpenStack-Nova-API-Version header is missing from response). Warning: Response may be incorrect. +----+---------+-------------------+ | Id | Name    | Availability Zone | +----+---------+-------------------+ | 1  | ag_Pod1 | az1               | +----+---------
13:44:22 <zhiyuan_> so you need to specify Region in your openrc/adminrc file
13:44:23 <BeanZhang> yeah, i know where is the reason
13:44:28 <BeanZhang> yes
13:44:30 <joehuang> Zhiyuan, I am afraid your script change for two nodes may lead to chaos
13:44:34 <Pengfei> haha~
13:44:55 <joehuang> or the admin-rc issue?
13:45:13 <BeanZhang> maybe in tricircle README, it should point out the suggested admin-openrc.sh
13:45:39 <joehuang> what's your admin-rc configuration?
13:45:42 <zhiyuan_> Yes, maybe you two submit a patch to improve README
13:45:46 <BeanZhang> we use the admin-openrc.sh from the install guide
13:45:52 <BeanZhang> ok
13:46:10 <Pengfei> ok~
13:46:12 <zhiyuan_> I always  specify os_region so didn't find this issue
13:46:13 <BeanZhang> http://docs.openstack.org/liberty/install-guide-ubuntu/keystone-openrc.html
13:46:36 <BeanZhang> this is our openrc file
13:46:46 <BeanZhang> and not specify the os_region
13:46:53 <joehuang> OK. let's continue this in #openstack-tricircle channel for daily discussion
13:47:06 <zhiyuan_> So, the problem is, without specifying os_region, nova client will try to find one nova service, thus it go to the wrong service
13:47:13 <joehuang> and please submit a patch for this
13:47:19 <BeanZhang> yeah
13:47:28 <BeanZhang> OK
13:47:35 <joehuang> Zhiyuan, your recent progress?
13:48:13 <BeanZhang> after specify the os-region-name, it echos Your request was processed by a Nova API which does not support microversions (X-OpenStack-Nova-API-Version header is missing from response). Warning: Response may be incorrect.
13:48:24 <BeanZhang> is it normal?
13:48:37 <joehuang> yes
13:48:43 <BeanZhang> ok~
13:49:01 <zhiyuan_> this can be changed later, since our nova api gateway doesn't support microversions currently
13:49:14 <zhiyuan_> so nova client raises a warning
13:50:11 <zhiyuan_> For my progress, the L3 N-S networking patch has been merged. Now I am working the security group part. It will take some times since it's also a big stuff......
13:50:33 <joehuang> please split into into smaller patch
13:51:05 <joehuang> and also for the quota part, we have already have a terrible big patch :(
13:51:15 <joehuang> please review that, thanks
13:51:23 <zhiyuan_> smaller patch? then maybe I can submit the RPC part first
13:51:45 <joehuang> yes
13:51:56 <zhiyuan_> ok, I will find time to review it tomorrow
13:52:25 <joehuang> ok, it's a little bit late, let's have more chat in #openstack-tricircle channel
13:52:42 <zhiyuan_> fine
13:52:44 <joehuang> thanks for your attendence
13:53:08 <joehuang> especially for Bean, Pengfei and Yipei, you are still in vacation
13:53:31 <joehuang> see you next time
13:53:56 <joehuang> #endmeeting