| *** caowei_ has joined #openstack-zun | 00:21 | |
| *** caowei has quit IRC | 00:24 | |
| *** caowei_ is now known as caowei | 00:24 | |
| *** zhurong has quit IRC | 00:45 | |
| *** kevzha01 has joined #openstack-zun | 01:25 | |
| *** kevzha01 is now known as kevinz | 01:25 | |
| *** Shunli has joined #openstack-zun | 01:26 | |
| openstackgerrit | Merged openstack/zun-ui master: Fix translate tag https://review.openstack.org/477311 | 01:38 |
|---|---|---|
| *** kiennt has joined #openstack-zun | 01:53 | |
| *** hongbin has joined #openstack-zun | 02:17 | |
| *** hongbin_ has joined #openstack-zun | 02:18 | |
| *** hongbin_ has quit IRC | 02:19 | |
| *** hongbin_ has joined #openstack-zun | 02:19 | |
| *** hongbin has quit IRC | 02:22 | |
| *** hongbin_ has quit IRC | 02:22 | |
| *** hongbin has joined #openstack-zun | 02:22 | |
| *** zsli_ has joined #openstack-zun | 02:25 | |
| openstackgerrit | Shu Muto proposed openstack/zun-ui master: Fix image pull action https://review.openstack.org/477458 | 02:27 |
| *** zsli__ has joined #openstack-zun | 02:27 | |
| *** Shunli has quit IRC | 02:27 | |
| *** zsli_ has quit IRC | 02:31 | |
| openstackgerrit | feng.shengqin proposed openstack/zun master: Validate security groups at zun-api https://review.openstack.org/475111 | 02:33 |
| openstackgerrit | Bin Zhou proposed openstack/zun master: PCI passthrough device modeling https://review.openstack.org/475913 | 02:33 |
| *** lakerzhou has joined #openstack-zun | 02:34 | |
| openstackgerrit | Merged openstack/zun-ui master: Imported Translations from Zanata https://review.openstack.org/477475 | 02:41 |
| hongbin | hi all, team meeting will start in about 15 minutes at #openstack-meeting channel | 02:45 |
| *** lakerzhou has quit IRC | 02:48 | |
| hongbin | hi all, team meeting is live now at #openstack-meeting | 03:00 |
| *** zsli__ has quit IRC | 03:02 | |
| *** zsli_ has joined #openstack-zun | 03:02 | |
| *** adisky__ has joined #openstack-zun | 03:17 | |
| hongbin | zsli_: mind giving me a paste for the error you saw? | 03:59 |
| zsli_ | un/servicegroup/zun_service_periodic.py:40 | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 ERROR zun.compute.manager [req-9f4006f8-8533-42d6-8a9d-7bea5731da96 admin admin] Unexpected exception: Docker internal error: 500 Server Error: Internal Server Error ("failed to create endpoint zun-sandbox-efae9dd7-75aa-4897-927e-26f0409c0c6e on network afb2459d-a5a0-45c7-9c30-eaec2d403233-df6ffdb34f20496692d221f2f2f7faaf: remote: Neutron net associated with identifier kuryr.net.uuid.lh:3a784f711e0eaf4673f883f93e4d5d76,kuryr.net | 03:59 |
| zsli_ | .uuid.uh:69baf057f8902f9da24f13ea38fce27c doesn't exist.").: DockerError: Docker internal error: 500 Server Error: Internal Server Error ("failed to create endpoint zun-sandbox-efae9dd7-75aa-4897-927e-26f0409c0c6e on network afb2459d-a5a0-45c7-9c30-eaec2d403233-df6ffdb34f20496692d221f2f2f7faaf: remote: Neutron net associated with identifier kuryr.net.uuid.lh:3a784f711e0eaf4673f883f93e4d5d76,kuryr.net.uuid.uh:69baf057f8902f9da24f13ea38fce27c doesn't | 03:59 |
| zsli_ | exist."). | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager Traceback (most recent call last): | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager File "/opt/stack/zun/zun/compute/manager.py", line 99, in _do_container_create | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager image=sandbox_image) | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager File "/opt/stack/zun/zun/container/docker/driver.py", line 557, in create_sandbox | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager return sandbox['Id'] | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager self.gen.throw(type, value, traceback) | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager File "/opt/stack/zun/zun/container/docker/utils.py", line 40, in docker_client | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager raise exception.DockerError(error_msg=six.text_type(e)) | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager DockerError: Docker internal error: 500 Server Error: Internal Server Error ("failed to create endpoint zun-sandbox-efae9dd7-75aa-4897-927e-26f0409c0c6e on network afb2459d-a5a0-45c7-9c30-eaec2d403233-df6ffdb34f20496692d221f2f2f7faaf: remote: Neutron net associated with identifier kuryr.net.uuid.lh:3a784f711e0eaf4673f883f93e4d5d76,kuryr.net.uuid.uh:69baf057f8902f9da24f13ea38fce27c doesn't exist."). | 03:59 |
| hongbin | zsli_: fyi, you could use this one for paste: http://paste.openstack.org/ | 03:59 |
| zsli_ | 2017-06-26 17:11:33.623 TRACE zun.compute.manager | 03:59 |
| zsli_ | i debugged the problem | 03:59 |
| zsli_ | seems kuryrlib-network send the add tag request to neutron. | 04:00 |
| hongbin | zsli_: ok, how neutron response it? | 04:01 |
| zsli_ | I also seen the neutron get the request, but cannot find the kuryr.net.uuid tag in db. | 04:01 |
| hongbin | i see | 04:01 |
| hongbin | could you list it manually? | 04:01 |
| zsli_ | then add tag failed, kuryrlibnetwor cannot find the proper network by tag. | 04:01 |
| mkrai | Is our gate also broken with this error? | 04:01 |
| zsli_ | yes, the network did not get the tag set. | 04:01 |
| mkrai | gate seems to work fine | 04:02 |
| hongbin | yes, our gate was broken by a neutron tag issue in before | 04:02 |
| zsli_ | hongbin: paste link here http://paste.openstack.org/show/613769/ | 04:02 |
| hongbin | zsli_: you have the kuryr log as well? | 04:03 |
| zsli_ | no, all log in the stdout. | 04:03 |
| hongbin | zsli_: also, could you type: "pip freeze" and paste the output to me as well | 04:04 |
| hongbin | ok | 04:04 |
| zsli_ | hongbin:http://paste.openstack.org/show/613770/ | 04:05 |
| zsli_ | hongbin: neutron update tag returned on this line https://github.com/openstack/neutron/blob/master/neutron/services/tag/tag_plugin.py#L121 | 04:09 |
| zsli_ | no tag updated to neutron db. | 04:09 |
| hongbin | zsli_: you might hit this bug: https://bugs.launchpad.net/zun/+bug/1694897 | 04:10 |
| openstack | Launchpad bug 1694897 in neutron "neutron tag xxx.xxx doesn't work correctly" [Low,Fix released] - Assigned to Ihar Hrachyshka (ihar-hrachyshka) | 04:10 |
| hongbin | zsli_: and your neutron server looks quite old, it might not pick up the fix yet | 04:12 |
| zsli_ | OK. i will rebase the patch and try again. | 04:12 |
| zsli_ | thanks hongbin. | 04:12 |
| hongbin | zsli_: np, perhaps, you could try the step in the bug, to see if you can reproduce it | 04:13 |
| hongbin | $ neutron tag-add --resource-type network --resource private --tag first.second | 04:13 |
| hongbin | $ neutron net-show private | 04:13 |
| zsli_ | ok. seem the same bug. | 04:13 |
| hongbin | ok | 04:13 |
| hongbin | yes, then possibly, you hit this bug | 04:14 |
| zsli_ | checked my devstack neutron code. it's 2017.3.16 | 04:14 |
| *** zsli_ has quit IRC | 04:14 | |
| hongbin | ok | 04:14 |
| *** zsli_ has joined #openstack-zun | 04:14 | |
| hongbin | mkrai: you have a docker issue? | 04:15 |
| zsli_ | thanks hongbin, will ping you later if there still problems. | 04:15 |
| hongbin | zsli_: sure | 04:15 |
| hongbin | mkrai: sorry, i have to leave now, ping me later or open a bug if you want, i will get back to it | 04:17 |
| *** hongbin has quit IRC | 04:22 | |
| *** zhurong has joined #openstack-zun | 04:36 | |
| *** diga has quit IRC | 04:56 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 05:12 | |
| *** kiennt has quit IRC | 05:43 | |
| *** kiennt has joined #openstack-zun | 05:55 | |
| openstackgerrit | Kevin Zhao proposed openstack/zun master: Remove the items for disable neutron https://review.openstack.org/477765 | 06:07 |
| openstackgerrit | Merged openstack/zun-ui master: Fix image pull action https://review.openstack.org/477458 | 06:16 |
| openstackgerrit | Kien Nguyen proposed openstack/zun master: Fix missing import _ https://review.openstack.org/477767 | 06:17 |
| *** kiennt has quit IRC | 06:41 | |
| openstackgerrit | Shu Muto proposed openstack/zun-ui master: Clean up doc/ and releasenotes/ https://review.openstack.org/477794 | 07:33 |
| *** egonzalez has joined #openstack-zun | 07:41 | |
| openstackgerrit | Kevin Zhao proposed openstack/zun master: Add websockify to requirements https://review.openstack.org/477802 | 07:47 |
| openstackgerrit | Shu Muto proposed openstack/zun-ui master: Clean up doc/ and releasenotes/ https://review.openstack.org/477794 | 07:51 |
| caowei | hongbin: hi | 08:05 |
| kevinz | caowei: I guess hongbin is sleeping :-) | 08:14 |
| caowei | haha | 08:15 |
| openstackgerrit | Shu Muto proposed openstack/zun-ui master: Improve docs build https://review.openstack.org/477813 | 08:18 |
| caowei | kevinz: can you help me? http://paste.openstack.org/show/613795/ | 08:23 |
| kevinz | caowei: OK I will try to help, but I'm not familiar with Kuryr :-) | 08:25 |
| kevinz | caowei: btw if you want to ping hongbin, I think in the morning he is always online | 08:27 |
| caowei | thanks | 08:30 |
| *** kzaitsev_ws has joined #openstack-zun | 08:37 | |
| zsli_ | caowei: could you paste the neutron-server log, seems the neutron port bind failed. | 08:46 |
| zsli_ | I just solved a problem like it, due to the neutron ovs-agent start up failed as there missed ovsdbapp lib. | 08:47 |
| caowei | http://paste.openstack.org/show/613801/ | 08:47 |
| zsli_ | cd neutron, and pip install -r requirements.txt and restart the neutron ovs-agent service. | 08:48 |
| zsli_ | the problem sovled. | 08:48 |
| *** diga has joined #openstack-zun | 08:50 | |
| zsli_ | caowei: Refusing to bind port 37714950-7141-4e8f-a241-5facfb9f845c to dead agent | 08:50 |
| zsli_ | could you check if the ovs-agent is alive、 | 08:50 |
| caowei | [root@kollaallinone kuryr]# neutron agent-list | 08:50 |
| caowei | neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. | 08:50 |
| caowei | +--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+ | 08:50 |
| caowei | | id | agent_type | host | availability_zone | alive | admin_state_up | binary | | 08:50 |
| caowei | +--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+ | 08:50 |
| caowei | | 8fe1defa-d400-4411-8a17-f8d09d6bf2cf | L3 agent | kollaallinone | nova | :-) | True | neutron-l3-agent | | 08:50 |
| caowei | | 9cea4508-4375-41e2-86d7-4fdab99d78e7 | Metadata agent | kollaallinone | | :-) | True | neutron-metadata-agent | | 08:50 |
| caowei | | d7d2b2a1-83d7-42d7-85cd-35f02d97dbaf | Open vSwitch agent | kollaallinone | | xxx | True | neutron-openvswitch-agent | | 08:50 |
| caowei | | fbb7a260-1556-4fbe-a29e-040238d038f3 | DHCP agent | kollaallinone | nova | :-) | True | neutron-dhcp-agent | | 08:50 |
| caowei | +--------------------------------------+--------------------+---------------+-------------------+-------+----------------+---------------------------+ | 08:50 |
| zsli_ | seems the ovs-agent is not ok. | 08:51 |
| zsli_ | restart it and try again? check the log if there is error? | 08:51 |
| openstackgerrit | feng.shengqin proposed openstack/zun master: Use default value about filters in object https://review.openstack.org/477836 | 08:55 |
| caowei | zsli_: neutron-openvswitch-agent.log http://paste.openstack.org/show/613802/ | 08:57 |
| zsli_ | caowei:could you check if the ovs-dbserver is alive? | 09:00 |
| zsli_ | seems agent command timeout, ovs-dbserver not alive. | 09:00 |
| zsli_ | make sure ovs-dbserver and ovs-vswitchd is alive. | 09:01 |
| *** kevzha01_ has joined #openstack-zun | 09:02 | |
| egonzalez | caowei, are you using kolla? when was the last time updated the repo? have properly configured etcd on each compute node docker systemd service? | 09:05 |
| *** kevinz has quit IRC | 09:06 | |
| caowei | [root@kollaallinone _data]# ps -ef|grep docker | 09:10 |
| caowei | root 877 1 0 6月26 ? 00:11:05 /usr/bin/dockerd --cluster-store=etcd://192.168.151.6:2379 --cluster-advertise=192.168.151.6:2375 | 09:10 |
| *** zsli_ has quit IRC | 09:11 | |
| *** Shunli has joined #openstack-zun | 09:11 | |
| caowei | egonzalez: I looked at the patches that you modified | 09:11 |
| egonzalez | caowei, missing -H tcp://<DOCKER_SERVICE_IP>:2375 -H unix:///var/run/docker.sock | 09:12 |
| egonzalez | to allow remote connection into docker api | 09:12 |
| *** zsli_ has joined #openstack-zun | 09:15 | |
| *** zsli_ has quit IRC | 09:16 | |
| *** Shunli has quit IRC | 09:17 | |
| caowei | egonzalez: [root@kollaallinone docker]# ps -ef|grep docker | 09:19 |
| caowei | root 25600 1 5 09:18 ? 00:00:01 /usr/bin/dockerd -H tcp://192.168.151.6:2375 --cluster-store=etcd://192.168.151.6:2379 --cluster-advertise=192.168.151.6:2375 | 09:19 |
| caowei | egonzalez: change done | 09:20 |
| egonzalez | caowei, that should work if etcd is running on each compute node, guess yes since kuryr docker network is created global | 09:20 |
| egonzalez | caowei, do kuryr images have openvswitch and bridge-utils packages installed? | 09:21 |
| *** Shunli has joined #openstack-zun | 09:21 | |
| Shunli | caowei: hexchat disconnected just now | 09:22 |
| Shunli | did your problem get sovled? | 09:22 |
| Shunli | caowei: time to leave, ttyl. | 09:24 |
| *** Shunli has quit IRC | 09:27 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:29 | |
| caowei | egonzalez: (kuryr)[root@kollaallinone /]# rpm -qa|grep openvs | 09:31 |
| caowei | openvswitch-2.6.1-4.1.git20161206.el7.x86_64 | 09:31 |
| caowei | (kuryr)[root@kollaallinone /]# rpm -qa|grep brid | 09:31 |
| caowei | bridge-utils-1.5-9.el7.x86_64 | 09:31 |
| caowei | (kuryr)[root@kollaallinone /]# | 09:31 |
| openstackgerrit | feng.shengqin proposed openstack/zun master: Optimize msg when exception https://review.openstack.org/477849 | 09:32 |
| openstackgerrit | Van Hung Pham proposed openstack/zun master: Switch from oslosphinx to openstackdocstheme https://review.openstack.org/477850 | 09:32 |
| *** kiennt has joined #openstack-zun | 09:33 | |
| egonzalez | caowei, still not working allowing remote api to docker? | 09:34 |
| caowei | egonzalez: I have manually merged your patch. | 09:34 |
| egonzalez | caowei, are already merged in master | 09:35 |
| caowei | yes , [root@kollaallinone ~]# ps -ef|grep docker | 09:36 |
| caowei | root 1195 1 1 09:26 ? 00:00:05 /usr/bin/dockerd -H tcp://192.168.151.6:2375 -H unix:///var/run/docker.sock --cluster-store=etcd://192.168.151.6:2379 --cluster-advertise=192.168.151.6:2375 | 09:36 |
| egonzalez | caowei, i mean, still not working kuryr/zun ? | 09:38 |
| caowei | yes | 09:38 |
| caowei | egonzalez: I re-create the network, try it | 09:40 |
| egonzalez | caowei, let you know that network should not have dhcp enabled | 09:42 |
| caowei | egonzalez: http://paste.openstack.org/show/613795/ | 09:43 |
| caowei | egonzalez: This is my configuration. | 09:43 |
| *** kevzha01_ is now known as kevinz | 09:46 | |
| caowei | egonzalez: Can not enable dhcp agent? | 09:52 |
| egonzalez | caowei, as been told by kuryr people not | 09:52 |
| egonzalez | caowei, try disabling dhcp in the subnet | 09:52 |
| egonzalez | caowei, are you using centos/ubuntu source or binary? | 09:53 |
| caowei | egonzalez: source | 09:54 |
| egonzalez | caowei, will try with centos source now | 09:54 |
| caowei | egonzalez: yes, centos source | 09:55 |
| *** kiennt has quit IRC | 10:02 | |
| egonzalez | caowei, give me some minutes to deploy with that config, i'm copying commands to show you my config | 10:04 |
| *** caowei has quit IRC | 10:14 | |
| *** caowei has joined #openstack-zun | 10:14 | |
| *** caowei has quit IRC | 10:15 | |
| openstackgerrit | feng.shengqin proposed openstack/zun master: Use default value about filters in object https://review.openstack.org/477836 | 10:47 |
| *** kevinz has quit IRC | 11:15 | |
| openstackgerrit | feng.shengqin proposed openstack/zun master: Optimize _get_security_group_id https://review.openstack.org/477895 | 11:22 |
| openstackgerrit | feng.shengqin proposed openstack/zun master: Drop duplicated exception handle code from docker commit https://review.openstack.org/477902 | 11:54 |
| *** janki has joined #openstack-zun | 12:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/zun-ui master: Updated from global requirements https://review.openstack.org/478127 | 12:25 |
| *** janki has quit IRC | 12:34 | |
| openstackgerrit | bhavani proposed openstack/python-zunclient master: switch to openstackdocstheme https://review.openstack.org/478133 | 12:44 |
| *** diga has quit IRC | 13:20 | |
| *** lakerzhou has joined #openstack-zun | 13:22 | |
| *** lakerzhou has quit IRC | 13:27 | |
| *** lakerzhou has joined #openstack-zun | 13:44 | |
| *** hongbin has joined #openstack-zun | 13:47 | |
| *** lakerzhou has quit IRC | 14:02 | |
| *** lakerzhou has joined #openstack-zun | 14:18 | |
| *** lakerzhou has quit IRC | 14:27 | |
| *** lakerzhou has joined #openstack-zun | 15:18 | |
| *** lakerzhou has quit IRC | 15:44 | |
| *** egonzalez has quit IRC | 16:00 | |
| *** kzaitsev_ws has quit IRC | 16:39 | |
| *** lakerzhou has joined #openstack-zun | 16:58 | |
| *** lakerzhou has quit IRC | 17:29 | |
| *** lakerzhou has joined #openstack-zun | 18:31 | |
| *** adisky__ has quit IRC | 19:10 | |
| *** lakerzhou has quit IRC | 19:11 | |
| openstackgerrit | Hongbin Lu proposed openstack/python-zunclient master: Re-enable osc tempest tests https://review.openstack.org/477027 | 20:15 |
| openstackgerrit | Hongbin Lu proposed openstack/python-zunclient master: Re-enable osc tempest tests https://review.openstack.org/477027 | 20:15 |
| openstackgerrit | Hongbin Lu proposed openstack/python-zunclient master: Remove unused code from zunclient/common/apiclient https://review.openstack.org/477032 | 20:18 |
| openstackgerrit | Hongbin Lu proposed openstack/zun master: Minor refactor of get_security_group_ids https://review.openstack.org/477705 | 22:01 |
| openstackgerrit | Hongbin Lu proposed openstack/zun master: Refactor create/delete sandbox code https://review.openstack.org/478316 | 23:04 |
| openstackgerrit | Hongbin Lu proposed openstack/zun master: [WIP] Make sandbox container optional https://review.openstack.org/471634 | 23:08 |
| openstackgerrit | Hongbin Lu proposed openstack/zun master: [WIP] Make sandbox container optional https://review.openstack.org/471634 | 23:09 |
| *** hongbin has quit IRC | 23:11 | |
| *** vikasc has quit IRC | 23:26 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!