Friday, 2017-01-27

*** rcernin has joined #openstack-neutron00:00
*** Swami has quit IRC00:03
*** mriedem has quit IRC00:03
*** mriedem has joined #openstack-neutron00:04
*** harlowja has quit IRC00:04
*** tommylikehu_ has joined #openstack-neutron00:08
*** liangy has quit IRC00:09
*** Swami has joined #openstack-neutron00:10
openstackgerritArmando Migliaccio proposed openstack/neutron-specs: Ocata release postmortem  https://review.openstack.org/42599000:13
*** tommylikehu_ has quit IRC00:13
openstackgerritArmando Migliaccio proposed openstack/neutron-specs: Ocata release postmortem  https://review.openstack.org/42599000:13
*** Leo_ has quit IRC00:22
*** fnaval has quit IRC00:23
openstackgerritJaivish Kothari(janonymous) proposed openstack/os-vif: Fix broken Link  https://review.openstack.org/42599400:23
*** masaki has joined #openstack-neutron00:25
*** mlavalle has quit IRC00:27
*** abhiraut has quit IRC00:27
*** john-davidge has joined #openstack-neutron00:28
*** john-davidge has quit IRC00:32
*** tommylikehu_ has joined #openstack-neutron00:33
*** abhiraut has joined #openstack-neutron00:33
openstackgerritJaivish Kothari(janonymous) proposed openstack/os-vif: Removing Deprecated hacking Check  https://review.openstack.org/42599600:36
*** rajinir has quit IRC00:36
openstackgerritSwaminathan Vasudevan proposed openstack/neutron: DVR: Add static routes to FIP namespace  https://review.openstack.org/30806800:37
*** boden has quit IRC00:40
*** abhiraut has quit IRC00:41
*** MasterOfBugs has quit IRC00:42
*** PramodJ has joined #openstack-neutron00:42
*** pramodrj07 has quit IRC00:42
*** iyamahat_ has joined #openstack-neutron00:42
*** iyamahat has quit IRC00:42
*** MasterOfBugs has joined #openstack-neutron00:42
*** jose-phillips has quit IRC00:43
*** thorst_ has joined #openstack-neutron00:45
*** hfu has joined #openstack-neutron00:47
*** radhikam has quit IRC00:47
*** claudiub has quit IRC00:47
*** abhiraut has joined #openstack-neutron00:48
*** hfu has quit IRC00:49
openstackgerritYAMAMOTO Takashi proposed openstack/neutron-vpnaas: gate_hook: Add "tempest" case  https://review.openstack.org/42600000:51
*** rcernin has quit IRC00:51
*** numans_ has joined #openstack-neutron00:52
*** belharar_ has quit IRC00:53
*** numans has quit IRC00:53
*** rcernin has joined #openstack-neutron00:54
*** thorst_ has quit IRC00:55
*** rcernin has quit IRC00:55
*** rcernin has joined #openstack-neutron00:55
*** abhiraut has quit IRC00:58
*** abhiraut has joined #openstack-neutron01:01
*** abhiraut has quit IRC01:01
*** ventifus has quit IRC01:04
*** ijw has quit IRC01:05
openstackgerritNate Johnston proposed openstack/neutron-fwaas: Use neutron-lib definition of neutron-fwaas API  https://review.openstack.org/42147201:13
Sukhdev_ijw: Ping01:21
*** leitan has joined #openstack-neutron01:22
*** mriedem is now known as mriedem_afk01:23
openstackgerritYAMAMOTO Takashi proposed openstack/neutron-lib: Add a space after comma  https://review.openstack.org/42099101:25
openstackgerritYAMAMOTO Takashi proposed openstack/neutron-lib: Add FIP64 extension from networking-midonet  https://review.openstack.org/41695101:25
openstackgerritYAMAMOTO Takashi proposed openstack/neutron-lib: Add router-interface-fip extension from networking-midonet  https://review.openstack.org/41000101:25
openstackgerritYAMAMOTO Takashi proposed openstack/neutron-lib: Migrate logging-resource extension from networking-midonet  https://review.openstack.org/40797401:25
*** ijw has joined #openstack-neutron01:26
*** leitan has quit IRC01:27
*** thorst_ has joined #openstack-neutron01:27
*** iranzo has joined #openstack-neutron01:28
*** iranzo has quit IRC01:28
*** iranzo has joined #openstack-neutron01:28
*** manheim has quit IRC01:30
*** ijw has quit IRC01:30
openstackgerritNate Johnston proposed openstack/neutron: IPtables enhancement for co-existence of SG and FWaaS v2 drivers  https://review.openstack.org/34817701:39
*** iranzo has quit IRC01:40
*** liangy has joined #openstack-neutron01:44
openstackgerritNate Johnston proposed openstack/neutron: IPtables enhancement for co-existence of SG and FWaaS v2 drivers  https://review.openstack.org/34817701:45
*** Swami_ has joined #openstack-neutron01:45
*** Swami_ has quit IRC01:46
*** tommylikehu_ has quit IRC01:47
*** Swami has quit IRC01:48
*** iwamoto has joined #openstack-neutron01:48
*** tommylikehu_ has joined #openstack-neutron01:48
*** tommylikehu_ has quit IRC01:53
*** tbachman has quit IRC01:54
*** Sukhdev_ has quit IRC01:54
*** tbachman has joined #openstack-neutron01:55
*** tbachman has quit IRC01:55
*** tbachman has joined #openstack-neutron01:55
*** tbachman has quit IRC01:59
*** hichihara has joined #openstack-neutron02:01
openstackgerritMerged openstack/neutron: Transition qos notification driver into qos driver  https://review.openstack.org/39665102:01
*** hichihara has quit IRC02:03
*** tbachman has joined #openstack-neutron02:04
*** hfu has joined #openstack-neutron02:05
*** tommylikehu_ has joined #openstack-neutron02:07
openstackgerritMerged openstack/neutron: adds support for vhost user reconnect.  https://review.openstack.org/34499702:08
*** tbachman_ has joined #openstack-neutron02:08
openstackgerritOpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements  https://review.openstack.org/42364502:09
*** tbachman has quit IRC02:09
*** tbachman_ is now known as tbachman02:09
openstackgerritMerged openstack/neutron: Use unique subnetpools in SubnetPoolPrefixDbObjectTestCase  https://review.openstack.org/42540902:09
*** MasterOfBugs has quit IRC02:11
*** dims_ has joined #openstack-neutron02:14
*** PramodJ has quit IRC02:14
*** dims has quit IRC02:14
*** jhershbe has joined #openstack-neutron02:15
*** thorst_ has quit IRC02:18
*** thorst_ has joined #openstack-neutron02:20
*** tommylikehu_ has joined #openstack-neutron02:21
*** tommylikehu_ has quit IRC02:22
*** tommylikehu_ has joined #openstack-neutron02:22
openstackgerritKevin Benton proposed openstack/neutron: Stop passing datapath as 'may_exist' argument  https://review.openstack.org/42603202:27
*** john-davidge has joined #openstack-neutron02:29
*** yamahata has quit IRC02:29
*** iyamahat_ has quit IRC02:29
*** kevo has quit IRC02:32
*** john-davidge has quit IRC02:33
*** baoli has joined #openstack-neutron02:35
*** gvrangan has quit IRC02:35
kevinbentonyamamoto: here :)02:35
kevinbentonyamamoto: i also respun https://review.openstack.org/#/c/419232/ . I remember why i had removed the fixture before02:36
kevinbentonyamamoto: the deepcopy it was using broke trying to deepcopy a weakref02:36
*** fzdarsky_ has joined #openstack-neutron02:36
kevinbentonyamamoto: i fixed it to use a regular copy02:36
yamamotoah, it makes sense.02:36
yamamotoi'll look02:36
kevinbentonyamamoto: i also have some ovs related things if you have a sec02:37
kevinbentonquestions mainly02:37
yamamotosure02:38
kevinbentonyamamoto: first, i found a small bug. not sure if it was really impacting anything or not. https://review.openstack.org/#/c/426032/02:38
*** jhershbe has quit IRC02:38
kevinbentonyamamoto: second, what leads to these "datapath invalid exceptions" http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/screen-q-agt.txt?level=DEBUG#_2017-01-26_19_53_42_41002:39
*** fzdarsky|afk has quit IRC02:39
kevinbentonyamamoto: third, I noticed in that line out_group=4294967295,out_port=4294967295, are both unsigned 32 bit int wrap-arounds of -1 (the invalid port) is that a problem?02:40
*** absubram has joined #openstack-neutron02:43
yamamoto"datapath invalid" is from ryu.  it usually means the bridge disconnected.02:43
*** baoli has quit IRC02:45
kevinbentonyamamoto: the openflow connection? or the bridge was removed?02:45
kevinbentonyamamoto: ah, which brings me to my last question02:46
kevinbentonyamamoto: 2017-01-26T19:53:25.306Z|00814|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 5 seconds, disconnecting02:46
kevinbentonyamamoto: http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/openvswitch/ovs-vswitchd.txt.gz02:46
kevinbentonyamamoto: corresponds to that same time02:46
*** dims_ has quit IRC02:47
yamamotoopenflow has special constant like 0xffffffff.  meaning any/all or something like that.02:47
kevinbentonyamamoto: is it possible there is a blocking issue where we don't respond to a probe on the OF channel?02:47
*** d-bark has joined #openstack-neutron02:47
kevinbentonyamamoto: ok02:47
*** absubram has quit IRC02:48
yamamotosure, if there's a blocking issue, it would explain those messages.02:49
*** absubram has joined #openstack-neutron02:49
kevinbentonyamamoto: it seems that command blocked everything02:50
*** dims has joined #openstack-neutron02:50
kevinbentonyamamoto: agent freezes for 20 seconds when it issues here http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/screen-q-agt.txt?level=DEBUG#_2017-01-26_19_53_18_84802:50
kevinbentonyamamoto: and during that freeze we get the inactivity probes02:50
*** tommylikehu_ has quit IRC02:52
*** tommylikehu_ has joined #openstack-neutron02:52
kevinbentonyamamoto: what i guess we can't see is if there is a hard freeze in some part of RYU or if RYU is properly yielding out and then ovsdb native is experiencing a block02:53
*** TMM has quit IRC02:54
*** TMM has joined #openstack-neutron02:54
yamamotosure, but i don't have any idea where it is.  it can be even kernel/scheduler issue.02:55
*** ijw has joined #openstack-neutron02:55
*** yuanying has quit IRC02:56
yamamotoiwamoto: do you have any idea?  i vaguely remember kakuma investigated similar pause, but it was long ago.02:56
kevinbentoni need to step away for a bit. yamamoto, iwamoto: send me any pointers if you do thing of possible causes.02:58
*** tbachman has quit IRC02:58
iwamotohi all02:59
iwamotoi need more context to ask kakuma-san02:59
*** ijw has quit IRC02:59
yamamotoiwamoto: no more context than what you can see in the backlog :-)03:01
*** tommylikehu_ has quit IRC03:02
yamamotomysterious pause of ovs-agent causes ovs disconnect.03:02
iwamototurning on ryu debug logging might reveal something03:04
*** tommylikehu_ has joined #openstack-neutron03:06
*** kuromagi^ has joined #openstack-neutron03:07
*** baoli has joined #openstack-neutron03:08
*** kuromagi has quit IRC03:10
kevinbentoniwamoto: what's the best way to do that?03:10
kevinbentoniwamoto: regular logging level config?03:10
iwamotokevinbenton: ryu logging was too verbose and armando changed that a while ago03:11
iwamotokevinbenton, 5743ed4679d43c325c5b3a2a4955088e35018c9703:12
*** yamahata has joined #openstack-neutron03:13
iwamotokakuma-san says increasing ovs timeout value helped :)03:16
*** numans_ has quit IRC03:22
*** numans has joined #openstack-neutron03:22
*** leitan has joined #openstack-neutron03:23
iwamotosudo ovs-vsctl set Controller br-int inactivity_probe=3000003:24
*** leitan has quit IRC03:28
*** mriedem_afk is now known as mriedem03:30
*** yarkot has joined #openstack-neutron03:31
*** pbandark has joined #openstack-neutron03:32
*** tbachman has joined #openstack-neutron03:37
*** thorst_ has quit IRC03:40
*** psahoo has joined #openstack-neutron03:41
*** tommylikehu_ has quit IRC03:41
*** amotoki has quit IRC03:43
*** gouthamr has quit IRC03:43
*** liangy has quit IRC03:47
*** kevo has joined #openstack-neutron04:12
*** nicolasbock has quit IRC04:17
*** mtreinish has quit IRC04:22
*** mtreinish has joined #openstack-neutron04:23
*** john-davidge has joined #openstack-neutron04:29
*** mdnadeem has joined #openstack-neutron04:30
*** nyechiel_ has joined #openstack-neutron04:32
*** john-davidge has quit IRC04:34
*** nyechiel_ has quit IRC04:45
*** iyamahat has joined #openstack-neutron04:46
*** tommylikehu_ has joined #openstack-neutron04:46
*** tommylikehu_ has quit IRC04:49
*** ayogi has joined #openstack-neutron04:50
*** radhikam has joined #openstack-neutron04:50
*** baoli_ has joined #openstack-neutron04:53
*** tommylikehu_ has joined #openstack-neutron04:55
*** baoli has quit IRC04:55
*** absubram has quit IRC04:55
*** reedip_ has joined #openstack-neutron04:56
*** udesale has joined #openstack-neutron04:58
openstackgerritReedip proposed openstack/neutron-fwaas: TrivialFix: Added link for modindex  https://review.openstack.org/42605504:59
*** iyamahat has quit IRC05:01
*** yamahata has quit IRC05:01
iwamotokevinbenton, yamamoto: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20invalid%5C%2205:04
iwamotonot good05:04
*** armax has quit IRC05:05
iwamotois there a bug report?05:05
*** tommylikehu_ has quit IRC05:06
*** nyechiel_ has joined #openstack-neutron05:08
yamamotoiwamoto: dunno05:08
*** ratailor has joined #openstack-neutron05:12
*** reedip_ has quit IRC05:17
*** janki has joined #openstack-neutron05:19
*** ataraday_ has joined #openstack-neutron05:23
*** leitan has joined #openstack-neutron05:25
*** gvrangan has joined #openstack-neutron05:27
*** tbachman has quit IRC05:29
*** leitan has quit IRC05:29
*** shausy has joined #openstack-neutron05:29
*** jchhatbar has joined #openstack-neutron05:30
*** janki has quit IRC05:33
*** psahoo has quit IRC05:35
openstackgerritYaohua Yan proposed openstack/neutron-fwaas: Deleting firewall in DB is directly operated by plugin, instead of agents' callback  https://review.openstack.org/42606705:41
openstackgerritGhanshyam Mann proposed openstack/neutron-fwaas: Switch to decorators.idempotent_id  https://review.openstack.org/42606905:41
*** david-lyle has quit IRC05:46
*** psahoo has joined #openstack-neutron05:51
*** karthiks has joined #openstack-neutron05:53
*** nmathew has joined #openstack-neutron05:55
*** Jack_V has joined #openstack-neutron05:56
*** thorst_ has joined #openstack-neutron05:56
*** wolverineav has joined #openstack-neutron05:57
*** absubram has joined #openstack-neutron05:58
*** jchhatbar is now known as janki06:00
*** thorst_ has quit IRC06:01
*** nagarjung has joined #openstack-neutron06:01
*** wolverineav has quit IRC06:02
*** absubram has quit IRC06:02
kevinbentoniwamoto, yamamoto: the bug is manifesting in different ways06:03
kevinbentonI think there is a bug under ovs timeouts06:03
kevinbentonSetting the inactivity probe high may be a reasonable workaround06:04
kevinbentonBut it seems like that's just working around the issue that something is blocking the agent completely06:04
*** gvrangan has quit IRC06:06
yamamotoi agree06:08
*** karthiks has quit IRC06:09
yamamotomaking log more verbose might help to isolate where it's blocking06:09
*** haleyb has quit IRC06:09
iwamotoif something is holding eventlet threads, it can be other than ryu-related06:11
iwamotobut we can try06:12
openstackgerritGhanshyam Mann proposed openstack/neutron-dynamic-routing: Switch to decorators.idempotent_id  https://review.openstack.org/42608606:12
*** pgadiya has joined #openstack-neutron06:14
iwamoto"no response to inactivity probe" shows a lot of hits. and aren't necessarily cause test failures06:15
kevinbentoniwamoto: but if we get a disconnect during a command it may lead to a failure06:16
*** ranjithd has quit IRC06:16
kevinbentoniwamoto: like the data path invalid one06:16
*** bobmel has quit IRC06:16
*** ranjithd has joined #openstack-neutron06:17
yamamotomaybe we can improve recovery06:18
kevinbentonyamamoto: does ryu have hierarchical exceptions so we could tell what errors are connection related?06:19
yamamotoit has some hierarchy but probably not useful for this purpose06:21
*** sridharg has joined #openstack-neutron06:22
*** claudiub has joined #openstack-neutron06:25
*** claudiub|2 has joined #openstack-neutron06:28
*** john-davidge has joined #openstack-neutron06:30
*** claudiub has quit IRC06:31
*** wolverineav has joined #openstack-neutron06:33
*** Sukhdev has joined #openstack-neutron06:34
*** john-davidge has quit IRC06:34
*** iyamahat has joined #openstack-neutron06:36
*** gcheresh_ has joined #openstack-neutron06:38
*** wolverineav has quit IRC06:38
*** psahoo has quit IRC06:40
*** donghao has joined #openstack-neutron06:41
*** horms has joined #openstack-neutron06:42
*** iyamahat has quit IRC06:47
*** moshele has joined #openstack-neutron06:48
*** wolverineav has joined #openstack-neutron06:50
*** gvrangan has joined #openstack-neutron06:50
*** horms has quit IRC06:51
*** eezhova has joined #openstack-neutron06:52
*** pgadiya_ has joined #openstack-neutron06:53
*** wolverineav has quit IRC06:54
*** sputnik13 has quit IRC06:55
*** sputnik13 has joined #openstack-neutron06:56
*** pgadiya has quit IRC06:57
*** moshele has quit IRC06:57
*** karthiks has joined #openstack-neutron06:58
*** psahoo has joined #openstack-neutron07:00
*** gcheresh_ has quit IRC07:02
openstackgerritMerged openstack/neutron: Fix a bad docstring in provisioning blocks module  https://review.openstack.org/42530807:04
*** yamamoto has quit IRC07:05
*** radhikam has quit IRC07:06
*** amotoki has joined #openstack-neutron07:07
*** hfu has quit IRC07:09
*** Sukhdev has quit IRC07:10
*** adriant has quit IRC07:11
kevinbentoniwamoto: how does RYU respond to those probes. is it using eventlet?07:15
kevinbentoniwamoto: to maintain a responder?07:15
*** hfu has joined #openstack-neutron07:17
*** tobias_fiberdata has joined #openstack-neutron07:20
*** wolverineav has joined #openstack-neutron07:21
*** rcernin has quit IRC07:24
*** leitan has joined #openstack-neutron07:26
*** wolverineav has quit IRC07:26
*** haleyb has joined #openstack-neutron07:26
iwamotokevinbenton, it's done around ryu.controller.ofp_handler.OFPHandler iirc. RyuApp is a eventlet thread07:26
*** donghao has quit IRC07:26
*** andreas_s has joined #openstack-neutron07:26
*** bbzhao has quit IRC07:27
*** korzen has joined #openstack-neutron07:27
*** bzhao has joined #openstack-neutron07:28
*** leitan has quit IRC07:30
*** gkadam has joined #openstack-neutron07:30
*** wolverineav has joined #openstack-neutron07:31
*** eezhova has quit IRC07:36
*** wolverineav has quit IRC07:36
*** gkadam has quit IRC07:42
*** gkadam has joined #openstack-neutron07:45
*** gkadam has quit IRC07:46
*** gkadam has joined #openstack-neutron07:47
*** ralonsoh has joined #openstack-neutron07:50
*** rubasov has joined #openstack-neutron07:55
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron: Enforce port QoS policies for all ports.  https://review.openstack.org/42528007:55
*** donghao has joined #openstack-neutron07:56
*** donghao has quit IRC07:56
*** thorst_ has joined #openstack-neutron07:57
openstackgerritThomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie  https://review.openstack.org/42575607:58
*** bfernando has joined #openstack-neutron07:59
*** wolverineav has joined #openstack-neutron08:01
*** yamamoto has joined #openstack-neutron08:01
*** thorst_ has quit IRC08:01
*** nyechiel_ has quit IRC08:02
*** amotoki has quit IRC08:05
*** wolverineav has quit IRC08:05
*** horms has joined #openstack-neutron08:08
*** mickeys has quit IRC08:14
*** mickeys has joined #openstack-neutron08:14
*** eezhova has joined #openstack-neutron08:15
*** john-davidge has joined #openstack-neutron08:16
*** fzdarsky_ is now known as fzdarsky08:16
*** gkadam has quit IRC08:16
*** john-davidge has quit IRC08:17
*** john-davidge has joined #openstack-neutron08:17
*** jpena|off is now known as jpena08:19
*** wolverineav has joined #openstack-neutron08:20
*** wolverineav has quit IRC08:25
*** gkadam has joined #openstack-neutron08:26
*** panda|Zz is now known as panda08:26
*** manheim has joined #openstack-neutron08:27
*** donghao has joined #openstack-neutron08:27
*** wolverineav has joined #openstack-neutron08:30
*** donghao has quit IRC08:31
openstackgerritYushiro FURUKAWA proposed openstack/neutron-specs: (Operator-only) Logging API for security groups  https://review.openstack.org/20350908:31
*** pmannidi has quit IRC08:32
*** jpena is now known as jpena|off08:32
*** nyechiel_ has joined #openstack-neutron08:34
*** wolverineav has quit IRC08:35
*** horms has quit IRC08:37
*** gcheresh_ has joined #openstack-neutron08:38
*** jpena|off is now known as jpena08:39
*** nyechiel_ has quit IRC08:41
*** manheim has quit IRC08:47
*** wolverineav has joined #openstack-neutron08:50
*** iranzo has joined #openstack-neutron08:50
*** iranzo has quit IRC08:50
*** iranzo has joined #openstack-neutron08:50
*** matrohon has joined #openstack-neutron08:50
*** wolverineav has quit IRC08:54
*** yamamoto has quit IRC08:58
*** zzzeek has quit IRC09:00
*** wolverineav has joined #openstack-neutron09:00
*** rmart04 has joined #openstack-neutron09:01
*** zzzeek has joined #openstack-neutron09:01
*** ygbo has joined #openstack-neutron09:01
*** iwamoto has quit IRC09:03
openstackgerritDaniel Alvarez proposed openstack/neutron: Include port_security check in fullstack tests  https://review.openstack.org/42581209:03
dalvarezajo bcafarel kevinbenton ^  may you have a look please? thanks a lot09:04
*** wolverineav has quit IRC09:04
bcafarelno more evil whitespace left? :)09:05
*** TMM has quit IRC09:06
*** claudiub has joined #openstack-neutron09:08
*** kevo has quit IRC09:09
*** wolverineav has joined #openstack-neutron09:10
*** claudiub|2 has quit IRC09:11
*** manheim has joined #openstack-neutron09:14
*** wolverineav has quit IRC09:14
*** Miouge has joined #openstack-neutron09:15
*** jlibosva has joined #openstack-neutron09:15
*** gkadam has quit IRC09:15
*** manheim has quit IRC09:16
Miougeajo: I can’t figure out why gate-neutron-dsvm-functional-ubuntu-xenial fails on https://review.openstack.org/#/c/42521809:17
MiougeAny tips?09:17
*** neiljerram has joined #openstack-neutron09:17
*** jlibosva has quit IRC09:18
*** oreillyd_ has joined #openstack-neutron09:20
*** masaki has quit IRC09:20
matrohonHi all09:21
*** masaki has joined #openstack-neutron09:21
*** efoley has joined #openstack-neutron09:22
matrohonwe have this bug on kolla :09:22
matrohonhttps://bugs.launchpad.net/kolla/+bug/165579409:22
openstackLaunchpad bug 1655794 in kolla "dnsmasq dhcp lease is not cleaned up after instance termination" [Undecided,New]09:22
matrohonI'd like to know if there is a corresponding bug on neutron09:22
*** masaki has quit IRC09:23
*** gkadam has joined #openstack-neutron09:24
*** moshele has joined #openstack-neutron09:26
*** jlibosva has joined #openstack-neutron09:26
*** leitan has joined #openstack-neutron09:27
*** vijaykc4 has joined #openstack-neutron09:27
*** donghao has joined #openstack-neutron09:28
*** manheim has joined #openstack-neutron09:28
*** wolverineav has joined #openstack-neutron09:29
*** lucas-afk is now known as lucasagomes09:32
*** leitan has quit IRC09:32
*** tmorin has quit IRC09:32
*** donghao has quit IRC09:33
*** wolverineav has quit IRC09:34
*** amotoki has joined #openstack-neutron09:35
*** karthiks has quit IRC09:35
*** horms has joined #openstack-neutron09:36
*** mickeys has quit IRC09:39
*** wolverineav has joined #openstack-neutron09:39
*** gkadam has quit IRC09:41
*** mhickey has joined #openstack-neutron09:41
ralonsohMiouge: this error in neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_bad_address_allocation is unrelated with your patch09:41
*** gkadam has joined #openstack-neutron09:41
MiougeIt sounds like that, but is it why Jenkins is voting -1?09:41
ralonsohMiouge: because there is an error, that's all09:42
*** wolverineav has quit IRC09:44
ralonsohMiouge: comment in your patch that the error in gate-neutron-dsvm-functional-ubuntu-xenial is unrelated, and be specific with the test failing09:44
Miougeralonsoh: ok, thank you for the help09:44
*** amotoki has quit IRC09:45
*** hfu has quit IRC09:47
*** hfu has joined #openstack-neutron09:48
*** hfu has quit IRC09:49
*** hfu has joined #openstack-neutron09:49
*** hfu has quit IRC09:49
*** rubasov has quit IRC09:50
*** hfu has joined #openstack-neutron09:50
*** hfu has quit IRC09:50
*** hfu has joined #openstack-neutron09:51
*** hfu has quit IRC09:51
*** karthiks has joined #openstack-neutron09:51
*** baoli_ has quit IRC09:51
*** hfu has joined #openstack-neutron09:52
*** hfu has quit IRC09:52
*** claudiub has quit IRC09:52
*** gabor_antal has joined #openstack-neutron09:52
ralonsohmoshele: ping09:52
mosheleralonsoh: pong09:53
ralonsohmoshele: about https://review.openstack.org/#/c/409526/09:53
*** iranzo has quit IRC09:53
ralonsohmoshele: in function "discover_devices"09:54
ralonsohmoshele: if you have two devices (dev1, dev2), both with the same pci address09:54
*** tommylikehu_ has joined #openstack-neutron09:54
*** claudiub has joined #openstack-neutron09:54
*** horms has quit IRC09:54
ralonsohmoshele: in "_create_emb_switch", you will execute two times this code09:55
*** hishh has joined #openstack-neutron09:55
ralonsohmoshele: and in both cases you'll have "embedded_switch = EmbSwitch(phys_net, dev_name, exclude_devices)" with the same info09:55
ralonsohmoshele: because, in your case, both PF can see all VFs09:55
*** nplanel has joined #openstack-neutron09:56
mosheleralonsoh: the dev_name is different09:56
ralonsohmoshele: I know09:56
ralonsohmoshele: but you'll have a list of all VFs with dev1 and another list with all VFs with dev209:56
ralonsohmoshele: how can you distinguish which VF belongs to each PFs??09:57
mosheleralonsoh: by the mac09:57
ralonsohmoshele: reading the VF mac gives you this info? how?09:58
*** thorst_ has joined #openstack-neutron09:58
mosheleralonsoh: because I now what is the mac I want to set  on the vm so I just need to find a vf that has that mac (but in the this case on both PFs)09:59
*** yamamoto has joined #openstack-neutron09:59
mosheleralonsoh:  I mean in this case I am looking on both PFs09:59
mosheleralonsoh:  it is a wired implementation of SR-IOV  in one of our legacy card :)10:00
ralonsohmoshele: is there a way to use the code you implemented but having two separate lists? or that's not what you are looking for?10:00
ralonsohmoshele: one list with VFs for dev1 and other with VFs of dev210:01
*** davidsha has joined #openstack-neutron10:01
mosheleralonsoh: sorry I don't follow10:02
ralonsohmoshele: I mean, when you build "embedded_switch = EmbSwitch(phys_net, dev_name, exclude_devices)"10:02
*** absubram has joined #openstack-neutron10:02
*** thorst_ has quit IRC10:02
ralonsohmoshele: "embedded_switch" contains all VFs, then remove those ones that doesn't belong to this device10:03
mosheleralonsoh: but the  EmbSwitch is per PF10:03
mosheleralonsoh: you want me to create an EmbSwitch that take 2 PFs?10:04
ralonsohmoshele: that's the problem: EmbSwitch is PER pci_address10:04
ralonsohmoshele: self.pci_slot_map[pci_slot] = embedded_switch10:05
*** ramishra has quit IRC10:05
*** TMM has joined #openstack-neutron10:05
*** absubram has quit IRC10:07
mosheleralonsoh: so EmbSwitch  represent a PF and I extend the mapping so that self.pci_slot_map  will hold list of EmbSwitch10:08
*** yamamoto has quit IRC10:08
*** yamamoto has joined #openstack-neutron10:08
openstackgerritOpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements  https://review.openstack.org/42364510:08
*** ramishra has joined #openstack-neutron10:09
*** wolverineav has joined #openstack-neutron10:09
ralonsohmoshele: I know what you want, but because both PFs have the same pci_address and you execute, for example, "clear_max_rate" filtering by pci_address10:10
*** horms has joined #openstack-neutron10:10
ralonsohmoshele: and embedded_switch.get_pci_device(pci_slot) will give you all VFs from both PFs, you don't need this change10:10
*** gkadam has quit IRC10:12
*** gkadam has joined #openstack-neutron10:12
*** wolverineav has quit IRC10:13
*** psahoo has quit IRC10:14
*** gcheresh_ has quit IRC10:14
mosheleralonsoh: yes but remember that if I have 3 VF on PF1 and 2 VF on PF2 I will have 5  VF on each PF. so if I will reset the rate on pci_address_x which is VF 3. it indeed will do it on both PF1 and PF2 but VF3 on PF2 is not in used10:15
sean-k-mooneymoshele: in linux netdevs to pci functions is a many to many relation ship with 1:1 being the most common. your legacy card hand sevarl phyicl fucntion netdevs associated with a single pci address. i have also seen the revices where a singel netdev spaned two pci endpoints and the embeded switch formed a lag/bond between them and passed that to the pf netdev10:15
sean-k-mooneynova tracks sriov allocation as a special case of pci-passhtough so it will not account for your legacy pf correctly as they both had the same pci address10:17
*** jlibosva has quit IRC10:17
*** d-bark has quit IRC10:17
*** yamamoto has quit IRC10:17
*** tommylikehu_ has quit IRC10:18
ralonsohmoshele: and take a look at your implementation. if you execute "_clear_rate" for a pci_address (pci_slot), you'll execute this twice in the same VF10:18
*** wolverineav has joined #openstack-neutron10:19
*** vijaykc4 has quit IRC10:19
*** yamamoto has joined #openstack-neutron10:20
moshelesean-k-mooney: yes we won't be able to do PF passhtough with those card, but with the pci regex support I can do VF passhtough https://review.openstack.org/#/c/199488/10:21
sean-k-mooneymoshele: why would you need the regex support for the connect x310:22
sean-k-mooneyyou ould have used the vendorid and product id with the pci adress10:23
sean-k-mooneyregex support just meand you have less entries in the pci whitelist10:23
*** wolverineav has quit IRC10:23
sean-k-mooneyactully for vf you just needed the vendor id and proct id of the vf which is different form the pf10:24
moshelesean-k-mooney: so I can use the second port for different for different physnet and also less entries in the pci whitelist10:24
openstackgerritArtur Korzeniewski proposed openstack/neutron: Addressing L3 HA keepalived failures in functional tests  https://review.openstack.org/42069310:24
sean-k-mooneymoshele: yes but you could have done that previously with out the regex just with more whitelist entriese so there is no hard dependecy on the regex support10:25
sean-k-mooneymoshele: its just a nice to have that make deployment eaisier10:25
*** yamamoto has quit IRC10:25
moshelesean-k-mooney: that true10:25
sean-k-mooneylooking at the code it looks like some vendor specific details have crept in10:27
*** karthiks has quit IRC10:27
sean-k-mooneythe sriov nic agent was never designed to work with intel nics originaly10:27
sean-k-mooneyit now does but you are assuming the embeed swtich is per pf.10:28
sean-k-mooneythat does not have to be the case10:28
sean-k-mooneyit can be per defvice10:28
*** wolverineav has joined #openstack-neutron10:28
moshelewhat do you mean per device ?10:29
*** gvrangan has quit IRC10:29
sean-k-mooneyper phyical pci addin card10:29
mosheleare you talking about multi-pf?10:30
sean-k-mooneyyes the embbed switch can span across multiple pfs10:30
sean-k-mooneyopenstack cant configur that but i am talking about at the silicon level10:31
*** murphy_zhao has quit IRC10:33
*** wolverineav has quit IRC10:33
sean-k-mooneyusing the flowdirector api im pretry sure you can directly switch traffic between 2 VFs allocated from different PFs via the embeded switch10:33
sean-k-mooneyprvided they are form the same addin card10:34
*** vijaykc4 has joined #openstack-neutron10:35
sean-k-mooneyif they are not from the same addin card they have to comunicate via the tor10:35
sean-k-mooneythe match api intel was working on would have exposed to the linux kernel i belive melenox were working on a similar switchdev? api in the linux kernel?10:36
*** vanzyla has joined #openstack-neutron10:36
*** psahoo has joined #openstack-neutron10:37
*** rossella_ has joined #openstack-neutron10:39
moshelewe have VF LAG, where you put both PFs in bond and also the traffic of the VF can go out from  2 VFs on different PFs although we passhtough  one of them  to the guest10:39
*** karthiks has joined #openstack-neutron10:39
*** tmorin has joined #openstack-neutron10:39
*** tommylikehu_ has joined #openstack-neutron10:40
*** tommylikehu_ has quit IRC10:42
moshelesean-k-mooney: now I am thinking if the agnet_required deprecation was a mistake :)10:42
mosheles/agnet_required/agent_required10:43
sean-k-mooneymoshele: hehe perhaps. or as the agent evolves we may require drivers to abstract so vendor specifics10:43
sean-k-mooneyi dont know about ye but we have a lot of silicon feature in currnt and past nics that arnt enabled in the linux driver10:44
sean-k-mooneye.g. you have to use the source forge driver, dpdk driver or windows driver to use them10:44
sean-k-mooneyfor example vlan trunking for sriov vf can only be enabled on intel niantic cards via the dpdk driver not the linux one...10:45
sean-k-mooneys/niantic/x520/10:47
moshelesean-k-mooney: so maybe we should define abstract driver and work pluggable drivers for that agent10:47
*** manheim has quit IRC10:47
moshelesean-k-mooney:  as I remember your dpdk driver requires other data than pci_address10:48
sean-k-mooneymoshele: i think that would be a good work item for pike. it would all others such as at&t who developed an sriov agent based on the dpdk driver to intergrate upstream in a common agent10:48
moshelesean-k-mooney: cool is it opensource can you point me to source code?10:49
openstackgerritArtur Korzeniewski proposed openstack/neutron: Add support for Keepalived VRRP health check  https://review.openstack.org/27354610:49
sean-k-mooneymoshele: yes it is opensouce i have not dug too deeply into how it works myself but ill grab the link its on github10:50
*** jhershbe has joined #openstack-neutron10:50
sean-k-mooneymoshele: this is the deamon its self https://github.com/att/vfd10:52
*** manheim has joined #openstack-neutron10:52
sean-k-mooneyi taught there was some openstack intergration code somewhate but i could be wrong10:53
moshelesean-k-mooney: I remember that code, someone in the SR-IOV sub meeting wanted to add support for this. So I suggested spec and POC code10:54
moshelesean-k-mooney: I never heard  from them since...10:54
sean-k-mooneyyes i think it provides an alternive command line that basicly the agent could invoke to do the configuration of the vfs10:54
sean-k-mooneyi have heard things on and off every so often but nothing since the winter break10:55
moshelesean-k-mooney: do you have some emails/names? maybe they started something10:56
*** dimak_ has joined #openstack-neutron10:56
*** yamamoto has joined #openstack-neutron10:57
*** vijaykc4 has quit IRC10:57
sean-k-mooneybackends like snabb switch are built around a similar concept in that they use the sriov silocon features in the nic and the embedded switch which they then expose as vhost-user interfaces to the guest.10:57
sean-k-mooneyam not to hand10:57
*** ociuhandu has quit IRC10:58
sean-k-mooneyralonsoh: do you?10:58
ralonsohsean-k-mooney: one sec10:58
sean-k-mooneymoshele: ralonsoh perhaps we should ping them on the mailing list and ask if they have something they could share.10:59
*** nmathew has quit IRC11:01
*** udesale has quit IRC11:01
*** dimak_ has quit IRC11:02
*** mvk has quit IRC11:06
*** panda is now known as panda|lunch11:07
*** hishh has quit IRC11:08
*** dsneddon is now known as dsneddon_afk11:08
*** mars[m] has quit IRC11:10
*** vcn[m] has quit IRC11:10
*** pavel_bondar[m] has quit IRC11:10
*** hishh has joined #openstack-neutron11:10
*** hfu has joined #openstack-neutron11:13
*** sambetts|afk is now known as sambetts11:14
*** horms has quit IRC11:14
*** pavel_bondar[m] has joined #openstack-neutron11:15
*** vijaykc4 has joined #openstack-neutron11:17
*** jhershbe has quit IRC11:17
*** jchhatbar has joined #openstack-neutron11:21
*** yamamoto has quit IRC11:22
*** janki has quit IRC11:24
*** jchhatbar is now known as janki11:24
*** chlong has joined #openstack-neutron11:24
*** hishh has quit IRC11:24
*** manheim has quit IRC11:26
*** vijaykc4 has quit IRC11:27
*** vijaykc4 has joined #openstack-neutron11:27
*** vijaykc4 has quit IRC11:28
*** leitan has joined #openstack-neutron11:28
*** vijaykc4 has joined #openstack-neutron11:28
*** donghao has joined #openstack-neutron11:29
*** vijaykc4 has quit IRC11:29
*** vanzyla has quit IRC11:30
*** leitan has quit IRC11:33
*** donghao has quit IRC11:33
*** murphy_zhao has joined #openstack-neutron11:34
*** tbachman has joined #openstack-neutron11:35
openstackgerritMerged openstack/neutron: [Live Migration] Extend ml2_port_binding table  https://review.openstack.org/40429311:36
*** murphy_zhao has quit IRC11:40
*** tmorin has quit IRC11:40
des_consoladomorning all - I'm quite confused with how to follow this http://docs.openstack.org/newton/networking-guide/config-bgp-dynamic-routing.html11:41
*** tbachman has quit IRC11:41
*** gcheresh_ has joined #openstack-neutron11:41
des_consoladoI've used kolla-ansible to deploy and I've got networking working to have instances talking to the outside world. I haven't yet re-configured or customised any services like this though.11:41
des_consoladoso on my controller node where neutron-server is running I can easily change the neutron.conf and restart the neutron-server container and I presume that deals with that part of things11:43
des_consoladoBut about this BGP agent... Well I'm running all services in containers, I guess I need to set up a new container for this agent...11:44
openstackgerritArtur Korzeniewski proposed openstack/neutron: Add support for Keepalived VRRP health check  https://review.openstack.org/27354611:45
*** moshele has quit IRC11:53
*** murphy_zhao has joined #openstack-neutron11:53
*** thorst_ has joined #openstack-neutron11:59
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron: Enforce port QoS policies for all ports.  https://review.openstack.org/42528011:59
*** manheim has joined #openstack-neutron12:01
openstackgerritSam Betts proposed openstack/neutron: Remove baremetal notification from nova notifier  https://review.openstack.org/42424712:01
*** bobmel has joined #openstack-neutron12:01
*** leitan has joined #openstack-neutron12:02
*** thorst_ has quit IRC12:03
*** catintheroof has joined #openstack-neutron12:04
*** mvk has joined #openstack-neutron12:04
*** nicolasbock has joined #openstack-neutron12:04
*** bfernando has quit IRC12:05
*** manheim has quit IRC12:05
*** bobmel has quit IRC12:06
*** manheim has joined #openstack-neutron12:08
korzenHi haleyb, I have address the comments in https://review.openstack.org/#/c/42069312:10
*** vcn[m] has joined #openstack-neutron12:10
*** mars[m] has joined #openstack-neutron12:10
korzenhaleyb, can you also take a look at: https://review.openstack.org/#/c/27354612:11
*** panda|lunch is now known as panda12:14
*** jpena is now known as jpena|lunch12:17
*** obondarev has joined #openstack-neutron12:17
*** dimak_ has joined #openstack-neutron12:18
*** sridharg has quit IRC12:19
*** lucasagomes is now known as lucas-hungry12:21
*** ganeshkaila has joined #openstack-neutron12:22
*** yamamoto has joined #openstack-neutron12:22
ganeshkailaHi @here, the SSH Tunnel 'ssh 8899:ip:80 user@ip' among the openstack instances is getting failed with the error 'bind: Address cannot be assigned'. It is getting succeed when I give command 'ssh -4 8899:ip:80 user@ip' which means forcing ssh to use ipv4. Is there a possibility to make the instances to use ipv4 all over?12:25
openstackgerritAnn Taraday proposed openstack/neutron: [WIP] Use writer for objects create  https://review.openstack.org/42621212:26
*** moshele has joined #openstack-neutron12:27
*** moshele has quit IRC12:29
*** amarao has joined #openstack-neutron12:29
*** yamamoto has quit IRC12:31
*** jhershbe has joined #openstack-neutron12:32
*** ranjithd has quit IRC12:33
*** dimak_ has quit IRC12:40
*** limao has joined #openstack-neutron12:40
openstackgerritMaxime Guyot proposed openstack/neutron: Apply QoS policy on network:router_gateway  https://review.openstack.org/42521812:40
*** jhershbe has quit IRC12:40
*** sdague has joined #openstack-neutron12:40
*** psahoo has quit IRC12:42
*** thorst__ has joined #openstack-neutron12:43
*** limao_ has joined #openstack-neutron12:44
*** limao has quit IRC12:44
*** tommylikehu_ has joined #openstack-neutron12:46
*** ociuhandu has joined #openstack-neutron12:46
*** sdague_ has joined #openstack-neutron12:52
*** tbachman has joined #openstack-neutron12:54
*** andreas_s has quit IRC12:55
*** limao_ has quit IRC12:56
*** tmorin has joined #openstack-neutron12:58
*** ratailor has quit IRC13:03
*** catintheroof has quit IRC13:05
openstackgerritThomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie  https://review.openstack.org/42575613:05
*** trown|outtypewww is now known as trown13:05
*** hishh has joined #openstack-neutron13:07
*** obondarev has quit IRC13:10
*** catintheroof has joined #openstack-neutron13:16
*** iranzo has joined #openstack-neutron13:16
openstackgerritThomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie  https://review.openstack.org/42575613:19
*** tommylikehu_ has quit IRC13:19
*** lucas-hungry is now known as lucasagomes13:19
*** hishh has quit IRC13:20
*** tommylikehu_ has joined #openstack-neutron13:20
*** ayogi has quit IRC13:20
*** jpena|lunch is now known as jpena13:22
*** mdnadeem has quit IRC13:23
*** sdague has quit IRC13:25
*** jlibosva has joined #openstack-neutron13:25
*** pbandark has quit IRC13:26
*** nmathew has joined #openstack-neutron13:26
tmorinralonsoh: I've removed the factory function and moved things in br_cookie  (https://review.openstack.org/#/c/425756/ ) -  thanks for your feedback!13:28
*** jhershbe has joined #openstack-neutron13:30
*** pgadiya_ has quit IRC13:34
*** jhershbe has quit IRC13:35
ralonsohtmorin: I'll review it now. Thank you13:35
tmorinralonsoh: many thanks!13:36
*** jheroux has joined #openstack-neutron13:40
*** thorst__ is now known as thorst_13:42
*** tommylikehu_ has quit IRC13:49
*** gouthamr has joined #openstack-neutron13:53
*** trown is now known as trown|brb13:57
*** nmathew has quit IRC13:58
*** ihrachys has joined #openstack-neutron14:01
*** nagarjung has quit IRC14:05
*** trown|brb is now known as trown14:09
*** ratailor has joined #openstack-neutron14:09
*** cleong has joined #openstack-neutron14:10
*** ratailor has quit IRC14:11
*** shausy has quit IRC14:11
*** sdague_ is now known as sdague14:12
*** absubram has joined #openstack-neutron14:14
*** kriskend has joined #openstack-neutron14:15
*** markvoelker has joined #openstack-neutron14:16
*** wolverineav has joined #openstack-neutron14:16
*** kriskend has quit IRC14:16
*** kriskend has joined #openstack-neutron14:16
*** wolverin_ has joined #openstack-neutron14:18
*** absubram has quit IRC14:18
*** murphy_zhao has quit IRC14:19
*** anilvenkata has joined #openstack-neutron14:19
*** gcheresh_ has quit IRC14:20
*** wolverineav has quit IRC14:20
dimskevinbenton : dasm : ataraday : jlibosva : ajo : a couple of jobs have a high rate of failure in the gate queue -  http://tinyurl.com/zwpv82314:20
ajodims looking thanks14:21
dimskevinbenton : dasm : ataraday : jlibosva : ajo : if you see http://status.openstack.org/zuul/ you can see we are trailing 12 hours because of these14:21
dimsthanks ajo14:21
ajoouch14:21
jlibosvadims: those data are from gate or check queue?14:22
ajodims do we know which specific failures are hitting us?, I will investigate otherwise14:22
dimsjlibosva you can see the 2 lines one from check and one from gate14:22
dimsmore worried about the gate than the check honestly14:22
jlibosvadims: oh, sorry. I'm stupid, it's written there in brackets :)14:23
dimsajo : nope. got this graph from sdague on infra channel :)14:23
ajowhen the load is higher we fail more14:23
ajo(race conditions, timeouts?)14:23
jlibosvathere is still this issue with memory depletion14:23
jlibosvadon't know if it's relevant but since full jobs run more tests, it might be related14:23
*** ranjithd has joined #openstack-neutron14:24
ajojlibosva I checked that, 17 failures in the last 24h14:24
ajoI wouldn't expect that to be a 50% failure rate14:24
ajothere must be something else14:24
*** amuller has joined #openstack-neutron14:24
jlibosvaajo: for full job?14:24
*** absubram has joined #openstack-neutron14:24
ajojlibosva I don't know time to look for failures, I'll start that14:24
*** jperry has joined #openstack-neutron14:24
ajojlibosva my thought on timeouts is, if it's happening in functional, nothing stops it from happening in full (under the hood)14:25
ajoit's a bad indicative14:25
jlibosvaajo: oh, I'm not talking about timeouts in ovsdb14:25
ajojlibosva oh, sorry, what did you mean with "jlibosva:ajo: for full job?" =14:26
jlibosvaajo: I thought you were checking hits for oom-killer: https://bugs.launchpad.net/neutron/+bug/165638614:26
openstackLaunchpad bug 1656386 in neutron "Memory leaks on Neutron jobs" [Critical,Confirmed] - Assigned to Darek Smigiel (smigiel-dariusz)14:26
jlibosvaajo: I asked whether you limited those hits to the neutron-full jobs only14:26
*** zhhuabj has quit IRC14:26
jlibosvaok, I see you commented there :)14:27
*** kriskend has quit IRC14:27
*** kriskend has joined #openstack-neutron14:27
*** kriskend has quit IRC14:29
*** kriskend has joined #openstack-neutron14:29
*** iranzo has quit IRC14:30
ajodims, is there anywhere where I can see the specific gate failures for neutron ?14:30
*** baoli has joined #openstack-neutron14:30
*** karthiks has quit IRC14:30
ajo(see which test failed, and logs, etc)14:30
dimsajo : this help? http://status.openstack.org/elastic-recheck/data/integrated_gate.html#gate-tempest-dsvm-neutron-full-ubuntu-xenial14:31
*** kbringard has joined #openstack-neutron14:36
*** efoley has quit IRC14:37
*** Dinesh_Bhor has quit IRC14:37
ajodims, oh, right!14:37
ajothanks14:37
*** efoley has joined #openstack-neutron14:38
kevinbentondims: several of the resets we were hitting yesterday were libvirt issues14:38
kevinbentondims: the only bug in the gate queue that I know of that is specific to neutron jobs is the oom killer14:38
*** zhhuabj has joined #openstack-neutron14:39
dasmkevinbenton: did your fix solve it?14:39
*** efoley has quit IRC14:39
*** tlian has joined #openstack-neutron14:39
dasmswappiness one?14:39
kevinbentonhttps://review.openstack.org/#/c/425961/14:39
ajokevinbenton do the oom kills happen only for neutron jobs?14:39
*** efoley has joined #openstack-neutron14:39
kevinbentonStill waiting for infra core14:39
ajohttps://etherpad.openstack.org/p/neutron-gate-failures-2017-01-2714:39
ajoI'm going over those14:39
ajolet's see what we find14:40
ajoif anybody want`s to take that list on reverse order, feel free ^ :)14:40
kevinbentonajo: it looks like it might14:40
dasmajo: i'll go in reverse order14:40
ajothanks dasm14:40
kevinbentonajo: but that's probably just because those jobs run extra services (neutron) to push memory usage higher14:41
ajokevinbenton yikes :S14:41
ajokevinbenton what do you mean?14:41
kevinbentonajo: memory usage of neutron procs is not as bad as some of the others14:41
kevinbentonajo: the neutron jobs run more services than non neutron14:41
ajokevinbenton yes, that's what I saw, compared to cinder we're nothing14:42
ajoah14:42
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 7  https://review.openstack.org/40955714:42
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 5  https://review.openstack.org/40955614:42
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 4  https://review.openstack.org/40955514:42
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 3  https://review.openstack.org/40955414:42
openstackgerritHenry Gessau proposed openstack/neutron: Decouple hook and func registration from CommonDbMixin  https://review.openstack.org/40955314:42
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 6  https://review.openstack.org/40783714:42
kevinbentonajo: a neutron server, neutron agents, etc14:42
ajoHenryG++14:42
*** janzian has joined #openstack-neutron14:42
HenryGajo: Just keeping my patches rebased with master. How are you my friend?14:43
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron: QoS NIC bandwidth manager for SR-IOV backend  https://review.openstack.org/40125414:43
ajoHenryG: good :D, I'm good :)14:44
otherwiseguykevinbenton, saw that you pinged me in #openstack-meeting last night?14:44
HenryGajo: Are going to the Boston summit?14:44
kevinbentonotherwiseguy: something seems to block up everything in the ovs agent14:45
kevinbentonotherwiseguy: so eventlet doesn't yield14:46
otherwiseguyOk. I'll take a look at removing eventlet. :p14:46
kevinbentonotherwiseguy: subsequently ryu and ovsdb native stops answering14:46
otherwiseguyhmm.14:46
kevinbentonOvs's phone calls14:47
kevinbentonAnd we get the disconnects14:47
*** dansmith is now known as superdan14:47
ajoHenryG not sure about boston, only Atlanta for now14:47
kevinbentonAt least that's one situation I've observed in the gate14:47
otherwiseguykevinbenton, got a quick link to log where it happens?14:47
HenryGajo: No worries. Any time you are near Boston, let me know and we'll meet for drinks.14:48
ajoHenryG I'd prefer not having to travel 4 times a year :D , my wife would hate me if she has to take care of kids for 1 month/year total while I'm abroad14:48
ajoHenryG oohh, so you're in boston now, ack :) I didn't know14:48
ajothanks HenryG I'll eventually be around,  :)14:48
ajootherwiseguy kevinbenton is it related to the ovs timeouts?14:49
kevinbentonajo: yes14:49
kevinbentonotherwiseguy: one sec14:50
kevinbenton2017-01-26T19:53:25.306Z|00814|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 5 seconds, disconnecting14:52
kevinbentonhttp://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/openvswitch/ovs-vswitchd.txt.gz14:52
kevinbentonagent freezes for 20 seconds when it issues here http://logs.openstack.org/41/425841/1/check/gate-tempest-dsvm-platform-centos7-nv/19326f0/logs/screen-q-agt.txt?level=DEBUG#_2017-01-26_19_53_18_84814:52
ajokevinbenton : http://logs.openstack.org/93/404293/44/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2197257/logs/screen-q-agt.txt.gz?#_2017-01-27_09_15_02_71214:52
ajoone of the unidentified gate failures ^14:52
ajodoes that ring a bell?14:52
kevinbentonotherwiseguy: you can see the ovs side disconnective from ovs agent14:53
ajowhen are we supposed to have guru meditation tracebacks there?14:53
kevinbentonotherwiseguy: in the ovs agent you see it try to issue an ofctl command with everything blocks up for 20 seconds14:53
*** iyamahat has joined #openstack-neutron14:56
*** mlavalle has joined #openstack-neutron14:56
*** armax has joined #openstack-neutron14:56
*** murphy_zhao has joined #openstack-neutron14:57
kevinbentonajo: that's just agent being stopped, no?14:57
ajokevinbenton  hmm, yes, it looks like that ':D14:57
ajonow I just realized14:57
ajothanks :D14:57
ajoIt's the end of execution of tests14:58
*** jlibosva1 has joined #openstack-neutron15:00
haleybis that from the monkey patching in eventlet_utils ?  more curious than anything15:02
*** jlibosva has quit IRC15:02
*** tbachman has quit IRC15:02
*** anilvenkata has quit IRC15:03
*** korzen has quit IRC15:03
*** tbachman has joined #openstack-neutron15:04
otherwiseguykevinbenton, that looks nasty. researching.15:05
*** efoley_ has joined #openstack-neutron15:05
kevinbentonotherwiseguy: iwamoto had mentioned that someone had run into this and increasing the timeout for the heartbeat was enough to paper over the issue15:06
*** absubram has quit IRC15:06
kevinbentonotherwiseguy: so maybe we do that for now...15:08
*** efoley has quit IRC15:09
openstackgerritAnindita Das proposed openstack/neutron: [WIP] Integration of Allocation/Endpoints OVO  https://review.openstack.org/36781015:10
openstackgerritVictor Morales proposed openstack/neutron: Integrate NetworkSegment OVO  https://review.openstack.org/38517815:11
*** yamahata has joined #openstack-neutron15:12
*** jamesdenton has joined #openstack-neutron15:14
prometheanfirejohn-davidge: updated the pd bug15:15
john-davidgeprometheanfire: Thanks!15:16
dasmkevinbenton: seatbelts fastened? sdague gave +W for swappiness :)15:16
john-davidgeprometheanfire: Does the rule get added if you remove and re-add the router interface? or not at all?15:16
kevinbentondasm: yep :)15:17
*** anilvenkata has joined #openstack-neutron15:19
*** gouthamr has quit IRC15:19
*** claudiub|2 has joined #openstack-neutron15:20
ajootherwiseguy I started this yesterday: https://review.openstack.org/#/c/425623/ but it's probably useless I guess15:21
*** claudiub|3 has joined #openstack-neutron15:23
*** claudiub has quit IRC15:23
openstackgerritOpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements  https://review.openstack.org/42364515:24
prometheanfirejohn-davidge: now my pd subnet's VMs are all missing their addresses :|15:24
john-davidgeprometheanfire: :/15:24
prometheanfire-A neutron-l3-agent-scope -i qr-7bc14490-05 -j MARK --set-xmark 0x4000000/0xffff000015:24
prometheanfiremight be another problem though (not related), checking15:25
*** jaosorior has joined #openstack-neutron15:25
*** claudiub|2 has quit IRC15:25
john-davidgeprometheanfire: no sign of a mangle rule starting -d <prefix> ?15:25
prometheanfirenope15:26
prometheanfirelike I said, this is having the patch applied to newton15:26
prometheanfireso maybe something there?15:27
prometheanfirel3 agent hands requests the subnet via pd right?15:28
otherwiseguykevinbenton, so the ofctl request is the last thing that is made before things freeze, at 19:53:18, we get the 5 sec inactivity probe failure at 19:53:25, so something weird happened between :18 and :20. The DbListCommand looks like it finishes.15:28
otherwiseguykevinbenton, is this fairly easy to reproduce/does it happen a lot?15:28
*** Leo_ has joined #openstack-neutron15:29
kevinbentonotherwiseguy: not easy to reproduce15:29
kevinbentonotherwiseguy: does seem to happen frequently15:29
kevinbentonotherwiseguy: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20Datapath%20invalid%5C%2215:29
*** crose has joined #openstack-neutron15:34
ajodasm , I need to leave: feel free to check my incomplete stuff if you had time: https://etherpad.openstack.org/p/neutron-gate-failures-2017-01-2715:35
dasmajo: sure. i'm on it15:35
ajomane of them seem nova related15:35
ajomany15:35
ajodims ^15:35
prometheanfirejohn-davidge: ya, my dibbler died15:36
ajothat's early conclusion, let's let dasm dig up further,15:36
dimsajo : thanks for looking, i will take a peek after the release team meeting15:38
john-davidgeprometheanfire: Ah, ok :) Hopefully that's all it was15:40
prometheanfirethat, and I upgraded the upstream router, breaking pd both up and down stream :P15:41
*** gouthamr has joined #openstack-neutron15:41
prometheanfirehttps://community.ubnt.com/t5/EdgeMAX/ipv6-Prefix-delegation-for-a-other-Router-Fritzbox/m-p/1535886#M10680315:42
*** gkadam has quit IRC15:42
prometheanfirethat's how to get PD working on and edgerouter15:42
prometheanfireneed to patch after every upgrade...15:42
openstackgerritMerged openstack/neutron-fwaas: TrivialFix: Added link for modindex  https://review.openstack.org/42605515:42
*** marst has joined #openstack-neutron15:47
*** princenana has joined #openstack-neutron15:48
*** absubram has joined #openstack-neutron15:49
otherwiseguykevinbenton, if it was super frequent, I'd just try forcing ovs-ofctl instead of native just to see if it is something in ryu or the ovsdb stuff that is getting hung.15:49
otherwiseguykevinbenton, trying to find out how the ryu stuff actually works wrt its event loop/responding to inactivity probes, etc.15:50
otherwiseguykevinbenton, I've never looked at the actual code of the the openflow side of neutron :p15:51
*** fzdarsky is now known as fzdarsky|afk15:51
*** amotoki has joined #openstack-neutron15:51
*** nplanel has quit IRC15:54
*** jlibosva1 is now known as jlibosva15:55
*** amotoki has quit IRC15:56
*** mvk has quit IRC15:58
*** nplanel has joined #openstack-neutron15:59
*** dave-johnston has joined #openstack-neutron15:59
*** ijw has joined #openstack-neutron16:00
jlibosvakevinbenton: http://logs.openstack.org/12/425812/2/check/gate-neutron-dsvm-functional-ubuntu-xenial/70f2d8a/logs/dsvm-functional-logs/neutron.tests.functional.agent.ovsdb.test_impl_idl.ImplIdlTestCase.test_post_commit_vswitchd_completed_no_failures.txt.gz#_2017-01-27_09_15_18_51316:03
*** murphy_zhao has quit IRC16:05
jlibosvakevinbenton: http://logs.openstack.org/12/425812/2/check/gate-neutron-dsvm-functional-ubuntu-xenial/70f2d8a/testr_results.html.gz16:05
*** efoley__ has joined #openstack-neutron16:06
*** TMM has quit IRC16:06
openstackgerritBertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue by using cast instead of fanout  https://review.openstack.org/42628716:06
*** efoley_ has quit IRC16:09
*** donghao has joined #openstack-neutron16:11
*** ventifus has joined #openstack-neutron16:11
*** kriskend has quit IRC16:12
*** efoley__ has quit IRC16:12
jlibosvakevinbenton: ihrachys otherwiseguy maybe we could enable debug on ovsdb server side for functional jobs? before the timeout there are some debug logs from python ovs - tcp:127.0.0.1:6640: received unexpected reply message16:13
openstackgerritBertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue in case of services_sync_needed  https://review.openstack.org/42455116:13
*** donghao has quit IRC16:13
otherwiseguyjlibosva, that message is repeated *a lot* throughout the logs. I remember tracking it down at some point and now I find that I can't remember what the cause was.16:14
* otherwiseguy thinks16:14
kevinbentonjlibosva: for this one16:14
kevinbentonhttp://logs.openstack.org/12/425812/2/check/gate-neutron-dsvm-functional-ubuntu-xenial/70f2d8a/logs/dsvm-functional-logs/neutron.tests.functional.agent.ovsdb.test_impl_idl.ImplIdlTestCase.test_post_commit_vswitchd_completed_no_failures.txt.gz#_2017-01-27_09_15_18_51316:14
kevinbentonjlibosva: something blocked the agent for 10 seconds16:15
*** gkadam has joined #openstack-neutron16:15
jlibosvakevinbenton: I was looking at http://logs.openstack.org/17/413017/23/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/f36e699/logs/screen-q-agt.txt#_2017-01-26_19_10_28_07816:15
*** iranzo has joined #openstack-neutron16:15
*** iranzo has quit IRC16:15
*** iranzo has joined #openstack-neutron16:15
openstackgerritBertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue by using cast instead of fanout  https://review.openstack.org/42628716:15
*** fnaval has joined #openstack-neutron16:16
*** liangy has joined #openstack-neutron16:16
otherwiseguyjlibosva, ah yes, the unexpected reply messages can happen if a transaction is destroyed before receiving a reply. I think this happens when we do a verify() call and it fails because a column has been updated while we are trying to set and have to retry?16:17
jlibosvakevinbenton: where do you see it was blocked (in the link you sent). The test doesn't even have an agent. It just sets up the ovsdb API and adds a bridge16:17
*** trevormc has joined #openstack-neutron16:18
kevinbentonjlibosva: that period of time where it is only showing _check_child_processes outputs16:18
otherwiseguyin a test environment when we have lots of things adding bridges and deleting them, we'd definitely see a lot of of those messages.16:18
kevinbentonjlibosva: 10 seconds or so16:18
*** HenryG has quit IRC16:19
kevinbentonjlibosva: followed by "neutron.agent.ovsdb.native.vlog [-] 10000-ms timeout __log_wakeup"16:19
*** nplanel has quit IRC16:20
*** HenryG has joined #openstack-neutron16:20
kevinbentonotherwiseguy: that native.vlog uses that custom ovs poller class16:20
kevinbentonit's supposed to be eventletized16:21
kevinbentonhttps://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py16:21
*** eezhova has quit IRC16:21
*** tbachman has quit IRC16:22
*** kriskend has joined #openstack-neutron16:22
prometheanfirejohn-davidge: for now, ignore my -1 til I get pd working again16:22
john-davidgeprometheanfire: ok sure, let me know if its still not working after that16:23
prometheanfireya16:23
otherwiseguykevinbenton, yeah, i wanted to be able to hook the python ovs idl lib's logs into our logging. it's a weird bastardization of python logging to make it match the rest of ovs's logging.16:24
otherwiseguythe ovs lib *should* be eventlet friendly, but ...16:24
*** tbachman has joined #openstack-neutron16:25
otherwiseguyand the native ovsdb stuff I tried to just avoid any kind of decision on an event loop but tried to make it eventlet friendly with threads, etc. which are monkey patched. but i've always been mildly uncomfortable just because of how much I hate eventlet :p16:26
kevinbentonotherwiseguy, jlibosva: this patch was basically added because of a hard block wasn't it? https://github.com/openstack/neutron/commit/c13d722f3913137945c27fcc74371d3316129f3016:29
*** teclator has quit IRC16:29
*** teclator_ has joined #openstack-neutron16:29
jlibosvakevinbenton: yes16:29
dalvarezjlibosva, kevinbenton, otherwiseguy i hit that issue today and i think it's  bug 1627106 in case it may help16:29
openstackbug 1627106 in neutron "TimeoutException while executing test_post_commit_vswitchd_completed_no_failures" [High,In progress] https://launchpad.net/bugs/1627106 - Assigned to Miguel Angel Ajo (mangelajo)16:29
*** david-lyle has joined #openstack-neutron16:29
jlibosvadalvarez: yeah, it is :)16:30
jlibosvakevinbenton: also16:30
jlibosvakevinbenton: I think we started seeing the error after we fixed a bug in that patch16:30
kevinbentonjlibosva: why is it blocking? is the connection already dead at that point?16:30
jlibosvakevinbenton: which was that timeout was in miliseconds or something16:30
dalvarez https://review.openstack.org/#/c/418120/      maybe some other eyes please?16:31
*** ralonsoh has quit IRC16:31
dalvarezjlibosva, otherwiseguy   https://review.openstack.org/#/c/425812/2         if you guys have some time pls :))))) <316:32
jlibosvakevinbenton: I don't know, by that time it just caused that all tests get stuck and the whole job got killed by global timeout16:32
jlibosvakevinbenton: https://github.com/openstack/neutron/commit/a01ca10216:32
*** thiagolib has joined #openstack-neutron16:34
jlibosvakevinbenton: iirc the ovs should have some mechanism of reconnecting in case something goes south16:35
jlibosvabut by that time of gate getting stuck, it didn't reconnect, so this is more like a workaround for this issue, which we might be still suffering16:36
*** panda is now known as panda|bbl16:37
jlibosvakevinbenton: also a good hint could be https://bugs.launchpad.net/neutron/+bug/1646107 - I think on ovs 2.6 it is more reproducible (or the xenial default)16:37
openstackLaunchpad bug 1646107 in neutron "Functional job times out on xenial" [Medium,Confirmed]16:37
kevinbentonack16:37
*** annegentle has joined #openstack-neutron16:40
*** janki has quit IRC16:40
otherwiseguykevinbenton, jlibosva: that timeout has always made me a bit uncomfortable since we have a queue for requests and responses that we read one at a time and if we timeout on a get(), it seems like we could get out of sync if we got a late result. I need to work out how to thoroughly test that.16:42
prometheanfirewell, can confirm client side is good, looks like I hosed the upstream server, will have to redo that16:42
*** tbachman has quit IRC16:43
openstackgerritKevin Benton proposed openstack/neutron: sanity check  https://review.openstack.org/42630216:45
*** absubram has quit IRC16:47
* otherwiseguy stares at functional test console for Kevin's sanity check16:49
*** Brenda has quit IRC16:49
*** gkadam has quit IRC16:49
kevinbentonotherwiseguy: of course that won't be the problem16:49
kevinbentonotherwiseguy: that would be too easy :)16:49
*** jlibosva has quit IRC16:49
kevinbentonotherwiseguy: but if i didn't do it, it would be the problem16:49
dasmotherwiseguy: is it very exciting?16:49
kevinbentonotherwiseguy: it's a quantum bug :)16:49
* dasm is thinking also about looking at console :)16:49
*** iranzo has quit IRC16:50
*** kbringard has quit IRC16:50
otherwiseguykevinbenton, I can't begin to express how much I hate debugging eventlet-related things. That probably means I just need to get better at it. But I find the monkey patching approach in general hard to stomach.16:50
* otherwiseguy tries to adopt a more open mind16:51
*** eezhova has joined #openstack-neutron16:52
otherwiseguykevinbenton, there seem to be quite a few passed functional tests at this point.16:58
*** anilvenkata has quit IRC16:59
*** crose has quit IRC17:00
*** slaweq has joined #openstack-neutron17:02
*** mhickey has quit IRC17:02
*** matrohon has quit IRC17:04
*** lnicolas1 has quit IRC17:05
*** rajinir has joined #openstack-neutron17:05
kevinbentonotherwiseguy: yeah17:05
kevinbentonotherwiseguy: the problem with eventlet is only takes one uncooperative function somewhere that blocks17:06
kevinbentonotherwiseguy: to bring the whole system to a halt17:06
*** rossella_ has quit IRC17:07
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 7  https://review.openstack.org/40955717:07
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 5  https://review.openstack.org/40955617:07
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 4  https://review.openstack.org/40955517:07
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 6  https://review.openstack.org/40783717:07
*** kbringard has joined #openstack-neutron17:10
*** iyamahat has quit IRC17:11
*** yamahata has quit IRC17:11
*** fzdarsky|afk is now known as fzdarsky17:12
*** slaweq has quit IRC17:12
openstackgerritBertrand Lallau proposed openstack/neutron-fwaas: Fix scale issue in case of services_sync_needed  https://review.openstack.org/42455117:18
*** Leo_ has quit IRC17:22
*** Leo_ has joined #openstack-neutron17:23
*** Jeffrey4l_ has quit IRC17:24
openstackgerritSindhu Devale proposed openstack/neutron: RouterPort OVO integration  https://review.openstack.org/42415417:24
openstackgerritSindhu Devale proposed openstack/neutron: Router, RouterPort and FloatingIP to OVO  https://review.openstack.org/30796417:24
openstackgerritShashank Kumar Shankar proposed openstack/neutron: Integration of Port Binding Level OVO  https://review.openstack.org/38203717:24
*** Jeffrey4l_ has joined #openstack-neutron17:24
*** eezhova has quit IRC17:27
*** eezhova has joined #openstack-neutron17:27
*** annegentle has quit IRC17:29
*** rmart04 has quit IRC17:29
openstackgerritKevin Benton proposed openstack/neutron: sanity check  https://review.openstack.org/42630217:30
*** annegentle has joined #openstack-neutron17:30
*** slaweq has joined #openstack-neutron17:33
*** iyamahat has joined #openstack-neutron17:34
*** iyamahat has quit IRC17:34
*** nlahouti has joined #openstack-neutron17:35
*** tbachman has joined #openstack-neutron17:35
openstackgerritMerged openstack/neutron: Add missing module-level SUBNET ref  https://review.openstack.org/41767217:36
*** kevo has joined #openstack-neutron17:37
*** ygbo has quit IRC17:38
*** Leo_ has quit IRC17:39
*** liangy has quit IRC17:41
*** Leo_ has joined #openstack-neutron17:42
*** mriedem is now known as mriedem_beach17:42
*** ociuhandu has quit IRC17:43
*** jpena is now known as jpena|off17:46
openstackgerritKevin Benton proposed openstack/neutron: sanity check  https://review.openstack.org/42630217:48
*** Swami has joined #openstack-neutron17:51
*** trown is now known as trown|lunch17:51
kevinbentonotherwiseguy: that timeout change makes less sense to me than before17:51
kevinbentonotherwiseguy: the "run" function is running in its own python native thread17:52
openstackgerritSwaminathan Vasudevan proposed openstack/neutron: DVR: Create router to fip namespace connection based on gateway state  https://review.openstack.org/28375717:53
kevinbentonotherwiseguy: so who cares if it blocks forever17:53
*** ijw has quit IRC17:54
*** ijw has joined #openstack-neutron17:54
*** radhikam has joined #openstack-neutron17:55
*** lucasagomes is now known as lucas-afk17:55
*** yamahata has joined #openstack-neutron17:56
*** ijw_ has joined #openstack-neutron17:56
*** ijw has quit IRC17:56
kevinbentonotherwiseguy: ah, nevermind17:57
*** harlowja has joined #openstack-neutron18:02
*** oreillyd_ has quit IRC18:02
*** hfu has quit IRC18:02
ihrachysarmax: we probably want to include https://review.openstack.org/#/c/425023/ into rc118:02
*** chlong has quit IRC18:03
openstackgerritAliaksandr Dziarkach proposed openstack/neutron: Pluggable IPAM rollback mechanism is not robust  https://review.openstack.org/39059418:06
*** mickeys has joined #openstack-neutron18:07
*** sambetts is now known as sambetts|afk18:07
*** jose-phillips has joined #openstack-neutron18:08
*** tbachman has quit IRC18:10
*** ataraday_ has quit IRC18:11
*** tbachman has joined #openstack-neutron18:12
*** ijw_ has quit IRC18:12
*** donghao has joined #openstack-neutron18:14
*** Leom has joined #openstack-neutron18:15
*** Leo_ has quit IRC18:15
*** manheim has quit IRC18:16
*** mvk has joined #openstack-neutron18:16
*** amarao has quit IRC18:17
*** donghao has quit IRC18:18
openstackgerritBrian Haley proposed openstack/neutron: Update in-tree code to use new neighbour functions  https://review.openstack.org/42020918:19
*** tbachman has quit IRC18:20
*** gvrangan has joined #openstack-neutron18:21
ihrachyskevinbenton: we fixed 'RuntimeError: Failed to start 2 workers.' functional test failure already right?18:23
*** tbachman has joined #openstack-neutron18:24
kevinbentonihrachys: i think so with the psutils patch18:24
ihrachysok makes sense. I will ignore those in my triaging then.18:24
*** jose-phillips has quit IRC18:26
openstackgerritThomas Morin proposed openstack/neutron: Fix WaitTimeout string representations  https://review.openstack.org/42592418:29
*** ranjithd has quit IRC18:32
*** vhosakot has joined #openstack-neutron18:32
*** davidsha has quit IRC18:33
kevinbentonihrachys, armax: time for end of cycle dhcp agent bugs :) https://bugs.launchpad.net/neutron/+bug/165991918:34
openstackLaunchpad bug 1659919 in neutron "dhcp agent race with port update and network create" [Undecided,New]18:34
* ihrachys rages in and merges all the things18:35
openstackgerritKevin Benton proposed openstack/neutron: Always acquire network.id lock in dhcp port update  https://review.openstack.org/42633918:38
*** david-lyle has quit IRC18:39
*** john-davidge has quit IRC18:39
openstackgerritKevin Benton proposed openstack/neutron: sanity check  https://review.openstack.org/42630218:42
*** abhiraut has joined #openstack-neutron18:43
*** slaweq has quit IRC18:45
*** dave-johnston has quit IRC18:45
*** iranzo has joined #openstack-neutron18:46
openstackgerritVictor Morales proposed openstack/neutron: Integrate Security Groups OVO  https://review.openstack.org/28473818:46
*** slaweq has joined #openstack-neutron18:47
*** slaweq has quit IRC18:52
*** slaweq has joined #openstack-neutron18:52
*** abhiraut has quit IRC18:53
*** abhiraut has joined #openstack-neutron18:53
*** ijw has joined #openstack-neutron18:56
*** beagles is now known as beagles-biab18:58
*** liangy has joined #openstack-neutron19:00
*** gcheresh_ has joined #openstack-neutron19:01
*** manheim has joined #openstack-neutron19:08
*** ranjithd has joined #openstack-neutron19:09
*** abhiraut has quit IRC19:10
*** absubram has joined #openstack-neutron19:10
*** obondarev has joined #openstack-neutron19:11
*** abhiraut has joined #openstack-neutron19:11
*** donghao has joined #openstack-neutron19:15
*** radhikam has quit IRC19:15
*** absubram has quit IRC19:15
*** hishh has joined #openstack-neutron19:16
*** Sukhdev has joined #openstack-neutron19:18
*** absubram has joined #openstack-neutron19:18
*** radhikam has joined #openstack-neutron19:19
*** slaweq has quit IRC19:19
*** donghao has quit IRC19:19
*** kbaegis has joined #openstack-neutron19:21
kbaegisHi all. Having some issue with the stable/newton odl plugin19:21
kbaegisMy devstack VM comes up and stacks fine, but about 80% of the time I lose network connectivity from br-ext to the Virtualbox networking stack19:23
openstackgerritboden proposed openstack/neutron-lib: Remove devref modindex ref  https://review.openstack.org/42636819:24
*** tbachman has quit IRC19:26
openstackgerritKevin Benton proposed openstack/neutron: Always acquire network.id lock in dhcp port update  https://review.openstack.org/42633919:26
kevinbentonhaleyb: ping19:26
*** trown|lunch is now known as trown19:27
*** obondarev has quit IRC19:28
*** gcheresh_ has quit IRC19:30
*** Matias has quit IRC19:32
*** Matias has joined #openstack-neutron19:33
haleybkevinbenton: pong19:35
kevinbentonhaleyb: can you take a quick look at https://review.openstack.org/42633919:36
kevinbentonhaleyb: traced a random gate failure down to it19:36
*** ikeyman has joined #openstack-neutron19:36
haleybkevinbenton: why was that done like that? to try and save on a lock? seems like a good change19:39
*** jheroux has quit IRC19:43
*** tbachman has joined #openstack-neutron19:47
*** ijw_ has joined #openstack-neutron19:47
*** jose-phillips has joined #openstack-neutron19:49
*** ijw has quit IRC19:49
openstackgerritIsaku Yamahata proposed openstack/neutron: Kill the metadata proxy process unconditionally  https://review.openstack.org/41156619:51
*** tbachman has quit IRC19:52
*** d-bark has joined #openstack-neutron19:53
*** david-lyle has joined #openstack-neutron19:53
*** david-lyle has quit IRC19:53
*** david-lyle has joined #openstack-neutron19:54
dasmarmax: ihrachys: fyi. i'm taking 2 weeks PTO, starting from Monday. Will be back around Feb 1319:57
ihrachysack19:57
armaxl19:57
armaxk19:57
dasmarmax: ihrachys i'll be checking irc/ml but won't be able to release rc119:57
*** crose has joined #openstack-neutron19:58
armaxack19:58
armaxthanks for the heads up19:58
dasmnp. have fun with last days of Ocata ;)19:58
*** manheim has quit IRC19:58
*** Jack_V has quit IRC19:58
*** manheim has joined #openstack-neutron19:59
*** beagles-biab is now known as beagles20:00
*** Leom_ has joined #openstack-neutron20:01
*** david-lyle has quit IRC20:01
*** david-lyle has joined #openstack-neutron20:01
*** tbachman has joined #openstack-neutron20:01
*** manheim has quit IRC20:01
*** Leom has quit IRC20:03
*** crose has quit IRC20:06
*** nlahouti has quit IRC20:08
*** ikeyman has quit IRC20:10
*** dsneddon_afk is now known as dsneddon20:12
*** MasterOfBugs has joined #openstack-neutron20:12
*** Miouge has quit IRC20:15
*** pramodrj07 has joined #openstack-neutron20:15
*** lnicolas has joined #openstack-neutron20:15
*** Leo_ has joined #openstack-neutron20:17
*** Leom_ has quit IRC20:17
ihrachysarmax: kevinbenton: ok I think I have a list of issues that happened Jan26+ at https://etherpad.openstack.org/p/neutron-upstream-ci line 20+20:18
* armax looks20:18
*** MasterOfBugs has quit IRC20:18
ihrachyswhat pops out is, multiple tests in test_dhcp_agent fails with device XXX does not exist20:18
armaxkill dhcp20:18
armaxmake it disabled by default and deprecate it as of now20:19
armaxproblem solved20:19
ihrachysthey fail a bit differently, but I think it's the same issue20:19
armaxDHCP is so yesterday anyway20:19
*** amarao has joined #openstack-neutron20:19
ihrachysthen we have a bunch of tests that just fail after test case timeout - absolutely random tests, from db migration to ovs agent20:19
ihrachysthere is a fix for a symptom of those in https://review.openstack.org/#/c/425924/ but that won't resolve the root cause20:20
ihrachysand finally we have ovs native (?) timeout on adding bridge, and metadata proxy startup failure20:21
ihrachysthat seems all there was in last 1d+20:21
ihrachysI will have a look at dhcp agent one, seems weird it's only that test, maybe we do something wrong in service setup20:21
ihrachysand it may be as well the same ovs add-bridge timeout issue, though I see at least one failure that complains about tap missing, not bridge20:22
*** abhiraut has quit IRC20:24
*** gvrangan has quit IRC20:27
*** slaweq has joined #openstack-neutron20:28
*** panda|bbl is now known as panda20:28
*** jose-phillips has quit IRC20:29
*** Leo_ has quit IRC20:30
*** kevo has quit IRC20:30
*** Leo_ has joined #openstack-neutron20:31
openstackgerritThomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie  https://review.openstack.org/42575620:32
ihrachysok, all dhcp failures are actually add-bridge-timeout failures20:33
*** dsneddon is now known as dsneddon_afk20:34
ihrachyskevinbenton: otherwiseguy: http://logs.openstack.org/43/408143/9/check/gate-neutron-dsvm-functional-ubuntu-xenial/8926e44/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2017-01-27_02_41_16_452 does it ring a bell?20:37
haleybi'm seeing a failure in the rally py35 tests now, timeout deleting dhcp port, what's with the rash of dhcp issues?20:38
haleybhttp://logs.openstack.org/09/420209/7/check/gate-rally-dsvm-py35-neutron-neutron-ubuntu-xenial/cc405f8/logs/screen-q-svc.txt.gz?#_2017-01-27_19_24_53_66820:39
*** Leo_ has quit IRC20:40
*** Leo_ has joined #openstack-neutron20:40
openstackgerritJohn Perkins proposed openstack/neutron: Integrate oslo.config validator  https://review.openstack.org/42639720:41
*** vnovakov has joined #openstack-neutron20:41
*** hishh has quit IRC20:43
openstackgerritAnindita Das proposed openstack/neutron: [WIP] Live Migration New Port Bindings Extension  https://review.openstack.org/41425120:46
openstackgerritThomas Morin proposed openstack/neutron: Refactor OVSCookieBridge to always apply extension cookie  https://review.openstack.org/42575620:46
*** claudiub has joined #openstack-neutron20:47
openstackgerritJohn Perkins proposed openstack/neutron: Integrate oslo.config validator  https://review.openstack.org/42639720:47
slaweqhaleyb: hello20:47
haleybslaweq: hi20:47
slaweqhaleyb: I'm checking what is reason of issue https://bugs.launchpad.net/neutron/+bug/1657260 and I think I found it20:47
openstackLaunchpad bug 1657260 in neutron "Established connection don't stops when rule is removed" [Undecided,New] - Assigned to Slawek Kaplonski (slaweq)20:47
slaweqit's missing CT zone in iptables config20:48
slaweqand conntrack is called with "-w 1" parameter20:48
slaweqthat don't remove entry from conntrack20:48
openstackgerritJohn Perkins proposed openstack/neutron: Integrate oslo.config validator  https://review.openstack.org/42639720:48
*** armax has quit IRC20:49
slaweqif I manually run this conntrack -D command on my host it works fine when it was without "-w 1"20:49
slaweqbut I'm not an expert of linuxbridge and iptables driver so I want to ask if it makes sense for You? :)20:49
*** claudiub|3 has quit IRC20:49
*** amuller has quit IRC20:49
*** fzdarsky has quit IRC20:50
*** gvrangan has joined #openstack-neutron20:50
haleybslaweq: so is the zone not created?  this was only with linuxbridge agent not ovs, right?  looking at bug again20:51
slaweqfor me it looks like zone is not created20:51
*** gcheresh_ has joined #openstack-neutron20:51
slaweqand in fact there is no code to create zone in IptablesFirewallDriver class20:52
slaweqit is only in OVSHybrid driver which is not used by linuxbridge20:52
*** jose-phillips has joined #openstack-neutron20:52
*** wolverin_ is now known as wolverineav20:53
haleybslaweq: i can't remember if that was intentional, it's been a while since that CT code was added20:54
*** hishh has joined #openstack-neutron20:54
haleybbut maybe finding why -w is used when there is no zone is quicker to get this fixed20:54
slaweqhaleyb: I don't know it too and I wanted to ask if I should try to add support for zones in Iptables Driver and linuxbridge agent or maybe (what IMHO should be easier) to remove this zone from conntrack command in case if zone is not used20:55
slaweqwhat You think?20:55
*** sdague has quit IRC20:56
haleybslaweq: let me see if i can find the change that added it, think it was kevinbenton, he adds all the cool stuff :)20:57
slaweqhttps://review.openstack.org/#/c/118274/20:57
slaweqit was probably this one20:57
slaweqit's quite old patch20:58
haleybslaweq: yes, and that was reverted and re-added i think20:58
*** jose-phillips has quit IRC20:58
slaweqso if it's that then this feature was not working for LB agent for quite long time :)20:58
slaweqhaleyb: I found it from "git blame" so it's in current code with this commit id20:59
*** Leom has joined #openstack-neutron20:59
*** Leo_ has quit IRC21:00
haleybslaweq: so the zones are required due to overlapping IP, and of course that commit message says "Currently this patch only works for OVS agent"21:01
slaweqyes21:01
slaweqand it cause bug with LB agent :/21:01
slaweqI think it should be quite easy to add zones to IptablesFirewallDriver class (but I might be wrong)21:02
slaweqwhat You think? is it good idea to try to go this way to fix this issue?21:02
*** sam_s has quit IRC21:02
*** liangy has quit IRC21:03
*** haplo37_ has quit IRC21:05
haleybslaweq: i don't know if it's easy or not from looking, but it might not be required, i'm thinking if it's partly due to DVR on computes21:06
slaweqhaleyb: ok, I will try to do something with it :)21:06
slaweqto fix this issue21:06
slaweqcan You maybe update status of this bug on launchpad? or it's not necessary?21:07
kevinbentonslaweq, haleyb: adding zones to linuxbridge shouldn't be needed21:07
*** sdague has joined #openstack-neutron21:07
*** vhosakot has quit IRC21:08
haleybkevinbenton: was it partly due to dvr?  i'm blanking21:08
kevinbentonhaleyb: it was because when two nodes communicated on the same host21:08
*** vhosakot has joined #openstack-neutron21:08
kevinbentonhaleyb: it would cross iptables twice21:08
*** thiagolib has quit IRC21:08
kevinbentonhaleyb: once on the filtering bridge between the sender and OVS21:09
kevinbentonhaleyb: and once again on the filtering bridge between the receiver and OVS21:09
slaweqkevinbenton: hello, so I should change IpConntrackManager class?21:09
haleybkevinbenton: ah, so the CT entries were "crossing" so to speak, with the wrong one getting hit21:09
kevinbentonhaleyb: well what would happen is when one side would generate a TCP reset, iptables would immediately removed the tracked connection on the first cross21:10
kevinbentonhaleyb: then when it would hit iptables on the receiver, it wouldn't match an active connection so the TCP RST would be dropped21:10
kevinbentonhaleyb: and the instance would never get the reset21:10
kevinbentonslaweq: yeah, is it easy to tell from there if there is no conntrack zone set?21:11
haleybkevinbenton: so it sounds like in this case, the zone isn't in use, but conntrack -D is using the argument when it shouldn't be21:11
kevinbentonhaleyb: yeah, sounds that way21:11
slaweqhaleyb: exactly21:11
*** cleong has quit IRC21:13
kevinbentonhaleyb, slaweq: however, upon re-explaining that, it occured to me that this can still happen with linuxbridge if the instances are on two different networks21:13
kevinbenton:)21:13
kevinbentonso we probably should add zone support in21:13
*** eezhova has quit IRC21:13
kevinbentonif they are on two networks communicating via a router, a single packet will flow through iptables twice (once on each network bridge)21:13
kevinbentonslaweq: i would proceed with the easy fix for now so we can back-port it21:14
kevinbentonthen we should probably add CT support to the regular iptables driver linux bridge uses21:14
slaweqkevinbenton: ok, so I will try to make easy fix to conntrack manager now21:14
slaweqthx for help21:15
kevinbentonno prob21:15
kevinbentonihrachys, otherwiseguy: does OVS support instant messenger lingo now?21:15
kevinbentonmaybe 'k' was meant to be an acknowledgement21:15
ihrachys:)21:16
haleybslaweq: does a grep -e "Assigned CT" in the log show just a single zone in use?21:16
*** donghao has joined #openstack-neutron21:16
*** haplo37_ has joined #openstack-neutron21:16
slaweqhaleyb: You mean in linuxbridge agent log?21:16
haleybslaweq: yes, i see a different zone for each port in the ovs agent log21:16
ihrachyskevinbenton: one other thing in the same log that I can't figure out is - we see that timeout happening for the bridge @ ...:16, but then later in the run, just before deleteBridge operation, @ :17, we have another attempt to add the bridge21:17
kevinbentonhaleyb: to answer your earlier question about my dhcp lock patch, yeah, the original thought was to save an unnecessary lock21:17
ihrachyshere: http://logs.openstack.org/43/408143/9/check/gate-neutron-dsvm-functional-ubuntu-xenial/8926e44/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2017-01-27_02_41_17_54421:17
haleybslaweq: or maybe there is no assigned zone of course21:17
ihrachysnote it happens after (re?) connect to ovsdb tcp port21:17
ihrachysa second later21:18
slaweqhaleyb: there is no such entry in log file21:18
*** vhosakot has quit IRC21:18
*** kevo has joined #openstack-neutron21:18
*** leitan has quit IRC21:19
ihrachyswhich probably correspond to https://github.com/openstack/neutron/blob/master/neutron/agent/ovsdb/native/connection.py#L14521:19
slaweqbut when I removed SG rule I saw that conntrack was called with "-w 1" for one port and for "-w 2" for second port from same network21:19
kevinbentonihrachys: this may be what otherwiseguy was worried about21:19
slaweqsoe it looks that conntrack manager assigns zone "per port"21:19
kevinbentonihrachys: us using timeout on the poller may leave the previous command in the queue21:19
ihrachysyea21:19
ihrachysthat's what I was about to suggest21:19
kevinbentonslaweq: yeah, zones are per port21:20
kevinbentonslaweq: the original per network zones were for overlapping IP issues21:20
*** donghao has quit IRC21:20
kevinbentonslaweq: the per port zones were added later for the double cross issue and TCP resets21:20
ihrachyskevinbenton: and also, shouldn't we see connecting messages to the port before first attempt to add the bridge?21:20
*** eezhova has joined #openstack-neutron21:20
ihrachysor do we reuse some connection pool in ovs lib?21:20
kevinbentonihrachys: connection is stored on class level attribute IIRC in ovs lib21:21
kevinbentonihrachys: so it is re-used21:21
*** panda is now known as panda|zZ21:24
*** nlahouti has joined #openstack-neutron21:24
*** dsneddon has joined #openstack-neutron21:25
*** armax has joined #openstack-neutron21:26
*** eezhova_ has joined #openstack-neutron21:28
*** eezhova has quit IRC21:29
*** tbachman has quit IRC21:31
*** absubram has quit IRC21:33
*** abhiraut has joined #openstack-neutron21:33
*** Jeffrey4l_ has quit IRC21:35
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 7  https://review.openstack.org/40955721:36
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 5  https://review.openstack.org/40955621:36
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 4  https://review.openstack.org/40955521:36
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 3  https://review.openstack.org/40955421:36
openstackgerritHenry Gessau proposed openstack/neutron: Decouple hook and func registration from CommonDbMixin  https://review.openstack.org/40955321:36
openstackgerritHenry Gessau proposed openstack/neutron: [WIP] Operation Pig Bristle, part 6  https://review.openstack.org/40783721:36
*** gcheresh_ has quit IRC21:40
*** vhosakot has joined #openstack-neutron21:40
*** Jeffrey4l_ has joined #openstack-neutron21:41
*** nplanel has joined #openstack-neutron21:41
*** tbachman has joined #openstack-neutron21:42
*** catintheroof has quit IRC21:44
*** catintheroof has joined #openstack-neutron21:45
kevinbentonslaweq, haleyb: yeah, i remember now that when the zones were originally added it was per network to solve the overlapping IP issue21:47
kevinbentonwhich linuxbridge didn't have at all21:47
kevinbentonThen when they were changed to per port to deal with the multiple table hits, I forgot to consider that linux bridge would have that issue as well21:48
openstackgerritSwaminathan Vasudevan proposed openstack/neutron: DVR: Add forwarding routes based on address_scopes  https://review.openstack.org/35506221:49
*** catintheroof has quit IRC21:49
*** ijw_ has quit IRC21:51
*** absubram has joined #openstack-neutron21:55
*** gvrangan has quit IRC21:57
kevinbentonihrachys: i wonder if the issue is that a reconnect wont' occur while in that block() function21:59
kevinbentonihrachys: so whenever a disconnect happens, it hangs the ongoing connection until we throw the timeout error22:00
kevinbentonihrachys: what is the magic incantation to make otherwiseguy appear? :)22:00
otherwiseguyi'm here!22:01
*** absubram has quit IRC22:01
* otherwiseguy reads the scrollback22:02
kevinbentonotherwiseguy: ihar observed a late re-issuance of the same command22:02
kevinbentonotherwiseguy: and it sort of sounded like exactly like what you were worried could happen22:03
kevinbentonotherwiseguy: the timeout left a command on the queue or something22:03
*** rmart04 has joined #openstack-neutron22:03
kevinbentonotherwiseguy: http://logs.openstack.org/43/408143/9/check/gate-neutron-dsvm-functional-ubuntu-xenial/8926e44/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_agent_mtu_set_on_interface_driver.txt.gz#_2017-01-27_02_41_16_45222:04
kevinbentonotherwiseguy: notice that test-brca7476e2 addbridge gets called again towards the end right before deletebridge is issued22:04
*** rmart04 has left #openstack-neutron22:04
otherwiseguyi wonder what caused the connection dropped (protocol error)22:05
ihrachysotherwiseguy: could it be something left by a previous test?22:06
ihrachysotherwiseguy: I started looking at logs of other native tests but haven't got far yet22:06
ihrachysI mean those executed by this test worker22:06
*** trown is now known as trown|outtypewww22:08
*** Leom has quit IRC22:08
*** Leo_ has joined #openstack-neutron22:08
kevinbentonihrachys: that reminds me. each test worker is a separate process22:08
kevinbentonihrachys: right?22:08
*** vhosakot_ has joined #openstack-neutron22:08
otherwiseguyihrachys, I mean, I guess it's possible. It obviously *shouldn't* be...and all of the actual creation of ovsdb protocol messages is done by OVS's lib, so why there would be one beginning with a 'k' instead of valid JSON is a mystery.22:09
ihrachyskevinbenton: yea22:10
kevinbentonotherwiseguy: we can't log the full message on a failure like that?22:10
kevinbentonihrachys: doesn't that mean each function process has its own ovsdb monitor?22:10
ihrachysprobably. what are you up for?22:11
*** jose-phillips has joined #openstack-neutron22:11
kevinbentonwhich one is the one we have to tell ovs to connect to22:12
kevinbentonand listen locally22:12
kevinbentonis that the native OFCTL replacement?22:12
*** vhosakot has quit IRC22:12
ihrachysit should be native22:12
kevinbentonhow do we pick port numbers for that?22:12
otherwiseguykevinbenton, the native.vlog messages are generated by the OVS lib itself.22:13
ihrachysdefine that22:13
*** gvrangan has joined #openstack-neutron22:13
ihrachysotherwiseguy: can we make the library log more?22:13
*** vishwanathj has joined #openstack-neutron22:13
otherwiseguyihrachys, more than "2017-01-27 02:41:16.452 27330 WARNING neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: error parsing stream: line 0, column 1, byte 1: invalid keyword 'k'"22:13
otherwiseguy?22:13
*** ihrachys_ has joined #openstack-neutron22:14
*** vishwanathj has quit IRC22:14
otherwiseguyI can look at the code and see if it has access to anything else.22:14
ihrachys_ok folks I gotta run. shoot anything relevant my way, I will check later in the evening. need to get out of the bay before the Traffic hits the road22:15
*** vishwanathj has joined #openstack-neutron22:15
*** ihrachys_ has quit IRC22:15
*** abhiraut has quit IRC22:15
*** thorst_ has quit IRC22:16
*** abhiraut has joined #openstack-neutron22:16
*** vishwanathj has quit IRC22:17
*** ihrachys has quit IRC22:17
*** vhosakot_ has quit IRC22:17
kevinbentonok22:18
*** limao has joined #openstack-neutron22:23
otherwiseguykevinbenton, It looks like we could probably add the logging of the current block of data being processed. There's a chance it wouldn't give anything new since the parse failure is showing at byte 1.22:24
kevinbentonotherwiseguy: yeah, i was thinking maybe if we could see the whole like it would be more obvious what leaked onto that22:24
kevinbentonwhole line*22:25
kevinbentonwhole blob22:25
kevinbentonwhatever you want to call it :)22:25
*** gouthamr has quit IRC22:27
*** armax has quit IRC22:27
*** kriskend has quit IRC22:28
*** baoli has quit IRC22:28
otherwiseguyMight be able to monkeypatch the ovs lib's jsonrpc.py's recv() function and have it dump self.input if it returns an error. Not sure if they'd want a patch adding the output to a debug log or not.22:28
*** amarao has quit IRC22:29
otherwiseguyBut since this doesn't reproduce easily, just having a testing patch that monkeypatches it might not be that helpful.22:29
otherwiseguykevinbenton, ^22:30
*** slaweq_ has quit IRC22:30
*** PramodJ has joined #openstack-neutron22:31
*** Leom has joined #openstack-neutron22:32
*** Leo_ has quit IRC22:33
*** pramodrj07 has quit IRC22:34
otherwiseguykevinbenton, weird, "error parsing stream" happens in a very concentrated group in the last 7 days: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20parsing%20stream%5C%2222:35
*** limao has quit IRC22:40
*** limao has joined #openstack-neutron22:40
*** abhiraut has quit IRC22:40
*** abhiraut1 has joined #openstack-neutron22:40
otherwiseguykevinbenton, wrt the AddBridgeCommand being called twice, the first one looks like it was told to TRY_AGAIN righ before the connection dropped. The situation that I was worried about is Req sent/processed off the queue, immediately it waits for a response. The response is delayed and times out. Then the response arrives in the queue. Then another command is called, and the queued result gets sent to it.22:41
*** jaosorior has quit IRC22:42
otherwiseguyApparently things are not working well when there is a disconnection. The OVS lib has a state machine to handle reconnects transparently in the background, but who knows...22:43
*** hishh has quit IRC22:46
*** ijw has joined #openstack-neutron22:47
*** ijw has quit IRC22:47
*** ijw has joined #openstack-neutron22:47
*** abhiraut1 has quit IRC22:47
*** nplanel has quit IRC22:47
*** mriedem_beach is now known as mriedem22:48
*** kbaegis has quit IRC22:48
*** kbaegis has joined #openstack-neutron22:49
*** kbaegis has quit IRC22:49
*** gouthamr has joined #openstack-neutron22:51
*** princenana has quit IRC22:51
*** Swami_ has joined #openstack-neutron22:53
*** kbringard has quit IRC22:53
*** Swami has quit IRC22:57
*** jamesdenton has quit IRC22:58
*** Swami_ has quit IRC23:00
*** jperry has quit IRC23:01
*** tbachman has quit IRC23:01
*** limao has quit IRC23:01
*** limao has joined #openstack-neutron23:02
*** slaweq_ has joined #openstack-neutron23:04
openstackgerritMerged openstack/neutron: Handle attempt to enable br_netfilter in namespace  https://review.openstack.org/42377723:05
*** annegentle has quit IRC23:06
*** greghaynes has quit IRC23:10
*** Leom has quit IRC23:10
*** Leo_ has joined #openstack-neutron23:11
*** abhiraut has joined #openstack-neutron23:11
*** nplanel has joined #openstack-neutron23:14
*** abhiraut has quit IRC23:16
*** marst has quit IRC23:20
*** tbachman has joined #openstack-neutron23:21
*** limao has quit IRC23:22
*** limao has joined #openstack-neutron23:23
openstackgerritOpenStack Proposal Bot proposed openstack/neutron: Updated from global requirements  https://review.openstack.org/42364523:23
*** trevormc has quit IRC23:24
openstackgerritSlawek Kaplonski proposed openstack/neutron: Clear conntrack entries without zones if CT zones are not used  https://review.openstack.org/42642923:29
*** eezhova_ has quit IRC23:30
slaweqhaleyb kevinbenton: ^^ I just make some patch to not use CT zones in Iptables driver23:31
slaweqif You will have a while it would be nice if You can have a look on it :)23:31
*** greghaynes has joined #openstack-neutron23:37
*** abhiraut has joined #openstack-neutron23:42
*** nplanel has quit IRC23:42
*** Swami has joined #openstack-neutron23:43
*** limao has quit IRC23:44
*** jamesden_ has joined #openstack-neutron23:44
*** Swami has quit IRC23:44
*** limao has joined #openstack-neutron23:44
*** pramodrj07 has joined #openstack-neutron23:44
*** Swami has joined #openstack-neutron23:44
*** PramodJ has quit IRC23:48
*** Swami_ has joined #openstack-neutron23:48
openstackgerritAradhana Singh proposed openstack/neutron: Refactoring config options for ml2 config opts  https://review.openstack.org/34022823:49
*** Swami has quit IRC23:52
*** john-davidge has joined #openstack-neutron23:52
*** janzian has quit IRC23:52
*** greghaynes has quit IRC23:52
*** tbachman has quit IRC23:55
*** john-davidge has quit IRC23:56
*** wolverineav has quit IRC23:57
*** wolverineav has joined #openstack-neutron23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!