| *** martial_ has quit IRC | 00:09 | |
| *** brinzhang_ has joined #openstack-meeting | 00:09 | |
| *** slaweq has joined #openstack-meeting | 00:11 | |
| *** vishalmanchanda has joined #openstack-meeting | 00:11 | |
| *** brinzhang has quit IRC | 00:13 | |
| *** slaweq has quit IRC | 00:15 | |
| *** igordc has joined #openstack-meeting | 00:21 | |
| *** yamamoto has joined #openstack-meeting | 00:23 | |
| *** brinzhang has joined #openstack-meeting | 00:25 | |
| *** yamamoto has quit IRC | 00:28 | |
| *** brinzhang_ has quit IRC | 00:28 | |
| *** macz_ has quit IRC | 00:33 | |
| *** igordc has quit IRC | 00:40 | |
| *** jmasud has joined #openstack-meeting | 00:42 | |
| *** jmasud has quit IRC | 00:53 | |
| *** brinzhang_ has joined #openstack-meeting | 00:55 | |
| *** jmasud has joined #openstack-meeting | 00:55 | |
| *** brinzhang has quit IRC | 00:58 | |
| *** ykatabam has quit IRC | 00:58 | |
| *** ykatabam has joined #openstack-meeting | 00:59 | |
| *** njohnston_ has joined #openstack-meeting | 01:01 | |
| *** njohnston_ is now known as njohnston|phone | 01:01 | |
| *** slaweq has joined #openstack-meeting | 01:11 | |
| *** slaweq has quit IRC | 01:16 | |
| *** masahito has joined #openstack-meeting | 01:17 | |
| *** igordc has joined #openstack-meeting | 01:20 | |
| *** njohnston|phone has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:30 | |
| *** igordc has quit IRC | 01:33 | |
| *** jamesmcarthur has quit IRC | 01:39 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:40 | |
| *** jamesmcarthur has quit IRC | 01:54 | |
| *** yamamoto has joined #openstack-meeting | 01:55 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:57 | |
| *** ykatabam has quit IRC | 01:59 | |
| *** rcernin has quit IRC | 02:02 | |
| *** masahito_ has joined #openstack-meeting | 02:04 | |
| *** masahito has quit IRC | 02:04 | |
| *** masahito has joined #openstack-meeting | 02:05 | |
| *** masahito_ has quit IRC | 02:05 | |
| *** masahito has quit IRC | 02:07 | |
| *** jamesmcarthur has quit IRC | 02:08 | |
| *** yamamoto has quit IRC | 02:09 | |
| *** gyee has quit IRC | 02:09 | |
| *** yamamoto has joined #openstack-meeting | 02:09 | |
| *** slaweq has joined #openstack-meeting | 02:11 | |
| *** slaweq has quit IRC | 02:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:16 | |
| *** brinzhang has joined #openstack-meeting | 02:31 | |
| *** brinzhang_ has quit IRC | 02:34 | |
| *** hyunsikyang has joined #openstack-meeting | 02:43 | |
| *** hyunsikyang__ has quit IRC | 02:46 | |
| *** diablo_rojo has quit IRC | 02:48 | |
| *** hongbin has joined #openstack-meeting | 02:50 | |
| *** ykatabam has joined #openstack-meeting | 02:51 | |
| *** rcernin has joined #openstack-meeting | 02:52 | |
| *** slaweq has joined #openstack-meeting | 03:11 | |
| *** jamesmcarthur has quit IRC | 03:13 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:14 | |
| *** slaweq has quit IRC | 03:16 | |
| *** apetrich has quit IRC | 03:18 | |
| *** links has joined #openstack-meeting | 03:19 | |
| *** ijw has quit IRC | 03:29 | |
| *** jamesmcarthur has quit IRC | 03:31 | |
| *** psachin has joined #openstack-meeting | 03:33 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:53 | |
| *** apetrich has joined #openstack-meeting | 03:56 | |
| *** dmacpher_ has joined #openstack-meeting | 04:03 | |
| *** dmacpher has quit IRC | 04:07 | |
| *** slaweq has joined #openstack-meeting | 04:11 | |
| *** slaweq has quit IRC | 04:17 | |
| *** hyunsikyang has quit IRC | 04:32 | |
| *** hyunsikyang has joined #openstack-meeting | 04:33 | |
| *** jamesmcarthur has quit IRC | 04:50 | |
| *** jmasud has quit IRC | 04:53 | |
| *** hongbin has quit IRC | 04:54 | |
| *** jmasud has joined #openstack-meeting | 04:55 | |
| *** artom has quit IRC | 04:59 | |
| *** diablo_rojo has joined #openstack-meeting | 05:04 | |
| *** haleyb is now known as haleyb|away | 05:05 | |
| *** slaweq has joined #openstack-meeting | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** igordc has joined #openstack-meeting | 05:38 | |
| *** yamamoto has quit IRC | 05:56 | |
| *** slaweq has joined #openstack-meeting | 06:11 | |
| *** yamamoto has joined #openstack-meeting | 06:11 | |
| *** igordc has quit IRC | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** diablo_rojo has quit IRC | 06:17 | |
| *** vishalmanchanda has quit IRC | 06:31 | |
| *** vishalmanchanda has joined #openstack-meeting | 06:55 | |
| *** e0ne has quit IRC | 06:55 | |
| *** lpetrut has joined #openstack-meeting | 07:05 | |
| *** lpetrut has quit IRC | 07:06 | |
| *** lpetrut has joined #openstack-meeting | 07:06 | |
| *** slaweq has joined #openstack-meeting | 07:11 | |
| *** slaweq has quit IRC | 07:15 | |
| *** yamamoto has quit IRC | 07:25 | |
| *** hjensas has quit IRC | 07:36 | |
| *** slaweq has joined #openstack-meeting | 07:38 | |
| *** JangwonLee_ has quit IRC | 07:41 | |
| *** slaweq has quit IRC | 07:43 | |
| *** bbowen_ has joined #openstack-meeting | 07:46 | |
| *** bbowen has quit IRC | 07:47 | |
| *** bbowen_ has quit IRC | 07:48 | |
| *** bbowen_ has joined #openstack-meeting | 07:48 | |
| *** witek has joined #openstack-meeting | 07:49 | |
| *** slaweq has joined #openstack-meeting | 07:49 | |
| *** kozhukalov has quit IRC | 07:51 | |
| *** hyunsikyang__ has joined #openstack-meeting | 07:58 | |
| *** yamamoto has joined #openstack-meeting | 08:01 | |
| *** hyunsikyang has quit IRC | 08:02 | |
| *** yamamoto has quit IRC | 08:06 | |
| *** masahito_ has joined #openstack-meeting | 08:09 | |
| *** tesseract has joined #openstack-meeting | 08:09 | |
| *** kozhukalov has joined #openstack-meeting | 08:13 | |
| *** jmasud has quit IRC | 08:13 | |
| *** maciejjozefczyk has joined #openstack-meeting | 08:14 | |
| *** rpittau|afk is now known as rpittau | 08:16 | |
| *** masahito_ has quit IRC | 08:16 | |
| *** masahito has joined #openstack-meeting | 08:17 | |
| *** jmasud has joined #openstack-meeting | 08:19 | |
| *** hjensas has joined #openstack-meeting | 08:21 | |
| *** yamamoto has joined #openstack-meeting | 08:32 | |
| *** ralonsoh has joined #openstack-meeting | 08:33 | |
| *** brinzhang_ has joined #openstack-meeting | 08:36 | |
| *** kozhukalov has quit IRC | 08:39 | |
| *** brinzhang has quit IRC | 08:39 | |
| *** brinzhang has joined #openstack-meeting | 08:40 | |
| *** brinzhang_ has quit IRC | 08:41 | |
| *** brinzhang_ has joined #openstack-meeting | 08:42 | |
| *** brinzhang has quit IRC | 08:44 | |
| *** brinzhang has joined #openstack-meeting | 08:48 | |
| *** dmacpher__ has joined #openstack-meeting | 08:49 | |
| *** brinzhang_ has quit IRC | 08:51 | |
| *** brinzhang_ has joined #openstack-meeting | 08:52 | |
| *** dmacpher_ has quit IRC | 08:53 | |
| *** jmasud has quit IRC | 08:53 | |
| *** brinzhang has quit IRC | 08:53 | |
| *** Liang__ has joined #openstack-meeting | 08:55 | |
| *** jmasud has joined #openstack-meeting | 08:56 | |
| *** e0ne has joined #openstack-meeting | 09:05 | |
| *** masahito has quit IRC | 09:11 | |
| *** masahito has joined #openstack-meeting | 09:15 | |
| *** brinzhang has joined #openstack-meeting | 09:24 | |
| *** brinzhang_ has quit IRC | 09:26 | |
| *** ociuhandu has joined #openstack-meeting | 09:35 | |
| *** JangwonLee has joined #openstack-meeting | 09:39 | |
| *** yamamoto has quit IRC | 09:44 | |
| *** apetrich has quit IRC | 09:52 | |
| *** brinzhang_ has joined #openstack-meeting | 09:52 | |
| *** brinzhang_ has quit IRC | 09:54 | |
| *** brinzhang_ has joined #openstack-meeting | 09:54 | |
| *** brinzhang has quit IRC | 09:55 | |
| *** masahito has quit IRC | 09:57 | |
| *** ociuhandu has quit IRC | 10:04 | |
| *** ykatabam has quit IRC | 10:08 | |
| *** ociuhandu has joined #openstack-meeting | 10:19 | |
| *** yamamoto has joined #openstack-meeting | 10:22 | |
| *** brinzhang_ has quit IRC | 10:22 | |
| *** yamamoto has quit IRC | 10:27 | |
| *** rubasov has joined #openstack-meeting | 10:34 | |
| *** ykatabam has joined #openstack-meeting | 10:48 | |
| *** rcernin has quit IRC | 11:08 | |
| *** rpittau is now known as rpittau|bbl | 11:32 | |
| *** psachin has quit IRC | 11:33 | |
| *** JangwonLee_ has joined #openstack-meeting | 11:35 | |
| *** JangwonLee__ has joined #openstack-meeting | 11:36 | |
| *** psachin has joined #openstack-meeting | 11:37 | |
| *** JangwonLee has quit IRC | 11:38 | |
| *** JangwonLee_ has quit IRC | 11:40 | |
| *** psachin has quit IRC | 11:49 | |
| *** JangwonLee_ has joined #openstack-meeting | 11:50 | |
| *** ykatabam has quit IRC | 11:53 | |
| *** JangwonLee__ has quit IRC | 11:54 | |
| *** apetrich has joined #openstack-meeting | 12:01 | |
| *** ykatabam has joined #openstack-meeting | 12:06 | |
| *** ykatabam has quit IRC | 12:08 | |
| *** raildo has joined #openstack-meeting | 12:19 | |
| *** yamamoto has joined #openstack-meeting | 12:27 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** vishalmanchanda has quit IRC | 12:31 | |
| *** jiaopengju2 has quit IRC | 12:32 | |
| *** johnthetubaguy has joined #openstack-meeting | 12:32 | |
| *** jiaopengju2 has joined #openstack-meeting | 12:32 | |
| *** TheJulia has quit IRC | 12:54 | |
| *** _erlon_ has quit IRC | 12:54 | |
| *** jmasud has quit IRC | 12:54 | |
| *** TheJulia has joined #openstack-meeting | 12:54 | |
| *** patrickeast has quit IRC | 12:55 | |
| *** _erlon_ has joined #openstack-meeting | 12:55 | |
| *** masayukig has quit IRC | 12:55 | |
| *** patrickeast has joined #openstack-meeting | 12:55 | |
| *** masayukig has joined #openstack-meeting | 12:56 | |
| *** jmasud has joined #openstack-meeting | 12:56 | |
| *** liuyulong has joined #openstack-meeting | 13:01 | |
| *** yamamoto has joined #openstack-meeting | 13:01 | |
| *** yamamoto has quit IRC | 13:06 | |
| *** jamesdenton has quit IRC | 13:07 | |
| *** hjensas has quit IRC | 13:11 | |
| *** nicolasbock has joined #openstack-meeting | 13:11 | |
| *** masahito has joined #openstack-meeting | 13:11 | |
| *** artom has joined #openstack-meeting | 13:19 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:23 | |
| *** vishakha has quit IRC | 13:23 | |
| *** jamesmcarthur has quit IRC | 13:23 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:24 | |
| *** xyang has quit IRC | 13:24 | |
| *** efried1 has joined #openstack-meeting | 13:24 | |
| *** xyang has joined #openstack-meeting | 13:25 | |
| *** larainema has joined #openstack-meeting | 13:25 | |
| *** efried has quit IRC | 13:25 | |
| *** efried1 is now known as efried | 13:25 | |
| *** hjensas has joined #openstack-meeting | 13:27 | |
| *** enriquetaso has joined #openstack-meeting | 13:30 | |
| *** jamesmcarthur has quit IRC | 13:32 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:33 | |
| *** vishalmanchanda has joined #openstack-meeting | 13:36 | |
| *** jamesmcarthur has quit IRC | 13:38 | |
| *** rpittau|bbl is now known as rpittau | 13:44 | |
| *** masahito has quit IRC | 13:45 | |
| *** Liang__ is now known as LiangFang | 13:56 | |
| *** sfernand has joined #openstack-meeting | 13:58 | |
| liuyulong | #startmeeting neutron_l3 | 14:00 |
|---|---|---|
| openstack | Meeting started Wed Mar 4 14:00:14 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:00 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:00 |
| *** rubasov has quit IRC | 14:01 | |
| liuyulong | Hi there | 14:02 |
| liuyulong | #topic Announcements | 14:02 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:02 | |
| slaweq | hi | 14:03 |
| liuyulong | #link https://www.openstack.org/events/opendev-ptg-2020/ | 14:03 |
| liuyulong | Hope I could get to Vancouver. | 14:04 |
| *** rbudden has joined #openstack-meeting | 14:05 | |
| liuyulong | I need a VISA. | 14:05 |
| liuyulong | I will try the community travel support. | 14:05 |
| *** yamamoto has joined #openstack-meeting | 14:06 | |
| slaweq | for now we also don't know how it will be, mostly due to this coronavirus :/ | 14:07 |
| liuyulong | #link https://etherpad.openstack.org/p/neutron-victoria-ptg | 14:07 |
| liuyulong | slaweq, maybe, but the Summer is coming. | 14:09 |
| liuyulong | Topics are wanted! ^^ | 14:09 |
| liuyulong | OK, no more announcement from me. | 14:11 |
| liuyulong | let's move on. | 14:11 |
| liuyulong | #topic Bugs | 14:11 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:11 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012766.html | 14:11 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/012926.html | 14:11 |
| liuyulong | Because I was not here last week, we have two lists now. | 14:11 |
| *** jamesmcarthur has joined #openstack-meeting | 14:11 | |
| liuyulong | First one: | 14:12 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1864963 | 14:12 |
| openstack | Launchpad bug 1864963 in neutron "loosing connectivity to instance with FloatingIP randomly" [Undecided,New] | 14:12 |
| liuyulong | I have left some questions about the reporters' deployment, that could help us to find out the real problem. | 14:12 |
| liuyulong | Mostly these questions are based on our local deployment. We met some issue on these fields. | 14:13 |
| slaweq | thx for taking care of this | 14:15 |
| *** psachin has joined #openstack-meeting | 14:15 | |
| liuyulong | slaweq, np | 14:16 |
| liuyulong | Next one | 14:16 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1865061 | 14:16 |
| openstack | Launchpad bug 1865061 in neutron "When neutron does a switch-over between router 1 and router2, the router1 conntrack flows shoud be deleted" [Low,Confirmed] | 14:16 |
| slaweq | that is something which our QE found during testing | 14:17 |
| slaweq | but it can br problem only if router will failover twice in short period of time | 14:17 |
| slaweq | and that's why it's set Low importance | 14:17 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| liuyulong | Yes, that is my question, how could that "twice" happen in real world? | 14:18 |
| liuyulong | https://bugs.launchpad.net/neutron/+bug/1865061/comments/1 | 14:18 |
| openstack | Launchpad bug 1865061 in neutron "When neutron does a switch-over between router 1 and router2, the router1 conntrack flows shoud be deleted" [Low,Confirmed] | 14:18 |
| liuyulong | We have "non-preemptive" settings for HA router keepalived. | 14:18 |
| liuyulong | So typically the "new-master" should work then. | 14:19 |
| liuyulong | The connections in the original host should be all broken. | 14:19 |
| slaweq | excactly, so I reported it there "just for the record" that such issue theoretically can happen | 14:19 |
| slaweq | but that shouldn't be in fact an issue in real world probably | 14:20 |
| liuyulong | extremely case is the HA networking is not stable. That could cause the HA router state change rapidly. For some deployment which running HA routers on hypervisors, the bad connection state could be a potential reason. | 14:23 |
| liuyulong | That could be another story. | 14:24 |
| liuyulong | OK, next one | 14:24 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1865891 | 14:24 |
| openstack | Launchpad bug 1865891 in neutron "Race condition during removal of subnet from the router and removal of subnet" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 14:24 |
| slaweq | yes, that one I'm working on now | 14:24 |
| slaweq | it seems that sometimes if You plug subnet to the router and in parallel remove subnet, Your router port will end up as port without fixed_ips | 14:25 |
| liuyulong | Alright | 14:25 |
| liuyulong | see my comment here: | 14:25 |
| liuyulong | https://bugs.launchpad.net/neutron/+bug/1865891/comments/2 | 14:25 |
| openstack | Launchpad bug 1865891 in neutron "Race condition during removal of subnet from the router and removal of subnet" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 14:25 |
| *** ricolin has quit IRC | 14:25 | |
| liuyulong | I can image another one is to add port as router interface and concurrently delete the port. | 14:25 |
| slaweq | I agree that maybe we will need to close it as "wontfix" | 14:26 |
| slaweq | but I want first to dig a bit more and see what can be done there | 14:26 |
| liuyulong | yes, it is indeed an issue. We just want to find out a balance. : ) | 14:28 |
| liuyulong | OK, next one | 14:28 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1865173 | 14:28 |
| openstack | Launchpad bug 1865173 in neutron "Revision number not bumped after update of router's description" [Low,Confirmed] | 14:28 |
| liuyulong | Tested on stable/queens, it is not reproducible. | 14:29 |
| slaweq | I was testing this on master branch | 14:29 |
| *** rubasov has joined #openstack-meeting | 14:31 | |
| liuyulong | Alright, a regression on router revision number. | 14:31 |
| slaweq | probably | 14:32 |
| slaweq | but I saw it only when I tried to bump router's description | 14:32 |
| liuyulong | Interesting... | 14:32 |
| slaweq | anyway, that's nothing really critical so I think it can stay in our backlog until someone will have some time to take a look at it | 14:33 |
| liuyulong | np, make sense to me | 14:33 |
| liuyulong | Next one: | 14:33 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1865557 | 14:33 |
| openstack | Launchpad bug 1865557 in neutron "Error reading log file from 'neutron-keepalived-state-change' in 'test_read_queue_send_garp'" [Low,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | 14:33 |
| ralonsoh | Just a logging problem | 14:33 |
| liuyulong | The fix is simple, and it is just fails the case not raise an exception. | 14:34 |
| ralonsoh | I found a problem only once, in a test | 14:34 |
| ralonsoh | as commented in the bug | 14:34 |
| ralonsoh | no no, we need to raise the exception | 14:34 |
| liuyulong | So I've +2ed that. | 14:34 |
| liuyulong | https://review.opendev.org/#/c/710850/1/neutron/tests/functional/agent/l3/test_keepalived_state_change.py | 14:34 |
| ralonsoh | ok, not an exception but a fail (the same effect) | 14:34 |
| ralonsoh | yes, I know | 14:35 |
| ralonsoh | because we are executing a test, it's better to use self.fail | 14:35 |
| ralonsoh | but the core of this patch is the extra log | 14:35 |
| liuyulong | OK, maybe I'm not clear here. | 14:35 |
| liuyulong | The fix is to just fail the case instead of raising an exception. | 14:36 |
| ralonsoh | the effect is the same | 14:36 |
| liuyulong | Yes | 14:36 |
| ralonsoh | the point is to increase the log info | 14:36 |
| ralonsoh | now we have the device list with the IP addresses | 14:36 |
| ralonsoh | inside the testing namespace | 14:36 |
| liuyulong | ralonsoh, great, thanks for working on this. | 14:37 |
| ralonsoh | yw | 14:37 |
| liuyulong | Alright, thag | 14:38 |
| liuyulong | Alright, that's all bugs from me today. | 14:38 |
| slaweq | I would like to talk about one also | 14:38 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1859832 | 14:38 |
| openstack | Launchpad bug 1859832 in neutron "L3 HA connectivity to GW port can be broken after reboot of backup node" [Medium,In progress] - Assigned to LIU Yulong (dragon889) | 14:38 |
| liuyulong | OK | 14:39 |
| slaweq | and those 2 alternative solutions proposed by me and liuyulong for it | 14:39 |
| slaweq | liuyulong: generally in Your approach I'm affraid those errors about fail to send garps during failover | 14:39 |
| *** ociuhandu has quit IRC | 14:40 | |
| slaweq | and the second potential issue is IMO if we will not increase downtime during failover as neutron-l3-agent has to be noticed that failover happened and bring gateway up then | 14:40 |
| slaweq | so 2 questions: | 14:40 |
| slaweq | 1. do You know if there is any way to delay sending of first garp, to avoid those errors from keepalived? | 14:41 |
| slaweq | 2. You said that You tested it in Your cloud, how long is downtime during failover with and without this patch? | 14:41 |
| liuyulong | I replied the comments in the patch set. Allow me quota it here: | 14:42 |
| liuyulong | We have run such code for a few months, no issue was found for such related log. Keepalived will send garp after a 60s delay by default [1], till then the L3 agent should have done qg-dev link up action. More details could be during the first phrase keepalived garp, do not send garp with no interval, it could have a 1 second delay (vrrp_garp_interval [2]). | 14:42 |
| liuyulong | [1] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/keepalived.py#L165 | 14:42 |
| liuyulong | [2] https://www.keepalived.org/manpage.html | 14:42 |
| liuyulong | Your first question could have the answer: vrrp_garp_interval. | 14:43 |
| *** puffc has joined #openstack-meeting | 14:44 | |
| liuyulong | The link up action is really quick, we have not seen any side effect on that. | 14:44 |
| slaweq | it's quick but if router has many other things to do, isn't it queued to be processed as other events? | 14:44 |
| liuyulong | More about that is the outside world also have ARP. | 14:45 |
| slaweq | e.g. if there would be many routers failovered in same time | 14:45 |
| liuyulong | HA state change does not have queue. | 14:45 |
| *** links has quit IRC | 14:45 | |
| liuyulong | It's not like the L3-agent main processing loop. | 14:46 |
| slaweq | ok, but can we maybe move this "set device up" action to the neutron-keepalived-state-change monitor process? | 14:46 |
| slaweq | so it would be done just after keepalived would configure VIP in the namespace | 14:46 |
| liuyulong | That "enqueue_state_change" actually does not have a "queue", it's just a list of functions. | 14:47 |
| slaweq | yes, but how about doing it here: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/keepalived_state_change.py#L89 | 14:48 |
| ralonsoh | slaweq, are we going to add net capabilities to the neutron-keepalived-state-change agent?? | 14:48 |
| ralonsoh | slaweq, I do not recommend it | 14:48 |
| ralonsoh | this should be just a monitoring process | 14:48 |
| slaweq | ralonsoh: look at the comment in https://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha.py#L166 | 14:49 |
| slaweq | according to it, such plans were already some time ago :) | 14:49 |
| liuyulong | That could be a heavy change. | 14:49 |
| ralonsoh | I still don't recommend it | 14:50 |
| ralonsoh | we'll have another service changing the net devices | 14:50 |
| ralonsoh | this should be in only one process: the l3 agent | 14:50 |
| liuyulong | We need router info from the l3-agent process to another monitor process. | 14:51 |
| *** nicolasbock has quit IRC | 14:51 | |
| slaweq | we already have keepalived which is also changing those interfaces | 14:51 |
| ralonsoh | yes | 14:51 |
| ralonsoh | but this is an external process not managed/programmed by us | 14:51 |
| slaweq | anyway, I really need to move forward with one of those potential fixes for this issue :) | 14:52 |
| ralonsoh | I know | 14:52 |
| slaweq | so first we should decide which one and then continue work on it | 14:52 |
| liuyulong | I prefer one fix for all drivers. | 14:53 |
| slaweq | liuyulong: yes, that's adventage for Your approach for sure | 14:53 |
| *** jamesmcarthur has joined #openstack-meeting | 14:53 | |
| ralonsoh | I still don't have a clear idea | 14:53 |
| ralonsoh | sorry | 14:54 |
| slaweq | what I'm affraid, is that this may cause some longer failover time | 14:54 |
| *** mmethot has quit IRC | 14:54 | |
| slaweq | but except that, I think that liuyulong's idea may be really better as it's more generic | 14:54 |
| liuyulong | And L3 issue should be handled in it's own scope by default. | 14:54 |
| liuyulong | slaweq, you have QA team I guess you mentioned in this meeting. : ) | 14:55 |
| *** priteau has joined #openstack-meeting | 14:55 | |
| slaweq | so ralonsoh what do You think if we will continue with liuyulong's patch? | 14:55 |
| liuyulong | We also have a QA team, I will try to make sure they have fully tested the fail-over time. | 14:56 |
| ralonsoh | I still need to check both again | 14:56 |
| slaweq | ralonsoh: ok, thx | 14:56 |
| slaweq | please check them | 14:56 |
| liuyulong | Another thing is I will try to add that "vrrp_garp_interval" for the VRRP of the HA router. | 14:56 |
| slaweq | liuyulong: and one more comment to this, can You remove config option from it? I don't think we really need such config option there | 14:57 |
| liuyulong | It will be an independent change. | 14:57 |
| slaweq | IMO this is internal implementation of HA routers and it shouldn't be configurable | 14:57 |
| liuyulong | slaweq, sure | 14:57 |
| *** david-lyle is now known as dklyle | 14:57 | |
| slaweq | ok, liuyulong please ping me if You will add this vrrp_garp_interval option | 14:58 |
| slaweq | I will test it again on my env | 14:58 |
| slaweq | and thx for working on this | 14:58 |
| liuyulong | slaweq, the config option is for our cloud locally, our operators would like to know the cloud code changes. | 14:58 |
| *** jamesmcarthur has quit IRC | 14:58 | |
| liuyulong | slaweq, np | 14:58 |
| slaweq | ok, that's all from my side | 14:58 |
| slaweq | thx | 14:58 |
| liuyulong | All right, we are out of time. | 14:59 |
| *** rh-jelabarre has joined #openstack-meeting | 14:59 | |
| liuyulong | let's end here. | 14:59 |
| liuyulong | Thank you guys for attending. | 14:59 |
| liuyulong | Bye | 14:59 |
| ralonsoh | bye | 14:59 |
| liuyulong | #endmeeting | 14:59 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:59 | |
| slaweq | bye | 14:59 |
| openstack | Meeting ended Wed Mar 4 14:59:31 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-03-04-14.00.html | 14:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-03-04-14.00.txt | 14:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-03-04-14.00.log.html | 14:59 |
| *** psachin has quit IRC | 15:00 | |
| *** nicolasbock has joined #openstack-meeting | 15:04 | |
| *** nicolasbock has quit IRC | 15:10 | |
| *** eharney has quit IRC | 15:10 | |
| *** ociuhandu has joined #openstack-meeting | 15:13 | |
| *** Lucas_Gray has joined #openstack-meeting | 15:15 | |
| *** ociuhandu has quit IRC | 15:18 | |
| *** lpetrut has quit IRC | 15:18 | |
| *** Lucas_Gray has quit IRC | 15:23 | |
| *** hjensas has quit IRC | 15:24 | |
| *** Lucas_Gray has joined #openstack-meeting | 15:25 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:28 | |
| *** Lucas_Gray has quit IRC | 15:29 | |
| *** ayoung has joined #openstack-meeting | 15:31 | |
| *** Lucas_Gray has joined #openstack-meeting | 15:31 | |
| *** jamesmcarthur has quit IRC | 15:33 | |
| *** psachin has joined #openstack-meeting | 15:40 | |
| *** TrevorV has joined #openstack-meeting | 15:46 | |
| *** yamamoto has quit IRC | 15:46 | |
| *** yamamoto has joined #openstack-meeting | 15:47 | |
| *** yamamoto has quit IRC | 15:50 | |
| *** masahito has joined #openstack-meeting | 15:50 | |
| *** eharney has joined #openstack-meeting | 15:52 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:55 | |
| *** masahito has quit IRC | 15:55 | |
| *** psachin has quit IRC | 15:56 | |
| *** ociuhandu has joined #openstack-meeting | 15:56 | |
| *** puffc has quit IRC | 16:01 | |
| *** Trevor_V has joined #openstack-meeting | 16:02 | |
| *** psachin has joined #openstack-meeting | 16:04 | |
| *** jamesmcarthur has quit IRC | 16:05 | |
| *** TrevorV has quit IRC | 16:06 | |
| *** ociuhandu has quit IRC | 16:11 | |
| *** ociuhandu has joined #openstack-meeting | 16:16 | |
| *** yamamoto has joined #openstack-meeting | 16:27 | |
| *** yamamoto has quit IRC | 16:33 | |
| *** ricolin has joined #openstack-meeting | 16:37 | |
| *** mattw4 has joined #openstack-meeting | 16:37 | |
| *** Lucas_Gray has quit IRC | 16:38 | |
| *** gyee has joined #openstack-meeting | 16:40 | |
| *** ociuhandu has quit IRC | 16:41 | |
| *** maciejjozefczyk_ has joined #openstack-meeting | 16:51 | |
| *** ociuhandu has joined #openstack-meeting | 16:51 | |
| *** maciejjozefczyk has quit IRC | 16:52 | |
| *** psachin has quit IRC | 16:53 | |
| *** maciejjozefczyk_ has quit IRC | 16:56 | |
| *** rubasov has quit IRC | 16:58 | |
| *** masahito has joined #openstack-meeting | 17:06 | |
| *** jamesmcarthur has joined #openstack-meeting | 17:11 | |
| *** masahito has quit IRC | 17:11 | |
| *** Trevor_V has quit IRC | 17:21 | |
| *** TrevorV has joined #openstack-meeting | 17:22 | |
| *** ayoung has quit IRC | 17:29 | |
| *** igordc has joined #openstack-meeting | 17:33 | |
| *** rpittau is now known as rpittau|afk | 17:44 | |
| *** njohnston has quit IRC | 17:53 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:54 | |
| *** ociuhandu has quit IRC | 17:57 | |
| *** njohnston has joined #openstack-meeting | 17:57 | |
| *** ociuhandu_ has quit IRC | 17:58 | |
| *** priteau has quit IRC | 17:59 | |
| *** igordc has quit IRC | 18:04 | |
| *** igordc has joined #openstack-meeting | 18:05 | |
| *** diablo_rojo has joined #openstack-meeting | 18:11 | |
| *** e0ne has quit IRC | 18:12 | |
| *** witek has quit IRC | 18:17 | |
| *** ijw has joined #openstack-meeting | 18:18 | |
| *** ijw has quit IRC | 18:18 | |
| *** ijw has joined #openstack-meeting | 18:19 | |
| *** ijw_ has joined #openstack-meeting | 18:20 | |
| *** ijw has quit IRC | 18:23 | |
| *** nicolasbock has joined #openstack-meeting | 18:24 | |
| *** igordc has quit IRC | 18:27 | |
| *** nicolasbock has quit IRC | 18:29 | |
| *** nicolasbock has joined #openstack-meeting | 18:30 | |
| *** igordc has joined #openstack-meeting | 18:33 | |
| *** tesseract has quit IRC | 18:55 | |
| *** jamesmcarthur has quit IRC | 18:56 | |
| *** sfernand has quit IRC | 19:17 | |
| *** larainema has quit IRC | 19:20 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:22 | |
| *** dmellado has quit IRC | 19:24 | |
| *** dmellado has joined #openstack-meeting | 19:25 | |
| *** jamesmcarthur has quit IRC | 19:33 | |
| *** ralonsoh has quit IRC | 19:38 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:47 | |
| *** eharney has quit IRC | 20:16 | |
| *** e0ne has joined #openstack-meeting | 20:17 | |
| *** alecuyer has joined #openstack-meeting | 20:35 | |
| *** jamesmcarthur has quit IRC | 20:49 | |
| *** patchbot has joined #openstack-meeting | 20:56 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Mar 4 21:00:13 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliverau | o/ | 21:00 |
| kota_ | hi | 21:00 |
| seongsoocho | o/ | 21:00 |
| clayg | I/ | 21:00 |
| alecuyer | o/ | 21:00 |
| rledisez | hi o/ | 21:00 |
| tdasilva | o/ | 21:01 |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke | i haven't got much planned, so straight to updates! | 21:01 |
| timburke | #topic concurrent EC | 21:01 |
| *** openstack changes topic to "concurrent EC (Meeting topic: swift)" | 21:01 | |
| timburke | clayg, how's it going? | 21:01 |
| clayg | Ok, I’ll push up my wip after the meeting. Folks can see t new approach. | 21:02 |
| *** raildo has quit IRC | 21:02 | |
| alecuyer | teasing us | 21:03 |
| clayg | I’ll probably do a new change? Might be easier to A/B | 21:03 |
| timburke | do you think it'll be in a ready-to-merge state, or still largely experimental? | 21:03 |
| timburke | (asking to set expectations, as much as anything) | 21:03 |
| clayg | alecuyer: no, it’s very WIP a lot of duplication that’ll need to be removed after we get it working | 21:03 |
| clayg | Not ready to merge. | 21:04 |
| timburke | 👍 | 21:04 |
| clayg | I’ll want some feedback | 21:04 |
| timburke | anything else you need now? | 21:04 |
| clayg | Not now. Next week. 👍 | 21:05 |
| timburke | sounds good | 21:05 |
| timburke | #topic 503 delays | 21:05 |
| *** openstack changes topic to "503 delays (Meeting topic: swift)" | 21:05 | |
| timburke | i wasn't sure if there was any other discussion needed here or not | 21:05 |
| clayg | I’m good on this topic. | 21:05 |
| *** ociuhandu has joined #openstack-meeting | 21:06 | |
| clayg | I think the whole idea is hostile to well behaved clients | 21:06 |
| clayg | I’m over it. | 21:06 |
| *** zaitcev has joined #openstack-meeting | 21:07 | |
| timburke | i guess the idea is that the client should ensure its own request smearing? | 21:07 |
| rledisez | i still see a point in it cause as a public cloud operator, we have no control on clients | 21:08 |
| timburke | my main questions are, would something as simple as sleep(random.random() * conf.max_unavailable_wait) work? defaulting it to zero, of course. and where would we want it to live? in the proxy, or somewhere in middleware? | 21:08 |
| *** masahito has joined #openstack-meeting | 21:08 | |
| timburke | in the proxy-server app, i mean | 21:09 |
| rledisez | I would say on the left of the pipeline because many things can happen without proxy-server being involved | 21:09 |
| * timburke nods | 21:10 | |
| timburke | probably ratelimit (for separation of concerns) or catch_errors, yeah? or its own thing | 21:10 |
| *** ociuhandu has quit IRC | 21:10 | |
| clayg | ratelimit seems reasonable | 21:11 |
| rledisez | ratelimit makes sense | 21:11 |
| *** jamesmcarthur has joined #openstack-meeting | 21:12 | |
| mattoliverau | +1 | 21:12 |
| timburke | oh yeah, i should look at the intereaction between ratelimit and s3api some more... | 21:12 |
| clayg | Hahaha | 21:13 |
| *** masahito has quit IRC | 21:13 | |
| timburke | i think it's not *terrible* following https://review.opendev.org/#/c/704659/ ? | 21:13 |
| patchbot | patch 704659 - swift (stable/stein) - s3api: Better handle 498/429 responses (MERGED) - 1 patch set | 21:13 |
| timburke | er, https://review.opendev.org/#/c/697535/ for the master version | 21:14 |
| patchbot | patch 697535 - swift - s3api: Better handle 498/429 responses (MERGED) - 1 patch set | 21:14 |
| clayg | Merged! | 21:14 |
| timburke | anyway | 21:14 |
| timburke | #topic losf | 21:14 |
| *** openstack changes topic to "losf (Meeting topic: swift)" | 21:14 | |
| clayg | I remember when things merged. | 21:14 |
| timburke | rledisez, how's it going? | 21:14 |
| rledisez | fine I guess, just an update about what's going on for LOSF | 21:14 |
| rledisez | as discussed at the last PTG in Shangai, the topic is not really lots of small files, but instead lots of big disks (having a consequence lots of files) | 21:14 |
| rledisez | this topic is back on our short-term roadmap cause we are experimenting with new hardware (36x14TB, 2 SSD, 96GB of RAM) | 21:14 |
| clayg | 🍿 | 21:15 |
| rledisez | we examined different option that could replace LOSF without having to maintaining them: | 21:15 |
| rledisez | XFS with rtdev option => not stable | 21:15 |
| rledisez | XFS with Intel CAS => not stable | 21:15 |
| rledisez | ZFS with metadata device => too much space needed for metadata + few operational issues | 21:15 |
| kota_ | wow | 21:15 |
| rledisez | in the end, we concluded that LOSF stays the best option. we made some changes that was discussed in Shangai: | 21:16 |
| rledisez | - new key format so that we can save a lot of CPU when doing replication | 21:16 |
| rledisez | - new option to store the LevelDB in a new path (on a fast device like SSD/NVMe) | 21:16 |
| rledisez | - store metadata in the LevelDB (TODO) | 21:16 |
| rledisez | and when I say we made some change, I mean alecuyer did it ;) | 21:16 |
| timburke | go alecuyer! | 21:17 |
| timburke | that all sounds great :D | 21:17 |
| rledisez | that's it for me on LOSF | 21:17 |
| alecuyer | heh, I guess :) So, I don't want to spend meeting time now on this ,but if anyone may be hitting these same issues we have, and want to guide the design/dev process, let's talk | 21:18 |
| clayg | It’s back on!!! | 21:18 |
| rledisez | so we're still involved in it, we just had to figure some stuff first | 21:18 |
| timburke | what were the issues you were seeing with the new boxes that brought it back around? RAM constraints still? | 21:18 |
| rledisez | yeah, the pattern will be the same => small files. if it was not working on 36x6TB, it will for sure not work on 36x14TB | 21:19 |
| alecuyer | inodes not fitting in the VFS cache | 21:19 |
| rledisez | also IO starvation because of replication because of cache miss in the VFS cache | 21:19 |
| timburke | make sense | 21:20 |
| rledisez | We have around 3 millions of objects per TB of disk. You can imagine the number of inode it implies (around 10M per TB of disks) | 21:20 |
| rledisez | well, maybe 8M, not 10 | 21:20 |
| timburke | #topic open discussion | 21:21 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:21 | |
| timburke | anything else to bring up today? | 21:21 |
| rledisez | zaitcev: I got your message, I loaded the two reviews. will have a look this week, I promess :) | 21:22 |
| zaitcev | rledisez: so, what can I help you with, then? You have any reviews for me? | 21:22 |
| timburke | oh, this was an interesting thread: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/012950.html | 21:22 |
| zaitcev | Hopefully not the whole LoSF | 21:23 |
| clayg | Do we need to auto retest failed tests in the gate before we fail the job? | 21:23 |
| rledisez | zaitcev: yeah, that would be unfair trading | 21:23 |
| kota_ | oh what | 21:23 |
| tdasilva | rledisez: are you still investigating those proxy level performance improvements? | 21:23 |
| timburke | clayg, yeah, i should look into that. i'm pretty sure there's a way to configure that... | 21:23 |
| rledisez | tdasilva: yes, I had to pause for the last 2 weeks, busy doing OVH-stuff, but I want to get back on it next week | 21:24 |
| tdasilva | rledisez: cool, looking forward to it.... | 21:24 |
| rledisez | tdasilva: I already have a review "ready" (I have a weird eventlet random failure in test). I'm now on MD5 replacement | 21:24 |
| tdasilva | rledisez: can you share gerrit # ? | 21:25 |
| rledisez | timburke: I admit I skipped the thread. in few words, any conclusion came out? | 21:25 |
| rledisez | tdasilva: https://review.opendev.org/#/c/697653/ actually this one is OK, I just need to rebase it | 21:26 |
| patchbot | patch 697653 - swift - Replace all "with Chunk*Timeout" by a watchdog - 5 patch sets | 21:26 |
| timburke | no real conclusion, just putting forward an idea of spreading the PTL responsibilities more (in part due to concerns about projects not having any self-nominees) | 21:26 |
| timburke | i don't expect *that* to be an issue for us ;-) | 21:27 |
| rledisez | tdasilva: this one has the random failure : https://review.opendev.org/#/c/704892/ | 21:27 |
| rledisez | running the test individually pass, running all of them at once fails | 21:27 |
| patchbot | patch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 1 patch set | 21:27 |
| timburke | right! i meant to dig into that but never got to it... | 21:27 |
| *** jmasud has quit IRC | 21:28 | |
| zaitcev | sounds like a challenge | 21:28 |
| timburke | so, i just realized we might want to poke more at libec -- see if we can avoid needing to do matrix inversion on every call to decode... | 21:29 |
| timburke | should be eminently cacheable | 21:29 |
| kota_ | hmm | 21:30 |
| timburke | https://github.com/openstack/liberasurecode/blob/master/src/backends/isa-l/isa_l_common.c#L211-L222 seems like we shouldn' | 21:30 |
| timburke | t need to do it on every call | 21:30 |
| tdasilva | just fyi, Vianney pinged me recently about quadiron lib patches, they are still waiting for reviews. I've been lacking on that, so if anyone has a chance | 21:31 |
| tdasilva | s/lib/libec | 21:31 |
| tdasilva | i think patch chain starts here: https://review.opendev.org/#/c/635603/1 | 21:32 |
| patchbot | patch 635603 - liberasurecode - fix: data access when having non-zero metadata size - 1 patch set | 21:32 |
| timburke | my trouble is that when i start doing libec reviews, i end up wanting to rewrite sizable parts of it all ;-) | 21:32 |
| rledisez | did anyone already looked into the glacier api, maybe to add it to s3api? | 21:33 |
| zaitcev | Sounds ambitious. | 21:33 |
| timburke | oh, and i should reach out to libphazr... i think that first patch would break them, but they don't have any CI set up with us... | 21:34 |
| timburke | all right | 21:38 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:38 |
| timburke | #endmeeting | 21:38 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:38 | |
| openstack | Meeting ended Wed Mar 4 21:38:36 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:38 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-03-04-21.00.html | 21:38 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-03-04-21.00.txt | 21:38 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-03-04-21.00.log.html | 21:38 |
| *** zaitcev has left #openstack-meeting | 21:38 | |
| *** patchbot has left #openstack-meeting | 21:38 | |
| mattoliverau | thanks timburke :) Sorry got lost reading the PTL thread. | 21:39 |
| timburke | no worries :-) i'm not even up-to-date on it, been distracted today | 21:39 |
| *** alecuyer has left #openstack-meeting | 21:42 | |
| *** yamamoto has joined #openstack-meeting | 21:43 | |
| *** jamesmcarthur has quit IRC | 21:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:44 | |
| *** bbowen_ has quit IRC | 21:46 | |
| *** bbowen_ has joined #openstack-meeting | 21:46 | |
| *** yamamoto has quit IRC | 21:49 | |
| *** eharney has joined #openstack-meeting | 21:52 | |
| *** jamesmcarthur has quit IRC | 21:56 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:57 | |
| *** enriquetaso has quit IRC | 21:59 | |
| *** jamesmcarthur has quit IRC | 22:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:10 | |
| *** slaweq has quit IRC | 22:21 | |
| *** jamesmcarthur has quit IRC | 22:22 | |
| *** rh-jelabarre has quit IRC | 22:26 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:32 | |
| *** nicolasbock has quit IRC | 22:35 | |
| *** jamesmcarthur has quit IRC | 22:37 | |
| *** rcernin has joined #openstack-meeting | 22:39 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:44 | |
| *** jamesmcarthur has quit IRC | 22:47 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:52 | |
| *** jamesmcarthur has quit IRC | 22:53 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:54 | |
| *** TrevorV has quit IRC | 22:55 | |
| *** jamesmcarthur has quit IRC | 23:02 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:09 | |
| *** slaweq has joined #openstack-meeting | 23:11 | |
| *** jamesmcarthur has quit IRC | 23:12 | |
| *** slaweq has quit IRC | 23:16 | |
| *** ykatabam has joined #openstack-meeting | 23:25 | |
| *** brinzhang has joined #openstack-meeting | 23:41 | |
| *** dklyle has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!