| *** macza has joined #openstack-meeting | 00:00 | |
| *** bobh has joined #openstack-meeting | 00:06 | |
| *** tetsuro has joined #openstack-meeting | 00:10 | |
| *** igordc has joined #openstack-meeting | 00:22 | |
| *** macza has quit IRC | 00:24 | |
| *** erlon has joined #openstack-meeting | 00:42 | |
| *** Liang__ has joined #openstack-meeting | 00:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:50 | |
| *** erlon has quit IRC | 01:05 | |
| *** _alastor_ has quit IRC | 01:05 | |
| *** igordc has quit IRC | 01:05 | |
| *** markvoelker has quit IRC | 01:08 | |
| *** slaweq has joined #openstack-meeting | 01:11 | |
| *** slaweq has quit IRC | 01:15 | |
| *** weshay is now known as weshay_PTO | 01:26 | |
| *** bobh has quit IRC | 01:40 | |
| *** jamesmcarthur has quit IRC | 01:52 | |
| *** markvoelker has joined #openstack-meeting | 02:09 | |
| *** hongbin has joined #openstack-meeting | 02:24 | |
| *** bbowen has quit IRC | 02:25 | |
| *** lbragstad_503 has quit IRC | 02:38 | |
| *** psachin has joined #openstack-meeting | 02:42 | |
| *** markvoelker has quit IRC | 02:43 | |
| *** bobh has joined #openstack-meeting | 02:44 | |
| *** hongbin has quit IRC | 02:58 | |
| *** hongbin has joined #openstack-meeting | 03:01 | |
| *** slaweq has joined #openstack-meeting | 03:11 | |
| *** apetrich has quit IRC | 03:14 | |
| *** slaweq has quit IRC | 03:15 | |
| *** hongbin has quit IRC | 03:16 | |
| *** hongbin has joined #openstack-meeting | 03:17 | |
| *** hongbin has quit IRC | 03:17 | |
| *** hongbin has joined #openstack-meeting | 03:17 | |
| *** hongbin has quit IRC | 03:20 | |
| *** hongbin has joined #openstack-meeting | 03:20 | |
| *** bobh has quit IRC | 03:21 | |
| *** hongbin has quit IRC | 03:21 | |
| *** bbowen has joined #openstack-meeting | 03:21 | |
| *** hongbin has joined #openstack-meeting | 03:21 | |
| *** hongbin has quit IRC | 03:22 | |
| *** hongbin has joined #openstack-meeting | 03:22 | |
| *** armax has joined #openstack-meeting | 03:23 | |
| *** rcernin has quit IRC | 03:35 | |
| *** markvoelker has joined #openstack-meeting | 03:41 | |
| *** baojg has joined #openstack-meeting | 03:48 | |
| *** tpatil has joined #openstack-meeting | 03:50 | |
| *** rcernin has joined #openstack-meeting | 03:50 | |
| *** lbragstad_503 has joined #openstack-meeting | 03:54 | |
| *** imsurit_ofc has joined #openstack-meeting | 03:54 | |
| *** rcernin has quit IRC | 03:57 | |
| *** rcernin has joined #openstack-meeting | 03:58 | |
| *** sagara has joined #openstack-meeting | 03:59 | |
| tpatil | #startmeeting Masakari | 04:00 |
|---|---|---|
| openstack | Meeting started Tue Jan 22 04:00:44 2019 UTC and is due to finish in 60 minutes. The chair is tpatil. Information about MeetBot at http://wiki.debian.org/MeetBot. | 04:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 04:00 |
| *** openstack changes topic to " (Meeting topic: Masakari)" | 04:00 | |
| tpatil | Hi All | 04:00 |
| openstack | The meeting name has been set to 'masakari' | 04:00 |
| sagara | Hi | 04:00 |
| tpatil | sagara: Hi | 04:01 |
| *** ShilpaSD has joined #openstack-meeting | 04:01 | |
| tpatil | #topic Critical Bugs | 04:01 |
| *** openstack changes topic to "Critical Bugs (Meeting topic: Masakari)" | 04:01 | |
| tpatil | #link https://review.openstack.org/#/c/627097/ | 04:02 |
| tpatil | I have voted +2 on this patch | 04:02 |
| tpatil | sagara: Can you please review this patch? | 04:02 |
| sagara | tpatil: Yes, I will review it later | 04:03 |
| tpatil | It's a critical issue so we should merge this patch as early as possible | 04:03 |
| tpatil | sagara: Thank you | 04:03 |
| tpatil | I don't see any other critical bug in LP | 04:05 |
| tpatil | Moving ahead | 04:05 |
| tpatil | #topic Stein workitems | 04:05 |
| *** openstack changes topic to "Stein workitems (Meeting topic: Masakari)" | 04:05 | |
| *** Bhujay has joined #openstack-meeting | 04:05 | |
| tpatil | Add event notification feature to masakari | 04:06 |
| *** Bhujay has quit IRC | 04:06 | |
| tpatil | We should merge the specs soon | 04:06 |
| tpatil | I can see the implementation patches are ready for review as well | 04:07 |
| *** Bhujay has joined #openstack-meeting | 04:07 | |
| ShilpaSD | tpatil: requesting you to review specs https://review.openstack.org/#/c/473057/6/ | 04:07 |
| tpatil | #link https://review.openstack.org/#/q/status:open+project:openstack/masakari+branch:master+topic:bp/notifications-in-masakari | 04:07 |
| ShilpaSD | tpatil: and patches related to it https://review.openstack.org/#/q/status:open+project:openstack/masakari+branch:master+topic:bp/notifications-in-masakari | 04:07 |
| *** tashiromt has joined #openstack-meeting | 04:07 | |
| tpatil | ShilpaSD: We will review these patches | 04:08 |
| *** Bhujay has quit IRC | 04:08 | |
| ShilpaSD | tpatil: thank you | 04:08 |
| *** Bhujay has joined #openstack-meeting | 04:08 | |
| *** bobh has joined #openstack-meeting | 04:09 | |
| ShilpaSD | tpatil: also propsed BP and patch for | 04:09 |
| ShilpaSD | Adding progress details for recovery workflow, kindly review the same | 04:09 |
| ShilpaSD | https://review.openstack.org/#/c/632079/1/ | 04:09 |
| *** tetsuro_ has joined #openstack-meeting | 04:09 | |
| tpatil | I have already reviewed these patches once, just waiting for other members to review as well | 04:09 |
| *** Bhujay has quit IRC | 04:09 | |
| ShilpaSD | tpatil: okay, thank you | 04:09 |
| tpatil | ShilpaSD: Will review this specs as well | 04:09 |
| *** Bhujay has joined #openstack-meeting | 04:10 | |
| ShilpaSD | tpatil: thank you | 04:10 |
| tpatil | I think we should target these two specs in Stein cycle | 04:10 |
| *** Bhujay has quit IRC | 04:11 | |
| *** tetsuro has quit IRC | 04:11 | |
| *** Bhujay has joined #openstack-meeting | 04:11 | |
| tpatil | Functional Tests | 04:12 |
| *** Bhujay has quit IRC | 04:12 | |
| *** baojg has quit IRC | 04:12 | |
| *** Bhujay has joined #openstack-meeting | 04:13 | |
| ShilpaSD | tpatil: we are in the process of making list of functional tests so that all code will get cover, and then will start implementing the same using masakari client | 04:13 |
| *** markvoelker has quit IRC | 04:13 | |
| *** Bhujay has quit IRC | 04:14 | |
| *** bobh has quit IRC | 04:14 | |
| tpatil | ShilpaSD: Ok,let's first add a functional CI job for running functional tests | 04:14 |
| *** Bhujay has joined #openstack-meeting | 04:14 | |
| ShilpaSD | tpatil: okay, will focus on adding CI job first | 04:15 |
| tpatil | There are already some functional tests written in openstacksdk which can be easily added in masakari quickly | 04:15 |
| tpatil | After adding those tests, then you can start identifying what other tests can be added | 04:15 |
| ShilpaSD | tpatil: okay, thats sounds good start | 04:16 |
| ShilpaSD | tpatil: thank you for this input | 04:17 |
| *** baojg has joined #openstack-meeting | 04:18 | |
| tpatil | Add masakari-monitors in devstack plugin | 04:18 |
| tpatil | slow progress, but plan is to add this support in stein cycle | 04:19 |
| tpatil | except running masakarihost-monitors which requires you to install pacemaker | 04:19 |
| tpatil | if pacemaker is not installed, it fails to start masakarihost-monitors | 04:20 |
| ShilpaSD | tpatil: ok, here we need to run masakari monitors process as a service similar to the devstack@n-cpu.service, we are looking into the same. | 04:23 |
| tpatil | ShilpaSD: Ok | 04:23 |
| tpatil | Does anyone wants to update about any Stein work items? | 04:23 |
| *** baojg has quit IRC | 04:23 | |
| tashiromt | I do not have update | 04:24 |
| tpatil | Ok, Let's end this meeting early then | 04:25 |
| ShilpaSD | one more from my side, pl review https://review.openstack.org/#/c/626541/ | 04:25 |
| tpatil | ShilpaSD: Will do | 04:26 |
| ShilpaSD | tpatil: thank you | 04:26 |
| tpatil | Ok, then ending this meeting early, Thank you all for joining. | 04:27 |
| tpatil | #endmeeting | 04:27 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 04:27 | |
| openstack | Meeting ended Tue Jan 22 04:27:17 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 04:27 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-01-22-04.00.html | 04:27 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-01-22-04.00.txt | 04:27 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/masakari/2019/masakari.2019-01-22-04.00.log.html | 04:27 |
| *** tetsuro has joined #openstack-meeting | 04:31 | |
| *** tetsuro_ has quit IRC | 04:31 | |
| *** sagara has quit IRC | 04:36 | |
| *** snowgoggles has quit IRC | 04:51 | |
| *** imsurit_ofc has quit IRC | 04:53 | |
| *** Bhujay has quit IRC | 04:58 | |
| *** tpatil has quit IRC | 04:59 | |
| *** whoami-rajat has joined #openstack-meeting | 05:04 | |
| *** _alastor_ has joined #openstack-meeting | 05:05 | |
| *** Bhujay has joined #openstack-meeting | 05:09 | |
| *** hongbin has quit IRC | 05:10 | |
| *** markvoelker has joined #openstack-meeting | 05:10 | |
| *** _alastor_ has quit IRC | 05:10 | |
| *** slaweq has joined #openstack-meeting | 05:11 | |
| *** baojg has joined #openstack-meeting | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** baojg has quit IRC | 05:22 | |
| *** sridharg has joined #openstack-meeting | 05:23 | |
| *** baojg has joined #openstack-meeting | 05:23 | |
| *** macza has joined #openstack-meeting | 05:24 | |
| *** macza has quit IRC | 05:29 | |
| *** slaweq has joined #openstack-meeting | 05:35 | |
| *** Bhujay has quit IRC | 05:37 | |
| *** ekcs has quit IRC | 05:38 | |
| *** markvoelker has quit IRC | 05:43 | |
| *** igordc has joined #openstack-meeting | 05:45 | |
| *** radeks has joined #openstack-meeting | 05:51 | |
| *** _alastor_ has joined #openstack-meeting | 05:52 | |
| *** e0ne has joined #openstack-meeting | 05:52 | |
| *** e0ne has quit IRC | 05:53 | |
| *** _alastor_ has quit IRC | 05:57 | |
| *** lbragstad_503 has quit IRC | 05:57 | |
| *** vishakha has joined #openstack-meeting | 06:06 | |
| *** imsurit_ofc has joined #openstack-meeting | 06:16 | |
| *** ijw has quit IRC | 06:21 | |
| *** dkushwaha has joined #openstack-meeting | 06:33 | |
| *** vishalmanchanda has joined #openstack-meeting | 06:34 | |
| *** markvoelker has joined #openstack-meeting | 06:40 | |
| *** Luzi has joined #openstack-meeting | 06:49 | |
| *** imsurit_ofc has quit IRC | 06:59 | |
| *** a-pugachev has joined #openstack-meeting | 07:08 | |
| *** lpetrut has joined #openstack-meeting | 07:12 | |
| *** markvoelker has quit IRC | 07:14 | |
| *** a-pugachev has quit IRC | 07:24 | |
| *** aojea has joined #openstack-meeting | 07:27 | |
| *** igordc has quit IRC | 07:27 | |
| *** bobh has joined #openstack-meeting | 07:29 | |
| *** bhagyashris_ has joined #openstack-meeting | 07:31 | |
| *** Liang__ has quit IRC | 07:34 | |
| *** bobh has quit IRC | 07:35 | |
| *** Liang__ has joined #openstack-meeting | 07:36 | |
| *** baojg has quit IRC | 07:36 | |
| *** tssurya has joined #openstack-meeting | 07:41 | |
| *** pcaruana has joined #openstack-meeting | 07:41 | |
| *** rtjure has quit IRC | 07:42 | |
| *** rtjure has joined #openstack-meeting | 07:44 | |
| *** hyunsikyang has joined #openstack-meeting | 07:46 | |
| *** tpatil has joined #openstack-meeting | 07:52 | |
| *** jaewookoh has joined #openstack-meeting | 07:55 | |
| *** JangwonLee has joined #openstack-meeting | 07:55 | |
| *** jaewook_oh has joined #openstack-meeting | 07:55 | |
| *** phuoc__ has joined #openstack-meeting | 07:56 | |
| *** hjwon has joined #openstack-meeting | 08:02 | |
| *** baojg has joined #openstack-meeting | 08:03 | |
| dkushwaha | #startmeeting tacker | 08:03 |
| openstack | Meeting started Tue Jan 22 08:03:08 2019 UTC and is due to finish in 60 minutes. The chair is dkushwaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:03 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:03 |
| *** openstack changes topic to " (Meeting topic: tacker)" | 08:03 | |
| openstack | The meeting name has been set to 'tacker' | 08:03 |
| dkushwaha | #startmeeting tacker | 08:03 |
| openstack | dkushwaha: Error: Can't start another meeting, one is in progress. Use #endmeeting first. | 08:03 |
| dkushwaha | #topic Roll Call | 08:04 |
| *** openstack changes topic to "Roll Call (Meeting topic: tacker)" | 08:04 | |
| dkushwaha | who is here for weekly meeting? | 08:04 |
| phuoc__ | hi dkushwaha | 08:04 |
| bhagyashris_ | O/ | 08:04 |
| bhagyashris_ | Hi | 08:04 |
| phuoc__ | I think we should end the old meeting first | 08:04 |
| phuoc__ | lest me try | 08:04 |
| phuoc__ | #endmeeting | 08:04 |
| dkushwaha | oh | 08:05 |
| *** macza has joined #openstack-meeting | 08:05 | |
| phuoc__ | nothing happens :) | 08:05 |
| dkushwaha | phuoc__, seems no another meeting started as I got error that: Can't start another meeting, one is in progress | 08:06 |
| phuoc__ | I think it's fine | 08:07 |
| dkushwaha | hello phuoc__ bhagyashris_ | 08:07 |
| bhagyashris_ | dkushwaha, phuoc___: Hi | 08:07 |
| dkushwaha | ok lets start.. | 08:08 |
| *** kopecmartin|off is now known as kopecmartin | 08:08 | |
| dkushwaha | #chair phuoc__ | 08:08 |
| openstack | Current chairs: dkushwaha phuoc__ | 08:08 |
| dkushwaha | #topic Denever CFP | 08:08 |
| *** openstack changes topic to "Denever CFP (Meeting topic: tacker)" | 08:08 | |
| *** macza has quit IRC | 08:09 | |
| dkushwaha | Tomorrow is last date for submitting proposal | 08:09 |
| *** apetrich has joined #openstack-meeting | 08:10 | |
| dkushwaha | I have prepared some initial topics for session | 08:10 |
| dkushwaha | #link https://etherpad.openstack.org/p/Tacker-CFP--Denever-summit | 08:10 |
| *** markvoelker has joined #openstack-meeting | 08:10 | |
| dkushwaha | please feel free to add your name there | 08:11 |
| dkushwaha | if you are adding talk, you can add that on link too | 08:11 |
| dkushwaha | phuoc__, bhagyashris_ if you have some more/better idea, please suggest | 08:13 |
| phuoc__ | I think one session about auto healing and monitoring VNF is good | 08:13 |
| phuoc__ | dkushwaha and bhagyashris_, you guys can make a presentation about it I think | 08:14 |
| bhagyashris_ | dkushwaha, phuoc__: sure | 08:15 |
| hyunsikyang | Hi:) I am late! | 08:16 |
| dkushwaha | phuoc__, I have 2 points on that: 1: we can add your both points in Topic #1 with existing points(as I missed it to add them). | 08:17 |
| dkushwaha | 2: prepare a separate session for only those topics in details | 08:17 |
| bhagyashris_ | phuoc__, I will prepare presentation for vdu autohealing . and are you asking about mistral monitoring ? | 08:18 |
| dkushwaha | hello hyunsikyang | 08:19 |
| phuoc__ | because VNF is important object, with monitoring and auto healing VNFs | 08:20 |
| phuoc__ | we can support managing VNFs better, | 08:20 |
| phuoc__ | IMO, it can be a separate session | 08:21 |
| dkushwaha | phuoc__, I am agree to keep a separate session on that. bhagyashris_ would you like to join that tession? | 08:22 |
| dkushwaha | or you plan to submit alone? | 08:23 |
| bhagyashris_ | dkushwaha: Not decided but someone fron NTT will join | 08:24 |
| dkushwaha | ok, So I just added the topic there | 08:26 |
| hyunsikyang | When is the deadline for this? | 08:27 |
| dkushwaha | Please feel free to add names and kindly help on abstract drafting | 08:27 |
| dkushwaha | hyunsikyang, deadline is by tomorrow | 08:27 |
| dkushwaha | we needs to be hurry | 08:27 |
| hyunsikyang | Ah ok Actually , i am working with phuoc. so we also consider for this. | 08:29 |
| *** ralonsoh has joined #openstack-meeting | 08:30 | |
| dkushwaha | thanks hyunsikyang . so please add your name | 08:30 |
| dkushwaha | if interested | 08:31 |
| hyunsikyang | ok:) | 08:31 |
| dkushwaha | moving next.. | 08:31 |
| dkushwaha | #topic BPs | 08:31 |
| *** openstack changes topic to "BPs (Meeting topic: tacker)" | 08:31 | |
| dkushwaha | Mistral workflow monitoring | 08:32 |
| dkushwaha | #link https://review.openstack.org/#/c/486924/ | 08:32 |
| *** hokeen has joined #openstack-meeting | 08:33 | |
| dkushwaha | functional test are failing on multinoe, so I am working on fixing that | 08:33 |
| phuoc__ | I hope that will be working soon :) | 08:34 |
| dkushwaha | phuoc__, hoping same :) | 08:34 |
| dkushwaha | nothing else update on this patch | 08:34 |
| dkushwaha | #topic vdu_autoheal | 08:35 |
| *** openstack changes topic to "vdu_autoheal (Meeting topic: tacker)" | 08:35 | |
| dkushwaha | #link https://review.openstack.org/#/c/612595/ | 08:36 |
| dkushwaha | bhagyashris_, thanks for your great work on this spec. | 08:36 |
| dkushwaha | Hope test case issue will be fixed soon | 08:37 |
| bhagyashris_ | dkushwaha: thank you :) | 08:37 |
| phuoc__ | Should we merge this patch first, https://review.openstack.org/#/c/631439 | 08:37 |
| bhagyashris_ | dkushwaha: yes i am working on fixing the test case issue | 08:37 |
| phuoc__ | And remove it in https://review.openstack.org/#/c/612595 | 08:37 |
| dkushwaha | phuoc__, It seems https://review.openstack.org/#/c/631439 is not fixing the issue | 08:39 |
| bhagyashris_ | dkushwaha, phuoc__: as on master the _hosting_vnfs is defined as the class level attribute but it is stired as instance level attribute in every place so i will prospse the change that the class level attribute will be stored as class level attribute only | 08:39 |
| bhagyashris_ | S/stried/stored | 08:40 |
| phuoc__ | I am looking for the patch that can fix the issue | 08:41 |
| *** markvoelker has quit IRC | 08:43 | |
| *** tashiromt has quit IRC | 08:43 | |
| dkushwaha | moving next.. | 08:44 |
| bhagyashris_ | I have just pushed the patch https://review.openstack.org/631439 | 08:45 |
| dkushwaha | bhagyashris_, great, will review it | 08:46 |
| bhagyashris_ | I have one doubt from the begging tacker used _hosting_vnfs as class level attribute but every where that is stored as instance level attribute so _hostingPvnfs should be class level or instance level attribute | 08:47 |
| dkushwaha | bhagyashris_, I didn't observed any issue related to that in existing code earlier, will check it again | 08:51 |
| bhagyashris_ | dkushwaha: ok | 08:52 |
| dkushwaha | phuoc__, any update from you? | 08:52 |
| phuoc__ | about force delete VNFs, because my server has problem so I will push patches this week | 08:53 |
| dkushwaha | phuoc__, cool | 08:54 |
| dkushwaha | #topic Open Discussion | 08:55 |
| *** openstack changes topic to "Open Discussion (Meeting topic: tacker)" | 08:55 | |
| phuoc__ | I see there is a guy he want to have demo about rolling upgrade | 08:55 |
| phuoc__ | OpenStack rolling upgrade with zero downtime to application | 08:55 |
| *** _JangwonLee_ has joined #openstack-meeting | 08:55 | |
| phuoc__ | he want to integrate Fenix and VNFM | 08:55 |
| phuoc__ | if you guy interested in this demo, we can do it :) | 08:56 |
| phuoc__ | it is for ONS North America Demo | 08:56 |
| *** electrofelix has joined #openstack-meeting | 08:56 | |
| *** rcernin has quit IRC | 08:56 | |
| dkushwaha | phuoc__, any link ? | 08:56 |
| *** priteau has quit IRC | 08:56 | |
| phuoc__ | it's OPNFV discuss | 08:57 |
| *** priteau has joined #openstack-meeting | 08:57 | |
| dkushwaha | phuoc__, seems interesting. | 08:57 |
| phuoc__ | I will find the mail, or forward it to you guys | 08:57 |
| dkushwaha | phuoc__, yes, please share | 08:58 |
| hyunsikyang | please share it thanks. | 08:58 |
| *** JangwonLee has quit IRC | 08:58 | |
| dkushwaha | hyunsikyang, do you have something to discuss? | 08:58 |
| hyunsikyang | yes | 08:59 |
| hyunsikyang | Actually i didn't submit blueprint yet. | 08:59 |
| phuoc__ | https://lists.opnfv.org/g/opnfv-tech-discuss/message/22674?p=,,,20,0,0,0::Created,,ONS+North+America+Demo+idea,20,2,0,29364359 | 08:59 |
| hyunsikyang | but, I want to extend networking features for C-VNF | 08:59 |
| hyunsikyang | So I will upload blueprint ASAP | 09:00 |
| dkushwaha | hyunsikyang, IMO there is already an existing BP for that | 09:00 |
| hyunsikyang | i didn't see it | 09:00 |
| hyunsikyang | i will check it again. could give me the info for that BP? | 09:00 |
| dkushwaha | hyunsikyang, ok let me check again | 09:01 |
| *** a-pugachev has joined #openstack-meeting | 09:01 | |
| dkushwaha | we are running out of time so closing this meeting here and we can continue on our IRC channel. | 09:02 |
| hyunsikyang | ok thanks | 09:02 |
| dkushwaha | Thanks Fols | 09:02 |
| dkushwaha | Thanks Folks.. | 09:02 |
| dkushwaha | #endmeeting | 09:02 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 09:02 | |
| phuoc__ | thank, see you guys | 09:02 |
| openstack | Meeting ended Tue Jan 22 09:02:39 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-01-22-08.03.html | 09:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-01-22-08.03.txt | 09:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-01-22-08.03.log.html | 09:02 |
| *** hokeen has quit IRC | 09:03 | |
| *** _JangwonLee_ has quit IRC | 09:04 | |
| *** JangwonLee has joined #openstack-meeting | 09:04 | |
| *** yahrens has joined #openstack-meeting | 09:04 | |
| *** tpatil has quit IRC | 09:04 | |
| *** armax has quit IRC | 09:23 | |
| *** tssurya has quit IRC | 09:25 | |
| *** tetsuro has quit IRC | 09:36 | |
| *** iyamahat_ has quit IRC | 09:40 | |
| *** markvoelker has joined #openstack-meeting | 09:40 | |
| *** Liang__ has quit IRC | 09:42 | |
| *** bhagyashris_ has quit IRC | 09:44 | |
| *** imsurit_ofc has joined #openstack-meeting | 09:46 | |
| *** _alastor_ has joined #openstack-meeting | 09:53 | |
| *** baojg has quit IRC | 09:57 | |
| *** baojg has joined #openstack-meeting | 09:58 | |
| *** _alastor_ has quit IRC | 09:58 | |
| *** iyamahat has joined #openstack-meeting | 10:01 | |
| *** erlon has joined #openstack-meeting | 10:08 | |
| *** baojg has quit IRC | 10:09 | |
| *** markvoelker has quit IRC | 10:13 | |
| *** erlon_ has joined #openstack-meeting | 10:23 | |
| *** e0ne has joined #openstack-meeting | 10:24 | |
| *** erlon has quit IRC | 10:26 | |
| *** imsurit_ofc has quit IRC | 10:32 | |
| *** hjwon has quit IRC | 10:34 | |
| *** tetsuro has joined #openstack-meeting | 10:34 | |
| *** davidsha has joined #openstack-meeting | 10:42 | |
| *** mahatic has joined #openstack-meeting | 10:51 | |
| *** imsurit_ofc has joined #openstack-meeting | 10:55 | |
| *** markvoelker has joined #openstack-meeting | 11:10 | |
| *** apetrich has quit IRC | 11:13 | |
| *** imsurit_ofc has quit IRC | 11:14 | |
| *** phuoc__ has left #openstack-meeting | 11:32 | |
| *** sridharg has quit IRC | 11:44 | |
| *** markvoelker has quit IRC | 11:44 | |
| *** bdperkin has joined #openstack-meeting | 11:46 | |
| *** sridharg has joined #openstack-meeting | 11:47 | |
| *** radeks_ has joined #openstack-meeting | 11:52 | |
| *** radeks has quit IRC | 11:55 | |
| *** apetrich has joined #openstack-meeting | 11:57 | |
| *** bbowen has quit IRC | 11:58 | |
| *** bobh has joined #openstack-meeting | 12:03 | |
| *** bobh has quit IRC | 12:08 | |
| *** macza has joined #openstack-meeting | 12:21 | |
| *** jaewook_oh has quit IRC | 12:21 | |
| *** ttsiouts has joined #openstack-meeting | 12:22 | |
| *** gary_perkins has quit IRC | 12:24 | |
| *** e0ne has quit IRC | 12:25 | |
| *** macza has quit IRC | 12:26 | |
| *** gary_perkins has joined #openstack-meeting | 12:26 | |
| *** e0ne has joined #openstack-meeting | 12:30 | |
| *** gary_perkins has quit IRC | 12:38 | |
| *** tssurya has joined #openstack-meeting | 12:38 | |
| *** markvoelker has joined #openstack-meeting | 12:41 | |
| *** gary_perkins has joined #openstack-meeting | 12:41 | |
| *** gary_perkins has quit IRC | 12:44 | |
| *** gary_perkins has joined #openstack-meeting | 12:46 | |
| *** tetsuro has quit IRC | 12:53 | |
| *** baojg has joined #openstack-meeting | 13:00 | |
| *** radeks_ has quit IRC | 13:06 | |
| *** radeks_ has joined #openstack-meeting | 13:06 | |
| *** markvoelker has quit IRC | 13:09 | |
| *** bbowen has joined #openstack-meeting | 13:11 | |
| *** mmethot has joined #openstack-meeting | 13:15 | |
| *** psachin has quit IRC | 13:16 | |
| *** mriedem has joined #openstack-meeting | 13:18 | |
| *** lpetrut has quit IRC | 13:20 | |
| *** lpetrut has joined #openstack-meeting | 13:28 | |
| *** mmethot has quit IRC | 13:39 | |
| *** mmethot has joined #openstack-meeting | 13:40 | |
| *** rtjure has quit IRC | 13:42 | |
| *** mjturek has joined #openstack-meeting | 13:42 | |
| *** rtjure has joined #openstack-meeting | 13:45 | |
| *** priteau has quit IRC | 13:46 | |
| *** arne_wiebalck_ has joined #openstack-meeting | 13:56 | |
| *** lbragstad_503 has joined #openstack-meeting | 13:56 | |
| *** armstrong has joined #openstack-meeting | 14:00 | |
| *** lbragstad_503 is now known as lbragstad | 14:03 | |
| *** e0ne has quit IRC | 14:05 | |
| *** e0ne has joined #openstack-meeting | 14:08 | |
| *** njohnston has joined #openstack-meeting | 14:15 | |
| *** mmethot has quit IRC | 14:15 | |
| *** mmethot has joined #openstack-meeting | 14:17 | |
| *** bbowen_ has joined #openstack-meeting | 14:18 | |
| *** TxGirlGeek has joined #openstack-meeting | 14:19 | |
| *** mriedem has quit IRC | 14:19 | |
| *** mmethot has quit IRC | 14:19 | |
| *** mmethot has joined #openstack-meeting | 14:20 | |
| *** bbowen has quit IRC | 14:21 | |
| *** mriedem has joined #openstack-meeting | 14:21 | |
| *** priteau has joined #openstack-meeting | 14:21 | |
| *** lbragstad has quit IRC | 14:23 | |
| *** mmethot has quit IRC | 14:23 | |
| *** mmethot has joined #openstack-meeting | 14:24 | |
| *** mmethot has quit IRC | 14:25 | |
| *** mmethot has joined #openstack-meeting | 14:25 | |
| *** lbragstad has joined #openstack-meeting | 14:26 | |
| *** mmethot_ has joined #openstack-meeting | 14:28 | |
| *** mmethot has quit IRC | 14:31 | |
| *** mmethot_ has quit IRC | 14:31 | |
| *** mmethot_ has joined #openstack-meeting | 14:32 | |
| *** ttsiouts has quit IRC | 14:39 | |
| *** TxGirlGeek has quit IRC | 14:39 | |
| *** ttsiouts has joined #openstack-meeting | 14:39 | |
| *** ttsiouts has quit IRC | 14:41 | |
| *** ttsiouts has joined #openstack-meeting | 14:41 | |
| *** mmethot_ has quit IRC | 14:43 | |
| *** mmethot_ has joined #openstack-meeting | 14:45 | |
| *** efried_mlk is now known as efried | 14:45 | |
| *** mmethot_ has quit IRC | 14:46 | |
| *** mmethot_ has joined #openstack-meeting | 14:47 | |
| *** mmethot_ has quit IRC | 14:48 | |
| *** mmethot_ has joined #openstack-meeting | 14:48 | |
| *** mmethot_ has quit IRC | 14:53 | |
| *** mmethot has joined #openstack-meeting | 14:54 | |
| *** mmethot has quit IRC | 14:59 | |
| *** mmethot has joined #openstack-meeting | 15:00 | |
| *** efried1 has joined #openstack-meeting | 15:00 | |
| *** efried has quit IRC | 15:01 | |
| *** efried1 is now known as efried | 15:01 | |
| *** awaugama has joined #openstack-meeting | 15:03 | |
| *** eharney has joined #openstack-meeting | 15:05 | |
| *** wwriverrat has quit IRC | 15:06 | |
| *** hongbin has joined #openstack-meeting | 15:09 | |
| *** Luzi has quit IRC | 15:10 | |
| *** liuyulong has joined #openstack-meeting | 15:13 | |
| *** _alastor_ has joined #openstack-meeting | 15:20 | |
| *** bobh has joined #openstack-meeting | 15:23 | |
| *** _alastor_ has quit IRC | 15:24 | |
| *** bobh has quit IRC | 15:26 | |
| *** ijw has joined #openstack-meeting | 15:28 | |
| *** eharney has quit IRC | 15:32 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:42 | |
| *** sridharg has quit IRC | 15:43 | |
| *** lpetrut has quit IRC | 15:46 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 15:46 | |
| *** eharney has joined #openstack-meeting | 15:46 | |
| *** jamesmcarthur has quit IRC | 15:50 | |
| *** _alastor_ has joined #openstack-meeting | 15:50 | |
| *** TxGirlGeek has joined #openstack-meeting | 15:52 | |
| *** dklyle has joined #openstack-meeting | 15:53 | |
| *** TxGirlGe_ has joined #openstack-meeting | 15:55 | |
| *** TxGirlGeek has quit IRC | 15:57 | |
| *** mmethot has quit IRC | 15:57 | |
| *** jamesmcarthur_ has quit IRC | 15:57 | |
| *** armax has joined #openstack-meeting | 15:57 | |
| *** yahrens has quit IRC | 15:58 | |
| *** mlavalle has joined #openstack-meeting | 15:59 | |
| *** arne_wiebalck_ has quit IRC | 15:59 | |
| *** efried has quit IRC | 16:00 | |
| slaweq | #startmeeting neutron_ci | 16:00 |
| openstack | Meeting started Tue Jan 22 16:00:32 2019 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 16:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 16:00 |
| mlavalle | o/ | 16:00 |
| slaweq | o/ | 16:00 |
| *** TxGirlGe_ has quit IRC | 16:01 | |
| slaweq | haleyb: njohnston hongbin: are You around for CI meeting? | 16:01 |
| haleyb | hi, i might have to leave early though | 16:01 |
| hongbin | o/ | 16:01 |
| slaweq | ok, lets start then | 16:02 |
| slaweq | #topic Actions from previous meetings | 16:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 16:02 | |
| slaweq | there wasn't too many actions from last week | 16:02 |
| slaweq | slaweq to make e-r query for bug 1811515 | 16:02 |
| openstack | bug 1811515 in neutron "pyroute2.NetNS don't work properly with concurrency in oslo.privsep" [Critical,Fix released] https://launchpad.net/bugs/1811515 - Assigned to Slawek Kaplonski (slaweq) | 16:02 |
| *** dmacpher has quit IRC | 16:03 | |
| slaweq | I didn't write this query but bug is now fixed (workarounded) so it's not necessary anymore | 16:03 |
| mlavalle | ++ | 16:03 |
| slaweq | next one: | 16:03 |
| slaweq | slaweq to check if oslo.privsep < 1.31.0 will help to workaround issue with SSH to FIP | 16:03 |
| mlavalle | it did | 16:03 |
| slaweq | as above, this is now fixed | 16:03 |
| slaweq | it did help but we workarounded it in different way | 16:03 |
| slaweq | :) | 16:03 |
| njohnston | o/ | 16:03 |
| slaweq | and the last one was: | 16:04 |
| bcafarel | o/ too | 16:04 |
| slaweq | slaweq to post more examples of failiures in bug 1811515 | 16:04 |
| openstack | bug 1811515 in neutron "pyroute2.NetNS don't work properly with concurrency in oslo.privsep" [Critical,Fix released] https://launchpad.net/bugs/1811515 - Assigned to Slawek Kaplonski (slaweq) | 16:04 |
| slaweq | I think I added link to logstash query from where more example could be found | 16:04 |
| slaweq | but it's already fixed | 16:04 |
| mlavalle | yeap | 16:04 |
| slaweq | so that was all from last week :) | 16:04 |
| mlavalle | you did | 16:04 |
| mlavalle | not quite | 16:04 |
| mlavalle | I had an action item | 16:04 |
| slaweq | ok, so I somehow missed it | 16:05 |
| slaweq | sorry mlavalle | 16:05 |
| slaweq | go on | 16:05 |
| mlavalle | continue working on https://bugs.launchpad.net/neutron/+bug/1795870 | 16:05 |
| openstack | Launchpad bug 1795870 in neutron "Trunk scenario test test_trunk_subport_lifecycle fails from time to time" [High,In progress] - Assigned to Miguel Lavalle (minsel) | 16:05 |
| mlavalle | which I did | 16:05 |
| mlavalle | submitted DNM patch https://review.openstack.org/#/c/630778/ | 16:05 |
| mlavalle | which I rechecked a few times and I got lucky | 16:06 |
| mlavalle | I got one succesful run and one failure right afterwards | 16:06 |
| mlavalle | that is allowing me to compare sucess / failure: | 16:06 |
| mlavalle | 1) When the test passes, the router is being hosted both in the copntroller and the compute | 16:07 |
| mlavalle | so we see the router in the logs of both L3 agents | 16:07 |
| mlavalle | 2) When the test fails, the router is never scheduled in the controller and it doesn't show up in its L3 agent | 16:08 |
| *** jamesmcarthur has joined #openstack-meeting | 16:08 | |
| slaweq | hmm, it should be scheduled to controller always as only there is dhcp port, right? | 16:08 |
| mlavalle | This is an example of L3 agent log in a failed execution in the controller: http://logs.openstack.org/78/630778/1/check/neutron-tempest-plugin-dvr-multinode-scenario/02391e0/controller/logs/screen-q-svc.txt.gz?level=TRACE | 16:08 |
| *** TxGirlGeek has joined #openstack-meeting | 16:09 | |
| *** efried has joined #openstack-meeting | 16:09 | |
| mlavalle | please note that the L3 agent is down according to the neutron server | 16:09 |
| mlavalle | I am seeing the same pattern in at least two cases | 16:09 |
| slaweq | why agent is down? | 16:10 |
| slaweq | do You know? | 16:10 |
| mlavalle | the agent is not actually down | 16:10 |
| mlavalle | it is running | 16:10 |
| mlavalle | but the server thinks is down | 16:11 |
| *** jamesmcarthur has quit IRC | 16:11 | |
| mlavalle | and my next step is to investigate why | 16:11 |
| mlavalle | so please give me an action item for next week | 16:11 |
| slaweq | sure | 16:11 |
| slaweq | #action mlavalle to continue investigate why L3 agent is considered as down and cause trunk tests fail | 16:12 |
| slaweq | thx mlavalle for working on this | 16:12 |
| slaweq | that is interesting why this agent is treated as down | 16:12 |
| mlavalle | yeah, in the same node | 16:12 |
| *** jamesmcarthur has joined #openstack-meeting | 16:13 | |
| slaweq | I wonder if agent is not sending heartbeat or neutron-server is not processing it properly | 16:13 |
| slaweq | IMHO it's more likely that agent is not sending it | 16:14 |
| slaweq | as heartbeats from other agents are ok on server | 16:14 |
| mlavalle | yes | 16:14 |
| slaweq | but we will see when You will check it :) | 16:14 |
| slaweq | ok, so now I think we can move on to the next topic | 16:15 |
| slaweq | right? | 16:15 |
| mlavalle | thanks | 16:15 |
| mlavalle | yes | 16:15 |
| slaweq | #topic Python 3 | 16:15 |
| *** openstack changes topic to "Python 3 (Meeting topic: neutron_ci)" | 16:15 | |
| *** dims has quit IRC | 16:15 | |
| slaweq | Etherpad: https://etherpad.openstack.org/p/neutron_ci_python3 | 16:15 |
| slaweq | in last week we merged 2 patches and now neutron-tempest-linuxbridge and neutron-tempest-dvr jobs are running on python 3 and using zuulv3 syntax already | 16:16 |
| slaweq | I plan to do the same with neutron-tempest-dvr-ha-multinode-full this week | 16:16 |
| mlavalle | Thanks | 16:17 |
| slaweq | there is also this neutron-functional job which needs to be switched | 16:17 |
| slaweq | and that is still problematic | 16:17 |
| slaweq | This week I sent email to ML: http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001904.html | 16:17 |
| slaweq | maybe someone more familiar with ostestr/subunit and python 3 will be able to help us with it | 16:18 |
| mlavalle | did you get any responses so far? | 16:18 |
| slaweq | nope :/ | 16:18 |
| slaweq | I will ask tomorrow on openstack-qa channel - maybe e.g. gmann will know who can help us with it | 16:19 |
| bcafarel | crossing fingers | 16:20 |
| njohnston | good idea | 16:20 |
| *** dims has joined #openstack-meeting | 16:20 | |
| slaweq | unfortunatelly except that I have no any other idea how to deal with this issue currently :/ | 16:20 |
| slaweq | and that's all related to python 3 from me | 16:21 |
| slaweq | njohnston: do You have anything else to add? | 16:21 |
| slaweq | how it's going with grenade jobs switch? | 16:21 |
| bcafarel | yeah pushing to add other limits to log output does not sound good as a viable fix :/ hopefully someone will fix the root cause and we get all our jobs py3-green | 16:21 |
| njohnston | No, nothing for this week. I have been focused on bulk ports, but I should be ablke to move forward on the grenade work this week | 16:21 |
| slaweq | njohnston: great, thx | 16:22 |
| *** factor has joined #openstack-meeting | 16:22 | |
| njohnston | #action njohnston Work on grenade job transition | 16:22 |
| slaweq | thx :) | 16:22 |
| slaweq | ok, so lets go to the next topic | 16:22 |
| slaweq | #topic Grafana | 16:22 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 16:22 | |
| slaweq | http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 16:23 |
| slaweq | there is peak on many jobs today, I don't know exactly why it is like that but: | 16:24 |
| slaweq | 1. I didn't found in today's jobs anything very bad, | 16:25 |
| slaweq | 2. as it is even in pep8 job, I think that it could be some "generic" issue, not related to neutron bug directly | 16:25 |
| *** ianychoi has joined #openstack-meeting | 16:26 | |
| slaweq | from other things I think that we again have one major issue | 16:26 |
| slaweq | and this week it is tempest-slow job :/ | 16:26 |
| mlavalle | yeah | 16:26 |
| mlavalle | it is evident in Grafana | 16:26 |
| slaweq | so, if You don't have anything else related to grafana, lets move to talk about this tempest jobs now | 16:27 |
| slaweq | fine? | 16:27 |
| slaweq | ok, I get it as yes :) | 16:28 |
| slaweq | #topic Tempest/Scenario | 16:28 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 16:28 | |
| slaweq | We have one major issue with tempest-slow job, it is described in bug report https://bugs.launchpad.net/neutron/+bug/1812552 | 16:29 |
| openstack | Launchpad bug 1812552 in neutron "tempest-slow tests fails often" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq) | 16:29 |
| slaweq | Today I was talking with seon-k-mooney about that | 16:29 |
| mlavalle | did he help? | 16:29 |
| slaweq | nope | 16:29 |
| bcafarel | slaweq: for today's jobs there was some general post failure ~20h ago that probably did not help in the graph (sorry lagging as usual) | 16:29 |
| slaweq | bcafarel: yes, that might be the reason :) | 16:30 |
| mlavalle | thanks anyway bcafarel | 16:30 |
| mlavalle | much appreciated | 16:30 |
| slaweq | ok, so tempest-slow job issue in details: | 16:30 |
| slaweq | 1. from grafana and logstash it looks that this issue was caused somehow by https://review.openstack.org/#/c/631584/ | 16:30 |
| slaweq | 2. I proposed revert of this patch https://review.openstack.org/#/c/631944/ and indeed this job passed 5 or 6 times already | 16:31 |
| slaweq | in fact it didn't failed even once with this revert | 16:31 |
| slaweq | which confirms somehow that this patch is culprit | 16:31 |
| slaweq | 3. I talked with sean today and we though that maybe this my patch introduced some additional race in ovsdb monitor and how it handles ports events on ovs bridges | 16:32 |
| mlavalle | so we are leaving interfaces un-plugged, right? | 16:32 |
| slaweq | but, what is strange for me is fact that it cause issues only in this job, and only (at least where I was checking it) with two tests which shelve/unshelve instance | 16:33 |
| slaweq | mlavalle: not exactly | 16:33 |
| slaweq | today I invesigated logs from one of such jobs carefully | 16:33 |
| slaweq | (please read my last comment in bug report) | 16:33 |
| slaweq | and it looks for me that port was configured properly | 16:34 |
| slaweq | and communication, at least from VM to dhcp server was fine | 16:34 |
| slaweq | which puzzled me even more | 16:34 |
| mlavalle | but the VM doesn't get the dhcp offer, right? | 16:35 |
| slaweq | I have no idea what is wrong there and how (if) this mentioned patch could break it | 16:35 |
| slaweq | mlavalle: it looks so | 16:35 |
| slaweq | but dnsmasq get DHCP Request from VM | 16:35 |
| slaweq | and sends this DHCP offer | 16:35 |
| mlavalle | so the the DHCP request gets to dnsmasq | 16:35 |
| slaweq | but this is somehow missed somewhere | 16:35 |
| *** ociuhandu has joined #openstack-meeting | 16:35 | |
| mlavalle | could it be a problem with the flows? | 16:36 |
| slaweq | possibly yes | 16:36 |
| slaweq | especially that we are using openvswitch fw driver there | 16:36 |
| mlavalle | we are dropping the offer | 16:36 |
| mlavalle | perhapds, that is | 16:37 |
| slaweq | it can be | 16:37 |
| slaweq | but basically I think that we should revert this patch to make tempest-slow into better shape | 16:37 |
| slaweq | *to get | 16:37 |
| mlavalle | yes, let's pull the trigger | 16:37 |
| njohnston | +1 | 16:38 |
| mlavalle | worst case, we revert the revert ;-) | 16:38 |
| slaweq | LOL | 16:38 |
| *** e0ne has quit IRC | 16:38 | |
| bcafarel | it's been some times since I saw a "revert revert revert revert revert ..." review | 16:38 |
| slaweq | I know that sean is going to release new os-vif version soon and it will have his revert of patches which caused that port was created twice during booting vm | 16:38 |
| slaweq | so maybe this my patch will not be really needed | 16:39 |
| *** e0ne has joined #openstack-meeting | 16:39 | |
| slaweq | but as a sidenote I want to mention that I think that I saw some similar issues from time to time already | 16:39 |
| *** mriedem is now known as mriedem_away | 16:39 | |
| slaweq | I mean issues that VM didn't have configured IP address and because of that was not reachable | 16:40 |
| slaweq | maybe it is same issue but just happens less often without this my patch | 16:40 |
| slaweq | please keep it in mind just :) | 16:40 |
| mlavalle | ok, thanks for the clarification | 16:40 |
| slaweq | ok, thats all from my side about tempest tests | 16:41 |
| slaweq | anything else You want to add? | 16:41 |
| mlavalle | not me | 16:42 |
| slaweq | ok, lets move on then | 16:42 |
| slaweq | next topic | 16:42 |
| slaweq | #topic fullstack/functional | 16:42 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 16:42 | |
| slaweq | today I found new bug in functional tests \o/ | 16:42 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1812872 | 16:42 |
| openstack | Launchpad bug 1812872 in neutron "Trunk functional tests can interact with each other " [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 16:42 |
| mlavalle | yaay | 16:42 |
| slaweq | fortunately so far we are lucky and it don't hit us in gate | 16:43 |
| slaweq | but I can reproduce it locally | 16:43 |
| slaweq | and I hit it in my patch where I want to switch functional job to python3 | 16:43 |
| slaweq | basically it is race between two trunk related tests | 16:44 |
| mlavalle | ahh | 16:44 |
| slaweq | in bug description there are details | 16:44 |
| slaweq | I have one idea how to "fix" it in ugly but fast way | 16:44 |
| slaweq | we can add lockutils.synchronized() decorator for those 2 tests and it should works fine IMHO | 16:44 |
| slaweq | what do You think about such solution? | 16:45 |
| *** jamesmcarthur has quit IRC | 16:45 | |
| mlavalle | so I assume they share a resource | 16:45 |
| mlavalle | right? | 16:45 |
| mlavalle | that is why you need the lock | 16:45 |
| slaweq | other than that is probably doing something similar to fullstack tests where we will need to monkey patch some functions | 16:45 |
| slaweq | they don't share resources in fact | 16:45 |
| slaweq | but each of them is listing ovsdb monitor events | 16:46 |
| slaweq | and such events aren't distingueshed between tests | 16:46 |
| mlavalle | well, that's the sahred resource, the stream of events | 16:46 |
| slaweq | so one test have got mocked handle_trunk_remove method to not clean trunk bridge | 16:47 |
| slaweq | but then second test is cleaning this bridge as it gets event from ovsdb monitor | 16:47 |
| slaweq | mlavalle: yes, so in that way they share resources | 16:47 |
| slaweq | so is such lock (with comment) acceptable for You? | 16:48 |
| slaweq | what You think about it? | 16:48 |
| njohnston | are trunk bridges vlan-specific? Would there be a way to plumb another fake vlan, thus making the trunk bridges in each test different from each other? | 16:48 |
| mlavalle | I don't see ehy not | 16:48 |
| slaweq | njohnston: trunk bridges are different | 16:48 |
| slaweq | they are created in setUp() method for each test | 16:49 |
| slaweq | but one of them can remove trunk bridge for another one because it is triggered by ovsdb event | 16:49 |
| mlavalle | so esentially another alternative is to come up with a way for each test to have iot's own stream events | 16:49 |
| slaweq | that was discussed some time ago I think in context of fullstack tests | 16:50 |
| njohnston | so the ovsdb event does not include info on which bridge it is related to? | 16:50 |
| mlavalle | would that be a lot of work? | 16:50 |
| slaweq | and we don't have (then we didn't have at least) easy way to tell - listen only events from this bridge | 16:50 |
| mlavalle | we have a lot of work to do | 16:51 |
| slaweq | but tbh now, when we switched ovsdb monitors to native implementation thx to ralonsoh work, maybe that would be possible | 16:51 |
| slaweq | I can check it | 16:51 |
| njohnston | cool | 16:51 |
| slaweq | if this will be too much to do, I will go for now with lock and TODO note how it might be fixed in fututr | 16:51 |
| slaweq | *future | 16:51 |
| slaweq | ok for You? | 16:51 |
| njohnston | +1 | 16:51 |
| mlavalle | if fixing this without locks is too labor intensive, let's go for the locks and leave a todo comment | 16:51 |
| slaweq | mlavalle: ++ :) | 16:52 |
| slaweq | #action slaweq to check if new ovsdb monitor implementation can allow to listen only for events from specific bridge | 16:52 |
| slaweq | ok, anything else You want to mention related to functiona/fullstack tests? | 16:53 |
| mlavalle | not me | 16:53 |
| slaweq | ok, lets move to next topic then | 16:53 |
| slaweq | #topic Periodic | 16:53 |
| *** openstack changes topic to "Periodic (Meeting topic: neutron_ci)" | 16:53 | |
| slaweq | since few days at least we have issue with jobs openstack-tox-py27-with-oslo-master and openstack-tox-py35-with-oslo-master | 16:53 |
| slaweq | Example: http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master/openstack-tox-py27-with-oslo-master/c787964/testr_results.html.gz | 16:54 |
| slaweq | it's only link to one python27 job | 16:54 |
| slaweq | but error is exactly the same in python 35 | 16:54 |
| slaweq | I didn't report this bug in launchpad yet | 16:54 |
| slaweq | is there any volunteer to report it on launchpad and fix it? :) | 16:54 |
| njohnston | never seen one like thta before | 16:54 |
| *** ttsiouts has quit IRC | 16:55 | |
| njohnston | I'll take a look | 16:55 |
| *** ttsiouts has joined #openstack-meeting | 16:55 | |
| slaweq | njohnston: it's probably some change in newest oslo_service lib and we need to adjust our code to it | 16:55 |
| slaweq | thx njohnston | 16:55 |
| njohnston | that's what I am thinking too | 16:56 |
| slaweq | #action njohnston to take care of periodic UT jobs failures | 16:56 |
| njohnston | thx | 16:56 |
| slaweq | Thank You :) | 16:56 |
| slaweq | ok, and that's all from me for today :) | 16:56 |
| slaweq | do You want to talk about anythin else quickly? | 16:56 |
| mlavalle | not me | 16:57 |
| slaweq | ok, so thanks for attending guys | 16:57 |
| slaweq | have a great week | 16:57 |
| slaweq | o/ | 16:57 |
| slaweq | #endmeeting | 16:57 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:57 | |
| openstack | Meeting ended Tue Jan 22 16:57:49 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:57 |
| mlavalle | Thanks, you too | 16:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-01-22-16.00.html | 16:57 |
| bcafarel | \o | 16:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-01-22-16.00.txt | 16:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-01-22-16.00.log.html | 16:57 |
| njohnston | \o | 16:58 |
| mlavalle | o/ | 16:58 |
| *** ttsiouts has quit IRC | 17:00 | |
| *** efried has quit IRC | 17:00 | |
| *** pcaruana has quit IRC | 17:02 | |
| *** macza has joined #openstack-meeting | 17:02 | |
| *** e0ne has quit IRC | 17:02 | |
| *** jamesmcarthur has joined #openstack-meeting | 17:05 | |
| *** macza_ has joined #openstack-meeting | 17:06 | |
| *** macza has quit IRC | 17:07 | |
| *** radeks__ has joined #openstack-meeting | 17:08 | |
| *** mlavalle has quit IRC | 17:08 | |
| *** radeks_ has quit IRC | 17:10 | |
| *** aojea has quit IRC | 17:15 | |
| *** jamesmcarthur has quit IRC | 17:20 | |
| *** a-pugachev has quit IRC | 17:21 | |
| *** igordc has joined #openstack-meeting | 17:28 | |
| *** davidsha has quit IRC | 17:31 | |
| *** radeks__ has quit IRC | 17:33 | |
| *** mattw4 has joined #openstack-meeting | 17:34 | |
| *** igordc has quit IRC | 17:39 | |
| *** radeks has joined #openstack-meeting | 17:47 | |
| *** efried has joined #openstack-meeting | 17:49 | |
| *** erlon_ has quit IRC | 17:49 | |
| *** radeks has quit IRC | 17:50 | |
| *** armax has quit IRC | 17:53 | |
| *** dmacpher has joined #openstack-meeting | 17:53 | |
| *** ekcs has joined #openstack-meeting | 17:57 | |
| *** pcaruana has joined #openstack-meeting | 18:01 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:02 | |
| *** ssbarnea|rover has joined #openstack-meeting | 18:02 | |
| *** jamesmcarthur has quit IRC | 18:03 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:04 | |
| *** ssbarnea|bkp2 has quit IRC | 18:04 | |
| *** priteau has quit IRC | 18:05 | |
| *** ociuhandu has quit IRC | 18:06 | |
| *** bbowen_ is now known as bbowen | 18:08 | |
| *** mmethot has joined #openstack-meeting | 18:12 | |
| *** jamesmcarthur has quit IRC | 18:14 | |
| *** snowgoggles has joined #openstack-meeting | 18:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:18 | |
| *** mmethot has quit IRC | 18:21 | |
| *** mmethot has joined #openstack-meeting | 18:21 | |
| *** yamahata has quit IRC | 18:22 | |
| *** iyamahat has quit IRC | 18:22 | |
| *** pcaruana has quit IRC | 18:23 | |
| *** mmethot has quit IRC | 18:25 | |
| *** mmethot has joined #openstack-meeting | 18:25 | |
| *** jamesmcarthur has quit IRC | 18:28 | |
| *** iyamahat has joined #openstack-meeting | 18:36 | |
| *** kopecmartin is now known as kopecmartin|off | 18:38 | |
| *** TxGirlGeek has quit IRC | 18:48 | |
| *** mmethot has quit IRC | 18:48 | |
| *** mmethot has joined #openstack-meeting | 18:49 | |
| *** armstrong has quit IRC | 18:50 | |
| *** armax has joined #openstack-meeting | 18:51 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:52 | |
| *** mmethot has quit IRC | 18:55 | |
| *** yamahata has joined #openstack-meeting | 18:55 | |
| *** mmethot has joined #openstack-meeting | 18:56 | |
| *** baojg has quit IRC | 18:57 | |
| *** baojg has joined #openstack-meeting | 18:57 | |
| *** baojg has quit IRC | 18:58 | |
| *** baojg has joined #openstack-meeting | 18:58 | |
| *** baojg has quit IRC | 18:58 | |
| *** baojg has joined #openstack-meeting | 18:59 | |
| *** jesusaur has quit IRC | 18:59 | |
| *** baojg has quit IRC | 18:59 | |
| *** baojg has joined #openstack-meeting | 18:59 | |
| *** TxGirlGeek has joined #openstack-meeting | 19:00 | |
| *** baojg has quit IRC | 19:00 | |
| *** baojg has joined #openstack-meeting | 19:00 | |
| *** baojg has quit IRC | 19:01 | |
| *** baojg has joined #openstack-meeting | 19:01 | |
| *** baojg has quit IRC | 19:02 | |
| *** ianw_pto is now known as ianw | 19:02 | |
| *** baojg has joined #openstack-meeting | 19:02 | |
| *** baojg has quit IRC | 19:02 | |
| *** baojg has joined #openstack-meeting | 19:03 | |
| *** baojg has quit IRC | 19:04 | |
| *** baojg has joined #openstack-meeting | 19:04 | |
| *** baojg has quit IRC | 19:05 | |
| *** baojg has joined #openstack-meeting | 19:05 | |
| *** mmethot has quit IRC | 19:05 | |
| *** baojg has quit IRC | 19:06 | |
| *** iyamahat has quit IRC | 19:06 | |
| *** baojg has joined #openstack-meeting | 19:06 | |
| *** iyamahat has joined #openstack-meeting | 19:06 | |
| *** baojg has quit IRC | 19:06 | |
| *** ociuhandu has joined #openstack-meeting | 19:07 | |
| *** e0ne has joined #openstack-meeting | 19:07 | |
| *** baojg has joined #openstack-meeting | 19:07 | |
| *** baojg has quit IRC | 19:08 | |
| *** baojg has joined #openstack-meeting | 19:08 | |
| *** jamesmcarthur has quit IRC | 19:09 | |
| *** baojg has quit IRC | 19:09 | |
| *** baojg has joined #openstack-meeting | 19:09 | |
| *** baojg has quit IRC | 19:09 | |
| *** baojg has joined #openstack-meeting | 19:10 | |
| *** baojg has quit IRC | 19:10 | |
| *** baojg has joined #openstack-meeting | 19:11 | |
| *** baojg has quit IRC | 19:11 | |
| *** TxGirlGeek has quit IRC | 19:11 | |
| *** baojg has joined #openstack-meeting | 19:11 | |
| *** baojg has quit IRC | 19:12 | |
| *** baojg has joined #openstack-meeting | 19:13 | |
| *** baojg has quit IRC | 19:13 | |
| *** baojg has joined #openstack-meeting | 19:14 | |
| *** TxGirlGeek has joined #openstack-meeting | 19:14 | |
| *** baojg has quit IRC | 19:14 | |
| *** baojg has joined #openstack-meeting | 19:15 | |
| *** baojg has quit IRC | 19:15 | |
| *** baojg has joined #openstack-meeting | 19:15 | |
| *** baojg has quit IRC | 19:16 | |
| *** baojg has joined #openstack-meeting | 19:16 | |
| *** baojg has quit IRC | 19:17 | |
| *** baojg has joined #openstack-meeting | 19:17 | |
| *** baojg has quit IRC | 19:17 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:18 | |
| *** baojg has joined #openstack-meeting | 19:19 | |
| *** baojg has quit IRC | 19:20 | |
| *** jamesmcarthur has quit IRC | 19:21 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:24 | |
| *** jesusaur has joined #openstack-meeting | 19:25 | |
| *** takamatsu has quit IRC | 19:27 | |
| *** tssurya has quit IRC | 19:36 | |
| *** e0ne has quit IRC | 19:42 | |
| *** iyamahat_ has joined #openstack-meeting | 19:43 | |
| *** TxGirlGeek has quit IRC | 19:46 | |
| *** iyamahat has quit IRC | 19:46 | |
| *** iyamahat__ has joined #openstack-meeting | 19:48 | |
| *** pcaruana has joined #openstack-meeting | 19:50 | |
| *** iyamahat_ has quit IRC | 19:51 | |
| *** whoami-rajat has quit IRC | 19:52 | |
| *** jamesmcarthur has quit IRC | 19:53 | |
| *** ralonsoh has quit IRC | 19:53 | |
| *** diablo_rojo has joined #openstack-meeting | 19:56 | |
| *** TxGirlGeek has joined #openstack-meeting | 19:58 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:58 | |
| *** jamesmcarthur has quit IRC | 20:02 | |
| *** diablo_rojo has quit IRC | 20:05 | |
| *** pcaruana has quit IRC | 20:11 | |
| *** mriedem_away is now known as mriedem | 20:16 | |
| *** mjturek has quit IRC | 20:24 | |
| *** pcaruana has joined #openstack-meeting | 20:24 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:36 | |
| *** martial__ has joined #openstack-meeting | 20:39 | |
| *** martial__ is now known as martial | 20:39 | |
| *** dims has quit IRC | 20:40 | |
| *** dims has joined #openstack-meeting | 20:42 | |
| *** oneswig has joined #openstack-meeting | 20:50 | |
| *** dims has quit IRC | 20:50 | |
| *** janders has joined #openstack-meeting | 20:50 | |
| *** isq has joined #openstack-meeting | 20:52 | |
| *** dims has joined #openstack-meeting | 20:52 | |
| martial | oneswig: I see blair in the unregistered channel ... hopefully fix it soon | 20:58 |
| oneswig | ok thanks martial | 20:59 |
| oneswig | #startmeeting scientific-sig | 21:00 |
| openstack | Meeting started Tue Jan 22 21:00:27 2019 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 21:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 21:00 |
| oneswig | Let us get the show on the road! | 21:00 |
| oneswig | #link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_22nd_2019 | 21:00 |
| oneswig | #chair martial | 21:01 |
| openstack | Current chairs: martial oneswig | 21:01 |
| *** trandles has joined #openstack-meeting | 21:02 | |
| oneswig | While we are gathering, I'll do the reminder - CFP ends tomorrow, midnight pacific time! | 21:02 |
| oneswig | #link CFP link for Open Infra Denver https://www.openstack.org/summit/denver-2019/call-for-presentations/ | 21:03 |
| oneswig | janders: you ready? | 21:03 |
| janders | I am | 21:03 |
| oneswig | Excellent, lets start with that. | 21:04 |
| oneswig | #topic Long-tail latency of SR-IOV Infiniband | 21:04 |
| *** openstack changes topic to "Long-tail latency of SR-IOV Infiniband (Meeting topic: scientific-sig)" | 21:04 | |
| oneswig | So what's been going on? | 21:04 |
| *** b1air has joined #openstack-meeting | 21:04 | |
| janders | Here are some slides I prepared - these cover IB microbenchmarks across bare-metal and SRIOV/IB | 21:04 |
| janders | https://docs.google.com/presentation/d/1k-nNQpuPpMo6v__9OGKn1PIkDiF2fxihrLcJSh_7VBc/edit?usp=sharing | 21:04 |
| *** artom has quit IRC | 21:04 | |
| oneswig | Excellent, thanks | 21:04 |
| b1air | o/ hello! (slight issue with NickServ here...) | 21:05 |
| janders | As per the Slide 2 we were aiming to get a better picture of low-level causes of the performance disparity we were seeing (or mostly hearing about) | 21:05 |
| oneswig | Hi b1air, you made it | 21:05 |
| oneswig | #chair b1air | 21:05 |
| openstack | Current chairs: b1air martial oneswig | 21:05 |
| b1air | tada! | 21:05 |
| janders | We were also thinking this will help decide what's best running on bare-metal and what can run happily in a (HPC)VM | 21:05 |
| martial | bad nick serve | 21:05 |
| janders | hey Blair! | 21:06 |
| martial | welcome Blair :) | 21:06 |
| janders | Slide 3 has the details of the lab setup. We tried to stay as generic as possible so the numbers aren't super-optimised, however we've done all the ususal reasonable things to do for benchmarking (performance mode, CPU passthru) | 21:07 |
| janders | Similar "generic" approach was applied to the microbenchmarks - we ran with the default parameters which are captured in Slide 4 | 21:07 |
| *** b1airo has joined #openstack-meeting | 21:07 | |
| janders | On that - the results once again prove what many of us here have seen - virtualised RDMA (or IB in this case) can match the bandwidth of bare-metal RDMA | 21:08 |
| martial | I asked Rion to join as well (since it is his scripts for Kubespray) | 21:08 |
| janders | We're at around 99% efficiency bandwidth wise | 21:09 |
| janders | With latency, however things things are different | 21:09 |
| oneswig | This is just excellent. :-) | 21:09 |
| janders | Virtualisation seems to add a fair bit there, about .4 us off from bare-metal | 21:09 |
| janders | (or ~24%) | 21:10 |
| b1airo | ok, i've managed to log in twice it seems :-) | 21:10 |
| martial | b1airo: cool :) | 21:10 |
| janders | the more the merrier :) | 21:10 |
| oneswig | Does your VM have NUMA passthrough? | 21:10 |
| *** b1air has quit IRC | 21:10 | |
| oneswig | #chair b1airo | 21:11 |
| openstack | Current chairs: b1air b1airo martial oneswig | 21:11 |
| janders | I haven't explicitly tweaked NUMA, however the CPUs are in passthru mode | 21:11 |
| b1airo | thanks oneswig | 21:11 |
| b1airo | hugepages...? | 21:11 |
| janders | I'm happy to check later in the meeting and report back - just run "numactl --show"? | 21:11 |
| b1airo | i note the hardware is relatively old now, wonder how that impacts these results as compared to say Skylake with CX-4 | 21:12 |
| *** jamesmcarthur has quit IRC | 21:12 | |
| oneswig | That ought to do it, or if you have multiple sockets in lscpu, I think that also means it's on | 21:12 |
| janders | I was considering applying NFV style tuning, but thought this would make it less generic | 21:12 |
| oneswig | janders: what kind of things? | 21:12 |
| janders | good point on hugepages though - just enabling that shouldn't have any negative side effects | 21:12 |
| janders | NFV style approach would involve CPU pinning | 21:13 |
| janders | however my worry there is we'll likely improve IB microbenchmark numbers at the expense on Linpack | 21:13 |
| b1airo | this could be a nice start to some work to come up with a standard set of microbenchmarks we could use... | 21:13 |
| *** deardooley has joined #openstack-meeting | 21:13 | |
| martial | oh here is Rion | 21:13 |
| oneswig | Hello Rion, welcome | 21:14 |
| janders | I hope to get some newer kit soon and I'm intending to retest on Skylake/CX5 and maybe newer | 21:14 |
| b1airo | surely if you were going to be running MPI code you'd want pinning though janders ? | 21:14 |
| deardooley | Hi all | 21:14 |
| * b1airo waves to Rion | 21:14 | |
| oneswig | Your page 6, plotting sigmas against one another. Are you able to plot histograms instead? I'd be interested to see the two bell curves superimposed. | 21:15 |
| b1airo | it's an interesting consideration though - how much tuning is reasonable and what are the implications... | 21:16 |
| janders | oneswig: thank you for the suggestion - I will include that when I work on this again soon (with more hardware) | 21:17 |
| oneswig | One potential microbenchmark for capturing the effect of long-tail jitter on a parallel job would be to time multi-node barriers | 21:17 |
| oneswig | I'm not sure there's an ib_* benchmark for that but there might be something in IMB. | 21:17 |
| janders | The second interesting observation is the impact of virtualisation on standard deviation in results | 21:17 |
| janders | interestingly this is seen across the board. In these slides I'm mostly focusing on ib_write_* but I put Linpack there for reference too | 21:18 |
| janders | whether it's Linpack, bandwidth or latency, baremetals are heaps more consistent | 21:18 |
| oneswig | Is that a single-node Linpack configuration? | 21:18 |
| janders | in absolute numbers the fluctuation isn't huge, but in relative numbers it's an order of magnitude | 21:19 |
| janders | yes, it's single node | 21:19 |
| *** pcaruana has quit IRC | 21:19 | |
| janders | I considered multinode but was getting mixed messages about the potential impact of interconnect virtualisation on the results from different people, so thought I better keep things simple | 21:20 |
| janders | this way we have the overheads measured separately | 21:20 |
| oneswig | I'd love to know more on the root causes (wouldn't we all) | 21:20 |
| janders | I think the variability could likely be addressed with NFV style tyning | 21:21 |
| janders | *tuning | 21:21 |
| janders | at least to some degree | 21:21 |
| *** baojg has joined #openstack-meeting | 21:21 | |
| janders | with the latency impact, I think the core of it might have to do with the way IB virtualisation is done, however I've never heard the official explanation | 21:21 |
| janders | I feel it likely gets better for larger message sizes | 21:22 |
| b1airo | seems to me that once you start doing cpu and/or numa pinning and/or static hugepage backing, then you really need to commit to isolating specific hardware for that job and you create just a few instance-types that fit together on your specific hardware nicely. then you probably also have some other nodes for general purpose. so perhaps there are at least three interesting scenarios to benchmark: 1) VM tuned for | 21:22 |
| b1airo | general purpose hosts; 2) VM tuned for high-performance dedicated hosts; 3) bare-metal | 21:22 |
| *** janders_ has joined #openstack-meeting | 21:23 | |
| janders_ | sorry got kicked out :( | 21:23 |
| janders_ | 1 2 3 testing | 21:23 |
| oneswig | Did you catch b1airo's message with 3 cases? | 21:23 |
| janders_ | unfortunatly not, I lost everything past "08:22] <janders> given the bandwidth numbers are quite good" | 21:24 |
| *** bbowen has quit IRC | 21:24 | |
| janders_ | b1airo: can you re-post please? | 21:24 |
| janders_ | sorry about that | 21:24 |
| martial | helping b1airo 16:23:00 <b1airo> seems to me that once you start doing cpu and/or numa pinning and/or static hugepage backing, then you really need to commit to isolating specific hardware for that job and you create just a few instance-types that fit together on your specific hardware nicely. then you probably also have some other nodes for general purpose. so perhaps there are at least three interesting scenarios | 21:25 |
| martial | to benchmark: 1) VM tuned for | 21:25 |
| martial | 16:23:00 <b1airo> general purpose hosts; 2) VM tuned for high-performance dedicated hosts; 3) bare-metal | 21:25 |
| janders_ | indeed | 21:25 |
| janders_ | have you guys had a chance to benchmark CPU pinned configurations in similar ways? | 21:26 |
| oneswig | This test might be interesting, across multiple nodes: IMB-MPI1 Barrier | 21:26 |
| janders_ | I wonder if pinning helps with consistency | 21:26 |
| *** janders has quit IRC | 21:26 | |
| janders_ | and what would be the impact of the pinned configuration on local Linpack? | 21:26 |
| *** baojg has quit IRC | 21:27 | |
| oneswig | janders_: yes, I did some stuff using OpenFOAM (with paravirtualised networking) | 21:27 |
| b1airo | thanks martial - i was in the bathroom briefly | 21:27 |
| janders_ | (I suppose we would likely lose some cores for the host OS - and I wonder to what degree the performance improvement on the pinned cores would compensate that) | 21:27 |
| oneswig | You can see the impact of pinning in simple test cases like iperf - I found it didn't increase TCP performance much but it certainly did help with jitter | 21:28 |
| b1airo | yes, i believe pinning does help with consistency | 21:28 |
| janders_ | I think the config proposed by Blair is a good way forward - my worry is if they users will know if they need max_cores configuration (20 core VM on a 20 core node) or the NFV configuration | 21:28 |
| b1airo | i think the question of reserved host cores is another interesting one for exploration... | 21:29 |
| janders_ | I tried Linpack in 18 core and 20 core VMs in the past and 20 core was still faster | 21:29 |
| *** efried has quit IRC | 21:30 | |
| janders_ | despite the potential of scheduling issues between the host and the guest | 21:30 |
| b1airo | i would contend that if most of your network traffic is happening via SR-IOV interface then reserving host cores is unnecessary | 21:30 |
| *** efried has joined #openstack-meeting | 21:30 | |
| oneswig | b1airo: makes sense unless they are working for the libvirt storage too | 21:30 |
| oneswig | OK, we should move on. Any more questions for janders? | 21:31 |
| oneswig | janders_: one final thought - did you disable hyperthreading? | 21:31 |
| b1airo | ah yes, good point oneswig - i guess i was thinking of storage via SR-IOV too, i.e., parallel filesystem | 21:31 |
| janders_ | yes I did | 21:32 |
| janders_ | no HT | 21:32 |
| oneswig | We found that makes a significant difference. | 21:32 |
| janders_ | I typically work with node-local SSD/NVMe for scratch | 21:32 |
| b1airo | to Linpack oneswig ? | 21:33 |
| janders_ | and a parallel fs indeed mounted via SRIOV interface | 21:33 |
| oneswig | b1airo: haven't tried that. Other benchmarks. | 21:33 |
| janders_ | on the local scratch it would be interesting to look at the impact of qcow2 vs lvm | 21:33 |
| janders_ | lvm helps with IOPS a lot, but in a scenario like the one we're discussing where there's little CPU for host OS, this might be even more useful | 21:34 |
| janders_ | so - b1airo - do you think in the configuration you proposed, would we need two "performance VM" flavors? | 21:34 |
| janders_ | max_cpu and low_latency (pinned)? | 21:35 |
| b1airo | janders_: we could talk more on this offline perhaps, i'd be keen to try and get some comparable benchmarks together as well | 21:35 |
| *** eharney has quit IRC | 21:35 | |
| oneswig | janders_: you need Ironic deploy templates... let's revisit that | 21:36 |
| janders_ | OK! being mindful of time, let's move on to the next topic. Thank you for your attention and great insights! | 21:36 |
| oneswig | OK time is pressing | 21:36 |
| *** wwriverrat has joined #openstack-meeting | 21:36 | |
| oneswig | #topic Terraform and Kubespray | 21:36 |
| *** openstack changes topic to "Terraform and Kubespray (Meeting topic: scientific-sig)" | 21:36 | |
| oneswig | deardooley: martial: take it away! | 21:36 |
| martial | so I invited Rion to this convesrsation | 21:36 |
| martial | but the idea is simple, I needed to deploy a small K8s cluster for testing on top of OpenStack | 21:37 |
| martial | internally we have used Kubespray to do so | 21:37 |
| martial | to deploy a Kubernetes (one master and two minion nodes) in an pre-configured OpenStack project titled nist-ace, using Kubespray and Terraform. | 21:37 |
| martial | the default Kubespray install requires the creation of pre-configured VMs | 21:38 |
| martial | #link https://github.com/kubernetes-sigs/kubespray/blob/master/docs/openstack.md | 21:38 |
| martial | Terraform has the advantage to pre-configures the OpenStack project for running the ansible playbook given information about networking, users, and the OpenStack project itself. Then Terraform handles the VM configurations and creations. | 21:38 |
| oneswig | Given Kubespray is Ansible, why the need for preconfiguration? | 21:38 |
| martial | #link https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md | 21:39 |
| martial | that was also my question, but the ansible script did not create the OpenStack instances | 21:39 |
| martial | terraform will | 21:39 |
| *** dklyle has quit IRC | 21:39 | |
| deardooley | @oneswig it's a pretty common pattern. terraform is much faster at pure provisioning that ansible, but config management is not it's strong suite. ansible is a good complement once the infrastructure is in place. | 21:40 |
| martial | you obviously need the OpenStack project's RC file | 21:40 |
| martial | once you have this sourced you are able to create the terraform configuration file to include master/minion number and names | 21:41 |
| martial | images, flavors, IP pools | 21:41 |
| *** priteau has joined #openstack-meeting | 21:41 | |
| *** mmethot has joined #openstack-meeting | 21:42 | |
| oneswig | How do you find Terraform compares to expressing the equivalent in Heat? | 21:42 |
| *** trandles has quit IRC | 21:42 | |
| *** mmethot has quit IRC | 21:43 | |
| martial | given that kubespray has its own ways of spawning on top of OpenStack, I did not try heat for this purpose | 21:43 |
| janders_ | in the context of a Private Cloud, would it make sense to disable port-security so that we don't need to worry about address pairs? | 21:44 |
| janders_ | or do you see value in having this extra layer of protection? | 21:44 |
| *** mmethot has joined #openstack-meeting | 21:44 | |
| martial | supposedely terraforms create the private network needed for indeed k8s communication | 21:44 |
| martial | on top of the OpenStack project | 21:44 |
| martial | 's own network | 21:44 |
| martial | once the configuration is done, calling terraform init | 21:45 |
| martial | followed by terraform apply | 21:45 |
| *** mmethot has quit IRC | 21:45 | |
| oneswig | What do you configure for Kubernetes networking? Does it use Weave, Calico, ? | 21:46 |
| deardooley | within the context of kubespray, you the terraform provisioner will handle all security group creation and managmeent for you as part of the process. You will need to implement any further security at the edge of your k8s apiservers on your own. | 21:46 |
| martial | (with a little extra obviously) creates the openstack | 21:47 |
| martial | I was checking in my configuration file and I do not see the k8s networking set | 21:48 |
| deardooley | it's pluggable. defaults to calico. there are some tweaks you need to make in your specific inventory depending on the existence of different openstack components in your particular cloud. | 21:48 |
| oneswig | Seems like a lot of interesting things are happening around blending the layers of abstraction, and interfacing with OpenStack to provide resources (storage, networking, etc) for Kubernetes - eg https://github.com/kubernetes/cloud-provider-openstack - does Kubespray support enabling that kind of stuff? | 21:48 |
| martial | the default seems to be Callico | 21:48 |
| oneswig | ... would be cool if it did | 21:49 |
| deardooley | for example, to plug into external loadbalancers, dynamically configure upstream dns, use cinder as persistent volume provisioner, etc. | 21:49 |
| oneswig | deardooley: that kind of stuff | 21:49 |
| deardooley | yeah. it's all pluggable with the usual caveates. | 21:50 |
| martial | (I am kind of tempting Rion here to think about a presentation at the summit on the topic of Kubespray deployment on top of OpenStack) | 21:51 |
| oneswig | Have you been using those cloud-provider integrations and is it working? | 21:51 |
| martial | I have not | 21:53 |
| deardooley | I use them on a couple different openstack installs. | 21:53 |
| deardooley | they work, but there are ways to get the job done, and there are ways to get the job done and keep all your hair and staff in tact | 21:54 |
| oneswig | deardooley: sounds familiar :-) | 21:54 |
| deardooley | it's likely anyone on this channel could pull it off in a day or two by pinging this list, but once you do, you'll appreciate the "Hey Google, create a 5 node Kubernes cluster" demo in a whole new way. | 21:55 |
| *** rbudden has joined #openstack-meeting | 21:55 | |
| deardooley | that being said, once you get your config down, it really is push button to spin up another dozen in the same environment. | 21:56 |
| oneswig | deardooley: in your experience, what does Kubespray do wrong / badly? Does it have shortcomings? | 21:56 |
| martial | after Terraform pre-configures everything is it simply the steps or running the ansible playbook | 21:56 |
| *** jamesmcarthur has joined #openstack-meeting | 21:56 | |
| b1airo | certainly sounds like it could be an interesting presentation topic | 21:57 |
| deardooley | it can build a production scale cluster for you. it can't do much to help you manage it. | 21:57 |
| b1airo | i think i need a diagram of the openstack - kubespray - terraform interactions | 21:57 |
| martial | see Rion "it could be an interesting presentation topic" :) | 21:57 |
| martial | (wink) | 21:58 |
| deardooley | as long as you treat nodes idempotently and get sufficient master quorem defined up front, it's not a huge issue. when something goes weird, just delete it, remove the node, and rerujn the cluster.yml playbook with a new vm. | 21:58 |
| oneswig | We are nearly at time - final thoughts? | 21:58 |
| b1airo | one other motivation type question? why not Magnum? | 21:58 |
| martial | in my particular case, the person I was working with wanted K8s to test containers ... truth is given how many containers they want, docker swarm might be enough | 21:59 |
| deardooley | flexibility, portability across cloud providers, secruity across multiple layers of the infrastructure and application stack, logging, monitoring, etc... | 21:59 |
| deardooley | +1 | 21:59 |
| oneswig | OK, we are at time | 22:00 |
| martial | and I will second Rion's comment of "remove" "rerun", that was very useful for testing things | 22:00 |
| oneswig | Thanks deardooley martial - interesting to hear about your work. | 22:00 |
| *** jamesmcarthur has quit IRC | 22:00 | |
| janders_ | great work, thanks guys! | 22:00 |
| oneswig | Until next time | 22:00 |
| oneswig | #endmeeting | 22:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:00 | |
| openstack | Meeting ended Tue Jan 22 22:00:50 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:00 |
| martial | happy to share :) | 22:00 |
| b1airo | thanks guys! | 22:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-22-21.00.html | 22:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-22-21.00.txt | 22:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-01-22-21.00.log.html | 22:00 |
| martial | bye | 22:01 |
| *** janders_ has quit IRC | 22:01 | |
| *** wwriverrat has quit IRC | 22:01 | |
| *** oneswig has quit IRC | 22:01 | |
| *** priteau has quit IRC | 22:08 | |
| *** rcernin has joined #openstack-meeting | 22:10 | |
| *** slaweq has quit IRC | 22:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:14 | |
| *** armax has quit IRC | 22:17 | |
| *** rcernin has quit IRC | 22:17 | |
| *** rcernin has joined #openstack-meeting | 22:19 | |
| *** jamesmcarthur has quit IRC | 22:20 | |
| *** TxGirlGeek has quit IRC | 22:24 | |
| *** armax has joined #openstack-meeting | 22:24 | |
| *** TxGirlGe_ has joined #openstack-meeting | 22:25 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:28 | |
| *** slaweq has joined #openstack-meeting | 22:29 | |
| *** TxGirlGe_ has quit IRC | 22:31 | |
| *** devananda has joined #openstack-meeting | 22:31 | |
| *** bbowen has joined #openstack-meeting | 22:33 | |
| *** slaweq has quit IRC | 22:33 | |
| *** lifeless_ is now known as lifeless | 22:34 | |
| *** TxGirlGeek has joined #openstack-meeting | 22:40 | |
| *** imacdonn_ has quit IRC | 22:48 | |
| *** imacdonn_ has joined #openstack-meeting | 22:49 | |
| *** baojg has joined #openstack-meeting | 22:52 | |
| *** efried has quit IRC | 22:53 | |
| *** TxGirlGeek has quit IRC | 22:59 | |
| *** slaweq has joined #openstack-meeting | 23:00 | |
| *** TxGirlGeek has joined #openstack-meeting | 23:04 | |
| *** slaweq has quit IRC | 23:05 | |
| *** mattw4 has quit IRC | 23:07 | |
| *** a-pugachev has joined #openstack-meeting | 23:08 | |
| *** armax has quit IRC | 23:31 | |
| *** ociuhandu_ has joined #openstack-meeting | 23:32 | |
| *** ociuhandu has quit IRC | 23:33 | |
| *** TxGirlGeek has quit IRC | 23:35 | |
| *** rbudden has quit IRC | 23:37 | |
| *** _alastor_ has quit IRC | 23:53 | |
| *** mattw4 has joined #openstack-meeting | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!