Tuesday, 2016-05-17

*** dimtruck is now known as zz_dimtruck00:52
*** arnoldje has quit IRC00:52
*** arnoldje has joined #openstack-performance01:25
*** rfolco has quit IRC01:52
*** zz_dimtruck is now known as dimtruck01:53
*** fredli has joined #openstack-performance02:02
*** dimtruck is now known as zz_dimtruck03:38
*** harlowja_at_home has joined #openstack-performance04:29
*** arnoldje has quit IRC04:52
*** fredli has quit IRC04:58
*** harlowja_at_home has quit IRC05:20
*** ad_rien_ has quit IRC06:30
*** haukebruno has joined #openstack-performance06:32
*** nineseconds has joined #openstack-performance06:44
*** nineseconds has quit IRC06:50
*** nineseconds has joined #openstack-performance06:51
*** rmart04 has joined #openstack-performance07:38
*** dmk0202 has joined #openstack-performance08:12
*** dmk0202 has quit IRC09:31
*** ad_rien_ has joined #openstack-performance11:30
*** ad_rien_ has quit IRC11:37
*** rfolco has joined #openstack-performance11:46
*** dmk0202 has joined #openstack-performance11:52
*** lappy-rook has joined #openstack-performance12:03
*** ad_rien_ has joined #openstack-performance12:09
*** rfolco has quit IRC12:36
*** rfolco has joined #openstack-performance12:36
*** ad_rien_ has quit IRC13:19
*** ad_rien_ has joined #openstack-performance13:20
*** lappy-rook has quit IRC13:20
*** openstackgerrit has quit IRC13:32
*** openstackgerrit has joined #openstack-performance13:33
*** nineseconds has quit IRC13:42
*** zz_dimtruck is now known as dimtruck13:44
*** dimtruck is now known as zz_dimtruck13:53
*** zz_dimtruck is now known as dimtruck13:57
*** med_ has joined #openstack-performance14:04
*** med_ has quit IRC14:04
*** med_ has joined #openstack-performance14:04
*** arnoldje has joined #openstack-performance14:17
*** rfolco has quit IRC14:23
*** rfolco has joined #openstack-performance14:24
*** markvoelker has joined #openstack-performance14:35
*** markvoelker has quit IRC14:35
*** markvoelker has joined #openstack-performance14:36
*** markvoelker has quit IRC14:39
*** markvoelker has joined #openstack-performance14:39
*** markvoelker has quit IRC14:51
*** markvoelker has joined #openstack-performance14:53
*** markvoelker has quit IRC14:57
*** markvoelker has joined #openstack-performance14:58
*** markvoelker has quit IRC15:04
*** dmk0202 has quit IRC15:33
*** dimtruck is now known as zz_dimtruck15:48
DinaBelovaMeeting reminder for everyone survived after the summit :)15:53
DinaBelovain 7 minutes15:53
*** haukebruno has quit IRC15:59
*** gokrokve has joined #openstack-performance15:59
DinaBelova#startmeeting Performance Team16:00
openstackMeeting started Tue May 17 16:00:10 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
openstackThe meeting name has been set to 'performance_team'16:00
*** rohanion has joined #openstack-performance16:00
DinaBelovahey folks o/16:00
ad_rien_Hi16:00
rohanionHi Dina16:00
rookhey16:00
DinaBelovawow, I'm glad people did not forget about the meeting :)16:00
rooklol16:00
DinaBelovait's a after-summit recovery :)16:00
DinaBelovalet's wait for a few more moments16:01
gokrokve0/16:01
DinaBelovait's sad I do not see AugieMena yet16:01
rookhm16:01
DinaBelovalet's hope he'll join16:02
*** zz_dimtruck is now known as dimtruck16:02
rookI can see if I can find someone who might know where he is16:02
DinaBelovarook ack16:02
DinaBelovaanyway, let's get started16:03
DinaBelovawith action items first16:03
DinaBelova#topic Action Items16:03
DinaBelovain fact we had two of them16:04
DinaBelovaone on myself gerarding Keystone performance multi node testing16:04
luzC o/16:04
DinaBelovasadly i do not have something specific to share regarding summit and my vacation >_<16:04
DinaBelovaI'll leave this on myself for the next time16:05
rooko/16:05
DinaBelova#action DinaBelova share status of Keystone multinode testing16:05
rookno word from Augie or Marcio16:05
DinaBelovaluzC o/16:05
DinaBelovarook ack16:05
DinaBelovaso I'll leavie his action itme with no changes16:05
DinaBelova#action AugieMena share details about some v3 limitations in Rally16:05
DinaBelovahe planned to share his results about some v3 limitations and related difficulties in rally16:06
DinaBelovaso let's keep this till next time16:06
rookI have heard of other users having v3 issues with Rally16:06
rookmostly around keystone.16:06
DinaBelovarook - any specific issues?16:06
rookI haven't personally hit it.16:06
rooknegative.16:06
* rook wonders if there are launchpads for the issues?16:06
DinaBelovarook can you please point the people boris-42 and I should talk about these issues with?16:07
DinaBelovarook it is - both for keystone and rally, but if there are some specific ones...16:07
DinaBelovaok, so anyway i guess it'll be better to find AugieMena :)16:08
DinaBelovaso I'll probably jump to the next topic16:08
DinaBelova#topic Summit follow-up16:08
DinaBelovain fact I loved how this summit went from the performance topic perspective16:09
*** msimonin has joined #openstack-performance16:09
DinaBelovaI want to quickly go through the interesting sessions16:09
DinaBelovafor folks who had no chance to attend - like rohanion and gokrokve16:09
DinaBelovafirst of all the working group session we had on Wednesday16:10
DinaBelova#link https://etherpad.openstack.org/p/newton-performance-team16:10
DinaBelovaI loved the way how it went - people seemed to be enthusiastic about the progress made by team and suggested several things to be done during Newton16:11
DinaBelovaI need to go through the suggestions and notes about some other performance benchmarks people are running16:11
DinaBelovafor instance there was a suggestion to take a look on the SPECCloud - a benchmark from the SPEC organization16:12
DinaBelovasadly I did not have a chance to take a look on it yet, but that's in my todo list for sure16:12
rookDinaBelova I helped build that workload so feel free to ask questions :)16:13
gokrokveThere are a lot of ideas for Newton. Do we have priorities for them?16:13
DinaBelovarook cool16:14
DinaBelovagokrokve - there were several suggestions about what can be tested in the networking test as well - like we planned to focus on Neutron, Contrail, Midokura, Calico - but folks asked to take a look on the OpenDaylight and DragonFlow as well if possible16:14
rookOVN too :D16:14
DinaBelovarook indeed16:14
DinaBelovagokrokve - I'll check this with ilyashakhat - as we surely have limited working cycles on this task16:15
DinaBelovaI'm sure we won't cover everything, so let's decide what can be covered16:15
DinaBelova#action DinaBelova ilyashakhat prioritize Networking testing ideas from https://etherpad.openstack.org/p/newton-performance-team16:15
gokrokveSure. We can actually vote for each item in this etherpad.16:15
DinaBelovagokrokve - good idea, let's do it16:16
gokrokveJust put +1 for an item which looks interesting and important for you.16:16
DinaBelovagokrokve ++16:16
DinaBelova#info let's vote for most interesting Networking testing ideas from https://etherpad.openstack.org/p/newton-performance-team with simple +116:17
DinaBelovain fact all other ideas were very welcomed16:17
DinaBelovaas well :)16:17
DinaBelovaI should mention several other sessions as well16:18
DinaBelovatwo of them were held as a pat of Large Deployment Team effort during the summit16:18
DinaBelova#link https://etherpad.openstack.org/p/AUS-ops-Large-Deployments-Team16:18
DinaBelovathere were mostly deployment and scale issues and interesting moments discussed, several performance related questions as well16:19
DinaBelovaI just recommend to go through this etherpad and take a look on the issues real operators and active LDT members are focused on16:20
DinaBelovato be on track16:20
DinaBelovathere was interesting test idea for our team proposed as well16:20
DinaBelovafolks were concerned about number of messages Neutron can generate16:21
DinaBelovaThings like security group changes can generate LOTS of messages - we can try to create some synthetic test via rally and measure number of messages that are trying to use messaging bus16:21
DinaBelovawe have synthetic numbers already16:22
rookI have a test to look at ovsct (new firewall driver) vs the iptables work.16:22
DinaBelovarook good idea16:22
rookWe have throughput numbers, but now we are looking at the control-plane as well.16:22
rookUsing Rally-plugins i created a scenario to create N networks and attachN networks to the guest.16:23
rookwhich will also bloat the firewall rules since they are mostly mac address driven.16:23
DinaBelova#info let's play around the Networking/Messaging test ideas proposed in https://etherpad.openstack.org/p/AUS-ops-Large-Deployments-Team16:23
DinaBelovarook ack, thank you sir16:24
DinaBelovathere is one more session that looks really interesting for myself16:25
DinaBelovanot the session in fact, but operators meetup16:25
DinaBelova#link https://etherpad.openstack.org/p/AUS-ops-informal-meetup16:25
DinaBelovait has super long etherpad as a result :D16:25
DinaBelovain fact I'll just recommend to go through the etherpad at least and see what topics are hot for now from the operators perspective16:26
DinaBelovalots of issues are related to upgradability and deployment, but at the same time there is a bunch of scalability / performance issues mentioned and discussed16:27
DinaBelovaright now I'm not ready to point on the specific ones, as lots of them (as it turned out) are either fixed in the latest OpenStack or have nice workarounds16:27
DinaBelovaso I need additional research on this etherpad now16:28
DinaBelovahope to do it soon16:28
DinaBelovaI think that's all I wanted to share after a summit - at least for now16:28
DinaBelovaany comments?16:29
ad_rien_Hi DinaBelova16:29
DinaBelovaad_rien_ hey :)16:29
ad_rien_Maybe we can discuss a bit about scalability evaluations we would like to perform at Inria16:29
DinaBelovaad_rien_ - good idea, let's move it to the end of meeting16:29
ad_rien_on top of Grid'5000 (see link available in the pad on just google for grid'5000)16:29
DinaBelovato the Open Discussions part16:29
ad_rien_thanks16:30
DinaBelovaack16:30
DinaBelovaok, so let's jump to the OSprofiler16:30
DinaBelova#topic OSProfiler weekly update16:30
DinaBelovarohanion hey o/16:30
rohanionHiiii16:30
rohanionso let's begin16:30
DinaBelovalooks like you had 3 weeks :D16:30
DinaBelovanot just one :)16:30
DinaBelovasure, the floor is yours16:30
rohanionOk, first of all - I've completed writing the new architecture16:31
rohanionnow it's time for code review16:31
DinaBelovanew drivers architecture?16:31
rohanionhttps://review.openstack.org/#/c/247005/ here's the base commit16:31
rohanionyes16:31
DinaBelova#link https://review.openstack.org/#/c/247005/16:32
rohanionalso I've split the ceilometer driver from the base commit16:32
rohanion#link https://review.openstack.org/#/c/251343/16:32
rohanionMongoDB driver is working and only small fixes have to be written16:33
rohanion#link https://review.openstack.org/#/c/310530/16:33
DinaBelovarohanion any numbers about MongoDB vs Ceilometer ?16:33
rohanionalso, I've commited changes to cinder, glance and horizon for them to support the new osprofiler api16:33
rohanion#link https://review.openstack.org/#/c/315676/16:34
rohanion#link https://review.openstack.org/#/c/316799/16:34
rohanion#link https://review.openstack.org/#/c/273085/16:34
DinaBelovaboris-42 harlowja ^^ we need to go through this bunch of commits :S16:34
DinaBelovarohanion thank you sir :D16:35
rohanionDinaBelova: yes, building a full report with mongodb driver takes 0.7 seconds16:35
DinaBelovait looks like I have looooooots of stuff to review after the vacation :D16:35
rohanioncompare that with 5 seconds for ceilometer16:35
DinaBelovarohanion I guess horizon folks are much happier with these numbers :)16:35
DinaBelovarohanion ok, I'll start reviewing on Thursday16:36
DinaBelovarohanion please ping other core reviewers as well - boris-42 and harlowja16:36
rohanionYes, Timur is very happy but there's an issue with messaging in horizon, we have to create the RPC transport ourselves and it's tricky as for me16:37
rohanionWill do, thank you!16:37
DinaBelovarohanion that's true16:37
DinaBelovaack16:37
DinaBelovaok, it looks like that's all for OSprofiler part16:37
DinaBelovalet's jump to the Open Discussions16:37
DinaBelova#topic Open Discussion16:37
DinaBelovaI have no specific items, but we have ad_rien_ from Inria today :))16:38
ad_rien_;)16:38
DinaBelova#link https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home16:38
ad_rien_Thanks16:38
ad_rien_To make a long story short, we are working on massively distributed clouds at Inria (from the research viewpoint) and we are interested to understand the current limit of the OpenStack core components16:38
ad_rien_We briefly exchanged with some folks during the summit and it looks that evaluating the zeroMQ solution16:39
ad_rien_make sense as a first TODO.16:39
DinaBelovaad_rien_ - in fact rohanion did several 0MQ researches already16:39
* DinaBelova looking if that was published to the performance-docs16:39
rohanionit was, as I remember16:40
rohanionI pushed the changes a long time ago16:40
DinaBelovarohanion ok, but I do not see them on the http://docs.openstack.org/developer/performance-docs/#16:40
DinaBelovaprobably because ilyashakhat was finishing your work16:40
ad_rien_It would be great if we can have a link to such results16:40
ad_rien_So we have few questions regarding that point: What can be a representative enough experimental protocol ?16:41
DinaBelovarohanion - can you please investigate if the 0MQ results will be publiched soon with Ilya?16:41
rohanionah, I remember. there's a bug that blocks HA for 0mq16:42
DinaBelovaad_rien_ we have composed Messaging Bus testing plan some time ago - http://docs.openstack.org/developer/performance-docs/test_plans/mq/plan.html16:42
rohanionsure I will16:42
ad_rien_Ok so what is the best way to move forward on that point ?16:42
DinaBelovawith several test cases - RPC Call Throughput Test, RPC Cast Throughput Test, Notification Throughput Test16:43
ad_rien_1./ wait for the rohanion's answer regarding the previous experiments16:43
rohanionI'll look for my notes, they have to be published somewhere16:43
DinaBelova2./ ad_rien_ - please go through the test plan http://docs.openstack.org/developer/performance-docs/test_plans/mq/plan.html and see if it looks ok for you16:43
ad_rien_2./ perform preliminary experiments according to the link you pasted DinaBelova16:43
rohanionI suggest doing both16:43
DinaBelovayes, indeed16:43
rohanionad_rien_16:43
DinaBelovaprobably you'll have some ideas to be added to this test plan16:44
ad_rien_ok if we have questions, is there a particular person to contact ?16:44
DinaBelovamyself, ilyashakhat and rohanion can answer them :)16:44
DinaBelovaad_rien_ ^^16:44
ad_rien_cool thanks16:44
rookad_rien_: might i ask why the ZeroMQ direction?16:44
rookad_rien_ is it due to the distributed environment you have?16:45
rohanionthe best openstack doctors recommend giving up on zmq until the sockets issue is fixed16:45
rookrohanion: exactly.16:45
DinaBelovaad_rien_ - please notice that this specific test plan is playing around oslo.messaging + SOME_DRIVER testing via oslo simulator tool16:45
ad_rien_We are working on a particular use-case related to Fog/Edge cloud computing solutions (i.e. a cloud that is spread over a significant number of small data centers)16:45
ad_rien_DinaBelova:  ok thanks16:46
DinaBelovaprobably you'll find out this is not 100% suitable for you - we can think either on the tool improvement or test plan extension, etc.16:46
ad_rien_We will give a look to the different information you gave us and come back to you soon16:46
ad_rien_The second experiment we would like to perform is a a scalability test of OpenStack16:46
ad_rien_with Rally.16:47
ad_rien_We are working on that right now16:47
DinaBelovaad_rien_ - currently 0MQ driver has one socket issue rohanion and rook mentioned ^^ - with node number growing up, 0MQ eats more sockets on the cloud controller nodes... too many16:47
ad_rien_we have several ways to achieve it on top of Grid'500016:47
DinaBelovarohanion - can you find a link to the bug?16:47
DinaBelovaad_rien_ ack16:47
msimoninthe socket issue is it a bug from zeroMQ ?16:48
ad_rien_or a way OpenStack is using it ?16:48
ad_rien_(thanks msimonin)16:48
DinaBelovamsimonin ad_rien_ - it's a logical continuation of 0MQ usage16:49
DinaBelovain openstack16:49
rohanionhttps://bugs.launchpad.net/oslo.messaging/+bug/155500716:49
openstackLaunchpad bug 1555007 in oslo.messaging "[zmq] ZMQ driver eats too many TCP sockets" [Undecided,Fix released] - Assigned to Oleksii Zamiatin (ozamiatin)16:49
rohanionhere we go16:49
ad_rien_Ok thanks16:49
DinaBelovatoo many peer-to-peer connectivity without any proxies from the 0mq driver that's written right now16:49
ad_rien_we are going to give a careful look16:50
rohanionproxies don't help as much as we want them16:50
DinaBelovarohanion that's true16:50
DinaBelovaI just used this as an example16:50
rohanionwe have spotted a slight decrease in the number of sockets, but still it eats tons of them16:51
rohanionbut ten thousand less than without using proxies16:51
rookI saw some blueprints for reusing sockets16:51
rookah ha.16:51
* rook sees rohanion posted it.16:51
DinaBelovaand that's normal for the 0MQ protocol, but not appropriate yet for the real usage16:51
rohanionwas it me? O_o16:52
DinaBelovarohanion, rook indeed16:52
DinaBelova:D16:52
ad_rien_Ok is there any other information we should know about the couple zeroMQ/openStack  before starting on that action :-P ?16:52
DinaBelovaad_rien_ nothing specific yet16:52
rohanionfor now bunny is much more reliable.16:52
msimonin:)16:52
DinaBelovawe'll check if something else can be found to help you16:52
ad_rien_ok just for your information we plan to work on that aspect during this summer (we are currently hiring an engineer that will work on scalability evaluations)16:53
DinaBelovaack, cool16:53
DinaBelovaanything else?16:53
ad_rien_so regarding the second experiments about the scalability evaluation, there are few items on the pad16:53
ad_rien_(see New tests planned section)16:54
ad_rien_#link https://etherpad.openstack.org/p/newton-performance-team16:54
msimoninWe had some discussions about qpid dispatch router at the summit16:54
DinaBelovaad_rien_ some of them we'll cover on Mirantis-Intel lab, but I think two labs will be even better :D16:54
ad_rien_yes that's exactly my poin16:55
ad_rien_t16:55
ad_rien_how can we merge the efforts.16:55
msimoninany thought on qpid dispatch router as brokerless communication solutioj16:55
msimoninsolution*16:55
harlowjaDinaBelova u alive???!!??16:55
DinaBelovaharlowja hahahha16:55
rohanionhehey joshua16:55
rohanionhow have u been?16:55
DinaBelovamsimonin - I dunno myself16:56
ad_rien_As it is mentioned on the pad, there are different ways to perform such evaluations (using docker, VM, tools like infrastructure emulators)16:56
DinaBelovaI guess we need to ping some oslo.messaging guys for this16:56
DinaBelovadims_ ?16:56
DinaBelovamsimonin - I'll check it with our oslo.messaging folks16:57
dims_hey DinaBelova16:57
DinaBelovahey :)16:57
ad_rien_it would be nice if we can use the same deployment scripts16:57
DinaBelovadims_ - it was a question from msimonin16:57
dims_what's up?16:57
dims_msimonin : DinaBelova : qpid dispatch router? we'll need kgiusti16:58
DinaBelovadims_ about 0MQ current driver limitations and other drivers and solutions like qpid dispatch router as brokerless communication and it's comparison16:58
DinaBelovadims_ ack16:58
msimonindims_:  ok thanks16:58
DinaBelovaad_rien_ - sure16:58
ad_rien_did you start to work on that on your side ?16:59
DinaBelovaonce we'll finalize test plans16:59
dims_DinaBelova : 0MQ will support direct connections for smaller number of nodes and broker-ed for larger number of nodes16:59
DinaBelovaad_rien_ - well, I did not find this out yet :D my vacation is till thursday16:59
DinaBelovaI'm a bit out of context16:59
DinaBelovaad_rien_ - I'll update you once I'll have any info16:59
ad_rien_ok so the first action is to work on the experimental protocol ?16:59
dims_DinaBelova : we should call kgiusti and ozamiatin for next meeting :)17:00
DinaBelovadims_ ack17:00
DinaBelovaad_rien_ i believe yes17:00
DinaBelovatogether with rohanion from our side17:01
DinaBelovaok, so we're out of time in fact17:01
DinaBelovaso let's end the discussion for now17:01
ad_rien_yes I understand, do not worry17:01
ad_rien_ok17:01
DinaBelovaand collect all info about the items discussed today :)17:01
DinaBelovathanks!17:01
DinaBelova#endmeeting17:01
openstackMeeting ended Tue May 17 17:01:45 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-05-17-16.00.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-05-17-16.00.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-05-17-16.00.log.html17:01
msimoninquoi ?17:02
msimonin(sorry wrong chan)17:02
harlowjabusy is how i've been :-P17:02
*** msimonin has left #openstack-performance17:03
*** rohanion has quit IRC17:05
*** ad_rien_ has quit IRC17:08
*** rmart04 has quit IRC17:22
*** wootehfoot has joined #openstack-performance17:55
*** wootehfoot has quit IRC19:07
*** wootehfoot has joined #openstack-performance19:08
*** rmart04 has joined #openstack-performance19:15
*** rmart04_ has joined #openstack-performance19:19
*** rmart04 has quit IRC19:20
*** rmart04_ is now known as rmart0419:20
*** gokrokve has quit IRC19:36
*** asti_ has joined #openstack-performance19:49
*** rmart04 has quit IRC20:12
*** msimonin has joined #openstack-performance20:13
*** wootehfoot has quit IRC20:15
*** dmk0202 has joined #openstack-performance20:29
*** rfolco has quit IRC20:47
*** rfolco has joined #openstack-performance20:47
*** rfolco has quit IRC21:01
*** dmk0202 has quit IRC21:10
*** markvoelker has joined #openstack-performance22:14
*** msimonin has quit IRC22:14
*** asti_ has quit IRC22:20
*** markvoelker has quit IRC22:21
*** markvoelker has joined #openstack-performance22:22
*** dimtruck is now known as zz_dimtruck22:27
*** arnoldje has quit IRC22:39
*** zz_dimtruck is now known as dimtruck22:45
*** markvoelker has quit IRC23:12

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!