Tuesday, 2016-12-20

« Monday, 2016-12-19 Index Wednesday, 2016-12-21 »
*** yujunz has joined #openstack-performance00:21
*** catintheroof has quit IRC00:29
*** dimtruck is now known as zz_dimtruck00:55
*** jkilpatr has quit IRC01:06
*** yujunz is now known as yujunz[away]01:12
*** yujunz[away] is now known as yujunz01:13
*** zz_dimtruck is now known as dimtruck01:16
*** tovin07_ has joined #openstack-performance03:41
*** yujunz has quit IRC04:08
*** dimtruck is now known as zz_dimtruck05:48
*** yujunz has joined #openstack-performance05:59
*** yujunz-zte has joined #openstack-performance07:00
*** yujunz has quit IRC07:02
*** pcaruana has joined #openstack-performance08:19
*** yujunz-zte is now known as yujunz[away]08:20
*** yujunz[away] is now known as yujunz-zte08:23
*** msimonin has joined #openstack-performance08:24
*** msimonin has quit IRC08:25
openstackgerrityunfeng zhou proposed openstack/performance-docs: add CONTRIBUTING.rst  https://review.openstack.org/41289808:48
*** yujunz-zte is now known as yujunz[away]09:04
*** yujunz[away] is now known as yujunz-zte09:05
*** yujunz-zte is now known as yujunz[away]09:05
*** yujunz[away] is now known as yujunz-zte09:05
*** yujunz-zte has quit IRC10:13
*** tovin07_ has quit IRC10:15
*** jkilpatr has joined #openstack-performance10:56
*** msimonin has joined #openstack-performance11:09
*** msimonin has quit IRC11:10
*** msimonin has joined #openstack-performance11:13
*** msimonin has quit IRC11:14
*** jkilpatr has quit IRC11:28
*** yujunz has joined #openstack-performance11:51
*** jkilpatr has joined #openstack-performance12:07
openstackgerritIlya Shakhat proposed openstack/performance-docs: Kubernetes density testing  https://review.openstack.org/41304812:12
*** jkilpatr has quit IRC12:20
*** jkilpatr has joined #openstack-performance12:20
*** yujunz has quit IRC12:26
*** catintheroof has joined #openstack-performance12:37
*** pcaruana has quit IRC13:01
*** pcaruana has joined #openstack-performance13:06
*** yujunz has joined #openstack-performance13:29
*** yujunz has quit IRC13:29
*** yujunz has joined #openstack-performance13:30
*** catinthe_ has joined #openstack-performance14:06
*** catintheroof has quit IRC14:08
*** pcaruana has quit IRC14:23
*** pcaruana has joined #openstack-performance14:37
*** Guest67717 is now known as med_14:48
*** med_ has quit IRC14:48
*** med_ has joined #openstack-performance14:48
*** zz_dimtruck is now known as dimtruck14:55
*** dimtruck is now known as zz_dimtruck15:05
*** tovin07_ has joined #openstack-performance15:11
*** vbala has joined #openstack-performance15:14
openstackgerritIgor Yozhikov proposed openstack/performance-docs: Test plan for k8s+OS+Cinder+Ceph  https://review.openstack.org/41193315:17
*** rcherrueau has joined #openstack-performance15:28
DinaBelova#startmeeting Performance Team15:30
openstackMeeting started Tue Dec 20 15:30:14 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.15:30
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:30
openstackThe meeting name has been set to 'performance_team'15:30
DinaBelovahey folks!15:30
akrzosHey DinaBelova15:30
rcherrueauo/15:30
tovin07_o/15:30
vbalaHi15:31
DinaBelovalet's wait for a few moments to ensure everyone who wanted joined :)15:31
lezbar__o/15:32
DinaBelovahey lezbar__ o/15:32
DinaBelovaso I guess we may get started15:32
DinaBelova#topic Action Items15:32
DinaBelovalast time we had only one action item on me15:32
DinaBelovaregarding verification of what grafana backend Mirantis is using15:33
DinaBelovain fact we're using right now plain Prometheus with its own database15:33
DinaBelovawe plan to add persistent time series storage (e.g. Cassandra or OpenTSDB) a bit later15:33
DinaBelovato store old monitoring data15:34
DinaBelovaand then we'll need to modify our grafana boards a bit15:34
DinaBelovato grab data from it15:34
DinaBelovabut right now it's plain prometheus15:34
DinaBelovaI don't remember who was asking this question, I believe it might be you, akrzos15:34
DinaBelovaso we may proceed to the current progress15:35
DinaBelova#topic Current progress on the planned tests15:36
DinaBelovarcherrueau it looks like you're only guy from inria today :)15:36
rcherrueau Yes, msimonin is on holiday, so I will speak for him/Inria.15:36
DinaBelovarcherrueau cool :)15:36
DinaBelovaplease go ahead15:36
rcherrueauWe are working on two stuff. First, deploy a multi-region OpenStack with kolla.15:36
rcherrueau15:36
rcherrueauThis almost works.15:37
DinaBelovaany issues met?15:37
DinaBelovaprobably we may list bugs here15:37
DinaBelovaif any15:37
rcherrueauWe have something we call the Administrative Region (AR) that contains Keystine, MariaDB (wth Keystone tables) and Memcached.15:37
rcherrueauThis AR also contains one HAProxy since we deploy with kolla.15:38
rcherrueau15:38
rcherrueauWe have then, n OpenStack Region (OSRn) that each contains Nova, Glance, Neutron, RabbitMQ, MariaDB and HAProxy15:38
rcherrueauEach OSR register itself to the AR Keystone. And when an operator connect itself to Horizon, he has to choose between all OSR15:38
rcherrueauTo do so, we have to patch kolla a little bit. We plan to make a mail on the kolla mailing list to share our experience with the community15:39
DinaBelovaso you have keystone separated from the OSR to separated region? just to make sure15:40
rcherrueauSo, no special issues except patches we have to do on the kolla-ansible code.15:40
rcherrueauYes, exactly15:40
DinaBelovarcherrueau ok, and those regions might be located in different locations theoretically15:40
rcherrueauYes, this is the idea15:41
DinaBelovaI think that keystone performance might be the issue in this case :/15:41
DinaBelovaI think although you'll test it anyway :)15:41
rcherrueauYes we will, and this comes to the second stuff we are working on15:41
DinaBelovaok, thank you rcherrueau - please keep us updated regarding your experiments :)15:42
rcherrueauAt the same time we are adding `netem` to our deployment and test tool15:42
DinaBelovaand the second?15:42
rcherrueau`netem` is a Linux tool that lets you emulate network latency, low bandwidth, packet loss ...15:42
akrzoswhat about setting latency via tc?15:43
rcherrueauThe idea is to make a several multi-region deployment on our G5k platform. Then use `netem` to simulate different locations with different latencies, bandwidth and see how OpenStack behaves15:43
DinaBelova#info Inria had to modify Kolla a bit to be able to proceed with their type of multisite deployment (Administrative Region and n OpenStack Regions)15:43
rcherrueauakrzos: netem is tc ;)15:43
akrzosah15:44
akrzos:D15:44
DinaBelova#info the second part of work is oriented on adding `netem` to their deployment and test tool - o simulate different locations with different latencies, bandwidth and see how OpenStack behaves15:44
DinaBelovaok, thanks rcherrueau15:44
rcherrueaumsimonin is working hard on this second part15:44
DinaBelovahope to see him next week :)15:44
DinaBelovaakrzos any update from you sir? afair you got new HW for the telemetry testing :)15:45
akrzosso beeing running into bottlenecks in telemetry services15:45
akrzosfirst was too few metricd workers15:46
akrzosthis is with 3 controllers, 4 ceph nodes, 10 computes15:46
akrzosbooted 1k instances15:46
DinaBelova#info akrzos has started work on telemetry testing following the test plan - http://docs.openstack.org/developer/performance-docs/test_plans/telemetry_scale/plan.html15:46
akrzosgnocchi backlog continously grows15:46
akrzos$os_Workers limits metricd workers to 6 on my controllers15:47
akrzos(24 logical cpu cores)15:47
akrzosso i redeployed overrideing it with 48 workers15:47
akrzosso 48 workers on each controller15:47
akrzosso 144 total metricd workers15:47
*** yujunz has quit IRC15:47
akrzosalso reduced metric processing delay15:47
akrzosfrom 60s to 30s15:47
akrzosand 1k instances is now handled in realtime15:48
akrzosin ceph there is 36 osds15:48
akrzosalso needed to tune pgs to avoid ceph health_warn15:48
*** yujunz has joined #openstack-performance15:48
akrzosthough the calculation for this is tricky using pgcalc15:48
akrzosso with this tuning i can now sustain 1k instances in the cloud aiwth gnocchi15:49
akrzoson low archival policy15:49
*** harlowja has joined #openstack-performance15:49
akrzosi attempted to scale further15:49
akrzos(wanted 2k)15:49
akrzosand got to ~1.9k before hitting new problems15:49
akrzosload avg on controllers is >core count15:49
DinaBelovawow15:50
DinaBelovait's huge load15:50
akrzosmemory is rising in both rabbitmq and ceilometer-collector15:50
akrzosat this scale now15:50
akrzosalso15:50
akrzosto get to 1.9 k15:50
akrzosi had to tune threads in gnocchi15:50
akrzosaggregation worker threads is default to 115:50
DinaBelovait looks like that potentially for ~2k VMs gnocchi and rabbit needs to be separated from each other to different nodes - with more nodes given to control plane side of the cloud15:50
akrzosmy concern now is the collector grows as i have seen in the past15:51
akrzosi thouigh there was a patch put in to limit the # of messages it grabs off rabbit15:51
akrzosto prevent growth15:51
akrzosbut i don't understand the problem enough right now15:51
DinaBelovaakrzos ack, thank you sir15:52
akrzosso another factor15:52
akrzosis the archival policy15:52
akrzoshigh policy might actually mean less aggregations being "Recalculated"15:52
akrzosand could actually be a lower workload15:52
akrzosdue to a finer grain "end" timeframe15:52
akrzosso i should retest with a new archival policy15:53
akrzosand maybe different number of aggregations15:53
akrzosso lots to try still15:53
akrzosanother thing i can share with the community is a collectd plugin i wrote to monitor gnocchi backlog15:53
akrzos#link https://review.openstack.org/#/c/411030/4/ansible/install/roles/collectd-openstack/files/collectd_gnocchi_status.py15:54
akrzosI think that summerizes the chaos i've been working on as of last week pretty well :D15:54
DinaBelovaack, really good job being done15:54
DinaBelovathanks akrzos15:54
akrzosthanks15:55
akrzosalso i agree separating telemetry from control plane for scale is a must15:55
DinaBelovayeah, I believe this is needed15:56
DinaBelovaon that scale of monitored resources15:56
DinaBelovaok, from mirantis side we've started uploading test plans / results for some recent researches15:56
DinaBelova#link https://review.openstack.org/41193315:56
DinaBelova#link https://review.openstack.org/41304815:56
DinaBelovathe first one is regarding Cinder performance with Ceph backend - in case of running OpenStack services on k8s15:57
DinaBelovaCeph is installed separately of course :)15:57
DinaBelovathe second one is related to max pods per host density testing15:58
DinaBelovain fact what we got was a bit disappointing15:58
DinaBelovaafter 200 pods being run on the host the overall process of scheduling, etc. becomes really slow15:58
DinaBelovaso 400 pods is almost the limit here15:58
DinaBelovawe think we may miss some pool / whatever configuration parameter15:59
DinaBelovaas we did not expect degradations to start that early (200 pods/node density)15:59
DinaBelovaso that's still in progress15:59
DinaBelovaalso right now we're still working on workloads testing16:00
DinaBelovaon 200 nodes16:00
DinaBelovawhen we're deploying heat stacks with various apps running on Vms and planning to run locust.io workloads against it16:00
DinaBelovastill on the deployment phase for now16:00
DinaBelovawe observed some strange issues with Heat support in the fuel-ccp - really bad performance16:01
DinaBelovaso we're debugging it right now to see what might be the reason for this issue16:01
DinaBelovaand I think that's pretty all from my side16:01
*** markvoelker_ has joined #openstack-performance16:01
DinaBelovaanything else to cover in test plans / test results topic?16:02
DinaBelovait looks like we may proceed to the Open Discussions16:02
DinaBelova#topic Open Discussion16:02
DinaBelovavbala tovin07_ I have an idea to finish the work on https://review.openstack.org/#/c/407967/ patch16:02
DinaBelovaand cut new osprofiler release16:03
*** markvoelker has quit IRC16:03
akrzosAny ptg updates?16:03
vbalavmware ci posted the result on that patch16:03
DinaBelovavbala tovin07 are you ok with it?16:03
tovin07_Yes, it’s from vbala16:03
vbalai'm ok with it16:03
tovin07_I think it’s ok16:03
DinaBelovaack, thanks :)16:04
DinaBelovaakrzos well :) from Mirantis side me and andreykurilin still coming :)16:04
andreykurilinhi hi16:04
DinaBelovaakrzos were you able to discuss it within your team?16:04
*** markvoelker has joined #openstack-performance16:04
DinaBelovarcherrueau the same question to you sir :) any updates on PTG side?16:04
akrzoswe are still looking into budget, but in an ideal world, we would have myself, rook, sai and justin on our team come16:04
DinaBelovaakrzos yay :) I hope this will happen :)16:05
akrzosand each have a performance topic we could cover/discuss16:05
rcherrueauno not right now16:05
DinaBelovaakrzos I think we may start preparing agenda16:05
akrzosso i was wondering if we would put together a schedule/agenda16:05
DinaBelovalemme create an etherpad for those purposes16:05
akrzosperfect16:05
tovin07_+116:05
DinaBelova#action DinaBelova create an etherpad for PTG  agenda collection16:06
DinaBelovaack, cool16:06
rcherrueauI have to discuss that with ad_rien16:06
DinaBelovarcherrueau sure16:06
DinaBelovaplease take your time16:06
DinaBelovaakrzos as said, I plan to focus on test ideas / tools roadmaps / etc.16:06
DinaBelovaok, one more thing to cover16:07
DinaBelovathere is  holiday season close to us16:07
*** markvoelker_ has quit IRC16:07
akrzosDinaBelova: got it16:07
DinaBelovaI wanted to check who's going to be available and when :)16:07
akrzoswe are out all next week, back january 3rd16:08
DinaBelovaI have a PTO for Dec 27 - Dec 3016:08
DinaBelovaok, so it looks like it makes sense to move our next meeting to Jan16:08
DinaBelovarcherrueau and you folks?16:08
rcherrueauMe also, I will be out next week. I don't know for msimonin16:08
DinaBelovaare you ok to meet on Jan 3rd?16:08
DinaBelovaack, let's agree on next meeting on Jan 3rd, already in the new year :)16:09
rcherrueauOK great16:09
DinaBelova#info next meeting to be on Jan 3rd, usual time16:09
tovin07_got it16:09
akrzosGreat Thanks!16:09
DinaBelovaand I think that's all from my side16:10
DinaBelovaanything else to cover?16:10
DinaBelovatovin07_ akrzos you're welcome :)16:10
DinaBelovaok, thank you folks! see you next year :D16:10
DinaBelovabye!16:10
tovin07_Bye16:10
DinaBelova#endmeeting16:10
vbalaBye16:10
openstackMeeting ended Tue Dec 20 16:10:45 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:10
rcherrueaubye!16:10
openstackMinutes:        http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-12-20-15.30.html16:10
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-12-20-15.30.txt16:10
openstackLog:            http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-12-20-15.30.log.html16:10
akrzosHappy Holidays all!16:11
*** rcherrueau has quit IRC16:14
*** vbala has quit IRC16:17
*** tovin07_ has left #openstack-performance16:33
*** yujunz has quit IRC16:35
*** catintheroof has joined #openstack-performance16:39
*** catinthe_ has quit IRC16:42
*** harlowja has quit IRC16:42
* rook just saw the pings16:44
rooksorry16:44
rookin other meetings16:44
rookDinaBelova: it would be good to get eyes  on https://review.openstack.org/#/c/412554/16:44
*** zz_dimtruck is now known as dimtruck16:56
*** dimtruck is now known as zz_dimtruck17:06
*** msimonin has joined #openstack-performance17:17
*** msimonin has quit IRC17:35
*** harlowja has joined #openstack-performance17:53
*** pcaruana has quit IRC17:59
*** catinthe_ has joined #openstack-performance18:03
*** catintheroof has quit IRC18:06
openstackgerritIgor Yozhikov proposed openstack/performance-docs: Test plan for k8s+OS+Cinder+Ceph  https://review.openstack.org/41193318:10
*** zz_dimtruck is now known as dimtruck18:13
DinaBelovarook ack18:23
DinaBelovaI've seen it already, did not have a chance to review yet18:23
rookDinaBelova: ack18:52
*** jkilpatr has quit IRC19:30
*** harlowja has quit IRC19:35
*** catintheroof has joined #openstack-performance19:37
*** jkilpatr has joined #openstack-performance19:39
*** catinthe_ has quit IRC19:40
*** catintheroof has quit IRC20:47
*** jkilpatr has quit IRC21:14
*** jkilpatr has joined #openstack-performance21:33
*** dimtruck is now known as zz_dimtruck23:52
« Monday, 2016-12-19 Index Wednesday, 2016-12-21 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!