Tuesday, 2017-01-03

*** rfolco has quit IRC00:42
*** tovin07 has joined #openstack-performance00:54
*** tovin07 has quit IRC00:57
*** markvoelker has joined #openstack-performance01:05
*** markvoelker has quit IRC01:09
*** tovin07 has joined #openstack-performance01:16
*** zz_dimtruck is now known as dimtruck01:42
*** yujunz has joined #openstack-performance01:46
*** markvoelker has joined #openstack-performance02:05
*** markvoelker has quit IRC02:10
*** dimtruck is now known as zz_dimtruck02:53
*** markvoelker has joined #openstack-performance03:06
*** markvoelker has quit IRC03:10
*** tovin07 has quit IRC03:34
*** tovin07 has joined #openstack-performance03:34
*** yujunz has quit IRC04:03
*** yujunz has joined #openstack-performance04:03
*** masber has joined #openstack-performance04:27
*** jkilpatr has quit IRC05:45
*** jkilpatr has joined #openstack-performance05:49
*** msimonin has joined #openstack-performance06:41
*** msimonin1 has joined #openstack-performance06:44
*** msimonin has quit IRC06:44
*** msimonin1 has quit IRC07:10
*** msimonin has joined #openstack-performance07:10
*** msimonin has quit IRC07:14
*** yujunz has quit IRC07:21
*** yujunz has joined #openstack-performance07:22
*** pcaruana has joined #openstack-performance07:41
*** brault has joined #openstack-performance08:17
*** msimonin has joined #openstack-performance08:26
*** yujunz is now known as yujunz[away]08:33
*** yujunz[away] is now known as yujunz08:36
*** yujunz has quit IRC08:46
*** yujunz has joined #openstack-performance08:47
*** msimonin has quit IRC09:24
*** msimonin has joined #openstack-performance09:29
*** yujunz has quit IRC09:38
*** yujunz has joined #openstack-performance09:39
*** tovin07 has quit IRC09:59
*** markvoelker has joined #openstack-performance10:10
*** markvoelker has quit IRC10:15
*** yujunz-zte has joined #openstack-performance10:16
*** rmart04 has joined #openstack-performance10:17
*** hieulq has quit IRC10:17
*** yujunz has quit IRC10:18
*** hieulq has joined #openstack-performance10:18
*** yujunz-zte is now known as yujunz[away]10:24
*** yujunz[away] is now known as yujunz-zte10:25
*** yujunz-zte has quit IRC10:25
*** klindgren_ has joined #openstack-performance10:29
*** klindgren has quit IRC10:32
*** yujunz has joined #openstack-performance10:56
*** yujunz has quit IRC11:00
*** yujunz has joined #openstack-performance11:00
*** msimonin has quit IRC11:01
*** msimonin has joined #openstack-performance11:03
*** msimonin has quit IRC11:03
*** yujunz has quit IRC11:06
*** msimonin has joined #openstack-performance11:07
*** markvoelker has joined #openstack-performance11:11
*** markvoelker has quit IRC11:16
*** yujunz has joined #openstack-performance11:21
*** yujunz is now known as yujunz[away]11:25
*** yujunz[away] is now known as yujunz11:34
*** jkilpatr has quit IRC11:39
*** rfolco has joined #openstack-performance11:40
*** jkilpatr has joined #openstack-performance12:03
*** markvoelker has joined #openstack-performance12:12
*** yujunz has quit IRC12:12
*** markvoelker has quit IRC12:16
*** fyxim has quit IRC12:19
*** serverascode has quit IRC12:19
*** zz_dimtruck has quit IRC12:19
*** dimtruckl has joined #openstack-performance12:19
*** dimtruckl is now known as dimtruck12:19
*** bvanhav has joined #openstack-performance12:21
*** fyxim has joined #openstack-performance12:22
*** serverascode has joined #openstack-performance12:24
*** msimonin has quit IRC12:48
*** msimonin has joined #openstack-performance12:50
*** msimonin has quit IRC12:50
*** msimonin has joined #openstack-performance12:52
*** markvoelker has joined #openstack-performance13:12
*** markvoelker has quit IRC13:17
*** yujunz has joined #openstack-performance13:20
*** yujunz is now known as yujunz[away]13:31
*** yujunz[away] is now known as yujunz13:32
*** yee37913 has joined #openstack-performance13:46
*** akrzos has quit IRC13:46
*** yee379 has quit IRC13:46
*** akrzos has joined #openstack-performance13:46
*** sai has quit IRC13:47
*** sai has joined #openstack-performance13:48
*** furlongm_ has quit IRC13:48
*** furlongm has joined #openstack-performance13:48
*** bvanhav_ has joined #openstack-performance14:02
*** bvanhav has quit IRC14:03
*** yujunz is now known as yujunz[away]14:11
*** markvoelker has joined #openstack-performance14:13
*** yujunz[away] has quit IRC14:15
*** markvoelker has quit IRC14:18
*** yujunz has joined #openstack-performance14:24
*** yujunz is now known as yujunz[away]14:28
*** yujunz[away] is now known as yujunz14:30
*** msimonin has quit IRC14:31
*** bvanhav_ is now known as bvanhav14:33
*** yujunz is now known as yujunz[away]14:34
*** yujunz[away] is now known as yujunz14:34
*** msimonin has joined #openstack-performance14:36
*** markvoelker has joined #openstack-performance14:38
*** yujunz has quit IRC14:48
*** rmart04_ has joined #openstack-performance15:06
*** rmart04 has quit IRC15:06
*** rmart04_ is now known as rmart0415:06
*** rmart04 has quit IRC15:12
*** rmart04 has joined #openstack-performance15:17
*** med_` is now known as medberry15:28
*** medberry is now known as med_15:28
*** med_ is now known as Guest4623215:29
*** Guest46232 is now known as medberry15:29
*** medberry has joined #openstack-performance15:29
*** tovin07 has joined #openstack-performance15:29
DinaBelova#startmeeting Performance Team15:30
openstackMeeting started Tue Jan  3 15:30:01 2017 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.15:30
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:30
openstackThe meeting name has been set to 'performance_team'15:30
DinaBelovahello folks!15:30
DinaBelovalet's see who has survived after holidays :D15:30
tovin07hello :D15:30
DinaBelovaakrzos tovin07 msimonin o/15:30
*** spn has joined #openstack-performance15:30
akrzosDinaBelova: hello15:31
msimoninHello15:31
spno/15:31
DinaBelovawow, we have quorum :D15:31
spnHappy new year to all :)15:31
*** rcherrueau has joined #openstack-performance15:32
DinaBelovayeah, first of all Happy New Year to all of you folks :) I hope holidays were as merry as they should :)15:32
DinaBelovaand it looks like we an start our first 2017 brand new meeting :)15:32
DinaBelova#topic Action Items15:32
DinaBelovalast time we only had one action item on myself15:33
DinaBelovaregarding creating an etherpad to start collecting PGT agenda15:33
DinaBelovahere it is15:33
DinaBelova#link https://etherpad.openstack.org/p/ptg-performance-team15:33
DinaBelova@akrzos @msimonin ^^15:33
DinaBelovalet's have two weeks or so to collect ideas15:34
DinaBelova#info let's use https://etherpad.openstack.org/p/ptg-performance-team to collect ideas to discuss on PTG15:34
spnyup15:34
DinaBelova#info let's have two weeks timeframe to prepare it15:35
tovin07(y)15:35
akrzos*nod*15:35
DinaBelova@msimonin do you have any news about Inria participance? are you guys going to attend in person?15:35
msimoninHum I don't think we'll go in person15:36
DinaBelovamsimonin ok, so we need to plan somehow your remote participance15:36
msimoninyes if it's possible15:37
msimonintechnically speaking this would be great15:37
spnis there a list to tell who all are attending in person?15:37
akrzoscan we use that etherpad to track15:37
DinaBelovaspn I and andreykurilin (Rally PTL)15:37
DinaBelovaakrzos  sure15:37
msimonin+1 to track on the etherpad15:37
spnI am attending in person too15:37
spn+1 to etherpad. I will update it15:38
DinaBelovaok, cool15:39
DinaBelovaI've added who's going to attend from mirantis side15:39
akrzosWe have budget for it. just not sure if who is going just yet15:40
DinaBelovaakrzos cool15:40
DinaBelovaplease update the etherpad once you'll know for sure15:40
DinaBelova:)15:40
akrzosI'll try and confirm before End of today15:40
DinaBelovaakrzos ack15:40
DinaBelovaso it looks like we may proceed to researches and updates15:41
DinaBelova#topic Current progress on the planned tests15:41
DinaBelovamsimonin any updates from you sir? (I do not think we all hav something sagnificant to talk about as it ere holidays, but still)15:42
msimoninYes :)15:42
msimoninHolliday was productive for other things than current actions :)15:42
DinaBelova** significant, sorry for the typo15:42
DinaBelova:D15:42
DinaBelovaok, I believe akrzos you have pretty the same situation :D15:42
DinaBelovafrom our side I have some items to share15:43
akrzospretty much just trying to get gnocchi on swift on the same hardware as ceph i did about two weeks ago15:43
akrzosthough already changes in gnocchi15:43
akrzos's defaults15:43
akrzos(again)15:43
DinaBelovaoh, I got it15:43
rookDinaBelova et all any more feedback on #link https://review.openstack.org/#/c/412554/15:44
DinaBelovaso waiting for the some updates next week I guess :)15:44
DinaBelovarook I see, not more reviews there15:45
DinaBelovalemme just merge it15:45
rookyup15:45
rooki want to post the results15:45
DinaBelovaas I feel pretty confident about it15:45
DinaBelovacool, lemme click merge button15:45
rookthx15:45
DinaBelovarook it should land in few moments15:45
DinaBelovafrom our side: several documents have been published on review: https://review.openstack.org/#/q/project:openstack/performance-docs+status:open15:46
DinaBelovaI guess one of the most interesting parts for you folks will be monitoring methodology15:46
DinaBelovaakrzos msimonin https://review.openstack.org/#/c/415563/15:46
rookoh fun, prometheus15:47
DinaBelovaplease review, we're going to continue this effort this year to improve it, so some feedback it welcome :D15:47
DinaBelovarook yeah :D15:47
msimoninDinaBelova: ack15:47
akrzosinteresting15:48
msimoninI see Heka for log collection15:48
DinaBelovamsimonin true15:48
msimoninKolla is dropping Heka probably in favour of fluentd15:48
tovin07Heka is now deprecated in some other projects as I see15:49
msimoninyes15:49
tovin07Yup, Kolla15:49
akrzosDinaBelova: I'll review and provide feedback though i don't have experience with promethous, definitely interested in it vs graphite or gnocchi or any other monitoring solution :)15:49
DinaBelovaakrzos ack, thank you sir :)15:49
*** rmart04 has quit IRC15:49
tovin07Kolla plans to move to Fluentd15:49
DinaBelovamsimonin tovin07 well, that's all matter of just what o choose :)15:49
DinaBelovalemme ask team go through this option as well :)15:49
DinaBelovathanks for letting know15:50
msimoninyes it was just  to give the information15:50
DinaBelova#action DinaBelova discuss Fluentd option for logs collecting in our monitoring solution15:50
DinaBelovathank you folks for the info :)15:50
openstackgerritMerged openstack/performance-docs: Performance of OpenStack API Workers test plan  https://review.openstack.org/41255415:51
rookinteresting...15:51
rookDinaBelova: do you guys have scale/perf data on these solutions?15:51
DinaBelovarook we have run this type of monitoring for 900 nodes environment15:52
rookhm ok cool, what are you collecting?15:52
rookall metrics? or a subset15:52
DinaBelovaall of them :)15:52
DinaBelovain that caae it was 13,5 million time series were collected15:52
rookper second?15:52
DinaBelovaand we needed to have two prometheus instances to survive this load15:53
rookhm ok15:53
rookcool!!15:53
DinaBelovarook not per second, but at all in Prometheus "list" of what needs to be collected15:53
DinaBelovaalthough prometheus is first of all monitoring and alerting tool15:53
*** rmart04 has joined #openstack-performance15:53
*** hieulq_ has joined #openstack-performance15:54
DinaBelovaand that means that there is high data retention being used by us15:54
akrzosso promethues is not just time-series data, but also collecting logs?15:54
akrzosor receiving logs from fluentd*15:54
DinaBelovaas if data was stored there for a long time, it became less responsive15:54
*** rmart04 has quit IRC15:54
DinaBelovaakrzos moniotirng only, we used elastic search for the logs in most common way. weka was used to parse some of the data from it and push as measurements to prometheus15:55
DinaBelovalike Kubernetes API response time15:55
DinaBelovathe only way you can gather this metric - calculate it from logs15:55
akrzosoh so some of the performance metrics from services is emitted only into the logs thus using the log collection component to pull the performance data and graph in time-series, thats very cool15:56
DinaBelovaakrzos indeed15:56
akrzosthought I'm an advocate for the performance data to be exposed outside logs (to begin with)15:56
akrzosthough*15:56
DinaBelovawell, I'll agree with you personally, but we were really interested in k8s API response time :D15:57
DinaBelovaso we had to find a way to collect it15:57
akrzosgotta do what you gotta do15:57
DinaBelovawe plan to continue this effort in new year - at very first we need to configure some external time series DB to collect all this data for longer time for post-experiments researches and calculations15:58
DinaBelovalike Cassandra or OpenTSDB15:58
DinaBelovaas right now we use pure promethues, and it's not really good in storing data for a ong time15:59
DinaBelovalong*15:59
akrzossame problem with graphite15:59
DinaBelovaakrzos yeah15:59
*** rmart04_ has joined #openstack-performance15:59
DinaBelovawe had issues like "oh, let's take a look on a data we had a month ago, oh sh*t no this data anymore"15:59
DinaBelovaso I really would like not having this issue in 2017 :D16:00
akrzosyeah or downsampling or what was done to "aggregation" of samples16:00
akrzossamples/measurements16:00
spnwhat is the backend for promethues?16:01
akrzosgrafana can even "change" what the measurements look likes based on the number of pixels you have available to display :O16:01
* akrzos assumes your using grafana for the visualization portion16:01
DinaBelovaspn prometheus has its own prometheus db (not sure what technology are they using under the hood), but if you want to have this data persistent - use some persistent DB prometheus can work with - it might be cassandra, opentsdb, influx16:02
DinaBelovaakrzos true16:02
spnDinaBelova: ok cool16:02
DinaBelovaso grafana is sometimes a challenge as well, that's true :D16:02
DinaBelovaanyway, so folks I'm kindly asking to review it16:03
spnakrzos: can we export the visualization portion of grafana so that it can be imported on a newer grafana elsewhere?16:03
akrzosspn: should be able to export the json and then push it into another grafana via the api16:04
akrzosi haven't tried the latest grafana to be honest16:04
DinaBelovamy team is going to be back around Jan 9th (Russian winder holidays are after the New year for about a week) so I really hope they'll have the feedback from you :)16:04
DinaBelovaok, so it looks like we may jump to open discussions16:05
DinaBelova#topic Open Discussion16:06
DinaBelovaanything else to share folks?16:06
spncan we discuss about spec in our face to face?16:06
spnI am using spec @ Dell for openstack16:06
DinaBelovaI believe we can16:06
DinaBelovaspn please propose this item to the list of what to discuss16:07
spnok sure16:07
DinaBelovaok, anything else folks?16:08
msimoninYep16:08
msimoninFrom Inria side16:08
DinaBelovasure, please go ahead :)16:09
msimoninAmong other things we'll use osprofiler16:09
msimoninmore precisely some interns will use it16:09
msimonin;)16:09
DinaBelovayay :)16:09
DinaBelovaI'm glad to read this16:09
tovin07That’s a good new16:10
msimoninI may be wrong but I thing there is still a review for its intergration in nova16:10
DinaBelovamsimonin that's right16:10
msimoninDinaBelova: ack16:10
DinaBelovaalthough right now it's tracked by our (Mirnatis) Nova team - they are really interesting in landing it16:10
msimoninok :)16:10
DinaBelovathis change is up to date in fact16:11
tovin07Not just Mirantis :))16:11
DinaBelovaand can be used16:11
DinaBelovatovin07 yes, you're right ;)16:11
msimoninok good to know16:11
DinaBelovamsimonin please keep me updated if any issues will occur, let's fix them :)16:12
msimoninThanks !16:12
*** rama_y has joined #openstack-performance16:12
*** rama_y_ has joined #openstack-performance16:12
*** rama_y_ has quit IRC16:12
DinaBelovamsimonin anything else to share?16:12
msimoninI think that's all for today16:13
DinaBelovayeah, thank you folks!16:13
DinaBelovahave a nice 2017 :)16:13
DinaBelova#endmeeting16:13
openstackMeeting ended Tue Jan  3 16:13:44 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:13
openstackMinutes:        http://eavesdrop.openstack.org/meetings/performance_team/2017/performance_team.2017-01-03-15.30.html16:13
akrzosThanks, you too!16:13
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/performance_team/2017/performance_team.2017-01-03-15.30.txt16:13
openstackLog:            http://eavesdrop.openstack.org/meetings/performance_team/2017/performance_team.2017-01-03-15.30.log.html16:13
msimoninyou too, thx16:13
tovin07Thanks! :D16:14
spnbye all16:14
*** rmart04_ has quit IRC16:23
*** bvanhav has quit IRC16:39
*** klindgren_ has quit IRC16:40
*** bvanhav has joined #openstack-performance16:40
*** tovin07 has quit IRC16:42
*** klindgren has joined #openstack-performance16:46
*** rama_y has quit IRC16:46
*** medberry is now known as med_17:14
*** msimonin has quit IRC17:35
*** rcherrueau has quit IRC17:37
*** bvanhav_ has joined #openstack-performance17:57
*** pcaruana has quit IRC17:58
*** bvanhav has quit IRC17:58
*** hieulq_ has quit IRC18:33
*** hieulq_ has joined #openstack-performance18:34
*** hieulq_ has quit IRC18:34
*** rama_y has joined #openstack-performance18:44
*** catintheroof has quit IRC19:33
*** catintheroof has joined #openstack-performance19:34
*** serverascode has quit IRC19:35
*** dimtruck has quit IRC19:35
*** dimtruck_ has joined #openstack-performance19:36
*** dimtruck_ is now known as dimtruck19:36
*** harlowja has joined #openstack-performance19:38
*** catintheroof has quit IRC19:38
*** rmart04 has joined #openstack-performance19:39
*** jkilpatr has quit IRC19:42
*** yee37913 has quit IRC19:48
*** yee379 has joined #openstack-performance19:49
*** rmart04 has quit IRC19:50
*** serverascode has joined #openstack-performance19:51
*** jkilpatr has joined #openstack-performance19:55
*** msimonin has joined #openstack-performance19:59
*** openstackgerrit has quit IRC20:03
*** dimtruck is now known as zz_dimtruck20:22
*** zz_dimtruck is now known as dimtruck20:22
*** dimtruck is now known as zz_dimtruck20:31
*** bvanhav_ has quit IRC20:39
*** zz_dimtruck is now known as dimtruck20:47
*** rfolco_ has joined #openstack-performance21:13
*** rfolco has quit IRC21:14
*** jkilpatr has quit IRC21:29
*** rfolco_ has quit IRC21:36
*** msimonin1 has joined #openstack-performance21:53
*** msimonin has quit IRC21:56
*** jkilpatr has joined #openstack-performance22:05
*** openstack has joined #openstack-performance23:15
*** msimonin1 has quit IRC23:41
*** msimonin has joined #openstack-performance23:41
*** msimonin has quit IRC23:41
*** msimonin has joined #openstack-performance23:41
*** msimonin has quit IRC23:42

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!