Tuesday, 2016-09-20

*** itsuugo has quit IRC00:00
*** itsuugo has joined #openstack-performance00:01
*** itsuugo has quit IRC00:06
*** itsuugo has joined #openstack-performance00:07
*** itsuugo has quit IRC00:14
*** itsuugo has joined #openstack-performance00:15
*** markvoelker has joined #openstack-performance00:29
*** itsuugo has quit IRC00:49
*** itsuugo has joined #openstack-performance00:50
*** itsuugo has quit IRC01:02
*** itsuugo has joined #openstack-performance01:03
*** itsuugo has quit IRC01:13
*** itsuugo has joined #openstack-performance01:14
*** itsuugo has quit IRC01:19
*** itsuugo has joined #openstack-performance01:20
*** haukebruno has quit IRC01:24
*** haukebruno has joined #openstack-performance01:26
*** zz_dimtruck is now known as dimtruck01:30
*** itsuugo has quit IRC01:33
*** itsuugo has joined #openstack-performance01:34
*** dimtruck is now known as zz_dimtruck01:40
*** itsuugo has quit IRC01:42
*** itsuugo has joined #openstack-performance01:43
*** itsuugo has quit IRC01:48
*** itsuugo has joined #openstack-performance01:48
*** itsuugo has quit IRC01:54
*** itsuugo has joined #openstack-performance01:55
*** markvoelker has quit IRC02:02
*** markvoelker has joined #openstack-performance02:02
*** itsuugo has quit IRC02:05
*** itsuugo has joined #openstack-performance02:06
*** itsuugo has quit IRC02:12
*** itsuugo has joined #openstack-performance02:13
*** itsuugo has quit IRC02:18
*** itsuugo has joined #openstack-performance02:20
*** zz_dimtruck is now known as dimtruck02:24
*** itsuugo has quit IRC02:24
*** itsuugo has joined #openstack-performance02:25
*** itsuugo has quit IRC02:38
*** itsuugo has joined #openstack-performance02:41
*** itsuugo has quit IRC02:52
*** itsuugo has joined #openstack-performance02:53
*** markvoelker has quit IRC03:02
*** markvoelker has joined #openstack-performance03:02
*** itsuugo has quit IRC03:07
*** itsuugo has joined #openstack-performance03:09
*** dimtruck is now known as zz_dimtruck03:47
*** itsuugo has quit IRC04:09
*** itsuugo has joined #openstack-performance04:11
*** itsuugo has quit IRC04:19
*** itsuugo has joined #openstack-performance04:20
*** itsuugo has quit IRC04:25
*** itsuugo has joined #openstack-performance04:27
*** itsuugo has quit IRC04:32
*** itsuugo has joined #openstack-performance04:33
*** itsuugo has quit IRC04:38
*** itsuugo has joined #openstack-performance04:39
*** markvoelker has quit IRC04:46
*** itsuugo has quit IRC05:03
*** itsuugo has joined #openstack-performance05:05
*** itsuugo has quit IRC05:14
*** itsuugo has joined #openstack-performance05:15
*** itsuugo has quit IRC05:20
*** itsuugo has joined #openstack-performance05:22
*** markvoelker has joined #openstack-performance05:47
*** itsuugo has quit IRC05:48
*** itsuugo has joined #openstack-performance05:49
*** msimonin has joined #openstack-performance05:51
*** markvoelker has quit IRC05:52
*** itsuugo has quit IRC05:54
*** itsuugo has joined #openstack-performance05:55
*** msimonin has quit IRC05:56
*** itsuugo has quit IRC06:02
*** itsuugo has joined #openstack-performance06:03
*** itsuugo has quit IRC06:11
*** itsuugo has joined #openstack-performance06:12
*** itsuugo has quit IRC06:17
*** itsuugo has joined #openstack-performance06:19
openstackgerritavnish proposed openstack/performance-docs: modify the home-page info with the developer documentation  https://review.openstack.org/37292306:23
*** itsuugo has quit IRC06:31
*** itsuugo has joined #openstack-performance06:32
*** itsuugo has quit IRC06:44
*** itsuugo has joined #openstack-performance06:45
*** markvoelker has joined #openstack-performance06:48
*** itsuugo has quit IRC06:50
*** itsuugo has joined #openstack-performance06:50
*** pcaruana has joined #openstack-performance06:50
*** markvoelker has quit IRC06:52
*** itsuugo has quit IRC06:59
*** itsuugo has joined #openstack-performance07:00
*** ad_rien_ has joined #openstack-performance07:10
*** itsuugo has quit IRC07:11
*** itsuugo has joined #openstack-performance07:12
*** msimonin has joined #openstack-performance07:13
*** ad_rien_ has quit IRC07:14
*** ad_rien_ has joined #openstack-performance07:16
*** itsuugo has quit IRC07:24
*** msimonin has quit IRC07:25
*** itsuugo has joined #openstack-performance07:25
*** msimonin has joined #openstack-performance07:27
*** itsuugo has quit IRC07:30
*** itsuugo has joined #openstack-performance07:31
*** markvoelker has joined #openstack-performance07:49
*** markvoelker has quit IRC07:53
*** itsuugo has quit IRC07:54
*** itsuugo has joined #openstack-performance07:55
*** itsuugo has quit IRC08:02
*** itsuugo has joined #openstack-performance08:03
*** itsuugo has quit IRC08:36
*** itsuugo has joined #openstack-performance08:39
*** markvoelker has joined #openstack-performance08:50
*** markvoelker has quit IRC08:54
*** lezbar has joined #openstack-performance09:01
*** itsuugo has quit IRC09:10
*** itsuugo has joined #openstack-performance09:11
*** itsuugo has quit IRC09:24
*** itsuugo has joined #openstack-performance09:26
*** itsuugo has quit IRC09:41
*** itsuugo has joined #openstack-performance09:42
*** ad_rien_ has quit IRC09:50
*** itsuugo has quit IRC10:00
*** itsuugo has joined #openstack-performance10:01
*** itsuugo has quit IRC10:47
*** itsuugo has joined #openstack-performance10:48
*** haukebruno has quit IRC10:49
*** ad_rien_ has joined #openstack-performance10:50
*** haukebruno has joined #openstack-performance10:50
*** ad_rien_ has quit IRC10:51
*** itsuugo has quit IRC10:58
*** itsuugo has joined #openstack-performance11:01
*** ad_rien_ has joined #openstack-performance11:02
*** ad_rien_ has quit IRC11:02
*** ad_rien_ has joined #openstack-performance11:03
*** ad_rien_ has quit IRC11:03
*** ad_rien_ has joined #openstack-performance11:03
*** ad_rien_ has quit IRC11:03
*** ad_rien_ has joined #openstack-performance11:05
*** ad_rien_ has quit IRC11:05
*** itsuugo has quit IRC11:18
*** itsuugo has joined #openstack-performance11:19
*** bvanhav has joined #openstack-performance11:23
*** itsuugo has quit IRC11:37
*** itsuugo has joined #openstack-performance11:38
*** Menthos has joined #openstack-performance11:39
*** itsuugo has quit IRC11:54
*** itsuugo has joined #openstack-performance11:56
*** itsuugo has quit IRC12:06
*** rfolco has joined #openstack-performance12:07
*** itsuugo has joined #openstack-performance12:07
*** ad_rien_ has joined #openstack-performance12:18
*** ad_rien_ has quit IRC12:21
*** markvoelker has joined #openstack-performance12:24
*** markvoelker has quit IRC14:02
*** markvoelker has joined #openstack-performance14:09
*** Guest14517 is now known as med_14:26
*** med_ has quit IRC14:26
*** med_ has joined #openstack-performance14:26
*** zz_dimtruck is now known as dimtruck14:28
*** Menthos has quit IRC15:13
*** msimonin has quit IRC15:27
*** ad_rien_ has joined #openstack-performance15:27
*** rohanion has joined #openstack-performance15:27
DinaBelova#startmeeting Performance Team15:30
openstackMeeting started Tue Sep 20 15:30:03 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.15:30
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:30
openstackThe meeting name has been set to 'performance_team'15:30
DinaBelovaok, so hello everyone!15:30
DinaBelovalet's see who's around :)15:30
ad_rien_o/15:30
DinaBelovaad_rien_ o/15:30
bvanhavhello15:30
DinaBelovarohanion, bvanhav o/15:30
ad_rien_but I will be the only one from Inria today (rcherrueau is finalizing his Phd thesis and msimonin had to go)15:31
DinaBelovaad_rien_ ack15:31
rohanionheeeey15:31
rohaniono/15:31
DinaBelovafrom my side I need to apologize as well: Mirantis folks are having big all hands meeting right now, so not so much tracktion from our side as well15:32
DinaBelovatraction *15:32
DinaBelovaok, so let's get started15:32
DinaBelova#startmeeting Performance Team15:32
openstackDinaBelova: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.15:32
DinaBelovaoh sorry :)15:32
DinaBelova#topic Action Items15:32
DinaBelovawrong command :)15:33
DinaBelovaso geragding action items - we had only one last time15:33
DinaBelovaad_rien_ it was about adding monitoring info to your etherpad15:33
DinaBelovaad_rien_ I see some details were added - cadvisor/influx/grafana15:33
ad_rien_yes but this is internal information15:34
ad_rien_From our side, we do not have much information to share today15:34
DinaBelovaad_rien_ ack15:34
DinaBelova#topic Current progress on the planned tests15:34
ad_rien_we are working on analyzing the traces we collected for15:34
ad_rien_(DinaBelova sorry you were too fast ;))15:35
ad_rien_the IDLE deployments15:35
ad_rien_and are still fighting issues with the 1000 fake drivers deployment15:35
DinaBelovaad_rien_ well, the test plans section is better for what you're discussing, so I switched to it15:35
DinaBelova:)15:35
ad_rien_running rally benchmark suite15:35
ad_rien_:-)15:36
ad_rien_We arranged a meeting on thursday with msimonin to prepare content for the presentation (and before our friday meeting)15:36
DinaBelovaad_rien_ cool! I think that I'll fill my part as well15:37
ad_rien_rcherreau developed a basic script to be able to see the influxdb informations we collected (i.e. this enables us to make postmorten analysis in a friendly way)15:37
DinaBelovaad_rien_ as for Alex part, he's helping one of our customers right now, so not sure if he'll be able to finish everything15:37
ad_rien_DinaBelova:  ack15:37
saihello, kinda late :)15:37
DinaBelovasai o/15:38
DinaBelovaok, so what can I share from our side15:38
DinaBelovalots of work in progress regarding preparing test plans and test results15:38
DinaBelovathat were performed during the q315:38
DinaBelova+ we're still fighting with OpenStack on top of k8s15:38
DinaBelovaright now we're having 350 nodes baremetal k8s installation15:39
DinaBelovawith Calico overlay15:39
DinaBelova+ OpenStack installed on top of this k8s cluster15:39
openstackgerritMerged openstack/performance-docs: modify the home-page info with the developer documentation  https://review.openstack.org/37292315:39
DinaBelovawe planned to run baseline tests against this cloud, but constantly observing the issues with k8s / docker15:39
DinaBelovadocker is not that scalable when we're talking about huge load running at once15:40
DinaBelova:)15:40
ad_rien_:-)15:40
DinaBelovaas for 1000 nodes experiment, we've decided to give a first try to non-containerized deployment15:40
DinaBelovaso we're having 1000 nodes being under installation with fuel + some dirty hacks15:41
DinaBelovato make it work smoothly15:41
ad_rien_it would be great if you can share the issues you faced15:41
DinaBelovaad_rien_ sure, first set of issues was related to installer itself15:41
DinaBelovawe used Kargo for this purpose (due to the right topology these folks are proposing)15:41
DinaBelovalemme share the links15:42
DinaBelova#link https://github.com/kubespray/kargo15:42
DinaBelovaso that's an installer15:42
DinaBelovawe filed two original issues15:43
DinaBelova#link https://github.com/kubespray/kargo/issues/47815:43
DinaBelova#link https://github.com/kubespray/kargo/issues/47915:43
DinaBelovathat later leaded to several more :)15:43
DinaBelovahttps://github.com/kubespray/kargo/issues/48715:43
DinaBelovahttps://github.com/kubespray/kargo/issues/48515:43
DinaBelovahttps://github.com/kubespray/kargo/issues/48015:43
DinaBelovaso the installation part is now ok, and we have observed several issues with the installed cluster15:43
DinaBelovathe most annoying is the following:15:44
DinaBelovacluster seem to degrade with time15:44
ad_rien_what do you mean by cluster ''degrades''  ?15:44
DinaBelovaOpenStack VMs were stuck in scheduling state - with more probability of this the more time k8s cluster is existing15:45
DinaBelovaso OpenStack was reinstalled several times with no issues, but for some reasons OpenStack services running as k8s apps began not to reach each other15:45
ad_rien_ok …. maybe some issues we observed previously with kolla (docker) but we did succeed to reproduce them on our side :( …. did you find a way to reproduce such issues ?15:45
ad_rien_s/we did succeed/did not succeed15:46
DinaBelovaad_rien_ only running OpenStack... one of our k8s engineers is suspecting the Docker in this behaviour15:46
DinaBelovaunder research right now15:46
DinaBelovabut nothing specific today, sorry15:46
ad_rien_the same on our side....15:46
DinaBelovaad_rien_ ok, so this seems to be common15:46
DinaBelovaI'll share details once (and I hope I will!) I'll have them15:47
ad_rien_thanks15:47
DinaBelovaso that's all from our side regarding the plans15:47
DinaBelovadoes anyone has anything else to share in this section?15:47
DinaBelovait looks like we may jump to the osprofiler15:48
DinaBelova#topic OSProfiler weekly update15:48
DinaBelovarohanion the floor is your15:48
DinaBelova:)15:48
rohanionhey guys15:48
rohanionso what we have done for the last week15:48
rohanion1. neutron patch is still waiting for CR15:49
rohanion2. cinder patch is documented and tested15:49
rohanion3. glance patch is finished, verified, waiting to merge15:49
DinaBelovarohanion do you need any help pinging the core reviewrs?15:50
rohanion4. sahara - still nothing, no updates15:50
rohanionYES PLEASE15:50
DinaBelovarohanion ack15:50
rohanionbecause it's a challenge finding who is responsible enough to merge the changes15:50
DinaBelovarohanion I'll make sure core reviewers are aware about these patches and ask for the help15:50
DinaBelovarohanion ack :)15:51
rohanion5. elasticsearch driver - reviewed, ready to merge, harlowja please look at it15:51
sairohanion: mind if i ask u what the elastic search dirver is abt15:51
saior if u can point me to some docs15:51
rohanionsure, I'll post a link right here15:52
rohanionhttps://review.openstack.org/#/c/340936/15:52
rohanionthis driver is basically another way of storing internal osprofiler data15:52
saigotcha15:52
saithanks15:52
DinaBelovarohanion anything else?15:53
rohanion6. redis driver - I reviewed it, I didn't like it, the setup will go nuts if the database server goes down15:53
rohanionI asked Ben if he could add a subclass that will use sentinel instead of a single redis instance15:53
rohanionno response since thursday15:54
DinaBelovabvanhav ^^15:54
bvanhavI'm working on making the suggested change15:54
DinaBelovabvanhav can you please take a look?15:54
DinaBelovaack!15:54
DinaBelovabvanhav good to know, thanks you :)15:54
rohanionthank you Ben!15:54
rookrohanion what about nova -- sorry if I missed it15:55
rohanionoh you mean Dina's patches?15:55
rookosprofiler w/ nova15:55
DinaBelovayeah, rohanion was going to continue my patches15:55
rohanionI have not started looking at them yet15:56
rohanionbut thanks for reminding me that, I'll start doing that tomorrow15:56
DinaBelovaas I am simply drown in the q3 deliverables we (Mirantis Scale RnD team) need to achieve15:56
DinaBelovaso thanks rohanion for grabbing a flag here15:56
rohanionhope to finish work on some of them till the next meeting15:56
DinaBelovarohanion ack15:57
DinaBelovaok, so it looks like we may proceed15:57
DinaBelova#topic Open Discussion15:58
DinaBelovaso I have some small summit news :)15:58
DinaBelova#info I was told we're goingto have separated time slot on summit for performance team needs15:58
saiDinaBelova: yeh wanted to ask abt that, are the perf guys meeting at summit?15:58
DinaBelovaby summit authorities15:58
DinaBelovasai yes we are :)15:58
saiwe are goin to have some presence from redhat perf, would be great to sync15:59
DinaBelovaand now (after some conversations) we'll have a separated time slot15:59
DinaBelovasai sure15:59
DinaBelova#info now exact time not available so far, most probably Thursday morning15:59
DinaBelovawill keep you folks updated15:59
saiawesome15:59
saiDinaBelova: so wrt DVR work seeing issues when trying to ping VM with floating ip when router is DVR16:00
DinaBelovasai ;)16:00
*** victorhe has joined #openstack-performance16:00
saisome cases takes 200 seconds16:00
DinaBelovasai ack16:00
saifiled https://bugs.launchpad.net/neutron/+bug/162533316:00
openstackLaunchpad bug 1625333 in neutron "Booting VM with a Floating IP and pinging it via that takes a long time with errors in L3-Agent logs when using DVR" [Undecided,New]16:00
DinaBelovaanything to share?16:00
DinaBelovaok, I see16:00
DinaBelova#link https://bugs.launchpad.net/neutron/+bug/162533316:00
DinaBelovasai lemme share this bug with our neutron team16:01
sairook: and i looked at it, seems to be an issue with floatingip assignment at l3-agent16:01
saisure16:01
DinaBelovasai cool, thank you for the update16:02
saialso heloing some folks at other places get started with shaker16:02
saihelping*16:02
saiseems to be good amount of interest16:02
DinaBelovasai our neutron team did not jump into the issue you filed during their experiments16:03
DinaBelovait's interesting16:03
DinaBelovaas they had plenty of tests done on various topologies16:03
DinaBelovaincluding DVR-based16:03
saiDinaBelova: I ahve linked the rally-plugin we used and rally results too16:03
DinaBelovasai ack, I'll take a look16:03
saiDinaBelova: our neutron team is also looking at it16:03
DinaBelova:) good to have neutron teams :)16:03
DinaBelova:D16:03
DinaBelovaok, anything else to share?16:04
sainothing from my end16:04
DinaBelovaad_rien_ rook ?16:04
ad_rien_nothing thanks16:04
rooknada16:04
DinaBelovaok, so thanks everyone for participating, stay tuned!16:04
DinaBelova#endmeeting16:05
openstackMeeting ended Tue Sep 20 16:05:02 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:05
openstackMinutes:        http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-09-20-15.30.html16:05
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-09-20-15.30.txt16:05
openstackLog:            http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-09-20-15.30.log.html16:05
bvanhavrohanian: didn't get a chance to mention earlier, I've worked some with the nova patch. So if you have questions about anything I did, feel free to ask me... if you just want to say, "you're an idiot for doing ___" that's fine too...16:06
*** Menthos has joined #openstack-performance16:24
*** Menthos has quit IRC16:24
*** JorgeCardoso has joined #openstack-performance16:25
*** rohanion has quit IRC16:31
*** ad_rien_ has quit IRC16:37
*** ad_rien_ has joined #openstack-performance16:44
*** JorgeCardoso has quit IRC16:55
*** itsuugo has quit IRC17:02
*** itsuugo has joined #openstack-performance17:03
*** sai is now known as sai|mtg17:08
*** pcaruana has quit IRC17:14
*** itsuugo has quit IRC17:36
*** itsuugo has joined #openstack-performance17:38
*** itsuugo has quit IRC17:45
*** itsuugo has joined #openstack-performance17:46
*** sai|mtg is now known as sai17:57
*** itsuugo has quit IRC18:49
*** itsuugo has joined #openstack-performance18:51
*** markvoelker has quit IRC19:03
*** markvoelker has joined #openstack-performance19:07
*** itsuugo has quit IRC19:12
*** itsuugo has joined #openstack-performance19:14
*** itsuugo has quit IRC19:28
*** itsuugo has joined #openstack-performance19:29
*** bvanhav has quit IRC19:33
*** itsuugo has quit IRC19:36
*** itsuugo has joined #openstack-performance19:37
*** itsuugo has quit IRC19:44
*** itsuugo has joined #openstack-performance19:45
*** itsuugo has quit IRC20:01
*** itsuugo has joined #openstack-performance20:03
*** beisner is now known as beisner-food20:16
*** beisner-food is now known as beisner20:17
*** victorhe has quit IRC21:40
DinaBelovafolks, here is the team slot: https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15504/openstack-performance-team-what-has-been-done-during-newton-cycle-and-ocata-planning22:17
*** itsuugo has quit IRC22:20
*** itsuugo has joined #openstack-performance22:21
*** itsuugo has quit IRC22:49
*** itsuugo has joined #openstack-performance22:50
*** markvoelker has quit IRC23:03
*** markvoelker has joined #openstack-performance23:03
*** itsuugo has quit IRC23:18
*** itsuugo has joined #openstack-performance23:18
*** itsuugo has quit IRC23:23
*** itsuugo has joined #openstack-performance23:24
openstackgerritDina Belova proposed openstack/performance-docs: Expand Mitaka results for L3 HA  https://review.openstack.org/37085923:25
openstackgerritMerged openstack/performance-docs: Add missed files for control plane 400 nodes report  https://review.openstack.org/37148923:26
*** Alexey_Abashkin_ has joined #openstack-performance23:45
*** Alexey_Abashkin has quit IRC23:46
*** itsuugo has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!