15:30:01 <DinaBelova> #startmeeting Performance Team
15:30:02 <openstack> Meeting started Tue Oct 11 15:30:01 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:05 <openstack> The meeting name has been set to 'performance_team'
15:30:12 <DinaBelova> hey folks :)
15:30:23 <rcherrueau> o/
15:30:30 <ilyashakhat_mobi> hi)
15:30:39 <DinaBelova> rcherrueau ilyashakhat_mobi msimonin o/
15:30:57 <DinaBelova> I want to say sorry again for 2 weeks without the meetings
15:31:26 <msimonin> o/
15:31:28 <DinaBelova> it were several time-consuming events that I was not able to control
15:31:55 <DinaBelova> but I'm back and I'm really glad to see all of you :)
15:32:17 <DinaBelova> msimonin is ad_rien going to join today?
15:32:42 <rcherrueau> DinaBelova: nop ad_rien is not comming today
15:32:53 <DinaBelova> rcherrueau ack, thank you sir
15:33:15 <DinaBelova> I think we may start
15:33:21 <DinaBelova> #topic Current progress on the planned tests
15:33:46 <DinaBelova> for the last 3 weeks we had lots of plans / results being pushed to the performance-docs
15:34:01 <DinaBelova> mostly that's a result of our Q3 activities :)
15:34:07 <DinaBelova> #link https://review.openstack.org/#/q/project:openstack/performance-docs+status:merged+after:2016-09-25
15:34:39 <DinaBelova> all these ^^ changes are merged already and presented on docs.openstack.org
15:35:06 <DinaBelova> rcherrueau msimonin please take a look, I believe you'll find something interesting for you
15:35:35 <msimonin> DinaBelova: ack, currently checking 1000 nodes :)
15:35:56 <DinaBelova> msimonin that was only baseline control plane
15:36:16 <DinaBelova> not some deep research, but we wanted to complete the picture we had for 200 and 400 nodes
15:36:41 <DinaBelova> + there are several changes on review
15:36:42 <DinaBelova> #link https://review.openstack.org/#/q/project:openstack/performance-docs+status:open+after:2016-10-07
15:37:19 <DinaBelova> I would say that there is *really* interesting research among them - https://review.openstack.org/#/c/361957/ - Neutron density test plan and results
15:37:35 <DinaBelova> this research was prepared and run by our Neutron team
15:38:07 <DinaBelova> thanks ilyashakhat_mobi for composing the test plan and results for performance-docs :)
15:39:04 <DinaBelova> msimonin rcherrueau this research (Neutron density) was run in terms of running as many VMs as possible on the environment and see what will fail and where keeping checking that dataplane is working ok
15:40:02 <DinaBelova> msimonin rcherrueau I know right now you're mostly focused on control plane, but this test plan has shown really cool results, so I'm highly recommending to go through it :)
15:40:28 <msimonin> DinaBelova: thanks !
15:41:16 * msimonin compiling doc …
15:41:32 <DinaBelova> also I have one more "favorite" change among the ones, that were merged
15:41:44 <DinaBelova> #link http://docs.openstack.org/developer/performance-docs/test_plans/reliability/version_2/plan.html
15:41:48 <ilyashakhat_mobi> DinaBelova: I'd also mention reliability reports - http://docs.openstack.org/developer/performance-docs/test_results/reliability/version_2/index.html#reports - something that we can be proud of :)
15:42:05 <DinaBelova> ilyashakhat_mobi ahaha, I was just mentioning them ^^
15:42:14 <DinaBelova> ilyashakhat_mobi yes, you're right
15:42:15 <ilyashakhat_mobi> yeah
15:43:23 <DinaBelova> so this test plan is version #2 of the reliability researches done inside Mirantis, thanks ilyashakhat_mobi for being able to collect so cool results
15:44:02 <DinaBelova> msimonin rcherrueau I believe you may be also interested in k8s / fuel-ccp related part
15:44:29 <DinaBelova> #link http://docs.openstack.org/developer/performance-docs/test_plans/fuel_ccp/plan.html
15:44:38 <DinaBelova> #link http://docs.openstack.org/developer/performance-docs/test_results/fuel_ccp/index.html#fuel-ccp-test-report
15:44:54 <msimonin> DinaBelova: so many things to read :)
15:44:55 <DinaBelova> #link http://docs.openstack.org/developer/performance-docs/test_results/container_cluster_systems/kubernetes/API_testing/index.html#results-of-measuring-of-api-performance-of-kubernetes
15:44:59 <DinaBelova> msimonin yeah :D
15:45:01 <DinaBelova> sorry :D
15:45:20 <msimonin> np :)
15:45:44 <msimonin> one quick question
15:45:54 <DinaBelova> sure
15:46:17 <msimonin> for  OpenStack control plane performance report for 1000 nodes, how many physical servers to host the virtual compute nodes were used ?
15:46:34 <msimonin> i guess it was 500 bu i can't find the information
15:46:58 <DinaBelova> physically it were 200 nodes in the lab, but controllers and storage nodes were physical, and computes were virtual
15:47:08 <DinaBelova> it's about 6 computes per node
15:47:17 <DinaBelova> msimonin hm, lemme check
15:47:23 <DinaBelova> I though this info is included
15:47:27 <ilyashakhat_mobi> the ratio was 7:1, so around 150 physical nodes for 1000 virtual computes + 3 hardware controllers + 20 hw ceph nodes
15:47:43 <DinaBelova> ilyashakhat_mobi oh, sorry, yes 7:1
15:48:06 <msimonin> ok thanks, how many cores the physical had ?
15:48:29 <msimonin> found : 24  ?
15:49:04 <DinaBelova> checking if the info published is true, it should be, yes
15:49:09 <msimonin> ok
15:49:54 <DinaBelova> yep, info presented should be ok
15:50:26 <DinaBelova> anyway, all other reports are interesting and cool too, I just don't want to list all of them and focused on my personal favorites
15:50:55 <DinaBelova> msimonin so any news from you sir?
15:51:29 <msimonin> So, these results could be included in the summit presentation ?
15:52:18 <DinaBelova> msimonin I think e can, but we did not have any monitoring being run for this specific test case
15:52:48 <DinaBelova> we just were running baseline control plane tests, so we can use this numbers, but in fact this data is not really representative
15:53:20 <DinaBelova> it's just a prove that control plane works ok on such scale, including real VMs running
15:54:28 <msimonin> yes even it's baseline control plane tests, it already proves that
15:54:42 <msimonin> it's worth mentionning :)
15:54:45 <DinaBelova> msimonin we can mention this on one slide I guess :)
15:54:48 <DinaBelova> ack
15:55:03 <msimonin> from our side
15:55:12 <DinaBelova> yes, please proceed :)
15:56:00 <msimonin> we ran seveal topologies / configurations to experiments 1000 computes nodes deployment (control place)
15:56:02 <msimonin> plane*
15:56:25 <DinaBelova> anything interesting you'd like to sharE?
15:56:48 <msimonin> we have tons of monitoring :s
15:56:53 <msimonin> tons of logs
15:56:54 <DinaBelova> :D indeed
15:57:00 <msimonin> but bascially it works :)
15:57:15 <DinaBelova> any suspicious trends? :)
15:57:19 <DinaBelova> or just works?
15:57:37 <msimonin> ok so we made some concurrency tests
15:57:44 <DinaBelova> (personally I prefer "just works", but there is always some moment to keep in mind) :D
15:58:25 <msimonin> with 1000 nodes and 500 nodes
15:59:19 <msimonin> we experiments differents concurrency parameteers to check if a 1000 nodes deployment could absorb the same "burst" as a 500
15:59:40 <msimonin> concurrency parameters  = concurrency in the rally benchmarks
15:59:55 <msimonin> and it doesn't
16:00:20 <DinaBelova> ok, some numbers? that were ok for 500 and not working for 1000?
16:01:00 <msimonin> like 50 was ok both but not 100
16:01:53 <DinaBelova> what was the rally scenario?
16:02:09 <msimonin> basically the first bottleneck was the scheduler
16:02:21 <msimonin> DinaBelova:  boot_and_delete / boot_and_list
16:02:30 <DinaBelova> I'm asking as for some operations keeping 100 rps is ok, and for some of them it's crazily huge load
16:02:32 <DinaBelova> ack
16:03:00 <DinaBelova> oh, it was constant rally runner?
16:03:03 <DinaBelova> or rps?
16:03:10 <msimonin> constant I guess
16:03:15 <DinaBelova> ack
16:03:22 <msimonin> we may use rps later
16:03:29 <DinaBelova> well, yeah, scheduler should be a blocker here
16:03:39 <DinaBelova> msimonin but rps should be really less :D
16:03:48 <msimonin> DinaBelova: yes sure
16:04:18 <DinaBelova> msimonin a tricky question: when are you going to publish your researches? :)
16:04:23 <DinaBelova> to the performance docs? :)
16:04:40 <msimonin> that's not tricky question, if everyone agrees we will
16:04:55 <msimonin> I mean from our side there's no issue at all at publishing them
16:05:11 <DinaBelova> msimonin as I said already you're more than welcome to do this :)
16:05:24 <DinaBelova> I'm asking mostly because there are 12 days left till summit, ideally it'll be good to have this data being at least published to review
16:05:29 <DinaBelova> before summit
16:05:42 <DinaBelova> as our presentation won't cover everything for sure
16:05:53 <DinaBelova> so full data needs to be presented somewhere
16:06:11 <msimonin> I agree
16:06:35 <msimonin> We can start with a first draft a a first experiment
16:06:42 <msimonin> of a
16:07:01 <DinaBelova> sure, please go ahead and publish the first version
16:08:13 <DinaBelova> msimonin anything else?
16:08:54 <msimonin> we met alwex for the summit presentation (but it's maybe out of the scope)
16:09:04 <DinaBelova> oh, I was disconnected for a moment
16:09:05 <DinaBelova> #topic Upcoming summit
16:09:06 <DinaBelova> it's in scope :)
16:09:08 <DinaBelova> but in this section :)
16:09:09 <DinaBelova> :D
16:09:15 <msimonin> ;)
16:09:39 <msimonin> any feedback from him ?
16:10:01 <DinaBelova> msimonin he agreed with your approach, working right now on adding the content :)
16:10:08 <DinaBelova> I'll do the same basically :)
16:10:26 <msimonin> Great !
16:10:29 <DinaBelova> msimonin rcherrueau fyi I'll be in EU time zone starting with Oct 19th
16:10:42 <DinaBelova> (well, if my passport with VIsa will be in time)
16:11:09 <DinaBelova> it's so difficult to grab Schengen visa for Russian person living in US :D
16:11:15 <msimonin> DinaBelova: ack, maybe easier to speak
16:11:26 <DinaBelova> msimonin yes, that's the reason :)
16:11:40 <ilyashakhat_mobi> DinaBelova: will we have some dedicated space at summit (as part of Rally program)
16:12:05 <DinaBelova> ilyashakhat_mobi we'll have it as a subteam of Large Deployments Team
16:12:13 <DinaBelova> #link https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15504/openstack-performance-team-what-has-been-done-during-newton-cycle-and-ocata-planning
16:12:27 <DinaBelova> I'll prepare etherpad for this session this week
16:13:08 <ilyashakhat_mobi> cool :)
16:13:49 <DinaBelova> I think that's all regarding the summit
16:13:54 <DinaBelova> #topic Open Discussion
16:13:56 <DinaBelova> anything else?
16:14:05 <rohanion> yeah...
16:14:37 <rohanion> I've finished working with the cinder patch
16:15:02 <rohanion> Still waiting for glance and neutron team to merge the changes
16:15:24 <DinaBelova> rohanion ack
16:16:03 <rohanion> I've started working on nova again
16:16:07 <msimonin> Sorry just to have more context: it's about osprofiler ?
16:16:13 <rohanion> Yeah
16:16:14 <DinaBelova> msimonin yes :)
16:16:16 <msimonin> ok :)
16:17:06 <msimonin> What will be the osprofiler intergation for Newton ?
16:17:34 <DinaBelova> msimonin alsmost all services support it right now
16:17:44 <DinaBelova> only Nova missing from the main OpenStack services :)
16:18:41 <msimonin> DinaBelova: ack
16:19:15 <DinaBelova> I think we may finish for now
16:19:33 <DinaBelova> thanks everyone for coming
16:19:34 <DinaBelova> #endmeeting