16:00:28 <mlavalle> #startmeeting neutron_performance
16:00:29 <openstack> Meeting started Mon Sep  9 16:00:28 2019 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:32 <openstack> The meeting name has been set to 'neutron_performance'
16:00:34 <njohnston> o/
16:00:36 <slaweq> hi
16:00:40 <jrbalderrama> hi !
16:00:43 <rubasov_> hi
16:01:03 <rubasov_> I'm sorry for missing some of these meetings
16:01:27 <mlavalle> hey how are you all!
16:01:40 <mlavalle> rubasov_: it was summer, so it's ok
16:01:46 <slaweq> good (but tired)
16:01:54 <rubasov_> good too, thank you
16:02:00 <mlavalle> now it's over, so back to reallity
16:03:14 <mlavalle> ok, let's get going
16:03:18 <mlavalle> #topic updates
16:04:03 <bcafarel> late o/
16:04:10 <mlavalle> does anybody have an update?
16:04:59 <jrbalderrama> small one
16:05:29 <njohnston> In our testing we have detected that something may have changed in bulk port creation that has slowed it down a bit, investigation just starting
16:06:14 <jrbalderrama> Ildiko from the openstack foundation, ask us to publish something about our collab at superuser.openstack.org. If you agree I can draft something.
16:06:33 <mlavalle> jrbalderrama: yes by all means
16:06:43 <mlavalle> I can help if you need
16:07:02 <jrbalderrama> great! I will keep you posted
16:07:19 <rubasov_> I intend to pick up where I left off: fixing the comments and merging the osprofiler vif plug patch
16:07:43 <mlavalle> ok, thanks for your updates
16:08:52 <mlavalle> I contnued working with with https://review.opendev.org/#/c/678438
16:08:52 <patchbot> patch 678438 - neutron - PoC: Add profiler decorator to Neutron - 3 patch sets
16:09:27 <mlavalle> to profile code and generate data viewable with KCacheGrind
16:10:04 <mlavalle> I spent Saturday evening trying to analyze some of the files I shared with you all last meeting
16:10:31 <mlavalle> and in many cases I couldn't make any sense of the data in KCacheGrind
16:12:08 <mlavalle> so I dug a little bit and discovered that the code that I am using in KCacheGrind class comes originally from here: https://people.gnome.org/~johan/lsprofcalltree.py
16:13:51 <mlavalle> I also discovered that an adaptation of it is used for django
16:13:53 <mlavalle> https://github.com/django-extensions/django-extensions/blob/8bc17b395b24a2b733d8035c270ae0c97e4909e4/django_extensions/management/commands/runprofileserver.py
16:14:35 <mlavalle> so since django is a mature framework, I decided to base my KCacheGrind class on theirs
16:15:17 <mlavalle> and that is what is reflected on the lastes revision of https://review.opendev.org/#/c/678438
16:15:17 <patchbot> patch 678438 - neutron - PoC: Add profiler decorator to Neutron - 3 patch sets
16:15:47 <mlavalle> I generated some profiles and seem to be in better shape
16:16:37 <mlavalle> btw, this is the doc explaining how this profiling is used in django: https://django-extensions.readthedocs.io/en/latest/runprofileserver.html
16:18:04 <njohnston> nice
16:18:09 <mlavalle> to be totally sure, my next step is to generate a  report file from cProfile
16:18:34 <mlavalle> and compare that report with what I can see with the KCachegrind gui tool
16:20:05 <mlavalle> and that got me thinking....
16:21:53 <mlavalle> if we generated profile data directly from cProfile and put it in a file, we could put this generation in a test running in the check queues to regualerly generate these data
16:22:45 <mlavalle> we could even have these test check if certain performance limist are esceeded and flag it some way
16:22:51 <mlavalle> what do you think?
16:24:07 <slaweq> mlavalle: will it require any changes in "production" code?
16:24:47 <mlavalle> my plam is to make the profiling configurable
16:24:52 <mlavalle> plan^^^^
16:24:52 <slaweq> ahh, ok
16:25:14 <mlavalle> for the time being, I am focusing on getting the stats right
16:25:28 <slaweq> can't it be done as additional middleware in pipeline?
16:25:32 <mlavalle> but eventually, I want to:
16:25:33 <slaweq> (just asking)
16:25:38 <mlavalle> 1) Make it configurable
16:26:08 <mlavalle> 2) or create a sort of fullstack test where we do profiling
16:26:33 <mlavalle> maybe the fullstack idea is better
16:27:40 <mlavalle> so set collect the profiling data in the wsgi controllers
16:28:04 <mlavalle> that way we don't have to add profile decorators in every single call
16:28:25 <slaweq> probably, but I'm not sure how it will work in such case tbh :)
16:28:43 <slaweq> that's why I'm asking if maybe You can try such approach at least
16:29:05 <mlavalle> and since I don't see deployers profiling code, maybe the fullstack test approach is better
16:29:24 <mlavalle> so here's my plan from now to the next meeting:
16:29:45 <mlavalle> 1) Make sure we are generating the right data
16:30:20 <mlavalle> 2) Modify the patch so we decorate the wsgi controllers
16:30:37 <mlavalle> keep in mind that all of this is PoC
16:30:42 <mlavalle> does it make sense?
16:31:05 <slaweq> yes :)
16:32:15 <mlavalle> we can even extend this to the agents
16:32:29 <bcafarel> sounds good, also we may then be able to get some initial findings from 1) data
16:32:32 <mlavalle> all of them are big loops, right?
16:33:02 <mlavalle> so we can collect data per iteration
16:34:04 <mlavalle> bcafarel: yesh, that is why I want to generate a report directly from cProfile without having to go through Kcachegrind
16:34:23 <mlavalle> a cProfile report shoud be our baseline
16:35:37 <mlavalle> and what we see in the Kcachegrind tool should reflect the cProfile report
16:36:29 <mlavalle> ok, that is all my update for today
16:36:39 <mlavalle> anything else we should discuss?
16:36:56 <slaweq> I don't have anything
16:37:18 <mlavalle> I am going to propose a PTG session on this topic
16:37:28 <mlavalle> so we can asses where we are on this topic
16:37:52 <mlavalle> slaweq: since you will be running the show at that point, is that ok with you?
16:38:24 <slaweq> mlavalle: sure
16:38:37 <mlavalle> ok, thanks for attending
16:38:43 <mlavalle> have a great week!
16:38:47 <bcafarel> that should be an interesting session :)
16:38:48 <mlavalle> #endmeeting