15:00:02 <rhochmuth> #startmeeting monasca
15:00:02 <openstack> Meeting started Wed Feb 17 15:00:02 2016 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:05 <openstack> The meeting name has been set to 'monasca'
15:00:07 <rhochmuth> o/
15:00:08 <rbak> o/
15:00:10 <Kamil> Hello
15:00:24 <rhochmuth> hi kamil
15:00:24 <shinya_kwbt> o/
15:00:46 <bklei> o/
15:01:18 <rhochmuth> agenda is as follows:
15:01:19 <rhochmuth> Agenda for Wednesday February 17, 2016 (15:00 UTC)
15:01:19 <rhochmuth> 1.	Grafana v2 update
15:01:19 <rhochmuth> 2.	Agent thread-pool not restarting.
15:01:27 <rhochmuth> link at, https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:01:49 <rhochmuth> we should also probably look at outstanding reviews
15:02:02 <ho_away> hello
15:02:10 <pradipm> Hello
15:02:10 <rhochmuth> hi ho_away
15:02:16 <rhochmuth> hi pradipm
15:02:34 <rhochmuth> please feel free to update the agenda at the link
15:02:47 <rhochmuth> #topic grafana v2
15:03:02 <rbak> Just a short update on this today
15:03:31 <rbak> Grafana has some concerns about having another auth method, so the patches are on hold for the moment.
15:03:49 <rbak> But they've offered to work with us to find a way to offer the integration.
15:04:13 <rbak> I have a call with them later this week.
15:04:24 <rhochmuth> sounds goo
15:04:25 <rbak> So this will happen, it just might take a little longer
15:04:39 <rbak> That's all I have for the moment.
15:05:23 <rhochmuth> ok, thanks, hopefully i'll get to trying this out soon
15:05:30 <Kamil> short question. What would be the other auth-method? O-Auth?!
15:06:12 <rbak> Possibly?  Or maybe working out some sort of pluggable auth system to keystone doesn't need to be part of the main repo.
15:06:17 <rbak> I'll know more next week
15:06:25 <Kamil> okay. Thx
15:06:35 <rhochmuth> let me know if you want me involved
15:06:41 <rbak> will do
15:06:51 <rhochmuth> i had a discussion with raj
15:07:03 <rhochmuth> and they are doing some really cool things with grafana
15:07:37 <rhochmuth> so, next topic then
15:07:39 <rhochmuth> ?
15:07:45 <rbak> sure
15:07:53 <rhochmuth> #topic agent thread-pool not restarting
15:07:59 <rbak> This one is also mine
15:08:02 <rhochmuth> that doesnt' sound good
15:08:32 <rbak> Basically in the collector monasca-agent uses a custom thread pool rather than a library.
15:08:49 <rbak> But the terminate function in the thread pool doesn't actually terminate threads
15:09:15 <rbak> So if a thread gets stuck it does a restart, which is terminate and join, and blocks there forever.
15:09:56 <rbak> I'm just wondering if anyone who wrote the original thread pool is around to fix it?
15:10:12 <rbak> Or maybe we need to switch to a thread pool library instead?
15:10:16 <rhochmuth> hmmm, probably not
15:11:05 <rhochmuth> so, if the thread is left in this state, what problem are you seeing, that we haven't seen after all this time
15:11:23 <rbak> We have issues with agents locking up like this every few days, and they require a manual restarts
15:11:33 <rbak> I'm not sure what causes threads to get stuck
15:11:43 <rhochmuth> hmmm, we haven't seen that
15:11:56 <rhochmuth> is it a specific plugin that is failing that we aren't running
15:12:02 <rhochmuth> possibly?
15:12:24 <rhochmuth> or is it any thread/plugin
15:12:25 <rbak> No, I've seen it on http_check
15:12:33 <rbak> And another that I can't remember
15:12:54 <rhochmuth> ok, so http_check was written by david schroeder
15:13:11 <rbak> I think those are probably bugs in their own right, but the thread_pool shouldn't lock up.
15:13:38 <rhochmuth> so, if you have a library in mind that you want to switch too that sounds good to me
15:13:53 <rhochmuth> just need to ensure licensing is ok on it
15:14:06 <rhochmuth> apache, bsd, …
15:14:10 <rhochmuth> bu not gplv2
15:14:39 <rhochmuth> if you want to touch-base with david, then i can get you in contact with him
15:14:40 <rbak> I tried switching to the multiprocessing module yesterday.  That seems to be what the monasca thread pool is based on.
15:14:54 <rbak> But there are architecture reasons that doesn't work
15:15:05 <rbak> I'll have to look around for other options
15:15:31 <rhochmuth> we haven't seen any problems with our http_check
15:15:41 <rhochmuth> that we are performing in helion, btw
15:16:34 <rhochmuth> ok, i guess next topic
15:16:48 <rhochmuth> #topic Mitaka Release planing (ansible-installer, devstack, etc.)
15:16:57 <Kamil> okay that's mine
15:17:06 <rhochmuth> hi kamil
15:17:21 <Kamil> Hi. Openstack uses devstack to install a test/dev environment. I know that there exist already an integration for monasca. Do we plan to contribute this to devstack?
15:17:37 <rhochmuth> it is already done
15:17:45 <Kamil> ah okay... haven't seen it yet
15:17:58 <rhochmuth> See, https://github.com/openstack/monasca-api/tree/master/devstack
15:18:09 <Kamil> and will be the monasca-vagrant (ansible) also a part of mitaka release?
15:18:31 <rhochmuth> There is also a Vagrantfile in that directory that you can use to build in one step
15:18:54 <rhochmuth> monasca-vagrant is another development environment that predates devstack
15:19:09 <rhochmuth> we are still using monasca-vagrant
15:20:00 <rhochmuth> but, as it is only a dev environment, that repo isn't a part of the official release
15:20:27 <Kamil> okay... but the devstack integration is still in monasca-api repo. Do we want to contribute it to the devstack repo?
15:20:52 <Kamil> i mean to this one: https://git.openstack.org/openstack-dev/devstack
15:20:56 <rhochmuth> there are only specific "core" projects that are part of the devstack repo
15:21:06 <rhochmuth> all the new projects are done as plugins
15:21:24 <rhochmuth> so, there are projects like nova and swift that are part of devstack
15:21:41 <rhochmuth> but newer projects, like murano and manilla are not
15:22:18 <Kamil> okay. Thx. So that means, that we will need an installation script for log functionality
15:22:24 <rhochmuth> i forget the actual specificis of whether a project is in or not
15:22:41 <rhochmuth> i think we need to build a devstack plugin for the log api
15:22:58 <rhochmuth> one was already in progress or done
15:23:08 <Kamil> Good. Thanks for the information. That's all from my side
15:23:22 <rhochmuth> actually, i might be confused
15:23:28 <rhochmuth> there are tempest tests for the log api
15:23:41 <rhochmuth> but i don't see a devstack plugin
15:24:07 <rhochmuth> i think it would be developed similar to the current monasca plugin
15:24:07 <Kamil> yes. I also haven't seen a plugin from log-api
15:24:27 <rhochmuth> i would like to move off of the monasca-vagrant environment
15:24:31 <rhochmuth> at some point
15:24:41 <rhochmuth> it is time consuming managing two environments
15:25:12 <rhochmuth> there are several advantages of the monasca-vagrant env though
15:25:51 <rhochmuth> not sure how to resolve that
15:26:04 <Kamil> monasca-vagrant is a very good dev environment for testing purposes
15:26:32 <rhochmuth> so, i should point out that the monasca devstack plugin is used by the openstack ci system
15:27:16 <rhochmuth> every code review goes through openstack ci, which builds an devstack vm with monasca and then runs all the tempest tests against it
15:27:40 <rhochmuth> currently, those tests are run against mysql and influxdb
15:28:04 <rhochmuth> if you wanted to test hibernate/postgres, you could potentially add that as a configuration
15:28:50 <rhochmuth> we test both the python and java apis
15:28:59 <rhochmuth> but not all variations of databases
15:29:05 <rhochmuth> just mysql and influxdb
15:29:17 <ddieterly> the gates only test the api, right?
15:29:44 <Kamil> i think yes
15:29:49 <rhochmuth> the gates test the api
15:29:54 <ddieterly> so, we don't have gate jobs on the other monasca componenents
15:30:24 <ddieterly> i think it would be great if somebody created a devstack plugin for each of the other monasca components
15:30:43 <ddieterly> i.e., persister, threshold, notification
15:31:06 <ddieterly> then we could get tempest tests and gate jobs on those componenents
15:31:30 <rhochmuth> i'm not sure what you are saying
15:31:42 <rhochmuth> usually teh tempest tests test the api
15:32:01 <ddieterly> it would be nice if each component had tempest tests
15:32:15 <rhochmuth> the persister doesn't have an api
15:32:15 <ddieterly> so, we could have gate jobs for each component
15:32:28 <ddieterly> does tempest only tests api's?
15:32:36 <rhochmuth> the tempest tests are usually for testing apis
15:32:42 <rhochmuth> they are integration or system tests
15:32:53 <ddieterly> ok, then other tests for the other components running in a gate job
15:33:20 <ddieterly> so, if you check in a review for the persister, then the gate tests are run
15:33:45 <rhochmuth> there are gates on the persister, which include tempest
15:34:31 <rhochmuth> the monasca devstack plugin is used by the monasca-persister
15:34:32 <ddieterly> does the persister have a devstack plugin?
15:34:42 <ddieterly> oh, it is shared
15:34:51 <rhochmuth> yes, it is shared
15:35:15 <ddieterly> it would be nice if it had it's own devstack plugin to isolate testing to just the persister
15:35:19 <rhochmuth> the devstack plugin in the monasca-api is used by multiple projects
15:36:24 <ddieterly> interesting
15:36:33 <pradipm> When we mention enable_plugin monasca-api git://git.openstack.org/openstack/monasca-api - all the monasca mini services are getting installed (persistor, monasca-threh .. etc. etc. ..) right?
15:36:50 <rhochmuth> correct
15:38:51 <rhochmuth> #topic other
15:39:04 <rhochmuth> ho_away: are you still looking at anomaly detection?
15:39:41 <ho_away> rhochmuth* yep but mainly discussion with my colleague to explain of current implementation
15:40:11 <rhochmuth> i saw an interesting article on fujitus applying deep learning to classification of time series data yesterday
15:40:40 <rhochmuth> i've also started spinning up again on nonparametric statistics
15:40:45 <ho_away> rhochmuth: really, can you point out the link?
15:41:40 <rhochmuth> http://phys.org/news/2016-02-fujitsu-deep-technology-time-series-high.html
15:42:43 <ho_away> rhockmuth: thanks! i will find out the writer :-)
15:42:55 <pradipm> I am seeking an export opinion for one usecase .. if possible in monasca ..........
15:42:57 <rhochmuth> please note, that joe keen made some improvements to the kafka consumer in monasca-common
15:43:32 <rhochmuth> this bumps the persister performance probably somewhere in the 10-20 K messages per sec
15:44:01 <ddieterly> java or python or both?
15:44:02 <rhochmuth> thanks ho_away
15:44:09 <rhochmuth> python
15:44:30 <ddieterly> nice
15:44:45 <rhochmuth> java was always fast
15:45:34 <rhochmuth> these improvements address the python persister which we hadn't done as much analysis on
15:45:53 <rhochmuth> so, does anyone want to talk about the log-api performance
15:47:41 <rhochmuth> ok, i've left my analysis and latest performance numbers
15:47:49 <Kamil> Unfortunaly, i am not aware of the current status of log-api. I think witek wrote you an email
15:49:48 <rhochmuth> ok, sounds like we are wrapping up
15:49:54 <pradipm> I am seeking an export opinion for one usecase .. if I can discuss once here.
15:50:00 <rhochmuth> sure
15:50:10 <rhochmuth> the floor is yours
15:50:29 <pradipm> Is there a way to run some simple rules on the alarm-history output?
15:50:51 <rhochmuth> like what
15:51:14 <pradipm> for example: over a time of say 5 hours, if IOPS have been breached in some pattern over 80%
15:51:53 <ddieterly> i don't believe that we have any filters/options on alarm-history at this time
15:51:54 <rhochmuth> there is a topic in kafka that any consumer can consume from
15:52:20 <rhochmuth> if you want to consume alarm state transition events that would be easy
15:52:39 <pradipm> I see. ... directly from kafka?
15:52:50 <pradipm> It's already in the alarm-state-history .. right?
15:53:12 <rhochmuth> You can also use the API
15:53:29 <rhochmuth> the entire alarm state history is in the API
15:53:59 <rhochmuth> you can also "subscribe" to alamrs via webhook notifications
15:54:00 <pradipm> thats correct. Is there any component in monasca that runs some form of rule-check on anywhere else?
15:54:12 <rhochmuth> no
15:54:26 <pradipm> Ok....
15:54:47 <rhochmuth> there has been some interest in various forms of this
15:55:06 <rhochmuth> for example, there is a group that is interested in alarm clustering
15:55:08 <pradipm> any suggestion if I want to define some simple rules (filters as in alarm-history .. or similar) ... what shd be the best approach?
15:55:22 <rhochmuth> the Vitrage project is also interested in alarm correlation
15:55:42 <ddieterly> get the alarms from the api and do something externally
15:55:44 <pradipm> Vitrage is in Monasca? I am not aware of .. sorry for ignorance.
15:55:57 <rhochmuth> Vitrage is another openstack project
15:56:05 <ddieterly> actually, get the alarm-history...
15:56:17 <rhochmuth> https://wiki.openstack.org/wiki/Vitrage
15:56:19 <pradipm> make sense ...ddieterly
15:56:46 <ddieterly> pradipm: there is an option in the cli to get the raw json, that would help with external tools
15:56:51 <pradipm> Thanks rhochmuth
15:56:53 <rhochmuth> i think understanding your specific use case would be good
15:57:14 <pradipm> sure ...
15:57:21 <rhochmuth> If you want to plugin to kafka the message schema is at, https://wiki.openstack.org/wiki/Monasca/Message_Schema
15:57:35 <rhochmuth> it depends on what you want to do though.
15:57:44 <pradipm> so basically I want to do some conformance on storage object underneath cinder/manila
15:57:54 <rhochmuth> you could filter alarm state transition events that are created by the threshold engine
15:58:15 <rhochmuth> so, i'm guess that would best be handled via the api
15:58:33 <rhochmuth> i should also point out that fabiog, who isn't here today, is adding support for Congress in Monasca
15:58:43 <rhochmuth> Congress is a policy engine
15:58:51 <pradipm> I see ... yeah .. something like that ...
15:59:04 <rhochmuth> Please check out, https://wiki.openstack.org/wiki/Congress
15:59:14 <rhochmuth> a fabiog would be the contact
15:59:20 <pradipm> can I configure a policy (like simple rules) and run on top of say alarm-state-history apu ..
15:59:23 <rhochmuth> this meeting is coming to an end i just realized
15:59:34 <pradipm> will try to chat to fabiog
15:59:59 <pradipm> thanks rhochmuth. thanks a lot for all the pointers.
16:00:07 <rhochmuth> by everyone
16:00:13 <ho_away> thanks rhochmuth and all :-)
16:00:29 <rhochmuth> #endmeeting