15:00:19 <rhochmuth> #startmeeting monasca
15:00:20 <openstack> Meeting started Wed May 11 15:00:19 2016 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:24 <openstack> The meeting name has been set to 'monasca'
15:00:27 <rhochmuth> o/
15:00:28 <rbak> o/
15:00:33 <witek> hello
15:00:33 <bklei> o/
15:00:38 <koji> hi
15:00:47 <jayahn> hi
15:00:54 <kamil_> hey
15:00:54 <flintHP> hi
15:01:07 <rhochmuth> hi flintHP
15:01:10 <ddyer> o/
15:01:17 <thatsdone> hi
15:01:22 <rhochmuth> hi ddyer
15:01:24 <ashwin> hi good morning
15:01:35 <rhochmuth> hi thatsdone
15:01:41 <rhochmuth> hi ashwin
15:01:47 <ddieterly> hi everybody!
15:01:47 <rhochmuth> hi everyone else
15:01:54 <ddieterly> that covers it
15:02:03 <rhochmuth> the agenda is at, https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:02:11 <rhochmuth> Agenda for Wednesday May 11, 2016 (15:00 UTC)
15:02:11 <rhochmuth> 1.	monasca-transform
15:02:11 <rhochmuth> 1.	https://wiki.openstack.org/wiki/Monasca/Transform
15:02:11 <rhochmuth> 2.	Reviews
15:02:11 <rhochmuth> 1.	https://review.openstack.org/#/c/307963/
15:02:12 <rhochmuth> 2.	https://review.openstack.org/#/c/289675/
15:02:12 <rhochmuth> 3.	https://review.openstack.org/#/c/306621/
15:02:13 <rhochmuth> 4.	https://review.openstack.org/#/c/286281/
15:02:13 <rhochmuth> 5.	https://review.openstack.org/#/q/topic:sporadic_metric
15:02:14 <rhochmuth> 6.	https://review.openstack.org/#/c/310511/
15:02:25 <rhochmuth> What I woudl like to cover first is monasca-transform
15:02:27 <s-kawabata_> o/
15:02:44 <rhochmuth> we have a few folks form hpe that are here today
15:02:57 <rhochmuth> ashwin, ddyer and flinHP
15:03:02 <rhochmuth> flintHP
15:03:14 <rhochmuth> so, if we could cover that in the first half hour that woudl be good
15:03:24 <rhochmuth> then we can look at some fo the reviews that have been posted
15:03:34 <bklei> +1
15:03:34 <rhochmuth> #topic monasca-transform
15:03:52 <rhochmuth> so, i'll let ashwin and team introduce you to the project and what is being done
15:04:00 <rhochmuth> and then open up for questions
15:04:09 <rhochmuth> ashwin u have the floor
15:04:16 <ashwin> hi all
15:04:22 <rhochmuth> hi ashwin
15:04:35 <ashwin> we will do a walkthrough new project monasca-transform
15:04:52 <ashwin> Blueprint is at https://blueprints.launchpad.net/monasca/+spec/monasca-transform
15:05:29 <ashwin> More detailed design is on the wiki at : https://wiki.openstack.org/wiki/Monasca/Transform
15:06:36 <ashwin> The main purpose of this project is to aggregate and transform metrics published to monasca's metrics topic in Kafka
15:07:05 <ashwin> and publish new aggregated metrics back to metrics topic in Kafka.
15:07:25 <ddieterly> what use cases does it cover that can't be covered by the alarm definition language?
15:08:06 <ashwin> There are several use cases which we can address e.g. aggregate individual metrics, combine multiple metrics to derive a new metric, find say a rate at which metric changes over a time interval etc
15:09:09 <witek> do you want to write back to the same topic?
15:10:17 <ddieterly> aggregrate individual metrics and combine multiple metrics to derive a new metric seem like functionality that already exists in monasca from the users point of view using existing tools/queries
15:10:40 <ashwin> yes the idea is aggregated metric is just like any other monasca metrics so you can set alarms etc on the aggregated metric
15:10:59 <ddieterly> find the rate and deriving metrics seem like new functionality
15:12:41 <ddieterly> why was apache spark chosen instead of using storm?
15:13:12 <ashwin> Deriving metrics using complex business logic For example consider the periodic heart beat metrics which come from a VM indicating what state the VM was in. One might have to look at how the state of  VM changed over the target time interval and then use some business logic come up with a new metric which would indicate the amount of time that the VM was in a usable state
15:14:38 <mrhillsman> o/
15:15:11 <bklei> will this all be project-scoped/multi-tenant?
15:15:29 <bklei> like for operators, as well as cloud customers?
15:15:53 <rhochmuth> aggregations can be defined based on any parameters
15:16:25 <rhochmuth> the project/tenant aggregations will be the normal one
15:16:45 <rhochmuth> for exampel calculating the number of swift bytes in/out per tenant
15:16:56 <rhochmuth> for metering/billing purposes
15:17:02 <bklei> and operators can aggregate across all projects?
15:17:17 <rhochmuth> yes, i believe so
15:17:23 <bklei> that would be good
15:17:34 <rhochmuth> i don't think that was the intended usage
15:17:40 <rhochmuth> but it is just code
15:17:59 <tomasztrebski> would I be able to compose mem.used_md with cpu.user_perc somehow with this ?
15:18:05 <tomasztrebski> it that the case ?
15:18:32 <ashwin> yes our initial use cases involve aggregating metrics over entire cloud, by all hosts, by each host, by all projects etc
15:18:33 <rhochmuth> not sure that was a target, but i'm not sure what you mean by that
15:18:48 <rhochmuth> tomasz ^^^
15:19:03 <bklei> nice ashwin
15:19:42 <ashwin> yes you could for example get mem.total_mb metrics which are per host metrics and aggregate to get total available memory across all hosts
15:19:43 <tomasztrebski> composite alarms - that's the name that at least read somewhere, the case is to compose two distinct metrics to say: trigger ALARM if cpu exceeds 50% and in the sime memory consumption goes above 70%
15:20:00 <rhochmuth> tomasz: that isn't the intended usage
15:20:08 <tomasztrebski> mhm, thx
15:20:53 <ericksonsantos> hello everyone o/
15:21:17 <tomasztrebski> hmm...my reviewer from sporadic metrics :D, hello there :-)
15:21:17 <iurygregory> hi people  =) we are late sorry
15:21:39 <ericksonsantos> tomasztrebski, haha
15:21:45 <rhochmuth> hi, we are in theprocess of reviewing the monasca-transform engine
15:21:59 <rhochmuth> let's stay focused for another 10 minutes
15:22:07 <iurygregory> tks for the update rhochmuth
15:22:17 <rhochmuth> then do intros and reivews
15:22:54 <ashwin> We are planning on using Spark, Spark streaming on collecting set of metrics and use generic transformation components to run series of aggregation operations over a time interval and publish the metrics back to Kafka.
15:23:12 <ashwin> all the components are written in python
15:23:33 <ashwin> All transformations will be data driven, and it should be possible to add new aggregation routines by configuration changes.
15:24:08 <ashwin> Generic transformation components will be re-usable and easy to change, add new ones or make changes to existing ones
15:24:36 <ashwin> Should be possible to write python unit tests  to test each transformation, also series of transformations (transformation pipeline) can be tested.
15:24:47 <ddieterly> was an evaluation done between spark and storm? monasca uses storm and the team has expertise in it. why was spark a better alternative to storm?
15:25:24 <slogan> Can I build the transform engine from code as well as from config?
15:25:57 <ddyer> spark was chosen for a couple of reasons:
15:25:57 <slogan> Er, pipeline
15:26:03 <ddyer> 1. python support
15:26:12 <kamil_> and monasca-ui will be extended to set/configure "new aggregations" ?
15:26:32 <rhochmuth> kamil_: there isn't an API at this time
15:26:33 <ddyer> 2. we had previous experience with hadoop and spark supported a lot of the same mechanisms
15:27:12 <ddyer> 3. it has a lot of momentum in terms of development and we wanted something others would be willing to invest in
15:28:00 <witek> how much of the code is ready? the repo is empty
15:28:08 <ddieterly> ddyer it would be good to update the wiki with those reasons so that people know why spark was chosen
15:28:20 <ddyer> good idea
15:28:24 <rhochmuth> kamil_: so you can deploy the service and specify the transform via configuration and new metrics will be created
15:28:34 <flintHP> we are in the process of preparing the code for submittal/review
15:28:47 <rhochmuth> witek: should be published soon
15:28:51 <ashwin> we found the ability to write unit tests in python to test components really useful (vs hadoop where this was impossible)
15:29:29 <rhochmuth> so, we are kinda getting to the end of a 1/2 hour
15:29:36 <rhochmuth> should probably talk next stpes
15:29:39 <rhochmuth> steps
15:29:45 <rhochmuth> follow-up
15:29:59 <rhochmuth> the code will be submited for review
15:30:08 <rhochmuth> and then folks can start using it
15:30:13 <rhochmuth> and reviewing it
15:30:35 <tomasztrebski> sweet :D
15:31:01 <rhochmuth> so, does anyone have any major issues with this code being submitted for review
15:31:04 <rhochmuth> and proceeding
15:31:14 <rhochmuth> would anyone like another session after reviewing the wiki
15:31:23 <witek> please squash the commits :)
15:31:30 <rhochmuth> lol
15:31:35 <ddieterly> we need to see it before we can make that determination
15:31:48 <slogan> :-)
15:31:57 <rhochmuth> so the code will end-up in the monasca-transform repo
15:32:12 <rhochmuth> as a single new review
15:32:17 <ddieterly> we eagerly await your review for submission ;-)
15:32:53 <rhochmuth> so, do we want to discuss again next week any issues?
15:33:02 <rhochmuth> we can use gerrit for comments and other questions
15:33:10 <rhochmuth> so, i think that would be preferred right now
15:33:57 <tomasztrebski> +1 gerrit
15:34:12 <flintHP> wow, we already have a +1 (grin)
15:34:20 <ashwin> thank you everyone!
15:34:35 <rhochmuth> ok, so ashwin and team please submit, and then we can review and discuss any subsequent follow-up
15:34:45 <rhochmuth> "submit the review" that is
15:35:02 <ashwin> sounds good thanks roland!
15:35:20 <rhochmuth> thanks ashwin et al
15:35:42 <ddieterly> roland is speaking latin again
15:35:54 <rhochmuth> the other topic listed on the agenda is reviews
15:36:02 <rhochmuth> #topic reviews
15:36:13 <rhochmuth> https://review.openstack.org/#/c/307963/
15:36:27 <bklei> that's me asking for status
15:36:36 <bklei> fix metric-list limits
15:36:40 <bklei> please to merge :)
15:36:59 <rhochmuth> yes, i think it is ready
15:37:01 <bklei> this is a blocker for us at twc
15:37:37 <rhochmuth> the problem from last week was that there were several performance issues identified
15:37:55 <rhochmuth> rbarndt did a huge amount of testing and performance analysis last week
15:38:01 <rhochmuth> but i think it is ready to go now
15:38:01 <bklei> last i heard from ryan brandt that he was ok now?
15:38:07 <rhochmuth> correct
15:38:30 <rbrndt> yup
15:38:55 <rhochmuth> https://review.openstack.org/#/c/289675/
15:39:10 <rhochmuth> this one is also ready, jsut waiting on some more testing
15:39:31 <rhochmuth> actually I =2'd that one, but the merge didnt' go through
15:39:34 <bklei> ok, rbak is excited to pull that in and make grafana changes
15:39:44 <rbrndt> that one is based off the metrics-list fix
15:39:48 <rbrndt> so need that first
15:40:08 <rhochmuth> right
15:40:39 <rhochmuth> i'll discuss after doen here, and possibly merge both
15:40:51 <bklei> sweet
15:40:58 <rhochmuth> https://review.openstack.org/#/c/306621/
15:41:09 <bklei> that's me
15:41:25 <bklei> i think it's ready -- no oustanding comments
15:41:30 <bklei> ovs plugin
15:41:38 <rhochmuth> can this be used in devstack
15:42:00 <bklei> i need to stand up devstack again -- does it use ovs?
15:42:12 <rhochmuth> i don't know
15:42:29 <bklei> i'll get the answer to that
15:43:13 <rhochmuth> has anyone else tested your plugin
15:43:31 <rhochmuth> all i can do at this point is a cursory review of code, ...
15:43:37 <bklei> tested i'm not sure -- i have been getting comments from folks that seem like neutron people
15:44:07 <bklei> we've tested it at twc in the labe with and without HA routers
15:44:13 <bklei> s/labe/lab
15:44:24 <rhochmuth> well, i'll take another look
15:44:39 <bklei> cool
15:44:46 <rhochmuth> i asked some folks in hpe india that are looking at vswitch metrics
15:44:59 <rhochmuth> some of them left comments
15:45:09 <bklei> aah, cool, that must be sonu and selvakumar
15:45:28 <rhochmuth> sonu yes
15:45:37 <rhochmuth> let's move on next review
15:45:37 <shinya_kwbt_> Openstak default is linuxbridge. I don't know devstack uses ovs.
15:45:38 <rhochmuth> https://review.openstack.org/#/c/286281/
15:46:16 <bklei> thx shinya -- so this won't help devstack if that's the default
15:47:17 <bklei> oh -- that kv hint is mine too
15:47:43 <rbrndt> I mentioned it in my comment, but I didn't see any differences in my testing of the hint
15:48:00 <rbrndt> And exploring the queries we use it seems like vertica is already satisfying everything locally
15:48:20 <rbrndt> It doesn't hurt anything though
15:48:27 <bklei> yeah -- i think it'll free vertica resource/bandwidth up more than anything
15:48:52 <bklei> according to artem at vertica -- the hint eliminates node chatter
15:49:05 <slogan> Devstack supports ovs, I think the default networking is ovs
15:49:31 <shinya_kwbt_> http://docs.openstack.org/developer/devstack/guides/neutron.html?highlight=openvswitch
15:49:32 <clarkb> slogan: default is still nova network which does not use ovs, but if you enable neutron then the default for neutron is ovs
15:49:54 <slogan> Just enable the q services, I might be able to help with a local.conf if needed
15:49:54 <rhochmuth> thanks shinya
15:50:15 <slogan> Right clarkb
15:50:50 <slogan> Novas net ID something I've never used so to me neutron iscalways on :-)
15:51:33 <rhochmuth> so let's move on
15:51:34 <rhochmuth> https://review.openstack.org/#/q/topic:sporadic_metric
15:51:41 <rhochmuth> tomasz, witek
15:52:10 <witek> Tomasz has moved the newest code to deterministic_alarm topic
15:52:28 <rhochmuth> i didn't get to review the latest round of changes
15:52:34 <rhochmuth> internet was out last night
15:52:45 <rhochmuth> but i heard that craig added comments
15:53:04 <witek> in sporadic_metric we have the additional sporadic field for metric, which can be persisted
15:53:34 <witek> so we can see, what kind of metric we are getting
15:54:04 <witek> is that functionality still useful after implementing deterministic_alarms?
15:54:19 <rhochmuth> it isn't required
15:54:44 <rhochmuth> as the deterministic_alarms are independent of sporadic metrics
15:54:44 <tomasztrebski> roland, I forgot to add you as reviewer to the newest changes, that's fixed right now
15:55:08 <rhochmuth> you can apply a deterministic_alarm to either a periodic or sproadic metric
15:55:17 <rhochmuth> and it is done independently
15:55:36 <rhochmuth> the only use that the sproadic metrics have at this point are to supply additional information about metrics
15:55:46 <witek> exactly
15:56:02 <rhochmuth> so it is still useful, but not required
15:56:11 <tomasztrebski> well, not exactly, we could "propose" user to use non-deterministic alarm for sporadic metrics
15:56:14 <tomasztrebski> directly in ui
15:56:50 <tomasztrebski> obviously the checkox would be mutable, so user could proceed as he wishes
15:56:50 <rhochmuth> so, i t hink the question is whether we shoudl keep sporadic metrics too
15:57:06 <tomasztrebski> I think we should with having the case above in my head
15:57:14 <rhochmuth> i see
15:57:37 <rhochmuth> i was assuming it woudl be removed just to simplify
15:57:40 <ericksonsantos> tomasztrebski, +1
15:57:46 <rhochmuth> ok, let me think about that some more
15:57:54 <witek> ok, thanks Roland
15:57:57 <rhochmuth> i'll discuss with craig
15:58:23 <tomasztrebski> that'd be up to the user to decide if he want to accept "the proposition" - so the checkbox (or something, I haven't decided yet) would be mutable in monasca-ui
15:58:26 <tomasztrebski> ok, thx
15:58:35 <rhochmuth> tomasz: are you planning on python changes too
15:58:50 <tomasztrebski> right now I am in middle of checking if I need to do anything for deterministic alarms
15:58:50 <rhochmuth> tomasz: also agree with last statements
15:59:11 <tomasztrebski> till now I found one "bug" in monasca-api documentation :D
15:59:26 <rhochmuth> so, i think we've run out of time again
15:59:34 <rhochmuth> this is happenign a lot
15:59:42 <rhochmuth> do we need a longer time-slot or two meetings
15:59:45 <tomasztrebski> a lot of interest in monasca :)
15:59:50 <rhochmuth> yup
15:59:53 <rhochmuth> that's good
16:00:01 <rhochmuth> i hate rushing through and skipping over stuff
16:00:14 <rhochmuth> we have a few folks that showed up today that we didnt' even get introduced
16:00:26 <rhochmuth> ok, i need to close the meeting down
16:00:37 <rhochmuth> sorry foks
16:00:43 <rhochmuth> too much to discuss today
16:00:46 <iurygregory> nps ^^
16:00:48 <ericksonsantos> no problem, we can introduce ourselves in the next meeting. :)
16:00:59 <witek> bye
16:01:00 <rhochmuth> i'll be in the openstack-monasca room
16:01:09 <rhochmuth> #endmeeting