15:00:17 <rhochmuth> #startmeeting monasca 15:00:17 <openstack> Meeting started Wed Oct 19 15:00:17 2016 UTC and is due to finish in 60 minutes. The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:19 <rhochmuth> o/ 15:00:20 <rhochmuth> https://etherpad.openstack.org/p/monasca-team-meeting-agenda 15:00:21 <openstack> The meeting name has been set to 'monasca' 15:00:35 <rhochmuth> Agenda for Wednesday October 19, 2016 (15:00 UTC) 15:00:35 <rhochmuth> 1. Summit Prep 15:00:35 <tomasztrebski> o/ 15:00:35 <rhochmuth> 2. Reviews 15:00:35 <rhochmuth> 1. https://review.openstack.org/#/c/388183/ 15:00:35 <rhochmuth> 2. https://review.openstack.org/#/c/375717/ 15:00:35 <rhochmuth> 3. Ask Charter Communications about the size of their deployment 15:00:35 <rhochmuth> 4. Lost & Forgotten [ help with testing, Tomasz :( ] 15:00:36 <rhochmuth> 1. https://review.openstack.org/#/c/361318/ 15:00:36 <rhochmuth> 2. https://review.openstack.org/#/c/315742/ 15:00:37 <rhochmuth> 5. Devstack [Openstack] integration 15:00:37 <rhochmuth> 1. https://review.openstack.org/#/c/387195/ 15:00:38 <rhochmuth> 2. https://review.openstack.org/#/c/385149/ 15:00:38 <rhochmuth> 6. Local deployment fix 15:00:39 <rhochmuth> 1. https://review.openstack.org/#/c/388017/ 15:00:46 <shinya_kwbt> o/ 15:00:47 <rbak> o/ 15:00:51 <hosanai> o/ 15:00:53 <cbellucci> o/ 15:00:57 <Kamil_> o/ 15:00:58 <rhochmuth> a bit of a higger agenda this week 15:01:07 <tomasztrebski> sorry :( 15:01:09 <rhochmuth> the first topic i woudl like to get to is summit prep 15:01:11 <koji> o/ 15:01:20 <rhochmuth> https://etherpad.openstack.org/p/monasca-barcelona-design-summit 15:01:25 <rhochmuth> please see the above link 15:01:26 <bklei> o/ 15:01:36 <rhochmuth> are there other items that we should add or expand on 15:01:47 <rhochmuth> i need to update the summit calendar/schedule 15:01:52 <rhochmuth> i woudl like to do that today 15:01:58 <rhochmuth> i'm pretty much already late on this 15:02:13 <rhochmuth> but, maybe unlike last time, only the people involved wwith monasca will show up 15:02:19 <rhochmuth> that could be a good thing 15:02:38 <rhochmuth> because last time in austin, a lot of folks showed up that weren't involved with monasca 15:02:44 <rhochmuth> and that diluted the meeting significantly 15:02:51 <rhochmuth> so, anyway 15:03:01 <rhochmuth> Here is what i have so far 15:03:02 <rhochmuth> 1. Monasca New Database discussion 15:03:02 <rhochmuth> 1. Cassandra 15:03:03 <rhochmuth> 1. https://blueprints.launchpad.net/monasca/+spec/cassandra-tasks 15:03:03 <rhochmuth> 2. https://etherpad.openstack.org/p/monasca_cassandra 15:03:03 <rhochmuth> 2. Locked alarms 15:03:03 <rhochmuth> 1. https://blueprints.launchpad.net/monasca/+spec/locked-alarms 15:03:03 <rhochmuth> 3. Logging 15:03:03 <rhochmuth> 1. Multi-tenancy support 15:03:04 <rhochmuth> 4. Properties/attributes for metrics/logs and sporadic metrics 15:03:04 <rhochmuth> 1. https://blueprints.launchpad.net/monasca/+spec/publish-logs-to-topic-selectively 15:03:05 <rhochmuth> 5. Monasca Analytics 15:03:05 <rhochmuth> 1. Anomaly Detection 15:03:06 <rhochmuth> 2. Alarm correlation/clustering 15:03:23 <rhochmuth> Obviousely, on the topic of Cassandra, based on the update from Monday, that disucssion needs to expand 15:03:46 <rhochmuth> because, Cassandra has got some difficult areas that we haven't been able to resolve 15:04:00 <rhochmuth> Item 2 is locked alarms 15:04:08 <rhochmuth> there is a blueprint for that one 15:04:31 <rhochmuth> but i woudl like to discuss that one in more detail and close on the design if possible 15:04:43 <rhochmuth> next item is multi-tenenacy 15:04:47 <rhochmuth> for logging 15:04:59 <rhochmuth> looking for fujitsu to lead that discussion 15:05:30 <rhochmuth> the nest item is properties/attribues for metrics/logs, which is related to support for sporadic alarms 15:05:57 <rhochmuth> i'll let you look at the remainder 15:06:30 <rhochmuth> if there are other topics, now is the change to add them 15:06:41 <rhochmuth> a blueprint would also be nice to accompany the discussion if that makes sense 15:07:04 <Kamil_> i think tomasz is in the meeting 15:07:08 <tomasztrebski> as far as I understood this is about discussing the design 15:07:13 <rhochmuth> correct 15:07:27 <tomasztrebski> I added one topic that I think is one thing that monasca would benefit from 15:07:30 <tomasztrebski> last point 15:07:40 <tomasztrebski> monasca maintenance 15:07:49 <rhochmuth> ok, i'll let you lead that discussion then 15:07:58 <tomasztrebski> cool 15:08:28 <tomasztrebski> so IMHO I think that what monasca does not have that could be done semi-automatic is muting certain alarm definitions 15:08:49 <tomasztrebski> based on defined period of time, whether this should be repeatable operations 15:09:19 <rhochmuth> ahhh, is that related to "flood detection/mitigation" 15:09:24 <tomasztrebski> like for example you got server maintenance over the night/weekend and you know certain alarm definitions will lit because of higher resource usage 15:09:33 <tomasztrebski> and you want to mute them 15:09:36 <tomasztrebski> maybe :D 15:10:09 <rhochmuth> so, you want to adjust the thresholds based on patterns 15:10:12 <tomasztrebski> the overall idea would be to expand data model to define maintenance_definition and have an engine to modify alarm_definitions a.k.a. change notifications_enabled flag status 15:10:13 <bklei> agree with the concept tomasztrebski -- that's lacking if you compare monasca alarming vs other tools like icinga/nagios 15:10:28 <bklei> we'd support a change like that for sure 15:10:56 <tomasztrebski> this ain't rocket science but there would be some work to be done in api / ui / client 15:11:01 <tomasztrebski> not sure about the threshold 15:11:16 <rhochmuth> i see, it is simpler than that 15:11:44 <tomasztrebski> and one thing that would need to be added (IMHO) would be extra component to change status because I think threshold is responsible for transitioning basicly 15:11:47 <rhochmuth> we've been interested in this area too 15:12:08 <tomasztrebski> rhochmuth: what do you mean by simpler ? 15:12:10 <rhochmuth> would it be more like alarm supression then 15:12:20 <tomasztrebski> btw: that would be all I had in my head about that topic 15:12:26 <bklei> even just a maintenance flag will go a long way 15:12:27 <rhochmuth> rhochmuth: simpler than adjusting the threshold dynamically 15:12:58 <rhochmuth> so going back to my question, is it related to alarm supression? 15:13:34 <tomasztrebski> if you mean to suppress transition - no, I would keep it the way it is, just disable any notifications there are 15:13:47 <tomasztrebski> but maybe suppressing transition could be an option to discuss - sure 15:13:59 <rhochmuth> ok, sounds like a good area to discuss 15:14:15 <rhochmuth> i also put a topic up for alarm flood detection/mitigation 15:14:32 <rhochmuth> it isn't exactly related, but I would like to cover that 15:14:48 <rhochmuth> as prometheus does that 15:14:50 <tomasztrebski> however I think that without notifications alarm is kind of meaningless, you could also add some sort of extra info to alarm like -> maintenance on or something similar to just hint the user 15:15:17 <rhochmuth> ok, so let's move on with agenda 15:15:22 <rhochmuth> thanks tomasz 15:15:29 <tomasztrebski> np 15:15:38 <rhochmuth> you are up for leading that session in barcelona 15:15:53 <tomasztrebski> :scared_cat_face 15:16:25 <rhochmuth> you will be scared 15:16:34 <rhochmuth> that was yoda 15:16:39 <tomasztrebski> :scared_like_hell_crying_cat_face :P 15:16:54 <tomasztrebski> heh, yoda rlx 15:16:55 <rhochmuth> #topic reviews 15:17:03 <rhochmuth> https://review.openstack.org/#/c/388183/ 15:17:16 <bklei> that's me 15:17:42 <bklei> i think we're making headway on that one -- very excited to pull it in. huge perf diff for vertica 15:17:53 <rhochmuth> we are excited about it too 15:18:14 <bklei> sounds like you saw similar 'minutes to seconds' improvement 15:18:15 <rhochmuth> rbrandt abadoned the other review we posted btw 15:18:27 <rhochmuth> or i thought he was going to abandon 15:18:28 <bklei> oh the projection hint? 15:18:28 <tomasztrebski> pity we (Fujitsu) cannot be excited - vertica does not like Fujitsu ;-) 15:18:44 <rhochmuth> correct bklei 15:18:59 <rbrndt> I'm running tests on the review in question right now. Functionally it looks fine, just testing performance on our loaded cluster 15:19:09 <bklei> you think the refactored query allows the optimizer to choose the right projection? 15:19:13 <bklei> or it doesn't matter 15:20:02 <rhochmuth> rbrndt? 15:20:28 <rbrndt> For the query we put the hint on? 15:20:52 <rbrndt> I think we can wait for the vertica fix, if it's not a prevalent problem 15:20:52 <bklei> right -- that query would be time qualified 15:21:29 <rhochmuth> we good? 15:21:42 <bklei> si 15:21:44 <rhochmuth> https://review.openstack.org/#/c/375717/ 15:22:16 <rhochmuth> yeah, so my only issue/concern is the number of dimensinos that are being added and potential performance hit 15:22:21 <bklei> me also -- i'll roll craig's feedback in today -- just an advertisement for publishing vm names and tenant names -- our infra engineers really want this one 15:22:33 <rhochmuth> those bastards 15:22:36 <bklei> valid concern. 15:22:58 <rhochmuth> can't they use a uuid 15:22:59 <bklei> should only change behavior if you add 'tenant_name' to metadata for either libvirt or ovs 15:23:00 <rhochmuth> by now 15:23:18 <rhochmuth> so, have you looked at performance at all 15:23:20 <bklei> and only perf hit for the keystone tenant list when cache gets updated 15:23:30 <bklei> tenant list is fast 15:23:31 <rhochmuth> or is it in the minutia 15:23:47 <rhochmuth> i see 15:23:58 <rhochmuth> well, i'll take a close look 15:24:04 <rhochmuth> i can't argue with it being useful 15:24:04 <bklei> we have something like 500 projects -- it returns fast 15:24:18 <bklei> one less translation step at 2:00 am :) 15:24:36 <rhochmuth> ok, i'll look it over 15:24:46 <rhochmuth> and get back with you if i have any other concerns 15:24:46 <bklei> many thanks -- i'll address craig's feedback today 15:24:51 <bklei> cool 15:25:12 <rhochmuth> #topic Ask Charter Communications about the size of their deployment 15:25:22 <bklei> size meaning? 15:25:25 <rhochmuth> i wanted to ask bklei about size of deployment 15:25:35 <rhochmuth> number of nodes, number tenants 15:25:40 <rhochmuth> if that could be shared 15:25:48 <bklei> generally yes 15:25:48 <rhochmuth> i'm just updating a slide for next week 15:25:54 <rhochmuth> metrics/sec 15:26:18 <bklei> 12K metrics/sec 15:26:30 <rhochmuth> wow, cranking it up since last time 15:26:32 <bklei> 105 Billion measurements in vertica 15:26:47 <rhochmuth> you'll pass mcdonald's soon 15:27:03 <bklei> :) according to vertica prof services -- we're small 15:27:04 <rhochmuth> how many compute nodes and vms? 15:27:08 <bklei> looking 15:27:57 <bklei> somewhere around 600 to 700 physical nodes, which run the agent 15:28:02 <bklei> that's two datacenters 15:28:21 <bklei> looking up vms/projects 15:28:24 <rhochmuth> 600-700 per two datacenters or aggregate 15:29:30 <bklei> aggregate 15:29:36 <rhochmuth> thx 15:29:42 <bklei> about 1000K vms per region/datacenter 15:29:54 <rhochmuth> awesome, thanks 15:29:59 <bklei> and about 250 customer projects per region/data center 15:30:09 <bklei> and another 50 per datacenter infra related 15:30:11 <rhochmuth> that is kinda in alignment with what we've been targeting with helion 15:30:16 <rhochmuth> you are a little higher 15:30:27 <bklei> colorado joke 15:30:40 <rhochmuth> but, we end up collecting a little more metrics per vm 15:30:50 <rhochmuth> our motto, you are a little higher 15:30:52 <haad1> hi 15:30:52 <tomasztrebski> that's the point where foreign key don't get a joke -_- 15:31:09 <rhochmuth> colorado is high in altitude 15:31:11 <bklei> recreational pot legal in colorado 15:31:18 <rhochmuth> that too 15:31:44 <rhochmuth> thanks bklei. 15:31:52 <bklei> yup 15:32:00 <rhochmuth> i'll add some details to one of the slides being presented in barcelona 15:32:11 <bklei> cool 15:32:20 <rhochmuth> #topic Lost & Forgotten [ help with testing, Tomasz :( ] 15:32:22 <bklei> i'll watch on the youtube 15:32:33 <rhochmuth> :-) 15:32:36 <rhochmuth> https://review.openstack.org/#/c/361318/ 15:32:51 <rhochmuth> should we +2 that one 15:32:55 <rhochmuth> seems like it is ready to go 15:33:34 <rhochmuth> rbrandt: is this one ready? 15:33:43 <rbrndt> Yeah, I think so 15:34:11 <rhochmuth> tomasz: are you ok with that? 15:34:17 <tomasztrebski> I was trying to test that, but seems like my monasca knowledge ..ehmm..sucks :D 15:34:34 <rbrndt> This one is a little deep into the testing/statistics call 15:34:45 <rbrndt> some weird behavior that needed to be cleaned up 15:34:53 <rhochmuth> if you have questions rbrndt is the expert 15:34:54 <tomasztrebski> well I brough it up for you (community) to look at, because I could not verify if this actually works or not 15:35:06 <tomasztrebski> no point for it to hang 15:35:39 <tomasztrebski> someone with better idea how this should work will do better job testing it I guess while I get more knowledge about it 15:36:03 <rhochmuth> i will test it later today, assuming i can get a devstack build to work again 15:37:51 <rhochmuth> https://review.openstack.org/#/c/315742/ 15:38:24 <tomasztrebski> same story from my side 15:38:29 <rhochmuth> rbrndt: so this one is ready too, i'm assuming 15:38:37 <rbrndt> Yeah, this one seems complete to me 15:38:48 <rhochmuth> just looking for another last sanity check 15:39:00 <rhochmuth> i'll try and get that done sometime today/tomorrow too 15:39:08 <rbrndt> tomasztrebski if we have specific queries that didn't work for you, I can try them here 15:40:47 <tomasztrebski> rbrndt I think that I did not spot the difference using any call using group_by with monascaclient 15:40:55 <tomasztrebski> though maybe client lacks something 15:41:00 <tomasztrebski> not really sure 15:41:32 <tomasztrebski> or maybe...again...I don't really understand how this should work 15:41:42 <rbrndt> Client should be up to date with that review, but I can double check that 15:41:52 <tomasztrebski> damn, I don't look very good here in public..telling all that :( 15:42:23 <tomasztrebski> rbrndt thx 15:42:58 <rhochmuth> rbrndt: so is the python-monascaclient all updated upstream and merged? 15:43:00 <haad1> guys does it make sense to put this patch into review process ? 15:43:08 <haad1> https://gist.github.com/haad/e31a94a3a36592952f64f6c2e47e3ab0 15:43:22 <haad1> monasca-persister will not start without it in mitaka branch 15:43:24 <rbrndt> to my knowledge the monascaclient is updated and merged for the group-by changes 15:44:10 <rhochmuth> haad1: if the persister is not starting in mitaka, we would glady accept the review 15:44:29 <haad1> ok then I will do it then 15:44:34 <rhochmuth> thanks 15:45:54 <rhochmuth> #topic Devstack [Openstack] integration 15:46:09 <rhochmuth> https://review.openstack.org/#/c/387195/ 15:46:23 <rhochmuth> how much faster is it? 15:46:34 <rhochmuth> but, it seems like a small change 15:46:41 <rhochmuth> i would have merged already if my devstack worked 15:46:49 <tomasztrebski> thx to Artur 15:46:54 <tomasztrebski> without flag: real 54m50.045s user 3m48.016s sys 0m17.780s 15:47:08 <tomasztrebski> with flag: real 38m50.415s user 3m53.688s sys 0m18.136s 15:47:22 <rhochmuth> wow, a lot faster 15:47:27 <tomasztrebski> once using git_clone enters mainstream branch it should speed up more 15:47:53 <rhochmuth> thx artur 15:47:54 <tomasztrebski> right now it only applies to all those projects that are cloned with git_clone wrapper, so all core openstack services 15:48:04 <haad1> https://review.openstack.org/388775 15:48:27 <rhochmuth> thx haad1 15:48:39 <rhochmuth> will review and merge asap 15:50:12 <rhochmuth> tomasz: i would merge https://review.openstack.org/#/c/387195/, but right now my devstack doesnt' come up do to the mysql issue i mentioned in one of the reveiws 15:50:21 <rhochmuth> do you know if that is resolved? 15:50:44 <tomasztrebski> https://review.openstack.org/#/c/388017/ 15:50:49 <tomasztrebski> answer lies here 15:51:05 <rhochmuth> i see, so if i test and have success with that review, i can merge it 15:51:12 <rhochmuth> then merge the other two 15:51:24 <rhochmuth> does that seem correct? 15:51:45 <tomasztrebski> yes, I just hope there are no merge conflicts 15:51:52 <tomasztrebski> if they are I will try to fix them ASAP 15:51:53 <rhochmuth> :-) 15:51:55 <rhochmuth> me too 15:52:06 <tomasztrebski> thx 15:52:07 <rhochmuth> ok, np 15:52:26 <rhochmuth> will do another devstack build and see what happens later today 15:52:40 <tomasztrebski> btw: I think that in overall status of monasca devstack has other areas to improve to match kind of standard way openstack proposes 15:53:20 <tomasztrebski> so if anyone has ideas, maybe we can follow up this topic to make monasca deployment there kind of more predictable for guys who now devstack approach and might expect certain things 15:53:27 <tomasztrebski> just an idea 15:53:36 <rhochmuth> i think it is a great area to cover 15:53:47 <rhochmuth> obviousely we weren't experts when we created that deploy 15:53:54 <rhochmuth> devstack deploy that is 15:54:03 <rhochmuth> and it sounds like there are more improvements to be made 15:54:29 <rhochmuth> is that work a separate discussion in the future 15:54:40 <rhochmuth> at a weekly meeting or in barcelona 15:54:43 <tomasztrebski> but now...tomasz arrived and he's breaking stuff up (yeah I know I am modest :P ) 15:55:25 <tomasztrebski> it might be, I have one bigger topic in terms of integration (however it is quite big and not really sure if I could work at this) 15:55:51 <tomasztrebski> I can add it to the design session if you like 15:56:08 <rhochmuth> sure, we can possibly cover that too 15:56:23 <rhochmuth> if we don't have time in a specific session we can cover while we are there 15:56:50 <rhochmuth> #topic Where to put examplary formatting files for notification=> https://review.openstack.org/#/c/349097/ 15:56:58 <rhochmuth> last one 15:57:01 <tomasztrebski> ok, just added it to the agend (not really sure if that fits there) but you can check out later 15:57:26 <tomasztrebski> yeah, I brought up a question where to put those examples for notifications methods 15:57:43 <haad1> I have one question 15:57:49 <haad1> can anyone tag new version on stable/newton and merge https://github.com/openstack/monasca-api/commit/e92a9524823863290065e781ad64b39086590647 there ? 15:57:49 <haad1> right now stable/newton is not running at all 15:58:42 <tomasztrebski> there are changes open with cherry pick on stable/newton 15:58:54 <tomasztrebski> might be the reason 15:58:54 <rhochmuth> haad1: i can do that 15:59:19 <haad1> rhochmuth: thanks 15:59:28 <rhochmuth> i'll need to look at it closer though after meeting 16:00:22 <rhochmuth> oops i need to end the meeting 16:00:27 <rhochmuth> just saw the clock 16:00:32 <rhochmuth> thanks everyone 16:00:35 <pratid> bye 16:00:35 <tomasztrebski> thx 16:00:36 <Kamil_> bye 16:00:38 <tomasztrebski> cheers and bye 16:00:39 <shinya_kwbt> bye 16:00:42 <bklei> cya 16:00:49 <rhochmuth> #endmeeting