15:00:24 #startmeeting monasca 15:00:25 Meeting started Wed Sep 9 15:00:24 2015 UTC and is due to finish in 60 minutes. The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:30 The meeting name has been set to 'monasca' 15:00:34 o/ 15:00:37 o/ 15:00:42 roll call 15:00:44 o/ 15:00:46 o/ 15:00:50 ajo: oops, totally missed your ping, anyways sounds like progress is being made, thats all good, happy to review a spec when it arrives 15:00:55 o/ 15:01:01 o/ 15:01:27 hello all 15:01:39 so, no one posted any agenda items 15:01:47 we need to start getting better at that 15:01:52 oops 15:02:09 we can discuss an agenda now 15:02:22 what would folks like to talk about 15:02:30 I can give an update on Ceilosca 15:02:39 if people are interested 15:02:40 :-) 15:02:43 +1 15:02:45 sounds good 15:02:47 ok, that sounds great, i'm interested 15:02:54 i've added to the etherpad at 15:03:10 i can give an update on TWC perf issues 15:03:16 https://etherpad.openstack.org/p/monasca-team-meeting-agenda 15:03:40 ok, i've added those to the etherpad 15:03:44 i've also added governance 15:04:04 johnthetubaguy: we will need some talk nova/neutron before a spec :) 15:04:12 I've got some question about Horizon on Angular. This has been discussed at the mid-cycle 15:04:13 to start on the good direction 15:04:28 I could write a very high level one as I don't know the low levels of nova 15:04:29 what is nova/neytron before a spec 15:04:31 if that's enough... I could do 15:04:57 ajo -- is this topic for monasca weekly meeting? 15:04:58 ok mroderus, hofizon andulga sounds good too 15:05:16 this is the weekly monasca meeting 15:05:29 ok, let's cover Ceilosca first 15:05:31 ajo: johnthetubaguy can you please move your conversation on #openstack-dev? 15:05:34 #agenda Ceilosca 15:05:42 ok 15:05:44 did i change topics correctly? 15:05:49 fabiog take it away 15:05:59 I think is #topic 15:06:05 but nevermind 15:06:09 #topic Ceilosca 15:06:17 ther u go 15:06:17 now is correct ;-) 15:06:21 sorry bklei , fabiog , *, I didn't realize I was on the meeting channel :/ 15:06:22 yes 15:06:32 ajo: np 15:06:42 so we have been working on the installer part 15:06:51 this is the location of the main repo 15:06:59 #link https://github.com/stackforge/monasca-ceilometer 15:06:59 o/ 15:07:15 but if you go in the #link https://github.com/stackforge/monasca-ceilometer/tree/master/deployer 15:07:47 deployer folder you can find a ceilosca.sh script that installs devstack enabling Ceilosca and performs a full installation of Monasca on a single VM 15:08:04 ok 15:08:06 in this way you can run it in a VM on your local box or in an Instance in the cloud 15:08:15 nice 15:08:21 it will take a while, but it is fully automated 15:08:44 so I would encourage people on the team to give it a go and provide feedback 15:08:55 so, this is installing devstack and monasca and deilosca? 15:09:03 yes 15:09:18 at the end you get a Ceilometer functionality but the data is stored in Monasca 15:09:22 in Influx DB 15:09:25 cool 15:09:32 is the entire Telemetry API covered? 15:09:43 I would say around 80% 15:09:48 that was my next request 15:10:09 if there are people interested in extend the API coverage please feel free to submit patches 15:10:23 so, the areas that are no covered 15:10:27 are? 15:10:36 in this file #link https://github.com/stackforge/monasca-ceilometer/blob/master/ceilosca/ceilometer/storage/impl_monasca.py 15:10:38 meta data, and alarms 15:10:43 there is the list of supported features 15:11:01 to clarify, Ceilosca only addresses Meters and Samples as today 15:11:06 so no Alarms or Events 15:11:28 is the meta data supported? 15:11:31 We could port events to Monasca now that there is an Event API 15:11:37 but that work has not been done yet 15:11:40 yes 15:11:48 you can perform queries on metadata 15:11:55 Pauline added support for that 15:12:00 in Resources and Samples 15:12:08 how do you describe the difference between what ceilosca.sh and monasca-vagrant 15:12:26 does? 15:12:47 ceilosca.sh is a bash script that requires at least to have Ubuntu and git installed 15:13:03 and then is a combination of shell commands and ansible 15:13:32 ok 15:13:32 the vagrant will do a similar thing but creating the instance 15:13:54 so if you have already a VM or Instance use Ceilosca.sh 15:13:59 i'm wondering how far we can leverage ceilosc.sh to integrate in with DevStack 15:14:24 some of the feedback from the TC geovernance review was that they would like to see Monasca in DevStack 15:14:38 do you think using ceilosca.sh would be a good start 15:14:57 actually you should look at #link https://github.com/stackforge/monasca-ceilometer/blob/master/deployer/devstack/ceilometer 15:15:10 that is the modified Devstack Ceilometer file 15:15:26 it will make all the changes to become Ceilosca during Devstack deployment 15:15:44 so instead of installing vanilla Ceilometer and then converting it into Ceilosca 15:15:52 it starts as Ceilosca 15:15:55 :-) 15:16:18 any questions from the team? 15:16:35 or you guys are cloning the repo :-) ? 15:16:36 i'll need to start digging in a little further into this 15:16:54 me too, will play with it in a week or two hopefully 15:17:00 i haven't tried to install yet, but that will be one of the items next on my list 15:17:04 great 15:17:14 do you have any more performance updates 15:17:21 if any of you can do some performance comparison tests 15:17:22 on ceilosca? 15:17:26 would be great 15:17:34 are you planning any 15:17:39 I am going to work on that next, but more sources is perfect 15:17:42 yes 15:17:55 I want to do a Ceilosca vs. Ceilometer with MongoDB 15:18:09 load tests and then use Rally for query performance test 15:18:10 sounds good, that seems to be the most important area 15:18:13 we just started 15:18:21 so any help is really appreciated 15:18:32 did you see gordon's commends related to ceilosca? 15:18:40 I did 15:19:33 ok, do we want to communicate more with ceilometer on this? 15:19:42 what can I say ... There have been several attempts to work together with Ceilometer. In Vancouver I asked the same question again and the answer was that because Monasca has java it is not possible 15:20:08 yes, understand 15:20:20 yes, we can 15:20:28 there is a meeting tomorrow 15:20:37 I can add that to the agenda 15:20:47 fabiog, so come back when there's no more required java dependency? 15:20:49 i wont' be able to attend in all day meetings 15:21:01 jimbaker: that was the kind of answer 15:21:33 well, i think we're doing an incredible amount of work to make Ceilometer viable by addressing the performance and scale issues 15:21:40 rhochmuth: we can add that to the following week 15:22:07 I spoke about Ceilosca when Ildiko and Gordon joined the meeting via Webex 15:22:19 as well as consolidate systems, so we don't have multiple services/components to support 15:22:35 yes, they are claiming that we didn't tell them anything 15:22:49 but this has been an area that has been communicated for a while 15:22:55 the Ceilosca name is new 15:23:09 rhochmuth: is all in the name :-) 15:23:09 but that was just a name that we came up with that was pretty cool 15:23:17 yeah 15:23:19 just good shorthand name 15:23:36 anyway, more communication would be good 15:23:47 jimbaker: maybe Monometer would have been less catchy and less problematic :-) 15:23:53 does the ceilometer project have any new performance numbers to share 15:23:58 :) 15:24:04 any deployments of significant size 15:24:11 for example, there is cern 15:24:15 rhochmuth: not that I am aware of 15:24:29 ok, that would be good information to have 15:24:31 rhochmuth: yes, I spoke with Cern, they basically have 2 Ceilometers 15:24:49 1 for alarms where they store data for less than 1 day 15:24:56 that is useless 15:25:09 and one for real metering where they store bare minimum data for 30 days 15:25:18 well that is harsh, that doesnt' meet our needs 15:25:30 and they have the big advantage of not running any Swift that is the real Ceilometer killer 15:25:37 fabiog, what's the ingest rate for CERN? 15:25:53 jimbaker: I don't have that number 15:26:07 fabiog, no worries - just want to multiply out one day's worth, etc 15:26:28 but I know that they had to have a separate RabbitMQ because with the Ceilometer re-publishing where going over the 18Mil messages and killing the queue 15:26:38 well, i'm wondering if ceilometer shoudl be considering ceilosca more closely 15:26:46 18 million/day? 15:26:58 they where trying to do 60s with ceilometer compute agent 15:27:03 18 M/day is not much 15:27:18 18Mil concurrently on the queue 15:27:33 basically the production rate was too big compared to what Ceilo could consume 15:27:38 ahh, got it 15:27:46 and the queue would fill fast and die after 18Mil messages 15:27:57 then is when they split the alarms from metering 15:28:11 because they needed the 60s interval for alarming and I guess autoscaling 15:28:28 but they use a once a day for the metering stuff (or something like that) 15:28:36 anyway .. enough of Ceilosca 15:28:50 any last question? 15:29:02 yes, so let's go connect with them on this and also get latest performance and deployment information on Ceilometer 15:29:11 seems like that data shoudl be readily available 15:29:15 at this point in time 15:29:47 #topic performance 15:30:02 that's me, yes? 15:30:09 that is u 15:30:40 ok -- quick overview for the group, TWC has a blocker issue from rolling out MaaS for our internal groups 15:30:57 vanilla use case -- display stock libvirt metrics via grafana 15:31:26 with less than 2 weeks worth of data, and a handful of instances in a project, takes up to a minute to render the page 15:32:01 you are about to say all the issues are solved, right? 15:32:12 bklei: is this a Vertical problem only or you think the same issue will appear using InfluxDB? 15:32:17 we're using vertica currently, biggest time spend on a query to figure out which definition_dimension ids to use in the measurements table query 15:32:23 not sure fabio 15:32:28 probably vertica only 15:32:35 so impacts HP Helion, TWC for now 15:32:51 so, we've got a list of potential solutions 15:32:53 not solved yet rhochmuth -- but progress 15:33:00 i tried 15:33:13 should we post your notes in this meeting 15:33:22 yes -- and further background, rbak and I were onsite at HP on Friday, thx for the meeting roland/deklan 15:33:26 from last Friday's face to face meeting 15:33:36 sure -- one sec 15:33:51 yes, i also got feedback for the governance TC about the face to face 15:33:53 my notes 15:33:59 Next steps: 15:33:59 1. Test changing grafana to pass merge flag to the statistics query 15:33:59 after the initial metrics: 15:34:01 1.1 Ryan to change grafana to pass merge flat 15:34:03 1.2 Brad to change the api to not do getdefids() with merge flag -- 15:34:05 might be harder than it seems, need thd defintion_dimension_id's 15:34:07 2. TWC to Fix current projections (Brad). 15:34:09 2.1 Change dimension projection to order by: 15:34:11 name, value, dimension_set_id 15:34:13 so, in the future we might not be able to have face to face meetings or use video conferencing 15:34:13 2.2 Change measurement projection to order by: 15:34:15 definition_dimension_id, time_stamp, value_meta 15:34:17 3. We're confused why the inner joins operate as poorly as they do 15:34:19 * Brad to open vertica support ticket and get feedback. 15:34:21 4. Deklan to code up the idea of splitting the dimensions table into 15:34:24 two, 15:34:25 removing the multiple of inner joins. 15:34:27 * TWC to come up with migration sql 15:34:29 5. Brad to experiment with adding constraints on each table to allow 15:34:31 pre-join projections (possible code ordering change). 15:34:33 (add PK=FK relationships) 15:34:35 6. FUTURE: Change to grafana to only do incremental statistics queries 15:34:37 for the elapsed time and append graph. 15:34:39 7. FUTURE: Add an API cache layer in front of (or part of API 15:34:41 to only query db once, and return from memory for subsequent 15:34:43 queries) 15:34:45 agreed 15:34:47 anyhoo, update on above list 15:34:55 #1 didn't pan out, still need to get ids for measurements lookup (bummer) 15:34:59 yes, please update 15:35:18 I'm working on #2, but no smoking gun there 15:35:34 #4 is not necessary given the latest query that we discussed 15:35:35 i opened vertica support ticket for #3 -- providing data over the past day 15:35:42 what about #5? 15:35:47 agree deklan 15:36:21 #5? 15:36:25 I'm working on #5 today, but using deklan's suggestion, can modestly improve stuff with https://review.openstack.org/#/c/221492/ 15:36:41 ok. 15:36:42 see what you think of that new query without multiple inner joins 15:36:51 bklei: did you get the projections right? 15:36:53 and -- I'll experiment with constraints today 15:37:09 i didn't get a chance to review 15:37:12 sorry 15:37:22 i re-ran the offending query through the DBD -- it suggestion segmented projections which were much worse :) 15:37:23 ddieterly did you review 15:37:31 have not had time 15:37:45 ok, so no feedback on your review 15:37:52 i would expect blazing peformance on that new query 15:37:52 i have another idea to add to the list 15:38:01 yes, i woudl too 15:38:05 update from the vertica engineer, he thinks there may be a bug in my current vertica version, providing a full dump to him today 15:38:13 sure, go for it 15:38:15 projections have to solve this 15:38:26 hmmm, that is interesting 15:38:58 anyway, the other idea was to cache to metric_defintion_dimension_ids 15:39:01 agree that projections should solve rhochmuth -- i'm hopeful for pre-join projections 15:39:34 so, if you have a query for some metric, that have a bunch of dimensions specified, you would cache that, as well as the exact ID, in memory 15:39:47 this would avoid the joins altogether 15:40:00 yeah -- that's a good idea too, completely skip going to the db for stuff 15:40:00 what do you think? 15:40:15 and it shouldn't be a huge list 15:40:20 correct 15:40:54 can give that more thought, but likely won't get to it this week, and am out next 15:40:56 a simple map of ((region, tenant id, metric name, dimensions), id) would work 15:41:10 yeah, that one woudl take more time to code 15:41:20 it isn't hard, but it isn't trivial either 15:41:35 right, but lots of bang for that buck 15:41:37 i think you are on the right path though 15:41:47 with the projectsions and potential vertica problem 15:41:57 yes, will keep pushing on that 15:42:03 so, best to continue with that, and then we can clook at caching it that doesnt' pan out 15:42:09 bueno 15:42:24 but, i'll be amazed if this isn't a projection issue or Vertica session bug as you say 15:42:34 i think that's my update -- any other questions 15:42:38 ditto 15:42:49 no further questions your honor 15:42:58 dismissed :) 15:43:04 #topic governance 15:43:25 MOnasca was not approved yesterday 15:43:37 the main issues appear to be 15:43:37 :( 15:43:45 IRC meetings 15:43:49 DevStack 15:43:53 CI/CD 15:44:05 at least those are the tansibgle ones that I took away 15:44:35 ddieterly do you other insights 15:44:56 one of the issues appears to be the face to face that we had on Friday 15:45:12 and the fact that we conferenced in Fabio for this too 15:45:26 no, you can read the summary in the patch 15:45:40 rhochmuth: hey .. you have to stop blaming me for everything :-) 15:45:53 lol 15:46:03 anyway, i'm trying to get further clarification on this 15:46:10 what's the issue with the f2f? everyone was invited to attend via webconf 15:46:23 that is what i thought 15:46:39 was this f2f critizised by anyone? 15:46:44 and I was the only one that was interested ... 15:46:59 no, everyone was invited that attended last weeks Monasca meeting in IRC 15:47:16 we were looking at Grafana, code, sql, … 15:47:21 does anyone know how to do that in IRC 15:47:30 if that wasn't clear, apologies to anyone who wanted to attend remotely and didn't get that message 15:47:56 but we bored fabiog within minutes :) 15:48:07 so, anyway, looks like we'll need to get further clarification 15:48:13 yes, that's true :-0 15:48:23 i thought it was just the weekly meetings that needs to be in IRC 15:48:39 rhochmuth: I think we can have other meetings 15:48:48 at least we did them in Ceilometer 15:48:53 That is what I thought too 15:49:03 my suggestion is to send an email in openstack-dev 15:49:04 but that wasn't the feedback 15:49:13 and post the link in this meeting 15:49:16 yeah, i was thinking about that too 15:49:16 my take -- as long as we advertise one-off meetings and allow remote attendance, should be OK 15:49:18 so there is a trace 15:49:25 yes 15:49:31 bklei: I agree 15:49:44 so is anyone interested in adding MOnasca to DevStack? 15:49:53 that is tangible and somethign we can act on? 15:50:14 DevStack integration will be the first step for integration testign via Tempest tests 15:50:24 we have Tempest tests in a separate repo 15:50:29 interested, but no BW 15:50:42 so getting those merged in with the Tempest repo would also be a good step 15:51:13 rhochmuth: so you will submit a patch of the existing one in the correct repo? 15:51:29 sorry, i dont' understand the question 15:51:40 fabiog 15:51:52 rhochmuth: you said that the tempest test for Monasca are available but in a separate repo 15:52:06 rhochmuth: does that mean simply to submit them all in the main tempest repo? 15:52:17 we forked in Github, just to get the development done 15:52:25 i believe you are correct 15:52:34 so, that should be the easy part 15:52:43 rhochmuth: right 15:53:01 ok, i'll try and get some resources on DevStack integration from HP 15:53:33 we are having overall planning meetings, so maybe there will be some availability 15:53:42 or, i can write some code... 15:54:03 new topic? 15:54:15 #topic horizon 15:54:27 I guess that's me 15:54:33 yes 15:54:48 i think we're going to run out of time 15:54:59 I remember that there were some discussions about Angular in Horizon and that our plugin may be affected 15:55:13 I was just curious if there are any links on more information on that 15:55:16 i'm not sure what the impact is 15:55:36 bklei do you know 15:55:43 can you check with Eric P. 15:55:52 if you don't know 15:56:01 is the move to Angular (or the plans for it) documented somewhere? 15:56:12 ok, thanks 15:56:14 this is a big topic in Horizon 15:56:24 so, it is discussed there 15:56:26 he tells me i don't want to know the answer to the question :) 15:56:33 lol 15:56:45 ask him if our panel will continue to work 15:57:00 is the answer no? 15:57:13 he doesn't think it'll break us 15:57:19 cool 15:57:32 ok, sounds good 15:57:39 we will need to test adn watch this 15:57:42 i guess we've got some angular stuff in our test env, and monasca-ui still works 15:57:59 bklei are you guys tracking this 15:58:15 as in using the latest horizon and involved with moves to angular? 15:58:18 ok, so if I have any specific questions, I will ask Eric P. 15:58:24 i guess indirectly by pulling in the latest horizon, we are 15:58:33 yes mroderus 15:58:34 yeah, that is waht I meant 15:58:43 ok, time is almost up 15:58:51 please add agenda items for next week 15:58:59 we have one item that we didn't get too 15:59:04 Grafana 15:59:11 In the last minute I just wanted to ask if anyone if working on grafana 2.0, and if so ping me. 15:59:15 so, we'll need to cover next week 15:59:30 not me 15:59:39 is fujitsu? 15:59:42 we not yet, but we're planning to 16:00:06 Alright, we're going to start looking at that in the next couple days at TWC 16:00:17 (thumbsup) 16:00:22 i think we are done 16:00:42 thx rhochmuth 16:00:51 thanks everyone 16:00:54 by everyone 16:00:55 thank you all 16:01:33 #endmeeting