15:00:06 <rhochmuth> #startmeeting monasca
15:00:07 <openstack> Meeting started Wed Sep 23 15:00:06 2015 UTC and is due to finish in 60 minutes.  The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:10 <openstack> The meeting name has been set to 'monasca'
15:00:15 <rhochmuth> roll call
15:00:25 <fabiog> o/
15:00:26 <jimbaker> o/
15:00:27 <rhochmuth> o/
15:00:31 <bklei> o/
15:00:34 <qwebirc77657> o/
15:01:04 <rhochmuth> Agenda looks a little light today at, https://etherpad.openstack.org/p/monasca-team-meeting-agenda
15:01:17 <rhochmuth> 1.	Review Gerrit bugs
15:01:17 <rhochmuth> 1.	Need more folks contributing to reviews
15:01:17 <rhochmuth> 2.	Update on DevStack
15:01:17 <rhochmuth> 3.	Update on Tempest Tests
15:01:37 <rhochmuth> Are there other items to discuss
15:01:48 <rhochmuth> How about an update on performance from TWC
15:01:53 <ddieterly> o/
15:01:53 <rhochmuth> as well as Grafana
15:01:58 <bklei> sure we can do that
15:02:27 <fabiog> I also have an update on Ceilosca
15:02:31 <rhochmuth> is anyone attending from fujitsu?
15:02:40 <rhochmuth> ok, let's talk about ceilosca too
15:02:48 <bklei> can we add timing of next tagging of monasca-agent and pub to pypi?
15:02:49 <witek> yes, but I have to leave earlier
15:04:15 <rhochmuth> ok, we'll discuss agent tags too
15:04:15 <rhochmuth> #topic reviews
15:04:16 <rhochmuth> One area I want to get help on is more reviews
15:04:47 <bklei> are people getting added to reviews, but not reviewing?
15:04:49 <witek> i think, i could help a bit
15:05:00 <rhochmuth> there are a bunch of reviews that I've +1'd, but have been just sitting for a couple of days
15:05:19 <rhochmuth> i would like to see the code flowing
15:05:22 <bklei> twc can be better too
15:06:04 <rhochmuth> in the old days, i would just +2 things, but it align with the overall process better we need multiple companies one each review commit
15:06:24 <bklei> agree
15:06:29 <ddieterly> i think we also needed automated tests
15:06:43 <rhochmuth> ideally, a company shouldn't be +2 their own work, but we're not quite there yet
15:06:46 <ddieterly> we just don't have enough tests to feel confident in submittals
15:06:50 <rhochmuth> but we have to move in that direction
15:06:51 <fabiog> ddieterly: +1 Every patch should have unit test
15:07:19 <ddieterly> i'd also like to see integration tests
15:07:50 <rhochmuth> ddieterly: so the integratino tests is what i'm working on
15:07:54 <rhochmuth> right?
15:08:10 <rhochmuth> that is what tempest tests are for
15:08:10 <ddieterly> yea, not sure how much coverage we have on those
15:08:51 <rhochmuth> well, i think coverage will be good at the API syntactic level
15:09:04 <rhochmuth> and basic functionality
15:09:14 <ddieterly> do the tempest tests use a 'real' db?
15:09:21 <rhochmuth> yes, they do
15:09:26 <rhochmuth> tempest uses the API
15:09:34 <rhochmuth> they are written against the Monasca API
15:09:44 <ddieterly> cool
15:10:02 <rhochmuth> so, getting back to reviews, just need more help in that area
15:10:28 <rhochmuth> witek, are all of tomasz's reviews then ready to get committed?
15:10:59 <witek> i will take a look at them again
15:11:04 <rhochmuth> he has about 5 or 6 reviews sittign in the queue,
15:11:05 <rhochmuth> thanks
15:11:25 <rhochmuth> sorry about the speed, we actually test each commit prior to +2
15:11:29 <rhochmuth> that is the theory
15:11:39 <rhochmuth> so the turn-aroudn time can be gated by testing
15:11:53 <rhochmuth> hopefully, when we have tempest tests up and running this will be automated
15:12:08 <rhochmuth> so, we'll have a little more confidence
15:12:15 <bklei> +1 on that
15:12:25 <fabiog> rhochmuth: in the meantime can you please post a comment when the tests have been done and were successful?
15:12:29 <ddieterly> are we going to require tempest tests for each patch?
15:12:30 <rhochmuth> #topic devstack
15:12:42 <fabiog> rhochmuth: I think this will give more confidence to the other reviewers that the patch works
15:12:42 <rhochmuth> sorry, i changed topics
15:12:54 <fabiog> np
15:12:58 <witek> #help Help is needed on reviews
15:13:20 <rhochmuth> ddieterly i think i need to get the first round of tempest tests done and integrated
15:13:40 <rhochmuth> then we can hopefully have a process where each feature has tests written againt it
15:13:49 <rhochmuth> so, i think the answer is yes
15:13:55 <ddieterly> +1
15:14:03 <jimbaker> +1
15:14:07 <ddieterly> what is the eta on the tempest stuff?
15:14:31 <rhochmuth> i'm going to try and have some code up for review this week
15:14:42 <rhochmuth> i have tests for metrics and alarm definitions complete
15:14:49 <rhochmuth> will work through the rests of the API
15:14:59 <rhochmuth> actually, i don't think everyhting will be done this week
15:15:15 <rhochmuth> but it will be a good start and time to merge and get feedback
15:15:35 <rhochmuth> #topic devstack
15:15:57 <rhochmuth> how is devstack going?
15:15:59 <ddieterly> ok, i'm in the process of converting all the ansible to bash in monasca-api/devstack
15:16:23 <ddieterly> hope to have a complete monasca deployment in devstack in 2-3 weeks
15:16:27 <rhochmuth> sounds exciting
15:16:33 <ddieterly> very
15:16:39 <rhochmuth> :-)
15:16:59 <rhochmuth> so, you've put some reviews up
15:17:10 <ddieterly> yea, already have
15:17:31 <fabiog> can you post it please?
15:17:40 <rhochmuth> #fabiog #ddieterly was asking about using ansible
15:17:47 <rhochmuth> i said can't do that
15:18:00 <Matthias_Christi> what is the reason to convert ansible to bash ?
15:18:01 <rhochmuth> has to be independent of any deployment tool
15:18:14 <ddieterly> https://review.openstack.org/#/q/owner:%22Deklan+Dieterly%22+status:merged,n,z
15:18:45 <rhochmuth> #Matthias_Christi i think the problem is they don't want Ansible in Devstack
15:18:51 <ddieterly> Matthias_Christi: it is the devstack way
15:19:09 <rhochmuth> #fabiog is that your understanding
15:19:17 <ddieterly> bash is the linqua franca of devstack
15:19:41 <jimbaker> for better or for worse...
15:20:25 <ddieterly> yea, c'est la geurre
15:20:25 <fabiog> rhochmuth: yep
15:20:46 <rhochmuth> so, basically 2-3 weeks on devstack, then we'll move onto ci/cd integration
15:20:58 <rhochmuth> ok, next topic?
15:21:08 <ddieterly> hopefully
15:21:15 <rhochmuth> we discussed the tempest tests already
15:21:22 <rhochmuth> so i'll move to performance update
15:21:26 <rhochmuth> #performance
15:21:31 <rhochmuth> #topic performance
15:21:47 <bklei> that's me
15:21:54 <rhochmuth> yes
15:22:10 <bklei> was out last week, in a conf this week, so not much movement, but
15:22:20 <bklei> vertica has agreed current perf is a bug
15:22:28 <rhochmuth> awesome!
15:22:35 <bklei> no fix yet
15:22:44 <rhochmuth> do they have an eta
15:23:04 <bklei> the bug is when 10 or so queries are in parallel
15:23:04 <bklei> no eta yet, pushing them on that
15:23:23 <rhochmuth> what do you mean by 10 queries in parallel
15:23:35 <bklei> asking them to back port to the current version (hot fix), instead of waiting for Q4 major release
15:23:46 <bklei> yes, 10 queries in parellel goes into the toilet
15:24:08 <ddieterly> that explains our puzzlement over the performance at least
15:24:08 <rhochmuth> omg
15:24:15 <bklei> so, i'm waiting on other projection work, until that's fixed
15:24:27 <bklei> (pre-join queries, schema change...)
15:24:32 <rhochmuth> your review is sitting up there, should we wait on that
15:24:40 <fabiog> bklei: wow! and no customers found that before ?
15:24:44 <bklei> exactly ddieterly
15:24:56 <rhochmuth> that switches from inner joins to ...
15:25:02 <bklei> apparently there is one other big customer.  but boo on their testing
15:25:04 <rhochmuth> should we wait on that
15:25:05 <ddieterly> i'm surprised that vertica had this bug
15:25:10 <ddieterly> kinda serious
15:25:48 <ddieterly> the vertica luster has been dulled a bit
15:25:58 <rhochmuth> also, would limiting the concurrency to <10 be a work-around
15:26:24 <rhochmuth> doesnt' sound like a good idea, but maybe it is faster
15:26:31 <bklei> the bug# is  VER-40005
15:26:45 <bklei> "Time in plan is high when 10+ queries are run concurrently on 3 node cluster"
15:27:01 <bklei> but the engineer thinks it's beyond just a 3 node cluster
15:27:23 <bklei> anyhoo, not much more to say on that till i get an eta
15:27:32 <bklei> will pressure them
15:27:51 <rhochmuth> do you want your review for inner join conversion to just wait then
15:28:09 <bklei> let's wait, not sure it's necessary, may be working around their bug
15:28:25 <rhochmuth> ok
15:28:29 <bklei> thx though
15:28:45 <rhochmuth> #bklei thanks for working through this
15:28:46 <bklei> it wasn't a dramatically better improvement
15:28:49 <bklei> sure
15:29:15 <ddieterly> i think we should still make the change because it is much simpler to implement and reaon about
15:29:17 <rhochmuth> nice to know we know how to code and that the bug was in Vertica
15:29:39 <ddieterly> we just need to make the change globally
15:30:37 <bklei> i'm not opposed to the change #ddieterly, just want a bug fix in vertica before spending much time on tweaking stuff above it
15:30:48 <ddieterly> yea, understood
15:31:01 <rhochmuth> so let's see how fast vertica can respond
15:31:14 <rhochmuth> and then we can measure again with fixes hopeflly
15:31:20 <rhochmuth> and then switch to new code
15:31:20 <bklei> agreed, hope to have better info next week
15:31:31 <rhochmuth> ok, cool
15:31:32 <bklei> yup
15:31:46 <rhochmuth> #topic grafana
15:32:02 <rhochmuth> rbak was adding support to Grafana 2.0
15:32:17 <rhochmuth> #bklei do you know how that is progressing?
15:32:19 <bklei> rbak is sort of here, but in a conf with me -- he's made some progress
15:32:31 <rhochmuth> ok
15:32:43 <bklei> more investigation, no eta or idea of the quantity of work yet.
15:32:54 <rhochmuth> ok
15:32:56 <bklei> but it's sexy, and i'm confident he'll get it done
15:33:12 <rhochmuth> sexy
15:33:14 <bklei> templates are awesome
15:33:15 <ddieterly> bklei: really?
15:33:35 <rhochmuth> ok, we'll check back next week
15:33:39 <bklei> very cool, and it doesn't sound too difficult to add us as a proper 'data source'
15:33:42 <bklei> yup
15:34:01 <rhochmuth> #topic ceilosca
15:34:32 <rhochmuth> #fabiog you are up
15:35:01 <ddieterly> fabiog: you there?
15:35:12 <ddieterly> crickets
15:35:14 <fabiog> rhochmuth: sorry
15:35:36 <fabiog> ok, so we have started to do tests with Ceilosca
15:35:54 <fabiog> and what we found is that Influlx 0.9.1 suffers all sort of issues
15:36:06 <fabiog> it times out with the Persister
15:36:15 <bklei> :(
15:36:17 <fabiog> and it prevents to store large amounts of data
15:36:29 <fabiog> but when we updated to 0.9.4.1 it worked really well
15:36:43 <fabiog> and we didn't have to change a single line of code or configuration
15:36:44 <rhochmuth> problem solved :-)
15:36:49 <rhochmuth> are we done
15:36:50 <fabiog> just switched the package
15:36:56 <ddieterly> fabiog: did you have a chance to test the api end points with 9.4?
15:36:56 <fabiog> rhochmuth: kind of
15:37:15 <ddieterly> i'm concerned that the api might be broken with 9.4
15:37:21 <fabiog> ddieterly: yes, because the ceilometer call in returns call the monasca API
15:37:25 <rhochmuth> so, there were changes i thoguht in .9.3 that were incompatible with .9.1
15:37:27 <fabiog> and we get data back
15:37:47 <rhochmuth> #bmotz was looking at this
15:37:48 <ddieterly> we need to check all the functionality of the end points
15:38:01 <fabiog> so I would like to see the mini-mon installation to be upgraded to use 0.9.4.1
15:38:07 <fabiog> would that be possible?
15:38:15 <ddieterly> fabiog: go for it
15:38:36 <rhochmuth> i don't have a problem updating
15:38:43 <fabiog> ddieterly: ok, so I will try to figure out what and where to change and I will submit a patch
15:38:44 <rhochmuth> just have to address any issues that occur
15:38:58 <rhochmuth> influxdb keeps changing their api, even though they are supposed to be frozen
15:39:00 <ddieterly> it is a very simple change in the ansible vars
15:39:13 <rhochmuth> yeah, so the change in Ansible is trivial
15:39:20 <ddieterly> the problem is finding all the problems
15:39:34 <ddieterly> goes back to comprehensive integration tests
15:39:51 <rhochmuth> #bmotz isn't here today, but you might contact directly to see what he might have uncovered
15:40:02 <fabiog> ddieterly: right. In the ceilosca case seems to work fine, so if there are bugs are not major
15:40:08 <rhochmuth> he was also looking at swithching from http to binary protocol
15:40:12 <fabiog> ddieterly: meaning stop working
15:40:51 <fabiog> rhochmuth: ok, I will get in touch with bmotz
15:40:57 <rhochmuth> thx
15:41:09 <ddieterly> does cceilosca exercise a good portion of the api?
15:41:11 <rhochmuth> if it is all working, then let's switch
15:41:24 <rhochmuth> we can also help to get some testing here
15:41:28 <rhochmuth> possibly
15:41:32 <fabiog> ddieterly: pretty much queries with dimensions
15:41:36 <rhochmuth> i'll need to coordinate with our test team
15:41:40 <fabiog> ddieterly: and post of new measurements
15:41:41 <ddieterly> alarm history, metrics, measurements, statistics
15:41:59 <fabiog> ddieterly: no alarm history or anything related to alarms
15:42:03 <fabiog> the rest yes
15:42:13 <ddieterly> merge-metrics flag, all parameters, etc
15:42:20 <rhochmuth> pagination
15:42:24 <fabiog> ddieterly: probably not all
15:42:29 <ddieterly> oh, yea!
15:42:30 <fabiog> and definitely not pagination
15:42:39 <ddieterly> how in the world did i forget that one
15:42:43 <rhochmuth> what, how can you forget paginatino
15:42:54 <rhochmuth> sorry, inside joke
15:43:57 <rhochmuth> #fabiog so i think plan to to upgrade
15:44:07 <rhochmuth> check with ben motz at cray
15:44:22 <rhochmuth> we'll need to get some testing done
15:44:33 <rhochmuth> to validate completely
15:44:35 <fabiog> rhochmuth: I will send an email in the mailing list asking him for issues
15:44:42 <rhochmuth> thx
15:44:52 <fabiog> but 0.9.4.1 is a huge step forward in performance
15:45:00 <rhochmuth> awesome!
15:45:04 <bklei> +1
15:45:12 <ddieterly> you might want to ask the influxdb guys if there is any compatibility issues that they know of
15:45:14 <rhochmuth> sounds like they are finally getting the streamign raft stuff done
15:45:34 <rhochmuth> #ddieterly can you send an email and cc folks
15:45:46 <ddieterly> sure
15:45:49 <rhochmuth> thx
15:46:01 <rhochmuth> #fabiog wnat are you seeing in performance
15:46:08 <rhochmuth> with the latests influxdb
15:46:42 <fabiog> rhochmuth: I see that Ceilosca is around 30/40% faster
15:46:57 <fabiog> rhochmuth: and it seems that is getting better when the data set is bigger
15:47:03 <fabiog> which is what I was expecting
15:47:10 <rhochmuth> cool
15:47:18 <fabiog> right now we are doing a sort of worst case scenario test
15:47:42 <fabiog> like we are doing open ended query of get samples with 1M measurements
15:47:53 <fabiog> so we need to switch to CURL instead of cello client
15:48:00 <fabiog> because the client times out
15:48:14 <fabiog> but up to 750K the client worked
15:48:20 <fabiog> in both cases
15:48:30 <fabiog> and in both cases failed at 1M
15:48:37 <rhochmuth> how is ceilosca comparing to native ceilometer
15:48:48 <rhochmuth> with the latests influxdb
15:48:58 <fabiog> my preliminary estimates is that is between 30 to 40% faster
15:49:08 <rhochmuth> nice
15:49:23 <fabiog> but I think the real advantage will be in the amount of data we can store in Ceilosca compared to Ceilometer
15:49:34 <rhochmuth> i would like to understand the tests and how that maps onto the overall system too
15:49:38 <fabiog> I want to test with 10M measurements and see what happens
15:49:46 <fabiog> I think Ceilometer will fail
15:49:49 <rhochmuth> 10M is small
15:50:16 <fabiog> right but we could not get Ceilo to work with more than 6M in the past
15:50:23 <rhochmuth> ohhh
15:50:44 <fabiog> let's see
15:50:59 <fabiog> the other thing I am not so sure is that Rally is really representative of the tests
15:51:07 <fabiog> I had strange results with it
15:51:22 <fabiog> so maybe we will use 10 runs using curl instead
15:51:36 <fabiog> it will be definitely more work but the results are more realistic
15:51:49 <fabiog> it will be undoubtedly the time a client will take to receive the data
15:52:26 <fabiog> bklei: how much data you have in Monasca in prod?
15:52:40 <fabiog> bklei: just to have an idea of a real prod env
15:53:06 <fabiog> so I can try to emulate that
15:53:12 <bklei> we've got around maybe 5 months of data, in vertica terms, getting close to 1TB
15:53:33 <fabiog> bklei: but in terms of measurements per metric
15:53:41 <fabiog> is 1M, 10M, 100K ...
15:54:04 <bklei> i can get that to you, don't know off the top of my head
15:54:20 <fabiog> bklei: I would appreciate that
15:54:28 <fabiog> thanks
15:54:35 <bklei> sure, will get u some #'s later today
15:54:39 <bklei> np
15:54:42 <rhochmuth> ok, 5 minutes left
15:54:54 <rhochmuth> should we continue with ceilosca or any other topics, or close
15:55:02 <fabiog> I am done unless there are questions
15:55:23 <bklei> i just want to plug a couple of libvirt plugin patches
15:55:26 <rhochmuth> no further questions your honor
15:55:32 <fabiog> rhochmuth: we need to start talking about summit topics and sessions
15:55:56 <fabiog> agenda for next week, I guess
15:55:58 <rhochmuth> we don't have any summit session
15:56:13 <rhochmuth> i think it will be hard to get, as they are over-booked and we aren't a openstack project yet
15:56:24 <fabiog> ok
15:56:32 <rhochmuth> but, we could reach out to other teams on various topics
15:56:56 <fabiog> yes, I want to talk to the Congress guys
15:57:02 <rhochmuth> there was some areas with the ceilometer team, such as alarming and componentization
15:57:12 <rhochmuth> Yes, Congress was the other one that comes to mind
15:57:27 <rhochmuth> There has been work in adding autoc-scaling to heat for MOnasca
15:57:37 <rhochmuth> I'll be getting an update on that tomorrow
15:57:55 <rhochmuth> Do you want me to contact Congress?
15:58:20 <fabiog> rhochmuth: sure I'll do that
15:58:28 <bklei> MaaS is our big priority at TWC, and so we're very focused on libvirt plugin -- would like to see https://review.openstack.org/#/c/225337/ and https://review.openstack.org/#/c/226405/ in soon and the agent tagged...
15:58:55 <rhochmuth> #bklei ok, i've given a +1 to one of those i believe
15:59:01 <rhochmuth> if others can review that would be great
15:59:05 <bklei> por favor
15:59:07 <rhochmuth> as soon as merged i can tag
15:59:16 <bklei> +1!!
15:59:17 <bklei> thx
15:59:27 <rhochmuth> i was hoping david schroeder at hp could take a look
15:59:32 <rhochmuth> not sure he was around this week
15:59:35 <bklei> me too, it's his baby
15:59:36 <rhochmuth> but will try and ping him
15:59:53 <bklei> thx
16:00:02 <rhochmuth> ok, coming to an end
16:00:05 <rhochmuth> thanks everyone
16:00:17 <ddieterly> ciao!
16:00:18 <bklei> thx #rhochmuth
16:00:26 <rhochmuth> #endmeeting