15:00:14 #startmeeting monasca 15:00:15 Meeting started Wed Apr 20 15:00:14 2016 UTC and is due to finish in 60 minutes. The chair is rhochmuth. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:18 o/ 15:00:19 The meeting name has been set to 'monasca' 15:00:19 see you in austin ;) 15:00:25 hello 15:00:27 o/ 15:00:31 o/ 15:00:39 morning 15:00:45 o/ 15:00:45 o/ 15:00:49 o/ 15:00:58 o. 15:01:00 o/ 15:01:12 agenda has been posted 15:01:14 Agenda for Wednesday April 20, 2016 (15:00 UTC) 15:01:14 1. Patches 15:01:14 1.1 https://review.openstack.org/#/c/298863/ 15:01:14 1.2 https://review.openstack.org/#/c/301355/ 15:01:14 1.3 https://review.openstack.org/#/c/307963/ (brad curious about status) 15:01:14 1.4 https://review.openstack.org/#/c/289675 (same) 15:01:14 2. Heads up on ovs plugin (router bandwidth metrics) 15:01:15 2.1 https://review.openstack.org/#/c/306621 (not fully baked) 15:01:15 3. https://review.openstack.org/#/c/302660/ 15:01:20 https://etherpad.openstack.org/p/monasca-team-meeting-agenda 15:01:35 hi everyone 15:02:04 so we have a few reviews that have been asked about 15:02:04 good morning! 15:02:21 so, let's go through that list and then discuss other items 15:02:28 https://review.openstack.org/#/c/298863/ 15:02:31 Those first couple reviews are mine. 15:02:44 That one just needs to get merged 15:02:56 It's been sitting around for two weeks with several +1 15:03:05 i see, looks like there are a bunch of reviews and +1s 15:03:16 i reviewed this morning, so it looks good to me 15:03:20 i can merge now 15:03:25 Great, thanks 15:03:51 then there is https://review.openstack.org/#/c/301355/ 15:03:52 The next one is the more complicated patch 15:04:11 This is the thread-pool replacement patch 15:04:11 i looked at this one too 15:04:28 there were some quesitons, but i'm not sure it has been resolved 15:04:38 I just wanted to talk about this one since it seems like there were some concerns 15:04:49 yes 15:04:58 is rbrabdt here? 15:05:03 rbrandt? 15:05:24 also michael hoppal and joe keen would be good 15:05:43 anyway, please ask questions 15:05:48 I'm open to making changes, I just want to keep this moving forward 15:05:52 not sure how much help i'll be 15:06:14 so, i'll need to ping all those names i mentioned 15:06:16 I just need clarification on what the concerns are so I can fix them. 15:06:49 i'll ping them, we were super busy last week on our own releases 15:07:02 so, hopefully they can take another look today 15:07:24 That's fine. I'll ping rbrandt later since it doesn't seem like he's on 15:07:48 We can move on for now. 15:08:07 the next two patches are rbrandt's -- and i was curious on status 15:08:11 actually i did this really cool thing 15:08:27 i walked down the aisle and talked to them using my voice 15:08:35 strange -- that works? 15:08:43 they are connecting in now 15:08:47 only sometimes with him 15:08:53 rbrandt u there 15:08:53 :) 15:08:59 hoppal u there 15:09:00 <14WAATBIX> I'm apparently under this name 15:09:04 im here 15:09:06 o/ 15:09:08 nice name 15:09:09 <14WAATBIX> no idea... 15:09:15 is that you rbrandt 15:09:19 <14WAATBIX> yeah 15:09:22 ok 15:09:32 rbak is asking about https://review.openstack.org/#/c/301355/ 15:09:41 this is the muleiprocessing review 15:10:01 he's trying to understand next steps and the feedback he;'s received so far 15:10:21 <14WAATBIX> Ah, yes. So the issue at the moment is the lost data if a check fails. 15:10:58 <14WAATBIX> Right now, as far as I am aware, we don't lose data except for the failed check itself 15:11:06 That's not correct 15:11:18 <14WAATBIX> jkeen actually may know more of the specifics 15:11:24 Right now, worst case, the agent hangs and everything is lost. 15:11:35 Until the agent is restarted by hand. 15:11:59 This patch doesn't actually change any behavior other than to allow the pool to restart as intended. 15:12:00 (and we've hit that a bunch and lost alot of data) 15:12:23 The triggering condition and the result of the restart are still the same. 15:12:30 so, with this patch, you fixed one big problem 15:12:39 <14WAATBIX> jkeen was exploring the code to see what we used to do and what this new code does 15:12:51 are there any other downsides 15:13:22 sorry, meant downsides or side effects 15:13:36 The only new downside I'm aware of is that it's noisy. Restarting the pool can throw a lot of errors. 15:13:51 you mean the log file 15:14:04 Yes. 15:14:12 <14WAATBIX> I know when I was testing it we regularly saw failures in the checks 15:14:23 Actually though, I think I finally realized what the miscommunication here is. 15:14:29 <14WAATBIX> which is what prompted the comments 15:14:45 There's two different ways this can be triggered, and we're talking about different conditions. 15:15:31 Previously, the pool restarted cleanly and no data was lost, unless a check was truly stuck in which case it never restarted and the entire agent hung. 15:16:23 In that worst case this version is better. If a check is just taking too long and the pool restarts, I suppose this does lose more data than before. 15:16:51 14WAATBIX: Why don't I talk to you after the meeting so we don't eat up too much time in the meeting. 15:17:07 should also get joe and michael 15:17:15 <14WAATBIX> Sure, I think we're getting synced now 15:17:24 please schedule a meeting with all three and anyone else that wants to attend 15:17:34 Sure, I can do that. 15:17:40 thx rbak 15:18:07 next review 15:18:09 https://review.openstack.org/#/c/307963/ 15:18:33 that's 14WAATBIX's -- just wondering status 15:18:44 i'm ready to test it locally when told ot 15:18:47 to 15:19:13 <14WAATBIX> It should be ready for testing at the moment, I'm trying to get it set up in a large scale environment 15:19:27 perfect -- will pull in this am and kick the tires 15:19:32 it didn't pass gates 15:19:39 is it really ready? 15:19:51 <14WAATBIX> its vertica code changes so the gate shouldn't be affected 15:20:03 <14WAATBIX> the python api tests failed in an alarm test 15:20:08 maybe just needs to be rechecked 15:20:09 <14WAATBIX> not sure why, but investigating 15:20:14 that seems like a flaky test 15:20:27 <14WAATBIX> it does, hopefully I can rewrite it 15:20:47 thx -- will give you feedback after i test your patch 15:20:58 thx 15:21:09 <14WAATBIX> thnx 15:21:20 https://review.openstack.org/#/c/289675 15:21:34 just wondering if that dude is close 15:21:38 (multiple metrics) 15:21:45 yup 15:22:02 <14WAATBIX> It might need to be rebased on the metric list changes 15:22:05 14WAATBIX 15:22:07 ? 15:22:28 so when metrics list merges you'll need to rebase 15:22:37 i'd guess -- poor 14WAATBIX 15:22:39 <14WAATBIX> I think so, but I'm not 100% sure 15:22:42 or which ever goes first 15:23:08 besides the rebase, are both reviews ready to go independently? 15:23:16 <14WAATBIX> I think so 15:23:37 bklei we are really interested in getting feedback on this one, namely performance analysis 15:23:55 hopefully, we've significantly improved things 15:24:03 this one will also impact any queries that you are doing 15:24:18 we can provide feedback on perf -- but not so much pre-merge :( 15:24:21 for example, grafana will need to be enhanced to use the group_by query parameter 15:24:28 our best perf tests are in prod 15:24:35 oh for sure 15:24:49 that's gonna make some graphs scream 15:24:50 well, you might want to test this one before production 15:25:07 a few more moving parts 15:25:15 the level of complexity is much higher 15:25:17 we can do a limited amount in the lab, but our best historic results are from prod, but we can get an indication 15:25:41 (we're talking multiple metrics -- not limit, yes?) 15:25:55 we did test this change against the ceilosca code 15:26:04 and saw a huge perf improvement 15:26:16 but, unfortuately no real numbers yet 15:26:26 i'm excited about this one 15:26:46 so, after this change, i'm not sure about the "caching" solution 15:27:03 multiple metrics might actually work better than pursuing caching 15:27:25 perhaps, things like metric-list would benefit, and that's a pig for us 15:27:48 ok, sounds like time to move on 15:27:48 metric-list is 1GB of data in our 'infra' project 15:28:14 but, with start/end time you limit that down to something managable 15:28:15 right 15:28:35 not as much as it did before -- with grafana 2 15:28:40 also, do you have automated cron josb to remove old data 15:28:40 and even 'current' is huge 15:28:44 yes 15:28:56 but infra keeps 13mos 15:29:04 so prune doesn't help that project much 15:29:09 i see 15:29:37 is there a way to control the range of data returned in grafana 2 15:29:43 i didn't see any 15:29:56 rbak can comment -- but i don't think so 15:29:58 it should just be the current range 15:30:00 right? 15:30:05 last hour 15:30:08 last day 15:30:19 I looked away for a moment. I'm not sure I followed the conversation 15:30:22 sounds like the data source doesn't have visibility to graph range 15:30:47 (providing timestamp to metric-list) 15:30:58 Templating doesn't have access to the time-range 15:31:04 so, when you are displaying metric names for example, can you limit the names that are returned to the current time range set in grafana 2 15:31:04 The graphs do. 15:31:21 It depends on how you do it. 15:31:30 and templating is the part that dispays the names and dimensions? 15:31:39 If you use templating, no 15:32:13 Templating is the feature that queries for dimensions and lets use insert them into the graphs dynamically. 15:32:28 I'm not sure if that answers your question 15:33:08 not sure, but we can move one 15:33:12 how about, Heads up on ovs plugin (router bandwidth metrics) 15:33:21 what is this about 15:33:24 i saw the review 15:33:28 yeah -- let's call this an advertisement 15:33:37 it's not ready, just want to gain visibility/support 15:33:45 our customers really want BW for virtual routers 15:33:58 we could do this locally, but thought others may want it too 15:34:08 people that use OVS/neutron 15:34:08 yes, this could be very interesting 15:34:28 it's pretty much working needs some cleanup though 15:34:33 so, syd will be interested and we're going to have a session next week with the neytron team too 15:34:49 so, is this a discussion that we want to have next week in austin? 15:35:06 ok -- i'll add syd to the review -- and i'll join that discussion if you let me know when/where 15:35:35 the fourth session on wednesday morning is the combined monasca/neutron/network discussion 15:35:40 perfect 15:35:49 it will be only 40 minutes long 15:36:00 but, we can also discuss more while we are there 15:36:17 sure, sounds good 15:36:18 but, i'm very interested in this development 15:36:25 oh -- one question on my review 15:36:30 so, i'll start taking a look at it 15:36:34 sure go ahead 15:36:36 is it required to add a 'detector'? 15:36:41 we don't use that process here 15:36:44 what do you mean? 15:36:56 for all plugins -- add a detector 15:37:04 detection plugin 15:37:09 right -- that 15:37:30 would be nice if it makes sense 15:37:32 is that required for all plugins? 15:37:38 not all plugins have deteciton plugins though 15:37:49 so not required 15:37:56 the detection plugins configure the agent 15:37:57 ok -- i'll add it if folks request 15:38:09 and are run seperately, usually at install/deploy time 15:38:22 sounds good 15:38:28 I stepped away for a bit, yes, plenty to talk about Wednesday 15:38:33 i'm not sure how detection woudl work in this case 15:38:33 yeah, and we don't use that process, but can add it if folks want it 15:38:45 yeah -- i'm not either 15:38:59 hi slogan, yes 15:39:22 you might want to take a look t that review if you haven't already 15:39:27 https://review.openstack.org/#/c/306621 15:39:30 will do 15:39:42 hre is the link, https://review.openstack.org/#/c/306621/ 15:40:01 (it's not fully baked -- but close) 15:40:33 but early comments welcome 15:40:58 ok -- i think that's it for me 15:41:14 https://review.openstack.org/#/c/302660/ 15:41:22 migrate from mysqldb to pymysql 15:41:27 it looks good to me 15:41:38 just need to run it once 15:41:49 there seems to be a problem with requirements 15:42:11 witek: so that sounds like it isn't ready 15:42:15 but I did not investigate 15:42:36 the gate job was successful previously 15:42:41 we are not so sure from where this error is comming 15:42:43 I just changed the commit message 15:42:51 ContextualVersionConflict: (Babel 2.3.3 (/home/jenkins/workspace/gate-monasca-agent-python27/.tox/py27/lib/python2.7/site-packages), Requirement.parse('Babel!=2.3.0,!=2.3.1,!=2.3.2,!=2.3.3,>=1.3'), set(['oslo.log'])) 15:43:50 it could be that something changes upstream 15:43:56 that is impacting the agent 15:43:57 not sure 15:44:11 i think it is, all of our agent reviews are having that issue 15:44:13 that is also my first assumption 15:44:37 i was going to try to look into it sometime today if i get a chance 15:44:47 that hoppalm 15:44:53 thx 15:44:59 not that 15:45:00 would be great, thanks michael 15:45:12 that hoppalm is such a great guy 15:45:17 :) 15:45:21 hotpalm :) 15:45:35 meanwhile roland is laughing as he types that 15:45:40 :) 15:45:54 we know him too much :) 15:46:10 witek: craig is reviewed tomasz's review 15:46:17 great! 15:46:27 he sent me an email this mroning with some general questions 15:46:36 but he is diving in on this 15:46:52 thanks a lot for that! 15:47:16 i'm not sure i'll run into him here today, but i'm expecting he'll post some comments to the review soon 15:47:45 that would be best, or if longer, per email to me and Tomasz 15:48:13 ok 15:48:41 so, are there any other reviews or questions? 15:49:20 for those in austin, i'm wondering if we should have dinner together one night 15:49:42 good idea 15:49:45 thursday is probably the best, but i'm open to suggestions 15:49:45 that would be great 15:49:54 just as long as bogdan doesn't pick the place 15:50:03 haha 15:50:19 actually, that was fun 15:50:20 Oo 15:51:17 i'll send an email out and try and coordinate 15:51:33 shinya, you are going to be there 15:51:54 Yes I will go to austin! 15:51:59 hosanai, you are going to be there too? 15:52:01 awesome 15:52:14 I'm looking forward to seeing monasca folks. 15:52:15 have you been looking at cassandra, btw 15:52:24 yep 15:52:24 me too 15:52:49 witek, rbak, bklei, slogan 15:52:50 Sorry not yet, but I will 15:52:59 no problem, was just checking 15:53:19 also, as i mentioned, we might want to consider ES too, 15:53:26 as that is a part of the logging solution 15:53:34 it would help with deploys 15:53:52 fabiog will be there too 15:54:09 if there is anyone else that is going to austin that i've missed please let me know 15:54:18 Kouji 15:54:36 great 15:54:45 my colleague, daisuke 15:55:09 thanks hosanai 15:55:16 i'll get him on the email list too 15:55:29 we'll coordinate while we are there, but just in case 15:55:30 thanks roland! 15:55:37 i wanted to have the names 15:56:06 the MOnasca bootcamp is on MOnday 15:56:13 if folks are available to help that would be great 15:56:24 my plan is to go thorugh an iPython notebook 15:56:32 which will have lot's of example of using the API 15:56:37 and cover other topics 15:56:38 I'll be there 15:56:44 ditto 15:56:49 it is supposed to he hands-on 15:57:02 what kind of help do you anticipate needing? 15:57:03 I'll be there 15:57:14 but given it is only 1 1/2 hour not sure that we can spend 1/2 hour just setting up VMs 15:57:23 thanks everyone 15:57:30 i'll introduce everyone that is there 15:57:52 also, please look at the other sessions invovling monasca 15:58:19 twc, fujitsu, cisco all have sessions 15:58:30 and don't forget wednesday morning 15:58:38 those are all the monasca working sessions 15:59:04 there are also some other sessions on monasca or involving monasca in some capacity that are showing up 15:59:08 I missed the ones by fujitsu and cisco, I'll go back and look for them 15:59:18 awesome 15:59:21 fujitsu is on the logging api/service 15:59:31 if you have a must see list you are keeping, maybe share that? 15:59:31 ciscso one is related to congress 15:59:35 and policy management 15:59:55 there are also some session on logging that look interesting 16:00:09 ok, i think that is done 16:00:14 bye-bye everyone 16:00:20 see you next week 16:00:21 i always cram until the end 16:00:37 #endmeeting