20:00:34 #startmeeting log_wg 20:00:34 Meeting started Wed Apr 15 20:00:34 2015 UTC and is due to finish in 60 minutes. The chair is Rockyg. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:35 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:00:38 The meeting name has been set to 'log_wg' 20:01:10 Hi guys! 20:01:14 hi 20:01:16 Hello 20:01:58 Hi! 20:02:51 So, I'll admit I haven't been keeping up on updating the minutes or agendas. 20:03:11 But it only took the better part of 4 days to get the spec into review 20:03:32 And I still have to figure out how to add a co-author 20:03:44 That said, suggestions for the agenda? 20:04:01 Rockyg: you should have the details in your e-mail ;) 20:04:15 for the co-author 20:04:31 I'll reread ;-) 20:04:54 There has been lots of passionate discussion on the spec proposal already 20:04:59 So, I have two agenda items: summit and spec 20:05:02 which is good 20:05:15 agreed, jokke_ 20:05:51 Lots of good info. I'm thinking we discuss that second and get some assignments for getting stuff together for the summit, though 20:06:00 So we get some summit work done ;-) 20:06:04 ++ 20:06:20 it's surprisingly close ... like 30 odd days 20:06:39 Yup, and I haven't done flight or hotel yet. 20:06:54 #topic prep for summit 20:07:11 We have an Ops session for logs, which is good. 20:07:35 I want to be organized to get lots of feedback/assignments out of the session 20:08:05 I think error code spec and requestor ID specs should be part of the discussion 20:08:43 Rockyg: do you know the slot for that session yet? 20:08:47 And maybe a good doc on what a log message should be and when one should be generated 20:09:14 Lemme check... though I think Tom has it sometime early on the first day 20:09:22 Which would be Tuesday 20:10:34 ok, just trying to ensure that I actually can be there ;) 20:10:49 #link https://docs.google.com/a/openstack.org/spreadsheets/d/1EUSYMs3GfglnD8yfFaAXWhLe0F5y9hCUKqCYe0Vp1oA/edit 20:11:32 make sure you go to the vancouver tab... 20:11:57 12:05 - 12:45 tuesday 20:12:17 yeah the how do we fix logging one 20:12:24 Big Room and everything ;) 20:13:17 Yeah! But no working group meeting on it yet. Hmmm. Have to fix that. 20:14:21 I think a big problem with log messages in OpenStack is the fact that devs are confused between error messages and log messages. I think I said this already. and the reviews are showing it. 20:15:12 So, getting devs to define what the *need* and when they need it in log messages might be good. An Ops guide for developers. 20:15:16 ++ 20:16:03 That would be worth discussing in some of the other sessions, too. 20:17:06 another issue I see is that there is a fuzzy concern for security that gets added to the discussion and specifics don't turn around and answer the concerncs. 20:18:12 Everyone seems to understand "operators" get to see lots of stuff and are mostly trusted, but "users" are either not ever to be trusted, or are the same as operators, which we should know is not true on both counts 20:18:26 there are few cards that tends to get pulled out when there is no real argument but person just does not like the transparency or concept and security is one of those 20:19:26 by or before the summit, I'd like to have a list of all the known log files the major projects generate, where they are, who writes to them, and other identifying categories 20:20:21 that is so deployment dependent, but I think we can pull some general logs for example 20:20:57 jokke_, right. and if we can *target* a good chunk of the logs as to who has access, and who is writing to them, we can address security issues appropriately 20:21:57 I don't think the logs themselves are so big concern as they are fairly well contained ... I think the biggest concern is what we let user to know about the issue 20:22:30 agreed, but that is all based on the security level of the log the message gets written to. 20:22:58 or let's put it in this way, if we put deployment data int othe logs and 3rd party gets access to those logs, the deployment information is one of the smallest problems there 20:23:04 So, any vm generated log message should be viewable by the tenant admin. and maybe the tenant users 20:23:32 But, most of the projects don't write to vm logs. 20:23:38 yeah there is difference between VM details that are available vs. underlying architecture logs 20:24:40 So, we identify the logs that the architecture logs write to, and we document level of security needed for *the logs* 20:25:36 By providing detailed audit docs, we reduce the FUD about the contents of the messages. 20:26:17 Anyway, for the summit: 20:27:06 1. Present cross project specs: Error codes for logs, error codes/messages for APIs, request IDs 20:27:26 2. Get Ops comments, volunteers. 20:27:48 3. Provide the spreadsheet of log files for ops to expand info about 20:28:05 4. Get lots of feedback on current efforts 20:28:21 5. Get Liberty targets to work on 20:28:27 How's that sound? 20:28:46 6. add as many devs as we can kidnap to the point 2 ;) 20:29:01 ;-) 20:29:40 I would really really like to have at least one item by the time I'm flying back home that I know I can start working on right away 20:30:17 I think that is the most important thing. One plus thing agreed to the level that implementation can start 20:30:31 I think that with the error codes, a lot of the documentation will come from the Ops group. If we can get them to sign up for detailed docs and we can get devs to put summaries into a repository, then I think we have a good docs story 20:31:27 And the dev summary will be their "words for error codes" stuff. Instead of camel case English, it can be an English phrase that is easily translatable ;-) 20:32:41 I'd like to have in each repo a doc that has list of error codes and next to used code is the detailed (to the complexity) description why that error was raised 20:32:48 Jokke_, (and kudryashova) I think if we have the error code spec in reasonable developer shape for the summit, we can have the groundwork for the supporting tools identified and spec/blueprint filed either by the end of the summit or shortly thereafter 20:33:21 jokke_, sounds like a good use of pydoc;) 20:33:29 if it's "User does not have privileges" or "Driver X.Y.X failed due to Z second timeout towards device A" 20:34:13 agreed. 20:34:53 what I'd like to enable is building of knowledgebase what you can prepopulate by dev insight on those docs and extend with real life experience 20:35:21 ++ 20:36:07 Funny, we seem to be back to the spec. 20:36:08 Just example something that Telco can use to build knowledgebase when they are supporting their customers without needing to pull engineering in each time someone couldn't get VM launched 20:36:13 sorry 20:36:16 yeah 20:36:33 So, it sounds like back to the summit, more planning. 20:36:56 It sounds like we need to plant people in the working group sessions 20:36:57 yeah ... does the API wg has their own sessions? 20:37:21 like telco, enterprise, big installations, and yes, api-wg 20:38:12 I think we seriously need to sync with those people before they get that "Lets dump gig of json to the user when they are in problems"-spec through 20:38:17 Maybe we can work with the product-wg to ensure either feedback or coverage. I'll bring that up in the product-wg meeting today 20:38:29 that sounds good 20:39:10 jokke_, it looks like there's a second cross project spec that is trying to address the request id in a different way 20:40:10 I'd really like to see the large installations folks in that session of ours ... we've seen already what devstack scale spits out ... I'd really like to hear from the big env folks how they a) tackle the current situation b) would like to do it 20:40:17 Rockyg: by whom? 20:41:25 good question. Looking... 20:41:28 btw did we loose Nikolay_St and kudryashova or do you guys have something in mind you'd like to tackle in the summit? 20:43:01 good question. 20:43:35 so, it looks to be the same spec, but it's getting developer eyes/comments because we've managed to raise its visiblity. 20:45:20 ++ 20:45:28 bit by bit 20:45:42 And it looks like Jamie Lennox needs someone to respond to his very recent comment 20:45:43 maybe we have everything in the Unicorn release after all 20:46:19 20:47:20 I think we need to discuss in the summit also where do we direct all this good mojo when the logs are amazing :D 20:47:42 Oh, good question! 20:47:59 Ah, the ops will have something else to complain about..... 20:48:21 So, enought on the summit 20:48:24 #topic spec 20:48:35 although we could, to void even more confusion (ref back to the spec) change the name of this wg to supportaility_wg rather than log, could that help on both problems? 20:50:38 Looking at the RFC, and the reason I don't want the code in the body, is there is a place in the Header for MSGID 20:51:32 on what header? 20:52:03 Then, if the "message" gets passed up to the next "app", the whole message is encapuslated and a new MSGID gets tacked to the front. 20:52:47 according to rfc5424, the message format is Header Structured-Data MSG 20:53:06 #link http://www.faqs.org/rfcs/rfc5424.html 20:53:44 Header has priority, version, timestamp, hostname, app-name, procid, msgid 20:54:15 section 6.2.7 is msgid 20:54:40 that is very syslog specific and I don't think python logging nor oslo.log has actually committed to follow such 20:54:54 1*32PRINTUSASCII 20:55:24 I bet you're right. But I also bet we could get oslo.log to add the field 20:55:48 We know that have Priority, host, timestamp 20:55:56 which is probably the reason why people have been so confused around that spec regarding the header reference ... that also does not help us to provide that information to the user, does it? 20:56:21 the header always gets printed out in the log message 20:56:33 And it is *before* the body 20:56:42 So, makes it easy to find 20:57:24 It's also something we might get oslo.log to default stuf to if they put it in and it isn't passed 20:57:58 like proj-class-nullnumber if no code specified 20:58:07 This field MAY be operator-assigned. 20:58:14 what ever that might mean 20:59:29 oh, that was speedy hour 21:00:12 Oh, sh*t. Yeah. But, review this RFC and I'll go look at python log stuff. I think this would work out great and solve some of the confusion 21:00:40 yeah ... we thanks and have a god one 21:01:07 And, the RFC allows for passing log messages through, and has some pretty good rules about it, which will leave the log trail back to the first loggable message through to the last., 21:01:23 YUup. better end the meeting.. 21:01:27 #endmeeting