20:01:55 #startmeeting log_wg 20:01:56 Meeting started Wed Apr 22 20:01:55 2015 UTC and is due to finish in 60 minutes. The chair is Rockyg. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:01:57 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:02:00 The meeting name has been set to 'log_wg' 20:02:32 Now, the question: do we have anything people want to discuss? The table is open for new discussions 20:03:41 not sure if this is worth discussing, but this crossed my mind recently 20:03:42 I really have nothing for today 20:04:04 nkrinner: well, better than napping -- go for it 20:04:52 Also, abviously, I have not gotten the next draft of error codes out yet. FYI. 20:05:04 for bugs we have the tag docimpact tag, which automatically notifies the doc team about changes that might affect them. do we want something like this for logs too? 20:05:52 Oh, yes, please! I started it, but going project by project, finding a bug that could use the tag, and adding it, was taking lots of time 20:06:09 If there's a more straightforward way, I'm all for it. 20:06:19 i was just browsing the code to see how and where to implement that. also trying to find out with who to talk about this. 20:06:49 so i suggest i will continue investigating this and report on it next week? 20:07:30 Sounds great 20:07:45 sounds good 20:07:59 so next week we hopefully see if it is worth the effort then. :-) 20:08:05 I think I was going the easy route with the tag "log" 20:08:49 nkrinner: I think it will be. Once it's available for ops folks, I think they will use it. Of course, that's also after educating them. 20:08:54 :) 20:09:04 yeah :-) 20:10:03 But, summit is soon. Wondering -- would it make sense to have a working session with ops folks just identifying, tagging and escalating log bugs? Sort of a log bug triage session? 20:11:40 I also find it difficult to find "log" bugs because yuu can't do a cross-project search for key words 20:11:52 ++ 20:11:52 sadly i can't make it to vancouver 20:12:02 Darn! 20:12:25 If you ever make it to San Jose, I'll buy you a beer. 20:12:33 :-) 20:13:34 FYI. some infra and other folks are looking at phabricator as a possible replacement for storyboard/launchpad. Take a look and see if it is better 20:14:00 #topic Rambling 20:14:08 thanks, will take a look 20:14:12 Just wanted to get in that ... 20:15:01 Some more rambling. We have "log files" scattered across the cloud. some log "log messages", some log error output, some log HTTP stuff. 20:15:20 I''ve got a list. I want to make it a matrix. 20:16:09 default log name, location, type of messages logged, audience (ops, tenant ops, app admin, etc), etc 20:16:48 ok 20:16:54 I'm thinking this could help clarify the differences between the various messages for one 20:17:34 It could also provide a checklist for devs as to what type of message they really want to generate based on classification 20:18:10 yeah I think that would be a good idea 20:18:12 It would aid everyone from dev to enduser in tracking down information 20:18:14 it would help making developers more aware about other peoples log needs 20:19:26 Let's brainstorm on what the columns should be. Of course the first one is the log file itself. /there are about 80? I think. 20:19:38 And location seems to be good for the second 20:20:14 project/service the log belongs to 20:20:22 maybe 20:20:59 https://docs.google.com/spreadsheets/d/1XTncfK_droY8E-Uy2icVuU-z9ya38ZBK_ZIRvGfPOXc/edit?usp=sharing 20:22:18 #link https://docs.google.com/spreadsheets/d/1XTncfK_droY8E-Uy2icVuU-z9ya38ZBK_ZIRvGfPOXc/edit?usp=sharing 20:22:37 viewable by? 20:22:41 i think that is the most important information 20:22:44 Audience? 20:23:14 I can see it at least 20:23:16 i think everything is only visible to admins? 20:23:25 Ops/Domain Ops/Tenant? 20:23:45 Well, all the vm log files are available to the tenant who spins up the vm 20:24:17 true 20:24:27 And if you have nested clouds, then the guy who controls the top of the provisioned cloud owns those cloud files 20:25:18 And who is supposed to use them, versus who can see them is where security issues happen 20:27:35 cloud users can do whatever they want with the logs they created i guess 20:27:35 Somewhere I had found a list that infra put together. Tryng to find it now. 20:28:27 Right. so it's a matter of what they have permission to create. It's the whole role thing. So I guess the question is: What is the scope of the Role? 20:30:11 i as a developer am mostly interested in what helps me debug issues with services and find all the infos i need in the log files. i have not really thought about user needs so far 20:31:40 Ture. Devs think about what they need to develop and Ops/End Users think about what they need to check on health. and hackers think of what info can be used to break in. 20:32:26 So, Purpose of the log is a good thing to know. It helps to inform us who the users will be. 20:32:38 i see 20:33:13 purpose alse depends a lot on the log level i guess 20:34:21 Yes. Not all logs are for log messages. Some are for notifications, the housekeeping on RESTful APIs, tracebacks, etc. 20:35:47 So, for instance, the Apache logs would get most/all API message logs, but then some of the project logs would also capture a log after processing the API response. 20:36:44 yeah and all the wsgi servers should log each request once to the service logs 20:38:47 So what jokke_ just said leads me to think we should add a field for message type, like wsgi notifications, HTTP, etc? 20:39:06 ok 20:40:21 to be honest that is all valuable information i can currently imagine. what else woulde we need to know? maybe a general 'does its job'/'doesn't do its job' rating? 20:40:23 that could work 20:41:27 for example i am not so happy with some of the logs i have had to look at, but it might be it was only because it were log files of services i am not familiar with 20:41:35 That's a good idea. Rating its usefulness 20:42:15 i think keystone had some quite ugly logs, but then i usually don't touch keystone, so maybe i was simply not educated enought to read them 20:42:45 nkrinner: make a note for us somewhere, example could be good idea 20:43:13 nkrinner: you can put a comment or note on the spreadsheet, or add to the note I put on it. 20:43:23 jokke_: true. this is what i remember vaguely 20:44:03 because if the logs on some specific services looks non intuitive, that's perhaps something we could affect 20:44:59 i have to look at it again, maybe i blamed the wrong service :-/ 20:45:07 Precisely. Or, more to the point, get them to be more precise;-) 20:47:47 So, I think getting this started is a good thing. Let me see if I can find that list of log files again to propagate the first column. Once we have that, we can announce to the ml for comments, concerns, help filling in. 20:48:36 yes, i think it is a good start 20:49:02 I think I'm also going to add a earliest release column for when the file first appears in OpenStack.. Optional, so last. 20:49:47 Gee. anything to get out of editing that feakin' spec ;-) 20:51:42 Jeez, just thought of another tool that ops guys would love for OpenStack. A log registry. If a new log is spun up, oslo has a library call that registers the log in a db or some such. 20:52:31 i think logs should be known and collected automatically anyway 20:53:41 Rockyg: automatic log finder would probably not work with syslog 20:53:42 nkrinner: you would think, but right now, it doesn't happen. Each project generates what it thinks it needs in the places it has decided upon. Wouldn't it be nice to have a central registry? 20:54:05 jokke_ don't want a finder, want a register 20:54:20 then devs could find logs they need, too. 20:54:47 hm. i think we have a tool for that already, let me check 20:54:51 Kind of like the murano app catalog, but for logs 20:56:08 it is a custom bash script, i think it is maintained manually 20:56:12 then that could be fed into logstash or whatever, with rules and create a customized best of collection of the distributed log files. 20:57:04 And if we know the type of log, that could help us determine how persistent it needs to be, also. 20:57:30 Some interesting thoughts. But that said, we're near the end. 20:57:51 #action nkrinner to look into tagging bugs. 20:58:06 yeah ... I've been thinking also, should we start log aggregator project to the OS or is there something doing that already 20:58:29 i know someone working on a logstash implementation afaik. maybe i have some news next week 20:58:48 nkrinner: sounds good, keep us posted 20:58:51 I found a launchpac query for log bugs from Tom Fifield: https://bugs.launchpad.net/openstack/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field. 20:58:51 bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=log&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search 20:59:19 It's supposed to find any bug tagged with "log" ? 20:59:19 ouch 20:59:34 Yeah. But, you work with whatch got. 20:59:42 It's a start. 20:59:55 so, Thanks everyone. a working session is a good session. 21:00:00 3endmeeting 21:00:03 yeah, and it's time ... thanks both of you ... good discussion today 21:00:07 #endmeeting