20:09:53 <Rockyg> #startmeeting log_wg
20:09:54 <openstack> Meeting started Wed Apr 29 20:09:53 2015 UTC and is due to finish in 60 minutes.  The chair is Rockyg. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:09:56 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:09:58 <openstack> The meeting name has been set to 'log_wg'
20:10:20 <nkrinner> Rockyg: but i think it is only the two of us
20:10:48 <Rockyg> Ok.  Well, it would be great if I could bounce some ideas off of you.
20:11:01 <Rockyg> I forget what project you're associated with?
20:11:25 <bknudson> I'm around.
20:11:26 <nkrinner> i also gathered some information on the docimpact/logimpact topic we discussed last meeting
20:11:36 <bknudson> you should send out a ping message when the meeting is started
20:11:44 <bknudson> to whomever wants to be notified
20:11:47 <Rockyg> Ah.
20:11:59 <Rockyg> jokke_:
20:12:02 <nkrinner> i'm not really associated with any project, but recently i comitted to sahara. but i am more interested in contributing to the oslo project
20:12:25 <Rockyg> Excellent.  nkrinner  You are just the person we need ;-)
20:12:37 <nkrinner> :-)
20:13:38 <Rockyg> I can't remember Nikolas/ handle, but I don't think he's on.  Same with Eugeniya
20:13:54 <Rockyg> topic:  what logging *really* needs
20:14:06 <Rockyg> #topic what logging *really* needs
20:14:53 <Rockyg> I've been working *a lot* on the spec, but it's been in responding to comments and changing the spec based on the responses.  I'm about 2/3 of the way there
20:15:03 <Rockyg> So, I haven't resubmitted it yet.
20:15:23 <bknudson> which spec?
20:16:26 <Rockyg> Two things the spec brings out:  1) we need someone to implement the oslo.log changes 2) we need projects to use new oslo.log error code stuff for new work, plus a while you're in there retrofit
20:16:33 <Rockyg> bknudson: lemme find it
20:16:58 <bknudson> there is a summit session proposed for oslo.log
20:17:22 <bknudson> see at the bottom here: https://etherpad.openstack.org/p/liberty-oslo-summit-planning
20:17:30 <bknudson> the working title is "Life, Liberty, and the pursuit of oslo.log changes"
20:17:55 <dhellmann> o/
20:18:20 <Rockyg> bknudson: got a line number?
20:18:32 <bknudson> #line 268
20:19:19 <Rockyg> #llink https://review.openstack.org/#/c/172552/5/specs/log_message_error-codes.rst,unified
20:19:25 <Rockyg> is the spec with comments.
20:20:04 <Rockyg> cool.  40minutes on oslo.log
20:20:11 <bknudson> lots of comments means people are interested
20:20:31 <Rockyg> that's Thursday?
20:20:47 <Rockyg> hey, dhellmann
20:21:05 <bknudson> this is the proposed schedule -- 5/20 is wed.
20:21:07 <dhellmann> hi, Rockyg
20:21:25 <Rockyg> I found a git repository with rfc5425 full compliance log message formatter in python
20:21:45 <Rockyg> OK.  Wednesday.
20:21:47 <bknudson> what's the license
20:22:04 <Rockyg> morgan fainberg's apache
20:22:07 <bknudson> https://tools.ietf.org/html/rfc5425 ?
20:22:38 <Rockyg> sorry 5424
20:22:47 <dhellmann> as a working session, that room is going to be pretty small and the discussion is expected to be very very focused
20:23:11 <bknudson> come to the session expecting to work.
20:23:18 <Rockyg> #link https://github.com/dpocock/python-rfc5424-logging-formatter
20:23:41 <dhellmann> Rockyg: that's GPL
20:23:54 <Rockyg> I've been doing a bunch of digging and reading, etc. to respond to the error code comments.  dhellmann: dang!
20:24:01 <bknudson> did morganfainberg write it?
20:24:15 <bknudson> we could probably twist his arm to dual license
20:24:36 <bknudson> although he'd have to talk to his old employer too.
20:24:38 <Rockyg> somehow, his name is on the license, but it's a fork of another repository, which, who knows, might be a fork of his.
20:24:39 <dhellmann> bknudson: looks like metacloud is involved, so we'd have to get them (or EMC, I think) to change it
20:25:07 <dhellmann> anyway, this is putting the cart before the horse because it's not certain it would be suitable for the changes needed
20:25:20 <bknudson> I thought oslo already had syslog formatter
20:25:23 <Rockyg> I'm thinking we can get the author to license under apache since it appears he just copied.  Might need to trace proveninance, though
20:25:29 <dhellmann> bknudson: we do
20:25:47 <dhellmann> Rockyg: let's focus on getting that spec approved before we worry about implementation
20:25:48 <Rockyg> bknudson:  but it's not 100%.  It's missing parts of the header
20:25:51 <dhellmann> we might not even need this
20:25:59 <Rockyg> agreed.
20:26:00 <bknudson> oops
20:27:00 <Rockyg> So, anyway, what I realized is that I would like the xproject time to define what gets written to which logs and when you write to any log.
20:27:16 <Rockyg> we have oslo logs, console logs, apache logs
20:27:48 <Rockyg> The api might need to write one thing to the oslo log, and pass log context to the project it is responding to.
20:28:43 <Rockyg> If we come up with a system flow and guideline for logs and logging, it will be much clearer where we need to fill gaps, change, or maybe even drop.
20:28:44 <bknudson> There's another log guideline that was already merged --
20:28:46 <bknudson> #link https://review.openstack.org/#/c/132552/
20:28:49 <jokke_> hi ... sorry got caught in middle of something
20:29:05 <bknudson> so this is a guildeline for logs and logging.
20:29:30 <Rockyg> Yup.  That is the basic "developers do this for logging so we have a chance of debugging this stuff" one
20:29:31 <bknudson> well, for logging... it didn't mention whether to use syslog or apache or whatever.
20:29:37 <dhellmann> yes, I think it makes sense to address these issues separately. Specs need to be small enough to understand and actually act on, either to implement them or to use the policies to help make a decision about something.
20:30:09 <dhellmann> bknudson: developers do not make that decision. *all* logging goes through the same API, and it can be configured by the deployer to end up wherever they want
20:30:52 <Rockyg> dhellmann: right.  so we need to get the logging "framework" documented/defined for OpenStack as an ecosystem.
20:30:53 <bknudson> y, developers need to know to not make an assumption
20:30:59 <bknudson> just because devstack gate does it that way
20:31:27 <dhellmann> Rockyg: let's start by defining the desired outcome, and work backwards to an API of some sort.
20:31:48 <Rockyg> great
20:32:03 <dhellmann> I think a lot of what you want is actually already possible, and some of what you want is going to meet quite a bit of resistance (error codes)
20:32:20 <Rockyg> Yes.
20:32:42 <Rockyg> I think error codes would go easier if we have an outline of how logging should work
20:33:17 <Rockyg> I kinda started from the middle, went down and am circling back to the top.
20:33:39 <dhellmann> yeah, it seems like there are several different issues, and we don't have to address them all in one spec
20:34:18 <Rockyg> Yes.  And if we get the principles down, the notification log stuff will be easier and smoother, too.
20:34:51 <Rockyg> I had started a google spreadsheet that I wanted to put the logs in and define certain characteristics
20:35:39 <Rockyg> #link https://docs.google.com/spreadsheets/d/1XTncfK_droY8E-Uy2icVuU-z9ya38ZBK_ZIRvGfPOXc/edit#gid=0
20:35:50 <dhellmann> we're not going to completely replace everything that is happening with logging, so we should be thinking in terms of small, incremental changes
20:36:11 <Rockyg> I thought if we could start with all the logs we have and categorize them, we would get patterns and classes
20:36:28 <Rockyg> I expect that console logs are not something we would want to address
20:36:58 <Rockyg> Except as a general "if it goes to the console, make the message meaningful"  kind of thing.
20:37:04 <dhellmann> console logs?
20:37:05 <jokke_> no, lets not touch anything in vm space unless they run nested openstack :)
20:37:12 <Rockyg> screen_
20:37:27 <jokke_> ah .... stdout and errout?
20:37:54 <Rockyg> jokke_: exactly.  Let's stay in the control plane, and out of the "data" plane
20:38:13 <dhellmann> so that's an example of what I was saying before -- the existing log configuration code is set up to use the console for devstack, the exact same code configures logging files as well
20:38:25 <Rockyg> So nova servers, compute nodes, neutron servers, keystone...
20:38:45 <Rockyg> dhellmann:  Oooh.
20:39:47 <bknudson> it's regular python logging
20:39:51 <dhellmann> all of this configuration stuff is built into the library already
20:40:00 <dhellmann> right, it's sitting on top of python's logging module
20:40:28 <Rockyg> hmmm.  how do you determine which logfile to write to?  I can understand by "project" but ....
20:40:51 <bknudson> the default config writes to stdout
20:41:02 <bknudson> if you want to override the default config to write to separate log files you can do that
20:41:19 <dhellmann> there are configuration options to control all of that stuff, and if the simple configuration options aren't enough a deployer can use python's configuration file format to do almost anything you can imagine
20:41:22 <Rockyg> dhellmann: are you by anychance going to have a log library tutorial?
20:41:41 <dhellmann> no, we don't have a session like that planned
20:41:48 <bknudson> here's a tutorial: https://docs.python.org/2/howto/logging.html#logging-basic-tutorial
20:41:53 <Rockyg> OK.  I know about the config file format stuff.  The operators use that to separate debug from everything else
20:41:53 <bknudson> from the python docs
20:42:23 <Rockyg> dhellmann: will that get me enough info to get deeper into oslo?
20:42:46 <dhellmann> the python.org docs?
20:42:50 <Rockyg> Yeah
20:43:01 <bknudson> Rockyg: you will want to understand that
20:43:16 <bknudson> Here's info on the config file: https://docs.python.org/2/howto/logging.html#configuring-logging
20:43:21 <dhellmann> understanding what the underlying code is capable of will help understand what oslo.log is doing
20:43:36 <bknudson> the oslo stuff is a small layer on top of python logging
20:43:52 <bknudson> for consistency, since it's easy to do everything differently
20:43:55 <Rockyg> I think I started it, but ah, that's what I need.  I'll review them both.  I'm also going to watch sdague's debug youtube talk again
20:44:03 <dhellmann> right, oslo.log mostly maps oslo.config options to behaviors in the standard logging library
20:44:44 <Rockyg> Yeah.  Theoslo.log  docs are good for figuring out what to put in your log message and how to build it from the dev perspective.
20:45:41 <Rockyg> so, putting the pieces together will give me the flow.  Which I have been trying to get and have not had much success.
20:46:03 <bknudson> I didn't know about the yaml config file... we should switch the sample.
20:46:32 <bknudson> http://git.openstack.org/cgit/openstack/keystone/tree/etc/logging.conf.sample
20:46:39 <Rockyg> I grabbed the file from openstack-infra/sys-config/modules
20:47:13 <dhellmann> bknudson: I think something has to read the yaml and convert it to dictionaries, so we would have to add code for that somewhere
20:47:47 <Rockyg> http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash
20:47:48 <bknudson> dhellmann: oh, it's not built-in, just easy to convert to the dict.
20:47:59 <dhellmann> bknudson: I *think* so? there's no yaml parser in the stdlib
20:48:26 <bknudson> Rockyg: logstash is a great example of why we need consistent format
20:48:39 <Rockyg> Yup.
20:48:58 <bknudson> people will get very angry at us if we make it difficult to debug gate issues.
20:49:05 <Rockyg> There is also a .erb file that infra uses to reconfigure the various message formats.
20:49:28 <Rockyg> So they can actually use logstash
20:49:57 <bknudson> not just openstack infra uses logstash, I'm sure there's deployments out there enjoying it too
20:50:05 <Rockyg> I've been collecting up all this data and trying to understand it to make it information.  I'm close.
20:50:20 <Rockyg> bknudson: yeah.  the operators grab that file, too.
20:50:27 <Rockyg> the .erb file
20:50:40 <dhellmann> Rockyg: it would be useful to see your notes, are they in an etherpad somewhere?
20:51:15 <Rockyg> I've got them in a few etherpads.  let me consolidate them.  I started, but then tried to get the spec out.
20:51:29 <dhellmann> ok, maybe just between now and the summit
20:51:34 <Rockyg> then the spec took longer because I kept searching for all the other stuff.
20:51:42 <Rockyg> dhellmann: definitely.
20:52:21 <Rockyg> dhellmann:  I'm about half way through addressing the comments in the spec.  I can work on that this week or the consolidation.  I can have one or the other out by early to mid next week
20:52:27 <Rockyg> Which would be better
20:52:42 <dhellmann> the spec, I think
20:52:54 <bknudson> get a new rev of the spec out so you can get more comments
20:53:19 <Rockyg> OK.  I'll make that happen then focus on getting the data down.
20:53:58 <dhellmann> ++
20:54:15 <Rockyg> But, I want that info so others can look at that, too, before the summit.
20:54:40 <Rockyg> Maybe all the different config and other files, tools and a flow and where they get used.
20:55:59 <Rockyg> I think if we have a reasonable understanding/flow/description of how/what things get logged now, it would be a good starting point in summit discussions of figuring out where we want to really go.
20:56:27 <bknudson> I'd find it interesting to know how it all works with elasticsearch, logstash, kibana.
20:56:30 <dhellmann> I thought the focus right now was the format of the messages?
20:57:33 <Rockyg> dhellmann: I think the format is a middle layer, but a really important one we can work on now
20:57:34 <bknudson> and whether some things have become irrelevant (syslog)
20:58:06 <Rockyg> Which is also why the spec is important.  another round of comments from the dev community and then get the ops guys on it.
20:58:47 <bknudson> hearing from ops how they want/need it done would be really great
20:58:58 <Rockyg> bknudson: I think the ops are driving the syslog stuff.  It's a more stringent standard, so easier for them to parse
20:59:16 <bknudson> syslog doesn't seem that stringent to me.
20:59:32 <bknudson> I've dealt with worse
20:59:39 <Rockyg> bknudson: I've got a bunch of stuff from the Paris summit in etherpads that the ops folks provided, comment on.
21:00:29 <Rockyg> bknudson: read the rfc and see what python.log is not enforcing.  One thing the operators want is "-" where a message field is left blank
21:00:51 <Rockyg> python.log lets you just skip that field
21:00:52 <bknudson> ah, that's where the formatter comes in.
21:00:57 <Rockyg> Yup.
21:01:35 <Rockyg> It really isn't that stringent, but is more tightly constricted for format
21:01:44 <jokke_> bknudson: it would also allow using the syslog header etc. fields more effectively providing the info out of those regardless the backend of the logging
21:02:24 <bknudson> defaulting to stringent syslog or having a syslog config option makes sense to me.
21:02:37 <Rockyg> Oops.  We're out of time.  do we want to continue on?  there's nobody after us.
21:02:52 <dhellmann> it's the EOD for me here, but you can continue on without me
21:03:18 <Rockyg> eod for dhellmann.  Think of poor jokke_ ;-)
21:03:34 <Rockyg> Thanks, dhellmann.  I really needed your input today.
21:03:58 <Rockyg> It will take me lots farther.
21:04:19 <dhellmann> np, Rockyg :-)
21:04:55 <Rockyg> so, I think this is winding down.  Let's get nkrinner's input on his research than call it a meeting
21:05:13 <nkrinner> ok
21:05:32 <Rockyg> #topic tags in launchpad
21:05:54 <nkrinner> so last week we discussed if we want something similar like the 'docimpact' tag for commit messages
21:06:47 <nkrinner> what is basically happening there is when a commit message contains a 'docinmpact', a bug is automatically filed against a project, here openstack-manual
21:07:09 <nkrinner> it is documented here: https://wiki.openstack.org/wiki/Documentation/DocImpact
21:07:41 <bknudson> there's also a SecurityImpact tag, which I think just sends email rather than filing a bug
21:07:53 <nkrinner> implementation happens mostly here: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/notify_impact.py#n161
21:08:27 <nkrinner> bknudson: that too, but i did not really look into that yet
21:09:00 <nkrinner> so, i think it would be something easy to implement. the hardest thing would be to get a spec approved to do so.
21:09:10 <jokke_> if anything I think the securityImpact model with automated mail to list is more what we would want
21:09:33 <bknudson> the bug is filed for docimpact because there's more work to do -- write the docs
21:09:46 <bknudson> the email is sent because they're asking for the ossg to look at the change
21:09:56 <nkrinner> i also wondered where to file the bug against if we go with the docimpact way
21:10:36 <nkrinner> so people prefer the securityimpact model?
21:10:40 <Rockyg> Yeah, we don't have a horizontal project for logs.  At least not yet.
21:11:06 <jokke_> We do not have a project and we do not have actual work to do if someone is changing their logging
21:11:27 <bknudson> what kind of changes do you want to see a LogImpact tag on?
21:11:36 <Rockyg> Well, translation would
21:11:53 <jokke_> it's definitely more in lines "We're doing damatical changes about what we spit out, please have a look"
21:11:56 <nkrinner> bugs where logging does not give meaningful information
21:12:19 <nkrinner> a way to tell "we need more/better logs here"
21:12:27 <bknudson> oh, it's for bugs, too, and not just commits?
21:12:32 <jokke_> Rockyg: the translation gets their strings through the i18n tooling anyways
21:12:38 <bknudson> The OSSG mailing list also gets email for security bugs.
21:12:56 <nkrinner> i prefer that, i think the main focus should be on bugs
21:13:12 <Rockyg> maybe operators would want mails on log bugs?
21:13:50 <Rockyg> at least on any marked critical
21:13:52 <bknudson> http://lists.openstack.org/pipermail/openstack-security/2015-April/subject.html
21:14:05 <bknudson> that's what the openstack-security list looks like
21:14:06 <jokke_> nkrinner: so did you find how to trigger those notifications out of bugs filed? Do we have tooling that FE looks for specific bug tags?
21:14:45 <nkrinner> jokke_: i did not look at that yet, i was looking at docimpact
21:15:06 <Rockyg> anyone can add a tag to a bug within a project, but is there a way to get the tag to pre-exist on all projects?
21:15:37 <nkrinner> when i proposed to investigate docimpact, i was not really aware what it does at that point
21:15:44 <jokke_> https://wiki.openstack.org/wiki/Bug_Tags
21:15:51 <nkrinner> bug tags: https://wiki.openstack.org/wiki/Bug_Tags
21:15:59 <jokke_> ;)
21:16:00 <Rockyg> nkrinner: from the other side, if you file a bug with docimpact, how does it get identified to the docs team?
21:16:21 <nkrinner> Rockyg: it only works with commit messages i think
21:16:27 <jokke_> Rockyg: in my understanding it does not ... the *Impact is commit message thing
21:16:44 <Rockyg> Hmmm.
21:16:50 <jokke_> the security bugs are special cupcake and gets all kind of bells and whistles anyways
21:16:54 <bknudson> I think the security tag causes an email
21:17:42 <nkrinner> would you like to have something similar with a log tag?
21:17:50 <nkrinner> sounds like a better idea to me
21:18:29 <jokke_> bknudson: I think it's not the tag but the classification of security bug in LP that triggers the e-mail (which is not sent to the public list)
21:18:37 <Rockyg> It would be worth polling the operators about it.  If they want an email, then we could get the tag across all projects
21:19:42 <nkrinner> do operators really want notifications about issues with logging? i want it as a developer though
21:20:05 <bknudson> if they're interested they can subscribe to the mailing list
21:20:54 <Rockyg> I wonder if there's a mailing list for the bugs with ops tag?
21:21:20 <bknudson> I didn't know about the ops tag.
21:21:41 <bknudson> are operators supposed to add that?
21:22:26 <Rockyg> anyone who thinks fixing the bug would make ops' lives easier.
21:22:50 <Rockyg> but I suspect, usually ops ;-)
21:24:21 <Rockyg> This is good info.  Now we just need to figure out what we should do with it.  Certainly it raises at least one question to ask ops at the summit
21:24:51 <Rockyg> Let's think more on this and what other info we want to get from ops at the summit.
21:24:59 <Rockyg> and let's call it a meeting?
21:25:26 <Rockyg> #endmeeting