21:06:29 <rockyg> #startmeeting log_wg
21:06:30 <openstack> Meeting started Wed Feb 24 21:06:29 2016 UTC and is due to finish in 60 minutes.  The chair is rockyg. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:06:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:06:34 <openstack> The meeting name has been set to 'log_wg'
21:06:40 <rockyg> jokke_, !!
21:07:31 <rockyg> #topic Ops midcycle
21:08:13 <rbradfor> rockyg, so how did it go
21:08:36 <rockyg> Great midcycle.  Turns out the docs lead for config ref was there
21:09:36 <rockyg> We talked.  I showed a bunch of ops the config output in the dev docs area.  They were happy
21:10:20 <rockyg> But most ops are still on icehouse/juno  so doesn't do *as* much for them
21:10:39 <rockyg> Yet.
21:11:10 <rockyg> I asked format and presentation questions.  They responded with preferences.  docs lead loved it.
21:11:10 <rbradfor> why would it not do as much, because they have not done upgrades?
21:12:18 <rockyg> Yeah.  Still on the older stuff.  *but* one of the most important section for ops that have been around a couple of releases is the section on changed opts:  new, deprecated, and who knows, maybed defaults changed
21:12:58 <rbradfor> rockyg, great you mentioned that.
21:13:24 <rbradfor> I'm working on a spec to "Better identify configuration option lifespan"
21:13:27 <rockyg> Yeah.  Trying to make upgrades easier, so Reno, and those changes sections are critical.
21:13:57 <rbradfor> so there are Released, Changed, Deprecated, Removal attributes per option, and those sections (expecially changed) are auto generated.
21:14:03 <rockyg> I think we can get you ops comments on a spec like that.
21:14:55 <rbradfor> rockyg, its good meta information that both adds developers and deployers/administrators. It's also a very low development cost when the support exists
21:15:17 <rbradfor> and when it removes manual documentation (including release-notes), you never miss anything.
21:15:42 <rockyg> Excellent.  What would really be cool, but just a nice to have would be a tool that could generate current install's opts and values and a diff of what will change with a selected "upgrade" release.  So, if they skip a release, they can get the info in one place.
21:16:12 <rbradfor> yes, that's a sub-part of it
21:16:38 <rockyg> Wow!  cool!  The ops guys will definitely want to read the spec.
21:17:47 <rockyg> It was pointed out in the prod wg meetup that online retailers can't change site code from about August until after the new year.  Except for fixing brokenness.
21:17:47 <rbradfor> as part of removing two deprecated options in oslo.log this lead to talking to docs team about updating.
21:18:14 <rockyg> I saw that depracation.
21:18:31 <rbradfor> good discussion with gpocentek on -doc about how in particular config sections are generated in documentation.
21:18:44 <rockyg> Wonderful.
21:19:05 <rockyg> The syslog deprecations and config opts in oslo.log are very confusing for me.
21:19:36 <rbradfor> it's in the same vein but I can see ways to make it quicker and easier to produce that content, and more closely align with the code that generate developer docs and sample config.    converging the code to a source makes it easier to remove inconsistencies
21:19:39 <rockyg> The documentation is there, but it doesn't really 'splain it.
21:20:24 <rockyg> Yeah.  One truth is more likely correct than multiple sources of truth
21:20:46 <rbradfor> it's all intertwined as I am finding out.
21:21:30 <rbradfor> having a means of generating the actual code content consistently, then puts more back on the documentators to format around that, different guides, master sections etc.
21:21:54 <rockyg> Yeah.  And those sample configs generated....ops would love that straightened out, too.
21:22:22 <rockyg> Plus, you get quicker feedback from docs folks if something isn't clear to them.
21:22:32 <rockyg> Shortens the feedback loop
21:24:30 <rockyg> That's excellent.  Also, I suspect at least some of what is happening for this release can be used more "standalone" as tools for ops on earlier releases
21:24:58 <rbradfor> so, sample configs is from my work across a few projects, hugely inconsistent.
21:25:28 <rbradfor> I believe there is more a goal to have these as part of docs, instead of in the code base.
21:25:44 <rbradfor> I'm all for that, as it would be a more consistent location.
21:25:57 <rockyg> Oh, yeah.  About a year, year and a half ago there was tons of bitching about those.  Problem was the projects needed to update in a different location when they changed something.  So, they were always out of date.
21:26:03 <rbradfor> and as you point out, having it each cycle enables the simple diff command to be your friend.
21:26:42 <rbradfor> it's just yet another thing to want to make consistent and work through projects with it.
21:26:50 <rockyg> Hmm.  That's interesting.  Opts located in docs....
21:27:10 <rbradfor> sample configs in docs
21:27:31 <rbradfor> which FYI are linkable (and therefore clickable) in the developer docs.
21:27:54 <rbradfor> how are they in the manuals now?
21:28:06 <rockyg> Oh, wow.  You're gonna get sick of me saying wow.
21:28:30 <rbradfor> its logical. now getting it done, that's political.
21:28:32 <rockyg> They are mostly just in tables in the manuals.  Just words.  No hyperlinks
21:29:10 <rockyg> You've gotta have a crossproject session at the summit.
21:29:12 <rbradfor> what I'm also  proposing for the docs, it would take zero extra time to also produce a sample config that manuals could injest if wanted.
21:30:01 <rockyg> So, what's the political part?
21:30:24 <rbradfor> political in having all projects follow a set way of doing things.
21:30:49 <rockyg> That's what cross project team is for.
21:30:59 <rbradfor> as a side issue for example, projects should have migrated from oslo incubator to oslo libraries (releases ago), a number of (including TC approved projects have not)
21:31:24 <rbradfor> agreed, but saying to do something and seeing it done would from my limited time around be very different things
21:31:43 <rockyg> The spec goes to them, gets finalized, then we start getting project specs and bps added to it.
21:31:53 <rbradfor> that incubator point FWIW, is needed to push through standard logging format strings across projects
21:32:34 <rockyg> Oh, yes.  Frinstance:  glance v2 won't make it into nova this release.  I think it's 4-6 releases and counting.  Cinder v2, also.
21:33:37 <rockyg> So, which projects are still using the incubator oslo_log?
21:34:01 <rbradfor> so, there are several consistency things, they don't result in huge visible wins for operators, but are the foundation for more work.
21:34:30 <rbradfor> solum is one of them.  it also lacked oslo.context (the important meaty bits)
21:34:47 <rbradfor> oslo.context chnage in solum just merged this week, log is getting there.
21:35:05 <rockyg> Just knowing they're in the pipeline will make ops happier.  Many skip a release between each upgrade, so they see the changes faster in some ways.
21:35:07 <rbradfor> the next is Manila (that uses incubutor oslo context)
21:35:27 <rockyg> Ah.
21:35:34 <rockyg> More important for manila.
21:35:52 <rockyg> Solum is close to single vendor.
21:36:27 <rbradfor> yeah, after the cutoff of oslo libraries for Mitaka (which is today), that's one of my top priorities (as I want it for a logging blueprint I'm working on)
21:36:44 <rockyg> In fact, some of the TC thought it was dead until the past couple of weeks.
21:37:27 <rockyg> Excellent!  So, I have some good news around that, too.
21:38:06 <rockyg> Oslo has not been part of the roadmapping effort.  But, I'm the CPL to get the infor for the roadmap from Oslo.  and it's easy since you guys freeze early.
21:38:47 <rockyg> We can highlight the efforts/timeline to get all projects onboard/up to date.
21:39:22 <rockyg> It might also serve to attract more dev attention through vendors who realize that some things move faster because of it.
21:40:50 <rockyg> So, should I let you go to deal with deadlines?  The one thing I'd love to see is a better description of what the default logformat gets you by way of an example.  And how you specify syslog format....
21:44:09 <rbradfor> rockyg, so for developers I've started to include examples of logging output, see http://docs.openstack.org/developer/oslo.log/usage.html#olso-logging-functions (later example)
21:44:49 <rbradfor> it's a pre-cursor to a more specific example that includes the differ types across nova for example (that passes instance info).
21:45:04 <rockyg> Kewl!
21:46:05 <rbradfor> there needs to be more info on the actual possible attributes in the format strings (e.g. %(request_id), %(instance), even %(color).   this is likely better in an operational document.
21:46:07 <rockyg> I've told some folks that I have to get the reqIDs in log message spec out so I'm committed to it.  We'll see if I can get it done before the summit.  Hope to.
21:46:37 <rockyg> Agreed.
21:47:15 <rbradfor> yes, that's an example.
21:48:05 <rbradfor> I'm working on the app agnositic parameters spec/bp. This is needed to provide both a consistency and a flexibility of the options in a logging string.  There are several novaisms (as described in code).
21:48:54 <rbradfor> something to work towards for next cycle to have a operator docs portion (in say config reference) that can actually tell you what you can set and change)
21:49:16 <rockyg> Oh, fyi, the ops folks have specifically said they want place holders [-] when an option is not used so parsing can be consistent.  I don't know if that made it anywhere as a bit of lore/knowledge or not
21:49:39 <rbradfor> I'd like to know more on that.
21:50:24 <rbradfor> because specifically there are situations of a single [-], or [- - - - -] (5 for unknown user identity, or - for not specified), my earlier link actually shows these
21:50:33 <rbradfor> but it's only for "context" details, not for all values
21:50:40 <rockyg> Mainly, just don't drop a log message field if not in use or not pertinent.  Fill it with - so the message parser doesn't have to be as complex.
21:51:22 <rbradfor> well, actual examples of messages people see now, makes it easier for me to understand, and then vett if it's a specific release issue and no longer a problem.
21:51:30 <rockyg> The other thing they bitched about was when a log message was passed inside another and all the duplications that arose in the payload from that, plus really long log messages
21:52:37 <rbradfor> I only have demo devstack environments, and I don't see the worst of these.  Some will be project specific, but you should just start a etherpad and ask for real contributions (minus any company specifics)
21:53:23 <rockyg> I can and will certainly do that.  I have some of that from old summit sessions, but will open up a new etherpad to collect up more.
21:53:54 <rbradfor> also, with regards to the [-], operators should provide the actual logging_context_format_string value used in the project also, because projects historically are inconsistent here.
21:54:10 <rbradfor> if it was just mentioned, good to get people to be active again.
21:54:52 <rockyg> there is another big painpoint in that there are certain times when an exception happens that puts megabytes to gigabytes into a single log message.  How to circumvent that from happening may take a bit of digging for more info, but I might be able to track down the specific ops who related the info.
21:55:31 <rockyg> Yeah.  And with summit coming up, now is the time to get ops' attention and feedback.
21:55:36 <rbradfor> I'm just guessing here, but it's the repetition of the exception trace that's the problem.
21:55:56 <rbradfor> some error 1000 times with 1000 traces of x lines.
21:56:34 <rbradfor> how do you create a "reasonable threshold" of repeating messages, a lot like linux does (saying  "message" repeated n times) in syslog
21:57:09 <rbradfor> I could only surmise that would be really hard btw.
21:58:40 <rockyg> Actually, it's not that hard, but the ops guys seemed to think the problem was more like a core dump into the log message rather than repeating.  The repeating ones repeat nicely.
21:58:51 <rockyg> Thousands to millions of times.
21:59:18 <rbradfor> something to be had sitting with a real operator to see what they see
21:59:41 <rockyg> I've been socializing the idea of take a dev to work
21:59:47 <rockyg> To the ops community.
21:59:58 <rockyg> We can make that happen for you...
22:00:27 <rbradfor> I've got a lot of code work ahead of me, I've only been at it full-time for a few months.  lots to learn.
22:00:58 <rockyg> Ah! time.  Anything you want to get in quick?  We can make the ops thing happen whenever you're ready.
22:01:15 <rbradfor> happy to listen to operators (I was one myself), but I can see without talking to them, some important foundational work needs to be done so I could legitimately say, I see that and I can fix that.
22:01:34 <rbradfor> time to get back to it!
22:01:43 <rockyg> Yup.  Same here.  The more I dig, the more I see missing ;-)
22:01:52 <rockyg> Thanks!
22:01:55 <rockyg> #endmeeting