21:02:09 #startmeeting crossproject 21:02:10 Meeting started Tue Jan 6 21:02:09 2015 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:02:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:02:12 * mestery waves at ttx again 21:02:15 The meeting name has been set to 'crossproject' 21:02:25 Our agenda for today: 21:02:32 #link http://wiki.openstack.org/Meetings/CrossProjectMeeting 21:02:42 mestery: live line edit fail 21:02:49 lol 21:02:58 #topic State of nova-network to neutron migration 21:03:04 anteaya: around? 21:03:14 This was raised in a ML thread before the holiday season 21:03:23 #link http://lists.openstack.org/pipermail/openstack-dev/2014-December/053355.html 21:03:26 o/ 21:03:28 Hey hey 21:03:32 ttx: happy new year=) 21:03:37 I am 21:03:44 boris-42: thx, glad you could join 21:03:50 In summary, while this is a stated priority for Nova and Neutron teams, the effort has stalled 21:03:53 ttx: yep from vacation=) 21:04:04 I think we have already taken steps to address this issue with a specific meeting and a champion to make sure progress is made 21:04:11 * nikhil_k joins in 21:04:18 we are working on collecting around a meeting time 21:04:22 ttx: Yes 21:04:23 I would just like to make sure there are no blockers at this point 21:04:28 so far myself and oleg have stated preferences 21:04:34 If there are, see if that meeting can help remove them 21:04:35 well today russian holidays 21:04:41 but that will come to an end 21:04:56 anteaya: do we have all the team-specific liaisons we need to make progress here ? 21:04:58 anteaya: the timing for setting up a meeting wasn't great with the holidays. hope to respond today 21:05:05 jogo: thank you 21:05:18 I know gus is interested too -- I'll make sure he's aware 21:05:18 we have jogo's and mikal's attention in nova 21:05:23 great and gus 21:05:25 ttx: yes.. we should now 21:05:36 and mestery and markmcclain and oleg in neutron 21:05:42 Do we need ops-side support ? 21:05:55 ttx good question, whom might you recommend? 21:05:58 like have a few of them in the workgroup 21:06:02 Someone from CERN I imagine 21:06:03 * anteaya turns down no offer of support 21:06:04 anteaya: Tim Bell? 21:06:12 oh yes, we have two cern devs 21:06:12 mikal: ++ 21:06:18 I have their email addresses 21:06:41 they attended the earlier impromtue meeting and have reviewd the current spec which is wip 21:06:58 anteaya: I can reach out to fifieldt_ in case he has someone else to suggest 21:07:10 the part I am still a bit unclear on is: what is required from nova 21:07:10 ttx would be great to have his participation 21:07:20 last I read the spec it was not very complete 21:07:30 jogo: right, there were some passages in the spec that needed detail 21:07:46 and unfortunately the main author oleg is on holidays still 21:08:02 anteaya: so we should know more next week ? 21:08:05 what's the link to the spec? 21:08:09 I sure hope to 21:08:12 * mestery thinks we really need to wait for Oleg here 21:08:35 found it: https://review.openstack.org/#/c/142456/ 21:08:42 o/ 21:09:01 anteaya: OK, we'll just let the workgroup make progress. If you are blocked, don't hesistate to raise a new topic for this meeing. Otherwise, we'll do a status update in this meeting in about a month ? Would that work ? 21:09:09 #link https://review.openstack.org/#/c/142456/ 21:09:09 mestery: well we can identify concerned parties, which we are doing 21:09:18 ++ 21:09:18 yes 21:09:20 thank you 21:09:27 thanks ttx 21:09:34 lets do two weeks from today 21:09:36 not a month 21:09:44 if we hit a wall I want folks to know 21:09:56 fair? 21:10:10 anteaya: if you hit a wall you can add the topic and we'll discuss it right away 21:10:14 Yep 21:10:17 okay fair enough 21:10:19 thank you 21:10:20 Many of these peoplre are also at LCA 21:10:26 yes 21:10:27 So you can lock them in a room if needed 21:10:36 not oleg though unfortunately 21:10:40 True 21:10:44 anteaya: if it just goes well we'll update in one month 21:10:49 very good 21:10:59 I won't stay quiet you do know that 21:11:17 #action ttx to schedule a status update on novanet2neutron one month from now, unless a more urgent status update is needed 21:11:25 will any neutron folks be at LCA? 21:11:32 markmcclain will 21:11:40 and I am expecting gus 21:12:14 markmcclain: can you be a proxy for Oleg ? 21:12:15 OK, anything else on that topic ? 21:12:19 jogo: http://lca2015.linux.org.au/media/news/115 21:12:25 so we can get some f2f stuff done 21:12:35 oh anteaya beat me to it... :) 21:12:39 ttx I'm done 21:12:42 mtreinish: :D 21:12:59 mtreinish: that was the previous markmcclain 21:13:01 jogo: yes 21:13:14 #topic Discuss openstack-spec: log guidelines 21:13:21 #link https://review.openstack.org/#/c/132552/ 21:13:45 I even did another run of it today, hopefully working out the last kinks 21:13:46 This is a proposed cross-project spec. It basically needs to be reviewed by project teams to make sure those guidelines are as applicable to them as possible 21:14:09 There weren't strong opposition to it so far, most comments are about details 21:14:18 * ttx updates 21:14:32 Really need to get some Ops eyes on this. 21:14:34 sdague: I think at this point its safe to stop fiddling with wording, unless there's a major issue to be addressed :-) 21:14:55 But that may mean most PTLs just didn't look into it yet 21:15:01 getting info from ops about if we're logging too much or not enough would be useful 21:15:02 Minor wording changes can happen post merge if needed. Specs are not set in stone. 21:15:03 Rockyg: +1 21:15:03 dhellmann: yeh, I hope so, I did try to explain the parts that I thought confused people a ton 21:15:04 But, that said, Ops can always do a new/addition to this once this is out. 21:15:12 morganfainberg: ++ 21:15:16 I know that the keystone logs are useless from my dev perspective. 21:15:36 sdague: yeah, some of the new bits are helpful 21:15:47 bknudson: we need to work on that in general. Keystone logs are far less useful from a ops perspective than they should be as well. 21:16:10 doesn't need to be addressed in the spec... the spec looks fine to me. 21:16:22 this spec is several months old at this point, and I don't see any point in continuing to wait for more reviews 21:16:32 ++ morganfainberg 21:16:37 I'll score the spec post this meeting. It looks good to me. 21:16:43 Rockyg: think you can gather ops feedback on it in a reasonable timeframe ? Don't want to stall those opsenstack-specss forever trying to get everyone's feedback before approving 21:16:48 the only complaint I would be worried about is dropping audit log level if some group is using it 21:17:08 yeah I'll be looking at it post meeting too 21:17:10 bknudson: realistically, we've never had push back on that point 21:17:18 bknudson: I think audit is a bad log level - cadf or info. 21:17:31 that was in there from the original draft back in May 21:17:32 bknudson: also you still get the log, just as info 21:17:46 #action PTLs make sure to review (or get reviewed) the log guidelines spec as it's considered pretty much ready by now 21:17:51 ttx: There is a midcycle meetup in March for Ops. I think if this gets out there as is, it will inspire the midcycle to improve it. and that's not too much lag time 21:17:52 If it really is audit info, hiding it in the logs is bad. 21:17:57 sounds good. I don't think I've seen it anywhere 21:18:00 bknudson: No one should be using that. If they are it should be changed. 21:18:24 bknudson: I thought there was effort to remove it a while back anyway. 21:18:46 Also, I'll post the review to the devops list. 21:19:00 Rockyg: do you anticipate major changes being proposed? 21:19:12 Rockyg: ok, maybe just relay that the window of opportunity to comment before approval is closing fast 21:19:16 jungleboyj: it was all the same effort, people were just digging in early based on early versions of this 21:19:47 I expect tweaks to existing and additions and/or clarifications 21:19:59 sdague: Ah, good to know. 21:20:35 Key here is that seandague has fought thes logs a lot in his position in QA, so what is there is goodness for anyone who has to use the logs 21:20:42 sdague: are we going to move to a global log config format (at least a global config file for logging) to ensure consistent output? 21:20:54 I'll comment in the review patch too, but is there ever Fatal? 21:21:24 asalkeld: that's beyond the scope of this 21:21:28 at least for now 21:21:32 ok 21:21:42 it is very high level 21:21:53 this is hopefully to get all the projects roughly doing the same things internally with logging information 21:22:03 yes, it's more a log philosophy than an implementation 21:22:09 right, agreed 21:22:23 annegentle_: fatal? Higher than crit? I think crit meets that by spec definition. 21:22:24 asalkeld: there is work happening in oslo.log and oslo.context to standardize the format 21:22:30 do we then have blueprints for each project? 21:22:38 we basically have the same default format if you use oslo.log too 21:22:44 morganfainberg: sure, but the link referenced in the blueprint mentions fatal so I ask. Noting in a comment. 21:22:49 clarkb: right agreed 21:22:59 Ah 21:23:09 sdague: so we could let that bake for one more week and then have the TC tally the votes and approve it 21:23:10 annegentle_: ack 21:23:20 ttx: I'm fine with that 21:23:33 sdague: unless a big objection comes out 21:23:36 So, do you guys see this as the basis for a "principles in Logging" for OpenStack? 21:24:24 Rockyg: I wasn't anticipating anyone writing another document. Is that what you mean? 21:24:28 #info Expect the TC to tally the votes on this one at next week meeting, unless a big objection is posted 21:24:52 #action ttx to add log guidelines openstack-spec talkly to next week TC meeting agenda 21:24:53 I'm thinking at a minimum, the intro on the Wiki page for Logging Standards 21:25:06 Rockyg: dhellmann: I just commented on the review I'd like discussion on where this gets documented 21:25:18 I thought the spec *was* the documentation for this. 21:25:31 that's the point of having something that goes through the review process, no? 21:25:34 that sounds like the right place 21:25:41 anngentle_ ++ if it only lives in the spec, it doesn't really live. 21:25:44 we have the Ops Guide http://docs.openstack.org/openstack-ops/content/logging_monitoring.html 21:26:05 so we do already discuss it -- and I htink the scope would be "how does OpenStack approach/think about logging" 21:26:14 so yes, I'd like that specifically addressed in the spec 21:26:18 these are instructions for developers to follow when adding new logging calls and reviewing other patches with new logging calls. I don't think it needs to go into a manual anywhere 21:26:35 if we want to replicate the information in a format appropriate for other consumers, that's independent of this spec 21:26:38 dhellmann: those are for the specs, sure. I'd just like to see an operator doc in addition 21:26:38 dhellmann: ++ 21:26:43 dhellmann: agreed 21:26:43 dhellmann: okay, sure. 21:26:45 annegentle_ Ooh, good! but it also needs to link to developer docs so devs know how to write the log code chunks 21:26:51 dhellmann: ++ 21:26:56 annegentle_: anyone can translate that into a "this is the spirit of openstack logs" doc 21:26:58 Rockyg: meh, it's a different audience 21:27:11 but yes, different audience 21:27:32 Rockyg: hence my thinking the Ops Guide (which is not read generally by contributor devs to my knowledge) 21:27:40 Different audience, but we need to make sure the stuff stays in sync between dev and users 21:27:56 just to be clear: AIUI, this repository is a place for us to agree on policy, like the governance repo is for the TC. Once something is approved here, it can be referenced as official. Is that right? 21:28:06 dhellmann: Rockyg: I think the loop is closed if I log a doc bug for openstack-manuals to make sure the spec info goes into the Ops Guide 21:28:10 Definition of log levels, in particular, would appear on both sides 21:28:10 dhellmann: that's my understanding 21:28:10 good enough 21:28:17 annegentle_: ok, cool 21:28:19 dhellmann: yeah that sounds right to me 21:28:20 dhellmann: my understanding as well 21:28:33 alright, last comments on that topic ? 21:28:55 annegentle_ (I'll get your nick right eventuall;) yeah. So, maybe links between the docs? We can discuss offline? 21:29:55 just caught up again. ++ Let's give it one more week, let ops folks know, then merge after comments addressed. 21:30:03 ok, moving on 21:30:11 ttx: is the output from this specs repo being published to specs.openstack.org yet? 21:30:11 #topic Discuss openstack-spec: OSProfiler 21:30:14 Rockyg: sure, let's start a bug for discussion 21:30:21 hey hey=) 21:30:24 dhellmann: hmm.. maaaaybe ? 21:30:34 ttx: I updated spec and reply on some comments 21:30:40 dhellmann: nothing was approved yet :) 21:30:48 #link https://review.openstack.org/#/c/134839/ 21:31:00 This is the second cross-project spec that's been posted for a while 21:31:09 so I would like us to make progress on it one way or another 21:31:21 dhellmann and me filed a few questions, but otherwise I suspect most PTLs overlooked it 21:31:41 this is mostly done - right 21:31:43 ttx: so actually there was a request from dns team 21:31:45 boris-42: I think there was some general concensus that we wouldn't put release in the paths in the repo 21:32:00 sdague: oh 21:32:05 * ttx checks rev2 21:32:06 sdague: didn't know, so I will update it 21:32:12 err rev3 21:32:12 dhellmann: right? 21:32:16 So I'll review that spec but it looks like a lot of the concerns from the proposed keystone spec will still be relevant there. 21:32:21 * boris-42 sdague: after meeting 21:32:27 sdague: yeah, I think we agreed to that after boris-42 submitted this one 21:32:48 dhellmann: sdague sorry guys I am still on holidays=) 21:32:53 there were a lot of reviews of the previous version https://review.openstack.org/103825 ... is the new one pretty much a straight copy of that? 21:32:54 but I will fix it 21:32:56 At a glance that is. 21:33:05 boris-42: np, it's an easy change 21:33:06 eglynn: yep with fixed issues 21:33:16 eglynn: that was on previous one 21:33:30 cool 21:33:33 So actually there is request from desingate team 21:33:42 to remove agrugments from api-paste.ini 21:33:58 as we are using CONF file of project we can do that 21:34:01 thoughts? 21:34:13 I heard that operators hates confs in api-paste.ini ;) 21:34:29 Pleas do not add arguments to be consumed from the paste-ini. We have had issues with typing etc from keystonemiddleware and its better to have options all in one place 21:34:51 auth_token middleware can get options from api-paste.ini or .conf 21:34:53 Easier to not need all sorts of magic to make it work right when the project conf already does. 21:34:59 So I'll need to make 2 new releases and some amount of patches to remove them 21:35:10 bknudson: we fixed the issues. But it was a lot of workaround code to do it right. 21:36:01 My main objection to it was if there was a performance penalty when turned off, but I guess evaluating "if None" takes negligible time 21:36:06 morganfainberg: btw guys I can remove data from request that have tokens and credentials and so on 21:36:40 ttx: actually even when it is turned ON 21:36:52 ttx: but not triggered the overhead is same to "If None" 21:37:11 ttx: so it can be turned of by default imho=) but okay step by step 21:37:25 boris-42: I'll make sure to comment and we can figure out how to do that as needed. But I think most of the keystone concerns still apply to this spec. I will review in depth later today. 21:37:41 morganfainberg: thanks 21:37:47 morganfainberg: if you have any questions just ping me=) 21:37:56 Aye 21:38:01 boris-42: who would you say is the main audience for the profiling results ? Ops ? Devs ? Both ? 21:38:11 ttx: both 21:38:27 ttx: Ops - when it will be finished and work via Horizon=) 21:38:32 Rockyg: so it would also make sense to get ops reviewing this one 21:38:37 ttx: sure 21:38:49 want to make sure they would use it 21:38:59 ttx: so it really helps to debug why part of requests works slow on proudction env 21:39:00 ttx: Yup. so I should put this link into the email to the list, too. 21:39:26 I hope it's not keystone that's the slow part. 21:39:28 It soujnds like a great thing to have overall, I just fail to see the crowd cheering yet 21:39:31 bknudson: it was=) 21:39:40 bknudson: in some cases=) 21:39:43 so it doesn't seem to get anyone very excited 21:39:45 bknudson: but not in all=) 21:40:04 last thing we want is to have it in and nobody using it :) 21:40:16 boris-42: I just commented inline, but have you considered using x-openstack-request-id for the header rather than another new trace header? 21:40:29 so I would very much like ops to say "THIS please" 21:40:38 annegentle_: I will add paragraph aobut that 21:40:39 ttx: I don't think you have to worry about no one using it. Many are, they just don't talk about it. 21:40:42 annegentle_: why we need new one 21:40:47 annegentle_: ok? 21:40:59 I am a little alarmed with the number of similar things we have for this: logging, osprofiler, notifcations (for ceilometer) etc 21:41:09 ttx from a dev pov this is very nice, I have used it quiet a bit 21:41:10 ttx: it will be usable via rally perf job 21:41:11 boris-42: I'd like to avoid more and more headers if possible, it's hard to document and difficult for users to know which is used when. Just wondered if it's usable. 21:41:25 annegentle_: yep but in this case it's impossbile (or I don't know) 21:41:43 boris-42: ok, just asking if you considered it, then would be nice to explain if it's not possible 21:42:16 annegentle_: yep so I agree that it should be noted in spec 21:42:25 boris-42: ok, thanks 21:42:28 ttx: like load + profiling 21:42:54 ttx: so I think many Devs will use it + I don't know about other companies but in Mirantis guys are wating for osprofiler 21:43:06 ttx: as zhiyan is helping around integrating it in ohter project 21:43:25 ttx: and jamespage was interested as well 21:43:39 ttx: so there are consumers=) 21:43:40 I know some Huawei devs are using it 21:44:22 boris-42: some teams (keystone?) have been rejecting their own profiler spec, do you remember what their objections were ? 21:44:44 and were steps taken to solve those ? 21:44:58 jogo: I don't think ceilometer is intended to scratch the same itch directly, though it does act as the default collector for osprofiler 21:45:27 eglynn: this is another thing, there are already patches to make MongoDB collector 21:45:36 boris-42: cool 21:45:39 but for OpenStack gates and some installation 21:45:50 it makes sense to have Ceilometer driver cause it simplifes life=) 21:46:01 yeap 21:46:04 ttx: hm need more info=) 21:46:14 when I last looked at osprofiler, I raised some questions about its use at public cloud scale 21:46:21 eglynn: right, but we have a several similar itches and a very piecemeal solution to them 21:46:26 are there any large operators evaluating / using it? 21:46:57 boris-42: I hear sensitive information leaking was one of those concerns. We certainly don't want to filter that same sensitive information 3 times (in logs, in notifications, in profiling) 21:46:57 dhellmann: I don't know, it's not well know thing yet 21:47:15 ttx: Ah about sensitive info 21:47:20 anither question for the ops list? devananda 21:47:25 ttx: yep it can be easily removed 21:47:27 we've got rally code in keystone already 21:47:52 bknudson: so + profiler and ulitmate tool to figth for better perf and scale 21:47:52 =) 21:48:17 Txt boris-42 I'll look at the spec and the one that was proposed for keystone to make sure appropriate comments are echoed in the cross-project one. 21:48:35 I am concerned about needing more custom filtering at each layer. 21:48:39 morganfainberg: probably I should add more details 21:48:48 morganfainberg: about how we can remove sensetive info in spec 21:48:52 morganfainberg: so to make it clear 21:49:08 Sure. Like I said I'll comment. 21:49:09 morganfainberg: like in DB requests don't send args 21:49:11 boris-42: you said there's a mongodb backend. does that replace the use of ceilometer to store/retrieve profiling data, or augment it? 21:49:25 devananda: yep we are wokring on that 21:49:36 devananda: so you'll be able to use directly mongo if you setup your profiler in such way 21:49:45 devananda: and that scales very very well 21:49:48 boris-42: is there support for apache or gpl licensed back ends in the works? 21:49:51 devananda: cause it's one table with 1 index 21:50:07 devananda: good question. 21:50:08 boris-42: looks like this one needs a bit more iterations, and ops feedback too. Hopefully this discussion will trigger that 21:50:11 devananda: nope not at this moment, but as far as we get first backend 21:50:18 devananda: there will be way to add more of them 21:50:21 boris-42: indeed. I'm pleased to see a pluggable back end there, like mongodb. but I also know that this limits its use 21:50:22 and make it fly above radar 21:51:00 ttx: ++ 21:51:33 Sounds like a good idea overall, just want to make sure everyone agrees it's also the right way to do it 21:51:56 alright, last comments on that topic ? 21:52:11 Please review and comment on the spec to help it move forward 21:52:50 ttx: thank you for making meeting=) 21:53:02 ttx: I will try to add more info to spec during next week 21:53:10 no problem. We'll regularly review proposed cross-project specs during this meeting 21:53:21 good topic for cross-project, thanks boris-42 21:53:27 #topic Open discussion & announcements 21:53:52 On release side we had 1:1 syncs today, mostly restarting the engine as we come back from the holiday period. 21:53:57 Logs at: 21:54:01 #link http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-01-06-09.02.html 21:54:23 question for other PTLs - how well have split timezone IRC meetings been working for folks? 21:54:33 has anyone fallen back to a single time slot after trying to split? 21:54:39 Project meetings? Like nova does? 21:54:43 yes 21:54:57 Ah keystone still has a single meeting. 21:55:06 devananda: works ok for Heat 21:55:07 devananda: ceilometer use to stagger meetings but I don't think they do any more, right eglynn ? 21:55:14 We haven't tried a split though. 21:55:25 devananda: we had alternating slots for the ceilometer meeting for a long time, but settled back to a single slot about a year ago 21:55:35 it's a difficult balance. You want to be inclusive, but not split them to the point where they are not reaching critical mass either 21:55:39 basically, Ironic split a couple months back, but I've seen less attendance overall, and am wondering if it just takes time for folks to adjust 21:55:54 * ttx copypastes snippets of wisdom cross-channels 21:56:11 or if, by trying to accomodate everyone, we ended up making it harder on everyone ... 21:56:19 (6am and 10pm for me tho') - the morning one is a bit quiet from the US 21:56:27 devananda: then I'd say come back to single meetings 21:56:34 eglynn: was there a deciding reason for returning to a single slot? 21:56:43 devananda: TBH I thought the split caused some loss of context 21:57:13 devananda: ... different line-up turning for each alternating meeting 21:57:36 devananda: ... but for a wide geo-spread of contributors, could be the only realistic option 21:57:36 eglynn: well that is the point 21:57:39 Are both meetings reflected in the calendar? I only see one, but I have sync issues 21:57:48 Rockyg: both are in the cal, yes. 21:57:58 if the point is to sync the team up, it's hard to do that if there are 2 separate groups meeting 21:58:25 I thought splitting was meant to put a little of the timezone pain on everyone, not create 2 separate groups of attendees 21:58:25 dhellmann: yeah, but just totally ignoring part of the team is not good either 21:58:35 asalkeld: sure 21:58:58 but when you split, you still want most of the team to come at both times, not create 2 separate groups 21:59:10 dhellmann: right, exactly 21:59:12 dhellmann: true, but for some folks though one of the alternating slots just wasn't possible to attend 21:59:34 eglynn: that's ok as long as it's a minority -- you're never going to get everyone to all of the meetings anyway 21:59:44 dhellmann: true that 21:59:57 ok, last minute 22:00:05 Anything else, anyone ? 22:00:10 thanks for the perspectives, all 22:00:22 for cielometer, we made it harder on ourselves by having the alternating slots on different days of the week also 22:00:39 leading to a short-week, long-week pattern 22:00:52 ok, time to close 22:00:55 thx everyone 22:00:59 #endmeeting