18:01:08 #startmeeting Keystone 18:01:08 Meeting started Tue Mar 3 18:01:08 2015 UTC and is due to finish in 60 minutes. The chair is morganfainberg. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:10 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:13 The meeting name has been set to 'keystone' 18:01:17 morganfainberg, you made it! 18:01:29 #topic Bandit - a security focused static code analysis tool 18:01:30 hi 18:01:34 lets just get right into it 18:01:37 tmcpeak1 o/ 18:01:47 hi all, I'm tmcpeak - I usually hang out in #openstack-security 18:01:48 Can you do Static analysis on Python? 18:01:54 sure 18:01:59 tmcpeak1, lies 18:02:01 Ha 18:02:05 lmao 18:02:06 well 18:02:06 * ayoung jaded 18:02:07 kinda 18:02:10 :D 18:02:17 ayoung, i can look at code in python... thats the same thing right? :P 18:02:22 so for those that haven't heard of Bandit, it's here 18:02:28 just use the @deprecation tag :) 18:02:33 * stevemar sneaks in late 18:02:35 * morganfainberg quiets up for tmcpeak1 to continue. 18:02:35 https://github.com/stackforge/bandit 18:02:41 static analysis for python must be tough with all the dynamic binding 18:02:42 #link https://github.com/stackforge/bandit 18:03:03 and #link https://wiki.openstack.org/wiki/Security/Projects/Bandit 18:03:19 so basically the idea here is to call attention to low hanging fruit in code that is checked in 18:03:31 we have about 40 tests 18:03:33 tmcpeak1, I assume that this is being run against Keystone, and any weaknesses found will be treated as a security bug until triaged? 18:03:38 so far, many of them are very very noisy 18:03:45 * lbragstad walks in late 18:03:59 yes, so the reason I'm here is that I'd like Keystone to volunteer to run Bandit in a gate 18:04:08 bknudson expressed some interest in this 18:04:11 tmcpeak1: as in, false positives? 18:04:18 tmcpeak1, hmmm 18:04:25 Gate is the wrong place 18:04:31 o/ : ) 18:04:34 we don't want to publically announce any problems 18:04:36 so the profile that I've made for Keystone is conservative and currently doesn't have any results 18:04:37 I've been working on a tox env -- https://review.openstack.org/#/c/157930/ 18:04:43 here's the issue 18:04:50 ayoung: these are publicly available anyway, Bandit is opensource 18:04:53 the point of putting it in the gate is to stop bad code getting in to begin with. 18:04:55 lets say Bandit gets smart about some new problem 18:04:56 you can just pull it down and find the same thing the gate test finds 18:05:07 and the gate now announces to the world :Keystone has vuln X 18:05:26 yeah, but that is very publicly available anyway 18:05:37 tmcpeak1, not really 18:05:39 tmcpeak1: it sounds like you're implying we have violations against the full suite of tests? 18:05:40 any attacker would already have access to that information 18:05:45 after the vuln is fixed and commited 18:05:54 tmcpeak1, this should be run, but.... 18:05:58 dolphm: yeah, the full suite of tests include some very noisy ones that need to be tuned 18:06:08 OK...so it would be something like this: 18:06:11 what about publishing those results to a private page? 18:06:22 1. run it privately, file bugs, get Keystone as secure as Bandit can say 18:06:23 I've created a bandit config file for use with Keystone 18:06:25 http://pastebin.com/SeBkRvCS 18:06:28 #link http://pastebin.com/SeBkRvCS 18:06:30 the plan is to have bandit work very similar to pep8 18:06:40 please check out that link 18:06:43 2. No upgrades to Bandit in Gate until private runs have the same level of fix 18:06:52 as with pep8, a new bandit wouldn't be run against keystone in the gate without a requirements bump. 18:07:07 tmcpeak1, you understand my concern? 18:07:15 ayoung: absolutely 18:07:27 We do things like this internally, too. We run tools like Valgrind etc. And this is fantastic 18:07:30 this means bandit would need to be locked separately from requirements 18:07:31 so with what in my opinion is a sensible profile, Keystone already has no vulnerabliities 18:07:37 here's output of recentish Keystone 18:07:38 since global reqs overrides requirements.txt 18:07:39 but we don't publish the results. We consume them internally and fix 18:07:41 #link http://pastebin.com/U0Fpjar1 18:07:46 tmcpeak1, i'm running it locally right now on keystone fwiw 18:07:47 ;) 18:07:52 sorry - here now. traffic was a killer 18:08:09 morganfainberg: oh cool, you're already using it? 18:08:25 so yeah, some of the tests in the full suite generate *tons* of false positives 18:08:31 tmcpeak1, just ran it 18:08:37 tmcpeak1, took me 30 seconds to enable 18:08:44 the idea with the profile I pasted above is that it will really only flag things you don't want in your code 18:08:53 so, i see sqla issues and complaining about popen 18:08:56 mostly 18:09:05 morganfainberg: use the above linked profile please 18:09:06 popen is awesome! 18:09:07 sql tests are noisy 18:09:08 and binding to all interfaces (desired) 18:09:09 tmcpeak1: please be clear - you're suggesting that there's no value in reporting the violations that bandit is currently seeing in keystone? 18:09:15 earlier it found a problem where regular random was used, so I posted a fix for that: https://review.openstack.org/#/c/157990/ 18:09:37 dolphm: no, not at all. I'd like to see folks using Bandit on a regular (manual) basis to close bugs 18:09:48 I also see value in an automated gate process which prevents egregious checkins 18:09:50 please see this 18:09:50 and it also complained about the use of 0.0.0.0 for default bind host: https://review.openstack.org/#/c/157975/ 18:10:09 (before I put this, please note this is all publicly available info, I'm not disclosing 0 days) 18:10:10 but I didn't have tmcpeak1's profile. 18:10:15 #link http://pastebin.com/G7iVFGZD 18:10:21 bknudson: I just made it this morning :) 18:10:38 we'd like to prevent this kind of code from getting into Keystone 18:10:48 just be glad we're not trove. 18:11:05 l 18:11:16 ayoung, so presumably something real caught in a gate test wouldn’t actually get merged until fixed. Would you still consider that a vulnerability in Keystone even if it hasn’t been merged? (I’ve also been hacking on Bandit) 18:11:22 so tmcpeak1 , with that profile i see no violations? 18:11:34 yeah, here's an output of Keystone with that profile 18:11:41 ljfisher, the concern isn't new code. we agree that is fine 18:11:41 command to run Bandit against Keystone with verbose profile: bandit -c keystone.yaml -p keystone_verbose ~/Documents/projects/keystone/ -r 18:11:48 Keystone Bandit conservative run: http://pastebin.com/U0Fpjar1 18:12:00 ljfisher, the issue is old code when bandit gets smarter 18:12:02 ljfisher, so my concern is that we need to know about new vulnerabilities before they show up in the gate. My concner is not with changes, but existing code. So...I think this is OK... 18:12:12 morganfainberg: for the gate we are pinning a version 18:12:14 but we need to have additional review done when Bandit gets update 18:12:18 new Bandit functionality won't change your gate 18:12:30 you'll pin to a version, and chose when to update 18:12:42 same as you do with other dependancies 18:12:42 pinning version is good 18:12:44 tmcpeak1, pins in requirements.txt are overriden by global requirements 18:12:45 FYI 18:12:48 o/ 18:12:52 you can't pin in requirements.txt 18:12:59 the pin will go into global requirements 18:13:03 M y concern is not Keystone, but Bandit. Before Bandit posts a new update...we'd want to know what it caught. 18:13:08 the pin for hacking is handled differently. 18:13:31 it's not automatically updated. 18:13:48 bknudson, we'd need this to fall into that same category then 18:13:52 OK...Gate is a red herring. I have no problem with running Bandit in gate 18:14:12 cool :) 18:14:17 ayoung: you still concerned about vulns though? 18:14:17 tmcpeak1, yeah i'd be fine with this being an additional check - i'd like this to not be lumped into pep8 though 18:14:28 tmcpeak1, if infra doesn't mine i'd like to see this as a specific job 18:14:33 morganfainberg: same, I'd like it to be a separate job 18:14:36 yeah 18:14:42 if there are concerns from infra we can put it in pep8 18:14:50 What we do need is to know about new classes of vulnerabilities before we make the tool general available. And that is not just a Keystone issue 18:14:53 but my opinion is 1st go w/ separate job 18:14:59 the requirements for this job are different than pep8 18:15:08 bknudson, exactly 18:15:11 since bandit doesn't need all the test-requirements like pep8 does. 18:15:28 and having it clearly be a separate failure would be very good. 18:15:32 ++ 18:15:35 tmcpeak1: have you successfully used bandit to identify and fix actual vulnerabilities? 18:15:40 ayoung: I definitely hear what you're saying. Another piece which I'm not mentioning here because it isn't directly related to my gate test mission is getting people to use it against their own projects 18:15:52 tmcpeak1, so i'm not opposed to this being enabled for keystone. 18:15:54 dolphm: absolutely. One place I've had great success is trove 18:16:13 let me dig up one for you 18:16:34 tmcpeak1, i assume the .yaml will be hosted in keystone's tree [for sanity sake] 18:16:47 https://bugs.launchpad.net/trove/+bug/1349939 18:16:49 Launchpad bug 1349939 in Trove "Multiple vulnerabilities in Couchbase implementation of restore strategy" [Critical,Fix released] - Assigned to Amrith (amrith) 18:16:55 so this bug in Trove was… really bad 18:16:57 not that i expect ot be changing it a bunch, but to save you a headache of updating. 18:17:08 I found this using Bandit and then tracing through the code 18:17:10 for every project in the future. 18:17:22 tmcpeak1, this has the potential to be a Pandora's box. But the box is already open I guess. 18:17:30 this would have resulted in VMT process if Trove had been in production already 18:17:41 no OSSA only because it was in master. 18:17:49 so it caught a bug before release. 18:17:51 morganfainberg: yes, I'd really like for each project to maintain their own profile 18:17:57 so... thanks! 18:17:59 tmcpeak1, cool 18:18:20 ayoung: yeah, box is open :) 18:18:57 tmcpeak1, so step 1 for Keystone, - experimental job in gate, lets get the keystone tree in shape for this, step 2 - non-vote, once we're happy - voting 18:18:57 cool, so I didn't want to take up all your time 18:19:02 tmcpeak1, the bug you mention has been fixed in trove, yes? 18:19:07 I remember doing it a while ago. 18:19:09 morganfainberg: perfect 18:19:22 amrith: yeah, you fixed it 18:19:24 tmcpeak1, ping me when you have the reviews to change project-config up, i'll +1 them as needed 18:19:29 tmcpeak1, thx 18:19:34 we need an official release of bandit, also. 18:19:41 morganfainberg: perfect, thank you 18:19:57 lets make sure bandit is released/in pypi etc 18:20:01 yeah, there are a couple of more steps we need, but I'll be reaching out to a few of you to push this along when we're ready 18:20:19 want to make sure the user (your) experience is rock-solid 18:20:37 so it's going to be clean to begin with 18:20:43 tmcpeak1: i just hope it doesn't prove to be a noisey tool spewing false positives at every contributor & reviewer 18:20:43 any new changes will eventually go through bandit 18:20:55 tmcpeak1, thanks. and make sure you get the experimental stuff in infra first, so we can do "check expirimental" as we implement the yaml etc in our tree 18:20:59 and if it's flagging things we need to decide if it's a false positive. 18:21:10 dolphm: if it is, we (Bandit team) have completely failed. We're going to test the crap out of it before we ask you guys to use it 18:21:23 tmcpeak1: is there a feature similar to # noqa for flake8 ? 18:21:24 There's a #nosec comment you can put on a line to tell bandit to ignore it. 18:21:30 ^ :) 18:21:47 our goal is not to have you guys using #nosec to get it to shut up 18:22:06 nosec should basically be like "yeah we understand this isn't good from security but it's really ok because xyz" 18:22:22 there are cases we need to noqa 18:22:27 there will be limited cases to nosec 18:22:27 tmcpeak1: i'd expect a small handful of legitimate # nosec with explanations 18:22:36 I've found bugs like that before too.. In nova they were generating a session ID in an uncryptographically safe way, turns out it didn't matter 18:22:40 but those will be the exception to the rule 18:22:59 so that's a perfect use for #nosec followed by a comment about why it doesn't matter 18:23:01 tmcpeak1, ok i look forward to seeing this become available. 18:23:24 the Nova guys actually had pushed a comment explaining why it isn't a security risk, so next guy doesn't come and panic about uncrypto session ID 18:23:34 cool, sounds good! 18:23:57 tmcpeak1: # nosec should require an inline comment at least! 18:24:03 definitely 18:24:09 tmcpeak1, yeah was just going to suggest that 18:24:13 tmcpeak1: # nosec: {why} 18:24:22 make it so #nosec can't be just bare on the line 18:24:28 morganfainberg: ++ 18:24:35 yeah, that's an awesome idea 18:24:51 morganfainberg: although you probably need more than just an inline comment to justify such a "why" 18:24:51 I'll add that to our dev list :) 18:25:01 yeah, they're probably going to be somewhat involved 18:25:15 honestly with current #nosec handling I'm not sure the best way to do it, but I agree we should require something 18:25:34 tmcpeak1: maybe # nosec requires a comment on the preceeding line? 18:25:46 we're dealing with AST and tying together multi-line comments is… at least a little challenging 18:26:05 I'll throw it on our backlog though 18:26:06 tmcpeak1: that's the fun! 18:26:17 tmcpeak1, dolphm throws down the gauntlet... ;) 18:26:30 and in the mean time we can have project reviewers throw liberal -1's for promiscuous #nosec usage 18:26:39 it shall be done :D 18:26:54 ok so we'll look for the infra changes and then the stuff to make bandit a reality 18:27:00 tmcpeak1, thanks! 18:27:04 sounds great! 18:27:05 thanks all 18:27:11 #topic Specs that will need Feature Freeze Exceptions (FFE) and/or be delayed until Liberty 18:27:22 So, it's that sad time of the cycle 18:27:29 :( 18:27:35 :( 18:27:35 :( 18:27:35 time to punt things to either FFE or next cycle 18:27:46 #link https://launchpad.net/keystone/+milestone/kilo-3 18:28:49 i never understood why this is a sad time... it'll go in a few week later is all :P 18:28:52 currently on the chopping block [for FFE unless it is gating today, no particular order]: x509, Domain-SQL, Reseller, Improve list role assignments filtering performance 18:28:57 So I really want to keep domain specific configs in Kilo….we are doing pretty well on reviews…but not sure all will get in by the 5th 18:29:34 if those are not gating today, they are getting a -2 (procedural) and require a FFE after k3 cuts, or a delay to liberty 18:29:40 fwiw i do think the domain-sql stuff is close 18:29:56 i think domain-sql is the best option for FFE, reseller is a close second 18:30:16 the others i'm not optimistic they will be accepted for FFE 18:30:23 remember FFE isn't just us, keystone-core 18:30:23 there is also this https://blueprints.launchpad.net/openstack/?searchtext=idp-id-registration 18:30:27 it's also rel-mngmt 18:30:34 and review that is held off by a bug. 18:30:35 for reseller, we need more eyes on it though, 0 reviews so far 18:31:13 we are about to send the last change of the chain 18:31:15 marekd, and that one i think will be pushed 18:31:30 marekd, as much as i'd like to see it. 18:31:41 morganfainberg: assuming the reviewers are happy with my updates, the core underpinnings of domain-SQL may well be gating today, not just the “switch on” at the top 18:31:55 henrynash, and thats why i said gating today 18:32:03 if it's gating actively, we'll make sure it lands 18:32:10 morganfainberg: me too, i wanted to discuss it later (post-meeting) 18:32:14 morganfainberg: ok, agreed 18:32:17 if it isn't lets plan a FFE 18:32:31 that one the FFE would be "yeah we couldn't land this in k3, but it's ready to go" 18:32:49 morganfainberg: agreed 18:33:12 henrynash, i'll take another look at your stuff today, but i looked at the comments you had made and i think you addressed all my concerns 18:33:25 stevemar: great, thx 18:33:32 other than that. 18:33:41 i have nothing else for the meeting 18:33:55 #topic Open Discussion 18:33:56 ok, so maybe i can use some of this time? 18:34:00 go for it 18:34:06 henrynash, https://review.openstack.org/#/c/160872/3 make a stand alone patch and I'd be ahppy to +2A it 18:34:20 if depends on Add API support for domain config and there is no need for that 18:34:36 ayoung, i'll rebase it on master and we can push it through? 18:34:39 https://review.openstack.org/#/c/152156/ this review is actually kind of blocked, it passes the unittests but will probably explode with mysql 18:34:47 stevemar, go for it 18:34:48 ayoung: true..but then the follow on patches can’t depend on two things 18:34:48 do every devstack installation following master 18:34:52 the bug is here: 18:35:04 henrynash, reverse the order 18:35:13 #link https://bugs.launchpad.net/keystone/+bug/1426334 18:35:14 Launchpad bug 1426334 in Keystone "DB migration problem with federation extension" [Undecided,In progress] - Assigned to Marco Fargetta (marco-fargetta) 18:35:14 put the trivial changes at the front of the patch-stack 18:35:25 ayoung: ok….yes, I’ll push it to the front, ++ 18:35:37 and as i talked with Marco, adding a migration script will not help 18:35:39 thanks 18:35:39 henrynash, hit that cherry pick button 18:35:50 as sanity_check in oslo_db will raise an exception. 18:35:55 If someone want review the reseller patches :) 18:35:58 #link https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:bp/reseller,n,z 18:36:02 marekd, and marco has done some work there. 18:36:03 stevemar: it’s standalne, justa reorder of dependant pacthes 18:36:23 marekd, i think we need docstrings for his fix and a bug against oslo to reference 18:36:38 marekd, my major complaint with his patch 18:36:43 https://review.openstack.org/#/c/159803/5/keystone/common/sql/migration_helpers.py ? 18:36:48 morganfainberg: ^^ 18:36:51 though the code could probably be claeaned up some 18:36:52 marekd, yeah 18:36:59 I wonder why the db migration problem doesn't show up in the gate? 18:37:35 sqlite maybe ? 18:37:41 bknudson, ^ 18:37:41 or theu use mysql? 18:38:16 morganfainberg: oh, i see, he is esentually recreating all the tables 18:38:26 bknudson, because the gate default assumes utf-8 iinodb 18:38:38 this is a case where someone has misconfigured their mysql then pushed in keystone 18:38:41 then get wedged 18:38:44 but if healthy actually? it may move lots of data in some rare cases. 18:38:46 morganfainberg: I thought they changed the gate defaults just to catch this 18:38:48 * jamielennox sneaks in the back 18:38:57 bknudson, clearly not :P 18:39:10 or we did something wonky 18:39:49 we've made fixes in the past to set tables to InnoDB 18:40:02 the issue here is it related to FKs. so it's ugly 18:40:03 I don't think we've tried to fix the charset. 18:40:11 bknudson, we did, migraiton 37 18:40:14 morganfainberg: for the utf-8 patch we would then need to remove 001, 002 files 18:40:15 migration* 18:40:40 marekd, no, we fix them in-place as we have then we do the exceptional case when it's broken to catch anyone who's run them 18:40:52 marekd, look at the _Fix_migration_37 example 18:40:55 this is all sorts of ugly 18:41:01 it's not pretty but it unwedges the deployer 18:41:13 but we need to figure out how to catch this earlier. 18:41:25 maybe a hacking check to ensure table definitions in migrations have innodb/utf8? 18:41:40 or a way to make the sanity check less sane in an exceptional case so it can be fixed. 18:41:51 we do have test already for table structure 18:42:00 bknudson, but not in migration. 18:42:04 and I think there's a "live" test for innodb. 18:42:23 yeah. anyway this needs some eyes to maek sure we solve it this time ;) 18:43:39 morganfainberg: where is that _Fix_migration_37 ? 18:43:48 in migration_helpers 18:43:53 marekd: line 142 18:43:59 https://review.openstack.org/#/c/159803/5/keystone/common/sql/migration_helpers.py 18:44:23 marekd, https://github.com/openstack/keystone/blob/master/keystone/common/sql/migration_helpers.py#L142-L181 18:45:15 morganfainberg: ok, i will add a docstring 18:45:26 so i think we could make that code in marco's patch better by dropping the FK constraints and, fixing the table, and re-adding the constraints 18:45:30 rather than recreating the tables 18:45:49 less data mangling involved 18:46:28 this need to be done all today? 18:46:34 marekd, that is a bug 18:46:39 that can happen anytime it lands 18:46:59 the dependant BP can't land until afterwards and might be FFE *or* just pushed until liberty 18:47:13 so i'd say fix the bug, ask for FFE if we want that in kilo 18:47:28 ++ 18:47:44 but start with the bug fix, which can land anytime before rc [sooner *is* better] 18:48:31 anything else? 18:48:43 before we close up the meeting and let infra have the channel early? 18:48:43 how do i ask for FFE? ml-thread? 18:48:47 with some explanation 18:48:49 marekd, yeah 18:49:18 why this should land in Kilo, why it wasn't able to land before FF, what the status of the code is 18:49:19 etc 18:49:25 sure. 18:49:29 ok enjoy your spare 10minutes! 18:49:33 go get a cup of coffee :) 18:49:37 morganfainberg: remind me, do we ship python-keystoneclient with the serber or separately? 18:49:40 jamielennox, you probably need the coffee 18:49:49 (server even) 18:49:52 henrynash, separate, but we try and do a final release around RC 18:49:57 morganfainberg: i could have stayed in bed :) 18:50:03 morganfainberg: ok 18:50:08 that is the releasr that ships with the named integrated release 18:50:11 jamielennox: it's....4am now? 18:50:24 also expect a new ksm / ksc to go out shortly into march here 18:50:29 no, nearly 6 - sydney time now 18:50:44 i'll be bugging people for ksc reviews and ksm stuff if we have something to release. 18:50:48 post k3 that is 18:50:54 thanks everyone! 18:50:58 #endmeeting