18:01:55 #startmeeting keystone 18:01:56 Meeting started Tue Jun 7 18:01:55 2016 UTC and is due to finish in 60 minutes. The chair is samueldmq. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:57 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:00 The meeting name has been set to 'keystone' 18:02:01 \o 18:02:07 welcome everyone 18:02:24 o/ 18:02:26 we have 4 topics to discuss today 18:02:35 #topic Mapping shadow users into projects and roles 18:02:41 dolphm: o/ 18:02:41 samueldmq: i have an open topic at the end to add 18:02:50 samueldmq: so just save me ~5m at the end pl 18:02:52 z 18:02:59 notmorgan: sure 18:03:24 is dolphm around ? otherwise we can just start with another topic and then circle back 18:03:36 this is autoprovisioning. 18:03:48 yes 18:03:50 We've had a request for this for a long time 18:04:02 dolphm: nice, the floor is yours 18:04:31 #link https://review.openstack.org/#/c/324055/ 18:04:46 so, this spec came directly out of the austin summit 18:05:15 the gist is that it makes a lot of sense to do some basic provisioning of authorization-related things in mapping, especially for federation use cases 18:05:24 since we have a chicken & egg problem otherwise 18:05:53 ayoung's "Federated query APIs" specification has a pretty good illustration of that chicken & egg problem 18:05:58 dolphm: imnsho +2. 18:05:58 dolphm +++ 18:06:10 (imnsho = in my not so humble opinion) 18:06:20 dolphm, question...is control of this going to be managed solely by the mapping file, or will we use an additional domain specific config to "gate" whether a projedct can be automatically created within a specific domain? 18:06:38 so, the solution here is basically to lazily provision things like projects and role assignments during the auth process - after a user has been authenticated, but before we give them a token for the first time 18:06:45 ayoung, there is a domain attribute 18:06:47 so that token can actually be immediately useful, with a project, roles, and all 18:07:11 Actually...I'd like to see a domain specific config value "allowed_idps" for mapping in general, and then autocreate is on top of that 18:07:30 ayoung: that is a fair request. 18:07:35 ayoung: i think. 18:07:41 ayoung: as part of the spec, i'm envisioning mapping files to become a bit more domain-specific, if that makes sense. i think domain admins should be able to manage their own mapping files 18:07:50 idps should be tied to domains since the beginning 18:07:50 dolphm: ++ 18:07:50 notmorgan, yeah, with the config options in the database, this is more practical, too 18:07:52 they are domains 18:07:56 dolphm: so (as you know) I support this (although have a follow up concern whcih is next on teh agenda, but that doesn’t stop the conceptual idea here) 18:08:00 dolphm, 10000% yes 18:08:09 so, there's a domain ID attribute in the mapping which is the key to that notion 18:08:21 henrynash_: and i have a very strong view on that. but i'll hold until your topic. 18:08:35 notmorgan: cool 18:08:46 so where is the disagreement here? 18:08:49 dolphm: could this eventually solve the "i project 1234 here and when i federation there i also get project 1234"? 18:08:58 shaleh: I don't think there is one :) 18:08:58 dolphm, I think that if we say "create the domain, create the domain specific config, specify idps for domain, allow auto provisioning" in that order we are on the right track 18:09:24 are we doing on demand creation to save DB space or something? 18:09:28 shaleh: this spec is new since last meeting, so i was just looking for an opportunity to socialize it, and answer any questions 18:09:40 dolphm: k 18:09:44 i'm on board 18:09:46 then the mapping can be uploaded safely, as it can only map to specific domains 18:09:55 shaleh: on demand or pre-create doesn't really change the nature of the spec. 18:10:04 shaleh: to save space versus pre-provisioning a bunch of projects you may not need? 18:10:08 notmorgan: agreed. I was asking about the detail 18:10:21 henrynash_, you are the Domain specific config guru. Does what I say make sense? is it practical? 18:10:32 and also to get around the weird ux of having a user hit keystone (uselessly) in order to assign them stuff 18:10:37 would it be a good idea to let admins explicitly set when a project is expected to exist ? 18:10:43 in the mapping ? ^ dolphm 18:11:01 samueldmq: expand on that 18:11:02 samueldmq: i replied to your comment in the review 18:11:15 samueldmq, good point 18:11:17 samueldmq: L141 18:11:28 * samueldmq nods, looking 18:11:54 on first read this sounds reasonable. I need to think about it some. 18:11:56 dolphm, I'll respond to the review request with these suggestions. 18:12:02 samueldmq: rodrigods: the suggestion implies a lot of complexity without a well defined use case to support it 18:12:04 ayoung: probably…but need to dig in deeper 18:12:19 shaleh: cool - you'll have plenty of time to digest :) 18:12:22 dolphm, good point too in the reply :) 18:12:28 dolphm, did you consider HMT? 18:12:37 i mean... the mapping is flat 18:12:44 rodrigods: nah, who wants that :-) 18:13:03 rodrigods: it's a list of projects, and each project object could have a parent ID, description, etc -- the other things a project requires 18:13:09 dolphm: default would be true, but if it's set to create=False and it exists... mapping invalid ? 18:13:11 dolphm, ++ 18:13:17 dolphm: that doesn't work, I agree that just adds complexity 18:13:24 still reading but in general i like it 18:13:42 dolphm, and... inherited roles? the same? 18:13:50 jamielennox: i believe it's what you were describing on the mailing list 18:14:02 in the thread with amakarov 18:14:20 dolphm: whilst i need to think about the domain admins managing this - why does that need to be part of this spec?? 18:14:22 dolphm: this seems to cover a chunk of the provisioning concerns. so i am a fan. 18:14:33 dolphm: yea, but i assumed it would be more difficult than this :) 18:14:57 rodrigods: i imagine inherited roles could be expressed as an attribute on the role object in the project definition? so for example, inherited=True (or whatever the attribute is called in the role assignment API) on L144 for example 18:15:11 dolphm, yeah, makes sense 18:15:19 dolphm, suggest just to add a note about them there 18:15:54 jamielennox: good question. the constraint is not a hard requirement, but it simplifies the mapping definition (otherwise, you'd have to specify a bunch of domain IDs in the mapping, for each project and all) 18:16:25 jamielennox: doing it once at the top seems to solve everyone's use cases that i've talked to, and simplifies the security model quite a bit 18:16:44 dolphm: ++ 18:16:54 notmorgan: a chunk? where would you want to take it next / what does it not cover? 18:17:02 nice, any other concerns on this spec ? I think we seem to have an agreement on it 18:17:05 notmorgan: i assume you agree this is MVP for the idea 18:17:09 dolphm: yes. 18:17:14 dolphm: and can be built on. 18:17:14 henrynash_, I'd be OK with, instead of them being domain specific config options, they go on the domain object in the resource backend. 18:17:23 * topol felt very MVP to me as well 18:17:31 dolphm: i don't see anything clearly needing to be added now. a good starting place 18:17:35 notmorgan: ack 18:17:36 samueldmq: thanks, we can move on 18:17:42 dolphm: perfect 18:17:45 dolphm: and like i said +2 :) 18:17:52 so please review the spec, and let's move on 18:17:53 ayoung: but could you write policy rule to control management? 18:17:58 dolphm: yea, in general i would think it simplifies most peoples case and helps the domain admin model, but there is some additional validation i think you would want to see like if you specify a domain id for a whole mapping then you shouldn't be able to specify a different domain id for any project or role 18:18:03 henrynash_, I don't think you need to. 18:18:07 #topic Should shadowing users be optional? 18:18:12 dolphm: to the point where it should almost be a different spec 18:18:13 henrynash_: o/ 18:18:16 * notmorgan grabs soapbox. 18:18:20 ah oops 18:18:21 henrynash_, it could be enforced by the rules engine at mapping time, or even confirmed when the mapping is uploaded 18:18:31 ayoung: later, i’;m up 18:18:49 also where is that domain name immutable flag 18:19:06 jamielennox, that is a global value 18:19:16 jamielennox: agree, and i forgot about that flag 18:19:18 ok, please coninute this convo on the spec 18:19:23 so my concern here is about whether we NEED to have shadow users…what If I don’t want local direct assigments…and have multidatcentres….don;t I know have another bigs table to replicate? 18:19:24 notmorgan: ++ 18:19:29 notmorgan: ++ 18:19:35 henrynash_: yes. we do. 18:19:39 let's focus on the current topic 18:19:44 henrynash_, would that be on a per Idp basis? 18:19:45 henrynash_: this should not be optional 18:19:49 What is this place? 18:19:53 notmorgan: becasue.... 18:20:09 the data doesn't move that fast, it simplifies out internal code paths AND security model 18:20:20 and it prevents us from many many extra edge cases 18:20:25 AND you say? 18:20:29 notmorgan: ++ 18:20:30 henrynash_: replication shouldn't be terrible unless you are adding/removing users at lightning speed 18:20:30 henrynash_: so... your use case is that you have separate clouds that should not have replicated identity data? 18:20:41 henrynash_: why are you replicating in the first place? 18:20:48 have multiple code paths, security models, etc all means we're battling a lot of possible paths of maintenance and optimisation 18:20:49 many many you say? 18:20:59 dolphm: I am assuming federation, but replication projects and assignments 18:21:11 yea, i don't think shadow users should be optional 18:21:15 optimization u say? 18:21:30 shadow users you say? 18:21:30 fungi, jeblair, mordred, might need a kick here in this channel (cc anyone who is op) 18:21:32 there are too many things that will hopefully rely upon it like direct role assignment 18:21:36 can we find a way to prevent the need to replicate? 18:21:58 shaleh, no, I don't think so 18:22:01 so why do we need direct role assignment if we have this mapping faciity we sjust discussed? 18:22:03 shaleh, actuallllly 18:22:04 yes 18:22:17 fungi, jeblair, mordred: will ping if it becomes really important. (thnx) 18:22:21 notmorgan: thanks 18:22:22 Do any of you know what are you are talkibg about or arw you just using fancy words with no meaning? 18:22:27 henrynash_, so, what if we forced the id_mapping approach on shadow users 18:22:30 BroNeed: are you here for an actual discussion, or are you just spamming the channel? 18:22:38 Yes 18:22:51 I asked what is this place 18:22:53 henrynash_: mapping relies on direct role assignments 18:22:54 (don’t ecnourage it) 18:23:01 dolphm, pretty sure it is a bot 18:23:08 ayoung: agree 18:23:09 Lol no 18:23:14 I am no bot. 18:23:22 BroNeed: please don't disrupt the meeting unless you have something to contribute 18:23:33 Just asking what this placw is 18:23:40 ^topic 18:23:44 Yes mam 18:23:52 ayoung: you think we can do away with the replication? 18:23:57 dstanek, yep 18:24:05 if the userid is deterministic 18:24:12 it will be the same in all places 18:24:16 for LDAP idMapping we do 18:24:18 henrynash_: i don't see the data being created by shadow users to be particularly burdensome for replication 18:24:19 redrose: if the mapping is going to create an assignment (whcih is what we just dsicused), why couldn’t it use teh epemeral user_id created by the mappoing 18:24:22 considering all the work Fernet puts into limiting replication, I'd rather not add more 18:24:31 sha256{userid, domainid} 18:24:42 we'd make Federation use idp instead 18:24:43 henrynash_: as usual, there will be bursts of writes when new users arrive, but that's true today anyway 18:24:47 sha256{userid, idpid} 18:24:54 with the userid value the value from the assertion 18:24:58 user id being deterministic doesn't get you things like role assignments that were created 18:25:06 jamielennox: ++ 18:25:21 but role assignments could be replicated without too much worry -- 18:25:29 notmorgan: so could user ids 18:25:35 jamielennox: sure. 18:25:43 jamielennox: i'd argue you'd probably replicate it all. 18:26:05 but i see this as non-optional, is my only point, not digging into replicate-or-not-to-replicate 18:26:17 shaleh: it's also the difference between rather static data (users, roles, projects), and highly disposable data (tokens) 18:26:18 jamielennox, but if there are explicit role assignments, those would have to be replicated anyway. henrynash_ 's complaint is that his setup does not have direct role assignments in the normal case 18:26:23 i’d just seems really odd that we have federation, then we require keystone to replaciate around shadow versions of this same info 18:26:26 i'm a firm -1 here because it just makes things too complicated. 18:26:34 dolphm: understood 18:26:36 at what scale is user replication problematic? 18:26:39 dstanek: -1 on optional? 18:26:47 yes 18:26:48 dstanek: or -1 on required? 18:26:53 dstanek: :P 18:26:57 it needs to be required 18:26:59 you'd have to be creating thousands of users a second, no 18:27:00 ? 18:27:01 ok cool. 18:27:08 lbragstad: maybe? 18:27:21 replication of even a few thousand rows is not that much 18:27:24 fwiw. 18:27:32 lbragstad: new group onboarding, M&A, etc. 18:27:33 is this an actual problem that we can quantify and solve or is it theoretical right now? 18:27:37 ayoung: but therefore what you can have keystones without replicating data? what about projects etc? 18:27:42 * notmorgan steps off the soapbox. 18:27:51 * notmorgan shoves the soapbox back under the desk 18:28:07 dstanek: ++ 18:28:12 you can also configure selective replication right? doesn't have to be all tables 18:28:22 dstanek: good question 18:28:25 jamielennox, it would work like this: 18:28:28 gyee: you can. 18:28:40 henrynash_: yeah, it's possible, but I think the goal is to eventually treat federated users like any other users and not have these special edge cases (ephemeral). 18:28:49 samueldmq: 30m in (fyi) 18:28:56 do we have use cases for all these fancy replication bells and whistles? 18:29:04 notmorgan, then we don't need to do anything special on the keystone side 18:29:07 don't replicate shadow user, but replicate role assignment./ User goes from instance 1, where they have a shadow user record, to instance 2 where they do now. Shadow user record is created, and mataches id in role assignment 18:29:10 gyee: exactly 18:29:12 notmorgan: thnaks, we're changing topics now 18:29:31 topol: replication is deployer choices and out of scope of keystone imo 18:29:34 so, we don't seem to have a 100% agreement on this right now, henrynash_ please post comments on the spec and continue discussion there 18:29:38 we need to move on 18:29:41 dolphm, do you have any strong objection to the userid generation scheme I proposed? 18:29:42 ayoung: assuming you don’t have something liek service users in your default domain, which you probably do 18:29:46 ayoung: that sounds like what we all discussed happening in Tokyo 18:29:49 we're likely going to need to figure out a way to replicate users, projects, etc., so that we can share tokens between datacenters. database replication doesn't work when the code is at different levels. 18:29:52 henrynash_, will still work 18:30:09 yea, -1 on making this optional, you already replicate a lot of data to make this work, users is not a big deal and simplifies our code a lot 18:30:11 please continue in dolphm's spec 18:30:15 #topic SAML2 Middleware 18:30:19 ok, thx 18:30:21 dstanek: o/ 18:30:30 bknudson, don't share, trust :-) 18:30:36 samueldmq: o/ 18:30:46 #link https://github.com/dstanek/keystone-saml2-experiment/tree/testshib-hacking (experimental hacking) 18:30:55 this is something i've been working on and wanted to give an update and collect some input 18:30:57 notmorgan yes those bells and whistles were scaring me a little 18:31:06 i need the IdPs that a Keystone SP can use to be highly dynamic and controlled by an API 18:31:16 i want to limit Apache restarts 18:31:23 ++ 18:31:24 i want to limit the need to use Keystone's API to manipulate thirdparty configuration 18:31:33 i want centralized control over the IdP configuration (read: database not distributed XML files) 18:31:58 so...i started working on middleware to replace mod_shib/shibboleth - that is mostly backed by pysaml2 18:32:22 dstanek very cool! 18:32:23 dstanek: less dependence on apache too??? 18:32:25 dstanek, could you tackle this in mod_auth_mellon instead 18:32:30 please 18:32:38 ayoung: in? 18:32:39 dstanek, we went over this awhile back 18:32:42 the world needs this. It is not a Keystone specific requirement 18:32:46 I remember we hit a few critical issues 18:32:47 right now i can almost do a round trip using testship (will be done by the end of the day) 18:33:04 dolphm, anything that needs SSO needs to be able to mananage the Idps 18:33:05 1) how to prevent replay attack 18:33:08 i did lots of experiments with shib and mellon, but not a lot of good results 18:33:29 ayoung: i will provide for the world! 18:33:31 gyee: a replay of what against what? 18:33:37 dstanek: this is a great idea 18:33:38 dstanek, we have a team that works on that kind of stuff. 18:33:48 dolphm, submit the saml doc twice 18:33:58 gyee: that's not terrible. you just need a list of open requests. 18:34:00 like managing the relay state 18:34:11 samueldmq: not my idea :-) i'm just hacking on it 18:34:13 its a python specific approach. I have real concerns with doing crypto heavy operations in python 18:34:29 but, that is not the real issue 18:34:42 dstanek: hacking on it in keystone is your great idea ;) 18:34:42 gyee: right now my demo code does this with a process cache, but in a real impl i would have to use the database 18:34:46 gyee: so after breaking the SSL connection between the client and keystone, you want to prevent a third party from sending that same request? 18:35:05 gyee: i don't remember there being a nonce or anything in saml2? is there? 18:35:08 the real issue is that having SAML handled by a module for Apache that needs config file changes is too static 18:35:20 dolphm: yes. because clients are stupid and may log saml or whatever 18:35:20 there's a relay state management in shibd 18:35:22 gyee: there's a validity period on the assertion, but nothing that the server does specifically for replays 18:35:42 it also includes trust originator 18:35:54 like the entire request URL is part of the signature itself 18:36:02 I don't want to get Keystone into the world of general SAML support unless we are going to use SAML to replace tokens 18:36:08 is this a new library or does it live in keystone? 18:36:14 I remember we were having trouble with it when fronted by a proxy 18:36:31 probably has to live in keystone if it's got a db 18:36:43 gyee: how does shib / mellon solve that problem? 18:36:56 bknudson: i would like it to live in keystone since i think it would be generally useful 18:37:10 dolphm, it validates what's in the relay state against the caller 18:37:21 this is the wrong community for this dstanek . SAML is only one of several Federation protocols. We've had more requests for openicd lately than SAML 18:37:27 like the request URL, time stamp, etc 18:37:28 ayoung: do you mean writing code to parse and interpret saml or writing code that uses a library to do that? 18:37:49 dstanek: is there a spec up for review so we can review/use for discussions ? 18:37:54 if we implement relay state in middleware that would be awesomer 18:37:54 dstanek, even the second part 18:38:01 its not a keystone specific problem 18:38:24 ayoung: we already do this for k2k - this just brings it full circle 18:38:34 dstanek, I know. and it was a mistake then 18:38:35 is restarting apache that big of a deal? you spin up a new instance and shut down the old one. 18:38:38 k2k is crippled saml 18:38:44 ayoung: the keystone specific problem is managing one or more trusted IdP's per domain via the API as a domain admin without having to be the cloud operator, etc, right? 18:38:50 samueldmq: no, but there can be 18:38:58 bknudson: as a domain admin, i don't have the ability to do that 18:39:01 dstanek: I think that would be useful 18:39:12 also k2k is an saml idp which is an easier problem 18:39:24 samueldmq: couldn't design it before digging in 18:39:25 dolphm, even that is a general problem, we just solve it as an abstraction layer for the rest of openstack 18:39:43 jamielennox: an SP is actually pretty easy too 18:39:46 just use PKI tokens :-) 18:39:47 well...ok you mean domain as an Openstack specific object, so yes you are right 18:39:50 dstanek: got it 18:39:55 ayoung: yes, that's what i meant 18:39:56 the hardest thing is dealing with shibboleth 18:40:06 ain't kidding 18:40:11 dstanek, I am working on a POC with a different IdP right now 18:40:14 dstanek, not worried about implementing saml but regarding the whole security of such authenticator involves 18:40:19 I've worked with both Ipsilon and Keycloak 18:40:25 gyee gave few examples 18:40:38 and the mod_mellon part is being extracted by jdennis 18:40:40 #action dstanek to post a spec on SAML2 Middleware up to review 18:40:51 rodrigods, PKI token are S/MIME doc essentially 18:40:55 ayoung: extracted to where? 18:41:00 This has got to be the gayest nerd network tranny shit I have ever stumbled across I am going back to Dalnet where the pussy is. 18:41:00 S/MIME's being around since the beginning of time 18:41:02 * notmorgan doesn't have a strong opinion on this front. 18:41:03 bknudson: the other issue is that it's easy for me to crash shib and i wouldn't want domain admins to be able to do that 18:41:04 there is registration code for mod_mellon to talk to Ipsilon as part of the ipsilon proejct, and he's done a standa lone for keycloak. 18:41:06 I'll link 18:41:34 rodrigods: is it any worse than what we already do? 18:41:51 https://github.com/jdennis/keycloak-httpd-client-install 18:42:07 does anyone have an hardcore "this is stupid" or "i hate you" type of feedback? 18:42:22 otherwise i'm just going to proceed 18:42:23 nkinder, dstanek is proposing moving the SAML processing to a middleware inside of keystone 18:42:27 no, middleware is a great idea! 18:42:37 if we get it to cover the bases 18:42:39 dstanek, I have a hard "please don't do it in keystone" feedback 18:42:40 dstanek: so first thing is i don't see any management urls, but i assume they are an easy problem after getting the authenticator to work 18:42:52 the problem is to know the whole base upfront 18:42:55 I don't think you are stupid and I don't hate you, dstanek 18:42:56 i've started looking into shibboleth's dynamic federations, but it seems to be too complicated for actual use 18:42:57 dstanek: i think you should proceed, it can just be another option 18:42:57 gyee, ^ 18:43:03 just keep in mind python sucks at crypto work 18:43:11 ayoung: do you object to the domain-admin use case for managing federations, or to the implementation? 18:43:12 so ayoung and dstanek (and others) please keep in the discussion in -keystone ? 18:43:12 dstanek: i've in the past said to implement it elsewhere and i know you've looked into it so if you really have to do it in middleware i'd prefer it to live upstream in keystone 18:43:22 dolphm, implementation 18:43:22 ayoung: not moving...this is all new stuff for a SP 18:43:27 well, likely will get better if we get some JIT added and/or asyncio+uvloop 18:43:28 it should have a general agreement so dstanek can proceed with the spec/implementation 18:43:31 we gotta move 18:43:32 but just keep that in mind. 18:43:47 #topic Performance testing patches up for review 18:43:49 dstanek: but mostly i'd want to make sure this becomes something in contrib that people _can_ use, i don't think we should ever recommend it as the default 18:43:59 lbragstad: o/ 18:44:03 ayoung: i'm not clear on how an alternative implementation would satisfy the domain-admin-federation-manager use case, then 18:44:08 so performance has been coming up ... a lt 18:44:09 jamielennox: ++ 18:44:10 lot* 18:44:15 dolphm, I want a general solution to SSO. I want to have mod_mellon do exactly what dstanek is proposing anyway 18:44:22 ayoung: or at least, i'm not clear on what that alternative should look like 18:44:25 samueldmq: can you timebox this at 10 min instead of 15 [for my last minute topic] 18:44:25 i wanted to try and get something written that would allow us to test performance the same way 18:44:32 that way we can get recreateable results 18:44:38 lbragstad: ++ and i like it 18:44:42 and hopefully tie it into reviews 18:44:47 with something like leaving commit 18:44:50 comment* 18:44:50 notmorgan: sure, I will do that and leave your 5 mins at the end 18:44:51 lbragstad, nice 18:45:00 lbragstad: you have 10 mins :) 18:45:04 like "recheck" but "check performance" 18:45:11 rodrigods, bases as in managing and validating relay states 18:45:12 dolphm, so, accepting a new IdP is an operator level operation. Lets assume we can make it far more managable than shibboleth does today. jdennis 's tool is for that exact problem in keuycloak 18:45:15 lbragstad: 3rd party CI can trigger on any event 18:45:16 and is based on the ipsilon approach 18:45:30 so I start contributing what i have to one of my own repos https://github.com/lbragstad/keystone-performance 18:45:38 the two really should be merged into a single tool that knows about multiple Idps, but the amount of Idp specific logic is high right now 18:45:48 ^ that will go through and set up keystone using openstack ansible 18:45:48 dstanek: even if it goes no where official, the code is worth exploring. 18:45:54 so...that is something that can be done by ansible fairly simply 18:45:56 ayoung: we need to continue on, this convo will need to continue in -keystone or on a spec 18:46:01 notmorgan, ++ 18:46:03 ayoung: sorry. just keeping things moving. 18:46:07 dstanek: maybe it gets folded into another project, maybe it becomes a project. Still of value. 18:46:14 lbragstad: so is this 3rd party or going to be on infra resources? 18:46:18 the performance tests are extremely basic but I wanted to throw it out there so that people can play wiht it 18:46:18 notmorgan: ayoung ++ 18:46:24 lbragstad: also i think it's beneficial 18:46:27 jamielennox i might be able to get some bare metal resources 18:46:30 jamielennox: it can't be in infra unless there is hardware dedicated to it. 18:46:31 lbragstad: it’s a GREAT start 18:46:37 lbragstad: this is going to be really useful 18:46:43 lbragstad: and we really neeed this 18:46:56 jamielennox: it needs to be consistent hardware that isn't affected by bad neighbor/different provider 18:47:02 we'll likely wind up implementing something like this for our cloud, too. 18:47:02 lbragstad: thanks for the initiative, and for sharing it 18:47:06 notmorgan: ++ 18:47:12 are we talking about rally gate? 18:47:13 jamielennox: or it needs to be run at least in consistent providers on consistent flavors 18:47:15 so please play wiht it, open bugs against the plays, etc.. 18:47:18 gyee: no. 18:47:18 and hopefully we can share the tests, although every deployment is going to be different 18:47:31 gyee: rally has failed to meet what lbragstad is proposing. 18:47:35 notmorgan: mostly i was just thinking it's a lot easier problem if we just consider it a 3rd party job 18:47:36 my next step is to write something that would kick of infra per patch 18:47:42 lbragstad: thanks for sharing the work you have 18:47:47 jamielennox: i would like this ot be 3rd party tbh 18:48:11 it needs dedicated hardware 18:48:15 yep 18:48:16 it is 18:48:26 jamielennox: i am happy to ask -infra about taking it on, but right now they aren't (afaik) in aplace for supporting this 18:48:29 it's hosted on rackspace dedicated bare metal servers 18:48:29 it needs to run lots of times, to get "real world" reports 18:48:39