17:00:36 <thinrichs> #startmeeting CongressTeamMeeting
17:00:36 <openstack> Meeting started Tue Mar 11 17:00:36 2014 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:39 <openstack> The meeting name has been set to 'congressteammeeting'
17:01:22 <thinrichs> Agenda for the day:
17:01:42 <thinrichs> 1) progress toward an alpha release
17:01:58 <thinrichs> 2) pointer to other policy projects.
17:02:03 <thinrichs> Let's start with 2 since it's quick.
17:02:14 * sarob_ lurking
17:02:33 <thinrichs> Last week gokul and others on the ML were asking about all the policy projects in OS.
17:02:58 <thinrichs> They put together a list of them with comparisons.  Probably incomplete.  But I took some time to go through them and put links to them on the Congress wiki.
17:03:08 <pballand> sarob_: you're in good company :)
17:03:18 <sarob_> ;)
17:03:46 <thinrichs> Here's a link to the wiki.  Let's make an effort to keep it up to date when we hear of new projects, etc.
17:03:47 <thinrichs> https://wiki.openstack.org/wiki/Congress
17:03:47 <pballand> the links are helpful, thank you
17:04:05 <thinrichs> Info is at the end of the page.
17:04:35 <thinrichs> Any comments/questions on that?
17:05:32 <pballand> it seems like these projects may play well, in the future
17:06:07 <pballand> current strategy appears complimentary (which is great)
17:06:39 <thinrichs> Yeah--I'm hoping that down the line we'll be able to carve off pieces of the Congress policy and send them down to these other efforts for enforcement.
17:07:04 <thinrichs> Moving to next agenda item now.
17:07:11 <thinrichs> #topic Progress toward alpha release
17:07:12 <rajdeep> hi
17:07:19 <thinrichs> rajdeep: hi!
17:07:22 <pballand> hi rajdeep - just in time :)
17:07:27 <rajdeep> yes,
17:07:55 <thinrichs> We had just quickly gone over the list of other policy efforts within OS that we've listed on the Congress wiki page.
17:08:00 <thinrichs> We'll be trying to keep that up to date.
17:08:05 <rajdeep> ok
17:08:20 <thinrichs> Now we've moved on to the main part of the agenda: progress toward an alpha release.
17:08:51 <thinrichs> We'll just go down a list of todos and discuss progress.
17:08:54 <rajdeep> great - what are the time lines
17:09:30 <thinrichs> Not sure what the right timeline is.  I was hoping to have something ready at the end of the month, but it's not clear if that's plausible.
17:09:40 <thinrichs> Let's revisit once we talk through the todos.
17:10:04 <thinrichs> First item is Data Integration, which you are working on rajdeep.
17:10:06 <rajdeep> where is the agenda posted?
17:10:27 <rajdeep> currently i am working on finishing the neutron integration
17:10:50 <rajdeep> should we have some kind of data model for each type of service we are integrating
17:11:04 <pballand> I failed to post it - Tim has a list of todos to talk about
17:11:11 <rajdeep> and populate those structures from the native objects
17:11:22 <rajdeep> i saw your comments on neutron
17:11:36 <pballand> the data model is just tuples
17:11:52 <pballand> tuples are lowest-common format for all data sources
17:12:05 <pballand> that allows the policy language to be common across all sources
17:12:22 <rajdeep> ok, so i need to convert networks, ports, subnets etc into tuples
17:12:30 <pballand> with that said, we should have a well defined (and commented) set of tuples for each data source
17:12:44 <thinrichs> rajdeep: yes
17:12:52 <gokul_> hi all:  this is gokul
17:12:56 <thinrichs> I put an example in the comments.  Did that make sense?
17:13:05 <thinrichs> gokul: hi!
17:13:09 <rajdeep> so i just convert dictionary into tuple and return
17:13:12 <rajdeep> yes go it
17:13:26 <gokul_> just beginning with congress...  hope to contribute :)
17:13:41 <pballand> gokul_: great - there is a lot to do :)
17:13:50 <thinrichs> rajdeep: not just a Python dict to Python tuple.  The tuples must only have strings/numbers as elements.  No embedded data structures.
17:14:03 <rajdeep> ok, got it
17:14:04 <kudva_> Hi, I am prabhakar, work with Gokul, joining as well
17:14:20 <thinrichs> kudva: welcome!
17:14:23 <pballand> kudva_: welcome
17:14:49 <rajdeep> so my plan is to cover neutron, nova, keystone and cinder
17:15:02 <thinrichs> We're in the midst of going thru the list of todos to get an alpha release ready.
17:15:30 <thinrichs> kudva, gokul: Why don't we keep going and then if anything sounds interesting to work on, let us know.
17:16:12 <rajdeep> any other component for openstack i missed?
17:16:17 <gokul_> yup -- sounds good.  just reading the conversation for now and "understanding".
17:16:24 <kudva_> okay, I have some questions as well, will save for another discussion. Let's go through the list
17:16:27 <thinrichs> rajdeep: I'd say that neutron and nova are musts and keystone and cinder are nice-to-haves
17:16:43 <rajdeep> good,
17:16:52 <rajdeep> should not be difficult
17:17:05 <rajdeep> i found some problems getting groups out of keystone
17:17:14 <rajdeep> apparently not many people use it as of now
17:17:31 <rajdeep> and all our examples tie users to groups
17:17:38 <rajdeep> which is more of AD/LDAP use case
17:17:44 <thinrichs> rajdeep: We probably want to at least understand how the tables/tuples that you're using to represent Neutron breaks/does-not-break the Neutron action descriptions that we have.
17:18:18 <rajdeep> ok-- so how do we plan the integration one data is in place
17:18:19 <thinrichs> rajdeep: I'd be happy to use AD/LDAP for groups -- to illustrate that Congress works with non-OS components.
17:18:54 <pballand> rajdeep: can you elaborate?
17:19:19 <rajdeep> do we need another component which ties the parser/runtime with the data_driver
17:20:14 <rajdeep> or will this be an example script outside the core code
17:20:18 <thinrichs> rajdeep: not sure if it's another component but yes we need a framework that glues the policy-engine together with the data drivers you are writing.
17:20:31 <thinrichs> rajdeep: I think we want it integrated.
17:20:33 <rajdeep> exactly
17:21:02 <thinrichs> Congress will need to be responsible for integrating the data necessary to make policy decisions.
17:21:05 <rajdeep> and we would need a place to do integration testing -- openstack + AD + congress
17:21:39 <thinrichs> rajdeep: that's another important issue and it's on the list of alpha todos
17:22:18 <rajdeep> ok
17:22:30 <rajdeep> let us move to the next item on the agenda
17:22:41 <thinrichs> Can someone look into the data integration "glue"?
17:22:45 <pballand> I would be happy to take that
17:22:54 <thinrichs> Great.
17:23:06 <thinrichs> #action pballand will look into the data integration framework.
17:23:46 <thinrichs> The next todo is the API infrastructure that connects the policy engine to the outside world.
17:24:01 <thinrichs> pballand: you worked on this--could you give us a status update?
17:24:15 <pballand> yeah, the code that is posted was pretty rushed
17:24:38 <pballand> basic idea was to use a simple API co-routine loop like other projects
17:24:59 <pballand> I'm not worried about a fancy framework right now, but getting the API abstractions right is important
17:25:13 <thinrichs> pballand: agreed
17:25:29 <pballand> still need to expose _basically everything_
17:25:57 <thinrichs> Could you estimate amount of time/work to hook everything up?
17:26:05 <pballand> I imagine a pretty good interface could be constructed with a few solid weeks of work
17:26:37 <thinrichs> pballand: do you have cycles to devote to that?
17:26:51 <pballand> I can definitely take a second pass at making the code more product-ready
17:27:16 <rajdeep> we would also need to tie the REST endpoints to the runtime engine
17:27:25 <rajdeep> or we want to keep them separate
17:27:27 <rajdeep> ?
17:27:49 <pballand> I think its best to keep a loose coupling between to two
17:27:55 <pballand> ... but in the same binary
17:28:10 <pballand> do you have thoughts?
17:28:27 <rajdeep> well it was hard for me to understand it initially
17:28:39 <rajdeep> how the two are linked -- currently only through tables
17:29:02 <rajdeep> perhaps we could put it in a readme
17:29:10 <pballand> (in my experience, APIs that evolve along with the underlying implementation become cumbersome over time)
17:29:27 <pballand> I'll try to make it clear as I develop the next iteration
17:29:36 <rajdeep> ok, great
17:29:45 <rajdeep> any plans to add persistence layer to tables
17:29:52 <rajdeep> ?
17:30:30 <thinrichs> rajdeep: persistence for policies is important.
17:30:35 <pballand> (to be fair, the current implementation is "barely-coupled", so it isn't surprising that it is hard to follow)
17:30:45 <thinrichs> persistence for tables is not.
17:31:03 <thinrichs> because once we turn off Congress, it'll need to resync with all the data sources anyway.
17:31:21 <rajdeep> ok..i meant policies
17:31:24 <pballand> rajdeep: congress is the source of truth for the policy, but is just a cache of other data (in the tables)
17:31:35 <thinrichs> pballand: agreed
17:31:45 <thinrichs> rajdeep: yes--we should do that.
17:32:07 <thinrichs> I was thinking that we'd do it via logging at first.
17:32:29 <thinrichs> So every time some inserts/deletes a policy statement, we'd log it in a way that we can replay the logs later to reconstruct the state of Congress.
17:32:43 <thinrichs> We'll need logging of some form for debugging/auditing anyway;.
17:32:50 <thinrichs> Anyone want to look into this?
17:33:08 <rajdeep> i can take it up after my data integration is over
17:33:20 <thinrichs> I don't think this is a show-stopper for an alpha release.
17:33:24 <thinrichs> rajdeep: great.
17:33:37 <rajdeep> we will use regular python logging?
17:33:41 <thinrichs> #action rajdeep will look into persisting policy after his data integration work.
17:33:48 <thinrichs> rajdeep: haven't thought that far ahead.
17:34:31 <thinrichs> Unless there are objection, we'll move onto a couple of quick alpha todos.
17:34:44 <rajdeep> sure go ahead
17:34:44 <pballand> please fire away :)
17:35:09 <thinrichs> There were some basic issues with the policy engine not doing really ANY syntax checking.
17:35:24 <thinrichs> I pushed those last week sometime.  So that todo is finished well enough for now.
17:35:49 <thinrichs> Another small todo was to startup some user docs so that people have a guide when using Congress for the first time.
17:35:56 <thinrichs> I just pushed for review the first cut at that.
17:36:01 <thinrichs> https://review.openstack.org/#/c/79383/
17:36:48 <thinrichs> I'll give you a minute to peruse.  Let me know when you're ready to move on.
17:37:15 <pballand> I'll try to review by tomorrow
17:37:17 <thinrichs> It's probably hard to look at in raw RST, but it gets compiled into HTML, PDF, etc.
17:37:30 <thinrichs> It's the standard thing to use in OS (I believe).
17:37:32 <rajdeep> wow thats lot of docs
17:37:50 <thinrichs> Lots of files, perhaps, but only a few pages of content.
17:37:53 <rajdeep> perhaps we could look at docbook format as well
17:38:21 <thinrichs> rajdeep: I have no idea what the list of supported formats are.
17:38:26 <sarob_> docs stores docbook in xml
17:38:31 <pballand> from the opestack wiki: "We use DocBook to create OpenStack's web documentation for deployers, admins, and CLI and API users. We use ReStructured Text (RST) files stored in source code for developer docs."
17:38:37 <sarob_> projects typically use RST
17:38:59 <sarob_> pballand: you got it
17:39:00 <thinrichs> sarob: that's what we're using.  Phew!
17:39:05 <rajdeep> so we are building developer docs "only"?
17:39:07 <pballand> :)
17:39:24 <pballand> it's a great place to start :)
17:39:37 <rajdeep> yes let us focus on getting alpha out
17:39:40 <sarob_> ive been dabbling with setting standards for RST
17:39:50 <pballand> I think there is confusion on whether these are dev or user docs, but our first users are developers :)
17:39:54 <sarob_> so we can pull the content into XML on build
17:40:04 <sarob_> pballand: right
17:40:15 <sarob_> pballand: status quo, better than new ground on docs
17:40:28 <sarob_> pballand: code perhaps first
17:40:30 <pballand> as we get closer to alpha, we can leverage the RST to create more formal user docs
17:40:42 <pballand> sarob_: agreed
17:40:59 <sarob_> pballand: as long as you are consistent in formal, no problems
17:41:06 <sarob_> pballand: arrg, format
17:42:04 <thinrichs> Moving on: we wanted to do some basic scaling tests.
17:42:24 <thinrichs> That was on my plate and I haven't gotten around to it.
17:42:47 <pballand> I was hoping to get an example dataset from a user with a big deployment, but they ran in to so hurdles
17:42:54 <pballand> hopefully more to come on this...
17:43:05 <pballand> s/so/some/
17:43:35 <thinrichs> Last item on my list for the alpha release.  (Probably others I'm missing.)
17:43:54 <rajdeep> thinrichs : what would be typical number of policy insertions in a real life use case?
17:44:25 <pballand> rajdeep: it seems to very *greatly* based on the deployment
17:44:41 <pballand> insertions are relatively infrequent
17:45:02 <rajdeep> ok and total policy elements
17:45:05 <pballand> data changes (group membership, for instance) are common, however, so incremental computation is imporatnt
17:45:30 <pballand> rajdeep: it's hard to say
17:45:40 <rajdeep> another question is how will congress scale
17:45:57 <rajdeep> will it be a multi node deployment?
17:47:03 <pballand> scale is important, but we need the operational model first
17:47:18 <thinrichs> With that, we need to move on to the last todo.
17:47:20 <pballand> I'd rather not scale a shitty model :)
17:47:29 <thinrichs> Before running out of time.
17:47:49 <thinrichs> The last issue is one of what *kinds* of policies we are having the user write.
17:47:56 <thinrichs> There are (at least) 3.
17:48:22 <thinrichs> 1. Classification + Action (what we're currently promoting).  The idea is to describe which *states* of the cloud are permitted by policy.
17:48:59 <thinrichs> 2. Access-Control (e.g. XACML, AD).  The idea is to describe what *actions* are *permitted* for whom.
17:49:24 <thinrichs> 3. Condition action: The idea is to describe what *actions* Congress should *execute* under what conditions.
17:49:37 <thinrichs> Do those 3 types of policies make sense?
17:49:58 <kudva_> In 1. do you mean *states* of a cloud as in compliance or reliability (failure), or a more generic state?
17:50:38 <thinrichs> kudva: by *state* I mean the collection of all the data Congress knows about (i.e. the collection of all the cloud services that we've hooked up to Congress)
17:50:53 <thinrichs> So, generic, I think.
17:51:50 <thinrichs> The reason I bring this up is that I believe there are reasons that people might want/need all 3 kinds of policy simultaneously.
17:52:02 <thinrichs> And perhaps we want to think of Congress as a policy engine that enables people to choose what they want and to mix and match.
17:52:08 <pballand> I think it's important that congress doesn't interpret things like failure - it simply computes the policy provided
17:52:24 <rajdeep> how are these related to each other
17:52:34 <rajdeep> 1 will lead to 3
17:52:51 <rajdeep> 2 will feed into 1?
17:53:00 <thinrichs> rajdeep: From 1 we can compute a bunch of possible 3s.
17:53:35 <thinrichs> E.g. if we want all networks connected to a specific kind of VM, and that policy is violated, then there may be many ways to correct the violations.
17:53:47 <thinrichs> We could disconnect an offending network, delete the VM, etc.
17:53:55 <kudva_> So condition would represent any python function (which uses os clients and monitored data, alarms etc)?
17:54:02 <thinrichs> Each of those choices could lead to a different policy 3.
17:54:57 <pballand> kudva: not python function; rather policy-directed combination of primitives from openstack components
17:55:27 <thinrichs> Policy type 2 is different than 1 b/c we might want to allow certain users to perform actions that cause certain types of violations.
17:55:41 <thinrichs> where by 'violations' I mean 'violations' according to policy type 1.
17:55:56 <pballand> (there is a function providing the data to congress, but the logic is in the way the policy language combines the tuples)
17:56:03 <thinrichs> So types 2 and 1 are different as well.
17:56:26 <pballand> kudva: does that make sense?
17:56:28 <gokul_> however (I brought up a similar qn on the ML), would it be correct to understand that at this point, based on how conditions evaluate to (pretty much Boolean), actions will be taken.    in some sense, something like an 'optimization policy' would not fit into this framework...   would that be a reasonable statement?
17:56:33 <thinrichs> I just noticed we're running short on time.
17:57:05 <thinrichs> gokul: yes--I think that's a reasonable characterization of the class of policies that Congress can *enforce* itself.
17:57:32 <thinrichs> I'm not sure at this point whether we could express a policy requiring optimization for enforcement within Congress.
17:57:52 <thinrichs> If we could express such a policy, we could potentially carve it out and send it to another component for enforcement.
17:57:58 <gokul_> thinrichs:   thanks for the clarification.  thats fine.  i just wanted to understand the scope.
17:58:06 <thinrichs> Hand-waving furiously.
17:58:29 <thinrichs> Let's wrap up since we're out of time.
17:58:42 <thinrichs> This has been a great discussion.  Let's continue on the ML.
17:59:08 <kudva_> Yes, I would like  to understand the relationship between policy primitives and python functions
17:59:15 <pballand> thanks everyone
17:59:18 <gokul_> great.  thanks!
17:59:22 <kudva_> thanks
17:59:25 <rajdeep> thanks!
17:59:32 <thinrichs> Thanks all!
17:59:39 <thinrichs> #endmeeting