16:00:33 <rakhmerov> #startmeeting Mistral
16:00:34 <openstack> Meeting started Mon Oct 21 16:00:33 2013 UTC and is due to finish in 60 minutes.  The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:37 <openstack> The meeting name has been set to 'mistral'
16:01:09 <tsufiev> hello
16:01:09 <rakhmerov> so this is the first community meeting for the newly started project Mistral
16:01:17 <rakhmerov> (I believe not the last :) )
16:01:22 <tnurlygayanov_> )
16:01:30 <rakhmerov> the agenda can be found at https://wiki.openstack.org/wiki/Meetings/MistralAgenda
16:02:21 <rakhmerov> it contains pretty high level items which are basically all about clarifying what the project is and who it's for
16:02:34 <rakhmerov> #topic Project Mission
16:03:07 <rakhmerov> btw, some useful links:
16:03:31 <rakhmerov> https://launchpad.net/mistral
16:03:45 <rakhmerov> https://wiki.openstack.org/wiki/Mistral
16:04:34 <rakhmerov> and the proposal that was created in parallel but we want to be in sync with: https://wiki.openstack.org/wiki/Convection
16:06:20 <rakhmerov> so we thought the project mission is to create a simple service for OpenStack based on using DSL for processing interrelated tasks
16:06:32 <rakhmerov> the main ideas are in this document: https://etherpad.openstack.org/p/TaskServiceDesign
16:06:57 <ilyashakhat> just curious, what is relation bw mistral and taskflow?
16:07:13 <devvesa> yep, i wanted to ask this question too
16:07:16 <rakhmerov> it's one of the items to discuss :)
16:07:35 <rakhmerov> let's get back to this a bit later, there's a reason for this
16:07:44 <rakhmerov> if you don't mind
16:07:51 <tnurlygayanov> lets push some links to mistral description )
16:08:04 <rakhmerov> I just did
16:08:09 <rakhmerov> the last one
16:08:54 <rakhmerov> so the main thing here is that the design is a little bit low level and is not so easy to explain for most of the people around
16:09:18 <stanlagun> I currently work on a more high-level design document
16:09:33 <rakhmerov> basically, the service is supposed to be just a generic state machine not performing any actions itself
16:10:18 <rakhmerov> stanlagun: right, we decided to work through the definition of it
16:10:39 <stanlagun> The motivation behind document is to clarify use cases, differences and relations to other task-related projects like TaskFlow
16:12:08 <rakhmerov> the questions that have raised: do we develop this service for developers exclusively? If yes, the level of abstraction would be different than if we offered some capabilities for people who drive them own VMs let's say
16:12:57 <rakhmerov> so now we see the opportunity to make it clear and understandable for a wide variety of users
16:12:58 <stanlagun> I'd say the service is not only for developers
16:13:05 <rakhmerov> not only for developers, yes
16:13:36 <stanlagun> but not necessary for novice users
16:14:25 <rakhmerov> and that leads us to the idea of introducing different levels: the low level as described in the design document (with some changes), and the set of tools and frameworks on top of this generic engine
16:14:34 <stanlagun> I suppose it is for a. developers b. system integrators and c. for administartors via additional toolset
16:14:49 <rakhmerov> stanlagun: agree
16:15:35 <stanlagun> Initial idea was to have a very low-level task management service
16:15:50 <rakhmerov> so the main value is to offload all the state management related with task processing onto this service that would provide all the cool features like HA, scalability and so forth
16:16:26 <stanlagun> But as we further discuss it it becomes clear that we need it to be human-friendly so that the tasks could be hand-written and not generated by some software
16:16:28 <rakhmerov> guys, any comments/questions on that?
16:17:13 <ameade_> can we go through a complete use case?
16:17:40 <rakhmerov> yes, even though we don't assume actions to be domain specific we can still introduce a set of actions (like run a script on a VM or hit a REST API) out of the box
16:17:50 <rakhmerov> ameade_: yes
16:18:44 <rakhmerov> #topic Use Cases
16:18:48 <stanlagun> There are number of very distinc use cases that can be addressed by mistral
16:19:51 <rakhmerov> stanlagun: go ahead please and then I'll add some more
16:20:38 <gokrokve> I see one of the simplest use cases. Mistral supports scheduling so the use case is the following. A user whants to schedule a simple script execution on multiple VMs. The user creates a simple task definition with schedule.
16:21:04 <rakhmerov> yes
16:21:31 <ameade_> so like a distributed cron? what executes the actual script?
16:21:38 <gokrokve> Another use case is live-migration. Mistral supports events, so the use case is the following. A user specifies commands for VM live migration in case of event from Ceilometer (CPU consumption 100%)
16:21:45 <stanlagun> Use case #1: I have some service that is doing some long-running business process on user request. I want to make it fault-tolerant so that if my app dies in the middle I want it to continue on another node from the point it died
16:23:07 <stanlagun> In this use case I split my buisness process into set of tasks and let Mistral handle them so that Mistral call me back with "Execute action X, here is the data". If my app dies then request just goes to another instance
16:25:23 <stanlagun> Use case #2: System integration. If activities of my service are described via published tasks then another app can hook to the task execution and trigger its own processing. Users can insert their own tasks into task graph
16:25:54 <stanlagun> This also gives new possibilities for monitorings and state management
16:26:25 <rakhmerov> I think I have one more use case: let's say I need to build an complex application for processing salary calculation. I have several computation steps that take place in different places/platforms. I calculate base salary in system #1, I calculate bonuses in system #2 and there may be even several steps like "A manager approves bonuses manually" which is considered a computation step too
16:26:52 <stanlagun> yep
16:27:07 <rakhmerov> so we can define a series of related tasks where execution of one task may be triggered by successful completion of another task
16:27:34 <rakhmerov> and at some point this flow ends once the definition of "salary is calculated" is met
16:28:48 <rakhmerov> so I may already have all the code that does all the work but I need to execute them in a distributed environment taking advantage of the cloud on-demand scaling feature
16:29:04 <rakhmerov> hence I need fault tolerance, HA and autoscaling
16:29:13 <rakhmerov> that's where Mistral comes into play
16:29:55 <rakhmerov> it serves a coordinator for these computational steps basically orchestrating (forgive me this overloaded term) what needs to be done when
16:30:12 <stanlagun> Use case #3: scheduling. Administrator can schedule some business-process to run on time (periodical events etc.) or on external alarm (Ceilometer etc). This can be done even if the program author was not aware of Ceilometer. This is an example of inversion of control because you can replace nagios with ceilometer without end app beeing even aware
16:30:34 <ameade_> is the actual execution of these tasks outside of Mistrals responsibility?
16:30:38 <rakhmerov> from this perspective it can be useful for people like data analysts and so on
16:31:01 <rakhmerov> ameade_: originally no, it's not
16:31:42 <stanlagun> Mistral cannot do salary calculations on its own. You do not upload any code
16:31:42 <rakhmerov> but as said before we can build a toolset on top of it to provide additional features to widen the scope of its applicability
16:32:18 <stanlagun> But it can send some sort of signal, call external REST API or just some URI, send RabbitMQ message to your app etc.
16:32:26 <rakhmerov> right, it's just supposed to manage the process via sending events
16:32:26 <gokrokve> What is the machanism of task execution?
16:32:39 <rakhmerov> right :)
16:33:06 <stanlagun> gokrokve: what do you meen by mechanism?
16:33:07 <rakhmerov> #topic Relationships with existing projects,  i.e. TaskFlow
16:33:28 <gokrokve> How it will call tasks?
16:33:36 <rakhmerov> We initially discussed that with harlowja
16:34:02 <rakhmerov> we appreciate his positive feedback on this initiative. He's actively working on TaskFlow
16:34:05 <gokrokve> As I see you propose call tasks through MQ, REST API. How scripts can be executed.
16:34:24 <rakhmerov> ooh
16:34:29 <stanlagun> gokrokve: it is not executing real tasks. It just signals other app to do so. But because this signaling can be a complex REST API call this can be considered an action of its own
16:35:24 <gokrokve> Can it be integrated with existing agents like Heat cfn-rool or triple-o os-config?
16:35:40 <gokrokve> They can execute some scripts on VMs.
16:35:42 <stanlagun> Scripts can be executed by apps that are out of scope of Mistral. For example you can use Mistral to send execution plan to Murano Agent or make it invoke Puppet manifest etc
16:35:44 <rakhmerov> however, we can provide the capability to allocate required number of workers with something like "VM agent" to do a specific work
16:35:45 <devvesa> /whois gokrokve
16:35:55 <devvesa> sorry
16:36:29 <rakhmerov> gokrokve: it can be a plugin mechanism to be able to integrate with existing agents
16:36:45 <rakhmerov> but it's all on top of generic service
16:36:48 <ameade_> +1 that makes sense
16:36:54 <stanlagun> The plan is to have built-in support for several popular deployment tools like chef, puppet, Murano etc. that can be triggered without additional workers
16:37:03 <rakhmerov> that doesn't know anything about any specific things
16:37:23 <rakhmerov> yes, a huge number of options here
16:37:33 <ameade_> a simple case could even be to have an SSH plugin...but I digress
16:37:59 <gokrokve> Yep. SSH might be a good option.
16:38:07 <rakhmerov> well, yes, it can be as long as we're ok with security kind of things..
16:38:48 <rakhmerov> so as an underlying technology we were considering (and still) TaskFlow mentioned above
16:38:53 <gokrokve> It usually depends on deployment. For some cases security is not a concern, so ssh might be useful.
16:39:19 <rakhmerov> that can be used in implementation
16:39:30 <ameade_> are there tools out there that already do what Mistral is trying to? And if so, how do they not fit our use cases?
16:39:31 <gokrokve> How do you plan to map DSL language in Mistral and TaskFlow code objects\approach?
16:39:35 <stanlagun> There are basicaly 3 requirement for Mistral actions that can be executed by Mistral itself. 1) They must be vere generic and very common 2) They must be safe to execute in shared environment under local root acc and 3) They must not block
16:40:06 <stanlagun> About Mistral-TaskFlow relations and integrations:
16:40:28 <rakhmerov> TaskFlow can also provide integration with Mistral making its local calls remote ones transparently for users thereby providing HA
16:41:23 <rakhmerov> #action stanlagun to describe the possible ways of executing particular actions on VMs in the tech specification
16:41:31 <stanlagun> TaskFlow and Mistral are very different and target different use cases. TaskFlow can be used for Mistral implementation (and may be not used either). TaskFlow is also probably would become one of Mistral clients so that you could schedule TaskFlow tasks over Mistral
16:41:31 <gokrokve> As for other tools like that. I think there are some. But they are not in OpenStack. So the idea is to have something integrated with openstack to use for example its security model via keystone.
16:42:49 <stanlagun> Mistral also going to provide declarative data-flow. That is something you wouldn't find anywhere
16:42:57 <gokrokve> As for task management. We saw some implementation of task management in nova conductor and glance.
16:42:57 <rakhmerov> yes
16:43:25 <gokrokve> But both nova and glance do this for their specific goals.
16:43:34 <rakhmerov> #topic DSL and the steps for clarifying it
16:44:27 <rakhmerov> as mentioned before we're going to use DSL for defining tasks, their dependencies and actions they're associated with
16:44:39 <gokrokve> Renat, please collect all use cases and publish them on wiki.
16:44:51 <rakhmerov> this DSL is supposed to be a generic one
16:45:08 <rakhmerov> #action Renat to collect use cases and publish them on wiki.
16:45:26 <rakhmerov> So regarding DSL
16:45:57 <rakhmerov> we're planning to modify in a number ways the existing technical specification we have and start discussing it with the community
16:46:13 <rakhmerov> stanlagun is working on that now
16:46:17 <rakhmerov> as well as I am
16:46:48 <stanlagun> rakhmerov: not exactly on that
16:46:53 <rakhmerov> #action Renat and Stan finish the modified specification and share it with the community
16:47:12 <rakhmerov> well, yes, it's one of the subitems
16:47:43 <rakhmerov> anyway, we'll merge all the ideas and efforts
16:48:22 <stanlagun> I want to write Q&A document that covers use cases, relations with other frameworks, existing services, development plans etc. so that you absolutely understand what Mistral is after reading it. This is not REST API or DSL spec
16:49:20 <rakhmerov> ok, that's good
16:50:00 <rakhmerov> then according to this document we'll need to clarify our vision of DSL and API
16:50:51 <rakhmerov> #action Stan (with Renat's help) to finish Q&A document that covers use cases, relations with other frameworks, existing services, development plans etc
16:51:01 <gokrokve> We need to have a DSL review with TaskFlow team
16:51:40 <rakhmerov> #action Renat, Stan, Timur to synchronize DSL and REST API with the latest conceptual changes
16:51:48 <rakhmerov> gokrokve: yes, sure
16:52:16 <stanlagun> DSL is probably gonna be YAML based and use some YAQL inside
16:52:17 <stanlagun> https://pypi.python.org/pypi/yaql
16:52:23 <rakhmerov> I have an action item to conduct a meeting with them, I'm going to do it soon
16:54:59 <rakhmerov> so just to clarify it for people who just got involved: Mistral is pretty orthogonal with TaskFlow. They are going to complement each other
16:55:08 <rakhmerov> I think we pretty much agreed on that
16:55:22 <rakhmerov> 5 mins left guys
16:55:27 <stanlagun> btw did we answered the question of TaskFlow-Mistral relations?
16:55:28 <rakhmerov> any questions?
16:56:10 <rakhmerov> I thought we did, but we can clarify something else if that's still not clear
16:57:12 <rakhmerov> of the agenda items we had we didn't discuss the initial design (with DSL and REST API)
16:57:32 <rakhmerov> but not sure if it's worth doing this now
16:57:50 <rakhmerov> instead we would answer some questions on it
16:58:29 <rakhmerov> are we done then?
16:58:58 <rakhmerov> ok, thanks to everyone
16:59:00 <stanlagun> We hope to hear more feedback from you guys as this is critical at the moment
16:59:06 <stanlagun> Nothing is fixed yet
16:59:09 <devvesa> clear for me the difference between mistral and taskflow
16:59:11 <rakhmerov> yes
16:59:12 <devvesa> thanks
16:59:13 <ameade_> thanks!
16:59:33 <rakhmerov> devvesa, let's go to #openstack-mistral
16:59:38 <rakhmerov> #endmeeting