16:00:40 <rakhmerov> #startmeeting Mistral
16:00:41 <openstack> Meeting started Mon Dec 16 16:00:40 2013 UTC and is due to finish in 60 minutes.  The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:44 <openstack> The meeting name has been set to 'mistral'
16:00:57 <igormarnat> Hi, guys!
16:01:09 <stanlagun> hi!
16:01:23 <rakhmerov> let's start
16:02:08 <akuznets_> hi
16:02:11 <rakhmerov> #topic Review AIs
16:03:00 <rakhmerov> btw, here's the link to agenda: https://wiki.openstack.org/wiki/Meetings/MistralAgenda
16:03:31 <rakhmerov> the first one:
16:03:34 <rakhmerov> rakhmerov, submit blueprints for TaskFlow about lacking functionality (i.e. asynchronous tasks)
16:04:23 <rakhmerov> for now I submitted 1 BP for TaskFlow related to asynchronous tasks processing: https://blueprints.launchpad.net/taskflow/+spec/async-tasks
16:04:50 <rakhmerov> this is the most important thing for us that TaskFlow folks are working on now
16:04:53 <NikolayM> hi!
16:05:01 <rakhmerov> hi )
16:05:59 <rakhmerov> like we discussed before we decided to implement Mistral engine without TaskFlow for now and work with TaskFlow guys on additional requirements
16:06:05 <igormarnat> Did you have a chance to discuss the BP with TaskFlow team? Any feedback?
16:06:51 <rakhmerov> well, we discussed in details the whole problem with Joshua and Ivan Melnikov and based on that I created a BP
16:07:04 <rakhmerov> and we're on the same page with them here
16:07:53 <rakhmerov> so yes, Ivan was going to implement some prototype soon and we need to check the status of that
16:07:58 <igormarnat> Ok, cool
16:08:15 <rakhmerov> #action rakhmerov, check the status of async tasks in TaskFlow with Ivan
16:09:03 <rakhmerov> we also hav an AI: Evaluate an interface of salt minion
16:09:21 <rakhmerov> we didn't get to it last week
16:10:01 <rakhmerov> we need to pas it on to this week, I think it will be possible by the end of the week
16:10:23 <rakhmerov> #action rakhmerov, Evaluate an interface of salt minion
16:10:56 <rakhmerov> the same thing about the next one: Work on joel_c requirement about communication with VM agent
16:12:03 <rakhmerov> we need to get in touch with him to discuss the details
16:13:05 <rakhmerov> gokrokve, can we set up a meeting with joel_c about that?
16:13:10 <gokrokve> There is a meeting with Joel at 9
16:13:20 <gokrokve> Yes. I will ask him about it
16:13:20 <rakhmerov> ooh
16:13:28 <gokrokve> You can also join today
16:13:46 <rakhmerov> I would prefer to have a separate one
16:13:59 <rakhmerov> and know about it in advance :)
16:14:35 <rakhmerov> like we discussed before it can be our next step in development
16:14:56 <rakhmerov> so we need to find out what he needs exactly to take it into account
16:16:55 <rakhmerov> #action gokrokve, set up a meeting with joel_c to find out about his requirements
16:17:09 <rakhmerov> let's move to discussing PoC status
16:17:11 <gokrokve> ok
16:17:20 <rakhmerov> #topic PoC status
16:18:11 <rakhmerov> at this point we implemented approximately 90% of what we planned
16:19:26 <rakhmerov> including Rest API, python client for Rest API, Scheduler (to run workflows according to cron patterns), authentication and lots of other infrastructural things
16:19:46 <rakhmerov> the main thing that is left is the engine itself
16:21:22 <rakhmerov> last Wed we started working on the 3rd version of it since, as we think, we finally came up with really simple idea of how it can be built with regard to further HA and Scalability requirements
16:22:49 <rakhmerov> so by now we've written all the required code for it and it's just a matter of waiting for a couple of commits (dependencies) to be merged to start testing it
16:23:08 <rakhmerov> I'm planning to start testing it all together tomorrow
16:23:43 <rakhmerov> so by the end of this week we're planning to demonstrate the results of what we did
16:23:58 <rakhmerov> and we agreed to release it with tag "poc"
16:24:56 <rakhmerov> so generally, the work on engine turned to be the most tricky thing but hopefully we found a good solution
16:25:21 <igormarnat> It expected to be the most tricky thing, isn't it?
16:25:31 <rakhmerov> yeah, right
16:25:48 <rakhmerov> from the very beginning we didn't have a clear vision of how it should be implemented
16:26:07 <rakhmerov> and I think it's a good achivement that we now have it (at least we think so)
16:26:36 <rakhmerov> even though we're a little bit behind the schedule
16:26:44 <rakhmerov> ok
16:27:01 <rakhmerov> one special thing that I was going to discuss again is Demo App for PoC
16:27:17 <rakhmerov> #topic Discuss Demo App for PoC
16:29:22 <rakhmerov> NikolayM started working on it and looks like we have to implement it in a different way if we want to get it done quickly, not like we planned initially (create VMs, volumes etc.)
16:29:58 <igormarnat> BTW, I didn't manage to find any description on wiki or in ether pads. Do we have any?
16:30:09 <NikolayM416> agree with Renat
16:30:25 <rakhmerov> igormarnat, let me doublecheck that
16:30:39 <rakhmerov> the problem is that in order to implement a simple scenario like "Create VMs", "Attach volumes" etc. we need something like a Data Flow already
16:31:25 <rakhmerov> because say if we create a VM as the first step we need a way to transfer a VM id on to the next task
16:31:47 <akuznetsov> can we simplify the scenario e.g. only backup of VMs
16:31:57 <rakhmerov> this is not yet clear from DSL perspective
16:31:57 <tnurlygayanov> yes
16:32:19 <igormarnat> Or we can create the very simple data flow, something really primitive, can't we?
16:32:30 <rakhmerov> akuznetsov, yes I think we can but let's ask NikolayM416
16:33:03 <rakhmerov> igormarnat, I think given our current activities it'll take time, I think ~1 week
16:33:21 <rakhmerov> if we can afford this then yes, we can follow that way
16:33:47 <rakhmerov> so let me list our options first and then we can decide which way to proceed
16:34:09 <igormarnat> Yep, I just tried to add something to the list of options, not to select one:)
16:34:13 <rakhmerov> 1. Create a simple workflow with OpenStack services that do not require Data Flow
16:34:45 <rakhmerov> 2. Create a simple implementation of Data Flow (~1 week estimation)
16:35:47 <rakhmerov> 3. Create a simple Demo App that exposes 1-2 endpoints, uploads a workflow with webhooks only leading back to those endpoints of the app
16:36:14 <rakhmerov> so option 3 does not assume having OpenStack installed
16:37:12 <rakhmerov> and it generally it can just work locally, when an endpoint gets a request to execute an action from Mistral we could just log it in a beautiful way and demonstrate how tasks are executed
16:37:24 <rakhmerov> in what order and with what parameters
16:37:44 <rakhmerov> so what are your thoughts guys?
16:37:55 <igormarnat> BTW, what does Stan say about this?
16:38:13 <rakhmerov> I think NikolayM has already started working on option 3
16:38:18 <rakhmerov> today
16:38:27 <rakhmerov> but we can change that direction right now
16:39:22 <rakhmerov> NikolayM416, what do you think about option with just making backups?
16:39:36 <rakhmerov> do you think it has some pitfalls?
16:41:15 <rakhmerov> for me it seems to be fine
16:41:47 <rakhmerov> I just want to make sure we will be able to implement it quickly
16:41:49 <NikolayM416> rakhmerov: I think it is good idea but it requires auth
16:41:53 <rakhmerov> people are waiting :)
16:42:08 <rakhmerov> authentication should be done
16:42:20 <igormarnat> I think .3 is fine, providing that we'll start working on implementation of PoC of data flow in parallel with discussion of results of current PoC
16:42:38 <rakhmerov> igormarnat, agree with you
16:43:18 <rakhmerov> ok, let me make a suggestion
16:43:32 <rakhmerov> given on the information that you guys provided
16:44:54 <rakhmerov> in my understanding, https://review.openstack.org/#/c/62037/ is the only thing we're waiting to be able to create simple (w/o Data Flow) workflows working with OpenStack, right?
16:45:21 <rakhmerov> it's now failing on the tests
16:45:32 <rakhmerov> I think we need to fix it somewhere
16:45:45 <rakhmerov> akuznetsov, can you please take a look at it tomorrow?
16:46:28 <akuznetsov> Yes
16:46:33 <rakhmerov> so my suggestion is to proceed with option 3 (simple app w/o OpenStack at all) for now, it should be quickly enough (~1-2 days)
16:47:09 <rakhmerov> and once we merge all required things for authentication and keystone trust mechanism we can create a simple workflow for backing VMs up
16:47:30 <rakhmerov> it should be pretty easy too
16:47:49 <rakhmerov> so, in other words, my suggestion is:
16:48:16 <rakhmerov> 1. High priority: simple demo app with endpoints to demonstrate Mistral w/o OpenStack
16:48:46 <rakhmerov> 2. Medium priority: simple workflow for backing up VMs on schedule
16:49:04 <rakhmerov> once we get auth done
16:49:09 <rakhmerov> sounds good for you?
16:49:53 <igormarnat> Looks good because it's going to work both with and w/o OS, and will not require changes in engine and app, only an addition of data flow, right?
16:50:03 <rakhmerov> right
16:50:09 <rakhmerov> my point here is to start testing engine as soon as possible
16:50:10 <akuznetsov> I think we should demonstrate Mistral with OpenStack because Mistral is OpenStack project, not an some abstract workflow as service
16:50:34 <rakhmerov> we will, now worries
16:50:38 <rakhmerov> no worries
16:50:46 <rakhmerov> as we get these things done with auth
16:51:33 <tnurlygayanov> rakhmerov, ok, when we will have the first prototype which will deploy something?
16:51:35 <igormarnat> no worries is much better than now worry:)
16:51:52 <rakhmerov> yes indeed :)
16:51:59 <rakhmerov> it just bothers me that we didn't start testing all main mechanisms together: REST API, client, Scheduler, engine
16:52:25 <rakhmerov> and that's why I think simple app w/o OpenStack would allow us to finally start doing it
16:52:46 <rakhmerov> I'm pretty sure we'll find a lot of problems once we start doing that
16:53:00 <tnurlygayanov> yes )
16:53:28 <rakhmerov> but again, don't get me wrong: I want to demonstrate it with OpenStack too
16:54:09 <igormarnat> Agree, this is the best way to go
16:54:54 <igormarnat> Anything else in agenda?
16:54:58 <rakhmerov> #action NikolayM, get a simple demo app done by Wed w/o OpenStack
16:55:34 <rakhmerov> #action akuznetsov, finalize work related with authentication and keystone trusts
16:55:58 <rakhmerov> well, this is the main thing I wanted to bring up actually
16:56:11 <rakhmerov> and I'm glad we agreed on that
16:56:32 <igormarnat> The audience is small for now, it helps sometimes:)
16:56:40 <rakhmerov> yep :)
16:56:51 <rakhmerov> the other thing I was going to talk about is the engine architecture itself but looks like we don't have time today
16:57:16 <rakhmerov> we can do it next time along with results we'll get done
16:57:19 <igormarnat> Yep, let's postponed it for the future. For now you can drop the link here, if any
16:57:45 <rakhmerov> my bad, I think it's still not published on the wiki
16:58:06 <rakhmerov> shame on me (I'm too deep into development now)
16:58:37 <rakhmerov> #action rakhmerov, publish POC Mistral Engine architecture on wiki
16:58:54 <rakhmerov> ok, let's wrap up for today
16:59:00 <rakhmerov> thanks to everyone!
16:59:09 <rakhmerov> #endmeeting