16:01:15 <rakhmerov> #startmeeting Mistral
16:01:16 <openstack> Meeting started Mon Dec 15 16:01:15 2014 UTC and is due to finish in 60 minutes.  The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:20 <openstack> The meeting name has been set to 'mistral'
16:01:24 <rakhmerov> hi all
16:01:27 <rakhmerov> again
16:03:17 <nmakhotkin> hi !
16:03:26 <dzimine> hi
16:03:27 <rakhmerov> hi
16:04:14 <rakhmerov> alright, let's go
16:04:19 <rakhmerov> #topic Review Action Items
16:04:22 <rakhmerov> 1. akuznetsova: check if task-defaults work for reverse workflows
16:04:39 <rakhmerov> done, there was actually a bug not only in reverse workflow
16:04:42 <rakhmerov> I fixed it
16:04:51 <rakhmerov> it's merged
16:05:05 <rakhmerov> 2. NikolayM, create new BPs for what's left on "for-each"
16:05:39 <rakhmerov> Nikolay, could you please insert the links here?
16:05:44 <rakhmerov> related to this work
16:05:59 <bhavenst> aloha
16:06:17 <rakhmerov> hi Bryan
16:06:20 <rakhmerov> how are you?
16:06:24 <nmakhotkin> yes, one second
16:06:37 <nmakhotkin> launchpad is too slow
16:06:51 <bhavenst> not bad, thanks
16:07:02 <nmakhotkin> one for parallelism - https://blueprints.launchpad.net/mistral/+spec/mistral-for-each-parallelism
16:07:21 <rakhmerov> ok
16:07:29 <rakhmerov> (to both of you :) )
16:07:45 <nmakhotkin> and (it was before) one for "fails-allowed" - https://blueprints.launchpad.net/mistral/+spec/mistral-for-each-cardinality
16:08:22 <rakhmerov> ok, I'll just add from myself. There's also a doc created by Nikolay https://docs.google.com/a/mirantis.com/document/d/1iw0OgQcU0LV_i3Lnbax9NqAJ397zSYA3PMvl6F_uqm0/edit
16:08:33 <nmakhotkin> yes
16:08:38 <rakhmerov> where here's writing on complete for-each specification
16:08:40 <dzimine> ^^ thanks Nikolay
16:08:43 <rakhmerov> please participate
16:08:52 <rakhmerov> ok
16:09:09 <rakhmerov> #topic Current status (progress, issues, roadblocks, further plans)
16:09:13 <nmakhotkin> dzimine, could you take a look on it?
16:09:38 <dzimine> yes I did already, and going to 1) have comments 2) run by our devops and get their take on it.
16:10:09 <dzimine> some things off the top I am not happy about is increased verbosity and 'computer-scienctific language' which not all normal people can understand.
16:10:14 <rakhmerov> my status: last week I completed what was left on "join", fixed that bug with "task-defaults" (didn't work if there was something in 'task-defaults' and nothing in tasks), discussed further plans with Dmitri
16:10:49 <rakhmerov> dzimine, what do you mean?
16:10:57 <rakhmerov> I'm really curious about it
16:11:29 <dzimine> I'll comment in the document, but e.g. 'concurrency' makes sense from inside, less from outside.
16:11:42 <bhavenst> Status: Finishing up pause-before (Renat's comments).  Realized I didn't test with pause-before set as a task-default so will do that too.
16:12:07 <nmakhotkin> status: I fixed some things in for-each, refactor it. Also my commits on docs were merged.
16:12:14 <rakhmerov> dzimine, all the terminology is up to discussion, nothing is decided yet
16:12:53 <rakhmerov> bhavenst, ok, good. I just left some minor comments, overall looks good
16:13:22 <rakhmerov> I wonder if Nastya is here?
16:13:32 <rakhmerov> akuznetsova?
16:14:25 <rakhmerov> she's in skype but she may have another meeting at the moment
16:14:40 <rakhmerov> ok, let's go on
16:14:50 <dzimine> info: my-status: 1) finally caught up with for-each functionality and spec and created all this mess with us needing to review and redo it; 2) with Winson, looked into 'runtime / global context' BP to figure how we want it. Still more design/thinking needed, need to talk it over with you guys. 3)
16:15:24 <rakhmerov> yes
16:15:30 <dzimine> @lakshmi from our team has joined the project, got the first patch  - fix the logging. Manas is working on better logging.conf sample
16:15:56 <rakhmerov> ooh, that's good
16:16:19 <rakhmerov> dzimine and others, on that "globals" etc. topic
16:16:41 <rakhmerov> my general comment is that it feels to me we're discussing it with too many minor technical details
16:16:58 <rakhmerov> so that we can't see the trees behind the forest
16:16:59 <dzimine> 4) we began to implement internally the 'client' to watch the workflow execution, and out of this looking to the API and will bring changes to help clients track the workflow with the reasonable number of API calls.
16:17:28 <rakhmerov> ok
16:17:57 <dzimine> 5) https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http - we learned that for a client this is not a reliable way to track the workflow execution, thus deferred the blueprint for now.
16:18:04 <rakhmerov> all the way around rather.. :) "forest behind the trees"
16:18:05 <dzimine> aha, one more:
16:18:28 <rakhmerov> yes, I moved it to the next cycle
16:18:38 <rakhmerov> I don't want to rush with this BP
16:18:49 <rakhmerov> because it touches lots of important things
16:19:22 <dzimine> that's it.
16:19:36 <rakhmerov> quick question
16:19:48 <rakhmerov> you said "will bring changes to help clients track the workflow with the reasonable number of API calls."
16:19:58 <dzimine> and rakhmerov, your fix on transaction DID seem to fix the 2CPU bug I mentinoed before.
16:20:11 <rakhmerov> ooh, really??
16:20:21 <rakhmerov> it may have actually, yes
16:20:33 <dzimine> I mean the current api seems about right, but if anything is suboptimal, we'll fix ti.
16:20:51 <rakhmerov> yes, ok
16:21:03 <rakhmerov> let's discuss it separately, if needed
16:21:12 <rakhmerov> ok, let's continue
16:21:29 <rakhmerov> #topic Release "Kilo-1" progress
16:21:43 <rakhmerov> I don't have much to discuss on that since last time
16:22:08 <rakhmerov> I included it just to have a chance to discuss something related to it quickly
16:22:29 <rakhmerov> so, one thing that's been already mentioned is that we moved "event-listeners" BP to the next cycle
16:22:42 <rakhmerov> as far as other things
16:22:58 <rakhmerov> Bryan is about to complete "pause-before" I think
16:23:23 <rakhmerov> Bryan, do you feel you can do it within the next couple of days?
16:23:44 <bhavenst> Yeah, I'll try to finish it today
16:23:53 <rakhmerov> seems like there shouldn't be much work but not sure about your work plans
16:23:58 <rakhmerov> ok, thanks a lot
16:24:00 <bhavenst> Don't see why I can't at this point.  Only if I hit something unexpected
16:24:11 <rakhmerov> yeah, ok
16:24:41 <rakhmerov> it would be nice if you could alter resume algo a little bit (should be not that hard) right in this patch
16:24:51 <rakhmerov> so that the change would be atomic
16:25:45 <bhavenst> I'm not sure if that's possible for the same reason I already experienced.  That I can't get a handle on existing task objects.
16:26:02 <bhavenst> But I'll try
16:26:26 <bhavenst> Adding IDLE is no problem there, but that doesn't seem to be sufficient to actually resume the task.
16:26:27 <rakhmerov> handle on existing task objects?
16:26:38 <rakhmerov> ooh
16:26:47 <rakhmerov> not sure I'm 100% right but
16:26:57 <bhavenst> I can pick up that a task is in IDLE but the only way to trigger it is to instantiate a new one
16:27:02 <bhavenst> which just gets paused again. :)
16:27:30 <rakhmerov> resume basically fetches tasks recursively and finds ones for which db tasks don't exist yet
16:27:32 <bhavenst> The task execution object is built and prepared, but exec was skipped.  So I need to call exec on that object itself
16:27:34 <rakhmerov> as far as I remember
16:27:49 <rakhmerov> so IMO we just need to change that condition a little bit
16:28:16 <rakhmerov> from "db instance doesn't exist" to "db instance doesn't exist or if it does then it's not in IDLE state"
16:28:21 <bhavenst> Yeah, I tried that already.
16:28:26 <nmakhotkin> AFAIK we can use another method of engine to run existing (and prepared) task
16:28:28 <bhavenst> Resulted in 2 of the same task
16:28:42 <rakhmerov> ook
16:28:54 <bhavenst> I could delete from db like you suggested, but would need to maintain the "already paused" state somewhere else
16:28:54 <rakhmerov> hm... maybe I really don't see something
16:29:02 <bhavenst> more likely I don't
16:29:03 <bhavenst> :)
16:29:11 <rakhmerov> :))
16:29:38 <bhavenst> I'll try it again following your comments here.
16:29:56 <rakhmerov> ok, anyway, please try to spend some time on that and if you get stuck with something let us know about your findings
16:29:59 <rakhmerov> sure
16:30:00 <rakhmerov> thanks
16:30:05 <bhavenst> will do
16:30:14 <rakhmerov> ok, another thing
16:30:21 <rakhmerov> HA testing is already going on
16:30:32 <rakhmerov> I mean we have a BP about that
16:30:46 <rakhmerov> it's in progress and Nastya already got some results
16:31:02 <rakhmerov> but, of course, the whole thing is really big
16:31:10 <rakhmerov> and this is just the beginning
16:31:16 <dzimine> I recall we asked Nastya to share the test plans. Right?
16:31:23 <rakhmerov> yes!
16:31:28 <rakhmerov> I asked her again
16:31:37 <rakhmerov> she's planning to do that this week
16:31:39 <dzimine> meaning, 'we tested it and it worked ok' doesn't give much specicifity / confidence
16:31:52 <dzimine> test first, plan last :)
16:31:59 <rakhmerov> #action akuznetsova, share all the work on HA testing & benchmarking
16:32:13 <dzimine> that will bring us to Test Driven Deveoplment point which I want in open discussion :)
16:32:14 <dzimine> :)
16:32:17 <rakhmerov> sure-sure
16:32:20 <rakhmerov> you're right
16:32:42 <rakhmerov> it's being taken care of, don't worry :)
16:32:56 <rakhmerov> I mean not only this testing ;)
16:33:30 <rakhmerov> we also have a couple of bugs that we'd like to fix in this cycle
16:33:54 <rakhmerov> one of them is https://bugs.launchpad.net/mistral/+bug/1401039
16:33:55 <dzimine> what are the bugs?
16:33:57 <uvirtbot> Launchpad bug 1401039 in mistral "Input inline syntax does not escape "=" symbol" [Medium,New]
16:34:16 <rakhmerov> looks like Nikolay knows how to fix it
16:34:36 <rakhmerov> but if Lakshmi wants to fix it himself he can try to do it
16:35:03 <rakhmerov> Nikolay, can you pls share your idea with Lakshmi about how this issue can be fixed?
16:35:39 <dzimine> AFAIK @lakshmi looked at it, he left the suggestions in the bug and passed it back to Nikolay
16:35:41 <rakhmerov> because what Lakshmi suggested in the ticket is not really going to work (IMO)
16:35:42 <dzimine> up to you two.
16:35:54 <rakhmerov> ok
16:36:00 <dzimine> if Nikolay knows how to do it just go ahead fix.
16:36:06 <rakhmerov> :)
16:36:07 <rakhmerov> ok
16:37:06 <rakhmerov> there's one more thing: https://bugs.launchpad.net/mistral/+bug/1324967
16:37:08 <uvirtbot> Launchpad bug 1324967 in mistral "Fix devstack back to using Rabbit when the RPC problem is fixed" [High,In progress]
16:37:09 <nmakhotkin> I've already sent the mail to Lakshmi
16:37:10 <rakhmerov> but it's really minor
16:37:14 <rakhmerov> and not urgent
16:37:20 <nmakhotkin> about how to fix this bug
16:37:24 <rakhmerov> but we may want to look at it
16:37:31 <rakhmerov> ok
16:37:43 <rakhmerov> maybe he'll want to do that by the end of the day
16:37:53 <rakhmerov> if not, then let's go ahead and fix it tomorrow
16:38:21 <rakhmerov> let's now try to discuss "for-each" a little bit
16:39:02 <rakhmerov> #action nmakhotkin, fix the bug with "=" sign in simplified syntax if Lakshmi doesn't fix it himself by the end of Monday
16:39:10 <rakhmerov> #topic "for-each" spec
16:39:31 <rakhmerov> the doc once again: https://docs.google.com/a/mirantis.com/document/d/1iw0OgQcU0LV_i3Lnbax9NqAJ397zSYA3PMvl6F_uqm0/edit
16:39:52 <rakhmerov> so is there something right now that we could discuss?
16:39:56 <dzimine> on 'for-each' - good news we understand the scope of the feature and Nikolay has captured it in the spec (almost0.
16:40:04 <rakhmerov> we don't have much time but anyway
16:40:13 <rakhmerov> at least we could scratch the surface
16:40:30 <dzimine> Bad news, as they say in Product Management, 'the answer is not in this building', meaning what makes sense to us turned out to make little sense to the users.
16:40:45 <dzimine> My concerns: increased verbosity.
16:41:11 <rakhmerov> give an example
16:41:24 <rakhmerov> where?
16:42:20 <rakhmerov> dzimine?
16:42:41 <dzimine> for each now itself has two subsections.
16:42:56 <dzimine> properties and input.
16:43:07 <dzimine> input is a duplication of task's input, thus confusing.
16:43:08 <rakhmerov> yes, but it's self-contained
16:43:33 <rakhmerov> I agree on "input", hence I suggested some different name
16:43:41 <rakhmerov> like "loop-expressoin" or something like that
16:43:42 <dzimine> well, we better do this off-line in the document, and also bounce this off the guys who are trying to use the DSL and have already give us the feedback on it's verbosity.
16:43:56 <rakhmerov> would be good to consult with native speakers on better alternatives :)
16:44:10 <rakhmerov> yes
16:44:11 <rakhmerov> ok
16:44:41 <rakhmerov> just one wish
16:44:54 <rakhmerov> please don't consider it as something that we're pushing
16:45:14 <rakhmerov> you are the equal participants of the discussion
16:45:22 <rakhmerov> suggest your options
16:45:32 <dzimine> there are 2 cases where we need to be good: 1) simplest form of for-each which is used in 80% cases thus shouldn't carry any heavy stuff, RIGHT DEFAULTS, and 2) all the complex variations - parallel limits (concurrency), combinations, etc.
16:45:50 <rakhmerov> yes, agree
16:45:52 <rakhmerov> ooh, btw
16:46:06 <rakhmerov> we actually had an interesting thought
16:46:12 <dzimine> I'll look at it today/tomorrow and get the native speaker devops feedback on it, too.
16:46:35 <rakhmerov> for-each: p1={$.array1} p2={$.array2}
16:46:42 <dzimine> Nikolay can you do me a favor and beef up the example so it's the whole 'task', not just for-each part of it, so it will make sense without explanation.
16:46:44 <rakhmerov> it's super concise
16:46:54 <rakhmerov> but it's not clear how to tune it
16:47:09 <rakhmerov> I mean provide other options like "parallelism" and others
16:47:17 <dzimine> got it.
16:47:23 <rakhmerov> so please think about it as well
16:47:31 <dzimine> if we figure the long form, turning it to short form is not a big deal.
16:47:45 <rakhmerov> dzimine, agree that we need a full example
16:48:05 <rakhmerov> #action nmakhotkin, add full example(s) to 'for-each' spec
16:48:16 <rakhmerov> ok
16:48:52 <dzimine> let's add another action to discuss the contexts, the other blueprint.
16:49:02 <dzimine> We can similarly write a spec and flash it out.
16:49:23 <dzimine> And for the last 10 min I want to discuss Test driven develpment.
16:49:56 <rakhmerov> I actually planned a topic about contexts (scoping) but ok
16:50:04 <rakhmerov> if you want we can skip it
16:50:38 <rakhmerov> #action all, keep discussing contexts (scopes) in the mailing list and other channels
16:50:48 <rakhmerov> #topic Test Driven Development
16:50:53 <rakhmerov> fire away
16:51:37 <dzimine> no point of re-stating the problem :)
16:51:42 <dzimine> or?
16:51:47 <dzimine> else I jump to solution.
16:52:12 <dzimine> We are editing the spec on for-each now, how about we try and make a test a spec.
16:52:36 <rakhmerov> whatever you think is needed Dmitri
16:53:09 <rakhmerov> ok, here's my input
16:53:16 <dzimine> and review the test as WIP on gerrit reveiw, make sure we agree on functionality and how it is tested, there, and add implementation.
16:53:36 <rakhmerov> I think in this case it makes sense to actually have a document
16:53:47 <rakhmerov> so that we could explain something, add examples, comments etc.
16:53:55 <dzimine> ^^ it does, yes.
16:54:00 <rakhmerov> at least we should agree on something..
16:54:04 <rakhmerov> before we start coding
16:54:10 <rakhmerov> I'm not saying on everything
16:54:21 <dzimine> yes on for-each, and may be on context
16:54:26 <rakhmerov> yes
16:54:36 <rakhmerov> but in 80% of the cases I agree
16:54:54 <rakhmerov> we can write a spec as a test first and then implement it
16:55:54 <rakhmerov> so may be it makes sense to repeat the main point
16:56:10 <rakhmerov> we're appealing to write tests!
16:56:12 <rakhmerov> and
16:56:21 <rakhmerov> not only to write them
16:56:29 <rakhmerov> but write before implementation
16:56:45 <rakhmerov> not always it's possible though
16:56:48 <dzimine> let's test-drive an learn to do it on for-each.
16:56:51 <rakhmerov> but we should try
16:56:56 <rakhmerov> yes
16:57:10 <rakhmerov> somethings, for example, when we need to research something it may not be possible
16:57:11 <dzimine> once we flash out the feature, review the test, once we happy with the test, we go to implementation.
16:57:16 <rakhmerov> which is ok
16:57:25 <rakhmerov> yes
16:57:28 <rakhmerov> I'm 100% for it
16:57:56 <rakhmerov> 2 mins left
16:57:58 <dzimine> and it's case-by-case, I am not proposing to do this for every BP.
16:58:00 <rakhmerov> anything else folks?
16:58:16 <rakhmerov> I would say in most cases it should work
16:58:20 <rakhmerov> this approach
16:58:31 <rakhmerov> from my previous experience
16:58:51 <rakhmerov> ok, 1 min left
16:58:56 <dzimine> ok, for-each - 1) docs 2) test, 3) impl
16:58:58 <dzimine> agreed?
16:59:08 <rakhmerov> yes, we did
16:59:14 <dzimine> Niklay, are you on board?
16:59:22 <rakhmerov> I hope Nikolay either did :)
16:59:35 <rakhmerov> ok, guys
16:59:37 <nmakhotkin> yes, I'am ok with it
16:59:38 <rakhmerov> it's time
16:59:44 <rakhmerov> thanks for coming
16:59:49 <nmakhotkin> bye!
16:59:49 <rakhmerov> bye
16:59:56 <rakhmerov> #endmeeting