08:01:50 #startmeeting mistral 08:01:51 Meeting started Fri May 4 08:01:50 2018 UTC and is due to finish in 60 minutes. The chair is d0ugal. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:01:53 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 08:01:55 The meeting name has been set to 'mistral' 08:02:12 rakhmerov, apetrich, bobh, mcdoker181818: ping! ^ 08:02:16 Happy friday everyone :) 08:02:22 I just need to get a coffee, back in a few mins. 08:02:30 d0ugal: hey 08:02:30 o/ 08:02:31 here 08:02:53 I don't have anything particular for the agenda, I'd like to go through and tidy up some of the blueprints maybe 08:02:59 Otherwise happy to discuss anything people have! 08:03:12 but first, coffee :) 08:09:25 https://blueprints.launchpad.net/mistral/ 08:09:34 So we have 140 blueprints, that is more than the number of bugs :) 08:09:54 I guess we should triage blueprints like we do with bugs. Lots of them are "New" and "Not Started" 08:18:02 :) 08:18:05 ok 08:21:14 d0ugal: I'm here 08:21:25 rakhmerov: Great 08:21:35 I'm just reading some blueprints, I have not seen many of these before :) 08:22:10 I am also reading https://help.launchpad.net/Blueprint - just so I can learn how blueprints should be used :) 08:22:23 It might be that we don't really need to do anything with them... I am not sure 08:22:27 ok 08:22:57 Hi, all. I created new bug ticket yesterday https://bugs.launchpad.net/mistral/+bug/1769012 . We can discuss it after triage blueprints. 08:22:58 Launchpad bug 1769012 in Mistral "Workflow pause with task retry policy" [Undecided,New] 08:23:10 well, in my understanding we just need to go over them and assign them to cycles and milestones according to our dev plans 08:24:33 rakhmerov: sure, but I don't think they should stay in "New" for design, like a bug that means they have not been triaged? 08:24:44 yes 08:24:45 they should either be approved or rejected I guess? 08:24:53 btw, did we clean up BPs for R-2? 08:25:11 rakhmerov: no, not yet. 08:25:12 d0ugal: I think they can be rejected, yes. If they are nonsense ) 08:25:16 :) 08:25:31 or Discussion looks like a useful status if they need to be talked about more 08:26:14 yes 08:26:21 It is just a task I want to do a little bit at a time - I think the bugs are much tidier now, so I'd like to do something similar with blueprints. 08:26:26 I'm not sure how to reject them properly though.. 08:26:34 Good question :) 08:26:34 maybe just mark them as "obsolete" 08:27:14 https://wiki.openstack.org/wiki/Blueprints 08:27:18 That also looks useful. 08:27:37 First I think we should discuss mcdoker1818's bug. 08:27:46 ok 08:27:54 as you wish, commandor ) 08:28:13 haha, I think it is something he wants to work on now. The blueprints can wait longer :) 08:28:21 yes 08:28:25 reading it.. 08:28:30 #link https://bugs.launchpad.net/mistral/+bug/1769012 08:28:34 Launchpad bug 1769012 in Mistral "Workflow pause with task retry policy" [Undecided,New] 08:28:34 I am too :) 08:29:05 woow, so many details.. 08:29:37 btw, just something I noticed immediately: the title of the bug doesn't describe the bug 08:29:39 :) 08:30:31 there's a good BLUF (bottom line up front) principle that I personally try to always use 08:31:20 mcdoker1818: Vitalii, can you please put a line right in the beginning of the description that reflects the matter of the bug 08:31:31 (or change the bug title) 08:32:26 Updated :) 08:32:57 ok, thanks 08:34:00 I wouldn't delete the details - just make sure there is an easy to understand summary at the top 08:34:50 I also slightly updated it 08:34:57 yep 08:35:09 you can add details after a short summary 08:35:17 but ok, I understand it now 08:35:24 Thanks! 08:35:47 I guess the bug should be pretty easy to fix 08:36:08 my assumption is that retry policy doesn't take PAUSED state into account for some reason 08:36:10 Yeah, so in summary - pausing a workflow breaks the retry policy? 08:36:20 right 08:36:42 maybe "Retry policy keeps iterating if the workflow is paused" :) 08:36:55 +1 08:37:13 https://github.com/openstack/mistral/blob/master/mistral/engine/tasks.py#L232 08:37:46 I changed the bug to "Confirmed" - how important is this bug for you? :) 08:38:05 mcdoker1818: yes, but DELAYED is not the same as PAUSED 08:38:30 rakhmerov: do we have a good desription of the states somewhere? I get confused with them sometimes. 08:38:30 Yep, I mean the retry executes before check the pause state 08:38:54 just to clarify: DELAYED is mostly an internal state needed to tell Mistral "this task is running but it's delayed due to some internal implementation reasons, like policy or something else" 08:39:08 d0ugal: I plan to fix it soon 08:39:16 PAUSED means that a user stopped it temporarily on purpose 08:39:31 mcdoker1818: thanks, I added it to rocky-2 08:39:37 d0ugal: probably we don't, let me check 08:39:43 ok 08:40:57 rakhmerov: https://github.com/openstack/mistral/blob/master/mistral/workflow/states.py#L18 :) 08:41:11 :DD 08:41:15 d0ugal: so we have some info about the states in the spec but that's definitely not full 08:41:45 d0ugal: haha :)) 08:42:06 rakhmerov: I guess the main problem how resume task iterations after resume execution 08:43:30 mcdoker1818: it shouldn't be a problem 08:43:54 we save info about retry iteration in the task 'runtime_context' field 08:44:04 under the key 'retry' or something like that 08:44:17 so we always know what the current iteration is 08:44:40 rakhmerov: As I know we resume tasks which has IDLE state 08:45:19 nope 08:45:21 PAUSED 08:45:21 arthur100 08:45:33 IDLE is for a different purpose 08:46:49 We don't change state task to PAUSED 08:47:59 why not? 08:49:26 so the states change in this case as follows: RUNNING or RUNNING_DELAYED [we pause wf] -> PAUSED [we resume wf] -> RUNNING or RUNNING_DELAYED 08:50:11 as far as RUNNING_DELAYED, you can perceive it as a sub state of RUNNING 08:50:17 let me check 08:50:26 so it's a flavor of RUNNING state 08:50:46 hardikjasani: hey 08:50:56 what is arthur100? :) 08:51:21 typed it in wrong window :D 08:51:31 Nothing critical of course 08:52:31 ok ) 08:52:53 hardikjasani: btw, how is it going? Progressing with your tasks? 08:55:26 rakhmerov: Ok, thank you for clarifying! 08:55:32 np 08:58:15 rakhmerov: going great! 08:58:22 ) 08:58:57 d0ugal, rakhmerov: What is about this ticket https://bugs.launchpad.net/mistral/+bug/1767830 ? Should I create blueprint for new api? 08:58:58 Launchpad bug 1767830 in Mistral "Execution and task specification can be out of date" [Medium,Confirmed] 08:59:21 Or is it a bug? 09:00:03 Sory, I got distracted for a moment there. 09:00:16 mcdoker1818: Looking. 09:00:34 http://localhost:8989/v2/execution/%ex_id%/spec 09:00:37 for example 09:00:44 Yeah, I remember now. 09:00:57 mcdoker1818: I am happy for you to treat it like a bug 09:00:59 hm.. 09:01:12 well, wait a second 09:01:23 so my concern is the following 09:02:02 the 'spec' field is an internal thing and it may look much different from the initial YAML text 09:02:06 that's the thing.. 09:02:19 and that is why it was not exposed in the first place 09:03:02 as a solution, we could keep a snapshot of the initial *YAML* chunk but that's extra space 09:03:08 yeah 09:03:18 we in many cases have many megs of data there 09:03:26 I guess the best solution is to keep old versions of a workflow while there is a related execution. 09:03:41 and imagine if that is stored for every execution 09:04:03 That is why I think it would be better to version the exectuion 09:04:09 > the 'spec' field is an internal thing and it may look much different from the initial YAML text 09:04:17 d0ugal: yeah, the most decent solution that I can think of is workflow versioning 09:04:21 Why do you think that it is a problem? 09:04:36 so we just have one unique copy of every workflow - then a each execution could have a reference to the workflow and a version id (the sha of the yaml contents?) 09:04:56 so that workflow definition keep versions and the reference should look like "WF id = 1-2-3-4, ver 5" 09:05:05 mcdoker1818: if we expose an internal data structure to users then we need to support and document it. 09:05:24 yep 09:05:40 I'd really hold on with this for now, honestly 09:05:41 Noop, we will not. We create a ExecutionSpec - resource 09:05:58 And we will expose it 09:06:14 mcdoker1818: from user perspective it doesn't make much sense 09:06:15 I don't follow. 09:06:20 we already have workflows 09:06:26 (i.e. workflow definitions) 09:06:46 why would user need to deal with one more thing that's called Spec something.. 09:06:47 ? 09:07:08 from the user perspective they just deal with different versions of workflows 09:07:16 but we don't support it properly now 09:08:04 it can be done on the user side though: just introduce the policy not to every update workflows and use special naming that includes a version 09:08:05 that's it 09:08:07 of you right, yes 09:08:19 all of you are right, yes 09:08:31 you can even forbid the corresponding operation in the policy.json file 09:08:33 yeah 09:09:11 That is a decent work-around for now. 09:09:52 Okay - so I guess we consider this bug to be invalid? 09:09:58 I'm sorry I missed what is wa? 09:10:27 mcdoker1818: What rakhmerov said. Don't let users update workflows via policy.json and then you can always get the original workflow. 09:10:37 hahahaah 09:10:47 Then instead of updating workflow you always create new workflows (and delete old ones later) 09:10:50 noop, it's not wa :) 09:11:06 Why not? 09:11:41 sorry, what is "wa"? 09:11:41 We use workflow by name in case of start sub-workflow (from task) 09:11:43 not following.. 09:11:51 work-around 09:11:58 ooh 09:12:06 Good point. That makes it harder. 09:12:21 sub-workflow names can be dynamic :) 09:12:25 but I understand, ok 09:12:34 but that would get ugly :) 09:13:46 mcdoker1818: Why do you need the original spec? 09:13:55 UI 09:13:55 What do you want to do with it? 09:13:59 I see 09:14:11 Would it fit in the workflow execution description? :) 09:14:59 no, I guess not. 09:15:03 Sorry, I don't understand you :) 09:15:36 Don't worry - the idea won't work. 09:16:25 the reason I'm arguing this much is that I don't want workarounds 09:16:28 I hate them ) 09:16:46 they have a trend to live there for long 09:17:04 if this is so important then why not implement it in the right way? 09:17:04 :D 09:17:14 I like the idea with the version 09:17:37 d0ugal: maybe we just need to write a spec and see so that it's backwards compatible and decently implement it? 09:17:45 rather than ending up with workarounds 09:18:02 rakhmerov: Right, that is why I am trying to think of a workaround from the users perspective 09:18:04 I have never used namespaces - but could a new, unique namespace be used each time? do sub-workflow executions look within a namespace? 09:18:45 rakhmerov: sure, I just wasn't sure how easy/realistic that was 09:18:55 but I am happy for workflow versioning to be added. 09:19:44 ok 09:20:03 so yes, my suggestion is at least write a spec and see how hard it would be 09:20:12 mcdoker1818: is this something you can work on? 09:20:30 let's do some research within a reasonable amount of time and then make a decision, how's that sound? 09:20:55 Sure, sounds good. I don't have the time for this, but somebody should go fot it :) 09:20:58 for it* 09:21:29 It's not a blocker for me right now, but I think it will be :) Yeap, I can do it a little later 09:22:18 Okay, sounds good 09:22:28 I'll mark the bug as invalid for now and sumarise this. 09:22:39 Ok 09:25:03 Thanks! 09:25:06 I am going to end the meeting bot before I forget, but I will be around for the rest of the day :) it is only 10:30am here. 09:25:08 #endmeeting