Tuesday, 2016-07-05

openstackgerritDigambar proposed openstack/watcher: Fix field type to audit_type  https://review.openstack.org/33639000:38
*** Kevin_Zheng has joined #openstack-watcher03:34
*** diga has joined #openstack-watcher04:16
openstackgerritlicanwei proposed openstack/watcher: Fix link error in base-setup.rst  https://review.openstack.org/33745606:40
openstackgerritJinquan Ni proposed openstack/watcher: Remove duplicate unittest  https://review.openstack.org/33746206:50
*** vincentfrancoise has joined #openstack-watcher07:08
*** danpawlik has joined #openstack-watcher07:12
digavincentfrancoise: Hi - https://review.openstack.org/#/c/336390/07:18
digaPlease review above patch07:18
*** dtardivel has joined #openstack-watcher08:03
digavincentfrancoise: acabot : Can someone can review my patch - https://review.openstack.org/#/c/336390/08:22
vincentfrancoisediga: hi08:29
vincentfrancoiseFYI I'll probably be doing some reviews this afternoon08:30
digavincentfrancoise: sure09:07
*** diga has quit IRC09:31
dtardiveldiga: review done09:39
tkaczynskihi guys. can I ask few questions about how watcher works internally (communication between watcher modules)?10:31
tkaczynskinot sure if anybody is around :)10:33
danpawliktkaczynski: I guess not :)10:37
tkaczynskiI'll wait then, I need some interaction as my questions might explode ;)10:38
*** vincentfrancoise has quit IRC11:17
*** acabot has quit IRC11:17
*** jed56 has joined #openstack-watcher11:33
*** vincentfrancoise has joined #openstack-watcher11:39
vincentfrancoisehi tkaczynski11:39
openstackgerritMerged openstack/python-watcherclient: Add license file  https://review.openstack.org/33734711:39
tkaczynskihi vincentfrancoise11:39
vincentfrancoisedid your comment from earlier already exploded?11:39
tkaczynskiso here's what I want to do: I 'm creating a scoring module for watcher now and I want decision engine to communicate with it. specifically, from some strategy I want to simply call some scoring engine and get the response from it11:41
tkaczynskino, the earlier thing is quiet ;)11:41
vincentfrancoisebut if you want to communicate, this means it will be tightly related to the scoring engine one will be using isn't it?11:42
tkaczynskiso I looked how is communication between modules already implemented and I see that it's using messaging, even though the clients are called "rpcapi" ;)11:43
tkaczynskiyes. if a strategy wants to use some scoring engine, it will be tightly coupled to it from that point11:43
vincentfrancoiseat the moment the whole messaging system is not really tidy11:43
vincentfrancoisein watcher I mean11:44
tkaczynskibut I don't think it will work in my case because I need really a synchronous call: send data to scoring engine and get the response11:44
tkaczynskiwhere the messaging is async really. the job is submitted and sometime later a notification comes11:45
tkaczynskiso implementing scoring module as, say applier, is probably not a good idea. do you agree?11:46
tkaczynskijed56: I would appreciate your opinion as well11:48
vincentfrancoiseWouldn't there be any other way of communicating with it?11:48
tkaczynskiwell, that's the next question :)11:49
tkaczynskiit seems that the reasonable choice might be a rest service, similar to the API11:50
vincentfrancoisetkaczynski: I'm digging a bit11:50
jed56you can do a sync call11:50
tkaczynskisync call using the existing messaging infrastructure?11:50
jed56     cast - the method is invoked asynchronously and no result is returned to the caller11:51
jed56    call - the method is invoked synchronously and a result is returned to the caller11:51
jed56yes with oslo messaging11:51
jed56However, if we have a huge volume of data this is not a good idea to use oslo messaing11:52
tkaczynskijed56: are you talking about this?11:53
tkaczynskithis is from applier/rpcapi.py11:53
jed56tkaczynski: yes for example11:53
tkaczynskiso the oslo messaging will take care of everything? even if I have multiple events being sent at the same time?11:55
jed56tkaczynski: yes11:56
jed56you have the notion of context11:56
tkaczynskicontext is kind of another problem - from what I see it comes from the pecan (so from the API)11:57
tkaczynskistrategies have no context inside, so I have nothing to put when calling the scoring engine RPC client11:58
jed56by context, I mean the context of your call (action_plan_uuid )11:59
jed56with the parameters11:59
jed56I  looking for a example for you11:59
tkaczynskiyes, but with scoring engine there might be many strategies calling the same scoring engine (same id), I still want to route the response to the right caller12:00
jed56one call one reponse one context12:01
jed56you don't have to manage the concurrency12:01
jed56this is manage by the broker12:01
tkaczynskiok, I can try that. this also means that there is no point of trying to use multithreading in scoring module12:03
tkaczynskibecause then it basically becomes asynchronous12:03
tkaczynskigoing forward - do you thing it makes a sense to have such module at all (meaning: as a separate service)?12:04
jed56IMHO, we should be a separate servic12:04
tkaczynskifor me it makes a perfect sense to introduce scoring engine concept within a decision engine service, strategies might and will consume it. but I don't see much benefits of having it as a separate service. is it for performance? using the fact that it will run in separate process, so potentially using another CPU core?12:06
tkaczynskiI don't have the real data yet, but it's possible that handling all the communication might actually take more CPU cycles than executing the calculate_score on the scoring engine. it kind of defeats the purpose of having it externally12:08
vincentfrancoisetkaczynski: feels like this is a debate that was already discussed although I can't remember what we concluded last time12:14
tkaczynskiwe concluded that we want to have a separate service :)12:15
jed56I'm searching the logs of the debate :p12:15
tkaczynskiI'm just asking this question again one more time as I am closer to implementation now, have more data and can revert from it :) and since we can't even have threads, I asked myself a question - why is it for?12:16
tkaczynskiand don't take me wrong: I already implemented this service, have it up ad running now :)12:16
tkaczynskijust no communicating ;)12:17
*** openstackgerrit has quit IRC12:19
*** openstackgerrit has joined #openstack-watcher12:19
tkaczynskiso it's not I'm lazy and don't want to do thing. I'm just double checking that we are doing the right thing. but since jed56 helped me with communication problem (I'm able to have sync calls), I don't need to re-implement it as a rest service or something12:19
jed56tkaczynski: if we are wrong we change that :)12:19
jed56this is an incremental process12:20
tkaczynskisure, but there is no guarantee that I will be able to do that once I finish implementation and move to other tasks :) just saying12:21
tkaczynskiso better get it right now and not delay it for future12:22
jed56I understand, IMHO this question is very dependent on the scope of the scoring engine12:23
jed56do we want that the scoring engine is in charge of prefetching the metrics ?12:24
jed56in this case, IMHO a service is better12:24
jed56moreover, we could avoid to restart the decision engine of each new prediction12:26
jed56I agree, for the first version of the scoring engine this a not a huge value to be a service12:27
jed56the good news, this is more easy to remove a service versus create it12:28
jed56tkaczynski: are you going to portland ?12:28
tkaczynskijed56: I'm not sure where is the question about "prefetching the metrics" coming from. scoring engine will be pretty simple and it should be by design.12:29
jed56The querying of the input data from any sources(ceilometer, monasca, kafka rabbitmq,ect) can take lot of time12:30
tkaczynskiit's not fetching any metrics by itself. all this work is done in strategy - the strategy needs to understand the input data and manage it. scoring engine is a representation of the machine learning algorithms or frameworks. It should be treated as a mathematical function, nothing else!12:31
tkaczynskiI fully understand that querying the data might take time, but it is NOT a responsibility of the scoring module12:32
tkaczynskion the high level this will be the integration with scoring engine from strategy code:12:33
tkaczynskiscoring_engine = scoring_engine_factory.get_scoring_engine('model-1')12:34
tkaczynskiresult = scoring_engine.calculate_score('[1, 2, 3, 4, 5, 6]')12:34
tkaczynski... do something with the result (which might be some prediction or something)12:35
jed56yes but this is possible to that in the strategy for scalability12:35
tkaczynskiwhat is not possible?12:35
jed56sorry :)12:36
jed56let's try again :p12:37
tkaczynskiI am all listening :)12:37
tkaczynskijed56: I won't be in Portland :(12:38
tkaczynskijed56: but Susanne will organize the video conference, so I will kind of participate ;)12:39
jed56I think we are agree, this is not a good idea to query the measurements in the strategies and then give it to the scoring engine.12:39
jed56so, who is in charge of that ?12:39
tkaczynskiwell, I agree that querying the measurements is a complex task. but the role of the scoring module is to integrate machine learning with Watcher. single responsibility principle12:41
tkaczynskiI don't have a good answer at the moment how the problem with querying the data should be solved, but this problem is valid for all strategies, whether they use scoring engines or not12:42
jed56we are not doing complex calculation with the metrics in the strategy moreover we will have soon luster-model-objects-wrapper12:43
jed56blueprint implemented12:45
tkaczynskibut cluster model is not covering the usage metrics, right?12:45
jed56tkaczynski: yes12:45
tkaczynskifor me it's a design choice: do we want each strategy requiring the access to the usage metrics to deal with this problem independently or do we want to solve it "globally" on the watcher level, so the code is shared12:47
tkaczynski... and possibly introduce some framework, which can be used by all strategies12:48
jed56IMHO, if we give the measurements to the scoring engine for example 1 week of data this is will required time to be computed bye the engine12:49
tkaczynskiare you saying that you would like the strategy to react (make optimizations) based on the data which is a week old?12:50
tkaczynskitypical scenario for optimization is that the data is analysed on-line (withing seconds or minutes) and then actions are taken, because cloud needs optimization12:52
tkaczynskithe past data might be useful for training the machine learning models of course, but not much for online analysis12:52
jed56we would like that the strategy uses the prediction which is based on 1 week of data12:52
*** jed56 has quit IRC12:55
tkaczynskiI think I know where are you going: you are mixing learning the model with making actual predictions. in machine learning learning the model happens "offline". data scientists collect the data (possibly from a week, a month or even years), they train the model using this data and then the model becomes a scoring engine and starts doing the analysis of the12:56
tkaczynskionline data12:56
tkaczynskiwhat watcher really needs if it wants to be a serious player in openstack optimization: a workflow for collecting and storing the data for future analysis12:57
*** jed56 has joined #openstack-watcher12:57
tkaczynskiand a separate workflow for online data, which will allow for cloud monitoring and optimization12:57
jed56sorry, I was offline12:58
tkaczynskiboth of these workflows are needed on the watcher level. scoring module needs both of them - one for learning, one for analysing. but it's not managing these workflows in any way12:59
*** alexchadin has joined #openstack-watcher13:01
tkaczynskimy recommendation would be: the watcher core members should look into this problem and prioritize it. the strategies will need that sooner rather than later. but honestly I don't think that scoring module is part of that problem. it's a consumer and only in the context of the strategy, because the strategy is the ultimate place where everything starts and13:03
tkaczynskiends. strategy developer must understand what she/he wants to do, what data is needed and what algorithms are required to perform the optimization13:03
tkaczynskiscoring module is a helper module which brings machine learning to the table, so the strategy has one (big on its own) problem less to solve13:04
jed56in this is case the scoring engine didn't need to be a service :)13:05
tkaczynskiI guess the future developer of scoring engine will come and ask: where is the historical usage data for this data center? watcher should have a way to provide that (possibly offline, even on a cd disc!), then the developer can create a great data model solving all problems in the world, predicting the next lotto numbers or whatever and once this is done it13:07
tkaczynskican be delivered as a scoring engine plugin, at which point some strategy might start to use that :)13:07
tkaczynskiand it's quite likely that the online data will be needed as for example the scoring engine might want to detect hot spots in the cloud by analysing the cpu, memory or whatever usage13:08
jed56tkaczynski:  IMHO, you could add a point for portland on that subject13:09
tkaczynskiall I'm trying to say is that this is much bigger problem than the scoring engine itself and needs to be solved!13:09
jed56current scoring engine and beyond13:09
tkaczynskiok, I will13:09
tkaczynskiso now coming back to original question: did we just agree that scoring module doesn't need to be a service? ;)13:10
jed56+2 :)13:11
tkaczynskisounds great, I can throw away my last commit then, only like 30 files ;)13:15
tkaczynskino worries, better this than polluting code base with unnecessary services and abstractions13:20
tkaczynskijed56 vincentfrancoise: thanks for your input, really appreciated!13:21
*** esberglu has joined #openstack-watcher13:23
*** diga has joined #openstack-watcher14:05
*** diga has quit IRC14:31
openstackgerritMerged openstack/puppet-watcher: Added watcher package ensure  https://review.openstack.org/33507214:40
*** alexchadin has quit IRC14:55
*** alexchadin has joined #openstack-watcher15:00
*** alexchad_ has joined #openstack-watcher15:04
*** alexchadin has quit IRC15:04
*** vincentfrancoise has quit IRC15:04
*** danpawlik has quit IRC15:07
*** openstackgerrit has quit IRC15:33
*** openstackgerrit has joined #openstack-watcher15:33
*** jed56 has quit IRC15:35
*** harlowja has joined #openstack-watcher15:40
*** alexchad_ has quit IRC15:50
openstackgerritDaniel Pawlik proposed openstack/puppet-watcher: Implement applier and decision-engine  https://review.openstack.org/33693517:37
*** wootehfoot has joined #openstack-watcher17:40
openstackgerritDaniel Pawlik proposed openstack/puppet-watcher: Implement watcher-db-manage commands  https://review.openstack.org/33693917:50
*** thorst_ has joined #openstack-watcher18:41
*** thorst_ has quit IRC18:41
*** thorst_ has joined #openstack-watcher18:42
openstackgerritDaniel Pawlik proposed openstack/puppet-watcher: Implement watcher-db-manage commands  https://review.openstack.org/33693918:43
openstackgerritDaniel Pawlik proposed openstack/puppet-watcher: Implement applier and decision-engine  https://review.openstack.org/33693519:02
*** dtardivel has quit IRC19:27
*** Zucan has joined #openstack-watcher19:42
*** Zucan has quit IRC19:47
*** Zucan has joined #openstack-watcher19:52
*** Zucan has quit IRC20:23
*** wootehfoot has quit IRC20:39
*** edleafe- is now known as edleafe21:36
openstackgerritTin Lam proposed openstack/watcher: Update docs links to docs.openstack.org  https://review.openstack.org/33372322:28
*** thorst_ has quit IRC22:47
*** thorst has joined #openstack-watcher22:48
*** thorst has quit IRC22:57
*** thorst has joined #openstack-watcher23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!