14:01:45 <boris-42_> #startmeeting Rally 14:01:47 <openstack> Meeting started Mon Dec 7 14:01:45 2015 UTC and is due to finish in 60 minutes. The chair is boris-42_. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:50 <openstack> The meeting name has been set to 'rally' 14:01:59 <ikhudoshyn> o/ 14:02:04 <oanufriev> hi all 14:02:18 <ylobankov> hi 14:03:14 <boris-42_> bochi-michael: hi 14:03:26 <bochi-michael> boris-42_, hi :) 14:04:16 <chenli> hi 14:05:02 <boris-42_> chenli: hi 14:05:06 <boris-42_> yingjun: hi 14:05:36 <yingjun> boris-42_ hi:) 14:05:43 <boris-42_> #topic Today's Agenda 14:05:52 <boris-42_> #link https://wiki.openstack.org/wiki/Meetings/Rally#Agenda 14:06:02 <boris-42_> #topic [ikhudoshyn] Preparation for a Distributed Runner 14:06:15 <ikhudoshyn> Hi everybody 14:06:18 <boris-42_> #link https://review.openstack.org/#/c/251889/ 14:06:22 <boris-42_> ikhudoshyn: hi there 14:07:02 <ikhudoshyn> I'm now working on distributed runner - the addition to Rally that will allow to generate load simultaneously by many nodes 14:07:33 <ikhudoshyn> Before implementing distributed runner itself I expect two preparation steps 14:08:06 <ikhudoshyn> 1. Runner should return chunks (or batches) of TaskResult's rather then individual ones 14:08:43 <ikhudoshyn> 2. Runner should aggregate SLA data and return SLAChecker instances 14:09:03 <ikhudoshyn> by the link https://review.openstack.org/#/c/251889/ there is a patch implementing step #1 14:09:36 <ikhudoshyn> While I'mm working on #2 I'd like to encourage everybody to take a look at it (#1) 14:09:38 <ikhudoshyn> eom 14:09:46 <ikhudoshyn> any questions? 14:10:16 <boris-42_> ikhudoshyn: so we will need to do SLAChecker instances pickable? 14:10:24 <ikhudoshyn> yes 14:10:24 <boris-42_> ikhudoshyn: we are going to send them via network in case of distributed runner? 14:10:32 <ikhudoshyn> yes, exactly 14:11:23 <chenli> ikhudoshyn: how runners talk with rally ? 14:11:45 <ikhudoshyn> right now Rally support only local runners 14:12:08 <boris-42_> ikhudoshyn: I believe you should send link to your spec 14:12:13 <ikhudoshyn> Runner communicates with TaskEngine via queue 14:12:15 <boris-42_> laurelm: to chenli 14:12:31 <ikhudoshyn> boris-42_, sure, I'll find it right now 14:13:10 <chenli> ikhudoshyn: queue ? message queue ? RPC like other projects ? 14:13:14 <ikhudoshyn> https://github.com/openstack/rally/blob/master/doc/specs/in-progress/distributed_runner.rst 14:13:30 <boris-42_> chenli: nope 14:13:36 <ikhudoshyn> chenli, no just plain multiprocessing.Queue 14:13:54 <ikhudoshyn> (or it might be Queue.Queue, I forgot) 14:14:00 <chenli> ikhudoshyn: o.... thanks for the link. 14:14:13 <boris-42_> chenli: it's just Queue 14:16:01 <chenli> it is said : communicates with remote RunnerAgents wia message queue (ZeroMQ) 14:16:19 <boris-42_> chenli: yep 14:16:20 <chenli> so the "ZeroMQ" is not the same as rabbitMQ 14:16:32 <boris-42_> chenli: currently we have only Local case 14:16:52 <boris-42_> chenli: it's just few processes 14:17:04 <chenli> boris-42_: o. 14:17:05 <boris-42_> chenli: that are communicating via shared memory 14:17:18 <boris-42_> chenli: using multiprocessing.Queue 14:17:27 <boris-42_> chenli: which you can see from rally.task.engine module 14:17:52 <chenli> boris-42_: ok 14:19:01 <boris-42_> chenli: so the goal of work that we are working now is to create a distributed Runner 14:19:24 <boris-42_> chenli: it means that we will be able to consume many physical hosts to generate load 14:21:47 <boris-42_> so seems like we have only one item in current agenda 14:21:53 <boris-42_> #topic OpenDiscussion 14:22:06 <boris-42_> somebody has something to discuss? 14:22:09 <ikhudoshyn> there is one more item from my side 14:22:13 <ikhudoshyn> https://review.openstack.org/#/c/253659/ 14:22:50 <ikhudoshyn> I'd like other guys join the discussion 14:23:15 <ikhudoshyn> it's a rather small issuse about coding style 14:23:48 <ikhudoshyn> I expected Tom (the author) to join our meeting but looks like he misunderstood me 14:24:28 <ikhudoshyn> putting the issue here is a lil'bit tedious so please come and review 14:24:50 <ikhudoshyn> eom 14:26:17 <boris-42_> ikhudoshyn: looking at it 14:27:15 <boris-42_> ikhudoshyn: so we already have such places 14:27:34 <boris-42_> ikhudoshyn: and we need to do such changes in backward compatible way 14:27:35 <boris-42_> ikhudoshyn: otherwise we will be break installations 14:27:54 <boris-42_> ikhudoshyn: in any case we need to create some guide about how to write plguins 14:27:57 <boris-42_> with such rules 14:28:24 <ikhudoshyn> boris-42_, i see 14:28:26 <boris-42_> ikhudoshyn: however why not using kwargs word? 14:28:35 <boris-42_> ikhudoshyn: I mean cinder_kwargs, nova_kwargs 14:28:40 <boris-42_> ikhudoshyn: seems ok for me 14:28:50 <ikhudoshyn> well my 'kwargs' point is secondary 14:29:27 <ikhudoshyn> I was against using cinder_kwargs for cinder and just **kwargs for nova 14:30:12 <ikhudoshyn> i.e. call method(cinder_kwargs={'param': 'val'}, nova_kwarg='param') looks ugly for me 14:30:40 <ikhudoshyn> i'd like it method(cinder_kwargs={...}, nova_kwargs={..}) 14:30:57 <chenli> ikhudoshyn: +1 14:30:59 <boris-42_> so you wrote the same=) 14:31:07 <ikhudoshyn> boris-42_, nope 14:31:14 <boris-42_> ah I see nova_kwargs='' 14:31:17 <ikhudoshyn> gimme a sec, i'll put it in detail 14:31:31 <boris-42_> ikhudoshyn: yep that's ugly 14:32:27 <ikhudoshyn> I realize that changing that will break compatibility with possible custom task configs 14:33:12 <boris-42_> ikhudoshyn: no we must not break compatiblity 14:33:13 <ikhudoshyn> yet it's ugly 14:33:24 <boris-42_> ikhudoshyn: so we can add new argument 14:33:24 <ikhudoshyn> boris-42_, I see 14:33:25 <ikhudoshyn> ( 14:33:35 <boris-42_> ikhudoshyn: write deprecation warnigns 14:33:49 <boris-42_> ikhudoshyn: and after few releases remove it 14:34:32 <ikhudoshyn> so we should both add nova_kwargs and keep **kwargs for a while 14:34:42 <ikhudoshyn> and merge the latter two 14:35:03 <ikhudoshyn> that sounds reasonable 14:35:17 <ikhudoshyn> now about '_kwargs' suffix 14:35:42 <ikhudoshyn> this one should get into task configuration 14:36:01 <ikhudoshyn> for me it would look unnatural there 14:36:18 <ikhudoshyn> since it's just python-specific term 14:36:41 <ikhudoshyn> _config or _args looks better 14:36:54 <ikhudoshyn> or _params 14:36:59 <boris-42_> ikhudoshyn: we can make args for example 14:37:00 <ikhudoshyn> like cinder_params 14:37:18 <ikhudoshyn> args are just fine 14:37:31 <ikhudoshyn> cinder_args, nova_args -- look good 14:38:31 <ikhudoshyn> but.. should we have service-specific names or method-specific 14:38:44 <ikhudoshyn> like nova_args or create_vm_args? 14:39:08 <ikhudoshyn> I mean there could be several nova- calls in scenario 14:39:23 <boris-42_> ikhudoshyn: yep params sounds good 14:39:24 <ikhudoshyn> which could require specific params 14:39:41 <boris-42_> ikhudoshyn: we need to standartize this across all plguins in rally 14:39:49 <ikhudoshyn> yes 14:39:52 <boris-42_> ikhudoshyn: could you report a bug? 14:39:57 <ikhudoshyn> thats what I'm talking about 14:40:09 <ikhudoshyn> yes 14:40:16 <ikhudoshyn> will do 14:41:12 <ikhudoshyn> but I guess spec looks more suitable here 14:41:24 <ikhudoshyn> or bug+spec :)) 14:42:08 <boris-42_> ikhudoshyn: ok 14:42:13 <ikhudoshyn> deal 14:42:24 <ikhudoshyn> that's all from my side 14:42:38 <chenli> I have a question, about when to use object and when to use dict 14:42:49 <chenli> I'm just a little confused now. 14:43:11 <ikhudoshyn> chenli, we need some more context 14:43:12 <chenli> We have module objects, but we want all db request return dict. 14:43:42 <chenli> and also a a Struct class in utils. 14:43:58 <rvasilets> o/ 14:44:32 <boris-42_> chenli: so objects must work with rally.common.db.api as a with objects 14:44:42 <boris-42_> chenli: we must not use there sqla models 14:45:32 <chenli> So, objects get return value from rally.common.db.api ==> get dict ? 14:47:05 <boris-42_> chenli: yep 14:47:19 <boris-42_> chenli: that was the goal of your task 14:47:25 <chenli> I can understand the usage of objects, because other projects do that too, such as nova and so on. but they would convert return value from db into the objects, and then use objects in other place. 14:48:17 <chenli> I don't really understand how we call objects.xx then get a dict. 14:48:31 <chenli> looks weird. 14:48:54 <boris-42_> chenli: it's not nova objects 14:49:03 <boris-42_> chenli: it's not even close to nova objects 14:49:35 <chenli> boris-42_: o ? 14:49:55 <boris-42_> chenli: did you look at code of nova objects? 14:50:03 <boris-42_> chenli: did you look at code of rally objects? 14:50:10 <boris-42_> chenli: no matching at all 14:50:29 <chenli> boris-42_: a little, it just call db api, and do nothing else. 14:50:31 <boris-42_> chenli: even in use cases 14:50:55 <boris-42_> chenli: in case of rally objects you are doing extra work (called business logic) in objects 14:51:18 <boris-42_> chenli: that doesn't belong to DB layer but that is used many time across the code 14:51:31 <boris-42_> chenli: that is why we have rally.objects 14:52:02 <boris-42_> chenli: rally.common.db.api should execute the DB queries 14:52:03 <boris-42_> chenli: and return dicts 14:52:23 <boris-42_> chenli: rally.objects will work with this dicts 14:52:59 <chenli> boris-42_: it should return dict too ? 14:53:50 <boris-42_> chenli: nope 14:53:58 <boris-42_> chenli: it should return what it returns 14:54:39 <boris-42_> chenli: so there are just few places in code where "sqla" models are used outside of db.api and we should remove it 14:54:56 <boris-42_> chenli: I am not sure what is unclear here? 14:55:06 <chenli> boris-42_: o, this is link to another question. cli module call api module, api call objects 14:56:25 <chenli> so if objects return a class created by objects, then cli module should not use it 14:56:39 <chenli> boris-42_: right ? 14:57:00 <boris-42_> chenli: it the same topic 14:57:33 <boris-42_> chenli: I said that rally.cli module shouldn't use rally.common.db.api directly and should use rally.common.objects insetad 14:57:54 <boris-42_> chenli: the another thing is rally.common.db.api should return plain dicts instead of sqlamodels 14:58:34 <boris-42_> oh no objects 14:58:35 <boris-42_> sorry 14:58:46 <boris-42_> rally.cli -> rally.api -> rally.bojects 14:58:55 <boris-42_> where rally.cli is on one host 14:59:09 <boris-42_> rally.api and rally.boject and rally.db on another 14:59:13 <boris-42_> this is required to create rally as a service 14:59:26 <boris-42_> okay we have to end meeting 14:59:29 <boris-42_> #endmeeting