08:00:28 <eranrom> #startmeeting storlets
08:00:29 <openstack> Meeting started Tue Mar 14 08:00:28 2017 UTC and is due to finish in 60 minutes.  The chair is eranrom. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:00:33 <openstack> The meeting name has been set to 'storlets'
08:00:35 <kota_> hey-yo
08:00:40 <sagara> hi
08:00:43 <eranrom> Hi
08:01:31 <eranrom> Let's start.
08:01:42 <akihito> Hi
08:02:20 <eranrom> akihito: Hi
08:02:28 <eranrom> I did not add anything to the agenda
08:02:51 <eranrom> There is something on stopping unused containers
08:02:59 <eranrom> which I am looking at
08:03:04 <takashi> hi
08:03:11 <kota_> takashi: !!!!
08:03:14 <akihito> takashi: Hi!
08:03:40 <kota_> we missed you takashi at the PTG
08:03:43 <takashi> all: hi
08:04:11 <takashi> kota_: sorry about that :-( I'll check your great discussion result...
08:04:54 <eranrom> takashi: Hi!
08:05:00 <takashi> There are no big topics from my side today. I think I should review some of the patches about agent refactoring
08:05:05 <takashi> eranrom: o/
08:05:22 <eranrom> takashi: great to see you here!
08:05:38 <takashi> eranrom: me too :-)
08:05:52 <takashi> it would be great if I can ask some review about https://review.openstack.org/#/c/426586/
08:05:58 <takashi> especially from kota_ :-P
08:06:09 <kota_> what's that?
08:06:23 <takashi> about range handling
08:06:28 <eranrom> #topic: Patches priority reviews
08:06:35 <kota_> oic
08:06:37 <takashi> eranrom: thx
08:07:05 <eranrom> :-)
08:07:14 <takashi> I'm afraid this is very affective bug, so I'll backport it to stable later.
08:07:18 <takashi> after we land the fix
08:07:35 <takashi> I think we also need to backport the patch to fix breaking unit test to stable
08:07:40 <eranrom> takashi: yep, good point.
08:08:06 <eranrom> takashi: I think I did that.
08:08:07 <takashi> which is submitted by kota_ and already merged to master
08:08:22 <takashi> eranrom: I'm afraid that I overlooked that. thx!
08:08:26 <kota_> if no big topic here, can we look at the patch together?
08:08:47 <kota_> what's the difference after the version i reviewed
08:08:48 <kota_> ?
08:09:07 <kota_> sorry, I have to re-install the change in my head :/
08:09:22 <akihito> I have 'stopping unused containers' topic.
08:09:27 <takashi> kota_: added some unit tests to assert my change
08:09:31 <sagara> I have resource limiting topic, but now I am writing that to etherpad.
08:09:47 <sagara> #link: https://etherpad.openstack.org/p/storlets-rate-resource-limiting-design
08:10:06 <takashi> kota_: in 	tests/unit/swift_middleware/handlers/test_obj.py and tests/unit/swift_middleware/handlers/test_proxy.py
08:10:13 <eranrom> ok lets spend few minutes on the patch, and then proceed with Sagara's and Akihito's topics
08:10:19 <takashi> eranrom: +1
08:10:24 <sagara> thanks
08:10:45 <akihito> Thank you! I wrote my topic in etherpad.
08:10:49 <akihito> <<https://etherpad.openstack.org/p/storlets-pike-design-summit>>
08:10:53 <akihito> No. 10 - 12
08:11:03 <eranrom> kota_: takashi: we want to go over this one https://review.openstack.org/#/c/426586/ right?
08:11:14 <kota_> takashi: quick question, current *file* object range shouldbe [start, end] but http range [start, end), right?
08:11:19 <eranrom> akihito: yes, I saw good topic!
08:11:21 <kota_> eranrom: yes
08:11:26 <takashi> kota_: yes
08:11:49 <kota_> k and the middleware (which in http layer) resolves (translates) to file
08:11:51 <kota_> make sense.
08:13:41 <kota_> takashi: can you make sure https://review.openstack.org/#/c/426586/2..4/tests/unit/swift_middleware/handlers/test_obj.py@258
08:13:41 <kota_> ?
08:14:02 <kota_> the http range calls 1-6 there
08:14:14 <takashi> kota_: oh, sorry
08:14:22 <kota_> and http range should be [1, 6) so that 1, 2, 3, 4, 5
08:14:22 <takashi> In http rfc, range should include end
08:14:38 <takashi> so http deals range as [start, end]
08:14:45 <kota_> that means, opposite?
08:14:52 <takashi> kota_: yes
08:15:00 <takashi> current file object deals range as [start, end)
08:15:08 <takashi> so we need some translation between them
08:15:24 <takashi> more precisely, current python file object
08:15:50 <kota_> ah, ok, the docs in middleware says, general range as (include start, and exclude end)
08:16:37 <takashi> kota_: I believe that is general range diffinition :-P
08:16:52 <takashi> definition
08:19:41 <kota_> done
08:20:03 <eranrom> kota_: Thanks!
08:20:22 <takashi> kota_: thx!
08:20:25 <eranrom> anything else on reviews?
08:20:38 <kota_> I still like clear cleanup for convert http range <-> general range (class or functions)
08:20:40 <kota_> though
08:21:10 <eranrom> kota_: My understanding is that the translation between http and files requires adding +1 to the end.
08:21:26 <kota_> eranrom: correct
08:21:42 <eranrom> previously that +1 was in the SBus layer, and Takashi changes that to happen in the middleware level
08:22:20 <takashi> eranrom: previously that +1 was done in Java SDaemon
08:22:35 <takashi> and not done in Python SDaemon (storlet-daemon)
08:22:35 <eranrom> oh, even worse...
08:22:47 <eranrom> takashi: right!
08:22:47 <takashi> with my patch, +1 is done in gateway side
08:22:49 <kota_> takashi: that means, Java SDaemon is now +2 for the range?
08:22:59 <takashi> kota_: I removed that +1 in my patch
08:23:03 <kota_> ok
08:23:27 <takashi> all range execution case python/java proxy/object is now tested in functional tests
08:24:55 <eranrom> anything else on ramges?
08:24:58 <eranrom> ranges?
08:25:19 <takashi> nothing more from me
08:25:34 <takashi> I'll submit backport patch to stable/ocata later, and ping you after that
08:26:18 <eranrom> takashi: sure. thanks!
08:26:48 <eranrom> anything else on patches before we continue?
08:29:09 <kota_> please go ahead eranrom
08:29:16 <takashi> let's move to the next topic
08:29:55 <eranrom> Since I have already read, lets continue with Akihito's topic.
08:30:18 <eranrom> #topic: Stopping unused containers / storlet demons
08:30:21 <akihito> Thank you!
08:30:42 <eranrom> I like the suggestion, I have two comments so far:
08:31:33 <eranrom> 1. We need to add a configurable timeouts for all 3 (I think you mention a timeout only for the third, which is the storlet app)
08:32:34 <eranrom> 2. I would like to understand how you plan to enforce the storlet app timeout. for the factory and storlet daemon it seems pretty streight forward
08:33:19 <eranrom> sorry one more comment:
08:33:55 <eranrom> Regarding file deletions. Have we agreed on having a storlet create a file?
08:35:20 <kota_> sorry, I'm missing the etherpad which we reffers right now
08:35:33 <akihito> eranrom: Thank you comments. first. Why we need to set Timeout for daemon_factory and agent process?
08:35:34 <eranrom> https://etherpad.openstack.org/p/storlets-pike-design-summit
08:35:52 <akihito> No. 10-12 :-)
08:36:19 <kota_> akihito: ah, you added to that one, ok thx
08:36:47 <kota_> please continue
08:36:50 <eranrom> akihito: because when we, e.g. start the container, and hence the daemon_factory there may be some time passing before any storlet_daemon is started
08:37:47 <akihito> Oh. I understand! OK. fix it.
08:37:49 <eranrom> similarly, there may pass some time after we start a storlet_daemon and before we get a storlet request
08:39:00 <akihito> for No2 comments. I want to add timeout for storlet app in agent_process.
08:39:10 <akihito> https:/In /github.com/openstack/storlets/blob/master/storlets/agent/daemon/manager.py#L223-L224
08:39:29 <akihito> sorry <<https://github.com/openstack/storlets/blob/master/storlets/agent/daemon/manager.py#L223-L224>>
08:40:24 <akihito> refer to <<https://review.openstack.org/#/c/410296/5/storlets/agent/common/server.py>>
08:40:49 <akihito> oh. sorry..
08:40:57 <akihito> https://review.openstack.org/#/c/410296/5/storlets/agent/common/server.py
08:43:15 <eranrom> Making a suggestion:
08:43:56 <eranrom> sorry..
08:44:23 <kota_> np
08:44:56 <eranrom> How about writing comments on the etherpads, and use the time now for questions.
08:45:14 <kota_> i support that idea
08:45:28 <eranrom> akihito: I will comment on the Etherpad regarding your answer to comment 2.
08:46:59 <akihito> eranrom: I see. Thank you.
08:49:11 <kota_> eranrom: on the question for you
08:49:27 <eranrom> kota_: yep
08:49:39 <kota_> eranrom: meaning "should we support that storlet app can write some files into the local in the container"
08:49:51 <kota_> eranrom: I strongly recommend to support that option
08:50:15 <eranrom> kota_: right, we started this discussion. and I think takashi also wanted to ask on this.
08:50:19 <kota_> eranrom: I know, we *don't* want such in-effective use case for our storlet streaming apps
08:51:15 <kota_> eranrom: however, from my experience with my custormers, absolutely we will have a clif (might be wall) when translating leagcy app to the storlet one.
08:51:55 <kota_> i'm calling *legacy* app is something like which can run on file system
08:52:09 <eranrom> kota_: I see.
08:52:50 <kota_> eranrom: i can manage the translation with my custormer but almost case for our poc starts with just running legacy app in the storlet container
08:53:03 <kota_> that is enabled to write the files to read
08:53:21 <eranrom> ffmpeg?
08:53:34 <kota_> no?
08:53:56 <kota_> i have certain cases for now,
08:54:05 <kota_> not only oss
08:54:11 <eranrom> kota_: I was asking because I had the same problem with ffmpeg for which I found a solution
08:54:20 <eranrom> kota_: got it
08:54:54 <eranrom> ok, note that cleaning up those files is as simple as restarting the container (and deleting the older instance)
08:55:14 <kota_> eranrom: maybe it's one of them. espically, a kind of hard-coded 'path-forced' input format.
08:55:29 <eranrom> kota_: yep
08:55:29 <kota_> like "progam -i file_to_path"
08:55:38 <eranrom> right
08:56:07 <eranrom> Will you be interested to setup another time this week to discuss the comments for Sagara's and Akihito's suggestions?
08:56:17 <eranrom> we can read it all and comment by then
08:56:28 <eranrom> we can meet in #openstack-storlets
08:56:34 <kota_> +1
08:56:49 <sagara> yes, I would like you to review resource limiting design.
08:57:02 <akihito> OK.
08:57:23 <eranrom> Tomorrow same time?
08:58:07 <kota_> i could have
08:58:44 <takashi> I'll join if possible
08:59:17 <takashi> I had some local discussions about the topics with sagara-san and akihito, so I think you can have the discussion without me.
08:59:18 <sagara> just a wait
08:59:56 <eranrom> takashi: ok, sure!
09:00:11 <sagara> I can join tomorrow meeting at same time.
09:00:18 <eranrom> sagara: greate thanks
09:00:22 <eranrom> I must end here.
09:00:32 <sagara> eranrom: thanks
09:00:34 <eranrom> Lets go to #openstack-storlets
09:00:37 <eranrom> #endmeeting