15:01:24 <alaski> #startmeeting Scheduler
15:01:25 <openstack> Meeting started Tue Oct 15 15:01:24 2013 UTC and is due to finish in 60 minutes.  The chair is alaski. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:28 <openstack> The meeting name has been set to 'scheduler'
15:01:43 <alaski> hi
15:01:50 <PaulMurray> hi
15:02:15 <alaski> So we don't have an official agenda today since I'm subbing in
15:02:26 <alaski> But we were going to continue the discussion from last week about summit sessions
15:02:33 <MikeSpreitzer> We have some discussion queued up from the ML
15:02:39 <MikeSpreitzer> that too
15:02:42 <glikson> hi
15:02:58 <alaski> yes, I figured we could get to ML after a summit discussion
15:03:03 <MikeSpreitzer> OK
15:03:13 <PaulMurray> I think we were going to get back to scheduler session too - if @PhilDay is here
15:03:26 <PhilD> Hi Folks
15:03:33 <PaulMurray> here he is
15:03:38 <alaski> Is https://etherpad.openstack.org/p/IceHouse-Nova-Scheduler-Sessions erroring for anyone else?
15:03:53 <alaski> #topic Summit sessions
15:04:08 <PhilD> I had trouble with it in Chrome - but could get to it in IE
15:04:20 <shanewang1> me too
15:04:21 <PhilD> I assume some problem with the Etherpad changes
15:04:25 <PaulMurray> Yes - it is for me @alaski
15:05:02 <alaski> ok, works in FF but not Chrome for me
15:05:21 <alaski> PhilD: do you want to recap your suggestion for sessions?
15:05:21 <glikson> works for me in Chrome..
15:05:29 <toan-tran> it works for me in chrome, version 30.0.1599.6
15:05:40 <MikeSpreitzer> As for content, I like Phil's revised trio
15:05:51 <MikeSpreitzer> for the case where we get only 3 sessions
15:05:58 <PhilD> Sure.   We have three guaranteed slots from Russell - so I was trying to make sure that we cover all of the major topics
15:06:20 <PhilD> Group scheduling,  Making other Metrics available, and performance
15:06:43 <PhilD> And then I tried to map teh proposed sessions (from both the EP and submitted) into those
15:07:08 <PhilD> Some other things will need to take thier chance against the general pool of sessions
15:07:24 <alaski> ok, so smart resource placement and instance group model would be combined?
15:07:46 <MikeSpreitzer> yes
15:07:52 <PhilD> I think the best we can do is say to RB "If there are only three slots - please cut it this way" - if tehre is an extra slot, please split this one up
15:07:56 <Yathi> I don't like the idea of combining
15:08:06 <russellb> alaski volunteered to help with building the schedule, so if he understands the priorities of this group, we should be set :)
15:08:09 <Yathi> It is a lot of big topics combined into a single session
15:08:34 <russellb> i'm pulling in some folks to help with some of the hard decisions ...
15:08:37 <PhilD> @alaski - its not great I agree,  but the alternatiove is that we don;t proivde any steer, and they they get combined in some other way instead
15:08:39 <MikeSpreitzer> Like Phil said, our top ask would be to split that session
15:08:42 <russellb> btw, i really appreciate the hard work you guys are doing with planning this stuff
15:09:02 <PhilD> Thanks Russell
15:09:04 <alaski> russellb: thanks
15:09:44 <PhilD> What surprised me was that Boris doesn't seem to have submitted a performance session yet ?
15:09:58 <alaski> yeah, I'm a little surprised too
15:10:06 <shanewang1> probably he forgot
15:10:08 <alaski> I'll try to ping him about that
15:10:15 <alaski> boris-42: are you here?
15:10:26 <boris-42> alaski always here=)
15:10:32 <alaski> boris-42: :)
15:10:53 <alaski> are you going to submit a session around scheduler performance, or should someone else?
15:11:12 <boris-42> alaski we almost finish work around refactoring scheduler
15:11:25 <boris-42> alaski and yes I will get results before & after
15:11:45 <russellb> boris-42: but a design summit session, did you plan to submit one?
15:12:02 <boris-42> russellb there is already for scheduler
15:12:16 <Yathi> PhilD: Debo and I felt combining Group APIs, Smart Resource Placement, Heat and Scheduling into a single session, it is going to be hard to give justice to everything..
15:12:24 <boris-42> russellb it seems that we have at least 3-4 concepts =)
15:12:27 <russellb> which, this one?  http://summit.openstack.org/cfp/details/34
15:12:34 <russellb> boris-42: yep
15:12:41 <MikeSpreitzer> oh
15:12:45 <alaski> Yathi: I think it would just be group apis and smart resource placement
15:12:50 <boris-42> russelbb I don't thing that this is right session
15:13:02 <russellb> boris-42: which session are you referring to then?
15:13:13 <boris-42> russellb wait a sec i will found
15:13:15 <russellb> ok
15:13:20 <boris-42> find*
15:13:40 <boris-42> russellb omg
15:14:17 <PhilD> @Yathi -  I underdstand the concern, really I do, but as always we have more material / ideas / proposed sessions than slots - so whatever way we cut this will be a compromise.
15:14:22 <boris-42> russellb oh I thought that there is a session
15:14:25 <boris-42> grayk ping
15:14:25 <glikson> I tend to agree with Yathi that having an in-depth discussion on all of that would be a challenge. ideally, we can narrow-down the scope to something that we actually plan to implement in Icehouse timeframe..
15:14:27 <boris-42> garyk ping
15:14:45 <alaski> boris-42: garyk isn't here today
15:14:53 <boris-42> alaski omg
15:14:59 <boris-42> =)
15:15:07 <boris-42> russellb there is a big thread in mailing list
15:15:18 <alaski> boris-42: do you want to propose a session for this?
15:15:22 <Yathi> Smart resource placement has a lot of people interested, and there might be even something to demo, using cross services theme
15:15:24 <boris-42> alaski yes
15:15:32 <boris-42> alaski we could discuss all approaches
15:15:33 <russellb> boris-42: right ... well these guys have been working hard trying to organize schedulre sessions, so that's why they were wondering
15:15:49 <MikeSpreitzer> Yathi: I would be happy to demo what my group has done
15:15:53 <boris-42> russellb I would like to be on their session imho
15:15:55 <alaski> boris-42: cool.  Can you do it today or tomorrow?
15:15:57 <PhilD> So which of teh other 2 sessions would you drop, scheduler performance or genearlised metrics / ceilometer ?
15:16:15 <boris-42> alaski russellb I will add just a session and link to that thread?)
15:16:31 <PhilD> Both of those topics also have a lot of interest, and a lot to discuss as well
15:16:37 <glikson> BTW, I also plan to submit a proposal on adding support for multiple scheduler configurations, as indicated in the Etherpad, to discuss the dilemas identified during the work on the blueprint in Havana
15:16:48 <alaski> boris-42: definitely link the thread if it has good information.  But please also summarize it in the proposal
15:16:50 <Yathi> MikeSpreitzer:  Same here, I will be happy to demo too, what our group is doing in this area.
15:17:17 <glikson> probably doesn't fit into either of the 3 themes
15:17:19 <boris-42> alaski Okay will do today but little bit later
15:17:30 <alaski> boris-42: that works, thanks
15:17:44 <PhilD> @glikson - that's fine, I think there can eb other sessiosn that take thier chance against all of th eother porpoosals.  What we're trying to do here is work out what the three "must have" scheduler sessions are
15:17:46 <boris-42> alaski russellb are you interested in benchmark results?
15:17:57 <boris-42> alaski russellb I am testing OpenStack agaisnt 1k servers
15:18:02 <glikson> PhilD: sure, makes sense
15:18:13 <MikeSpreitzer> With Boris submitting a session, PhilD's revised trio still looks good, right?
15:18:29 <alaski> boris-42: benchmarks are good.  everyone likes real numbers
15:18:36 <PhilD> Yep - I'd assumed that Boris would submit a session;-)
15:18:44 <boris-42> alaski actually I don't want to speak about numbers
15:18:52 <MikeSpreitzer> ?
15:18:54 <alaski> MikeSpreitzer: makes sense to me
15:19:08 <boris-42> alaski I would like to speak about how to get such results on your PC or small amount of servers using Rally
15:19:09 <boris-42> =)
15:19:24 <boris-42> alaski https://wiki.openstack.org/wiki/Rally
15:19:55 <MikeSpreitzer> oh
15:19:58 <alaski> even if you don't spend a lot of time on numbers it would be good to hear what you saw
15:20:36 <shanewang1> yes, we all are interested in that.
15:20:41 <PhilD> One thing we have to try and get smart on is how to do the cross team sessions (Nova/Heat), Nova/Celiometer, etc
15:21:05 <Yathi> are these cross team sessions - separate from the three you are trying to pick here ?
15:21:07 <MikeSpreitzer> boris-42: I think we were anticipating a session on how to improve scheduling, not how to measure the improvement
15:21:19 <PhilD> Feels like those are always hard to schedule - and generally slow because both groups have to come up to speed.
15:21:52 <alaski> MikeSpreitzer: agreed.  That's what I'm expecting
15:22:17 <PhilD> I'm asking the question / trawling for ideas really on what we can do to make those effective.
15:22:40 <boris-42> MikeSpretizer I am speaking about another one
15:22:41 <shanewang1> MikeSpreitzer: better have some numbers to compare those methods on how to improve scheduling
15:22:55 <alaski> PhilD: My thinking has been that we should start with what needs to be done in each project, and limit the cross team concerns
15:23:03 <MikeSpreitzer> PhilD: cross anything is tough, you first have to spend more time than you expected learning each other's vocabulary and way of thinking
15:23:04 <boris-42> MikeSpreitzer scheduler is one thing, OpenStack benchmark system another
15:23:33 <MikeSpreitzer> boris-42: OK, two sessions, got it
15:23:42 <MikeSpreitzer> Yes, I definitely like measuring!
15:23:56 <boris-42> MikeSpreitzer yeah we have to simplify it
15:24:18 <MikeSpreitzer> boris-42: now I'm lost, simplify what?
15:24:33 <PhilD> Ok - so then we can exclude the Heat aspects from the Group scheduler session and the Celiometer aspects from the metrics one ?
15:24:48 <PaulMurray> boris-42 do you mean benchmark system to be in scheduling sessions or elsewhere?
15:24:48 <boris-42> MikeSpreitzer simplify benchmarking of openstack deployments.  Then we will be able to start continues working around performance
15:24:57 <boris-42> PaulMurray NOO
15:24:59 <boris-42> =)
15:25:05 <boris-42> PaulMurray I will bring only numbers=)
15:25:13 <MikeSpreitzer> boris-42: understand remark about simplification; thanks
15:25:23 <PaulMurray> boris-42 thanks :)
15:25:36 <alaski> PhilD: I think that makes sense if we're trimming to the minimum possible
15:26:09 <alaski> So just to summarize, we're looking at "Group scheduling,  Making other Metrics available, and performance"
15:26:23 <alaski> group sheduling would be instance groups and smart resource placement
15:26:31 <alaski> boris-42: will open a performance session
15:26:33 <MikeSpreitzer> I'm not sure I understand the remark about trimming #2
15:26:35 <PhilD> Yep - that would be my core trio
15:27:00 <MikeSpreitzer> What is the proposed in scope, what is out, for #2 ?
15:27:05 <boris-42> alaski so should I make 2 session?
15:27:08 <boris-42> sessions*
15:27:39 <PhilD> And if there is a chance of a 4th core scheduler session we split group scheduling into "API" and "scheduling implementation" - rather than pick a 4th topic
15:27:40 <alaski> boris-42: it sounds like it.  We want a session on improving scheduler performance, with numbers.  And then you can open a session on Rally
15:27:55 <boris-42> alaski Nice thanks
15:28:01 <shanewang1> is "making other metrics available"   Generic  Scheduler Metrics and  Celiometer?
15:28:02 <Yathi> Ok.  if there are extra sessions available it will be good to see the API discussion separately
15:28:13 <MikeSpreitzer> Yathi: agreed
15:28:15 <Yathi> PhilD: that will be good
15:28:22 <MikeSpreitzer> In fact, quite desired
15:28:27 <MikeSpreitzer> I hope alaski pushes hard for it
15:29:23 <alaski> I'll be ready to go with that.  But I haven't seen how many sessions are competing for slots right now
15:29:37 <alaski> I would like more than 3 scheduling sessions if possible
15:29:48 <Yathi> Smart resource placement topic is too big involving several areas, and also covers cross-services. I hope we have a good start with this during the session allotted
15:30:15 <PhilD> @shenewangs1 yes - although it sounds as if were agreed to go with focusing on what can be done inside Nova first, rather than risk a cross-team session
15:30:39 <PhilD> Maybe we should go for a two week summit next time ;-)
15:30:43 <alaski> Keep in mind that what we really need to cover is work that can be expected to be done in IceHouse, which may help limit scope
15:30:48 <shanewang1> @Phil thanks.
15:31:30 <russellb> alaski: +1 :-)
15:31:42 <Yathi> alaski: Agreed.  Our discussions so far in the ML and in scheduler meeting, is to start simple.. get somethign that works within Nova first
15:31:52 <glikson> maybe we can try arranging additional 'unconference' sessions to deep-dive into the more complicated topics..
15:31:57 <shanewang1> @Phil Maybe we should have nova summit, ceilometer summit:)
15:32:27 <alaski> Yathi: cool, that seems the best approach
15:32:34 <MikeSpreitzer1> Right.  But with an eye towards going holistic.
15:32:47 <Yathi> I am thinking these 'unconference' sessions will be where we will get to demo our initial progress
15:33:05 <shanewang1> @glikson I like that idea
15:33:05 <MikeSpreitzer1> But they conflict with official sessions, right?
15:33:28 <glikson> MS: yes
15:33:44 <Yathi> It is my first summit.. don't know how they work
15:33:52 <MikeSpreitzer1> sorry, networking glitch, I missed what shanewang1 likes from glikson
15:33:58 <alaski> So are we pretty much settled and ready to move on?
15:34:15 <MikeSpreitzer1> what did glikson say that s. likes?
15:34:24 <shane-wang> "unconference"
15:34:33 <MikeSpreitzer1> thanks
15:34:43 <russellb> if you want to grab space there, do it early
15:34:47 <russellb> it fills up fast
15:34:47 <MikeSpreitzer1> my first summit too.  Will stop whining now.
15:35:03 <shane-wang> my second:)
15:35:32 <Yathi> how long do the official sessions last ?
15:35:41 <shane-wang> 20-30min?
15:35:52 <alaski> Yathi: 50-60 min I think
15:36:18 <MikeSpreitzer1> How does one get an unconference slot?
15:36:21 <Yathi> scheduler sessions will all get 50-60 mins ?
15:36:22 <shane-wang> :-X
15:36:49 <alaski> Yathi: each session will get that time, yes
15:37:12 <glikson> could also make sense to try agreeing on acceptable time slots for the unconference before the summit, among the 'core' interested folks (maybe using doodle)
15:38:04 <Yathi> We have to make sure we get interested people to attend the interesting 'unconference' sessions
15:38:23 <MikeSpreitzer1> sounds like an interesting idea
15:39:00 <glikson> probably would need to happen during the last week before the summit, after the design tracks agenda is finalized
15:39:32 <Yathi> how do you reserve a 'unconference' session? Is it on the spot ?
15:39:41 <glikson> Yathi: yes, a whiteboard
15:39:47 <shane-wang> do we have enough rooms for "unconference", and can two scheduler "unconference"s happen at one time?
15:39:53 <MikeSpreitzer1> when does the whiteboard open?
15:40:12 <MikeSpreitzer1> I would not want two competing unconferences for us
15:40:17 <glikson> MS: when the summit begins
15:40:42 <alaski> So why don't we come back to this after some/all of the sessions have been scheduled.  Then we can see what might need to go into an unconference
15:40:43 <shane-wang> omg
15:40:55 <MikeSpreitzer1> alaski: sounds good to me
15:41:01 <glikson> russellb: do you know how many 'unconference' rooms will be available? one or more?
15:41:08 <MikeSpreitzer1> shane-wang: ?
15:41:09 <shane-wang> alaski: agree
15:41:23 <shane-wang> omg "when the summit begins"
15:41:31 <MikeSpreitzer1> ?
15:41:45 <glikson> alaski: sure. probably would make more sense to have follow-ups rather than completely new topics
15:41:49 <alaski> MikeSpreitzer1: how would you title the current ML discussion?  I'm not exactly sure where it left off
15:41:58 <shane-wang> I bet a lot of competition
15:42:24 <russellb> glikson: not sure
15:42:37 <russellb> have to run for lunch, ttyl guys!
15:42:41 <MikeSpreitzer1> Well, there have been a few topics in ML.  I am most directly involved in discussion of API (including model) for policy to inform joint decision-making
15:43:04 <MikeSpreitzer1> (at first just joint in nova)
15:43:24 <alaski> #topic APIs for Smart Resource Placement
15:43:51 <MikeSpreitzer1> this is step 1 on garyk's three step roadmap; later steps include supporting lower level APIs and implementation.
15:44:15 <MikeSpreitzer1> I think we are pretty converged on the model
15:44:31 <MikeSpreitzer1> Biggest open question is whether Policy gets its own lifecycle...
15:44:44 <alaski> MikeSpreitzer1: I just want to clarify that the reason to target Nova first, or any project really is just to get some work done.  This topic has come up before but never really got beyond discussion, except for the current Instance Group work
15:44:48 <MikeSpreitzer1> I do not think it ruins anything, I just think it's an unnecessary nuisance.
15:44:59 <MikeSpreitzer1> alaski: +1
15:45:19 <MikeSpreitzer1> I am willing to live with Policy having its own lifecycle, if that's what it takes to get an agreement.
15:45:46 <MikeSpreitzer1> The other question about Policy is scope.  That's really an airy thing, at most a matter of naming, at this point.
15:46:45 <Yathi> It is the idea of a Policy model applying outside the scope of an INstance group
15:46:56 <Yathi> Hence the suggestion for a Policy to have its own lifecycle
15:47:21 <Yathi> Not sure if we want to start that discussion here.
15:47:22 <MikeSpreitzer1> yes
15:47:37 <Yathi> In terms of the ML discussions for identifying
15:47:41 <MikeSpreitzer1> Well, does anybody have anything to add?
15:47:54 <alaski> this is a good place for it.  we can hash things out more quickly than on the ML
15:47:55 <Yathi> 'unconference' topics - are we do that here ?
15:48:28 <alaski> Yathi: I think we should see what scheduler sessions are accepted and then look at the unconference
15:48:31 <Yathi> alaski: agreed.. the last 3 - 4 scheduler meetings we have made good discussions, and I really liked it
15:48:59 <Yathi> alaski: ok
15:49:20 <MikeSpreitzer1> So, anything to add to the ML discussion about Policy objects?
15:49:41 <toan-tran> hi
15:49:42 <Yathi> so do you like the latest update to the model diagram
15:49:48 <toan-tran> I'm new to th group
15:50:00 <toan-tran> I've read the desc of policy
15:50:12 <toan-tran> i agee with yathi
15:50:25 <toan-tran> that it's indepdent and can have its own cycle
15:50:38 <Yathi> Like I expressed in the email,  let's limit the INstanceGroupPolicy to have only a reference to a policy that is persisted separately and with a lifecycle of its own
15:51:00 <Yathi> this common "Policy" object can be used by other InstanceGroups if applicable..
15:51:14 <toan-tran> Yathi: +1
15:51:35 <MikeSpreitzer1> Yathi: I understand your argument, and can live with your revised model.  If nobody has any additional considerations, I move for unanimous consent on that model.
15:52:09 <Yathi> alaski: do you have any comments ?
15:52:15 <MikeSpreitzer1> (except that maybe there is a little redundancy that I mentioned in private email)
15:52:49 <alaski> Unfortunately other work considerations have kept me from completely understanding the ML discussion right now
15:53:00 <MikeSpreitzer1> (both policy_uuid and policy_id field in reference object)
15:53:04 <alaski> so I'm okay with the general consensus
15:53:34 <Yathi> MikeSpreitzer1: I think we have a general consensus
15:54:13 <Yathi> regarding the other objects in the API model - I hope they look okay too?
15:54:48 <glikson> Yathi, MS: do we have a clear view of instance groups may evolve to include things other than servers?
15:54:50 <MikeSpreitzer1> Yes, the rest of the model is OK.  I sent a note about generalizing attachment from groups to allow resources, don't know if anybody noticed or cared.
15:55:14 <alaski> glikson: great question, I'm also interested to understand that
15:55:16 <MikeSpreitzer1> glikson: I think it's easy, just exapnd the set of resources that can appear in a group
15:55:18 <Yathi> yeah the instance like I mention in the document, can refer to any resource
15:55:41 <glikson> ..only that no other resources exist in Nova DB :-)
15:55:45 <Yathi> an InstanceGroupMember has a member_id and this id can refer to any resource's UUID
15:55:56 <Yathi> glikson: we are starting within Nova
15:56:05 <MikeSpreitzer1> glikson: right, expanding kinds of resources allowed is connected to later generalizing beyond nova
15:56:08 <Yathi> but we expect to have a global state repository -
15:56:12 <alaski> Is there a path to breaking this out of Nova though
15:56:28 <Yathi> this is part of the Smart Resource Placement bigger idea
15:56:30 <MikeSpreitzer1> ooh, good question
15:56:36 <Yathi> but starting within Nova first
15:56:58 <MikeSpreitzer1> I think it can be relatively easy to evolve the API, it will be harder with the implementation.
15:57:06 <glikson> yeah, so the question is whether merging all the schedulers into one is the only way going forward.. I was wondering whether a more distributed approach would make sense.
15:57:29 <MikeSpreitzer1> Nobody is proposing to change the existing schedulers, I think
15:57:32 <toan-tran> one small question: where does user add Connection & Metadata to a group?
15:57:34 <Yathi> glikson: we have not fully come to that conclusion yet
15:57:45 <alaski> yeah, the API is what i'm interested in.  What does that need to look like as this moves out of Nova
15:57:46 <Yathi> also, the existing schedulers will continue to work as is
15:57:47 <toan-tran> I saw POST instance, but not connection
15:57:47 <MikeSpreitzer1> I note that the existing schedulers can all (more or less) be told a decision already was made by client
15:58:10 <glikson> Yathi: I don't see how the two can co-exist.. can you elaborate?
15:58:18 <alaski> MikeSpreitzer1: sort of.  They can, but there are no guarantees that it will listen
15:58:30 <alaski> s/it/they/
15:58:40 <MikeSpreitzer1> alaski: I see making that fully true as a relatively small and easy evolution
15:59:19 <Yathi> glikson: no concrete thoughts yet, but the idea is for a pluggable scheduler, depending on the current request, hence the existing schedulers can exist
15:59:34 <MikeSpreitzer1> alaski: I mean, let us (when we add holistic scheduling) make the individual existing schedulers all willing to really accept a decision from the client
15:59:41 <Yathi> the POC code I pushed for a smart solver-based scheduler is a new driver that works within Nova
16:00:01 <alaski> MikeSpreitzer1: agreed.  In my opinion I would like to rework that API though.  get right of scheduler hints and design a better api, since ripping out scheduler hints isn't much at this point
16:00:41 <Yathi> guys I think we will need to continue this next time or in the ML
16:00:41 <MikeSpreitzer1> alaski: OK, refining APIs is good.  But later in our roadmap.  For now I am satisfied to expect it can be done.
16:00:47 <alaski> well, we're at time
16:00:47 <glikson> alaski: any chance you can come up with a brief summary of your thoughts on this?
16:01:19 <alaski> glikson: which one?  multiple schedulers?
16:01:23 <MikeSpreitzer1> I'll be watching the ML
16:01:41 <glikson> alaski: no, reworked API, hints, etc
16:02:18 <alaski> glikson: basically I find the hints api lacking.  Based on current work let's see what sort of placement api makes sense and move towards that
16:02:21 <Yathi> we are yet to get to the implementation details behind how the API work will lead to the actual scheduling
16:02:33 <alaski> but we should give up the channel now
16:02:39 <alaski> #endmeeting