11:00:27 <oneswig> #startmeeting scientific-wg
11:00:28 <openstack> Meeting started Wed Sep 13 11:00:27 2017 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
11:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
11:00:32 <openstack> The meeting name has been set to 'scientific_wg'
11:00:33 <oneswig> #chair martial
11:00:34 <openstack> Warning: Nick not in channel: martial
11:00:35 <openstack> Current chairs: martial oneswig
11:00:48 <oneswig> #chair martial_
11:00:49 <openstack> Current chairs: martial martial_ oneswig
11:01:00 <oneswig> Greetings
11:01:04 <martial_> thanks Stig
11:01:20 <martial_> what's on our agenda this am?
11:01:25 <oneswig> #link Agenda for today https://wiki.openstack.org/wiki/Scientific_working_group
11:01:32 <oneswig> Good morning martial_
11:01:37 <oneswig> bright and early today
11:01:41 <priteau> Good morning
11:01:48 <priteau> Or afternoon
11:01:52 <oneswig> Hi priteau
11:02:07 <martial_> Hi Pierre
11:03:17 <b1airo> evening
11:03:19 <martial_> #chair b1airo
11:03:20 <openstack> Current chairs: b1airo martial martial_ oneswig
11:03:31 <b1airo> two martials?
11:03:43 <oneswig> Typo on my part.
11:03:54 <oneswig> He gets two votes :-)
11:04:01 <martial_> I know, strange
11:04:03 <b1airo> ah hh
11:04:20 <b1airo> how is your new job going martial_ ?
11:04:30 <oneswig> just four of us thus far. verdurin sends his apologies - can't make it today
11:04:38 <martial_> b1airo: keeping me busy but that is expected
11:04:48 <oneswig> Same for enol, sends his apologies but tells me the federation section is coming together well
11:05:05 <oneswig> Apparently, expecting a case study from Nectar today...
11:05:26 <oneswig> martial_: what is your new role at DataMachines?
11:05:55 <b1airo> i see belmoreira is online, maybe we can quiz him on Ironic at CERN...
11:06:48 <martial_> oneswig: I will start shaping the research outreach of company in time
11:06:56 <daveholland> (apols for lateness)
11:07:05 <b1airo> hi daveholland
11:07:10 <oneswig> Hi daveholland, welcome - just getting going
11:07:29 <priteau> Congratulations on the new job martial_
11:07:40 <martial_> priteau: thanks
11:07:58 <oneswig> b1airo: good idea - we are missing a case study on infrastructure management...
11:08:08 <martial_> we should also talk ORC and P2302 on today's meeting
11:08:29 <b1airo> P2302...?
11:08:30 <oneswig> martial_: lets do that.
11:08:36 <oneswig> So without further ado
11:08:40 <oneswig> #topic Taking advantage of opportunistic capacity on OpenStack
11:08:57 <b1airo> ah the standard on interop
11:09:18 <martial_> b1airo: yep
11:10:42 <oneswig> So last week we discussed a little about how to make the most of unused available resource, but I don't think there was a clear example of people nailing this.
11:10:48 <b1airo> so, anybody here offering opportunistic / preemptible instances on their clouds in some way?
11:10:49 <oneswig> Unless b1airo you recall otherwise?
11:11:11 <b1airo> no, seems to be in the category of "yes please!"
11:12:10 <b1airo> there was a talk proposed for the summit which i think got in, will see if i can find it...
11:13:59 <b1airo> yep here it is
11:14:03 <b1airo> #link https://www.openstack.org/summit/sydney-2017/summit-schedule/events/19990/htc-service-on-underutilized-openstack-cloud-resources?BackURL=https%3A%2F%2Fwww.openstack.org%2Fsummit%2Fsydney-2017%2Fsummit-schedule%2Fglobal-search%3Ft%3Dresearch%23eventid%3D19990
11:14:21 <martial_> (and maybe SC17 BoF)
11:15:13 <martial_> b1airo: seems interesting indeed
11:15:56 <oneswig> ooh.  Nice.  Be good to hear what the MOC people are up to.
11:16:16 <b1airo> the approach mentioned in that abstract is almost exactly what i sketched out here with one of my colleagues, we just haven't actually tried implementing it though
11:16:36 <martial_> Interesting how they use slurm to dispatch the jobs, makes sense
11:16:42 <oneswig> Something I am not sure about with the SLURM interface is that when there is a change in cluster configuration, a new config file must be propagated among all nodes.
11:17:18 <b1airo> yes, i'm curious how they glue it in and whether the new resource provider scheduling model helps
11:18:27 <b1airo> oneswig, i guess in that case the slurm cluster is probably running across the hypervisors and so quite static
11:19:37 <b1airo> or maybe it is a single node slurm only used for scheduling... hopefully we'll find out!
11:19:42 <priteau> They had a lightning talk in Boston: https://www.openstack.org/summit/boston-2017/summit-schedule/events/18991/hpchtc-and-cloud-making-them-work-together-efficiently
11:19:45 <oneswig> Isn't this about opportunistic compute instances?  I'd expect them to appear and disappear with little warning
11:20:01 <priteau> #link https://www.openstack.org/videos/boston-2017/hpchtc-and-cloud-making-them-work-together-efficiently
11:20:29 <b1airo> i thought of the instances as jobs from slurms perspective
11:20:47 <b1airo> priteau, oh wow, how did i miss that!
11:22:02 <b1airo> ah no, you're right oneswig
11:22:25 <b1airo> i didn't read the last sentence about slurm user experience
11:23:00 <oneswig> #link video for the lightning talk: https://www.openstack.org/videos/boston-2017/hpchtc-and-cloud-making-them-work-together-efficiently
11:23:06 <b1airo> my idea was to use slurm under the hood to manage nova scheduling and resource capacity
11:24:06 <b1airo> to get preemptible instances available as a service, so to speak
11:24:07 <oneswig> b1airo: that approach would add to job startup and put quite a bit more work through the openstack control plane.  It does get used as a model however - works for long-running jobs.
11:24:51 <b1airo> yeah, the users i am thinking of do not want a queuing system interface
11:25:37 <oneswig> b1airo: how would they interact and what is slurm queueing in this scenario/
11:27:57 <martial_> maybe you will be able to see the result in the video or we can ask them at the next summit I guess :)
11:28:45 <b1airo> well, one idea was to reserve nova capacity for this use case and then offer it to a set of users. they would (at least in the mvp) have to submit a slurm job, but that job would describe their nova create, then slurm would (eventually, and according to all its rich scheduler goodness) boot their instance
11:30:11 <b1airo> if that proved useful it might be more directly integrated with nova
11:30:47 <oneswig> b1airo: sounds like the SLURM-V model used by Ohio State University.
11:30:50 <b1airo> might need a new instance state though :-/
11:30:56 <oneswig> Try looking that up
11:31:06 <b1airo> will do, thanks
11:31:31 <priteau> So in essence it would provide a batch job queue, with Slurm doing the scheduling but with OpenStack doing the actual VM lifecycle management?
11:32:11 <oneswig> I guess what it adds is a queue to VM creation, rather than failing when no capacity is available
11:32:59 <b1airo> yes, and users can do whatever they like with their VM - it is not forced into a HTC job model
11:34:06 <martial_> must be interesting in the flavor availability
11:35:30 <b1airo> the other idea was more a pipe dream about what might be possible if something like SLURM was underneath Nova handling some of the resource providing
11:35:34 <priteau> b1airo: I suppose they still provide a walltime for the job, and the end of which the VM is killed?
11:36:09 <oneswig> How about we invite the speaker Rajul Kumar to attend the next WG meeting?
11:36:11 <b1airo> priteau, yes that would be crucial, e.g., in our case we'd limit walltime to something like 24 hours or so
11:36:31 <martial_> oneswig: should good
11:36:44 <b1airo> good idea oneswig
11:36:47 <martial_> oneswig: plus he is in Boston
11:36:54 <oneswig> #action Stig to track down Rajul Kumar and invite him along
11:36:59 <martial_> let me reach out to him
11:37:07 <martial_> (or Stig)
11:37:07 <oneswig> #action Martial to do it :-)
11:37:16 <oneswig> excellent.
11:37:19 <oneswig> thanks martial_
11:37:30 <oneswig> Let's move on...
11:37:33 <martial_> of course
11:37:41 <oneswig> #topic Private cloud capacity meter
11:38:11 <martial_> #link https://status.rc.nectar.org.au/capacity/
11:38:18 <oneswig> Second airing of this one... johnthetubaguy created a tool for using new Nova APIs to query available private cloud capacity
11:38:22 <martial_> b1airo interesting
11:38:29 <oneswig> #link https://github.com/johngarbutt/os-capacity
11:38:41 <oneswig> Share and enjoy :-)
11:39:08 <daveholland> I need some Copious Free Time, those would be good graphs to produce for the deployment here
11:39:15 <oneswig> There are even better API capabilities for measuring available capacity in Pike, apparently.
11:39:57 <oneswig> but this one was developed against Ocata, and may be compatible further releases back
11:40:11 <oneswig> Copious Free Time - luxury :-)
11:40:11 <daveholland> (crosses fingers for Newton and Liberty)
11:41:01 <oneswig> We are planning to convert to monitored metrics, much in the manner of that nice NeCTAR page
11:41:33 <oneswig> ... which really does make you wonder if they do *anything* at Melbourne uni
11:42:10 <oneswig> #topic Updated edition of the Scientific OpenStack book: Progress report
11:42:44 <oneswig> Right, this is going well, there has been a load of activity updating entries.
11:42:53 <oneswig> Thanks to everyone who has come forward.
11:43:23 <oneswig> Reminder for those yet to do so - end of next week is the deadline for contributions
11:43:36 <daveholland> O/S+Lustre: James has produced a draft, I am editing it, we are at ~2300 words +diagrams
11:43:56 <oneswig> daveholland: that sounds great.  2300 words seems on the large side?
11:44:15 <daveholland> it can't decide if it's a high-level case study or a how-to :(
11:44:29 <oneswig> I haven't sized any of the other case studies for word count.
11:44:47 <oneswig> daveholland: I can help with some guidance on that - will mail you
11:44:54 <daveholland> oneswig: thanks
11:45:19 <oneswig> We have two other matters here:
11:45:43 <oneswig> 1) Would be good to have an additional study on bare metal infrastructure management (the Cray one is alas no longer valid)
11:45:58 <oneswig> Any volunteers?  I can put one together on the SKA prototype if need be.
11:46:14 <oneswig> 2) Cover art proposals.
11:46:29 <oneswig> These must fit some very relaxed ownership criteria.
11:46:52 <oneswig> "As an open project, we only use no cost, royalty-free, Creative Commons attribution only, commercial use allowed images. We don't license images. You can see how the V1 cover image fits that bill in its attribution on the book's back cover"
11:47:43 <oneswig> If anyone would like to propose an image from their institution, lets create a list on the etherpad and we can put it to a vote next week
11:48:42 <martial_> sounds good
11:49:18 <oneswig> #link Cover art proposals, add to this etherpad: https://etherpad.openstack.org/p/scientific-computing-book-update-2017
11:49:39 <oneswig> OK, thanks again for everyone's help.  I think that's all to cover on this
11:49:58 <oneswig> #topic Scientific SIG
11:50:16 <oneswig> There was some discussion last week on the differences about being a SIG and they seemed minimal.
11:50:35 <oneswig> One I am not sure of is taking discussion off openstack-operators list - that doesn't seem like a great move to me.
11:51:34 <martial_> there is a meeting tomorrow (middle of the night for me) I believe to continue this discussion?
11:52:07 <martial_> (among other topics)
11:52:58 <oneswig> martial_: ah, thanks for the reminder. 6am.
11:53:54 <oneswig> Let's follow up after that.
11:54:11 <oneswig> OK lets try to make time for the AOB?
11:54:16 <oneswig> #topic AOB
11:54:59 <martial_> YES
11:55:01 <martial_> ORC: Amsterdam meeting arriving very soon
11:55:01 <martial_> #link https://www.egi.eu/blog/second-international-open-research-cloud-congress/
11:55:03 <martial_> Ildiko Vancsa from OpenStack will be participating, but nobody from the SWG currently?
11:55:03 <martial_> They were asking about a Scientific prospective if possible
11:55:17 <martial_> and #2
11:55:28 <martial_> IEEE P2302 & NIST Public Working Group.
11:55:28 <martial_> This is an effort on Cloud Federation (global, not Research, so different from the ORC).
11:55:29 <martial_> You can find more details at
11:55:29 <martial_> #link http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/FederatedCloudPWGFC
11:55:31 <martial_> #link http://sites.ieee.org/sagroups-2302/
11:55:31 <martial_> During the kick off OpenStack was cited as a cloud for the testing effort.
11:55:33 <martial_> I reached out to David but have had no further information at this point.
11:55:59 <martial_> Stig, are you planning to go to the next ORC meeting?
11:57:03 <daveholland> (got to dash, next meeting looms)
11:57:13 <dweaver> Just dropped by to let you know that at the London Days UK event you have a meeting slot scheduled for 15:30 - 17:30 in room Bishopsgate 1 - is that time suitable for you as we have the room available for most of the day?  The only clash I can see is John Garbutt is giving his talk at 15:45 - 16:15, but everyone else doing HPC should be free.
11:57:43 <oneswig> martial_: I am not planning to at present but may change plans...
11:57:48 <oneswig> Hi dweaver, welcome :-)
11:58:02 <dweaver> oneswig, Hi, I've been lurking :)
11:58:57 <martial_> not much more from me on AOB
11:59:10 <martial_> and we have only a minute left
11:59:11 <oneswig> Thanks for the scheduling details.  I think John might want to be involved in the WG (and WG members to see his talk).  How about we take the second hour of the slot?
11:59:33 <oneswig> Is the schedule posted online/
11:59:59 <dweaver> oneswig, OK, 16:30-17:30 should be fine.  Yes, on the website here: https://openstackdays.uk/2017/
12:00:20 <oneswig> Thanks dweaver, will take a look
12:00:32 <oneswig> We are alas out of time - busy session as always
12:00:40 <oneswig> Thanks everyone
12:01:04 <martial_> #endmeeting