11:00:03 <oneswig> #startmeeting scientific-sig
11:00:04 <openstack> Meeting started Wed Nov 21 11:00:03 2018 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
11:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
11:00:07 <openstack> The meeting name has been set to 'scientific_sig'
11:00:18 <oneswig> Hello \o/
11:00:40 <priteau> Good morning
11:00:41 <oneswig> #link Agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_21st_2018
11:00:42 <mgoddard> \o
11:00:43 <verdurin> Morning.
11:00:48 <oneswig> Greetings all
11:00:58 <dh3> morning
11:01:20 <Elwell> Hi Folks, Mostly lurking for #3 on agenda
11:01:28 <oneswig> It's thanksgiving over the pond, we'll not be hearing from them I guess
11:01:44 <oneswig> Hi Elwell - good to see you
11:02:15 <oneswig> dh3: I heard from Komathy, she's waiting for confirmation from Sanger on their visit
11:02:36 <dh3> oneswig: OK, I will prod Pete to give a definite "yes" (I thought I had already)
11:02:58 <oneswig> Elwell: We could move #3 to #1 - no problem at all
11:03:06 <Elwell> I can wait :-)
11:03:06 <oneswig> Let's tee off with that?
11:03:29 <oneswig> I'm curious myself!
11:03:41 <oneswig> #topic Lustre and Manila
11:04:01 <oneswig> Elwell: can you describe what you're looking for?
11:04:10 <Elwell> I've been tasked with "investigate how possible it is to export our lustre filesystem(s) to on-prem openstack, securely"
11:04:42 <oneswig> IB or IP?
11:04:53 <Elwell> My current plan is probably to reproduce the NCI (manual NFS export) way, but wondeered how automatic Manila would help
11:05:02 <janders> g'day everyone
11:05:12 <oneswig> hey janders, you made it home
11:05:36 <dh3> Sanger are interested in the Manila approach too but haven't got further than looking at the generic manila plugin and going "not enough time". We have Lustre connected to OpenStack in two ways though
11:05:39 <Elwell> Storage is IB, cloud is IP, so I'm expecting to either throw router or (ib client, nfs export to IP) into mix
11:05:50 <janders> I indeed have, however I may still be on Singapore time
11:06:03 <dh3> 1. "isolated" - Lustre FS connected to a separate VLAN which is exposed to a single project as a provider network
11:06:36 <dh3> 2. "secure Lustre" - in the scientific sig book - proper multi-tenant using Lustre routers, full isolation but costs in terms of performance (and complexity to set up)
11:06:49 <dh3> we are all IP/ethernet though
11:07:04 <oneswig> We did a project at Cambridge University for https://www.euclid-ec.org/ which pretty closely matched #1, but all-IB and bare metal OpenStack
11:07:51 <janders> could Lustre work with a purely hardware based IB router?
11:08:12 <janders> http://www.mellanox.com/page/products_dyn?product_family=255&mtag=sb7780
11:08:16 <janders> this sort of a thing
11:08:16 <Elwell> ah by router I mean LNET routeer
11:08:24 <dh3> different beast :)
11:09:02 <oneswig> Elwell: The book dh3 is referring to is here: https://www.openstack.org/assets/science/CrossroadofCloudandHPC.pdf
11:09:09 <Elwell> our storage is still on FDR, we have a mix of EDR and OPA lustre clients (and Aires)
11:09:18 <oneswig> Sanger contributed their method for multi-tenant Lustre
11:09:55 <Elwell> https://nci.org.au/wp-content/uploads/2016/07/LUG-2016-sjjfowler-hpc-data-in-the-cloud.pdf is probably what I'll follow for now
11:10:06 <oneswig> Aries + OpenStack, I remember something faintly about that...
11:10:28 <Elwell> <shudder>
11:10:47 <oneswig> Thanks for the link - will study it!
11:11:30 <Elwell> OK - so looks like noone's using Manila in that role yet, but a few sites interested (if lacking time/manpower) right?
11:11:46 <oneswig> Elwell: what's the ideal mode of usage?
11:12:45 <Elwell> openstack user / project mapped to their own / project part of lustre tree and can read / write there from cloud.
11:13:06 <oneswig> As I understand it, with Manila we can provision filesystems on the fly for cloud use.  I think we can also use Manila as a broker for providing access to existing "provider filesystems"
11:13:08 <Elwell> so pre-post processing on cloud, HPC on $bigsystems,
11:13:40 <dh3> Elwell: do you want the client to be a Lustre client or an NFS client? (we provide images with Lustre ready-baked for less technical users)
11:14:00 <Elwell> tbh, probably just NFS
11:15:01 <Elwell> as we can all_smash permissions down, rather than letting random cloud user hit us with a real lustre client
11:15:13 <Elwell> all_squash even
11:15:35 <dh3> the Lustre uid mapping is neat for that
11:16:00 <Elwell> yep, but only 2/3 of our systems are on 2.10, the other is still 2.5
11:16:24 <dh3> fair point, the majority of ours are 2.5 too
11:16:25 <Elwell> (appliance so unlikely to be upgraded
11:16:46 <oneswig> I assume it's easier to develop this NFS interposer than it is to upgrade Lustre
11:17:54 <oneswig> Elwell: your OpenStack is virtualised or bare metal?
11:18:31 <Elwell> I *think* virtualised - I'm not in the nimbus team
11:19:58 <oneswig> The NFS re-export is a well established pattern in Manila.  But I think currently the solution is not designed to scale out.
11:20:22 <oneswig> There are plans for that, I believe, but it's not there today.
11:21:00 <Elwell> OK - that's probably not too big a blocker for us at the moment as it's more a proof-of-concept for now
11:21:31 <Elwell> Thanks all.
11:21:39 <oneswig> Elwell: do you think you / your group would be up for developing this poc?
11:22:23 <oneswig> I'm pretty sure we could gather together a decent group of interested people.
11:22:45 <Elwell> It'll depend on staff availability I guess. We're about to kick off some heavy procurements and not sure how much time that'll suck up
11:23:02 <Elwell> (OT- we're recuiting a project manager...)
11:23:07 <oneswig> Sounds familiar.
11:23:57 <Elwell> but yeah - happy to share progress with others
11:24:13 <oneswig> OK, I'll make a note, we should come back to this again.  Are you in Perth?
11:24:38 <Elwell> I'm actually in Tasmania - I work remotely
11:25:05 <oneswig> There are few places more remote, from here...
11:25:14 <Elwell> (and I'm normally loitering on IRC anyway)
11:25:24 <oneswig> #action oneswig to follow up on Lustre + Manila interest group
11:26:03 <oneswig> OK - thanks Elwell, I hope we can gather some requirements in the next couple of weeks.
11:26:09 <oneswig> Move on?
11:26:15 <Elwell> +1
11:26:31 <oneswig> Thanks!
11:26:41 <oneswig> #topic Berlin wash-up
11:27:13 <oneswig> For those that joined in Berlin, thanks for taking part, it was on a large scale and good with it, I thought
11:27:32 <janders> +1
11:27:44 <oneswig> janders: how was the wine? :-)
11:27:55 <oneswig> A well-deserved win
11:28:30 <janders> it's waiting in the cellar for a good occasion :)
11:28:46 <oneswig> #link Etherpad from the meeting https://etherpad.openstack.org/p/Berlin-Scientific-SIG-Meeting
11:28:54 <janders> I think it will go perfectly for a spicy Asian pork dish like bun cha
11:29:08 <oneswig> Sounds great :-)
11:29:42 <janders> thanks oneswig :)
11:29:55 <oneswig> There were several topics discussed in the meeting that seemed to get a positive response
11:30:53 <oneswig> Ongoing developments with federation being a major one.  It was great that Colleen joined us, from the Keystone project
11:32:44 <oneswig> Probably the biggest area of discussion was on how to manage one infrastructure for virt, bare metal and containers (in particular, kubernetes)
11:33:32 <janders> keystone - ocp/k8s integration sounds promising however I believe it has some limitations (especially if we put Kuryr into the picture)
11:35:00 <oneswig> what specifically?
11:35:26 <janders> I believe keystone projects can translate 1:1 to ocp/k8s projects
11:35:27 <janders> however
11:35:42 <janders> kuryr creates all per-project networks in one designated "kuryr" tenant
11:35:51 <janders> that's my understanding
11:36:27 <janders> there might be some confusion here as the kuryr guys aren't 100% across the keystone integration part but that was their take on it
11:37:16 <miloa> Hi, i was at the scientific-sig meeting at Berlin. I would like to give if it is also part of this discution some feedbacks as it was my first scientific-sig meeting.
11:37:30 <oneswig> Hi miloa - welcome!
11:37:43 <miloa> thanks :)
11:38:02 <oneswig> janders: so on the openstack side there is 1 project, on the kubernetes side, many projects?
11:38:17 <oneswig> miloa: go ahead
11:38:58 <miloa> It was very interresting, thanks for the meeting. We were a lot of people from different scientific area
11:39:35 <miloa> I was wondering if it would be possible to have a sort of Scientific SIG On boarding session to not be get lost :)
11:40:51 <oneswig> miloa: That's an interesting point.  The meeting session should cover that, but perhaps it does not cover it enough.
11:41:11 <miloa> And if it would be possible to for example do little group organized in scientific aera (biology, physics...) to know each others and discuss then go back in big group and share ?
11:41:25 <oneswig> Are you thinking of something like a presentation to introduce the SIG and what it does?
11:42:16 <miloa> yes :) something like that, and a presentation of the communication tools the SIG is using for example.
11:42:19 <oneswig> miloa: small spin-off groups, that can sometimes work.  We don't get much time to do it in 40 minutes.  I recall we did something like this with the session at the Austin summit a couple of years ago
11:42:42 <miloa> ok i didn't know :)
11:43:00 <oneswig> I think it's a good idea.  Perhaps we should just get people sitting together anyway!
11:43:37 <oneswig> miloa: where are you from?
11:43:42 <miloa> France
11:43:49 <oneswig> bienvenue :-)
11:43:55 <miloa> merci :)
11:44:24 <oneswig> I was really encouraged to see how many new people were at the SIG for the first time.
11:44:44 <oneswig> You make a good point, we should focus more on introductions
11:45:43 <miloa> Perhaps scientifics SIG breakfirst and/or speed-mentoring will help too.
11:45:44 <mgoddard> there is https://wiki.openstack.org/wiki/Scientific_SIG
11:46:15 <mgoddard> for information about communication etc.
11:46:17 <oneswig> Thanks mgoddard, good link :-)
11:46:46 <miloa> thanks mgoddard :)
11:47:01 <oneswig> The other major topic we discussed in the session was AI/ML and in particular GPUs.  There were ~10 people there representing institutions interested in this.
11:47:54 <oneswig> I don't think there are major gaps at an infrastructure level here, only a gathering of all the current best practice to help share knowledge.
11:47:54 <janders> oneswig: (sorry for the delayed response, didn't want to interrupt the valuable onboarding thread) my understanding is that each ocp/k8s namespace/project can have its own private neutron network
11:48:20 <janders> however all of these networks live in a designated openstack project assigned to kuryr
11:48:55 <oneswig> janders: that does seem strange.  It's like the transposition of what Magnum does.
11:49:11 <janders> which is a little weird on its own but gets weirder if there is ocp/k8s keystone integration
11:49:24 <janders> I will try to work with RHAT to see if we can propose something more sane
11:49:46 <oneswig> I saw Kuryr on several RH slides, they do seem to be pushing it.
11:50:15 <oneswig> ok, thanks janders, would be great to understand more about how these fit together
11:50:49 <janders> I will do my best to progress on this in the coming weeks/months and I shall report back
11:51:10 <oneswig> On the GPU side, I was wondering if we might prevail on Blair - but now he's in NZ the meetings really are at antisocial hours for him.
11:51:24 <janders> perhaps next week?
11:51:34 <janders> it'll be the morning meeting, easier for NZ peeps
11:51:56 <oneswig> I'll drop him a mail - I think it's at a time when he's often en route to the airport for an early morning flight!
11:52:18 <janders> good call
11:52:38 <janders> I didnt mean to interrupt the lustre-manila chat but as we're approaching the end of the meeting:
11:52:45 <janders> what's the appetite for manila-beegfs integration
11:52:46 <oneswig> #action oneswig to contact Blair re state of knowledge on GPUs
11:53:01 <janders> as well as wider adoption of beegfs as an openstack storage backend?
11:53:19 <janders> I think beegfs is perfect material for ceph of HPC OpenStack
11:53:22 <dh3> janders: we keep hearing anecdotal good things about beegfs but have no experience. Potentially interested
11:53:23 <oneswig> janders: plenty of appetite over here, that's for sure.  Can it do the necessary functions?
11:53:40 <Elwell> janders: "possible, depends if we go down beegfs"
11:54:06 <janders> I havent done much prep, however my thinking is - if GPFS can do it, why beegfs can't?
11:54:10 <verdurin> I would consider it strictly for scratch-type storage at the moment
11:54:24 <janders> some R&D would need to happen but I think it's well worth it
11:54:30 <oneswig> I'm hoping to meet Sven the BeeGFS creator at a conference in a few weeks (ciuk).  Perhaps if we can figure out what's needed of it, I can put it to him.
11:54:58 <janders> I had a chat to RHAT about it but I also feel we need to speak to BeeGFS guys first
11:55:24 <janders> personally I think manila+BeeGFS-OND = nirvana of HPC Cloud stoarge
11:55:32 <oneswig> janders: would you mind looking into what's needed?
11:55:40 <janders> not at all
11:56:06 <oneswig> #action janders to research BeeGFS functional requirements for Manila
11:56:17 <oneswig> thanks janders, already looking forward to it :-
11:56:23 <oneswig> just time for ...
11:56:28 <oneswig> #topic AOB
11:56:30 <janders> I also think that what you guys have going with ansible (no manila involved) is a pretty powerful demo of what OpenStack+BeeGFS are capable of
11:56:59 <oneswig> thanks janders - it's proving quite popular!
11:57:59 <oneswig> Had a really useful session yesterday in London - the UKRI Cloud Working Group.  Totally recommend it for next time for the UK contingent here.
11:58:43 <verdurin> Glad it was useful, oneswig - couldn't attend, unfortunately
11:58:46 <janders> re AOB - just wanted to say it was great to see some of you in Berlin. Oneswig: thanks for organising a great dinner meetup, really enjoyed it (and heard more good feedback from others)
11:59:01 <janders> good stuff
11:59:03 <oneswig> You're very welcome.  It was great to see so many
11:59:22 <dh3> next time James and I will do the dinner not the bar crawl :)
11:59:36 <janders> how many were brave enough to try the cheese?
11:59:39 <janders> was pretty full on
11:59:41 <janders> good though!
12:00:03 <oneswig> Ah, the cheese.  Despite the alpine theme, I didn't get round to it.
12:00:16 <oneswig> one last thing - have you seen this box - http://www.eenewsanalog.com/news/graphcores-toon-tips-dell-ai-server
12:00:19 <verdurin> You can't miss the cheese.
12:00:37 <oneswig> Makes the DGX-1 look like a mere abacus :-)
12:00:54 <janders> was about to say - is this the DGX killer?
12:01:06 <oneswig> who knows...
12:01:10 <oneswig> OK we are over time, alas
12:01:11 <janders> DGXes seem quite capable of killing themselves :P
12:01:13 <verdurin> Bye.
12:01:15 <oneswig> thanks all
12:01:17 <oneswig> #endmeeting