Wednesday, 2017-11-15

*** crushil_ has joined #openstack-cyborg00:35
*** crushil_ has quit IRC01:36
*** crushil_ has joined #openstack-cyborg01:36
*** crushil_ has quit IRC02:23
*** crushil_ has joined #openstack-cyborg02:50
-openstackstatus- NOTICE: Due to an unexpected outage with Zuul (1 hour), you'll need to recheck any jobs that were in progress. Sorry for the inconvenience.02:56
*** masber has joined #openstack-cyborg07:53
masbergood afternoon, I just found you project, is cybord ready to be installed?08:03
masberI would like to be able to use FPGAs on openstack, can I get some help within this channel?08:04
zhipengh[m]Hi masber , thx for the interest !08:21
zhipengh[m]Could u introduce a little bit about yourself and your requirement ?08:22
masberoh yes sorry08:22
masberI work for a medical research intitute08:22
masberand we are playing with openstack/ceph in order to see if it can fit out use cases08:23
masbernow we are doing a project with intel08:23
masberand we will borrow some FPGA cards from them08:23
zhipengh[m]That sounds interesting :)08:24
masberI would like to see if we can make those cards to work on openstack, I mean been able to access the FPGAs from the instance08:24
masberI did a talk in the openstack summit in sydney08:24
masberI talk about what we do etc08:25
masberI am not an openstack expert just, installed it and played with it a little bit08:25
masberbut now with these FPGAs coming I feel I should start getting my hands dirty08:26
zhipengh[m]Cool ! Can you provide the title of your Sydney topic ? So that I could search on youtube :)08:26
zhipengh[m]As for cyborg08:26
masberso, if I am not mistaken we are receiving some nallatech 385a08:27
zhipengh[m]Our current code base cannot function yet, and we are undergoing active development08:27
masberthe name of the talk was "building a genome future with openstack"08:27
zhipengh[m]I will definitely check it out :)08:28
masberit is quite basic talk08:28
zhipengh[m]So at the moment you could run up the three main cyborg processes, but no function yet08:28
masberI see08:29
zhipengh[m]We are now try to define data models08:29
zhipengh[m]And would be great to have input from you08:29
zhipengh[m]For example on FPGA08:29
masberok, I am not an user with lots of expertise on these kind of things, but happy to give it a go08:29
zhipengh[m]What is your main use case for that and what would you like to see FPGA resource represented08:30
zhipengh[m]That'd be great !08:30
masberthis is to run a tool called GATK408:30
masberit is a best practice set of tools to run genome analysis08:30
masberso this tool can work on HPC or in a SPARK cluster08:31
masberour goal is to see how much efficiency we can get by running these tools on FPGAs08:31
masbercompared to running on Apache Spark08:31
masberthe thing is I am receiving the cards tomorrow from the intel guy who is coming from overseas (not sure from singapose, china or US)08:32
masberand I am not sure for how long I can keep the cards either08:32
zhipengh[m]that is very interesting08:33
masberanother problem is that currently I am deploying openstack using kolla...08:34
zhipengh[m]I think we should first look into GATK4 , see what are the resource requirements for running it on the FPGA card08:36
zhipengh[m]Those requirements could compose what we call accelerator capability in cyborg08:37
masberI think I can get that information from intel I think08:37
masberI see08:37
zhipengh[m]And then I think our team could come up the very first data model outline for FPGA08:37
masberI think intel got GATK4 running on FPGAs08:38
zhipengh[m]With a real use case :)08:38
masberyes my use case is quite real because we are planing to deploy this pipeline in production08:38
masberwe are actually running GATK3 in production08:38
zhipengh[m]BTW we will have video conference about 6:20 hour later08:38
zhipengh[m]Would it be possible for you to join ?08:38
masberin 6.20 hours08:39
masberlet me check08:39
zhipengh[m]Haha right08:39
masber2:00 am my time08:39
masberyes I can join08:39
zhipengh[m]Wow that is pretty late08:39
zhipengh[m]Anyways I will send out the details later on openstack-dev mailing list08:40
masberyep, well as I said Im not a quite competent openstack user so most of the time I have to stay till late so I can get help from the community08:40
masbermany nights working until 4am08:40
masberbut it is ok, I am happy to try and see if I can help you08:41
zhipengh[m]Man that is hardcore08:41
masbermy boss gives me free days08:41
masberit is ok08:41
masberplus I'm a pragmatic person which means I learn by experience so for me it is quite good to get in contact with people in get things done08:42
masberyeah I will join the call so if there any doubt I can talk with the intel guy tomorrow08:42
masberhe is giving me the cards and then we will have a meeting08:43
masberso maybe I can ask questions in your behalf08:43
masberso how do you do the video conference?08:46
masbergoogle hangouts?08:47
zhipengh[m]Very good quality08:50
masberok I will see you later, Im going home now08:53
zhipengh[m]Take care man !08:59
*** crushil_ has quit IRC10:30
*** mpaolino has joined #openstack-cyborg14:36
*** zhipeng has joined #openstack-cyborg14:48
*** crushil has joined #openstack-cyborg14:49
*** masuberu has joined #openstack-cyborg14:55
masuberuis the meeting happening?14:56
zhipenghi will start momentarily14:57
masuberuI don't know how to access tho14:58
masuberuand also I can't talk as it is 2am and everybody is sleeping at home but I can chat and listen14:59
masuberujust ping me when people is ready15:00
zhipengokey no problem, let me send the zoom link here15:00
zhipeng#startmeeting openstack-cyborg15:00
openstackMeeting started Wed Nov 15 15:00:29 2017 UTC and is due to finish in 60 minutes.  The chair is zhipeng. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: openstack-cyborg)"15:00
openstackThe meeting name has been set to 'openstack_cyborg'15:00
zhipengjust did a little bit chatting with crushil, that we will keep irc meeting today, and move the zoom meeting to a later time slot for dev specific issue15:06
zhipengwill keep everybody posted15:06
masuberuso no zoom, just irc for now?15:07
*** shaohe_feng_ has joined #openstack-cyborg15:08
zhipengmasuberu yep :)15:08
zhipengcrushil you there ?15:08
*** Li has joined #openstack-cyborg15:08
shaohe_feng_zhipeng: what's the zoom conference ID?15:09
*** Li is now known as Guest6987115:09
zhipengshaohe_feng_ will do zoom tomorrow morning15:09
shaohe_feng_zhipeng: got it.15:09
zhipengcould everyone log their names ?15:10
zhipeng#info Howard15:10
zhuliCan’t get my pc connected, have to use cellphone15:10
mpaolino#info Michele15:10
Guest69871#info Li15:10
zhipengzhuli we will just irc tonight15:10
zhuli#info zhuli15:10
zhuliGot it15:10
Guest69871Do we have another meeting tomorrow?15:11
crushilI'm here15:11
zhipengokey seems like full house almost :)15:11
crushil#info Rushil15:11
zhipeng#topic GATK4 FPGA requirement discussion15:11
*** openstack changes topic to "GATK4 FPGA requirement discussion (Meeting topic: openstack-cyborg)"15:11
zhipengthe first order of business today is that we have masuberu15:12
zhipengsystem engineer from one of the largest genome testing reserach testing centers in the world15:12
zhuliWelcome masuberu15:13
masuberuthank you all for having me here15:13
zhipengto talk with us about their upcoming trail with Intel FPGA card to run GTAK4 application for genome analysis15:13
zhipengthe floor is yours masuberu :)15:13
masuberuthan you15:13
masuberuI work for a medical research intitute15:14
*** shaohe_feng__ has joined #openstack-cyborg15:14
masuberuand as sequence and analyse genomes15:14
shaohe_feng__#info shaohe15:14
*** shaohe_feng_ has quit IRC15:14
masuberuright now we are in a project with intel15:14
masuberuthey are helping us to build an environment to run GATK4 which a set of tools of best practice genome analysis15:15
masuberutomorrow a guy from intel is coming to my office to give me FPGA cards15:15
masuberubecause GATK4 can either work on HPC cluster, Spark cluster and GDC15:15
masuberuso my idea is to test GATK4 on FPGAs through openstack and compare performance with apache spark15:16
masuberuobviously FPGA should be much faster but we want to see how much faster15:16
masuberuand I was looking for documentation about how to integrate FPGAs on openstack and I found your project15:17
zhipengshaohe_feng_ is from Intel :)15:17
masuberuso wanted to say hi and see if this would be a possibility of collaboration15:17
masuberuwell intel is quite big15:18
masuberuI guess15:18
shaohe_feng__masuberu: hi15:18
zhipenghaha yes15:18
Guest69871Hi Masuberu, This is Li from Huawei, our team has a lot of experience using FPGAs in Openstack15:18
masuberuthe person coming tomorrow is from either China or Singapore I think15:18
zhipengmasuberu could you describe a possible setup ?15:18
zhipengFor example if you are gonna have a VM connected to the FPGA15:19
zhipengor just feeding data directly to GTAK4 on that FPGA ?15:19
masuberuwhat ever is easier, I was thinking of giving a go and see if could be possible to integrate FPGAs with nova and see if the vms could see it15:20
shaohe_feng__masuberu: I'm also working on FPGA. And some of us has started to support FPGA on k8s15:20
masuberuotherwise I am happy of using different methods15:20
masuberuI am totally open to suggestions15:20
masuberuthe setup is15:20
masuberu4 compute nodes with AMD cpus (56 cores) 512 RAM and ~4.5TB spinning disks 10k15:21
Guest69871Do you have the developed IPs to use for the FPGA?15:21
zhipeng#info masuberu from Garvan Institute of Medical Research15:21
masuberuand 3 nodes with intel cpus (numa 2 sockets 14 cpus each) 512 GB RAM and 20 TB nvmes15:21
masuberuall connected through 2x25GB bond network15:21
masuberunetwork is mellanox ethernet15:22
masuberumellanos switch 100GB15:22
*** Guest69871 has quit IRC15:22
masuberuit is not big setup but I think it is a good starting point15:22
*** LiLiu_ has joined #openstack-cyborg15:23
LiLiu_#info Li15:23
masuberuI don't know how many FPGAs I will get, I will know tomorrow15:23
zhipengshaohe_feng_ any tips on nova integrates with FPGA ?15:23
masuberuso I am not sure if that explains most of the questions? please feel free to ask15:24
zhipengmasuberu will GTAK4 run as an IP for the FPGA ?15:24
masuberutomorrow I have a meeting with Albert from intel and he will brief me about how it works15:25
LiLiu_Question: is there multiple IPs to be used in the GATK415:25
masuberuLiLiu_, what do you mean by multiple IPs?15:25
LiLiu_I mean different bitstreams15:26
masuberuso GATK4 is a bunch of tools bioinformatitians use to create pipelines15:26
masuberuyou can run a tool at a time or multiple ones at the same time, depends on what you are doing15:27
LiLiu_OK, I assume you will have multiple types of IPs in your system. Do you chain the FPGAs together or each FPGA only talk to the tied CPU ?15:28
masuberuok, I will ask that question to the intel guy tomorrow15:29
masuberuI honestly never used a FPGA before15:29
zhipengshaohe_feng__ any tips on nova integrates with FPGA ?15:29
LiLiu_I see, my team works a lot on FPGA heterogeneous system in Openstack, we may talk more offline if you are interested, masuberu.15:31
zhipengdid some googling15:33
LiLiu_I wanna know more about the scope/size of the system you guys are targeting15:33
masuberuit will depends on the performance of this tests15:33
zhipengseems like the pairHMM will be the part that need acceleration from FPGA or GPU ?15:33
masuberufor instance we have capacity to sequence 1200 genomes per month so it all depends how much hardware we need for that15:34
shaohe_feng__zhipeng:  actually, every accelerator in Intel's FPGA is a sriov device.15:34
masuberuzhipeng, yep I got that document yesterday from Albert15:35
zhipengshaohe_feng__ how about the altera ones ?15:35
zhipengthe same ?15:35
shaohe_feng__zhipeng: altera means intel :)15:35
masuberunallatech 385A is what I am getting15:36
shaohe_feng__zhipeng: there maybe 3 method to support FPGA.15:36
masuberugetting for testing15:36
shaohe_feng__zhipeng: 1. pre-programming FPGA on the host.15:36
shaohe_feng__zhipeng: 2. the orchestration program FPGA15:38
shaohe_feng__zhipeng: 3. VM program FPGA.15:38
shaohe_feng__zhipeng: I have got a FPGA server, and I'm trying to do some test on it15:38
masuberushaohe_feng__, the person coming tomorrow is called AlbertZQ Wang, not sure if you know him...15:38
shaohe_feng__zhipeng:  AlbertZQ Wang from intel?15:39
shaohe_feng__^ masuberu:15:39
shaohe_feng__which site? China or US15:40
masuberuI guess China but not 100% sure15:40
masuberumaybe either of them15:40
masuberuany questions?15:41
shaohe_feng__masuberu: haha, we can connect each other convenience in intel's internal connection tool.15:41
masuberuyeah they showed me that tool last time they came to my office15:42
masuberuany plan?15:42
zhipengi think this is definitely a very interesting use case for us15:42
zhipengit would be great if you could come back with more details from the Intel guys tomorrow15:43
zhipengand we could hammer out details for future work15:43
zhipengi think getting Nova connect with FPGA is easy15:43
zhipengbut how to sched and manage your FPGAs is what we will be working on15:44
masuberuzhipeng, you mean because of SRIOV?15:44
masuberuit can be nova or k8s15:44
masuberuor another method15:44
zhipengyes because the current support for Nova is good enough15:45
zhipengto set it up15:45
masuberuwhat about overhead?15:45
zhipengwhat type of overhead you are thinking about ?15:45
masuberuis it going to have much overhead due to virtualization?15:45
zhipengi think that depends15:45
zhipenghow you setup the workload between VM and FPGA15:46
masuberuok, is there anything I can read about that? because I don't know much to be honest15:46
zhipengfor example if you gonna have FPGA do all the GATK heavy lifting and VM just for running a data pipe application15:46
masuberuand I want to get the most of this project15:46
zhipengi don't think there will be much overhead15:47
masuberuI don't know for how long I can have the FPGAs15:47
masuberulets say we decide to run on the numa nodes, then I can even enable cpu afinity15:47
masuberuto make sure process use local memory15:48
masuberubut that is a different story15:48
masuberuI am more worry about the lack of documentation about openstack+FPGAs15:48
zhipengyou can just go through the nova pci support docs15:49
masuberualso we have another project coming to run GPUs for machine learning stuff15:49
masuberubut that will come in 3-4 months time15:49
zhipengcool :)15:49
masuberuanother thing I normally deploy openstack using kolla...15:50
masuberukolla-ansible, not sure if that will be an issue15:50
zhipengany thoughts on making cyborg kolla-able ? :P15:50
crushilzhipeng, My coworker who presented for us is kolla-kubernetes core15:51
crushilI'm working with him to get Cyborg kollafied15:51
zhipengmasuberu there you go :)15:51
masuberuthat's cool15:51
zhipengwe have everyone you need in cyborg :P15:51
crushilIt's already on my plate for a future sprint15:51
masuberuI am quite flexible we can do whatever we want with this nodes as they are running nothing15:51
zhipengmasuberu just give a howler any time you have an issue on this channel15:52
zhipengcore members are always logged in , so as long as any of us are awake, we will respond :)15:52
crushilBtw you should play with this tool I will be integrating Cyborg into this tool15:52
zhipengyou could also just send email to the openstack-dev mailinglist, or openstack-ops15:53
masuberuI think I know rwellum15:55
masuberuby the way GATK4 is just a jar file15:56
masuberuvery simple to install and run15:56
masuberuis it possible to share an FPGU with multiple instances?15:57
masuberuor do I need to assign 1 FPGU card per vm?15:57
LiLiu_if the FPGA has multiple PFs/VFs, then yes15:57
*** zhipeng has quit IRC15:57
LiLiu_It really depends on how the IP is developed15:58
*** zhipeng has joined #openstack-cyborg15:58
masuberuok, and the FPGUs has cores?15:58
masuberuI mean can I break the resources of a FPGU card and assign it to multiple vms?15:59
LiLiu_FPGA can be virtualized and shared by multiple VMs15:59
LiLiu_you are right16:00
masuberufor example with memory and cpus I can "define" flavors, can I do same with FPGUs?16:00
LiLiu_That's one of the most common ways of managing FPGA in Openstack16:01
masuberulike for instance 0.3Teraflops for one instance and 0.7Teraflops to another16:01
LiLiu_That's totally doable16:02
masuberudo you have experience with openhpc?16:02
LiLiu_I don16:02
masuberuwe run rocks cluster but the development is quite slow, I am wondering whether openhpc would be better option16:02
LiLiu_but I know someone in my team might do16:02
masuberuok, thats ok, let try and focus on these tests comparing GATK4 on FPGUs vs Apache Spark16:03
masuberumaybe this type of tests doesn't even makes sense, we will see16:03
masuberuso I will get the cards tomorrow and connected to the servers16:04
masuberuand from there if anyone know any document about how to integrate nova with fpgus or if there is any document/tool within cyborg project just let me know16:05
LiLiu_before cyborg is ready to use, I think the most naive way is just use PCI passthrough16:06
crushilzhipeng, Don't forget to close the meeting16:13
zhipengof course :)16:13
zhipengif nothing else, we had a great discussion today16:14
zhipengthe meeting is adjourned16:14
zhipengand I will let everyone know the new ZOOM meeting time16:14
*** openstack changes topic to "#openstack-cyborg"16:14
openstackMeeting ended Wed Nov 15 16:14:28 2017 UTC.  Information about MeetBot at . (v 0.1.4)16:14
openstackMinutes (text):
masuberuthank you all16:22
masuberuand sorry I hijack the whole meeting, I guess you probably had other things to talk16:22
masuberuit is 3.22am here so I am going to sleep now16:22
masuberuplease feel free to send me an email if any idea pops up or masber@hotmail.com16:23
zhipengnite :)16:26
*** zhipeng has quit IRC16:28
jkilpatrdid we start at 10am this time? I was off by an hour :(16:50
*** shaohe_feng__ has quit IRC16:56
*** masuberu has quit IRC16:57
*** masuberu has joined #openstack-cyborg17:06
*** masber has quit IRC17:15
*** crushil has quit IRC17:21
*** crushil has joined #openstack-cyborg17:32
*** crushil has quit IRC17:49
*** mpaolino has quit IRC17:51
*** crushil has joined #openstack-cyborg18:13
*** LiLiu_ has quit IRC18:54
*** crushil has quit IRC19:31
*** crushil has joined #openstack-cyborg19:47
*** masber has joined #openstack-cyborg23:32

Generated by 2.15.3 by Marius Gedminas - find it at!