Wednesday, 2018-04-04

*** Duth has joined #openstack-cyborg03:19
*** Duth has quit IRC03:44
openstackgerritLi Liu proposed openstack/cyborg master: Added bitstream metadata standardization spec
*** zhipeng has joined #openstack-cyborg06:20
*** goldenfri has quit IRC08:27
*** zhipeng has quit IRC08:50
*** circ-user-ojqFa has joined #openstack-cyborg13:55
*** Sundar has joined #openstack-cyborg13:58
*** Li_Liu has joined #openstack-cyborg14:02
*** zhipeng has joined #openstack-cyborg14:02
*** Melissa_S has joined #openstack-cyborg14:02
*** circ-user-ojqFa has left #openstack-cyborg14:03
zhipeng#startmeeting openstack-cyborg14:03
openstackMeeting started Wed Apr  4 14:03:00 2018 UTC and is due to finish in 60 minutes.  The chair is zhipeng. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:03
*** openstack changes topic to " (Meeting topic: openstack-cyborg)"14:03
openstackThe meeting name has been set to 'openstack_cyborg'14:03
zhipeng#topic Roll Call14:03
*** openstack changes topic to "Roll Call (Meeting topic: openstack-cyborg)"14:03
zhipeng#info Howard14:03
Sundar#info Sundar14:03
SundarHi Howard!14:03
Li_Liu#info Li_Liu14:03
zhipengHi Sundar :)14:04
Melissa_S#info Melissa14:04
*** circ-user-ldzTg has joined #openstack-cyborg14:04
zhipengHi Melissa_S14:05
*** Helloway has joined #openstack-cyborg14:05
zhipengCan you introduce yourself a little bit ?14:05
Melissa_SHi! I am new to this chat - my name is Melissa Sussmann. Sundar and I actually used to work together at Intel a while back. I am at Xilinx now and sitting in for Dutch :)14:06
zhipengah nice to meet you :)14:06
Li_LiuWelcome Melissa!14:06
Melissa_SLikewise :)14:06
SundarHi Melissa, welcome! Good to have you here.14:07
Melissa_SThanks Sundar :)14:07
*** Yumeng__ has joined #openstack-cyborg14:08
kosamara#info kosamara14:08
*** chucksong has joined #openstack-cyborg14:09
zhipengHey !14:09
Yumeng__#info Yumeng__14:09
zhipengHi Yumeng :)14:09
Yumeng__Hi zhipeng14:09
chucksongHi everyone!14:10
zhipengHi chucksong14:10
zhipengdon't believe we met before14:10
zhipengcan you introduce yourself a little bit ?14:10
chucksongyes, I am Chuck Song of Xilinx14:11
chucksongCurrently taking charge of the FaaS team14:11
chucksongI suppose we are going to talk about the Cyborg project.14:11
zhipengFPGA as a service ?14:11
SundarHi Chuck, welcome!14:11
Li_LiuHi Chuck14:11
Li_LiuWe have a lot people joining :)14:12
chucksonghi Sundar and LiLiu14:12
zhipenglet's get into business :)14:12
zhipeng#topic sub-team report14:12
*** openstack changes topic to "sub-team report (Meeting topic: openstack-cyborg)"14:12
zhipengI think we could start with release subteam14:13
zhipeng#info cyborg has been added for milestone based release14:14
zhipengthat was part of Rocky goal14:14
SundarIs Apr 19 the spec freeze day for Cyborg too?14:15
zhipengyes Sudnar the week of Apr 16 to Apr 2014:16
zhipengwill be our very first deadline for spec freeze :)14:16
Melissa_Swhat do you need delivered on the Xilinx side before freeze?14:16
Melissa_SPardon me - I am new to the project :)14:17
zhipengMelissa_S altho currently all the drivers are in-tree which means Xilinx driver also needs a spec14:18
Li_LiuI think the major thing we need is Xilinx Device driver. But that doesn't have to be before spec freeze I think14:18
zhipengbut driver spec freeze should be expected for Rocky MS-214:18
zhipengwhich will be week of Jun 04-Jun0814:18
Li_LiuDoes the vendor specific driver need a spec Howord?14:18
zhipengLi_Liu at the moment yes14:18
zhipengbut yes as Li_Liu said, plz don't wait for the deadline :)14:19
SundarNot to jump the gun :) but could I ask folks to review so that I can update ?14:19
SundarThis is based on the latest proposal from Nova community14:19
zhipengSundar will go into that very soon :)14:19
Li_Liusure, I will take a look14:19
zhipengLet's finish the subteam report first14:20
zhipengMelissa_S chucksong plz refer to
zhipengfor the general milestone sched14:20
SundarThanks, Li_Liu and Howard14:21
Melissa_SYes - thank you.14:21
zhipengCyborg's specific sched could be found
zhipengokey moving on14:22
zhipengYumeng__ could you update the progress on your side ?14:22
Yumeng__I sent out a update about the high time synchronisation card on the mail list14:23
Yumeng__Pls check14:24
Yumeng__And launchpad migration to storyboard will be done soon.14:24
zhipengdid you just send the clock driver proposal ?14:25
zhipengor maybe I missed it ?14:25
Yumeng__1 second14:25
zhipengBTW in case everyone not follow the storyboard migration, plz refer to14:26
Yumeng__Pls go ahead, I will send out the right link14:27
zhipeng#info storyboard migration for cyborg14:28
zhipengthx Yumeng__ :) doc subteam has been very responsive and active14:28
zhipengMelissa_S chucksong Dutch signed on as co-lead for the driver sub-team14:29
zhipengdoes one of you guys maybe interested in take the role ?14:29
zhipenghelp update and coordinate the driver development for Rocky ?14:29
Melissa_SChuck and I can work on this14:30
zhipenggreat :)14:30
Melissa_SI think you could put Chuck's name down - he is on the apps side14:31
zhipengduly noted :)14:31
Melissa_Swe will find the correct internal team for this project14:31
*** Yumeng has joined #openstack-cyborg14:31
zhipeng#topic critical rocky spec review14:31
*** openstack changes topic to "critical rocky spec review (Meeting topic: openstack-cyborg)"14:31
zhipengfirst up, Sundar's Nova-Cyborg interaction spec14:32
chucksongzhipeng, yes I will take the role14:32
zhipengSundar could you help get everyone up to speed ?14:32
SundarSure, Howard14:33
zhipeng#info Sundar cyborg-nova spec discussion14:33
SundarWe had arrived at a way of representing FPGA components in Nova, and worklfows for all use cases based on hat representation, back in the PTG14:34
SundarThe spec that Howard provided above codifies that14:34
*** dolpher has joined #openstack-cyborg14:34
SundarHowever, there was a race condition for one use case, which has been debated in the community. An analogy was found with a vGPU issue, and a joint solution was proposed14:35
SundarBased on that, I worked out the details, in a way that should cover GPUs etc. but probably focuses more on FPGAs.14:36
SundarI think this is a better solution than the PTG flow :) both for Nova and Cyborg14:36
SundarFor Cyborg, this means that we don't need to keep accelerator usage details in Cyborg DB -- just use placement info :)14:37
SundarEventually, if/when Nova supports preferred traits, maybe we don't need the weigher -- but we can evaluat that when it becomes real14:38
*** Duth has joined #openstack-cyborg14:38
SundarFor the new folks in Cyborg, this may sound cryptic :) I am open to walking folks through the proposal in a call if needed14:38
SundarHoward, do you want me to explain the crux of the proposal here?14:39
Li_LiuSundar, regardless, we still need to track individual assignment for each device in Cyborg, right?14:39
zhipengSundar it would be great to have a brief walk through and the central take aways :)14:40
SundarLi_Liu: Yes, we will be handling acceleratr -PF/VF matching14:40
SundarThat is not involve din the scheduling, because Placement counts accelerator resources14:40
SundarAs instances are spawned and terminated, Cyborg agent can track PF/VF usage14:41
SundarHoward, sure.14:41
SundarThe brief summary is this:14:41
*** shaohe_feng_ has joined #openstack-cyborg14:42
Melissa_SSundar - I am open to scheduling a call and walkthrough of your proposal.14:42
SundarWe want to expose FPGAs in 2 ways: FPGA as a Service (FPGA aaS), and Accelerated Function as a Service (AFaaS)14:42
SundarFor FPGAaaS, user/flavor asks for a device type (or region type) and optionally a bitstream ID that needs to be applied by Cyborg before the instance comes up.14:43
SundarThis is similar to Amazon flow.14:43
SundarFor AFaaS, the user/flavor asks for a function/algorithm e.g. 'ipsec' + some indication of what device family the VM has the drivers for14:44
SundarTo cover both cases, we say that an accelerator can be a device/region or a function14:44
SundarWe represent a generic accelerator with the custom resource class (RC) CUSTOM_ACCELERATOR14:45
SundarWe also represent FPGAs and their inner regions as nested Resource Providers (RPs)14:45
SundarSo, a region RP can provide N instances of a CUSTOM_ACCELERATOR class. Most commonly, N=114:45
SundarAlso, each region RP has traits: region type (e.g. CUSTOM_FPGA_<vendor>_REGION_...)14:46
Sundarpossibly function type (e.g. CUSTOM_FPGA_<ipsec_uuid>)14:47
Sundarand device family (e.g. CUSTOM_FPGA_XILINX_<product/part>...)14:47
Li_LiuSundar, for AFaaS, on top of asking for a function/algorithm, I think users can also specify the minimum kpi/capability for the requested resources14:47
Li_Liuyou keep going14:48
Li_LiuI just throw some of my thoughts here14:48
SundarLi_Liu, NP, I'll get back to that14:48
SundarWith this background, here's how a flavor can ask for FPGA aaS:14:48
Sundarresource:CUSTOM_ACCELERATOR=1; trait:CUSTOM_FPGA_<vendor>_REGION_<uuid>=required14:49
Sundaroptionally, one more extra spec: bitstream:3A56D4=required14:49
SundarThis gets Placement to choose all matching devices based on trait. Once  a node is selected, the Cyborg agent in the node notes the extra spec and applies the bitstream14:50
SundarBefore I go to AFaaS, does this make sense for FPGA aaS/14:50
zhipengit does for me, this is awesome Sundar14:50
Li_Liuyup, it looks great14:51
SundarCool :) Now for AFaaS flavor14:51
chucksongIt sounds good Sundar. I need to know more about this project though. I just joined this project and still have a lot to learn14:51
DuthYes agreed. Sundar, can you share some more information on this proposal for FPGA aaS/14:52
zhipengDuth let's schedule another dedicated call for it14:52
Sundarresource:CUSTOM_ACCELERATOR=1; trait:CUSTOM_FPGA_INTEL_<ipsec-uuid>=required; trait:CUSOTM_FPGA_INTEL_<product>=required14:52
zhipengvideo conf would be better for a deep dive walk through14:52
SundarYes, we can do more in a video call14:53
zhipengSundar plz carry on the AFaaS scenario14:53
SundarSome folks don't like the UUIDs. I think UUIDs make it very concrete and we can make it more user-friendly later. Also, for AFaaS, we can avoid region UUIDs and use just product name14:54
SundarOK, back to AFaaS :)14:54
zhipengstring name could do the trick ?14:54
zhipengsorry go ahead14:54
SundarZhipeng: We can discuss strings in more deatail, may be in an email?14:56
zhipengSundar absolutely14:57
*** Kevin_Zheng has joined #openstack-cyborg14:57
zhipengplz carry on, sorry for the interruption :)14:57
SundarOK. AfFaaS flavor: resource:CUSTOM_ACCELERATOR=1; trait:CUSTOM_FPGA_INTEL_<ipsec-uuid>=required; trait:CUSOTM_FPGA_INTEL_<product>=required14:57
SundarThis picks all devices that have the required function. Makes sense, except that if no free instance of that function is available, the request will just fail, rather than have Cyborg pick an available region and reprogram it14:58
SundarThat is ok if that's what the operator wants: he may want to prevent reprogramming for whatever reason.14:59
SundarBut, in the general case, we want the ability to reprogram if needed.14:59
SundarSo, what we want is: trait:CUSTOM_FPGA_INTEL_<ipsec-uuid>=preferred (not required)15:00
SundarBut Nova does not support preferred traits today. I am told that it is not even close15:00
SundarSo, the next best thing: resource:CUSTOM_ACCELERATOR=1; trait:CUSOTM_FPGA_INTEL_<product>=required15:01
Sundaranother extra spec: function:CUSTOM_FPGA_INTEL_<ipsec-uuid>=required15:01
SundarNow, Placement chooses all devices that match the product, whether or not they have the function15:01
SundarA Cyborg weigher can check the allocation candidates to see which ones have the function -- based on the function trait -- and rank them higher15:02
SundarSo, Nova is likely to pick a device that has the function. if not, Cyborg agent in the compute node  will note the requested function is not present in the selected device/region RP15:03
SundarIt will contact Glance to get a matching bitstream and program it, before the instance comes up15:03
zhipengwhere would the weigher be implemented ?15:03
SundarThis is the core of the proposal, and covers situations where each bitstream implements only one function15:04
SundarZhipeng: the weigher would initially be a Cyborg weigher in my understanding -- but it is not Cyborg-specific. It could possibly become Nova in-tree hopefully15:04
SundarThe weighe ris looking at a list of RPS and choosing those with a specific trait. It is generic15:05
zhipengi mean the weigher will be part of the agent ?15:05
SundarIt is a weigher in the Nova controller -- like all other filters/weighers in Nova framework15:06
Li_LiuThe weigher has to be in Cyborg Controller I think15:06
shaohe_feng_yes, it should. And seems nova agree on the weigher15:07
SundarLi_Liu: OK, what I meant is, it runs in the controlle ralong with Nova/Cyborg. The operator must update nova.conf to use this weigher15:07
Li_LiuRight :)15:07
shaohe_feng_yes. just a config option in nova15:08
zhipengSundar I think it is definitely fine to update the spec patch based upon the current discussion conclusion15:08
zhipengand let's schedule another video conf for a detailed discussion with Xilinx team15:09
SundarThe part of this proposal which needs further review from Nova is when one bitstream has multiple functions, say crypto and compression. I think that is for the future and may not be needed in rocky. Does that sound agreeable?15:09
zhipengto see if there is any further improvements15:09
shaohe_feng_but weigher may be not very high priority. It is help to speed up the creation of a VM. but no helpful for the VM performance after VM start15:09
SundarThanks, Howard!15:09
zhipengSundar yes that would be maybe next release :)15:09
edleafeJust an FYI - it's not likely that nested RPs will be complete in Rocky15:10
zhipengshalhe_feng_ agree, let's implement the custom rc and traits first15:10
zhipengedleafe it is possible that we try with the nrp first right ?15:10
edleafezhipeng: sure, but it looks like the earliest nrp will be available will be in Stein15:11
Sundaredleafe: There is a release notes in Queens for nRPs, right?
edleafeSundar: that is the very beginnings of the structural changes needed for nrp15:12
shaohe_feng_edleafe: so cyborg just can support a fpga resource class in node provider in this release?15:13
edleafethe full model we need is still far away15:13
Sundaredleafe: The backup optoion would be to apply the RCs and traits to the compute node RP. But, when there are multiple devices in the same node, that can result in issues.15:13
edleafeshaohe_feng_: I'm not sure how that would work if you have multiple devices per node15:13
Sundaredleafe: We crossed. :)15:14
edleafeI'm on a call right now - I just wanted to set your expectations15:14
zhipengedleafe thx :)15:14
shaohe_feng_edleafe:  IMHO, we can support multiple devices later.15:14
zhipengyes let's be flexible15:14
zhipeng#action Sundar update the spec according to the ml discussion conclusion15:15
SundarWe can support multiple devices with some restrictions, which may satisfy immediate needs and still give freedom to operaors15:15
zhipengI would especially thanks Sundar for his initiative on the mailing list15:15
*** dolpher has quit IRC15:16
shaohe_feng_Sundar: any code plan for the spec?15:16
zhipengand also the spec discussion with Nova team15:16
Li_LiuOr can we put a simple version of multiple device support in Cyborg for now?15:16
Sundarshaohe, yes.15:16
zhipengshaohe-feng_ your PoC code could be used right ?15:16
SundarThanks, Howard15:16
shaohe_feng_zhipeng: yes. I think so.15:16
SundarThe POC code does not publish RCs and traits15:16
SundarBut we can build on that15:17
zhipengyes that's what i meant15:17
shaohe_feng_Sundar: it publish15:17
SundarAlso, the notion of using PFs and VFs as resources is something the Nova/PTG folks didn;t want ;)15:17
SundarShaohe: sorry, to clarify, it publishes PCI functions as RCs right?15:18
zhipenglet's take the details offline :)15:18
Sundarok :)15:18
shaohe_feng_Sundar: the poc is similar to nova teams conclusion.15:18
zhipengnext up, Li Liu's metadata spec15:18
zhipeng#info metadata standardization spec15:18
shaohe_feng_and we know nrp is not ready, so simply it.15:19
zhipengfolks plz review it15:19
zhipenganything you want to add, Li_Liu ?15:19
chucksongzhipeng, sorry for interrupt, what's the expected freeze date for the Xilinx driver spec? april 19 or Jun.4?15:19
zhipengchucksong Jun 415:19
Li_Liuzhipeng, I made some modifications based on shaohe's comments couple days ago. Waiting for more suggestions15:19
zhipengLi_Liu okey :)15:20
chucksonggood, thanks!15:20
zhipengokey moving on15:20
Li_LiuMy next spec for programmability is on the way. within the week I think15:20
zhipengLi_Liu you are the rock star man15:20
SundarLi_Liu: I had some high level comments on the spec. We can discuss them in more detail when you want15:20
zhipengI will cry myself to sleep tonight :P15:20
zhipeng#info cyborg-spec setup15:21
Li_Liusure sundar, wehcat/skype/phone/email whatever you want man15:21
zhipengYumeng__ has been great to setup the cyborg-spec repo15:21
zhipengI think the current patch look good, so if plz any core give a +2, I will land it this week15:22
zhipengokey folks we still have planned specs missing for rocky15:23
zhipeng#info quota and os-acc spec still missing15:23
zhipengI will talk to indicidual owners to see how to push forward15:23
zhipengdeadline is less than three weeks away :)15:23
zhipeng#action Howard to track the missing quota and os-acc spec15:24
zhipeng#topic open patches that need attention15:24
*** openstack changes topic to "open patches that need attention (Meeting topic: openstack-cyborg)"15:24
zhipeng#info shaohe_feng's devstack fix15:24
Yumengsorry to interrup. Here is the more detailed clock-driver use case description link I sent in the mail list:
zhipenggreat work from shaohe, I will have zhuli help merge this by the end of the week15:25
shaohe_feng_zhipeng: this should be land first.  :)15:25
zhipengYumeng thx !15:25
YumengDriver proposal would be provided next.15:25
zhipengYumeng I think you could directly put it up as a driver spec :P15:25
shaohe_feng_shaohe_feng_: It can help to avoid other debugs coming.15:26
Yumengzhipeng: okey.that would be great.15:26
openstackgerritLi Liu proposed openstack/cyborg master: Implemented the Objects and APIs for vf/pf
zhipengshaohe_feng_ I will nag zhuli :P15:26
zhipengnext, which Li_Liu just updated15:26
zhipeng#info object and apis for vf/pf15:27
Li_Liulol.. I just addressed some comments from Shaohe yesterday15:27
zhipengOkey I think that is all the important stuff we need to discuss today15:28
Li_LiuSundar, I hear you said Nova folks does not like vf/pf representations15:28
*** Duth has quit IRC15:28
shaohe_feng_Li_Liu: it is internal implementation, Why do not like?15:29
Li_Liubut I think vf/pfs are more friendly to vendor drivers. And it can be used interchangeably with Deployables15:29
shaohe_feng_Li_Liu: yes.15:29
SundarLi_Liu: yes, this is PTG feedback: e.g. Line 2115:30
shaohe_feng_Li_Liu: but your vf/pf representations is not expose15:30
Li_Liushaohe_feng_, I know, just wanna point it out, in case nova folks have questions on them. :)15:30
SundarOur implementation still uses PFs/VFs of course15:30
SundarWe can correlate regions/functions with PFs/VFs15:31
zhipeng#topic Vancouver Forum15:31
*** openstack changes topic to "Vancouver Forum (Meeting topic: openstack-cyborg)"15:31
zhipengalmost forget this15:31
zhipengplz feel free to propose forum topics for Vancouver Summit15:32
zhipengI might not be able to attend that Summit due to several reasons15:32
Melissa_Swhat is the vancouver summit?15:32
zhipengLi Liu will be my double there :P15:32
zhipengOpenStack Vancouver Summit15:32
Melissa_SChuck went to bed and I am here to keep track of actions :)15:33
*** chucksong has quit IRC15:33
Melissa_She is in China right now15:33
zhipengwould love to chat if he travel to shenzhen15:34
zhipengor beijing next week15:35
Melissa_Soh ok - I will let him know15:35
Melissa_Swhat email do I use for you?15:36
zhipengmy cell is +86-1857665896615:37
Melissa_Sokay - I sent you his cell phone number and our info via email15:40
shaohe_feng_zhipeng:  if not merged, I can not create the client repo?15:40
SundarZhipeng, shall we propose a talk on Cyborg architecture?15:41
zhipengyes shaohe_feng_15:42
shaohe_feng_zhipeng: it is ready.  wait for merge.15:42
zhipengSundar definitely a good idea :)15:42
shaohe_feng_I have tried to push the repo, but failed15:43
SundarCool ;)15:43
*** goldenfri has joined #openstack-cyborg15:45
zhipengokey if there is no other topics15:48
zhipenglet's conclude the meeting today :)15:48
Melissa_Ssounds good - any additional actions for us?15:48
zhipengnuh just keep in touch would be great :)15:50
zhipengoh one more thing, kosamara will you have the bandiwidth to work on the GPU spec ?15:54
zhipenglet me just close the meeting first in case I sleep over it again :P15:57
*** openstack changes topic to "OpenStack Cyborg Project Discussion"15:57
openstackMeeting ended Wed Apr  4 15:57:33 2018 UTC.  Information about MeetBot at . (v 0.1.4)15:57
openstackMinutes (text):
*** zhipeng has quit IRC15:59
*** Li_Liu has quit IRC16:01
*** circ-user-ldzTg has quit IRC16:19
*** Sundar has quit IRC16:43
*** Helloway has quit IRC16:55
*** Kevin_Zheng has quit IRC17:07
*** Yumeng__ has quit IRC17:57
*** shaohe_feng_ has quit IRC18:38
*** Melissa_S has quit IRC19:21
*** edleafe- has joined #openstack-cyborg23:31
*** edleafe has quit IRC23:33
*** edleafe- is now known as edleafe23:33

Generated by 2.15.3 by Marius Gedminas - find it at!