Tuesday, 2017-08-29

*** edmonds__ has joined #openstack-powervm00:00
*** edmondsw_ has quit IRC00:01
*** edmondsw_ has joined #openstack-powervm00:01
*** edmondsw_ has quit IRC00:01
*** edmondsw_ has joined #openstack-powervm00:02
*** edmonds__ has quit IRC00:04
*** edmondsw_ has quit IRC00:26
*** edmondsw has joined #openstack-powervm00:29
*** edmondsw has quit IRC00:31
*** edmondsw has joined #openstack-powervm00:31
*** edmondsw has quit IRC00:36
*** thorst_afk has joined #openstack-powervm00:51
*** thorst_afk has quit IRC00:59
*** edmondsw has joined #openstack-powervm01:40
*** edmondsw has quit IRC01:44
*** apearson has joined #openstack-powervm01:59
*** thorst_afk has joined #openstack-powervm02:00
*** thorst_afk has quit IRC02:06
*** thorst_afk has joined #openstack-powervm03:01
*** thorst_afk has quit IRC03:06
*** apearson has quit IRC03:21
*** thorst_afk has joined #openstack-powervm04:02
*** thorst_afk has quit IRC04:07
*** thorst_afk has joined #openstack-powervm05:03
*** thorst_afk has quit IRC05:08
*** thorst_afk has joined #openstack-powervm06:03
*** thorst_afk has quit IRC06:09
*** k0da has quit IRC06:45
*** thorst_afk has joined #openstack-powervm07:05
*** thorst_afk has quit IRC07:10
*** edmondsw has joined #openstack-powervm07:40
*** edmondsw has quit IRC07:45
*** edmondsw has joined #openstack-powervm07:49
*** k0da has joined #openstack-powervm08:03
*** thorst_afk has joined #openstack-powervm08:05
*** thorst_afk has quit IRC08:10
*** edmondsw has quit IRC08:29
*** thorst_afk has joined #openstack-powervm09:06
*** thorst_afk has quit IRC09:11
*** edmondsw has joined #openstack-powervm09:35
*** edmondsw has quit IRC09:47
*** thorst_afk has joined #openstack-powervm11:07
*** thorst_afk has quit IRC11:12
*** edmondsw has joined #openstack-powervm11:48
*** chhavi has joined #openstack-powervm11:48
*** edmondsw has quit IRC11:52
*** thorst_afk has joined #openstack-powervm12:06
*** miltonm has joined #openstack-powervm12:23
*** apearson has joined #openstack-powervm12:52
*** esberglu has joined #openstack-powervm13:02
openstackgerritMatt Rabe proposed openstack/nova-powervm master: Change NVRAM manager and Swift APIs to accept UUID  https://review.openstack.org/47192613:02
*** esberglu has quit IRC13:02
*** esberglu has joined #openstack-powervm13:03
efriedesberglu's IRC client is going haywire; he asked me to...13:03
efriednever mind13:04
esberglu#startmeeting powervm_driver_meeting13:04
openstackMeeting started Tue Aug 29 13:04:01 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.13:04
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:04
openstackThe meeting name has been set to 'powervm_driver_meeting'13:04
efriededmondsw is at VMWorld.13:04
efriedthorst_afk - you going to be here?13:04
thorst_afknot really13:04
thorst_afkwill be catching up on the feed periodically13:05
esberglu#link https://etherpad.openstack.org/p/powervm_driver_meeting_agenda13:05
esberglu#topic In Tree Driver13:05
esberglu#link https://etherpad.openstack.org/p/powervm-in-tree-todos13:05
efriedOkay.  I guess it'll be mostly a status update.  esberglu when done, send link to minutes out so others can catch up.13:05
esbergluI tested the config drive patch with FORCE_CONFIG_DRIVE=True13:06
esbergluEverything seemed to be working as expected13:06
efriedSweet.  Have you seen my email?13:07
esbergluefried: Yep. Ran it last night, 0 failures13:07
efriedWell, so here's the funky thing...13:07
esbergluNeed to port that to the IT patch13:07
efriedhttps://review.openstack.org/#/c/498614/ <== this passed our CI.13:07
efriedIt oughtn't to have.13:07
esbergluefried: Weird...13:08
efriedWe would have been sending down AssociatedLogicalPartition links like http://localhost:12080/rest/api/uom/ManagedSystem/None/LogicalPartition/<real_uuid>13:08
efriedBut... perhaps the REST side is just parsing the end off without looking too closely at the rest of it.13:09
efriedThe only side effect I can think of would have been that we would be ignoring fuse logic in mappings13:09
efriedwhich just means every mapping ends up on its own bus.13:09
efriedwhich wouldn't manifest any problems in the CI, really.13:09
esbergluefried: Want me to post the CI logs from the manual CI run to see if there's anything different going on?13:10
efriedNo, if it passes, we won't be able to see anything useful in there.13:10
efriedAnyway, yeah, esberglu you want to pick up those two changes and run with 'em?13:11
efriedfinish up UT and whatnot?13:11
esbergluefried: Sure13:11
esberglu#action esberglu: Port host_uuid change to IT config drive patch13:11
esberglu#action esberglu: Finish UT for OOT host_uuid patch13:11
esbergluthat's it for IT?13:12
efriedFor completeness, the pypowervm side is 5818; the nova-powervm change to be finished up and ported to the in-tree cfg drive change is https://review.openstack.org/#/c/498614/13:12
efried#action esberglu UT for 5818 too13:13
efriedIt's passing right now, but needs some extra testing for the stuff I changed.13:13
esbergluefried: ack13:13
efriedOh, and following up on vfc mappings.  I don't have anything fibre-channely to test with.  Do you?13:13
esbergluDon't think so13:14
efriedNeed to make sure the same logic (posting ROOT URIs to create mappings) also works for vfc, then make the same change in the vfc mapping bld methods.13:14
efriedCan be a followon change, I suppose13:15
efriedFor the moment, we're just using it to trim down the cfg drive stuff, which is always vscsi.13:15
esbergluefried: Ok. We can loop back to that after this first wave is done and either add it in or start a new change13:16
efriedyuh.  Perhaps someone from pvc can lend us a fc-havin system for a day or two.13:16
efriedmdrabe you got anything like that?13:16
mdrabeBut as far as lending I'm not sure :/13:17
efriedI actually would literally need it for half an hour13:17
mdrabeAll be consumed for pvc stories atm, in a few days they should be free I think13:17
efriedand would need a free fc disk I could assign to an LPAR (even the nvl)13:17
efriedAssuming that disk is free, the testing would be nondestructive.13:18
efriedI just need to create a mapping in a certain way and make sure it works.13:18
mdrabeTesting with devstack though right?13:18
efriedjust pypowervm13:18
efrieddon't even care what level13:18
mdrabeoh mkay13:18
mdrabeThat's good then, dm me13:19
efried#action efried to validate vfc mappings work the same way (with ROOT URIs for AssociatedLogicalPartition) using mdrabe's setup.13:19
efried#action efried to continue thorough review of cfg drive change13:20
efriedAt some point I reckon we're gonna need thorst_afk to review it too.13:20
efriedAll three of us have had hands on it, so approval is going to be like supermajority consensus.13:20
efriedOh, wait, this is in tree.13:21
efriedSo we all just get to +1 it anyway.13:21
efriedAnd community approval is going to entail...13:21
efried#action esberglu to drive pvm in-tree integration bp for q.13:21
esbergluefried: Yep13:22
efriedNot sure if you caught my parting shot yesterday on that, but: may want to ask mriedem in -nova whether he wants a fresh bp or just re-approve existing.13:22
esbergluefried: Yep saw that was planning on putting that in motion today13:22
esberglu#topic Out Of Tree Driver13:23
mdrabehttps://review.openstack.org/#/c/471926 passed functional testing13:23
mdrabeThere's one issue uncovered from the testing left to be ironed out, but it's unrelated to the change13:23
efriedWhat was the issue?13:24
efriedLove it when test finds bugs.13:24
efriedIt kinda justifies the whole existence of testing as a thing.13:24
mdrabeOne evacuation failed with the dreaded vscsi rebuild exception...13:25
efriedgot bug?13:25
mdrabeCan't link here, dming, sec13:26
efriedIf it's RTC, I don't care.13:26
mdrabeHeh k13:26
efriedIs it not a bug in the community code?13:26
efriedIf so, we ought to have a lp bug for it.13:26
mdrabeIt's been some time since I've looked at it13:26
efriedoh, is it an old bug?13:27
* efried confused13:27
mdrabeNo, I just mean within a weeks timeframe13:27
mdrabeI forget things quickly, sorry13:27
efriedmdrabe Okay, well, I'm not in a huge hurry to get a lp bug opened, but if the changes are going into nova-powervm, that should happen eventually (before we merge it).13:28
mdrabeefried: The exception that was raised was this one: https://github.com/powervm/pypowervm/blob/develop/pypowervm/tasks/slot_map.py#L66513:28
mdrabeFor 1 out of 5 evacuations13:29
efriedAs in, we couldn't find one of the devices on the target system?13:29
efriedSo first of all, 1/5 ain't good.13:29
mdrabeAnd I _think_ I recall seeing LUA recovery failures in the logs13:29
efriedSecond, upon what are you basing your assertion that this is unrelated to your change?13:30
mdrabeBecause it's not related to the slot map13:30
efriedeven though ten out of the 13 or so LOC leading up to that exception have 'slot' in 'em?13:31
mdrabeI'll -1 WF until we resolve it13:32
mdrabeefried: fair?13:32
efriedI had put a +2 on it, but yeah, I think we should follow up first.13:32
esbergluReminder that pike official release is tomorrow13:34
esbergluThat it for OOT?13:34
efriedother than pci stuff, I think so.13:34
esberglu#topic PCI Passthrough13:35
efriedLots to catch up on here since last week.13:35
efriedFirst of all, last week I got a prototype successfully claiming *and* assigning PCI passthrough devices during spawn.13:36
efriedWere any of y'all in the demo on Friday?13:36
efriedThe nova-powervm code is here: https://review.openstack.org/#/c/496434/13:37
efriedAnd I'm actually not sure ^ relies on any pypowervm or REST changes, as currently written.13:38
efrieddespite what the commit message says.13:38
efriedREST has merged the change that lets us assign slots on LPAR PUT.  Which means I can remove the hack here: https://review.openstack.org/#/c/496434/3/nova_powervm/virt/powervm/vm.py@57313:39
efriedAlso the much-debated PCI address spoofing I think I'm gonna keep in nova-powervm (abandoned 5755 accordingly) because...13:40
efriedAll of this is going to be temporary13:40
efriedIt may not even survive queens, gods willing.13:40
mdrabeefried: I forget, through what API do we assign PCI devices after spawn?13:41
efriedmdrabe Before that REST fix?  IOSlot.bld and append that guy to the LPAR's io_config.io_slots.  Then POST the LPAR.13:41
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: ci check  https://review.openstack.org/32831513:42
mdrabeefried: And that's triggered by an interface attach from an openstack perspective?13:42
efriedmdrabe No, actually, I'm not sure what happens during interface attach - should probably look into that.13:43
efriedNo, in openstack the instance object we get passed during spawn contains a list of pci_devices that have been claimed for us.13:43
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: CI Check2  https://review.openstack.org/32831713:43
efriedVia the above change sets, we're culling that info and sending it into LPARBuilder (curse him).13:44
efriedmdrabe Is that what you were looking for?13:44
mdrabeI'm just trying to understand the flows affected13:45
efriedSure, definitely worth going over in more detail, let's do that.13:45
mdrabeYea, I've been meaning to take some time to stare at this stuff, I'll probably ask better questions after I do that13:46
efriedNova gets PCI dev info from three places:13:46
efried => get_available_resource (in the compute driver - code we control) produces a list of pci_passthrough_devices as part of the json object it dumps.13:46
efried => The compute process looks in its conf for [pci]passthrough_whitelist, which it intersects with the above to filter down to only devices you're allowed to assign to VMs.13:47
efried => The nova API process looks in its conf (which may not be the same .conf as the compute process - took me a while to figure THAT one out) for [pci]alias entries, which it *also* uses to filter the above.13:48
efriedThe operator sets up a flavor.  In the flavor extra_specs he sets a field called pci_passthrough:alias whose value is a comma-separated list of <alias>:<count>13:48
*** edmondsw has joined #openstack-powervm13:49
efriedThe <alias> names come from the [pci]alias config, and are how the op identifies what kinds of devices he wants on his VM.  Those [pci]alias entries just map the alias name to a vendor/product ID pair.13:49
efriedAnd the <count> is how many of that kind of dev you want.13:49
efriedWhen you do a spawn with that flavor, nova looks at the pci_passthrough:alias in the flavor, maps it to the vendor/product ID, and then goes and looks in the filtered-down pci_passthrough_devices list for devices that match.13:50
efriedMeanwhile it's keeping track of how many of those kinds of devices it has claimed and whatnot.13:51
mdrabeOk so adding/removing PCI devices is triggered through resize13:51
efriedSo assuming it finds suitable devices, it decrements their available count and assigns 'em to your instance.13:51
efriedYes, I believe that's the case, though I haven't explicitly tried it yet.13:51
mdrabeThat makes me wonder how this works with SR-IOV13:52
efriedTo come full circle: nova puts the specific devices it claimed into your instance object that it passes to spawn, which is where our code again gets control.13:52
efriedYeah, SR-IOV is going to be a different story13:52
efriedEspecially since we're not doing the same thing nova does with SR-IOV.13:52
efriedBut much of the flow is the same.13:53
*** edmondsw has quit IRC13:53
efriedpci_passthrough_devices is *supposed* to register each VF as a child of its respective PF.13:53
efriedSo you could claim a VF and the matching is done based on the parent.13:54
efriedBut when you're doing that as part of network interface setup, things go off the rails a bit.13:54
efriedNow it starts looking for a physical_network tag on your device and trying to bind a neutron port with that network and all that jazz.13:55
efriedIn the rest of the world, you have to pre-create VFs, and they're passed through explicitly one by one and assigned directly to the VM.13:55
efriedIn our world... we don't have the VFs until we need 'em, and even then, they're not assigned directly to the VM.13:56
efriedSo we have to fool the pci manager by spoofing "fake" VFs in our pci_passthrough_devices list.  We just create however many entries according to the MaxLPs on the PF.13:56
mdrabeRight okay, I'm stuck in the PowerVM perspective13:57
efriedYeah, so when we do a claim with SR-IOV, nova actually hands us one of those fake VFs, but we ignore it and just create our VNIC on the appropriate PF.13:58
efriedThis is probably enough historical treatise.  The aforementioned PoC code gives me confidence that we can make this work in q without community involvement.  Which is not bad.13:58
efriedBut it also ain't pretty.13:59
efriedThe main ugliness is that we have to spoof our PCI addresses.13:59
efriedBecause nova refuses to operate without a Linuxy PCI address in <domain>:<bus>:<slot>.<func> format.13:59
efriedOur devices don't have those.  We have DRC index and location code.13:59
efriedLinuxy PCI addresses are 32-bit.  DRC index is 64-bit.14:00
mdrabeWhat determines the DRC index for us?14:00
efriedSo I started down a path of suggesting some changes to nova's pci manager that would allow us to use our DRC index (or location code, or whatever we wanted) to address and identify devices.14:01
efriedIt was basically shot down as being an interim hackup that would be superseded by the move to placement and resource providers.14:02
efriedWhich is really what I was going for in the first place.  I wanted to garner some attention and discussion that would get us moving in that direction.14:03
efriedThe upshot is that we (I believe Jay is the nova core most invested in this) want to make devices (not just PCI - any devices) managed through the placement and resource provider framework.14:04
efriedIn that nirvana, our compute driver provides a get_inventory method, which replaces get_available_resource.14:04
efriedThe information contained therein is able to represent any resource generically, and the nova code doesn't try to introspect values and do stuff with 'em like it is doing today for PCI addresses and whatnot.14:05
mdrabeThat sounds like the way to go14:05
efriedThat work is off the ground at this point in nova, for resources like vcpu, mem, and disk.14:06
efriedThere's also some support for custom resource classes.14:06
efriedJay and I are working up content for discussion at the PTG toward making devices managed by the same setup.14:06
esbergluGood discussion. We ready to move on?14:08
efriedA resource provider would describe the devices it has available; those devices would have qualitative and quantitative properties.  Nova would get a spawn request asking for a device with certain qualitative and quantitative properties.  Placement and scheduler and claims and family would just match those values (again, blindly, not introspecting the values) and give us the resources.14:08
efriedAnd we get the helm back in our driver and do whatever we want with those claimed resources.14:08
mdrabeI feel much more informed than I did an hour ago14:09
efriedSo my action this week is going to be collating some of these notes and stuff, creating an etherpad for the PTG, and perhaps putting some of it down in a blueprint https://blueprints.launchpad.net/nova/+spec/devices-as-resources whose spec is here: https://review.openstack.org/#/c/497978/14:10
mdrabeefried: Is the resource provider change targetted for q?14:10
efriedWell, that's what I don't know.14:11
efriedI'm sure it will be targetted for q.14:11
efriedWhether it will get done in q is another question.14:11
efriedWe need to be prepared to move forward with our hacked version14:11
efriedAnd we can transition over as able.14:11
efriedIt's a big piece of work.14:11
efriedSo I suspect that even if it gets done in q, it'll get done late in the cycle, possibly too late for us to exploit it fully ourselves.14:12
efriedThe really good news here is that Jay is very invested in this, and it fits with the overall direction nova is moving wrt placement and resource providers, so I don't doubt it's going to get done... eventually.14:12
efriedIt's not just us whining "we need this for PowerVM".14:12
efriedOkay, I think that's probably enough of that for now.  Any further questions, or ready to move on?14:13
efried#action efried to write etherpad and/or spec content for nova device management as generic resources.14:14
esbergluI might have questions later, I need to look through the code still14:14
esberglu#topic PowerVM CI14:14
esbergluNot much to report here14:14
esbergluStill waiting for the REST change for the serialization issue14:15
efriedesberglu It's been prototyped, though?14:15
efriedAnd run through CI?14:15
esbergluefried: Prototyped and run through CI, but not with the latest version of the code14:15
esbergluefried: I think it requires the related changes as well. Not 100% sure though, hsien deployed it14:17
esbergluOther than that the compute driver was occaisionally failing to come up on CI runs. The stacks on the undercloud for a few systems were messed up14:18
esbergluI redeployed, haven't seen it since, gonna keep an eye out14:19
esbergluThose were the only failures hitting CI consistently, so failure rates should be pretty low now14:20
esbergluWell not now, once that rest fix is in14:20
esbergluThat's all I had CI14:20
esberglu#topic Driver Testing14:20
esbergluJay isn't on. But he was having problems stacking last week14:21
esbergluI got his system stacked, not sure if any further testing has been done on it yet14:21
esbergluNothing else to report there14:22
esberglu#topic Open Discussion14:22
esbergluThat's it for me14:22
efriednothing else here14:22
esbergluAlright. See you here next week14:23
openstackMeeting ended Tue Aug 29 14:23:50 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:23
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-08-29-13.04.html14:23
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-08-29-13.04.txt14:23
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-08-29-13.04.log.html14:23
*** edmondsw has joined #openstack-powervm14:42
*** edmondsw has quit IRC14:54
*** edmondsw has joined #openstack-powervm14:58
*** edmondsw has quit IRC15:02
*** k0da has quit IRC15:11
*** edmondsw has joined #openstack-powervm15:54
openstackgerritMatt Rabe proposed openstack/nova-powervm master: Change NVRAM manager and Swift APIs to accept UUID  https://review.openstack.org/47192615:55
*** edmondsw_ has joined #openstack-powervm15:58
*** edmondsw has quit IRC15:58
*** edmondsw_ has quit IRC16:00
*** thorst_afk has quit IRC16:01
*** thorst_afk has joined #openstack-powervm16:05
*** thorst_a_ has joined #openstack-powervm16:28
*** tjakobs has joined #openstack-powervm16:30
*** thorst_afk has quit IRC16:32
*** efried has quit IRC16:40
openstackgerritTaylor Jakobson proposed openstack/nova-powervm master: Add resize attached volume support  https://review.openstack.org/49886416:43
*** thorst_a_ has quit IRC16:51
*** thorst_afk has joined #openstack-powervm16:52
*** efried has joined #openstack-powervm16:53
*** thorst_afk has quit IRC16:56
*** thorst_afk has joined #openstack-powervm17:04
*** k0da has joined #openstack-powervm17:17
*** k0da has quit IRC17:29
*** edmondsw has joined #openstack-powervm17:54
*** edmondsw_ has joined #openstack-powervm17:58
*** edmondsw has quit IRC17:59
*** edmondsw_ has quit IRC17:59
*** edmondsw has joined #openstack-powervm18:00
*** edmondsw has quit IRC18:07
*** chhavi has quit IRC18:23
*** edmondsw has joined #openstack-powervm18:23
*** edmondsw has quit IRC18:27
openstackgerritSatish Yadav proposed openstack/nova-powervm master: Add enable LPAR metric parameter  https://review.openstack.org/49890218:28
*** edmondsw has joined #openstack-powervm18:41
*** edmondsw has quit IRC18:45
*** edmondsw has joined #openstack-powervm19:03
*** edmondsw has quit IRC19:04
*** edmondsw has joined #openstack-powervm19:07
*** edmondsw has quit IRC19:08
efriedesberglu thorst_afk Can I get a second look at 5806 please?19:39
openstackgerritEric Berglund proposed openstack/nova-powervm master: Remove host_uuid from config drive paths  https://review.openstack.org/49861419:40
esbergluefried: Sure19:40
*** k0da has joined #openstack-powervm19:43
esbergluefried: LGTM19:53
efriedesberglu thx19:53
*** edmondsw has joined #openstack-powervm19:55
*** edmondsw has quit IRC19:56
efriedesberglu You working on 5818?19:56
esbergluefried: Yeah19:58
*** edmondsw has joined #openstack-powervm20:08
*** edmondsw has quit IRC20:13
*** thorst_afk has quit IRC21:22
*** thorst_afk has joined #openstack-powervm21:23
*** apearson has quit IRC21:24
esbergluefried: I removed the WIP for removing VFC mappings from 5818. I was thinking a separate patch. Cool with you?21:24
esberglu* for removing host_uuid from the VFC mappings21:25
*** thorst_afk has quit IRC21:27
*** k0da has quit IRC21:28
*** esberglu has quit IRC21:43
*** esberglu has joined #openstack-powervm21:44
*** k0da has joined #openstack-powervm21:44
*** esberglu has quit IRC21:48
*** thorst_afk has joined #openstack-powervm21:49
*** thorst_afk has quit IRC21:53
*** esberglu has joined #openstack-powervm21:57
efriedesberglu Yes, cool.22:00
efriedStill need some more UT in that guy, I think.22:01
efriedIt may be overkill, but something proving host_uuid is unused in all those bld methods.22:01
*** esberglu has quit IRC22:01
*** edmondsw has joined #openstack-powervm22:02
*** edmondsw has quit IRC22:04
*** edmondsw has joined #openstack-powervm22:04
*** miltonm has quit IRC22:07
*** esberglu has joined #openstack-powervm22:07
*** tjakobs has quit IRC22:08
*** edmondsw has quit IRC22:23
*** esberglu has quit IRC22:27
*** esberglu has joined #openstack-powervm22:27
*** esberglu_ has joined #openstack-powervm22:28
*** esberglu has quit IRC22:31
*** esberglu_ has quit IRC22:35
*** edmondsw has joined #openstack-powervm22:41
*** edmondsw has quit IRC22:45
*** edmondsw has joined #openstack-powervm22:58
*** edmondsw has quit IRC23:03
*** k0da has quit IRC23:13

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!