Tuesday, 2017-04-04

*** thorst has quit IRC00:31
*** thorst has joined #openstack-powervm00:31
*** thorst has quit IRC00:35
*** esberglu has quit IRC00:44
*** esberglu has joined #openstack-powervm00:44
*** esberglu has quit IRC00:49
*** esberglu has joined #openstack-powervm00:51
*** esberglu has quit IRC00:52
*** esberglu has joined #openstack-powervm00:52
*** esberglu has quit IRC00:56
*** thorst has joined #openstack-powervm01:07
*** thorst has quit IRC01:15
*** thorst has joined #openstack-powervm02:15
*** thorst has quit IRC02:35
*** jwcroppe has quit IRC02:50
*** jwcroppe has joined #openstack-powervm02:51
*** jwcroppe_ has joined #openstack-powervm03:11
*** jwcroppe has quit IRC03:13
*** esberglu has joined #openstack-powervm04:21
*** esberglu has quit IRC04:26
*** thorst has joined #openstack-powervm04:32
*** thorst has quit IRC04:37
*** shyama has joined #openstack-powervm04:38
*** esberglu has joined #openstack-powervm05:16
*** esberglu has quit IRC05:20
*** thorst has joined #openstack-powervm05:33
*** thorst has quit IRC05:38
*** tjakobs_ has joined #openstack-powervm05:53
*** tjakobs_ has quit IRC05:57
*** esberglu has joined #openstack-powervm06:10
*** esberglu has quit IRC06:15
*** openstackgerrit has joined #openstack-powervm06:18
openstackgerritShyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map  https://review.openstack.org/43232206:18
openstackgerritShyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map  https://review.openstack.org/43232206:19
openstackgerritGautam Prasad proposed openstack/ceilometer-powervm master: Adhere to updated Inspector interface  https://review.openstack.org/45302406:27
*** thorst has joined #openstack-powervm06:34
*** thorst has quit IRC06:38
*** thorst has joined #openstack-powervm07:34
*** thorst has quit IRC07:39
*** esberglu has joined #openstack-powervm07:59
*** esberglu has quit IRC08:04
*** k0da has joined #openstack-powervm08:35
*** thorst has joined #openstack-powervm08:35
*** thorst has quit IRC08:55
*** edmondsw has joined #openstack-powervm09:14
*** edmondsw has quit IRC09:18
*** esberglu has joined #openstack-powervm09:48
*** thorst has joined #openstack-powervm09:52
*** esberglu has quit IRC09:52
*** thorst has quit IRC09:56
*** k0da has quit IRC09:56
*** thorst has joined #openstack-powervm11:30
*** smatzek has joined #openstack-powervm11:52
*** edmondsw has joined #openstack-powervm12:06
*** jpasqualetto has joined #openstack-powervm12:11
*** jpasqualetto has quit IRC12:21
*** esberglu has joined #openstack-powervm12:31
*** jpasqualetto has joined #openstack-powervm12:34
a1fisherMorning guys, I'm seeking some clarification on getting a nova-link vm to talk to an existng openstack controller. I've had some success getting my controller to see nova-link and can almost deploy an instance from Horizon but I'd like to check my approach and clarify what openstack components are needed to be installed on nova-link. I installed nova-compute and neutron-openvswitch-agent. Is this the right approach or have I12:36
a1fishergone down the wrong path?12:36
*** esberglu has quit IRC12:36
a1fisherSorry, should have added that I have installed the networking-powervm and nova-powervm drivers (with  networking-powervm also going on the controller)12:39
thorsta1fisher: well, it really depends on the environment that you want to create.  Are you doing typical I/O for PowerVM (meaning a Shared Ethernet Adapter on the VIOS)?12:41
thorstor are you trying to do some SDN'y things?12:42
walerit's a simple config thort - single VIOS, single SEA, one SSP12:42
thorstwaler: OK - so then you need networking-powervm installed.  There is a pvm-sea-agent in there to run12:42
thorstalso, due to how neutron works, if you're not running Linux Bridge or OVS, then you need to also install the 'agent package' on the neutron server.12:43
waler(we kept things simple, in the hope it would avoid any additional complications)12:43
thorstso install networking-powervm on the controller as well.  You don't need to do anything with it there, it just tells the neutron server what a SEA is12:43
thorstwhat version of OpenStack are you using?12:43
thorstI ask because we have a known bug in trunk that efried is working on  :-(12:44
a1fisherthats been done. The bit I think I'm missing is running pvm-sea-agent. We're using Mitaka12:44
thorstMitaka should be fine.  I think around Ocata we did add the ability to use the Open vSwitch agent if you'd like.  But in Mitaka, stick with the SEA agent12:45
a1fisherso, all we need to do from a networking point of view is install networking-powervm and run the agent?12:45
thorstpretty much.12:45
thorstunless you have multiple SEAs12:45
walernot at the moment :)12:46
thorstif you have multiple SEAs, you need to start specifying the bridge mappings and what not12:46
a1fishercool, I'll give that a go.12:46
thorstdon't forget - install it on both the novalink and your controller12:46
thorstrun the agent just on the novalink12:46
thorstbut install the package on both12:46
thorst(and restart neutron-server after you installed it on the controller)12:46
walerthorst : does it add much to the OpenStack setup/config, if you go from a single VIOS, to a redundant pair of VIOS?12:47
a1fisherOK. regarding the nova-compute part, does that need to be on the nova-link vm?12:47
a1fisherthats the only way I can think that nova-link will talk to the controller12:47
thorstwaler: It depends.  But generally no.  Yes if you start doing multiples of things (multiple SSPs for instance)12:47
thorsta1fisher: nova-compute is meant to run on the novalink vm12:47
thorstit is not supported in any other fashion.12:47
a1fisherI'll give that a go and see far we can get12:48
walermany thanks thorst12:48
thorstgood luck!12:48
*** jwcroppe_ has quit IRC12:52
*** nbante has joined #openstack-powervm12:55
*** apearson has joined #openstack-powervm12:56
*** esberglu has joined #openstack-powervm12:57
esberglu#startmeeting powervm_driver_meeting13:00
openstackMeeting started Tue Apr  4 13:00:41 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
openstackThe meeting name has been set to 'powervm_driver_meeting'13:00
esberglu#topic Out Of Tree Driver13:03
esbergluocata is broken13:03
thorstesberglu: the upload thing?13:03
thorstcrap...how did that get back ported...13:03
thorstI thought we tested it earlier  :-/13:04
thorstin the staging env...13:04
esbergluSo CI is still down, I can redeploy again with newton or we can get this figured out and be without CI until then13:05
thorstesberglu: well it seems like we're going to need a new pypowervm for it13:05
esbergluI think I may have missed some of the convo yesterday13:05
thorstand we're going to have to bump that back in ocata...which is awful13:05
esbergluBut it looked like efried was expecting this13:05
thorstwell, efried thought this could occur.  I didn't think it could13:06
thorstit was based off of whether or not whatever broke us was back ported13:06
thorstefried will be on in 15 min and we can discuss more then13:06
esbergluDo you know what broke us?13:07
thorstthat's the irony...and what I'm frustrated about13:07
thorstwe're trying to fix for something that we don't know why we broke13:07
esbergluWell lets look at what got into ocata in the last week13:07
esbergluBecause I deployed ocata on staging at the end of last week no problem13:08
esbergluThere's nothing that has gone into ocata since I deployed on staging successfully13:10
esbergluin the 3 powervm projects13:10
thorstit'd be in nova itself13:10
esbergluYep I'm looking there now13:10
adreznecIt would have to be a bugfix at that point that broke us, right?13:10
adreznecI mean Ocata's been cut for some time now13:10
thorstI'd assume13:10
thorstnothing much in the past week there tho13:11
*** mdrabe has joined #openstack-powervm13:11
thorstthought maybe a global req change13:11
esbergluNone of that looks suspect13:11
thorstbut nothing much there13:11
thorstpbr updated...13:11
adreznecthorst: yeah, once ocata gets cut reqs are pretty much frozen13:11
adreznecUnless there's a major breaking issue13:11
*** jwcroppe has joined #openstack-powervm13:12
thorstesberglu: I'm now curious if newton is hosed.13:12
esbergluI hope not. That means CI is down for the count until this is resolved13:13
thorstadreznec: does concurrent.futures use greenlet or eventlet?13:13
thorstdo you know?13:14
adreznecthorst: not sure offhand13:15
adreznecisn't concurrent.futures a stdlib13:16
thorstadreznec: yeah, but that's where efried sees us hanging13:17
adreznecneither eventlet or greenlet are builtins, so it can't use either by default13:17
thorstso efried wants to switch to all eventlet I think...13:17
*** efried has joined #openstack-powervm13:18
thorstso the net is...I think that is now our highest priority13:18
efriedNot sure if I missed anything, but I have a plan for the broken upload in ocata.13:18
thorstand we should work it first oot13:18
thorstrather than it (so we don't create a bunch of misc reviews for the core side)13:18
thorstefried: net is, CI is down13:18
thorstwhat I think we're curious about is did something change in OpenStack...or was it perhaps even lower than that13:19
thorstlike eventlet or somewhere else13:19
adreznecthorst: is this futures in py2.7 or py3?13:19
esbergluStaging CI is still up. So if we need to run something through we can still do it there13:19
efriedFor OOT ocata, we need a bug opened; then we a) port to ocata OOT the change that moves from FUNC back to IO_STREAM; b) update the pypowervm requirement to 1.1.1.  And, of course, we'll need to release
adreznecI'll admit I'm not totally in the loop on the upload issue13:19
thorstefried: what is the 'fix'13:20
thorstdown in pypowervm13:20
efriedYeah, so something changed in eventlet recently - I still haven't nailed down exactly what, but sdague gave me some vague pointers last week.13:20
thorstwell shit13:20
adreznecefried: something that would have changed since ocata was released?13:20
thorstthat'll affect things way back13:20
efriedThe fix is two-sided, unfortunately.  In pypowervm, we have to kill coordinated upload.  In community, we have to kill FUNC.13:21
adreznecotherwise shouldn't it be pinned by version in reqs?13:21
efriedBecause there's no way to do FUNC without threads, and there's no way to do coordinated without threads.13:21
efriedThe alternative is to retool pypowervm to use eventlet instead of futures.13:21
thorstefried: so is concurrent.futures just dead?13:21
thorstbecause of a change in eventlet?13:22
efriedNo, it's just incompatible with greenlet.13:22
efriedAlthough that might not be entirely true.13:22
thorstso func is still viable, just not in an OpenStack env.13:22
thorstand it also calls into question if we need a change for your VIOS Task thingy13:22
thorstwhich also uses concurrent.futures13:22
efriedI don't disagree with that.13:22
thorstnon-disagreement is as close as we can hope for a resounding agreement13:23
thorstefried: so are you focused on that change today?13:23
thorstand we just keep CI down while we fix that?13:23
efriedWhich change?  Get rid of FUNC and coordinated in ocata OOT?13:23
efriedOr convert to greenlet?13:23
efriedPerhaps we should take a couple of minutes and go over what I (sort of, maybe) know so far about the underlying cause.13:24
thorstefried: yes, lets do that13:24
efriedSo from my research with mdrabe yesterday, I *think* it goes like this:13:24
efriedThere's two kinds of multiprocessing models available: threads and greenlets.13:24
adreznecSo just to clarify13:24
adreznecWhat version of pypowervm is this using13:25
adreznecWith Ocata13:25
efriedI don't fully understand the difference between them, but they're totally different animals, not just different implementations on top of the same underlying threading model.13:25
efriedOpenstack uses greenlets throughout.  They even have a hacking check in place to make sure you're using eventlet through their nova.utils wrapper of it.13:25
efried(adreznec, not sure, but to fix this we'll need to release 1.1.1 and bump the ocata req to that.  Is that even legal?)13:26
esbergluocata is using I believe13:26
adreznecWe can technically do that for Ocata13:26
adreznecI guess...13:26
adreznecin the future... please, let's never have to deal with that13:27
efriedYeah, the req bump will need to happen regardless of which way we fix this.  Unless we can figure out some as-yet-unknown way to fix it purely in the community code.13:27
adreznecI'm just wondering if this is only broken with some combination of versions13:27
adreznece.g. only with pypowervm 1.1.0 because that's where we require futures>=3.013:27
adreznecvs just "futures"13:28
adreznecwith no version req13:28
efriedMm.  And presumably openstack doesn't require futures?13:28
adreznecsorry, yes, they do13:28
adreznecthat's where the >3.0 req came from13:28
adreznec(walking to a meeting)13:29
efriedAnyway, threading in python apparently has this GIL (global interpreter lock) which actually makes it so that literally only one thread runs at a time - the others are stopped.13:29
efriedNormally this is okay because threads can yield and allow other threads to run, so as long as your actual programming doesn't have deadlocks in it, you're aaight.13:30
efriedThis sucker is blocking on a syscall.13:30
efriedWhich doesn't yield.13:30
efriedSo all the other threads - including the greenlets, including the one that would kick the REST server to do its open, which would unblock the write side - are frozen.13:31
efriedNow, there is apparently a way to explicitly release the GIL.13:31
efriedThat might be the least disruptive path, if we can figure out how to do it.  But a) it's going to be a hack (more on that in a bit), and b) it might not work in the context we would need to do it in - that is, it might only work if we can do it right at that open() call, which is in code we don't own.13:33
thorstefried: to me...lets just fix it proper...13:33
thorstgreenlet (not eventlet)?13:33
efriedYeah, so I don't know what the difference is there - those are different libs - but they use greenlets / green threads under the covers.13:35
efriedWhereas anything that says "thread" - like the native thread library, or concurrent.futures - uses the other kind of threads.13:35
thorstI'd assume we can use greenlets for most things...but the pipe may not be able to use greenlet...13:36
efriedThat's an unknown at this point.  But I imagine it's gotta be possible.13:37
thorstso the net is, due to this, we need to bump pypowervm...get a bug for nova-powervm...and possibly back port this way (enough) back13:37
efriedHowever, given that we're going to need to release a new pypowervm and bump the OOT req to it anyway, I would just as soon do the fix that avoids threading altogether.13:38
thorstwaler is testing on Mitaka, so he could actually probably tell us if Mitaka is impacted  :-)13:38
*** tjakobs_ has joined #openstack-powervm13:38
thorstefried: for the upload?  Sure.  VIOS Feed Task stuff...not so sure13:38
efriedthorst Agree, but I think what saves us there is that what's running in that thread is non-blocking.13:39
thorstalright...so I guess that's priority 1...13:39
thorstyep...I agree that is probably not highly impacted.13:39
efriedSo maybe it hitches the process while it's doing that POST, but as soon as the POST comes back, we keep truckin.13:39
thorstyeah, kinda ick, not uber ick13:40
efriedNot ideal, and perhaps something we should look into for the future, but not first priority, right.13:40
thorstalright...so gameplan build out here...13:40
thorst1) esberglu - would you be willing to make the bug and tag at least back to ocata13:40
thorstI honestly suspect that newton / mitaka may still be impacted...would love to know that if you have time to redeploy with newton13:41
thorst2) efried you're updating the pypowervm bits?13:41
thorst3) should I do the nova-powervm bits to swap off func?13:42
esbergluthorst: I should be able to do that in the background today13:42
efriedthorst Actually, let me do it.13:43
efriedTake a look at this delta: https://review.openstack.org/#/c/443189/15..16/nova/virt/powervm/disk/ssp.py13:43
efriedIt'll be like that, except we won't actually need the IterableToFileAdapter.13:43
thorstwe would need to basically revert into that change for both localdisk13:43
thorstand ssp13:43
thorstwhy not?13:43
efriedBecause I'm gonna make a change to pypowervm ;-)13:43
efriedSince we're going to need a new version of that anyway.13:43
efriedBackward compatible.13:44
thorstdon't make it even more complicated  :-)13:44
efriedBut eliminating the need for IterableToFileAdapter.13:44
efriedIt makes it less complicated, really.13:44
efriedThe HTTP request expects an iterable.  Glance gives us an iterable.  For some reason we had pypowervm expecting a file and converting it to an iterable, so the community had to convert the iterable to a file just so pypowervm could convert it back.13:45
efriedwhich is stupid.13:45
thorstwell, I guess I'll let you do magic and be on point to be a reviewer?13:45
thorsthow do we make actions in the meeting?13:46
thorst(for the meeting minutes)13:46
esberglu#action esberglu: Open bug for upload issue13:46
*** k0da has joined #openstack-powervm13:46
efriedI'm going to use 5083, which is already most of the way there.  Just need to add the iterable killer.13:48
thorst#action efried drive pypowervm and nova-powervm fixes for upload issue13:48
thorst#action esberglu determine if newton is impacted13:48
thorstefried: 5083 - we should tag that with the bug that esberglu is making13:50
thorstso we have one bug capturing this whole nightmare...13:50
thorst#action adreznec ship out a new pypowervm once this whole fiasco is solved  :-)13:50
adreznecmight need to sync that with julio13:50
thorstwhat else do we have for the meeting?13:53
esbergluCool sounds like we have a plan. Meeting is almost up, anyone have anything else? I don't have anything for CI13:53
esbergluefried: anything in-tree?13:53
esbergluJust waiting for reviewers at this point correct?13:53
thorstnbante and jay are still doing testing...  I know jay is hitting issues, I'm trying to help out once I get in the env.  nbante I thik is stuck on something with tempest in OSA.13:53
efriedI think by the time we hit the SSP change set, we'll need to bump the in-tree reqs to the new pypowervm.13:53
*** tjakobs_ has quit IRC13:54
nbantecorrect..adreznec: check once if we have to uncomment anything in user_config.yml to get that work13:55
adreznecnbante: you'll likely have to experiment on your own there today. I'm pretty much swamped in meetings until later this afternoon13:56
nbantesure..I already tried most of parts. But will give shot in 2-3 hours. If not work, will send you note13:57
esberglunbante: I may have time to take a look today. I will let you know13:58
thorstnbante: one thought I had...13:59
thorstI don't think we care if tempest is deployed via OSA13:59
thorstor run from a separate server...13:59
thorstyou have a cloud, we just need to run tempest against it13:59
thorstso that gives you options to try13:59
thorstbut esberglu is more familiar than I am...so maybe he'll figure it out in 2 mins14:00
nbanteI never tried tempest so not sure how it worked. In SVT, we have own framework14:00
*** jwcroppe has quit IRC14:00
esbergluthorst: I haven't got tempest working yet either.... so I'm pretty much in the same boat as nbante right now for OSA CI14:00
thorstesberglu: right...but my thought is14:00
thorstwe have tempest working for IT CI14:01
thorstor OOT CI14:01
thorstso...uh...how'd we set it up there?14:01
thorstand can we do the same here?14:01
*** jwcroppe has joined #openstack-powervm14:02
esbergluAt least some tweaks will be needed. We can discuss more when I really dive into it14:02
thorstnbante: are you deployed with Cinder?14:03
nbanteusing local disk only14:03
thorstso we're just getting back to where we were?14:03
nbanteafter local disk, I worked on tempest where I stuck14:04
nbantedo you want me to parallely work on configuring cinder?14:05
thorstahh, right...so we were getting tempest and then moving to iSCSI cinder14:05
thorstsorry, I'm getting my wires wrong  :-)14:05
thorstOK - I'll also catch up with Jay...14:06
thorstcan you check with him to get his IRC working?14:06
nbantesure..will check14:07
thorstok - I didn't have anything else.14:07
openstackMeeting ended Tue Apr  4 14:11:13 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:11
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-04-04-13.00.html14:11
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-04-04-13.00.txt14:11
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-04-04-13.00.log.html14:11
shyamathorst: efried https://review.openstack.org/#/c/432322 the is_volume_on_vios is used by slot.py to rebuild vscsi slots14:16
efriedshyama See new review14:34
*** mdrabe has quit IRC14:37
*** mdrabe has joined #openstack-powervm14:43
efriedthorst adreznec esberglu 5083 PS214:47
esbergluefried: https://bugs.launchpad.net/nova-powervm/+bug/167972014:47
openstackLaunchpad bug 1679720 in nova-powervm "Openstack image upload failure" [Undecided,New]14:47
esbergluI basically just summarized what happened in IRC14:48
thorstefried: loking14:48
efriedthorst How do you want me to deprecate FUNC?  With a comment, or actually check for its usage and issue DeprecationWarning?14:48
esbergluI tried to just summarize the IRC chat for that bug, might be missing some stuff since I don't understand the issue as well as you14:48
esbergluFeel free to update14:48
efriedthorst k, will address in PS3; meanwhile please review PS2 for any other changes needed.14:49
*** tjakobs_ has joined #openstack-powervm14:49
efriedesberglu Regarding the bug - does the upload actually fail with a 500?  Doesn't just hang?  Can you put a stack trace in the bug?14:54
esbergluefried: Yeah it got a 500 error when I was trying to stack yesterday. But I'm dumb and forgot to save off the logs before trying to deploy on newton14:55
efriedThat's... weird.14:55
esbergluSo I don't have a stack trace, and it looks like newton is working14:55
efriedI need to see that 500.  Could be a separate issue.14:55
efriedAssume it was during the primer LU creation14:56
efriedWe might not need to fix ocata.  Which would be kinda nice.14:56
esbergluWell once I confirm newton works, I can go back and try ocata again14:56
esbergluBut here's what happened14:56
esbergluIt downloaded the image from the remote location14:56
esbergluThen tried to run the openstack create image command14:57
esbergluAnd got a 500 error there14:57
efriedopenstack create image doesn't touch our code.  That 500 would have had to be from glance.  Does that sound familiar?14:57
efriedthorst esberglu adreznec Until we nail that down, I'm going to modify this bug to represent the hang we know about.14:59
esbergluOkay. I'm restacking with ocata now15:00
*** jwcroppe has quit IRC15:05
*** jwcroppe has joined #openstack-powervm15:07
*** nbante has quit IRC15:07
efriedthorst Feedback on PS2?15:16
efriedStill working at it?15:16
efriedesberglu We're going to have a tricky-ish time testing this, because the nova-powervm change set is going to have a requirements update to pypowervm 1.1.1, which hasn't been released yet; but we don't want to release that until we've tested it.  Can you finagle a way around that?15:18
esbergluefried: Yeah should be able to just pull in the pypowervm patch and then remove pypowervm from the reqs15:19
thorstI +1'd it15:20
efriedthorst PS3 is up.15:20
efriedAdded Partial-Bug to the commit message too.15:20
*** shyama_ has joined #openstack-powervm15:22
thorstLGTM now15:24
*** shyama has quit IRC15:24
*** shyama_ is now known as shyama15:24
openstackgerritEric Fried proposed openstack/nova-powervm master: create_disk_from_image: IO_STREAM instead of FUNC  https://review.openstack.org/45323315:45
efriedthorst adreznec esberglu ^^15:45
thorstefried: ok.  I'll review that while you review shyama's (if you haven't already)15:45
efriedNote that we're going to have to bump g-r pypowervm also before this merges, or our shiny new bot is going to propose a reversion to 1.1.015:45
efriedthorst I did.  Needs more changes.15:46
thorstshyama: FYI15:46
efriedesberglu See if you can pull ^^ (453233) and pypowervm 5083 into the CI.15:46
esbergluefried: Okay. I will do a manual run on staging. That's really my only option right now15:47
esbergluefried: Manual run is started. Should see results in 30 min or so16:08
esbergluIt also looks like the ocata stack is going to go through for CI undercloud. Not done yet, but past the point it failed previously16:08
esbergluJust a weird coincidence that I hit it during the image upload16:09
*** shyama has quit IRC16:17
efriedesberglu Does that guy use the same SSP as others?16:18
esbergluNo it has its own16:18
efriedOkay, good.  Cause we won't test the code path in question unless we're actually uploading a new image LU.16:20
*** shyama has joined #openstack-powervm16:33
openstackgerritShyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map  https://review.openstack.org/43232216:46
shyamaefried: done with the changes. please check the new change set https://review.openstack.org/#/c/43232216:47
esbergluefried: Passed the run16:56
thorstshyama: I posted a comment in there17:02
thorstsmall one, otherwise looks good to me17:03
efriedthorst https://review.openstack.org/#/c/453233/17:04
efriedshyama I posted new comments as well.17:07
*** nbante has joined #openstack-powervm17:19
shyamaefried: https://review.openstack.org/#/c/432322/7/nova_powervm/virt/powervm/volume/driver.py@120 I will make it " Needs to be implemented within the subclass to support rebuild." does that sound ok?17:20
efriedshyama Ah, yes, that makes more sense.  Thanks.17:20
openstackgerritShyama proposed openstack/nova-powervm master: Update File i/o to support live-migration and use slot map  https://review.openstack.org/43232217:35
efriedthorst esberglu FYI, got a bug in Content-Length header.  Looks like glance's iterable answers to len() with a zero.17:40
thorstefried: still think we can get away without a facade?17:41
efriedYou suck17:41
thorstefried: wasn't being sassy17:41
thorstjust asking  :-)17:41
efriedThing is, even if it returned 0, the requests code should still be doing the right thing.17:42
efriedMust be something else.  Still looking...17:42
*** apearson has quit IRC18:00
*** apearson has joined #openstack-powervm18:03
efriedthorst I am stumped.18:05
efriedWhen I reinstate IterableToFileAdapter and use chunkreader(), it works.18:05
efriedI even tried sending the glance iterable through chunkreader().18:05
efriedWhich should be NO different.18:06
thorstso the IterableToFileAdapter had some weird-ness to it18:08
thorstthat I didnt' see until like 5th pass throgh it18:08
thorstspecifically around that 'remaining_data' bit.18:08
thorstif it passed size in, it would get part of it, but otherwise it got all of it...18:09
efriedWell, we are passing size in.18:09
efriedAnd it's being hidden through chunkreader() anyway.18:10
thorstdo you catch the StopIteration?18:11
efriedThink I've got a version that works now.18:12
thorstnot sure then...18:12
efriedI think it's because the glance thingy isn't really an iterable per se.18:13
efriedI mean, it is and it isn't.18:13
thorstthey're multi meg chunks18:13
efriedor... whatever.18:14
thorsthmmm...I thought they made them bigger18:14
efriedoh, perhaps so, and the IterableToFileAdapter is forcing them to 64K18:15
efriedRather, the chunkreader is forcing them to 64K through the IterableToFileAdapter.18:16
efriedAnyway, I believe I've got a version that works without IterableToFileAdapter.  toxing it now, will post in a bit.18:17
efriedthorst Check out PS418:20
efriedthorst adapter.py best compared against base.18:20
thorstso you're basically saying upload the whole chunk unless they have a read method...18:21
thorstbut then we ignore the chunksize parameter altogether that way...18:22
thorstseems like we'd want to specify that somehow...18:22
efriedTime to change your password.18:22
adreznecNow's my chance!18:23
esbergluTime to change my gmail password18:23
efriedthorst I'm not a fan of the overhead associated with rechunking.  We're getting rid of that by getting rid of IterableToFileAdapter.18:23
* adreznec furiously logs in as esberglu everywhere18:23
*** apearson has quit IRC18:23
efriedthorst And chunksize isn't overridden anywhere.  That 64K is pretty arbitrary, almost certainly an artifact left over from the k2operator days.18:24
*** nbante has quit IRC18:25
thorstefried: totes18:26
thorstJust thinking its another thing to note in the method signature18:26
thorst(which is non-existent at present)18:26
efriedthorst Tested with SSP, works.  Gonna test with localdisk now.18:26
efriedthorst Yeah, not a terrible idea to add a docstring there.18:27
efriedthorst But it's gonna be a big 'un.18:28
thorstefried: just capture the delta.18:30
thorstdon't boil the adapter.py docstring ocean18:30
efriedthorst But, but, but... my OCD!!!18:30
*** mdrabe has quit IRC18:30
thorsthandle your OCD in a separate change set.18:31
*** shyama has quit IRC18:33
efriedthorst PS518:45
efriedMan, we don't have a lot of logging on the localdisk side.18:45
thorstwe probably never cleaned it up like we did ssp18:46
thorstit's just such a lower used path18:46
efriedthorst Oh, crap, sonar.18:46
*** jwcroppe has quit IRC18:47
*** jwcroppe has joined #openstack-powervm18:48
*** apearson has joined #openstack-powervm18:49
*** mdrabe has joined #openstack-powervm18:50
*** jwcroppe has quit IRC18:52
efriedthorst Interesting stat: upload to localdisk too 434s.  Versus about 100s less to SSP.18:52
thorstis localdisk truly using a localdisk?18:53
thorstor a hdisk backed by FC?18:53
thorstthe SAS drives aren't as fast as your FC.18:53
efriedthorst Dunno.18:54
efriedBut I would have expected the "network" to be the bottleneck anyway.18:54
efriedThe RMC limiter18:54
thorstheh, well maybe it isn't18:55
efriedthorst Sonar-fixin PS6 posted.18:57
efriedRetesting for sanity with localdisk.18:57
efriedhold on.18:58
thorstefried: I've got a meeting for 60 min18:58
thorstso I'll recheck then  :-)18:59
efriedthorst Okay, PS7 is read now.18:59
efriedswhat I get for developing from two different machines.18:59
efriedadreznec We're going to need to release pypowervm 1.1.1 now.  You got a handle on that?19:00
adreznecefried: I'd need to talk with Julio on the specifics required now with the new flows19:01
adreznecHe handled everything the last time around19:01
efriedthorst I'd like to get 5037 in there as well.  Please add that to your review queue19:01
adreznecNot sure how he handled tagging/version bumping19:01
*** mdrabe has quit IRC19:02
*** mdrabe has joined #openstack-powervm19:04
efriedesberglu So ocata is green?19:10
esbergluefried: Finishing up the stacks on the compute nodes, hit some package versioning issues there19:11
esbergluBut yeah it appears to be19:11
efriedesberglu Has it made the primer LU yet?19:11
esbergluefried: No not yet19:22
efriedesberglu Okay, so we won't know if we're busted there until that happens.19:22
openstackgerritEric Fried proposed openstack/nova-powervm master: create_disk_from_image: IO_STREAM instead of FUNC  https://review.openstack.org/45323319:50
thorstefried: hit up the reviews19:54
thorstI think we need to push a new pypowervm before we push https://review.openstack.org/#/c/453233 through19:55
efriedthorst Yes, before that will verify.  I wouldn't mind a look-see now, though.19:55
efriedthorst mdrabe Do we want to get 4975 in?19:57
efriedThat guy has (or will have) some fixage to the retry logic.19:59
efriedRight now, if a streamed upload fails partway, we'll retry it.19:59
efried...with a stream that's partially consumed.19:59
efriedWhich is bogus.19:59
efriedthorst And what about 5052, which appears to be ready?20:00
thorstefried: 5052 - yes20:01
thorstI can submit that now20:01
efriedmdrabe How close are we on 4975?20:08
*** thorst has quit IRC20:35
mdrabeefried: Sorry been in a meeting, you're dependant on 4975 ya?20:36
*** smatzek has quit IRC20:37
*** mdrabe has quit IRC20:40
*** mdrabe has joined #openstack-powervm20:48
*** thorst has joined #openstack-powervm21:07
esbergluefried: Hit a few more hiccups getting the compute nodes to stack21:15
esbergluImage template is built now, ready nodes are spawning21:15
*** tjakobs_ has quit IRC21:20
efriedmdrabe Not dependent; but it did uncover a definite bug.21:24
efriedmdrabe It's pretty low-probability, and if the upload breaks partway through, we'd wind up failing anyway.  Your fix would just fail us sooner and probably prettier.21:25
efriedSo it's not urgent or strictly necessary.21:25
mdrabeK should have it up in a bit21:26
mdrabeIf UT is will play21:26
efriedmdrabe See my response to thorst comment.21:29
efriedAnyway, no hurry; I don't think we care too much about getting this into 1.1.121:29
mdrabeefried So you thinkin keep all the retries then?21:30
efriedmdrabe Keep the retry on the func path - unless thorst still disagrees.21:30
efriedWe may want to talk it out first so we don't end up thrashing the change set.21:31
mdrabeIght I'll leave that be for now21:31
*** edmondsw has quit IRC21:31
*** edmondsw has joined #openstack-powervm21:33
*** edmondsw has quit IRC21:38
*** smatzek has joined #openstack-powervm21:58
*** esberglu has quit IRC21:58
*** thorst has quit IRC22:01
*** mdrabe has quit IRC22:03
*** smatzek has quit IRC22:12
*** esberglu has joined #openstack-powervm22:13
*** k0da has quit IRC22:13
*** tjakobs_ has joined #openstack-powervm22:25
*** tjakobs__ has joined #openstack-powervm22:29
*** apearson has quit IRC22:29
*** tjakobs_ has quit IRC22:32
*** apearson has joined #openstack-powervm22:38
*** tjakobs__ has quit IRC22:42
*** esberglu has left #openstack-powervm22:45
*** jpasqualetto has quit IRC22:53
*** apearson has quit IRC22:53
*** thorst has joined #openstack-powervm23:01
*** jpasqualetto has joined #openstack-powervm23:05
*** thorst has quit IRC23:06
*** jpasqualetto has quit IRC23:13
*** jpasqualetto has joined #openstack-powervm23:26
*** jpasqualetto has quit IRC23:28
*** svenkat has joined #openstack-powervm23:45

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!