Tuesday, 2018-04-24

*** tonyb has quit IRC00:33
*** tonyb has joined #openstack-powervm00:35
*** esberglu has joined #openstack-powervm01:53
*** esberglu has quit IRC03:01
*** edmondsw has joined #openstack-powervm04:43
*** edmondsw_ has joined #openstack-powervm04:44
*** edmondsw has quit IRC04:48
*** edmondsw_ has quit IRC04:59
*** edmondsw has joined #openstack-powervm05:41
*** edmondsw has quit IRC05:45
*** edmondsw has joined #openstack-powervm06:02
*** edmondsw has quit IRC06:06
*** edmondsw has joined #openstack-powervm07:24
*** edmondsw has quit IRC07:28
*** edmondsw has joined #openstack-powervm07:45
*** edmondsw has quit IRC07:49
*** AlexeyAbashkin has joined #openstack-powervm08:05
*** edmondsw has joined #openstack-powervm08:05
*** edmondsw has quit IRC08:09
*** edmondsw has joined #openstack-powervm08:26
*** edmondsw has quit IRC08:30
*** adi_____ has quit IRC12:15
*** adi_____ has joined #openstack-powervm12:15
*** efried has joined #openstack-powervm12:16
*** edmondsw has joined #openstack-powervm12:28
*** edmondsw_ has joined #openstack-powervm12:29
*** edmondsw has quit IRC12:33
*** edmondsw_ has quit IRC12:44
*** edmondsw has joined #openstack-powervm12:48
*** edmondsw has quit IRC12:52
*** edmondsw has joined #openstack-powervm12:59
*** edmondsw has quit IRC13:01
*** edmondsw has joined #openstack-powervm13:27
*** edmondsw has quit IRC13:31
*** esberglu has joined #openstack-powervm13:40
*** edmondsw has joined #openstack-powervm13:59
esbergluefried: You around?14:00
efriedesberglu: yup.14:00
esbergluefried: I was wrong yesterday. It isn't just an issue with our mocks bmd.is_volume doesn't work live either14:01
esbergluPops an attribute error "AttributeError: 'dict' object has no attribute 'is_volume'"14:02
efriedesberglu: Yeah, it looked to me like DriverVolumeBlockDevice takes a BlockDeviceMapping and strips away all the special-ness.14:03
efriedesberglu: And I can't figure out why we would do that.14:03
esbergluefried: But the actual bdm being passed into that method doesn't have that special-ness either14:04
*** edmondsw has quit IRC14:04
esbergluFrom what I can tell14:04
efriedright, swhat I'm saying.  So either we need to re-bdm-ify the thing so we can ask is_volume, or we need to duplicate the logic, which is by asking if that particular field is set to "volume".14:04
*** edmondsw has joined #openstack-powervm14:04
efriedYou should bring this question to mriedem.14:05
*** edmondsw has quit IRC14:09
*** edmondsw has joined #openstack-powervm14:09
*** tjakobs__ has joined #openstack-powervm14:13
*** edmondsw has quit IRC14:14
*** AlexeyAbashkin has quit IRC14:25
*** edmondsw has joined #openstack-powervm14:35
*** edmondsw has quit IRC14:40
*** edmondsw has joined #openstack-powervm14:42
*** edmondsw has quit IRC14:46
*** esberglu has quit IRC15:04
*** esberglu has joined #openstack-powervm15:05
*** AlexeyAbashkin has joined #openstack-powervm15:08
esbergluefried: git restack question15:19
esbergluI checked out the localdisk IT patch and ran git restack15:19
esbergluAnd have done all of the edits on the vSCSI change15:19
esbergluAnd git added all of them15:20
esbergluThen what do I do?15:20
esberglugit rebase --continue says that I have uncommitted changes in my working tree15:21
esberglugit commit --amend takes me into the localdisk commit message15:21
efriedIt sounds like you weren't actually on the VSCSI change when you did your edits.15:21
efriedgit restack brings you into an editor.  You change `pick` to `edit` for the commit you want to fix up.  Did you do that for the VSCSI commit?15:21
esbergluefried: Yes15:22
efriedThen git commit --amend should have put you in the VSCSI commit message.15:22
efriedSomething got messed up in there.15:22
efriedHow hard to redo the vscsi changes?15:22
esbergluefried: I'll just copy all the files I thouched15:23
efriedYou can also save your branch (rename it to something to move it out of the way) and then re-download and re-restack.15:23
efriedThen you've got the branch with your changes and you can diff/patch, which may be easier.15:24
esberglugit stash to the rescue15:29
efrieddude, that's like the fourth time this morning I've put my 'u' in the wrong place.15:32
efriedI wonder if I'm having a stroke.15:32
*** edmondsw has joined #openstack-powervm15:33
esbergluefried: You remember why we couldn't test vSCSI in CI?15:37
esberglu^ or edmondsw if you're there15:37
esbergluSomething to do with remote?15:39
efriedI don't think I ever understood that, no, sorry.15:39
esbergluefried: New vSCSI commit is up15:42
efriedesberglu: snapshot was a manual rebase?15:43
edmondswesberglu efried I'm sitting next to thorst, I'll get him to sign in and explain15:43
*** thorst has joined #openstack-powervm15:43
esbergluefried: 1 line was conflicting...15:44
thorstwe don't have a SSP cinder driver.  We have a storwize one, but we don't have a good cleanup mechanism for failed CI jobs and our v7k would eventually die (and also they can't handle 100 servers SSH'ing to it at the same time)15:44
thorstI think that's the answer to the question I can't see  :-)15:44
edmondswthorst question was "why we can't test vSCSI in CI"15:45
thorstyeah, it's basically we need a cinder driver issue15:45
esbergluthorst: Yep thank you15:45
thorstI think that there are some opportunities on the roadmap, but may be a few months out15:45
edmondswthorst how does SSP relate to this?15:49
edmondswI thought it was more to do with how our CI is structured15:51
thorstvscsi can connect to either SSP or a FC backend (and more - like GPFS and what not).  So you just need a cinder backend that is reasonable for the CI to test vSCSI16:00
*** esberglu has quit IRC16:05
*** esberglu has joined #openstack-powervm16:05
*** esberglu has quit IRC16:10
*** thorst has quit IRC16:30
*** AlexeyAbashkin has quit IRC17:01
*** edmondsw has quit IRC17:04
*** edmondsw has joined #openstack-powervm17:56
*** edmondsw_ has joined #openstack-powervm18:03
*** edmonds__ has joined #openstack-powervm18:05
*** edmonds__ is now known as edmondsw__18:06
*** edmondsw__ is now known as edmondsw___18:06
*** edmondsw has quit IRC18:06
*** edmondsw___ is now known as edmondsw18:06
*** edmondsw_ has quit IRC18:09
*** esberglu has joined #openstack-powervm18:17
*** esberglu has quit IRC18:17
*** esberglu has joined #openstack-powervm18:18
efriedesberglu: Have we reproduced in tree yet?18:45
efriedesberglu: Are you running in tree with the pypowervm patch?18:45
esbergluefried: Yeah right now both IT and OOT CI are running with patched pypowervm18:46
esbergluefried: Not reproduced IT yet18:46
esbergluWait do you mean reproduced as in have we hit the issue IT? If so then yes18:47
esbergluWe haven't pushed a nova patch yet18:47
efriedright, that's what I was asking.18:47
efriedOkay, so definitely affects nova too.18:47
esbergluefried: edmondsw: I'll be taking down production CI later this week to upgrade the undercloud to queens19:02
esbergluProbably friday morning19:02
esbergluShouldn't be any longer than a normal redeploy assuming no roadblocks19:03
edmondswesberglu ack19:07
*** esberglu has quit IRC19:11
*** esberglu has joined #openstack-powervm19:14
efriededmondsw: I meant to talk to you about something else here.19:17
efriededmondsw: We're no longer constrained by the requirements project/bot.  Does that apply to queens as well?19:17
efriededmondsw: If so, we don't need that monkey-patch that Gautam put up this morning.19:17
edmondswefried what monkey patch?19:18
edmondswI thought the requirements changes were only for rocky and forward19:18
edmondswso probably moot19:19
efriededmondsw, esberglu: 6505 is back up for review.19:34
*** edmondsw_ has joined #openstack-powervm21:11
*** edmondsw has quit IRC21:13
esbergluefried: Just got around to verifying the is_volume change21:32
efriedwell shit21:33
efriedesberglu: I think I know why.21:33
efriedesberglu: I think you need to skip https://review.openstack.org/#/c/526094/46/nova/virt/powervm/driver.py@213 and pass block_device_info into _vol_drv_iter.21:34
efriedesberglu: Hum, maybe not.21:37
esbergluefried: Why? I'm not following. block_device_info contains multiple block_device_mappings, and is_volume could be different between them21:37
efriedesberglu: Yeah, we need to figure out wtf block_device_info is, once and for all.21:39
efriedesberglu: Got a way to pdb into this sucker?21:39
esbergluCan you pdb into a running openstack?21:39
efriedesberglu: Yes, as long as you're running the relevant process in the foreground.21:40
efriedesberglu: Or you could use remotepdb21:40
efriedwhich would probably be better considering n-cpu is threaded.21:40
efriedimport remotepdb; remotepdb.set_trace()21:41
efriedWatch the logs for a message that tells you it hit.  The message will give you a port number.  Telnet to that port number on localhost and you'll be attached to the pdb session at that point in that thread.21:41
efriedYou'll probably have to install remotepdb21:41
efriedpip install remote-pdb21:42
edmondsw_oh wow, there is an equivalent to rpdb that handles connections properly? awesome21:47
edmondsw_looks like it's remote_pdb though (missing underscore)21:50
efriedoh, my bad21:53
*** edmondsw_ has quit IRC21:55
esbergluefried: Looks like block_device_info is just a standard dict21:56
efriedesberglu: Well wtf.21:57
efriedesberglu: so you're sitting there right before you would encounter the error, yes?21:58
esbergluefried: Yeah21:58
efriedesberglu: Okay, stand by, let me noodle this.21:58
esbergluefried: Ok. I'm relocating, should be back online in 15 or so22:00
*** esberglu has quit IRC22:04
*** edmondsw has joined #openstack-powervm22:05
*** edmondsw has quit IRC22:05
*** tjakobs__ has quit IRC22:09
*** esberglu has joined #openstack-powervm22:17
efriedfrom nova.objects import block_device as bdmobj22:17
efriedstep into _vol_drv_iter, into the loop over bdms;22:17
efriedbdm = bdmobj.BlockDeviceMapping(**bdm)22:17
efried...and then see if the is_volume line passes.22:17
efriedThis would be something of a hack that shouldn't be necessary, but if it works...22:17
esbergluefried: on it22:18
esbergluefried: It works, but the connection_info is a dict, so it has to be converted to a string first (required by BlockDeviceMappin)22:29
efriedesberglu: This is bizarre.  The block_device_info is coming straight from the spawn call.  In the ironic code path they're *also* not touching it before doing that _bdm_obj.is_volume business on it.  So why are we getting a dict where they're getting a DriverVolumeBlockDevice?22:29
esbergluWait no it still failed sorry22:29
esbergluefried: I have no idea. I'll post another note on the patch and we can see what mriedem has to say22:30
efriedesberglu: I left a note on the review.  It's bizarre to me that block_device_info is a plain dict.  I think we wait for mriedem to look again.22:30
efriedyeah, that.22:30
efriedI'm out.22:31
*** efried has quit IRC22:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!