Friday, 2017-02-17

*** kriskend has quit IRC00:03
*** svenkat has quit IRC00:16
*** svenkat has joined #openstack-powervm00:44
*** efried has quit IRC00:48
*** jwcroppe has quit IRC00:57
*** efried has joined #openstack-powervm01:02
*** thorst_afk has joined #openstack-powervm01:17
*** svenkat has quit IRC01:21
*** thorst_afk has quit IRC01:41
*** thorst_afk has joined #openstack-powervm01:56
*** thorst_afk has quit IRC02:50
*** thorst_afk has joined #openstack-powervm03:03
*** thorst_afk has quit IRC03:04
*** thorst_afk has joined #openstack-powervm03:35
*** thorst_afk has quit IRC03:35
*** mdrabe has joined #openstack-powervm03:57
*** seroyer has joined #openstack-powervm04:34
*** seroyer has quit IRC04:49
*** thorst_afk has joined #openstack-powervm04:57
*** thorst_afk has quit IRC05:02
*** mdrabe has quit IRC05:05
*** tjakobs has joined #openstack-powervm05:08
*** tlian has quit IRC05:32
*** thorst_afk has joined #openstack-powervm06:59
*** thorst_afk has quit IRC07:03
*** thorst_afk has joined #openstack-powervm08:59
*** k0da has joined #openstack-powervm09:03
*** thorst_afk has quit IRC09:04
*** tjakobs has quit IRC09:07
*** edmondsw has joined #openstack-powervm09:54
*** k0da has quit IRC09:56
*** edmondsw has quit IRC09:58
*** thorst_afk has joined #openstack-powervm11:01
*** thorst_afk has quit IRC11:05
*** k0da has joined #openstack-powervm11:14
*** smatzek_ has joined #openstack-powervm11:39
*** smatzek_ has quit IRC11:40
*** smatzek has joined #openstack-powervm11:41
*** viclarson has joined #openstack-powervm12:01
*** seroyer has joined #openstack-powervm12:38
*** thorst_afk has joined #openstack-powervm12:43
*** Esha has joined #openstack-powervm12:53
EshaThe stable/ocata PowerVM CI isue was fixed, Thanks Eric ! Now seeing an issue with stable/newton PowerVM CI in
*** tblakes has joined #openstack-powervm12:58
*** svenkat has joined #openstack-powervm13:03
*** edmondsw has joined #openstack-powervm13:07
*** jpasqualetto has joined #openstack-powervm13:16
*** jpasqualetto has quit IRC13:48
*** dwayne has joined #openstack-powervm13:48
*** Esha has quit IRC13:53
*** mdrabe has joined #openstack-powervm13:54
*** viclarson has quit IRC13:55
*** viclarson has joined #openstack-powervm13:55
thorst_afkmdrabe: did you have a chance to look at that slot error from the other day?13:56
*** thorst_afk is now known as thorst_13:56
mdrabeYea it didn't look like a slot error13:56
mdrabethorst_ It was that the disk couldn't be found13:56
thorst_which to me indicates that the rebuild map was called before the hdisk was discovered.13:57
thorst_the hdisk was probably there (IMO), it probably just didn't run the VIOS command yet to discover it13:57
thorst_does that make sense?13:58
thorst_so my question is this working in other envs13:58
mdrabeBut there were two errors printed as debug statements13:58
mdrabeone for each VIOS, that volume X couldn't be found on this VIOS13:58
thorst_right...because the slot map checks each vios13:59
mdrabeBut the error isn't coming from the slot map13:59
mdrabeNo ITLs for VIOS vio1-38w for volume 35d30504-6a77-4fb2-acab-286a9cd067ad. _discover_volume_on_vios /usr/lib/python2.7/dist-packages/nova_powervm/virt/powervm/volume/
*** jpasqualetto has joined #openstack-powervm14:00
mdrabeI don't think that should be happening at all, regardless of the slot map14:01
*** apearson has joined #openstack-powervm14:01
thorst_that's what you're working off of?14:01
mdrabeNo from the LP bug14:01
openstackLaunchpad bug 1665407 in nova-powervm "unable to migrate vm with attached volumes" [Undecided,New]14:02
*** svenkat has quit IRC14:04
thorst_yeah, see comment 2:  vif.plug() -> slot_mgr.build_map() -> slot_mgt.init_recreate_map() -> slot_mgr._pv_vscsi_vol_to_vio() -> vol_drv.is_volume_on_vios()14:04
thorst_o damn...that does the discover.14:05
mdrabeOh boy the slot map is doing discover...14:05
thorst_yeah, I knew we had to be doing that.14:05
thorst_otherwise this would be borked to hell elsewhere...14:05
thorst_now viclarson is running stable/mitaka.  I wonder if there was a change missing or something back there.14:07
openstackgerritMerged openstack/nova-powervm master: Add imagecache information to devref
mdrabeIt is possible, a lot of that stuff went in right at the end of mitaka14:08
thorst_yeah, but I thought we did get it all back ported.14:09 my current theory is that maybe cinder is not attaching it to the new host in time.14:09
mdrabethis is also resize to a different host, which wasn't looked at as much I don't think14:10
thorst_our testing was primarily resize on same host or remote rebuild14:13
thorst_viclarson: you around?14:13
thorst_so current theory on the bug you opened.14:14
thorst_at least my current theory...14:14
thorst_This is a resize to a different host.  Which means that Cinder needs to attach the volume to the other host before that 'finish_resize' method is called.14:15
thorst_can you do the test again, and make sure that during the resize op, the new host has the volume attached to it on the backing v7000 (is that right?) when it calls that finish_resize14:16
viclarsoni changed slot_mgt to14:16
viclarsonother_slot_mgr(vol_drv_iter=None) for PlugVif and PlugManagmentVif14:17
viclarsonnow it hungs on connections14:17
viclarsoni think it because vm stays in active state14:18
thorst_well you won't get your cinder volumes really there...14:18
viclarsoncinder mapped volumes to hosts in time14:18
viclarsononce again attach, detach, resize with attached volumes works fine (when nvram_store is none)14:19
viclarsonmb i'm wrong14:20
viclarsonand it's just unstable behaviour14:20
thorst_so what the error is telling us is that the other host doesn't have a path to the volumes14:20
thorst_the VIOS can't find the volumes...14:20
thorst_this generally indicates that something is up between the target host and the storage subsystem14:20
thorst_maybe a FC zone is off (probably not the case?) or the cinder driver isn't mapping to it in time14:21
thorst_we're trying to definitively rule that out...14:21
*** burgerk has joined #openstack-powervm14:28
viclarson16:31:14 mkvdiskhostmap target host14:30
viclarson16:31:15 Operation finish migration14:30
*** dwayne has quit IRC14:35
viclarson<thorst_> maybe a FC zone is off (probably not the case?)  **** or the cinder driver isn't mapping to it in time ****14:37
viclarsonwhat to check?14:37
*** jwcroppe has joined #openstack-powervm14:43
*** esberglu has joined #openstack-powervm14:44
thorst_viclarson: so that can be an easy check...can you deploy new VMs on each host and attach new volumes?  If so, then its not a zoning issue14:44
*** tjakobs has joined #openstack-powervm14:45
viclarsonyes i can14:48
thorst_how many hosts do you have?14:48
thorst_can you put one into maintenance...then resize the VM14:49
thorst_it should do a local resize14:49
thorst_want to see if that flow works.14:49
viclarsoni switched out local resize14:49
viclarsondont work14:49
thorst_same error eh?14:49
viclarsonthe same behaviour as i have now for remote resize14:50
viclarsonvm hangs in state finishing resize14:50
viclarsonthe wrong word ... boring...14:53
*** kriskend has joined #openstack-powervm14:53
*** tlian has joined #openstack-powervm14:53
thorst_heh, wish I could get in the env.  :-)14:56
thorst_viclarson: you going to be at the PTG next week?  If so, I could dig in on it with you14:56
*** seroyer has quit IRC14:58
viclarsonno im not going and unfortunately its impossible. isolated environment. the bank14:58
*** k0da has quit IRC15:03
thorst_let me see if I can work through this a bit more...I'll work off code inspection15:04
thorst_what we're thinking is somehow the volume id is changing15:04
thorst_which is super weird.15:04
*** tjakobs has quit IRC15:11
*** dwayne has joined #openstack-powervm15:15
*** tblakes has quit IRC15:17
*** seroyer has joined #openstack-powervm15:19
*** tblakes has joined #openstack-powervm15:20
*** tjakobs has joined #openstack-powervm15:41
*** viclarson_ has joined #openstack-powervm15:52
*** viclarson has quit IRC15:57
*** kylek3h has quit IRC15:58
*** kylek3h has joined #openstack-powervm15:59
*** kylek3h has quit IRC16:01
*** kylek3h has joined #openstack-powervm16:01
*** tlian2 has joined #openstack-powervm16:10
*** jpasqualetto has quit IRC16:11
*** tlian has quit IRC16:12
*** tlian2 has quit IRC16:15
*** jpasqualetto has joined #openstack-powervm16:27
*** svenkat has joined #openstack-powervm16:30
efriedthorst_ adreznec esberglu First five in-tree change sets are ready for your reviews.16:47
*** jpasqualetto has quit IRC16:54
*** kjw3 has joined #openstack-powervm16:58
*** tblakes has quit IRC17:03
*** tblakes has joined #openstack-powervm17:03
*** jpasqualetto has joined #openstack-powervm17:06
*** jpasqualetto has quit IRC17:23
*** mdrabe has quit IRC17:24
*** mdrabe has joined #openstack-powervm17:32
*** jpasqualetto has joined #openstack-powervm17:57
viclarson_finally i changed slot_mgr = slot.build_slot_mgr(18:02
viclarson_instance, self.store_api, adapter=self.adapter) #without vol_dev_it18:02
viclarson_vm appears with attached volumes ip and hostame18:03
viclarson_what will broke?18:04
*** tlian has joined #openstack-powervm18:18
*** viclarson_ has quit IRC18:25
thorst_o man...just missed viclarson18:51
thorst_I'll have to catch him Monday18:51
thorst_efried: if you see him in IRC while I'm at PTG - basically the volumes won't be rebuilt at the same LUA index.  This may confuse some OSes.  It is in line with other platforms, but the purpose of the volume iterator is so that we can put the volumes back in the identical location.18:52
thorst_efried: working on your reviews now18:52
efried"confuse some OSes" by making the disk names come up different, right?18:53
*** kylek3h is now known as kylek3h_away19:02
efriedthorst_ 4883 when you get a chance too.  Little 'un.19:02
thorst_review.o.o is a pain today19:03
thorst_;ppls ;ole tjeu19:03
thorst_looks like they're restarting it...19:03
thorst_oo, I've been waiting for 4883...19:03
-openstackstatus- NOTICE: Restarting gerrit due to performance problems19:04
efriedYeah, r.o.o has been choppy last few minutes anyway.19:04
thorst_had one comment on 488319:05
esbergluthorst_: efried: adreznec: Are there any *-powervm changes we are trying to get in before I cut the RC at the end of the day?19:11
adreznecNothing from me19:14
thorst_Not on my side...I'm still working on master19:19
*** k0da has joined #openstack-powervm19:23
*** k0da has quit IRC19:37
efriedesberglu Sorry, missed the above.  I wouldn't mind but certainly not urgent/critical.19:43
esbergluOkay. If you want it in you have to go through that cherry-pick to ocata business19:45
thorst_efried: I think I'm done with reviews20:12
*** smatzek has quit IRC20:13
*** jpasqualetto has quit IRC20:18
*** jpasqualetto has joined #openstack-powervm20:35
efriedthorst_ Take another look at please?20:43
openstackgerritEric Fried proposed openstack/nova-powervm master: In-tree backports: Task UT, Delete args
*** jpasqualetto has quit IRC21:17
*** kriskend_ has joined #openstack-powervm21:25
*** kriskend has quit IRC21:28
*** burgerk has quit IRC21:34
*** kjw3 has quit IRC21:44
*** thorst_ has quit IRC21:53
*** svenkat has quit IRC22:01
*** tjakobs has quit IRC22:12
mdrabeefried: Is the OpenStack instance UUID _always_ supposed to match the LPAR's UUID?22:15
mdrabeignoring the greater bit, the rest of the UUID should be identical I mean22:16
*** edmondsw has quit IRC22:23
*** kriskend_ has quit IRC22:26
*** esberglu has quit IRC22:29
*** tblakes has quit IRC22:32
*** seroyer has quit IRC22:34
*** jwcroppe has quit IRC22:40
*** esberglu has joined #openstack-powervm22:47
*** esberglu has quit IRC22:52
*** edmondsw has joined #openstack-powervm22:55
*** edmondsw has quit IRC22:59
*** dwayne has quit IRC23:05
*** jwcroppe has joined #openstack-powervm23:05
*** jwcroppe has quit IRC23:09
*** mdrabe has quit IRC23:10
*** apearson has quit IRC23:24
*** thorst has joined #openstack-powervm23:27
*** thorst has quit IRC23:32
*** k0da has joined #openstack-powervm23:47
*** smatzek has joined #openstack-powervm23:58

Generated by 2.14.0 by Marius Gedminas - find it at!