Wednesday, 2016-07-27

*** wangqwsh has quit IRC00:09
*** wangqwsh has joined #openstack-powervm00:09
*** esberglu has joined #openstack-powervm00:34
*** thorst has joined #openstack-powervm00:36
*** thorst has quit IRC00:42
*** thorst has joined #openstack-powervm00:43
openstackgerritDrew Thorstensen proposed openstack/networking-powervm: Support VIOS CNAs
openstackgerritDrew Thorstensen proposed openstack/networking-powervm: Support VIOS CNAs
wangqwshthorst: ibmveth6 is the new NIC that you help attach to neo4, right?00:55
*** esberglu has quit IRC00:59
thorstwangqwsh: nope00:59
thorstits the en 000:59
thorstwangqwsh: yep.  That sounds right.  You have to use Ashana's controller though01:00
thorstits got the right VLANs to the powervm novalink01:01
*** arnoldje has joined #openstack-powervm01:15
*** svenkat has quit IRC01:28
*** openstackgerrit has quit IRC01:33
*** openstackgerrit has joined #openstack-powervm01:34
*** tsjakobs has joined #openstack-powervm01:42
*** jwcroppe has quit IRC01:42
*** tsjakobs has quit IRC01:47
*** thorst has quit IRC01:48
*** thorst has joined #openstack-powervm01:48
*** thorst has quit IRC01:57
*** thorst has joined #openstack-powervm02:06
*** thorst has quit IRC02:06
*** svenkat has joined #openstack-powervm02:12
*** svenkat has quit IRC02:33
*** svenkat has joined #openstack-powervm02:35
*** svenkat has quit IRC02:49
*** svenkat has joined #openstack-powervm02:59
*** wangqwsh has quit IRC03:01
*** thorst has joined #openstack-powervm03:07
*** svenkat has quit IRC03:15
*** thorst has quit IRC03:16
*** seroyer has joined #openstack-powervm03:37
*** thorst has joined #openstack-powervm04:15
*** thorst has quit IRC04:21
*** seroyer has quit IRC04:24
*** kotra03 has joined #openstack-powervm04:49
*** seroyer has joined #openstack-powervm04:51
*** seroyer has quit IRC04:58
*** arnoldje has quit IRC05:09
*** thorst has joined #openstack-powervm05:20
*** thorst has quit IRC05:26
*** kotra03 has quit IRC05:58
*** thorst has joined #openstack-powervm06:01
*** thorst has quit IRC06:08
*** kotra03 has joined #openstack-powervm06:41
*** thorst has joined #openstack-powervm07:08
*** thorst has quit IRC07:14
*** adreznec has quit IRC07:35
*** adreznec has joined #openstack-powervm07:39
*** thorst has joined #openstack-powervm08:11
*** k0da has joined #openstack-powervm08:12
*** thorst has quit IRC08:18
*** thorst has joined #openstack-powervm09:16
*** thorst has quit IRC09:23
*** k0da has quit IRC09:52
*** k0da has joined #openstack-powervm10:00
*** thorst has joined #openstack-powervm10:23
*** thorst has quit IRC10:29
*** svenkat has joined #openstack-powervm11:05
*** thorst has joined #openstack-powervm11:26
*** thorst has quit IRC11:31
*** thorst has joined #openstack-powervm11:49
*** thorst has quit IRC11:56
*** thorst has joined #openstack-powervm12:02
*** Ashana has joined #openstack-powervm12:15
*** kotra03_ has joined #openstack-powervm12:37
*** kotra03 has quit IRC12:40
*** kotra03_ is now known as kotra0312:40
*** seroyer has joined #openstack-powervm13:00
*** tblakeslee has joined #openstack-powervm13:05
*** mdrabe has joined #openstack-powervm13:11
*** seroyer has quit IRC13:14
*** seroyer has joined #openstack-powervm13:16
*** edmondsw has joined #openstack-powervm13:17
*** seroyer has quit IRC13:23
*** lmtaylor has joined #openstack-powervm13:28
*** apearson has joined #openstack-powervm13:38
*** efried has joined #openstack-powervm13:40
Ashanaadreznec: I just ran your repo-discovery.yml and I got an error saying13:49
AshanaThe conditional check 'ansible_architecture not in \"{{ groups['repo_all'] |  map('extract', hostvars, 'repo_master') | list }}\"' failed.13:49
Ashanaadreznec: for that conditional should there be brackets around ansible architecture13:50
adreznecAshana: Are you just running repo-discovery on it's own? Or as part of the whole repo-build flow?13:51
-openstackstatus- NOTICE: Gerrit is being restarted now to relieve performance degradation13:53
Ashanaon its own that was on the list of Qing Wu playbooks that he wanted me to run13:54
thorstadreznec: Qing Wu ran them all as independent playbooks13:54
adreznecOh, I hadn't run it that way. I bet it's an issue with the fact for repo_master not being set then13:55
adreznecBecause that gets set to false when the repo_server setup happens13:55
adreznecRunning them all independently is a weird use case13:55
adreznecFor now just try running all of setup-infrastructure and see if that fixes it13:56
adreznecIf so I'll try and find a way to make the playbooks independent again13:56
AshanaOk I will try that thank you!13:59
*** lmtaylor has quit IRC14:03
*** jwcroppe has joined #openstack-powervm14:03
*** seroyer has joined #openstack-powervm14:05
*** arnoldje has joined #openstack-powervm14:14
efriedthorst, mdrabe and I were discussing the bug about mappings not being cleaned up on destroy_evacuated.  Wanted to get your input on an approach we considered.14:15
*** dwayne has joined #openstack-powervm14:16
efriedMy take was: we're being real gentle and careful about how we clean up mappings on destroy - generally feeding everything through Disconnect etc.  Is there some reason we shouldn't just feed everything through the LPAR-specific scrub on destroy?14:17
*** tsjakobs has joined #openstack-powervm14:19
*** lmtaylor1 has joined #openstack-powervm14:20
*** mdrabe has quit IRC14:57
thorstefried: I don't really think there would be an issue with that...except isn't the LPAR there?14:57
thorstisn't the LPAR scrub generally used when we know the LPAR, but before mappings are attached to it?14:57
efriedCorrect on both counts.14:58
thorstand doesn't the destroy really try to do what the scrub does?14:58
efriedNot exactly, no.  Disconnect is gentler (and slower).14:58
efriedIt involves discover_hdisk, for one thing, which I don't think we need.14:58
efriedBut I could be wrong - mdrabe will have to investigate to verify.14:59
thorstwell, wouldn't we need that to clear out the hdisk14:59
efriedThe scrubber removes the storage (if unused) as well, in a post-execute.  I think that's better for this situation because, if it fails, the mappings are gone anyway.14:59
*** kotra03 has quit IRC14:59
openstackgerritTaylor Jakobson proposed openstack/nova-powervm: [WIP] Add iscsi support
efriedwhich would resolve this problem.14:59
*** esberglu has joined #openstack-powervm14:59
thorstimplicitly on the covers I don't see anything wrong15:01
thorsthow hard is it to write up and let me stew on in gerrit?15:01
efriedGuess that's a mdrabe question, and he seems to be afk at the moment.15:02
*** mdrabe has joined #openstack-powervm15:05
openstackgerritEric Fried proposed openstack/nova-powervm: Rebase on pypowervm.tasks.hdisk refactor
thorstefried: I'm not digging that the pypowervm change requires us to update nova-powervm15:09
thorstthat smells bad.15:09
efriedthorst, did you look at the change I just posted?15:09
thorstwhoops, just the name15:09
efriedBasically, we were doing something semi-wrong in our UT anyway.15:10
thorstby going to lua_recovery instead of discover_hdisk15:10
efriedYeah.  I think what happened is that we lazily neglected to change those mocks after we wrapped lua_recovery in discover_hdisk (to put in that retry)15:10
openstackgerritEric Fried proposed openstack/nova-powervm: Rebase on pypowervm.tasks.hdisk refactor
*** dwayne has quit IRC15:14
thorstsvenkat: please?15:15
tsjakobsthorst: do you know if there is a good way to get the udid of a device from the device_name in nova-powervm?15:28
tsjakobsthe iscsi discovery only returns the dev_name15:28
efriedtsjakobs, what's the schema type of the device?15:33
efriedDoes it show up as a PhysicalVolume?15:33
tsjakobsya, i think so15:33
thorsttsjakobs: is it not passed in as part of the connection_info?15:33
thorst(since this is a cinder volume)15:33
tsjakobsthe connection_info has host_ip and IQN. Then we do the iscsi discovery/login then a query to get the dev_name15:34
tsjakobsbest i can tell15:34
efriedIt sounds like you will need to GET the VirtualIOServer with the ViosStorage extended attribute group, then loop through the PhysicalVolumes therein looking for the one with a matching dev_name.15:35
efriedWhat do you need to do with the UDID?15:35
tsjakobsI'm using it in the connect/disconnect of iscsi. It looks like I could probably get away with just the device_name, it just seemed like the refactor i did worked better15:37
tsjakobsif the udid was easily available15:38
efriedtsjakobs, I think it would be a new thing if we were mapping a PV directly to a VM.  thorst, keep me honest here.15:49
efrieddamn, that CAN'T be right.  Looking more...15:49
*** dwayne has joined #openstack-powervm15:51
efriedOkay, tsjakobs, for the existing disk type, we're also just using the device name to create the mappings.15:53
efriedSee nova_powervm/virt/powervm/volume/
*** jwcroppe has quit IRC15:55
tsjakobswhat about nova_powervm/virt/powervm/volume/
tsjakobswould i have to make a dev_name_to_scsi_mapping?15:57
tpeoplesfyi, broke migration for the nova-powervm driver15:57
tpeoplesefried: fyi15:57
tsjakobsefried: ^^^15:57
thorsttpeoples: is that just a rename of a method?15:57
thorstnothing more severe?15:57
tpeoplesyeah, so easy fix15:57
thorstthank goodness.15:57
thorstthx dude.15:57
thorstmind making a bug for it?15:57
tpeoplesbut it'll blow up15:57
thorstthx...easy fix.15:58
thorstefried tsjakobs: stuck in meeting...haven't read the previous stuff.15:58
adreznecthorst: esberglu: That reminds me... once we get the CI stablized, we should probably look at getting the multi-node live migration jobs working in the CI15:59
adreznecDoesn't have to be for every run on Nova, but even just a periodic job we set up somewhere or just on nova-powervm runs would give us more heads up in this area16:00
thorstadreznec: yeppers16:00
adreznecSomething to keep on the back burner for after our CI is stable16:01
adreznecSo next week, right esberglu?16:01
thorstyour optimism knows no bounds16:01
*** thorst is now known as thorst_afk16:01
esbergluadreznec: The only thing that is really still holding us back is the slow instance build times16:02
efriedtpeoples, fix is ready.  Got that bug report yet?16:07
openstackLaunchpad bug 1606981 in nova-powervm "Driver does not implement cleanup_live_migration_destination_check method" [Undecided,New]16:07
openstackgerritEric Fried proposed openstack/nova-powervm: Refactor: cleanup_live_migration_destination_check
efriedtpeoples thorst ^^16:09
*** k0da has quit IRC16:10
*** jwcroppe has joined #openstack-powervm16:13
adreznecesberglu: How slow is slow?16:13
esbergluI’ve seen as bad as 500 seconds16:14
thorst_afkesberglu: with the new code that tells us each step...where is it spending the majority of its time now?16:25
*** thorst_afk is now known as thorst16:27
esbergluthorst_afk: Task plug_vifs completed in 155 seconds16:31
thorstugh...that's awful16:36
thorstOK - time to fix that.16:36
adreznecesberglu: What was the rest of the breakdown?16:38
adreznecJust trying to wrap my head around a 500 second spawn...16:39
*** Ashana has quit IRC16:39
*** Ashana has joined #openstack-powervm16:40
*** Ashana has quit IRC16:44
*** Ashana has joined #openstack-powervm16:47
esbergluadreznec: This one was only 384 seconds! crt_disk_from_img took 43, cfg_drive took 23. Those are the big ones16:51
*** Ashana has quit IRC16:51
*** Ashana has joined #openstack-powervm16:53
*** Ashana has quit IRC16:58
thorstesberglu: link?17:09
esbergluFilter by that if you want the build I was looking at17:11
thorstI was just looking at all the instances17:12
thorstso in general, we have two time sinks17:12
thorstplugging vifs for some reason17:12
thorstI'll sign up to take that17:12
thorstthen crt_disk_from_img and connect_disk are wild17:13
thorstefried: Maybe that's one you take and work with clbush on?17:13
thorstesberglu: somethings wrong with the networking-powervm agent...I'm not getting any debug  :-(17:15
thorstI thought we had enabled that?17:16
esbergluthorst: Crap. I just had it on a local branch and forgot to add it back in last time I redeployed.17:17
thorstcan we get that added and then I can actually debug that?17:17
*** Ashana has joined #openstack-powervm17:21
esbergluthorst: Is that something we want to permanently enable?17:22
*** Ashana_ has joined #openstack-powervm17:23
*** Ashana has quit IRC17:26
*** Ashana_ has quit IRC17:28
*** Ashana has joined #openstack-powervm17:29
*** Ashana has quit IRC17:34
*** Ashana has joined #openstack-powervm17:34
Ashanaadreznec: I ran the setup-infrastructure and there was an error with the  repo_package_sync.yml I got this error failed: [infra01_repo_container-e0986c2a] (item=/var/www/repo/pools) => {"changed": true, "cmd": ["rsync", "--delay-updates", "-F", "--compress", "--archive", "--rsh", "ssh -l nginx -o stricthostkeychecking=no", "--out-format=<<CHANGED>>%i %n%L", "/var/www/repo/pools", "nginx@compute01_repo_container-1b7b4616:/17:42
thorstesberglu: Yeah, perm17:52
thorstAshana: does the compute node have an ssh key pair to the source?18:01
thorstand adreznec does it?18:01
adreznecAshana: Ok.... what was the actual error?18:02
adreznecThat just says it failed18:02
adreznecYou'll probably have to run again with -vvv18:02
adreznecand yes thorst, the repo containers all have SSH keys shared between them18:02
thorstadreznec: do the playbooks set that up18:03
thorstor ahead of time is that done?18:03
adreznecDone in the playbooks18:03
adreznecI'm just using the existing ones put in place during repo server setup18:03
openstackgerritMerged openstack/networking-powervm: Support VIOS CNAs
efriedthorst, are you saying crt_disk_from_img and connect_disk are performing poorly?19:36
*** Ashana has quit IRC19:41
openstackgerritDrew Thorstensen proposed openstack/networking-powervm: Simplify host_uuid and gets
thorstefried: yeppers19:45
thorstsorry for delay19:45
efriedthorst, What's the context here?  CI run?19:47
thorstso we're seeing two major slow downs.19:47
thorst1) Networking -> I'm on it.  esberglu is increasing log levels and then I get to figure it out19:47
thorst2) crt_disk_from_image and connect_disk.  Hoping you, esberglu and clbush can dive a little bit into?  We need to figure out if only solution is to decrease SSP size...19:48
*** Ashana has joined #openstack-powervm19:48
efriedthorst, we're seeing a slowdown from previous runs, or we're just now finding a bottleneck that has always existed?19:49
thorstefried: so we opened it up to all of nova19:49
thorstso much more throughput19:49
thorstso its a bottleneck that has always existed that we just never percieved19:50
thorstwe were able to open up to all of nova after we solved other bottlenecks19:50
efriedSo the bottleneck must be at the SSP, because that's the only shared component?19:50
thorstlikely....I think we just need to get clbush and changh on it...and get them whatever data they need.19:51
openstackgerritEric Fried proposed openstack/nova-powervm: WIP/Proposal: Simplify storage deletion on destroy
*** Ashana has quit IRC19:53
*** Ashana has joined #openstack-powervm19:54
*** Ashana has quit IRC19:55
*** Ashana has joined #openstack-powervm19:56
efriedthorst, mdrabe - take a look at the approach suggested by ^^19:56
efriedAnd try to break it.19:56
Ashana@thorst just got back and I ran it with -vvv but it still19:56
Ashanaadreznec: it says  [WARNING]: Consider using synchronize module rather than running rsync  then it was on the infra01_repo_container-e0986c2a19:57
adreznecThat's not the error though, that's just a warning19:58
adreznecDo you want to shoot me the full log in a pastebin19:58
AshanaYes I will20:02
openstackgerritMerged openstack/nova-powervm: Refactor: cleanup_live_migration_destination_check
*** miltonm has quit IRC20:31
*** thorst has quit IRC20:55
*** svenkat has quit IRC20:58
*** k0da has joined #openstack-powervm20:58
*** Ashana has quit IRC21:05
*** seroyer has quit IRC21:07
*** lmtaylor1 has left #openstack-powervm21:27
*** apearson has quit IRC21:46
*** esberglu has quit IRC21:55
*** tblakeslee has quit IRC21:59
*** mdrabe has quit IRC22:03
*** efried has quit IRC22:04
*** svenkat has joined #openstack-powervm22:10
*** tsjakobs has quit IRC22:13
*** edmondsw has quit IRC22:25
*** arnoldje has quit IRC22:25
*** svenkat has quit IRC22:27
*** k0da has quit IRC22:28
*** svenkat has joined #openstack-powervm22:32
*** jwcroppe has quit IRC22:40
*** jwcroppe has joined #openstack-powervm22:41
*** jwcroppe has quit IRC22:45
*** thorst_ has joined #openstack-powervm22:47
*** seroyer has joined #openstack-powervm22:54
*** thorst_ has quit IRC22:55
*** thorst_ has joined #openstack-powervm22:56
*** svenkat has quit IRC23:03
*** thorst_ has quit IRC23:04
*** seroyer has quit IRC23:18

Generated by 2.14.0 by Marius Gedminas - find it at!