Tuesday, 2016-08-16

*** svenkat has joined #openstack-powervm00:11
*** thorst_ has joined #openstack-powervm00:28
*** smatzek has joined #openstack-powervm00:38
*** smatzek has quit IRC00:49
*** esberglu has joined #openstack-powervm01:06
*** esberglu has quit IRC01:17
*** thorst_ has quit IRC01:38
*** thorst_ has joined #openstack-powervm01:39
*** thorst_ has quit IRC01:48
*** thorst_ has joined #openstack-powervm02:02
*** thorst_ has quit IRC02:03
*** mdrabe has joined #openstack-powervm02:39
*** dwayne has quit IRC02:53
*** thorst_ has joined #openstack-powervm03:03
*** dwayne has joined #openstack-powervm03:06
*** thorst_ has quit IRC03:11
*** svenkat has quit IRC03:28
*** kotra03 has joined #openstack-powervm03:49
*** kotra03 has quit IRC03:59
*** thorst_ has joined #openstack-powervm04:10
*** dwayne has quit IRC04:16
*** thorst_ has quit IRC04:17
*** dwayne has joined #openstack-powervm04:27
*** dwayne has quit IRC04:46
*** mdrabe has quit IRC04:54
*** dwayne has joined #openstack-powervm05:02
*** thorst_ has joined #openstack-powervm05:15
*** dwayne has quit IRC05:17
*** thorst_ has quit IRC05:22
*** dwayne has joined #openstack-powervm05:31
*** dwayne has quit IRC05:47
*** dwayne has joined #openstack-powervm06:19
*** thorst_ has joined #openstack-powervm06:20
*** dwayne has quit IRC06:24
*** thorst_ has quit IRC06:27
*** dwayne has joined #openstack-powervm07:20
*** dwayne has quit IRC07:27
*** thorst_ has joined #openstack-powervm07:28
*** thorst_ has quit IRC07:32
*** kotra03 has joined #openstack-powervm07:41
*** dwayne has joined #openstack-powervm07:49
*** k0da has joined #openstack-powervm07:54
*** dwayne has quit IRC07:56
*** dwayne has joined #openstack-powervm08:17
*** dwayne has quit IRC08:26
*** thorst_ has joined #openstack-powervm08:30
*** thorst_ has quit IRC08:37
*** dwayne has joined #openstack-powervm08:42
*** dwayne has quit IRC08:57
*** kwazar is now known as quasar09:23
*** dwayne has joined #openstack-powervm09:35
*** thorst_ has joined #openstack-powervm09:35
*** thorst_ has quit IRC09:41
*** dwayne has quit IRC09:43
*** dwayne has joined #openstack-powervm09:57
*** dwayne has quit IRC10:13
*** dwayne has joined #openstack-powervm10:27
*** thorst_ has joined #openstack-powervm10:41
*** smatzek has joined #openstack-powervm10:42
*** dwayne has quit IRC10:43
*** thorst_ has quit IRC10:47
*** dwayne has joined #openstack-powervm11:22
*** dwayne has quit IRC11:28
*** thorst_ has joined #openstack-powervm11:35
*** dwayne has joined #openstack-powervm11:52
*** dwayne has quit IRC11:59
*** svenkat has joined #openstack-powervm12:13
*** dwayne has joined #openstack-powervm12:19
*** dwayne has quit IRC12:23
*** burgerk has joined #openstack-powervm12:27
*** mdrabe has joined #openstack-powervm12:29
*** dwayne has joined #openstack-powervm12:37
*** tblakes has joined #openstack-powervm12:58
*** edmondsw has joined #openstack-powervm13:11
*** seroyer has joined #openstack-powervm13:17
*** dwayne has quit IRC13:22
*** catintheroof has joined #openstack-powervm13:36
*** esberglu has joined #openstack-powervm13:36
*** catintheroof has quit IRC13:43
*** burgerk has quit IRC13:54
*** apearson has joined #openstack-powervm13:56
*** efried has joined #openstack-powervm14:06
*** dwayne has joined #openstack-powervm14:17
*** kotra03 has quit IRC14:23
adreznecMan, the OVH gate is so unstable14:28
adreznecI feel like it's basically a game of "recheck until your jobs run in the OSIC or Infra clouds" at this point...14:29
openstackgerritEric Berglund proposed openstack/nova-powervm: DNM: ci check  https://review.openstack.org/32831514:30
*** lmtaylor1 has joined #openstack-powervm14:31
*** mdrabe has quit IRC14:34
*** mdrabe has joined #openstack-powervm14:35
*** burgerk has joined #openstack-powervm14:37
thorst_adreznec: that bad?15:06
*** tjakobs has joined #openstack-powervm15:09
openstackgerritEric Berglund proposed openstack/nova-powervm: DNM: CI Check2  https://review.openstack.org/32831715:11
adreznecthorst_: Yeah... it varies throughout the day15:11
adreznecBut there are periods where checks fail just installing apt packages because their internal mirrors are unreachable15:12
thorst_wonder what'll be like with POWER at somepoint  :-)15:13
thorst_efried: cna_vlan_sync up for review again15:25
efriedthorst_, ack.15:26
*** k0da has quit IRC15:30
efriedthorst_, couple comments not addressed.  Want to discuss?15:35
efriedPerhaps you have a response queued up but haven't submitted yet?15:36
thorst_efried: probably don't...probably just missed15:39
efriedthorst_, looks like you're gonna need a rebase anyway.15:44
*** dwayne has quit IRC15:45
*** seroyer has quit IRC16:02
*** seroyer has joined #openstack-powervm16:28
*** dwayne has joined #openstack-powervm16:50
thorst_efried: 3752 - fired back at you.16:57
thorst_we may need a third party arbiter16:58
thorst_like btang16:58
efriedIt was btang's idea16:59
thorst_to raise up a legit error?16:59
thorst_rather than force through?16:59
efriedSee his second comment in PS317:00
efriedPS4, rather17:00
thorst_efried: OK - I pinged him to let him know17:01
thorst_said he can pick which way we go17:01
efriedYou still need to rebase.17:02
thorst_yep yep17:03
thorst_just want to know direction before I do17:03
efriedSo what's the behavior as written?  It'll fail by default if there's a mkvterm open?17:04
thorst_yeah, that's the default17:06
thorst_I think we're arguing what we have nova-powervm do, but in the pypowervm change set17:06
adreznecThat's not very transparent thorst_17:07
adreznecWhere's your community spirit17:08
adreznecGuess you'll have to go to a session on learning to contribute at the next summit17:09
thorst_did that get accepted?17:09
adreznecHasn't been announced yet17:09
thorst_I was going to say...17:09
adreznecAug 25th iirc17:09
thorst_adreznec: how's OSA stuff going?  Should I help out?17:11
adreznecthorst_: I'm not doing OSA stuff right now. Trying to figure out issues with storage deletion right now17:11
efriedthorst_, so what are you advocating?  Changing it back to what it was before, no force flag, blow away the existing vterm by default?17:12
thorst_leave force flag in, have nova-powervm set it to false17:12
thorst_change if we get feedack17:12
adreznecMappings aren't getting cleaned up, and Hsien says the POSTs to do so aren't coming down to REST17:12
efriedthorst_, so did you mean to have the pypowervm default the force flag to True?17:13
thorst_efried: nope17:13
thorst_I've a few changes to make there17:13
thorst_I'll default to false.17:14
thorst_and I have to add it elsewhere too17:14
thorst_there's a wrapper method17:14
thorst_so I'll make that change in a few hours.17:14
efriedSo then I'm confused.  I thought you were wanting to make the force the default behavior.17:15
efriedI'm fine with this as you've written it.17:15
thorst_OK. let me get a change in...we're likely in agreement then17:16
efriedWould like to see the nova-powervm side convert to that ConsoleUnavailable error btang mentioned, if that'd help the user experience.17:16
efriedGuess I'm confused by your last comment, saying you don't want to do that.  Why?17:16
thorst_well, I didn't want to.  btang and you convinced me to let it be OK.  Agree, nova-powervm needs to convert the error to something better.17:17
efriedSeems to me like this ought to be a fairly rare occurrence, no?17:17
thorst_it depends.17:17
thorst_right now, since people don't have console, its happening a lot17:18
efriedadreznec: do these need to be updated in light of recent events?  https://github.com/openstack/nova-powervm/blob/master/devstack/local.conf.aio17:18
thorst_there is a lot of 'mkvterm'ing right now17:18
adreznecefried: Definitely. Maybe esberglu could fix them up based on the work he's doing for the CI nodes17:18
efriedadreznec: are you still working on getting a golden AIO going?17:18
adreznecWon't be identical, but should be close17:19
adreznecefried: I had one working, but mine used linux bridge17:19
adreznecI'd imagine you're more interested in a VIOS-based one17:19
efriedI'm about to try to stack svenkat's system, so I'm going to need one real soon.17:19
thorst_efried: mine worked pretty much as is17:19
thorst_a few days back at least17:20
efriedwith the above link?17:20
*** smatzek has quit IRC17:20
thorst_I just had to change to DATABASE_PASSWORD17:20
adreznecesberglu was working on the updated CI one17:20
efriedBut didn't we have to take out the enabling of neutron and stuff?17:20
*** smatzek has joined #openstack-powervm17:20
efriedsmh, I just know I'm about to dive into this whirlpool of stack, unstack, tweak, restack.  Typically takes me like three days to get a freakin stack going.17:21
efriedNot gonna lie, one of the most frustrating parts of my job.17:23
efriedadreznec: What did you do to get rabbitmq working?17:42
efriedI copied the sources.list straight over from neo3417:42
efriedThe version appears to be the same17:43
efriederlang appears the same too17:44
adreznecefried: iirc, I purged your rabbit/erlang installs/config, cleaned the apt cache, replaced your apt config with a good copy, did a full upgrade of everything, then reinstalled rabbit/erlang/dependencies17:51
efriedMay have to ask you what some of that means in a minute.  Gonna try "upgrade everything", cause that's the only bit I understand.  If that doesn't work...17:52
*** k0da has joined #openstack-powervm17:58
thorst_efried: You have time to tackle a minor bug?18:06
thorst_its a doc thing we got from a user18:06
efriedthorst_, okay, may as well while I'm trying to stack.18:06
openstackLaunchpad bug 1613723 in nova-powervm "Required group: pvm_admin" [Undecided,New]18:07
thorst_its really quite simple18:07
thorst_1) We didn't doc that these drivers require that the process running the PowerVM Compute driver be part of the pvm_admin user group18:07
thorst_2) We now have users directly trying to use the docs18:08
adreznec3) Fix docs18:08
adreznec4) Profit???18:08
thorst_adreznec: this gets updated with each master merge?  http://nova-powervm.readthedocs.io/en/latest/#nova-powervm-devref18:08
adreznecShould be18:08
adreznecthorst_: Man, VIF plugging seems really slow...18:09
adreznecIs Slack down for anyone else?18:10
tjakobsSlack is down for me18:13
esbergluadreznec: efried: Sorry just catching up on this thread. I can update the local.conf.aio based on the one I am using for CI18:21
thorst_slack being down would be great.18:36
thorst_adreznec: what type of vif plug?18:36
*** k0da has quit IRC18:38
adreznecthorst_: linux bridge18:49
adreznecTakes like 200 seconds...18:49
efriedthorst_, adreznec: Where is the source for that devref?18:49
efriedIt doesn't appear to be in nova-powervm/doc/source/devref18:49
*** k0da has joined #openstack-powervm18:49
efriednm, found it.18:50
thorst_adreznec: 200 seconds?18:53
thorst_that's ridiculous18:53
thorst_we need to profile that.18:53
thorst_it was like 2 seconds before...18:53
adreznecYeah, at least I think thats what's taking the time. Haven't dug deeper into it18:55
thorst_efried: mind taking a pass at 3696 when you get a sec18:55
efriedthorst_, ack18:55
thorst_thx dude18:55
*** apearson has quit IRC18:59
*** apearson has joined #openstack-powervm19:01
openstackgerritEric Fried proposed openstack/nova-powervm: devref usage: n-cpu must run from pvm_admin group  https://review.openstack.org/35609919:04
efriedthorst_ ^^ Not sure if this is what you had in mind.19:04
efriedthorst_, 3696 needed a rebase; did that and added +1.  Who else should look at this before merge?19:07
esbergluthorst: I’m also still hitting very slow vif plugging on the CI. Most of the failing tests error out waiting for VIFs to be created19:08
adreznecMaybe a neutron change19:09
esbergluNope I’ve been hitting this for a while. I thought it was already on thorst’s radar19:09
esbergluAt least not a recent change19:09
thorst_efried: adreznec should look at 369619:13
thorst_esberglu: the slowness on the CI for networking-powervm would've been something different19:14
thorst_but, its weird cause I'm 100% not hitting it in my testing19:14
thorst_I did 200 VMs on 4 hosts in 8 minutes.19:14
esbergluAlmost every time plug_vifs takes 0 or 1 second. But when it doesn’t it19:16
esberglujust gets out of control19:16
esbergluI’ve seen 400+ seconds19:16
seroyerthorst_ adreznec: I see plug_vifs completed in 28 seconds from your last spawn19:17
seroyerLooking in the complete history, I see several spawn attempts and plug_vifs took between 11 and 28 seconds.  Most in the mid to upper teens.19:18
seroyerThe times in spawn order were: 11, 14, 16, 14, 14, 15, 18, 19, 2819:21
adreznecseroyer: So what else are you seeing for timings?19:21
adreznecSomething else long running? Image copy?19:21
seroyerWorking backwards from the last spawn: cfg_drive 8, crt_disk_from_img 34,19:23
seroyerEverything else was smallish, like 0-2 seconds.19:24
seroyer(plus the plug_vifs of 28 seconds)19:24
seroyerNow, I’m not looking at timestamps, just the “Task xxxx completed in X seconds” messages.19:26
thorst_seroyer: I'm not super worried about 11-28 seconds.  I bet most of that is the REST19:37
thorst_seroyer: and yeah, those 'Task xxxx completed in' was put in specifically to debug those19:37
seroyerI’m don’t have much experience with these logs, but it looks to me like a the beginning of the spawn, I see it taking about 16 seconds to audit available compute resources.  That seems like a long time to me.19:43
*** miltonm has joined #openstack-powervm19:49
seroyerInterestingly enough, the time that plug_vifs to 28 seconds (longest), was the shortest overall spawn time (111 seconds total).  Other total spawn times were all around 200-220 seconds.19:50
*** apearson has quit IRC19:52
thorst_seroyer: that is a long time.19:57
thorst_are you using volumes?19:57
seroyerEphemeral only.19:57
thorst_too long then19:57
thorst_if cinder was involved...I could see it19:57
thorst_is the NL backed up?19:58
thorst_cpu wise, io wise, etc?19:58
seroyerDisk is RAID 1 SSD19:58
seroyer6 dedicated CPUs, 16GB memory.  devstack deploy with compute only.19:58
seroyerNo, separate controller.19:59
thorst_ps aux | grep meta19:59
thorst_how many?19:59
seroyerzero, but adreznec might have just unstacked…19:59
thorst_zero seems more or less right20:00
adreznecNo, didn't unstack yet20:02
*** apearson has joined #openstack-powervm20:02
adreznecAnd yeah, no metadata service should be fine20:02
adreznecI am unstacking now though20:03
*** svenkat has quit IRC20:30
thorst_tblakes: I think this needs to add a 'name=instance.name'20:31
*** tblakes_ has joined #openstack-powervm20:32
thorst_tblakes: its actually...20:32
*** tblakes has quit IRC20:35
*** tblakes_ is now known as tblakes20:35
efriedthorst_, should we run both pvm_sriov and pvm_sea network agents, always?20:40
efrieddevstack-wise, that is20:40
efriedToday we run pvm_sea always20:40
thorst_that seems reasonable20:40
efriedOr should we try to introduce logic to config ml2_conf.ini based on local.conf settings?20:40
thorst_well, its not a ml2 conf setting20:41
thorst_ml2 conf tells a process how to run20:41
efriedWhich agents are running...20:41
thorst_you're figuring out whether or not to run20:41
thorst_and I'd say....sure, lets always run the agent20:41
thorst_as long as it doesn't bomb out if there are no SR-IOV ports.20:41
thorst_if it does that ... then maybe auto detect or something20:41
efriedtenant_network_types = vlan20:41
efriedextension_drivers = port_security20:41
efriedtype_drivers = vlan20:41
efriedmechanism_drivers = pvm_sea,pvm_sriov20:41
efriedThat's in ml2_conf.ini20:42
thorst_well, that's for the server side neutron, isn't it?20:42
efried"server side".20:42
thorst_to know which type of agents the server supports20:42
thorst_neutron server process...not agent process20:42
efriedI don't know.  Was going to ask you and svenkat to help me understand which bits run where.20:43
thorst_yeah, its kinda confusing cause its like one config file for everything20:43
efriedWhat I do know is that right now, pvm-q-agt is running the sea agent only.  We're going to need two separate processes now.  Thinking renaming pvm-q-sea-agt and pvm-q-sriov-agt20:43
adreznecefried: Can you actually run both at once?20:44
thorst_adreznec: only the SR-IOV agent20:44
thorst_can't run SEA and OVS for instance at same time20:44
adreznecThat's why I was asking20:44
thorst_would be awesome if we could  :-D20:47
thorst_but neutron doesn't expose that port detail to us20:47
thorst_adreznec: you have a multi-node local.conf handy?20:55
efriedthorst_, do we want csg to incorporate SR-IOV community blueprints?20:55
adreznecthorst_: With linux bridge, sure20:55
thorst_adreznec: send to me on slack?20:55
thorst_sure  :-)20:55
*** smatzek has quit IRC20:58
adreznecI think that's a great idea20:58
*** thorst_ has quit IRC21:02
*** tblakes_ has joined #openstack-powervm21:07
efriedadreznec: Any reason this should be hanging?21:09
efried+lib/neutron_plugins/ovs_base:neutron_ovs_base_cleanup:46  sudo ovs-vsctl del-br br-int21:09
adreznecefried: Uh... are you running OVS?21:10
efriedhell if I know21:10
adreznecThis is regular novalink, right?21:11
*** tblakes has quit IRC21:11
*** tblakes_ is now known as tblakes21:11
efriedadreznec, yeah21:14
efriedthough not sure what "regular" means.21:14
efriedOh, as opposed to SD*?21:14
adreznecefried: yeah, that's what I meant21:19
adreznecYou're standard VIOS environment, so you shouldn't have any OVS in there21:20
*** lmtaylor1 has left #openstack-powervm21:25
*** tjakobs has quit IRC21:34
*** thorst_ has joined #openstack-powervm21:39
*** thorst_ has quit IRC21:43
*** dwayne has quit IRC21:50
*** thorst_ has joined #openstack-powervm21:54
*** edmondsw has quit IRC21:58
*** apearson has quit IRC22:02
*** seroyer has quit IRC22:04
thorst_efried: It finally happened!22:06
thorst_Sonar caught something useful!22:06
thorst_its a miracle!22:06
efriedwow, awesome.22:06
efriedMark this day22:06
thorst_it shall go down in the history books22:07
thorst_adreznec: can you review 3696 sometime tomorrow?22:07
thorst_its got the +1 (had the +1) from efried, but he justifiably wants a second set of eyes22:07
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Update devstack local.conf  https://review.openstack.org/35614622:11
*** esberglu has quit IRC22:16
*** mdrabe has quit IRC22:17
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: Wrap console failure message  https://review.openstack.org/35614922:24
*** tblakes has quit IRC22:24
*** thorst_ is now known as thorst22:49
*** burgerk has quit IRC22:51
openstackgerritDrew Thorstensen (thorst) proposed openstack/networking-powervm: Fix bridge mapping change  https://review.openstack.org/35615923:11
*** thorst has quit IRC23:17
*** k0da has quit IRC23:17
*** thorst has joined #openstack-powervm23:17
*** thorst has quit IRC23:26

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!