Thursday, 2017-01-19

*** dwayne__ has joined #openstack-powervm00:13
*** kylek3h has joined #openstack-powervm00:14
*** jwcroppe has quit IRC00:14
*** jwcroppe has joined #openstack-powervm00:15
*** jwcroppe has quit IRC00:19
*** svenkat has quit IRC00:24
*** svenkat has joined #openstack-powervm00:28
*** k0da has quit IRC00:30
*** svenkat has quit IRC00:30
*** thorst has joined #openstack-powervm00:39
*** thorst has quit IRC00:42
*** svenkat has joined #openstack-powervm00:54
openstackgerritSridhar Venkat proposed openstack/nova-powervm: Migration scenario, SR-IOV vif contains source physical ports
*** thorst has joined #openstack-powervm01:11
*** thorst has quit IRC01:11
*** edmondsw has joined #openstack-powervm01:18
*** svenkat has quit IRC01:20
*** edmondsw has quit IRC01:23
*** thorst has joined #openstack-powervm02:15
*** thorst has quit IRC02:20
*** kylek3h_ has joined #openstack-powervm02:33
*** kylek3h has quit IRC02:41
*** dwayne__ has quit IRC02:41
*** dwayne__ has joined #openstack-powervm02:50
*** tlian has quit IRC03:27
*** thorst has joined #openstack-powervm03:30
*** thorst has quit IRC03:36
*** thorst has joined #openstack-powervm04:33
*** thorst has quit IRC04:38
*** nbante has joined #openstack-powervm05:04
*** Jay1 has joined #openstack-powervm06:03
*** Jay1 has left #openstack-powervm06:04
*** nbante has quit IRC06:27
*** thorst has joined #openstack-powervm06:34
*** thorst has quit IRC06:39
*** nbante has joined #openstack-powervm06:48
*** k0da has joined #openstack-powervm07:40
*** openstackgerrit has quit IRC08:33
*** thorst has joined #openstack-powervm08:35
*** thorst has quit IRC08:39
*** thorst has joined #openstack-powervm10:35
*** thorst has quit IRC10:40
*** xia has joined #openstack-powervm11:19
*** XiaYan has joined #openstack-powervm11:23
*** smatzek has joined #openstack-powervm11:43
*** nbante has quit IRC11:52
*** edmondsw has joined #openstack-powervm12:07
*** thorst has joined #openstack-powervm12:37
*** tblakes has joined #openstack-powervm13:08
*** tlian has joined #openstack-powervm13:14
*** svenkat has joined #openstack-powervm13:23
*** wangqwsh has joined #openstack-powervm13:27
*** esberglu has joined #openstack-powervm13:27
esberglu#startmeeting powervm_ci_meeting13:28
openstackMeeting started Thu Jan 19 13:28:40 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:28
openstackThe meeting name has been set to 'powervm_ci_meeting'13:28
esberglu#topic status13:31
esbergluYesterday I worked through some fixes for adreznec's DNS / apt patch13:31
esbergluIt's pretty much working now, a couple small changes are still needed13:32
thorstesberglu: pretty much working in that test runs have used it or...13:32
esbergluYeah. Test runs have used it on staging13:32
thorstfaster stronger?13:33
esbergluThey were faster, mostly finishing in under an hour13:35
thorstweird...and neat13:35
thorstcan you PM me the zuul from one of them?13:35
thorstI'd like to compare devstack times.13:35
esbergluBut the times are usually a little different between staging and prod13:35
thorstyeah, but devstack times shouldn't be too different13:36
thorstrun times...definitely different13:36
esbergluAs part of deploying that patch I realized that the CI nodes (mgmt, ctrl, log) were 14.04, I thought we had upgraded a while back13:37
esbergluBut apparently not13:37
adreznecYeah, we'll have to do a longer term comparison to be sure13:37
adreznecHow did the upgrade go esberglu?13:37
esbergluNot as well as I was hoping13:37
esbergluThe ctrl node is fine13:38
adreznecYeah, was afraid of that13:38
esbergluTo 16.0413:38
esbergluOn the staging CI servers13:38
thorstdid we capture the VM images for the controller nodes before upgrading?13:38
thorstI could just snapshot them13:38
thorstso you can fall back safely13:38
thorsttoo late for staging, but maybe something for prod?13:38
adreznecWell worst case scenario we just run the playbooks again right? :)13:38
thorstword...that's why we built them13:39
esbergluSeeing some issues with the log and mgmt nodes13:39
adreznecWhat went awry? bad config? missing packages?13:39
esbergluFor some reason puppet is downloading all the modules to the / directory on the log node13:39
esbergluWhen running install_modules.sh13:39
adreznecHmm. Maybe the path it was trying to use doesn't exist or something...13:40
esbergluThe mgmt node had some bad packages and some stuff with starting/stopping services13:40
esbergluI just upgraded them last night so I haven't dug too deep into the problems yet13:41
esberglu#action esberglu: Get staging CI nodes running on 16.0413:41
adreznecOk. Yeah these are pretty typical issues due to the upstart/systemd conversion13:41
esbergluAnd after that start hitting the dual CI13:42
esberglu#topic Dual CI13:42
esbergluI have some questions about this13:42
esbergluHow do we want to handle voting?13:43
thorstno vote yet.  Similar to what nova-powervm does (once stable of course...don't publish until stable)13:43
thorstdifferent CI name though.  Something that indicates in tree status13:43
esbergluOkay. Has somebody been testing the in tree with devstack?13:45
esbergluSo I can steal their local.conf?13:45
* thorst looks to efried13:45
adreznecThe PowerVM Oak CI. Or maybe maple? elm?13:45
efriedI have a local.conf13:45
thorsthar har har...13:45
efriedI don't get it13:46
efriedgot it13:47
efriedbore (in-tree) vs bark (oot)13:47
efriedor is it bole?13:47
adreznecOk, so efried is going to give his local.conf to esberglu and we'll go from there13:48
esbergluThat's pretty much all I have for CI. Get the staging nodes up and get the dual CI running there13:49
esbergluI might have some q's as I get further into the dual CI stuff13:49
thorstxia: I know you were working with wangqwsh.  I know that wangqwsh is helping efried with the in tree driver, but has experience with the CI.13:49
efriednot sure how we handle the fact that you have to spawn with no network13:49
thorstwondering out loud if it would make sense for you to become familiar with the CI as well.13:49
thorstefried: we can deploy w/ network, that network just won't get attached to the VM.13:50
thorstso it'll fake work13:50
efriedit broke when i tried it13:50
thorstwhat the what?  OK...I'll check out the error later.13:50
efriedmaybe a code change is needed.13:50
wangqwshthrow a lot of error about networking13:50
thorstahh, I bet I know what it you'd get around it using the SEA agent for now...13:51
efried#action efried & thorst to investigate spawn "with" network on the first few in-tree change sets13:51
thorstno, that'd be broken too.13:51
*** nbante has joined #openstack-powervm13:51
efriedback to #topic wangqwsh helping with CI?13:52
thorstwell, actually it would be xia getting some experience there from wangqwsh13:52
thorstwe clearly need backups for esberglu when he decides to go on vacation13:52
thorstand not just backups, but also driving some of that CI dev13:53
esbergluYeah with all 5 days I have left for the year...13:53
thorstnot sure its right to do that now...but at some point.13:53
thorstsomething to consider.13:53
esbergluI have been adding wangqwsh to the neo-os-ci reviews. I can add xia as well. Just to give some exposure13:54
thorstyeah, exposure is a good start.  I know wangqwsh and xia, you're both also in the driver code as well13:54
thorstso if that's the focus now...keep it as the primary13:54
thorstthat's all I had  :-)13:55
esbergluAnyone have anything else?13:56
efriednot I13:56
openstackMeeting ended Thu Jan 19 13:58:20 2017 UTC.  Information about MeetBot at . (v 0.1.4)13:58
openstackMinutes (text):
thorstefried: in other news...I posted up a first pass at fileio13:58
*** mdrabe has joined #openstack-powervm13:58
thorstit was really Shyama's, I just changed a couple things and posted it to review.o.o13:59
thorstbut it's damn simple.13:59
thorstand I totally dig that.13:59
efriedyuh, will look13:59
*** dwayne__ has quit IRC13:59
thorstadreznec: did you have a chance to get back to nbante on the ansible bits?14:00
*** jwcroppe has joined #openstack-powervm14:18
*** kjw3 has joined #openstack-powervm14:23
*** tblakes has quit IRC14:25
*** smatzek has quit IRC14:26
*** esberglu has quit IRC14:32
*** esberglu has joined #openstack-powervm14:32
*** esberglu has quit IRC14:36
*** mdrabe has quit IRC14:37
*** tblakes has joined #openstack-powervm14:38
efriedthorst: can you give a second +2 for svenkat here?
*** mdrabe has joined #openstack-powervm14:46
thorstefried: hasn't passed any CI yet14:46
thorstlooks like gate is borked14:46
*** apearson has joined #openstack-powervm14:48
*** dwayne__ has joined #openstack-powervm14:49
*** openstackgerrit has joined #openstack-powervm14:53
openstackgerritEric Fried proposed openstack/nova-powervm: Mock data cleanup 1
*** edmondsw_ has joined #openstack-powervm14:55
*** edmondsw has quit IRC14:56
*** esberglu has joined #openstack-powervm14:59
efriedthorst: you want to put an eyeball on 4729 before we merge it?15:01
*** burgerk has joined #openstack-powervm15:12
adreznecesberglu: did you manage to get the named.conf.options templating between prod and staging figured out yesterday?15:13
adreznecWasn't sure if that last syntax I had you try worked or not15:14
*** smatzek has joined #openstack-powervm15:14
esbergluNo that syntax didn't work. I just hardcoded it. That's the only thing left to get working on that patch15:14
esbergluBut I think there is an "all" keyword that I might be able to use there15:15
esberglufor host in groups['all']15:15
adreznecHmm, even if they're in separate inventories?15:15
*** smatzek_ has joined #openstack-powervm15:15
esbergluI'm using the log server as the cache node for staging15:16
esbergluSo it wouldn't need separate inventories15:16
adreznecAh right15:16
adreznecI was thinking otherwise we could just do an if/elif block15:16
thorstefried: looking at 4729 now.15:17
adreznecsaying if host in group prod do this, elif in staging do this, else fail15:17
*** smatzek has quit IRC15:19
thorstefried: LGTM15:20
*** tjakobs has joined #openstack-powervm15:24
*** kylek3h_ is now known as kylek3h15:33
*** mdrabe has quit IRC15:48
efriedthorst: got some time now to look at the spawn-with-no-network thing?15:58
*** mdrabe has joined #openstack-powervm16:00
*** apearson has quit IRC16:00
efriedhm, it's possible it's still trying.  We'll know in about 2 minutes.  I guess it won't interrupt the boot disk upload?  So it'll either fail/revert after that, or it'll "succeed".  And if it "succeeds", maybe it ends up in error state because of the network failure?16:01
*** jwcroppe has quit IRC16:01
efriedAnyway, the error is:16:01
efriedPortBindingFailed: Binding failed for port a6ee9db3-ed17-4c79-84c2-a03428b65d5e, please check neutron logs for more information.16:02
*** jwcroppe has joined #openstack-powervm16:02
efriedAnd in q-svc:16:02
efriedFailed to bind port a6ee9db3-ed17-4c79-84c2-a03428b65d5e on host neo40 for vnic_type normal using segments [{'segmentation_id': 1054, 'physical_network': u'default', 'id': u'262f1eb9-722e-4150-bbbf-f447c02cfa2e', 'network_type': u'vlan'}]16:02
*** jwcroppe_ has joined #openstack-powervm16:03
thorstesberglu: see openstack-nova convo between markus_z and myself.  Looks like we missed a milestone16:04
thorstefried: yeah, so what's happening is there is no agent binding the port16:04
thorstso to work, you'd need an agent that basically 'no-ops'.  OVS won't work (not a no-op).  SEA won't work now either because it expects the server to send a message...16:05
thorstso I think I was wrong on that...and we do need network stuff.16:05
thorstmaybe as soon as PS-1 or 2?16:05
thorstunless there is a deploy with no network test...16:05
*** jwcroppe has quit IRC16:06
adreznecthorst: Yeah, we should be more diligent on that, but there's no point in adding 4.0.0b1 now. We'll just have to do extra well at hitting 4.0.0b3 next week :)16:06
thorstadreznec: agree.16:06
*** dwayne__ has quit IRC16:07
esbergluYeah. I totally forgot about b1 until it was already b2. I have the reminders now for milestones, etc.16:08
efriedthorst: So either we find the set of tests that do deploys with no NIC (if any) or we have to put a PlugVifs into the first change set?16:10
thorstI'd say, look for the tests without any nics first16:10
thorstcause I really don't want to do the later.16:11
thorstits too big already...16:11
thorsthell, maybe even propose a tempest test that does that.16:11
thorstI did a no network deploy yesterday, so I know its possible.16:11
thorstat least from the API16:11
efriedYeah, that's how I tested the first change set.16:13
efriedthorst esberglu adreznec We have another problem.16:13
efriedI hadn't yet looked at how the CI was failing on the in-tree change sets.16:13
efriedReady for this?16:13
efriedDuplicateOptError: duplicate option: proc_units_factor16:13
efriedThe OOT and in-tree drivers are (of course) trying to register the same opts in the same groups.16:14
efriedI think this is going to be aaight for the in-tree driver cause the OOT driver won't be there to conflict.  But when the first in-tree change set merges, the OOT CI (and for that matter, any consumption of the OOT driver) will start failing with this error.16:14
efriedWe could rename the group for one or the other, which is kind of gross.16:15
thorstfix it once it merges in the OOT16:15
thorstbut wait till it happens16:15
efriedokay, but fix it how?16:15
efriedI guess we can just take that opt out of the OOT, eh?16:15
efriedThat's kind of weird.16:15
thorstcause once it's merged, its inline16:15
efriedBut okay.16:15
thorstit doesn't matter where the opt long as it's doc'd16:16
efriedPITA for someone trying to find the source for a given opt.16:16
thorstadmins shouldn't be looking (or shouldn't need to) in our code for that info16:16
efriedOkay, fair enough.16:16
thorstmeh, growing pains16:16
efriedIdea being we'll eventually have all of the opts in-treee.16:16
thorstnot sure we'll do all....I'm not sure how I feel about proposing that FC nightmare in tree16:17
thorstbut we'll cross that bridge when we get to it16:17
thorstthat FC stuff is a real...meh16:17
*** thorst is now known as thorst_afk16:19
*** jwcroppe_ has quit IRC16:37
*** thorst_afk is now known as thorst_16:37
*** jwcroppe has joined #openstack-powervm16:39
thorst_adreznec: you may be the one to want to answer markus_z's latest q in openstack-nova16:39
*** apearson has joined #openstack-powervm16:40
*** XiaYan has quit IRC16:46
*** xia has quit IRC16:46
wangqwshefried: you can use this patch set to fix opts conflict issue.
openstackgerritboden proposed openstack/networking-powervm: Use neutron-lib portbindings api-def
*** jwcroppe has quit IRC16:59
*** jwcroppe_ has joined #openstack-powervm17:02
*** dwayne__ has joined #openstack-powervm17:02
*** wangqwsh has quit IRC17:10
efriedthorst_ ^^ see my comments.  Should we go ahead and offer up subsequent patches or wait for boden to do it?17:23
thorst_efried: well, he did us a favor, but might be worth pinging him to ask17:23
efriedDon't see him in IRC.  Should I email him?17:24
efriedah, got him in #openstack-neutron17:25
*** apearson has quit IRC17:33
*** nbante has quit IRC17:45
*** nbante has joined #openstack-powervm17:45
openstackgerritTaylor Jakobson proposed openstack/nova-powervm: First pass at imagecache
*** jwcroppe_ has quit IRC18:03
openstackgerritSridhar Venkat proposed openstack/nova-powervm: Migration scenario, SR-IOV vif contains source physical ports
*** jwcroppe has joined #openstack-powervm18:07
*** nbante has quit IRC18:19
openstackgerritTaylor Jakobson proposed openstack/nova-powervm: First pass at imagecache
thorst_grrr...figured out our volume connector decision is across the whole host19:14
thorst_it shouldn't be like that...19:14
*** openstackgerrit has quit IRC19:33
*** k0da has quit IRC19:38
*** k0da has joined #openstack-powervm19:38
*** kjw3 has quit IRC20:14
*** kjw3 has joined #openstack-powervm20:14
efriedthorst_: you in nova mtg?21:21
efriedI had to make a statement on when we anticipated the split-CI would be running and passing.  esberglu - do you have thoughts on that yet?21:22
*** svenkat has quit IRC21:27
adreznecefried: seems like it went over pretty well regardless. onus is back to CI then21:28
efriedYes - esberglu you around?21:28
*** smatzek_ has quit IRC21:30
thorst_and yeah, I wasn't in the nova meeting, but +1 on how efried handled it (which was - he handled it well)21:37
*** openstackgerrit has joined #openstack-powervm21:38
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: WIP: File I/O Volume Driver Implementation
openstackgerritDrew Thorstensen (thorst) proposed openstack/nova-powervm: WIP: File I/O Volume Driver Implementation
thorst_efried adreznec: ^^21:39
thorst_couple things on that one...21:39
thorst_1) I added gpfs, local and nfs.  The model matches how libvirt does it, which is kinda neat.21:39
thorst_2) I reswizzled the volume driver picker stuff.  I want to follow up on that in a separate change set too21:40
thorst_but before I go too far into UT, take a peak?21:40
thorst_one of the other changes in there is that a given VM should be able to support cinder volumes of different types.  Some could be iSCSI, others could be NFS, others could be GPFS21:41
thorst_I think that this basically allows that...except we have to force a binary decision between FC and iSCSI for the host name resolution...which stinks.21:42
*** thorst_ has quit IRC21:45
*** jwcroppe has quit IRC21:51
*** jwcroppe has joined #openstack-powervm21:51
*** jwcroppe has quit IRC21:55
*** thorst_ has joined #openstack-powervm22:11
esbergluthorst_: efried: adreznec: I started working through some of the basics for the dual CI today.22:14
esbergluMost of the stuff can be reused from the other job22:14
efriedGood deal.22:14
esbergluThere is gonna be some changes needed to the prep_devstack script (or a new script, haven't decided yet)22:14
efriedTo level-set: the in-tree CI needs to be your first and only priority until it's green.  If you get distracted by anything else, let us know so we can deflect.22:15
*** thorst_ has quit IRC22:15
esbergluAnd some changes to the powervm-ci repo. I think the script used there only supports a blacklist, not a whitelist of tests22:16
efriedThe nova cores are hesitant to start reviewing our code until we have it passing.  And if we are to have any chance of getting the first change set into ocata, that needs to happen far enough in advance of code freeze for them to get all the way through it.22:16
esbergluHow are we creating the whitelist of tests? Do we have a plan for that?22:17
efriedYou mean how are we going to see which tests will pass?22:19
efriedThat would be a good start to constructing the whitelist - get the OOT CI to run on in-tree-only (without loading up the OOT stuff) and see what passes (if anything)22:19
efriedThen we can go through and add tests we think should also be passing, if we see any - but that would be a great baseline.22:20
efriedAt the moment, the in-tree change sets are failing to get started because the compute driver sees conflicting config opts.  If the OOT code wasn't present, we would get past that22:20
efried...and on to the next failure ;-)22:21
*** k0da has quit IRC22:29
*** thorst_ has joined #openstack-powervm22:40
*** tblakes has quit IRC22:42
*** smatzek_ has joined #openstack-powervm22:43
*** thorst_ has quit IRC22:43
*** jwcroppe has joined #openstack-powervm22:47
*** smatzek_ has quit IRC22:48
*** esberglu has quit IRC22:50
*** esberglu has joined #openstack-powervm22:50
*** esberglu has quit IRC22:55
*** burgerk has quit IRC22:56
*** dwayne__ has quit IRC23:01
*** mdrabe has quit IRC23:02
*** esberglu has joined #openstack-powervm23:04
*** esberglu has quit IRC23:09
*** esberglu has joined #openstack-powervm23:13
*** esberglu has quit IRC23:17
*** esberglu has joined #openstack-powervm23:19
*** esberglu has quit IRC23:23
*** tjakobs has quit IRC23:39
*** thorst_ has joined #openstack-powervm23:44
*** thorst_ has quit IRC23:48

Generated by 2.14.0 by Marius Gedminas - find it at!