15:01:12 <lennyb> #startmeeting third-party
15:01:12 <openstack> Meeting started Mon Feb  6 15:01:12 2017 UTC and is due to finish in 60 minutes.  The chair is lennyb. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:15 <openstack> The meeting name has been set to 'third_party'
15:01:20 <lennyb> Hello
15:01:45 <asselin_> hi
15:01:50 <mmedvede> o/
15:01:56 <mptacekx> hi
15:03:02 <lennyb> hello all, any questions/updates ?
15:04:27 <mptacekx> nothing from my side, thanks
15:04:56 <asselin_> nothing from me either
15:05:21 <mmedvede> same here
15:07:08 <pots> i have a question...
15:07:22 <lennyb> go ahead pots
15:07:50 <pots> i'm seeing failures where the problem seems to be that tempest.conf is not being updated with the parameters for my cinder driver
15:08:35 <pots> I have a 'test-config' secction in my local.conf and am setting the TEMPEST_* variables.
15:09:06 <pots> The failure is intermittent, which is really odd.
15:09:42 <pots> anyone seen this before?
15:10:26 <lennyb> pots, what version of devstack are you using?
15:10:49 <pots> master
15:11:46 <pots> i just found this so i haven't had a chance to really dig in to the logs yet
15:12:03 <lennyb> pots, can you share your tempest.conf?
15:13:23 <lennyb> pots, this is our tempest.conf and local.conf http://13.69.151.247/89/428789/2/check-cinder/Cinder-ISER-LIO/10f21d2/tempest.conf.gz  http://13.69.151.247/89/428789/2/check-cinder/Cinder-ISER-LIO/10f21d2/local.conf.gz
15:14:37 <mmedvede> pots: one thing I can suggest is to look through your devstack.log. It might be possible to see when/who is changing tempest.conf. You can start by grepping for 'tempest.conf'
15:16:31 <pots> i'll dig around some more and get back to you.
15:16:32 <mmedvede> pots: e.g. it is possible that tempest.conf is being cleared after you set your parameters. But it is indeed strange that the behavior is intermittent
15:17:31 <mptacekx> I see that it's propagated correctly from local.conf to tempest.conf
15:17:42 <mptacekx> what the problem in there ?
15:18:04 <pots> that's just it, sometimes it's not propagated.
15:18:46 <pots> I have a local.conf section and TEMPEST_VOLUME_VENDOR and TEMPEST_STORAGE_PROTOCOL set
15:19:08 <pots> and it works most of the time
15:19:59 <pots> when it failed, the tempest.conf.gz that was uploaded didn't included the changes to the [volume] section
15:20:24 <pots> and i fail the test_volume_crud_with_volume_type_and_extra_specs tests
15:21:08 <pots> e.g. here: http://os-logs.tristero.net/39/429439/1/check/lenovo-iscsi/22e59ae/
15:22:02 <pots> but i haven't had a chance to read through all the logs yet
15:23:05 <lennyb> pots, I cant find stack.sh.log
15:23:33 <mmedvede> lennyb: it is devstacklog.txt.gz
15:23:45 <lennyb> mmedvede, thank
15:25:42 <mptacekx> sorry, I still see all records from [[test-config|$TEMPEST_CONFIG]] in tempest.conf if you miss some other config values from [[post-config]] or [[post-extra]] those might be overwritten by devstack
15:27:52 <pots> it's missing some iniset commands that appear in the logs of a successful run, sorry i wasn't more prepared--i thought this might be a known issue
15:29:46 <pots> i don't see a section that's doing 'iniset $TEMPEST_CONFIG ...'
15:30:01 <mptacekx> we hit similar behaviour as we're also using iniset in our setup for tempest.conf changes but due to some change in devstack tempest is now configured after those runs and it's rewritting it. Therefore we had to move to this new test-config phase with our adjustments and keept iniset just in stable branches
15:30:41 <mmedvede> this ^, we did similar adjustments
15:31:15 <pots> me too
15:31:33 <mmedvede> so maybe the intermittent behavior is due to stable vs master patches?
15:32:36 <pots> in the failing test, i see there is no "run_phase test-config" section
15:35:11 <pots> does my test-config section look like it should?  http://os-logs.tristero.net/39/429439/1/check/lenovo-iscsi/22e59ae/logs/local.conf.txt.gz
15:38:57 <mmedvede> pots, not sure, but unexpanded variables look suspicious to me
15:39:21 <lennyb> pots, looks ok, I can suggest trying 'RECLONE=yes' and deleting destination stack folder, just to be sure. I do not see any errors ( except nova db ) in the stack log
15:40:36 <lennyb> mmedvede, what do you mean by 'unexpanded' ?
15:40:54 <mmedvede> pots: e.g. are you sure that "[[test-config|$TEMPEST_CONFIG]]" would have $TEMPEST_CONFIG substituted correctly?
15:41:08 <pots> it seems to work most of the time
15:41:30 <pots> e.g. here's a good run: http://os-logs.tristero.net/35/427435/14/check/lenovo-iscsi/e92ae46/logs/
15:41:37 <mmedvede> ok, just looks suspicious, as it requires those vars to be defined
15:42:00 <mptacekx> $TEMPEST_CONFIG is not defined on computes
15:42:31 <pots> also, would RECLONE and removing the dest folder apply to VMs launched by nodepool with the openstackci puppet scripts?
15:43:29 <pots> i'll get out my reading glasses and compare the devstacklog.txt files more closely
15:45:15 <mmedvede> RECLONE is devstack setting (or d-g?), not sure how openstackci puppet configuration is relevant here
15:45:32 <lennyb> ok, any thing else on this issue ?
15:45:46 <pots> no, thanks.
15:46:10 <mptacekx> since this commit https://review.openstack.org/#/c/409123/ if such variables ar enot filled, stacking won't fail, just print warning
15:46:11 <lennyb> any other issues/questions ?
15:46:16 <mmedvede> I have a couple of things to bring up, in case people have missed those
15:46:45 <lennyb> mmedede, pls go
15:46:52 <mmedvede> there were a few changes last week that could have an effect on CIs
15:46:59 <mmedvede> #link [devstack] remove deprecated IRONIC_IPMIINFO_FILE https://review.openstack.org/#/c/427960/1
15:47:26 <mmedvede> this one is for ironic CIs. We hit that problem, and the fix is to use the new name
15:47:34 <mmedvede> #link placement-api is now required http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html
15:48:20 <mptacekx> I have one comment to that
15:48:28 <mmedvede> In case if you have ENABLED_SERVICES manually set in your gate, you now have to also have placement-api there
15:49:02 <mptacekx> on computes it should be rather placement-client, but it won't work on stable branches like stable/newton
15:49:24 <mptacekx> at least as per our observation
15:50:37 <mmedvede> mptacekx: this is a very good bit of information. We only have a single node gate where we were manually setting services
15:51:44 <mptacekx> mmedvede: I hit that problem recently with multinodes, therefore I am glad that I can share my findings here
15:52:30 <mmedvede> ok, I think our multinode is mostly red now. Maybe this is why! I have a few greens that could be stable branch tests
15:52:38 <mmedvede> mptacekx: thanks for that!
15:52:49 <mptacekx> mmedvede: you're welcome
15:53:35 <mmedvede> that is all for my announcements
15:53:42 <lennyb> thanks mmedvede, mptacekx
15:53:58 <mptacekx> mmedvede: it should work also with placement-api on compute, to be honest I didnt find difference but placement-client is a dummy service which is more appropriete as we just need to configure placement section of nova.conf on all machines with n-cpu
15:55:15 <mmedvede> mptacekx: do you mind giving a linky to your logs, for my own reference?
15:55:35 <mmedvede> if you have that public
15:56:18 <mptacekx> sure, e.g. http://intel-openstack-ci-logs.ovh/2017-02-06/304621/11/check/tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial/3bdb2bd/
15:56:25 <lennyb> mmedvede, we are working with placement-api
15:56:34 <mmedvede> thanks mptacekx
15:56:50 <lennyb> mmedvede, what do you need?
15:57:36 <mmedvede> lennyb: I wanted to have a reference for working multinode config in case I have problems fixing it
15:57:46 <lennyb> mmedvede,  http://13.69.151.247/21/304621/11/check-mlnx-multinode/Nova-ML2-Sriov-Multinode/a9a0268/   our MultiNode CI with everything
15:58:06 <mmedvede> lennyb: awesome!
15:58:45 <lennyb> OK, out time is runnig out. thanks everybody for attending. See you next week.
15:59:02 <mmedvede> thanks lennyb
15:59:19 <lennyb> #endmeeting