15:01:12 #startmeeting third-party 15:01:12 Meeting started Mon Feb 6 15:01:12 2017 UTC and is due to finish in 60 minutes. The chair is lennyb. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:15 The meeting name has been set to 'third_party' 15:01:20 Hello 15:01:45 hi 15:01:50 o/ 15:01:56 hi 15:03:02 hello all, any questions/updates ? 15:04:27 nothing from my side, thanks 15:04:56 nothing from me either 15:05:21 same here 15:07:08 i have a question... 15:07:22 go ahead pots 15:07:50 i'm seeing failures where the problem seems to be that tempest.conf is not being updated with the parameters for my cinder driver 15:08:35 I have a 'test-config' secction in my local.conf and am setting the TEMPEST_* variables. 15:09:06 The failure is intermittent, which is really odd. 15:09:42 anyone seen this before? 15:10:26 pots, what version of devstack are you using? 15:10:49 master 15:11:46 i just found this so i haven't had a chance to really dig in to the logs yet 15:12:03 pots, can you share your tempest.conf? 15:13:23 pots, this is our tempest.conf and local.conf http://13.69.151.247/89/428789/2/check-cinder/Cinder-ISER-LIO/10f21d2/tempest.conf.gz http://13.69.151.247/89/428789/2/check-cinder/Cinder-ISER-LIO/10f21d2/local.conf.gz 15:14:37 pots: one thing I can suggest is to look through your devstack.log. It might be possible to see when/who is changing tempest.conf. You can start by grepping for 'tempest.conf' 15:16:31 i'll dig around some more and get back to you. 15:16:32 pots: e.g. it is possible that tempest.conf is being cleared after you set your parameters. But it is indeed strange that the behavior is intermittent 15:17:31 I see that it's propagated correctly from local.conf to tempest.conf 15:17:42 what the problem in there ? 15:18:04 that's just it, sometimes it's not propagated. 15:18:46 I have a local.conf section and TEMPEST_VOLUME_VENDOR and TEMPEST_STORAGE_PROTOCOL set 15:19:08 and it works most of the time 15:19:59 when it failed, the tempest.conf.gz that was uploaded didn't included the changes to the [volume] section 15:20:24 and i fail the test_volume_crud_with_volume_type_and_extra_specs tests 15:21:08 e.g. here: http://os-logs.tristero.net/39/429439/1/check/lenovo-iscsi/22e59ae/ 15:22:02 but i haven't had a chance to read through all the logs yet 15:23:05 pots, I cant find stack.sh.log 15:23:33 lennyb: it is devstacklog.txt.gz 15:23:45 mmedvede, thank 15:25:42 sorry, I still see all records from [[test-config|$TEMPEST_CONFIG]] in tempest.conf if you miss some other config values from [[post-config]] or [[post-extra]] those might be overwritten by devstack 15:27:52 it's missing some iniset commands that appear in the logs of a successful run, sorry i wasn't more prepared--i thought this might be a known issue 15:29:46 i don't see a section that's doing 'iniset $TEMPEST_CONFIG ...' 15:30:01 we hit similar behaviour as we're also using iniset in our setup for tempest.conf changes but due to some change in devstack tempest is now configured after those runs and it's rewritting it. Therefore we had to move to this new test-config phase with our adjustments and keept iniset just in stable branches 15:30:41 this ^, we did similar adjustments 15:31:15 me too 15:31:33 so maybe the intermittent behavior is due to stable vs master patches? 15:32:36 in the failing test, i see there is no "run_phase test-config" section 15:35:11 does my test-config section look like it should? http://os-logs.tristero.net/39/429439/1/check/lenovo-iscsi/22e59ae/logs/local.conf.txt.gz 15:38:57 pots, not sure, but unexpanded variables look suspicious to me 15:39:21 pots, looks ok, I can suggest trying 'RECLONE=yes' and deleting destination stack folder, just to be sure. I do not see any errors ( except nova db ) in the stack log 15:40:36 mmedvede, what do you mean by 'unexpanded' ? 15:40:54 pots: e.g. are you sure that "[[test-config|$TEMPEST_CONFIG]]" would have $TEMPEST_CONFIG substituted correctly? 15:41:08 it seems to work most of the time 15:41:30 e.g. here's a good run: http://os-logs.tristero.net/35/427435/14/check/lenovo-iscsi/e92ae46/logs/ 15:41:37 ok, just looks suspicious, as it requires those vars to be defined 15:42:00 $TEMPEST_CONFIG is not defined on computes 15:42:31 also, would RECLONE and removing the dest folder apply to VMs launched by nodepool with the openstackci puppet scripts? 15:43:29 i'll get out my reading glasses and compare the devstacklog.txt files more closely 15:45:15 RECLONE is devstack setting (or d-g?), not sure how openstackci puppet configuration is relevant here 15:45:32 ok, any thing else on this issue ? 15:45:46 no, thanks. 15:46:10 since this commit https://review.openstack.org/#/c/409123/ if such variables ar enot filled, stacking won't fail, just print warning 15:46:11 any other issues/questions ? 15:46:16 I have a couple of things to bring up, in case people have missed those 15:46:45 mmedede, pls go 15:46:52 there were a few changes last week that could have an effect on CIs 15:46:59 #link [devstack] remove deprecated IRONIC_IPMIINFO_FILE https://review.openstack.org/#/c/427960/1 15:47:26 this one is for ironic CIs. We hit that problem, and the fix is to use the new name 15:47:34 #link placement-api is now required http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html 15:48:20 I have one comment to that 15:48:28 In case if you have ENABLED_SERVICES manually set in your gate, you now have to also have placement-api there 15:49:02 on computes it should be rather placement-client, but it won't work on stable branches like stable/newton 15:49:24 at least as per our observation 15:50:37 mptacekx: this is a very good bit of information. We only have a single node gate where we were manually setting services 15:51:44 mmedvede: I hit that problem recently with multinodes, therefore I am glad that I can share my findings here 15:52:30 ok, I think our multinode is mostly red now. Maybe this is why! I have a few greens that could be stable branch tests 15:52:38 mptacekx: thanks for that! 15:52:49 mmedvede: you're welcome 15:53:35 that is all for my announcements 15:53:42 thanks mmedvede, mptacekx 15:53:58 mmedvede: it should work also with placement-api on compute, to be honest I didnt find difference but placement-client is a dummy service which is more appropriete as we just need to configure placement section of nova.conf on all machines with n-cpu 15:55:15 mptacekx: do you mind giving a linky to your logs, for my own reference? 15:55:35 if you have that public 15:56:18 sure, e.g. http://intel-openstack-ci-logs.ovh/2017-02-06/304621/11/check/tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial/3bdb2bd/ 15:56:25 mmedvede, we are working with placement-api 15:56:34 thanks mptacekx 15:56:50 mmedvede, what do you need? 15:57:36 lennyb: I wanted to have a reference for working multinode config in case I have problems fixing it 15:57:46 mmedvede, http://13.69.151.247/21/304621/11/check-mlnx-multinode/Nova-ML2-Sriov-Multinode/a9a0268/ our MultiNode CI with everything 15:58:06 lennyb: awesome! 15:58:45 OK, out time is runnig out. thanks everybody for attending. See you next week. 15:59:02 thanks lennyb 15:59:19 #endmeeting