14:59:02 <kumarmn> #startmeeting trove
14:59:03 <openstack> Meeting started Wed Sep 27 14:59:02 2017 UTC and is due to finish in 60 minutes.  The chair is kumarmn. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:59:06 <openstack> The meeting name has been set to 'trove'
14:59:47 <kumarmn> pinging @amrith, @smatzek, @lukebrowning, @johnma, @slicknick
15:01:24 <kumarmn> #topic new ptl confirmed
15:01:27 <kumarmn> https://review.openstack.org/#/c/503786/
15:03:55 <kumarmn> #topic gate check failures
15:04:41 <smatzek> the gates are failing on multiple fronts.
15:05:10 <kumarmn> what are the nature of the failures and why now?
15:06:09 <smatzek> the py* gates are failing for two reasons.  1. about 2 weeks ago openstack changed the ostestr to 1.0.0 which changed its internals to use a project call stestr.  This drives changes into our tox.ini.  2. stestr has a bug where it doesn't work if you pass it --serial and a test blacklist
15:06:39 <smatzek> this review fixes and works around those:  https://review.openstack.org/#/c/507087/
15:07:17 <smatzek> I just got that mostly wrapped up yesterday evening and this morning.  Next up I've looked at the functional/scenario gate.
15:08:03 <smatzek> Historically these have been failing due to the 4x nested virtualization in the infra systems.  However, recently, I'm not sure when but in the last week or so, something else has changed and they are not even running the tests. They are failing during devstack setup.
15:08:44 <smatzek> amrith, could you take a few minutes and check out one of the failing logs and see if anything sticks out to you?
15:10:03 <smatzek> [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/ptyprocess-0.5.2.dist-info/top_level.txt is the error in devstack, failing somewhere in keystone or in glance before it outputs the glance line to the summary.
15:10:42 <smatzek> why that, and why now I can't answer.  I've looked at Cinder and Nova devstack logs and they look way different than ours.  I'm probably going to have to seek out help on this one.
15:11:29 <kumarmn> okay. will it help if we can run these locally?
15:11:37 <smatzek> no
15:11:50 <kumarmn> I believe lukebrowning is close to getting devstack/trovestack running locally
15:12:04 <smatzek> it would help with debug but I still expect it to fail the same way
15:12:26 <smatzek> once we get the tests running again I would like to pursue the strategy we talked about at PTG, of temporarily skipping/disabling the tests that always cause failures due to the nexting.  Likely upgrade and possibly backup.
15:12:49 <kumarmn> right, good idea.
15:13:22 <kumarmn> there are some other fixes queued up, that we could review/recheck as well once the gates are ok
15:13:27 <smatzek> that is a less than ideal thing to do, but we can mitigate the risk by not merging code that would directly impact those.  Regressions could still sneak through though, requirements updates that would mysteriously only hit those paths, etc
15:14:03 <smatzek> all the gate fixing is going to have to go in one review
15:14:40 <smatzek> brenttang, how much devstack experience do you have?  right now the gates are dead even before they run tests, in devstack initial setup
15:15:17 <smatzek> the community is also bringing zuul v3 on line and I"m not sure if that is at play here or not.
15:19:01 <brenttang> A little, been a while since digging thru the flow to add stuff to devstack though... I have a fix for the gate-trove-python failures (that keep forgetting to submit a weekend when it might pass), but haven't seen the devstack-gate failures yet
15:19:39 <smatzek> I have a review up for the python* failures and a bug open against stestr for it as well.
15:20:27 <smatzek> see ^, the devstack based gates are dead in the water and so even doing a recheck on a weekend won't help
15:27:54 <kumarmn> Any more discussion on the gate? If not I would like to move on to the next topic.
15:28:02 <smatzek> done
15:28:33 <kumarmn> #topic getting devstack/trovestack working on other archs
15:29:12 <kumarmn> I believe lukebrowning is working on this topic. He was making good progress. He had some issues where disk device naming was inconsistent.
15:29:49 <kumarmn> due to some forks in devstack, device names were showing up as sdb on one arch and trove requires them all to be vdb
15:31:27 <smatzek> device names show up as vd? on ppc64le clouds  It may only be nested where it's happening
15:44:43 <kumarmn> lukebrowning was saying that the problem was with devstack. and it was showing up as vdb on amd64 and sdb on ppc64le
15:50:44 <smatzek> it may be, I don't know the specifics but he pm-ed me about it as well.  I'm firing up a nested DBaaS instance right now to check its disks.
15:53:31 <kumarmn> #topic currency (OS and DB)
15:55:11 <smatzek> previous topic update.  On my nested VM on ppc64le I have vd? disks.  Whatever is causing sd? is not specific to ppc64le and must be a setting in devstack.  My OSA deployed Trove works fine
15:55:12 <kumarmn> brenttang, were you and lukebrowning able to synch up on your fixes?
15:56:26 <kumarmn> smatzek, you might have to get with lukebrowning separately. he showed up a section in devstack where ppc64le was using libvirt-vscsi
15:59:49 <kumarmn> thanks smatzek, brenttang, emtrevin for attending. signing off.
16:00:00 <kumarmn> #endmeeting