16:00:37 <xarses> #startmeeting fuel
16:00:37 <xarses> #chair xarses
16:00:37 <xarses> Todays Agenda:
16:00:37 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:37 <xarses> Who's here?
16:00:38 <openstack> Meeting started Thu Jun  2 16:00:37 2016 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:42 <openstack> The meeting name has been set to 'fuel'
16:00:43 <openstack> Current chairs: xarses
16:00:45 * xarses waves back
16:01:19 <xarses> o/
16:01:48 <iberezovskiy> hi
16:02:06 <gardlt> o/
16:02:10 <vsakharov> hi
16:02:20 <mwhahaha> hi.
16:03:11 <xarses> I just realized that the agenda wasn't updated for todays meeting
16:03:23 <xarses> please take a moment to move your topics =)
16:04:01 <xarses> #topic Telco Team Status (vsakharov)
16:04:07 <vsakharov> Hello!
16:04:10 <vsakharov> Our team continues fixing bugs:
16:04:10 <vsakharov> * Done from the last update - 4
16:04:10 <vsakharov> * In progress (in development and on review) - 12
16:04:10 <vsakharov> Our second activity is scoping features for 10.0.
16:04:10 <vsakharov> For now there are two features in work: SR-IOV+DPDK and DPDK+VXLAN. There are no any updates from last meeting yet.
16:04:11 <vsakharov> That's all.
16:04:28 <vsakharov> Any questions?
16:04:31 <degorenko> hi
16:04:50 <ezpz> o/
16:04:57 <xarses> thanks vsakharov
16:05:15 <xarses> #topic ubuntu_vlan_ha job doesn't actually test the patch set - do we need it? (iberezovskiy, degorenko)
16:05:33 <iberezovskiy> this week me and degorenko found out that ubuntu_vlan_ha job
16:05:36 <iberezovskiy> (which runs on each patch set to fuel-library) rebuilds fuel-library
16:05:40 <iberezovskiy> package on the patch but it doesn't actually update it on master node.
16:05:43 <iberezovskiy> as a result we have deployed environment without patch applied. and as
16:05:46 <iberezovskiy> result the patch isn't actually tested in ha configuration.
16:05:51 <iberezovskiy> e.g. patch https://review.openstack.org/#/c/321086/ (#1 patch set)
16:05:56 <iberezovskiy> #link https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.smoke_neutron/5369/
16:06:02 <iberezovskiy> #link https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/9387/
16:06:05 <iberezovskiy> so, our suggestion is the one of:
16:06:08 <iberezovskiy> 1. update this jobs to make patch set actually tested
16:06:11 <iberezovskiy> or 2. remove this jobs totally because it doesn't show what's wrong with the patch
16:06:14 <iberezovskiy> thoughts?
16:06:29 <degorenko> and we have same issue for puppet upstream jobs i guess
16:06:50 <xarses> what isn't applied the new fuel library?
16:07:07 <degorenko> yes, master node is not updated during tests
16:07:31 <degorenko> so, actually we don't have our patch for ubuntu_vlan_ha job
16:07:49 <mwhahaha> i'm pretty sure it is applied
16:07:59 <mwhahaha> if not it seems that we need to fix the job not remove it
16:08:09 <xarses> mwhahaha: +1
16:08:17 <degorenko> that's why we raised this topic
16:08:19 <mwhahaha> DEBUG: Executing command: ['/usr/bin/yum-builddep', '--installroot', '/var/lib/mock/epel-7-x86_64/root/', '--releasever', '7', '/var/lib/mock/epel-7-x86_64/root//builddir/build/SRPMS/fuel-library10.0-10.0.0-8493.2.gerrit321086.1.gitc62b3ac.src.rpm'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'LC_MESSAGES': 'C', 'PROMPT_COMMAND':
16:08:20 <mwhahaha> 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'HOSTNAME': 'mock'} and shell False
16:08:32 <xarses> but we patch it properly in smoke_neutron?
16:08:38 <degorenko> mwhahaha, just take a look on links shared by iberezovskiy
16:08:41 <iberezovskiy> this jobs builds package, but doesnt update the master node with them
16:08:44 <mwhahaha> or is it an artifact from no more docker
16:08:48 <degorenko> you will see - that one job has old fuel-library package
16:09:23 <mwhahaha> yea i'm trying to track it down in the output :D
16:09:28 <mwhahaha> anyway it should be fixed
16:09:43 <degorenko> bookwar_, mattymo are you here?
16:10:05 <iberezovskiy> if I'm not mistaken, mattymo said that's it's expected behavior, that's why we're asking
16:10:12 <degorenko> yes
16:10:24 <degorenko> and we were surprised to hear that
16:10:36 <xarses> seems that we should fix the job
16:10:44 <xarses> we still need to test neutron vlan
16:10:56 <iberezovskiy> bookwar_, ^
16:11:42 <iberezovskiy> +1 to fix of course, I wanna see every patch set tested in ha mode
16:12:09 <degorenko> also +1 for fix
16:12:19 <xarses> ok, lets plan on fixing it then, who's taking lead for that?
16:12:47 <degorenko> ci-team?
16:12:52 <mwhahaha> we need to verify that it's not actually being updated cause if it wasn't shouldn't tests always pass but they fail
16:13:17 <xarses> degorenko: can you file a bug?
16:13:29 <iberezovskiy> I can
16:13:49 <xarses> #action iberezovskiy will file a bug to fix ubuntu_vlan_ha job so that it tests the right fuel-library package
16:14:06 <xarses> #topic Response to BVT failures (mwhahaha)
16:14:15 <mwhahaha> I'd like to bring to attention the way we handle BVT failures. I believe that if we identify the patch that causes the BVT failure, the correct course of action is to revert the offending patch to unblock BVT.
16:14:15 <mwhahaha> Then we can merge whatever fix is required to put the functionality back in after BVT has been unblocked.
16:14:15 <mwhahaha> I'm raising this because BVT was broken yesterday (starting 6/1 @3:39PM Jenkins time) due to a change in the way astute handles DHCP checker responses, https://review.openstack.org/#/c/322201/.
16:14:15 <mwhahaha> Instead of merging the revert, https://review.openstack.org/#/c/324053/, we proceded to merge two changes to fix the DHCP checker, https://review.openstack.org/#/c/323352/ and https://review.openstack.org/#/c/323437/.
16:14:15 <mwhahaha> While I agree that this particular issue needed to get addressed, I do not believe that we should have spent so much time to address the original bug causing the BVT failure when a revert followed on by the DHCP checker update could have been done afterwards.
16:14:15 <mwhahaha> The main issue I have here is that we're so close to HCF for 9.0 and we've now lost over an entire day for fixes to land in master and be backported because we spent time trying to fix the way DHCP checker works rather than going back to the original state.
16:14:16 <mwhahaha> I was concerned that what if the DHCP checker update introduces more BVT blocking issues.
16:14:16 <mwhahaha> I believe in the future we should lower our risk by reverting the offending patch and then doing additional testing rather than force it in because BVT is blocked.
16:14:17 <mwhahaha> I thought this was the established workflow for such things, if not I would propose that this be written down and followed as the correct way to handle such failures in the future. Thanks.
16:15:45 <xarses> mwhahaha: it was my undersanding that revert first was the correct course of action
16:16:01 <mwhahaha> mine as well, but this was not followed.
16:16:12 <mwhahaha> additionally the offending original patch was merged while bvt was red
16:16:15 <iberezovskiy> as far as I know the correct flow was always to revert the patch
16:16:32 <xarses> in the rare case that it's a typo, it might make sense to fix it. but still we should revert back to passing first
16:16:51 <holser_> o/
16:17:01 <mwhahaha> hopefully bvt will clear up (it's still running), but we need to pay attention to this better
16:17:12 <mwhahaha> additionally when BVT is broken, it needs to be a priority to all cores
16:17:23 <xarses> mwhahaha: it should be clear on the wiki, if not feel free to update it and send out a mail about it
16:18:23 <mwhahaha> oh a side note that we need to find a solution for the wiki
16:18:38 <mwhahaha> my understanding is that it's going away at some point
16:18:46 <xarses> oh?
16:19:49 <xarses> can you take an action to figure out the next steps there?
16:19:53 <mwhahaha> https://www.mail-archive.com/openstack-dev%40lists.openstack.org/msg83371.html
16:21:24 <mwhahaha> i know puppet openstack is moving off of it as well. just thought i'd mention it. if more details surface i'll take an action item to come up with something else
16:21:31 <mwhahaha> anyway that's all i've got
16:21:35 <xarses> #topic open discuss
16:21:51 <xarses> any one have anything else to raise?
16:23:09 <xarses> ok, thanks everyone
16:23:19 <xarses> #endmeeting