16:00:09 <xarses> #startmeeting fuel
16:00:09 <openstack> Meeting started Thu Feb 18 16:00:09 2016 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <xarses> #chair xarses
16:00:09 <xarses> Todays Agenda:
16:00:09 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:09 <xarses> Who's here?
16:00:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:12 <angdraug> o/
16:00:13 <openstack> The meeting name has been set to 'fuel'
16:00:13 <fzhadaev> Hi!
16:00:14 <vkramskikh> hi
16:00:15 <openstack> Current chairs: xarses
16:00:15 <mwhahaha> hi
16:00:19 <rmoe> hi
16:00:20 <kozhukalov> hi
16:00:23 <igorbelikov> o/
16:00:25 <akasatkin> hi
16:00:31 <ashtokolov> o/
16:00:39 <HeOS> hi!
16:00:54 <asvechnikov> hi
16:00:56 <dguryanov> Hi!
16:01:20 <SheenaG> o/
16:01:28 <IvanBerezovskiy> hi
16:01:33 <mihgen> hi
16:01:43 <xarses> #topic action items from last week
16:01:44 <aglarendil> \o/
16:01:57 <xarses> maximov to coordinate with other teams and find a volunteer for puppet-openstack noop job
16:02:27 <evgenyl> hi
16:02:54 <xarses> xarses will bring up viability of version information (was version.yaml) for environment triage
16:02:54 <xarses> xarses to create thread over QEMU/KVM on ML
16:03:18 <xarses> neither are done, and I am quite annoyed I forgot these again =(
16:03:34 <xarses> on to the main event
16:03:44 <xarses> #topic UI Team status (vkramskikh)
16:03:47 <monester> hi all
16:03:50 <vkramskikh> Hi! Here is our status for 9.0 features:
16:03:51 <vkramskikh> 1) https://blueprints.launchpad.net/fuel/+spec/remove-vendor-code - the removal request has been merged; request to restore them in the downstream is in progress
16:03:51 <vkramskikh> 2) https://blueprints.launchpad.net/fuel/+spec/redesign-of-node-roles-panel - implemented
16:03:51 <vkramskikh> 3) https://blueprints.launchpad.net/fuel/+spec/network-requirements-popup - implemented, spec is still not merged, requires +2 from holser
16:03:51 <vkramskikh> 4) Ability to show all network groups - part of https://blueprints.launchpad.net/fuel/+spec/multirack-in-fuel-ui which wasn't done in 8.0 - implemented
16:03:54 <vkramskikh> 5) https://blueprints.launchpad.net/fuel/+spec/allow-choosing-nodes-for-provisioning-and-deployment - the design and implementation is in progress
16:03:57 <vkramskikh> 6) https://blueprints.launchpad.net/fuel/+spec/separate-fuel-ui-repo - some progress since the last week, currently trying to make a separate test runner using test runner tools: https://review.openstack.org/#/c/276814/
16:04:01 <vkramskikh> 7) NFV stuff - implementation for node attributes (for Nova and DPDK CPU pinning) is close to be implemented
16:04:04 <vkramskikh> Questions?
16:04:30 <xarses> are we in good shape for the NFV elements?
16:04:32 <yottatsa> o/
16:04:38 <vkramskikh> xarses: yes we are
16:05:01 <xarses> thanks
16:05:08 <yottatsa> xarses vkramskikh actually dpdk specs is still no merged
16:05:35 <vkramskikh> I know :) but I'm talking about UI stuff only in my report
16:05:39 <yottatsa> oh, thats about UI
16:05:42 <yottatsa> sorry, nwm
16:05:44 <xarses> yottatsa: link? or is this the item lower in the agenda?
16:05:50 <yottatsa> yep
16:06:00 <xarses> ok
16:06:09 <xarses> #topic Mixed Team Status (sslypushenko)
16:06:27 <sslypushenko> Mixed team is working on multipath featute https://blueprints.launchpad.net/fuel/+spec/support-for-multipath-disk. Spec is almost agreed.
16:06:31 <sslypushenko> https://review.openstack.org/#/c/276745 Development is in progress.
16:06:45 <sslypushenko> Questions?
16:07:01 <xarses> good progress to be done in this iteration?
16:07:33 <sslypushenko> What do mean about this iteration?
16:07:58 <xarses> before feature freeze anyway
16:08:19 <sslypushenko> yeap, I hope we have enough time
16:08:41 <sslypushenko> this not much things for development here
16:09:16 <angdraug> fyi iterations are marked in our release schedule:
16:09:19 <xarses> #topic Bugfix team status (dpyzhov)
16:09:19 <angdraug> #link https://wiki.openstack.org/wiki/Fuel/9.0_Release_Schedule
16:09:23 <dpyzhov> hi guys. for 9.0 we have 28 high priority bugs in python and 15 in library.
16:09:52 <dpyzhov> also we have 118 low priority bugs in python and 42 in library
16:10:10 <ashtokolov> angdraug, FF is in the middle of iteration
16:10:24 <dpyzhov> we have only 1 high priority bug  targeted to 8.0 and it will be fixed in plugins
16:10:49 <dpyzhov> so we are still good in library and more or less ok with high priority python bugs
16:11:17 <dpyzhov> plan is to move two guys from library bugs to python in order to fix the balance
16:11:18 <angdraug> ashtokolov: yes, that's when mitaka-3 is scheduled, and I have a feeling we're going to need that extra week
16:11:19 <aglarendil> dpyzhov: what about the bug with reboot networking sequence issues?
16:11:53 <dpyzhov> aglarendil: could you share the link?
16:12:11 <aglarendil> #link https://bugs.launchpad.net/fuel/+bug/1546947
16:12:11 <openstack> Launchpad bug 1546947 in Fuel for OpenStack 8.0.x "Corosync doesn't start on boot if balance-rr bonding is configured: 'No nodelist defined or our node is not in the nodelist'" [High,Confirmed] - Assigned to Dmitry Bilunov (dbilunov)
16:12:32 <dpyzhov> plan is to target it to maintenance update
16:12:55 <aglarendil> dpyzhov: I am not sure that we could afford it as we do not have clear analysis yet
16:14:04 <dpyzhov> aglarendil: analysis is still in progress, you are right
16:14:59 <dpyzhov> but we agreed to move it to mu1
16:15:09 <aglarendil> okay, it is your call, but it can backfire
16:15:38 <dpyzhov> any other questions?
16:16:11 <xarses> #topic Enhancements Team status (ashtokolov)
16:16:25 <ashtokolov> 1. Task Based Deployment with Astute v1 (production ready) - https://blueprints.launchpad.net/fuel/+spec/enable-task-based-deployment Engine has been merged, we are working on advanced settings support and global concurrency policy.
16:16:34 <ashtokolov> 2. Unlock "Setting" tab and rerun deployment Tasks in post-deployment - https://blueprints.launchpad.net/fuel/+spec/unlock-settings-tab in design, we are discussing how to implement it w/o ConfigDB
16:16:42 <ashtokolov> 3. Stop/Restart deployment w/o environmentment reset - https://blueprints.launchpad.net/fuel/+spec/graceful-stop-restart-deployment on review
16:16:49 <ashtokolov> 4. Deployment Tasks idempotence with Fuel Mixed Team https://blueprints.launchpad.net/fuel/+spec/granular-task-idempotency https://blueprints.launchpad.net/fuel/+spec/granular-task-ensurability WIP
16:16:54 <ashtokolov> 5. Fixing backend bugs to unlock Separated deployment/provisioning of nodes on UI https://blueprints.launchpad.net/fuel/+spec/specify-nodes-for-provisioning-and-deployment-in-ui 2/3 on review, 1/3 - merged
16:17:23 <ashtokolov> that's all from my side
16:18:34 <mihgen> for #1, do we see a drop in CI time required?
16:18:38 <bookwar> ashtokolov: looks like task based deployment increased the time used by smoke tests - is there a known issue regarding deployments on not as fast servers?
16:19:46 <ashtokolov> bookwar let's investigate it, when we changed ISO on Fuel CI?
16:20:04 <aglarendil> bookwar: let's look into it
16:20:20 <aglarendil> bookwar: do you have numbers for what is the difference?
16:20:28 <mihgen> action item for ashtokolov )
16:20:43 <mihgen> For https://blueprints.launchpad.net/fuel/+spec/unlock-settings-tab, are there estimates what would it take to implement required nailgun changes without configdb/solar?
16:20:57 <bookwar> ashtokolov: yesterday. it might be smth else affected the test runs i checked, so i am just asking maybe there some known issues
16:21:03 <bookwar> let's investigate then
16:21:17 <xarses> #action ashtokolov will look into increased deployment time in smoke tests
16:21:32 <bookwar> basically bvt went faster then smoke_neutron and that's surprised me
16:22:01 <xarses> #topic Reviewers needed for https://review.openstack.org/#/c/243340/ (aglarendil)
16:22:34 <aglarendil> this is about non-root spec author dnikishov1 seeking reviewers
16:22:36 <dnikishov1> Hello
16:22:58 <ashtokolov> mihgen we have to save deployment info w/o ConfigDB, it looks like short task, but we don't want to do it as a workaround
16:23:00 <dnikishov1> So basically the spec is ready and so is most of the code
16:23:09 <aglarendil> so far, dnikishov1 , I think holser had some questions about the scope of the feature
16:23:28 <dnikishov1> Ok
16:23:54 <aglarendil> is it about only deactivation of root login or is it about something more
16:24:01 <aglarendil> e.g. running all fuel pieces as non-root?
16:24:08 <aglarendil> e.g. nailgun-agent and mcollective agent?
16:24:39 <dnikishov1> This spec is about disabling root login into slave nodes
16:25:12 <dnikishov1> Running fuel as non root is planned for the future
16:25:36 <dnikishov1> But it isn't in scope at the moment
16:25:37 <ikalnitsky> dnikishov1: is it about running? i thought we run openstack services using non-root users
16:25:47 <ikalnitsky> oh, you meant fuel
16:26:47 <angdraug> commit is linked to the bp, bp description defines the scope quite clearly, what's in question here?
16:27:19 <aglarendil> angdraug: dnikishov1 is seeking for reviewers for a long time, so he kindly asked us to finally pay attention to the spec
16:27:34 <aglarendil> I think this is a broader question about SLA for specs review
16:28:05 <mihgen> dnikishov1: do you have capacity to finish all code required by FF? Or you'd need people to help with some parts, like QA?
16:29:16 <dnikishov1> I think I'd need qa folks to review the commits which are already on the review and give their opinions
16:29:30 <angdraug> I see a long list of reviews linked from bp, looks like it's code-complete and is only blocked by lack of reviewers
16:30:08 <dnikishov1> The only thing missing is ostf,  but I can finish it by eow
16:31:01 <ikalnitsky> dnikishov1: please, do not hesitate to ping folks in fuel-python irc channel
16:31:22 <dnikishov1> Ok
16:31:46 <mihgen> to finish with the spec, whom do we need?..
16:32:14 <ikalnitsky> mihgen: as usual - +2 from me and holser, i assume
16:32:41 <xarses> moving?
16:32:56 <mihgen> so then actions on you guys to help or find someone to help to move it forward.. ?)
16:33:06 <ikalnitsky> yep
16:33:36 <xarses> #topic NFV status (yottatsa)
16:33:55 <yottatsa> SR-IOV progress is good — library and agent code is almost done, we need reviewers #link https://review.openstack.org/#/q/topic:bp/support-sriov
16:34:03 <yottatsa> DPDK development is just started:
16:34:03 <yottatsa> * Spec is on final review, pls +2 #link https://review.openstack.org/273084
16:34:03 <yottatsa> * I’ve done two patches for library, third is on the way, we have agent, we have no nailgun
16:34:04 <yottatsa> * We’ve run into problem with OVS+DPDK package #link https://bugs.launchpad.net/fuel/+bug/1544354 , which is solved now
16:34:04 <yottatsa> And DPDK feature is depended on Huge, where no patches was done #link
16:34:04 <yottatsa> https://review.openstack.org/#/q/topic:bp/support-hugepages
16:34:04 <mihgen> xarses: can you make a formal action item for ikalnitsky, holser about that non-root access feature.. ?
16:34:05 <openstack> Launchpad bug 1544354 in Fuel for OpenStack "netconfig task fails when adding br-floating with ERROR: This system does not support "SSSE3"." [Critical,In progress] - Assigned to Aleksander Mogylchenko (amogylchenko)
16:34:19 <xarses> mihgen: sure
16:35:49 <xarses> #action ikalnitsky holser to follow up with dnikishov on the spec for non-root https://review.openstack.org/#/c/243340/
16:36:25 <xarses> yottatsa: do have any questions that we can address here?
16:36:29 <xarses> on the spec
16:37:31 <yottatsa> xarses spec is covered all the questions, except what to use to store constants: openstack.yaml or settings.yaml
16:38:57 <xarses> #topic Telco Team Status (fzhadaev)
16:39:03 <fzhadaev> Hi all.
16:39:03 <fzhadaev> For now, Telco team has three main activities for 9.0:
16:39:03 <fzhadaev> 1) Support NFV features:
16:39:03 <fzhadaev> 1.1) Huge pages [1]
16:39:03 <fzhadaev> 1.2) NUMA/CPU pinning [2]
16:39:03 <fzhadaev> Both specs were merged. 5 patches merged, 9 - on review
16:39:03 <fzhadaev> 2) Daemon Resource Allocation Control [3]
16:39:04 <fzhadaev> Spec is on review. 1 patch is on review.
16:39:04 <fzhadaev> 3) Removing Mirantis-specific code from fuel code
16:39:05 <fzhadaev> 3 patches are on review.
16:39:05 <fzhadaev> [1] https://blueprints.launchpad.net/fuel/+spec/support-hugepages
16:39:06 <fzhadaev> [2] https://blueprints.launchpad.net/fuel/+spec/support-numa-cpu-pinning
16:39:33 <fzhadaev> [2] https://blueprints.launchpad.net/fuel/+spec/support-numa-cpu-pinning
16:39:33 <fzhadaev> [3] https://blueprints.launchpad.net/fuel/+spec/cgroups
16:39:33 <fzhadaev> Any questions?
16:40:12 <mihgen> dpdk seems to be blocked by hugepages, any way to unblock?
16:40:46 <asvechnikov> it's already not blocked
16:41:13 <mihgen> quoting yottatsa: And DPDK feature is depended on Huge, where no patches was done #link
16:41:25 <asvechnikov> because specs for huge pages and numa/cpu pinning are merged
16:42:00 <asvechnikov> and dpdk may work without huge pages and cpu pinning
16:42:10 <alex_didenko> there's plenty of work related to DPDK that is not blocked by anything
16:42:13 <angdraug> yottatsa: so was it only blocked on specs, not on code?
16:42:54 <yottatsa> angdraug mihgen it is blocked on code: DPDK requires Hugepages to be configured on host
16:43:52 <xarses> since several teams are working on NFV, can you sync and provide a single update on NFV feature status?
16:44:04 <dklenov> to get DPDK performance gain HugePages are needed
16:44:34 <dklenov> although as alex_didenko mentioned, there is plenty of other work that can be done w/o huge pages
16:44:50 <dklenov> so it is end-to-end blocking
16:44:58 <dklenov> DPDK cannot be finished with HP
16:45:02 <dklenov> but can be started
16:45:19 <dklenov> *without HP
16:45:30 <yottatsa> dklenov Without Hugepages, openvwsitch with DPDK can not be launched at all.
16:45:48 <xarses> #action fzhadaev yottatsa to sync on NFV status and provide combined update
16:45:54 <xarses> ok, we need to move
16:45:55 <yottatsa> ok
16:45:58 <fzhadaev> ok
16:46:02 <xarses> #topic Switch OpenStack puppet modules to upstream master - https://review.openstack.org/#/c/279460/ (IvanBerezovskiy)
16:46:06 <IvanBerezovskiy> As many of you knows we've switched to Mitaka ISO yesterday,
16:46:06 <IvanBerezovskiy> Now we are ready to be fully aligned with upstream puppet modules
16:46:11 <IvanBerezovskiy> #link https://review.openstack.org/#/c/279460/4
16:46:11 <IvanBerezovskiy> As soons as this patch merged we can enable Fuel CI under upstream modules (igorbelikov)
16:46:11 <IvanBerezovskiy> I guess it will be non-voting for now, but it's a big step to make it voting soon.
16:46:11 <IvanBerezovskiy> If we merge this patch now we should start to look at each patch to upstream modules first of all to analize their affection on fuel-library.
16:46:11 <IvanBerezovskiy> We should be ready to react immediately on each problem which can come from modules commits.
16:46:11 <IvanBerezovskiy> My suggestion is to organize a duty under upstream modules. Patches to modules should be monitored by Fuel-library and MOS puppet teams.
16:46:11 <IvanBerezovskiy> Also, as addition to CI I suggest to add an ability to test parallel commits to fuel-library and to puppet-modules (e.g. using depends-on in commit message) - wip (igorbelikov)
16:46:12 <IvanBerezovskiy> Are we ready for this? :)
16:47:01 <igorbelikov> We're ready to fire up Fuel CI for upstream modules tommorrow
16:48:23 <angdraug> igorbelikov: what about depends-on -- is that included?
16:49:30 <mihgen> +1 for duty..
16:49:31 <igorbelikov> angdraug: Not sure about tomorrow, but this will follow shortly
16:49:59 <angdraug> how do we configure the ci duty?
16:50:40 <IvanBerezovskiy> at the beginning I think we should have at least to people from each team
16:50:46 <angdraug> first of all, we need a gerrit dash for upstream commits with -1 from fuel ci
16:50:47 <IvanBerezovskiy> s/to/two
16:51:32 <igorbelikov> angdraug: Fuel CI won't be able to leave -1 yet
16:51:37 <IvanBerezovskiy> angdraug: are we allowed to make it woting?
16:51:46 <angdraug> IvanBerezovskiy: do you mean 4 people should be on duty on any given day?
16:51:46 <IvanBerezovskiy> s/woting/voting
16:51:51 <bookwar> angdraug: if we just start posting -1 on all commits in puppet upstream - people will get angry very fast
16:52:06 <IvanBerezovskiy> at the beginning - yes
16:52:10 <angdraug> ok, s/-1/failure/
16:52:21 <bookwar> i'd rather set it in non voting silent mode for some time to see the stats
16:52:32 <degorenko> ^ agree
16:52:35 <IvanBerezovskiy> +1
16:52:48 <igorbelikov> and without leaving actual score I don't think it's possible to configure a gerrit dash for that
16:53:01 <angdraug> if we set it to silent, duty engineers will have to watch jenkins instead of a gerrit dashboard
16:53:38 <igorbelikov> angdraug: email notifications to the rescue?
16:53:45 <bookwar> (don't see a big problem with that)
16:53:55 <angdraug> no, not a big problem
16:54:05 <bookwar> we have rss
16:54:06 <angdraug> just thinking about how we estimate amount of work that the duty would take
16:54:18 <bookwar> and we can subscribe irc bot to it btw
16:54:50 <angdraug> can irc bot post failures to #fuel-tracker?
16:54:55 <mwhahaha> do we have that bot subscribed to bvt failures yet? :D
16:55:06 <angdraug> mwhahaha: +1
16:55:12 <bookwar> mwhahaha: not yet, but we should
16:55:21 <mihgen> not #fuel-tracker please, but #fuel-bugs
16:55:27 <mihgen> tracker tracks all patchsets
16:55:36 <mihgen> it's hundreds of messages per day
16:55:45 <angdraug> why #fuel-bugs?
16:55:55 <angdraug> and not e.g. #fuel-ci?
16:56:13 <angdraug> or just #fuel-infra?
16:56:16 <bookwar> fuel-dev :)
16:56:18 <IvanBerezovskiy> why can't we use #fuel-dev in case of such failures?
16:56:21 <mihgen> whatever, but if you mix something in to fuel-tracker - you won't notice anything
16:56:45 <angdraug> I'd rather not spam #fuel-dev
16:56:51 <mihgen> for BVT failures on ci.fuel-infra.org, I'd support #fuel-dev
16:56:56 <mihgen> it happens rarely
16:57:07 <mihgen> and makes sense to attract attention of everyone
16:57:14 <angdraug> commits to puppet-openstack happen very often
16:57:24 <xarses> 3 min
16:57:24 <aglarendil> folks, can we switch back to task based deployment speed? we went through CI logs and did not find any regression. for example new CI tests for smoke_neutron have more than 15-25 % increase
16:57:37 <ashtokolov> bookwar about your question: it looks like master with task-based is faster than 8.0 w/o task-based:
16:57:49 <xarses> aglarendil: there is an action to investigate it
16:57:50 <ashtokolov> Task finished. Took 2192.38510299 seconds. - for master vs Task finished. Took 2824.64024019 seconds. - for 8.0
16:57:57 <angdraug> ok, lets wrap up
16:58:02 <ashtokolov> https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.smoke_neutron/2218/consoleFull vs https://ci.fuel-infra.org/job/8.0.fuel-library.pkgs.ubuntu.smoke_neutron/298/consoleFull
16:58:15 <angdraug> lets merge the commit to switch to upstream master no later than tomorrow
16:58:32 <angdraug> and launch fuel ci for upstream in silent mode also tomorrow
16:58:54 <angdraug> igorbelikov will monitor the failures for the first couple of days
16:58:55 <IvanBerezovskiy> we can merge right now, it's already reviewed
16:59:08 <angdraug> yes, now would be better
16:59:33 <angdraug> IvanBerezovskiy: please assign igorbelikov in monitoring the failures until EOD Monday
16:59:45 <angdraug> after that we'll have some data to estimate and structure the duty effort
16:59:47 <IvanBerezovskiy> sure
16:59:57 <xarses> thanks all
17:00:01 <xarses> #endmeeting